The question this page answers

CustomDataManager.export_all writes a config.json and one or more .mcf files side by side. Why two formats instead of one, and how does a CSV column end up as a predicate on a Data Commons observation? For the mechanics of calling add_input_file, add_variable_to_mcf, and friends, see Preparing data.

The short answer

config.json and .mcf do different jobs. config.json tells the importer how to read your CSVs: which column holds the variable, which holds the date, which holds the entity a row is about. MCF tells the importer what new graph nodes exist that aren't already in the base Data Commons graph: your StatVars, your groups, your sources and provenances, and, for genuinely new concepts, custom entity types, event types, properties, units, and measurement methods. Every input file dcp-tools produces is variable-per-row (one observation per row). That's the only layout the current importer accepts, so there's no format to choose. The columnMappings block in config.json uses dcid:observationAbout for a row about one entity, and custom:<name> for a row about several. That split exists because a single dcid reference can't represent a bilateral relationship.

What a Data Commons Platform deployment needs

A Data Commons Platform instance serves an organisation's own data next to the base Data Commons knowledge graph. The platform's ingestion workflow reads a declarative bundle (config, CSVs, MCF) and turns it into graph nodes and observations. dcp-tools builds that bundle. It doesn't talk to a database and it doesn't run the ingestion workflow itself. CustomDataManager assembles the files, and run_ingestion_workflow hands them to the platform.

The full shape of the bundle format, including fields not covered here, is documented at docs.datacommons.org/custom_dc/custom_data.html. The part dcp-tools exists to make easier to construct is narrower: how a CSV column becomes a predicate, and why the schema-defining nodes live in a separate file format.

The reasoning

config.json declares column roles, not data

Every entry in config.json's inputFiles list points at either a CSV of observations or an MCF file of node definitions. A CSV entry carries a columnMappings block. The block doesn't hold data. It holds a role assignment, naming which column in this specific CSV plays which part in every observation the importer will build from it. An MCF entry carries no mappings, since it has no columns to assign. It carries only the provenance its nodes belong to.

from dcp_tools import CustomDataManager
import pandas as pd

manager = CustomDataManager()
manager.add_source(dcid="ONEData", url="https://data.one.org")
manager.add_provenance(
    dcid="ONEClimateFinance",
    url="https://datacommons.one.org/data/climate-finance-files",
    source="ONEData",
)

commitments = pd.DataFrame({
    "country": ["Germany", "France", "Japan"],
    "year": [2023, 2023, 2023],
    "variable": ["climateFinanceProvidedCommitments"] * 3,
    "value": [2_100, 1_450, 3_200],
})

manager.add_input_file(
    "climate-finance/commitments.csv",
    provenance="ONEClimateFinance",
    data=commitments,
    column_mappings={
      "observationAbout": "country",
      "date": "year",
      "variable": "variable",
      "value": "value",
    },
)

The resulting config.json entry:

{
    "filename": "climate-finance/commitments.csv",
    "provenance": "dcid:provenance/ONEClimateFinance",
    "columnMappings": {
        "dcid:variableMeasured": "variable",
        "dcid:observationDate": "year",
        "dcid:value": "value",
        "dcid:observationAbout": "country"
    },
    "format": "variablePerRow"
}

Each key in columnMappings is a role the importer already understands (observationAbout, date, variable, value, unit, scalingFactor, measurementMethod, observationPeriod). The string you supply is the name of the column that fills that role in this file. add_input_file accepts the short names (variable, date, and so on). config.json stores all but scalingFactor under their dcid:-prefixed aliases because that's the predicate form the importer reads them as. Two files can map the same role to differently named columns (one CSV might call its date column year, another fiscal_year) because the mapping is declared per file.

What columnMappings cannot do is invent a column that isn't there, or tell the importer what a value like climateFinanceProvidedCommitments means. That's the MCF file's job. It defines the climateFinanceProvidedCommitments StatVar node once, and every row across every CSV that references it by that dcid inherits the definition.

Why bilateral data needs a different mechanism from observationAbout

observationAbout takes exactly one column, because it fills exactly one predicate: "this observation is about this entity." That works for the commitments file above, one country per row. It stops working the moment a row describes a relationship between two entities instead of a fact about one.

Take provider-to-recipient climate finance flows. Each row already has two country columns, neither of which is uniquely "the" entity the row is about. Continuing with the same manager from above, register a StatVar that names the two dimensions, then register the file:

flows = pd.DataFrame({
    "provider": ["Germany", "France", "Japan"],
    "recipient": ["Kenya", "Senegal", "Vietnam"],
    "year": [2023, 2023, 2023],
    "variable": ["climateFinanceProvidedFlows"] * 3,
    "value": [340, 210, 480],
})

manager.add_variable_to_mcf(
    dcid="climateFinanceProvidedFlows",
    name="Climate finance provided (bilateral flow)",
    stat_type="dcid:measuredValue",
    observation_properties=["dcid:providerCountry", "dcid:recipientCountry"],
)

manager.add_input_file(
    "climate-finance/flows.csv",
    provenance="ONEClimateFinance",
    data=flows,
    column_mappings={
      "variable": "variable",
      "date": "year",
      "value": "value",
      "custom:providerCountry": "provider",
      "custom:recipientCountry": "recipient",
    },
)

A row with custom:<name> keys needs no observationAbout, as the flows file above shows. Each custom:<name> key becomes a dcid:<name> property reference on the observation. The StatVar's own observationProperties list tells the importer which properties to expect and validate against. Here that's providerCountry and recipientCountry, declared once on the StatVar rather than re-derived per file. A single-entity StatVar doesn't need this list at all. It's specific to the multi-entity case, because the importer has no other way to know that a row with two country columns is deliberately structured that way and not missing an observationAbout.

Two dcid rules, and why they aren't the same rule

Names you pass to add_source and add_provenance go through a different minting rule than names you pass to add_entity_type, add_property, add_unit, or Node/populationType/etc. on add_variable_to_mcf. Both rules turn a bare token into a dcid:-prefixed one, but only one of them inserts a namespace segment:

• add_source(dcid="ONEData", ...) mints dcid:source/ONEData.
• add_property(dcid="providerCountry", ...) mints dcid:providerCountry, no property/ segment.

The difference tracks what's being identified. A source or provenance name is usually a short organisational label: "ONEData", "WEO", "IMF". Two different datasets, even two teams at the same organisation, can plausibly both register a source called "WorldBank". Nesting the name under source/ or provenance/ means that collision has to happen inside a much smaller namespace (one organisation's list of sources), and it keeps a source's identity separate from a provenance's even if someone reuses the same word for both.

A custom entity type, property, unit, or measurement method doesn't have that problem, because defining one is the act of claiming a unique identifier. When you call add_property(dcid="providerCountry", ...), you declare that dcid:providerCountry is now a Property node. The typeOf: dcid:Property on the node itself is what disambiguates it from a same-named source or StatVar. That's why these builders use ensure_dcid, which only adds the dcid: prefix, while sources and provenances use mint_dcid, which adds a namespace segment too.

Consequences

columnMappings is always required on a CSV entry, because it is how the importer finds your columns.

Related

• Preparing data walks through CustomDataManager end to end: sources, provenances, variables, input files, and export.
• CLI tools covers csv2mcf, for building MCF node files straight from a CSV without going through CustomDataManager.