Documentation

Python packages / DCP tools

Changelog

v1.0.1 (2026-10-13)

  • Fixed a fresh install failing to import dcp_tools.gcp_utilities, which broke the dcp-tools CLI and run_ingestion_workflow. datacommons-admin 1.1.3 moved IngestionJobClient to a new module, and 1.1.6 removed the job_name argument dcp-tools passes. The dependency is now capped at datacommons-admin<1.1.3 until dcp-tools supports the new client.

v1.0.0 (2026-10-13)

First stable release under the name dcp-tools. The API is unchanged since v1.0.0a5, and from this release on, breaking changes wait for a new major version. To upgrade from bblocks-datacommons-tools 0.1.1, follow the migration notes in the v1.0.0a1 to v1.0.0a5 entries below, which describe each change in detail.

  • Breaking: renamed the package from bblocks-datacommons-tools to dcp-tools, the import path to dcp_tools and the CLI to dcp-tools. The old distribution installs dcp-tools and re-exports it with a DeprecationWarning.
  • Breaking: ingestion runs on the Data Commons Platform. run_data_load is now run_ingestion_workflow, the ingestion settings are renamed, and the Cloud SQL and redeploy settings and commands are removed.
  • Breaking: the Python API is snake_case throughout, the node models drop their MCF prefix, and add_source and add_provenance take separate dcid and name parameters.
  • Breaking: export_all writes the complete bundle and empties the target directory first.
  • Breaking: the variable-per-column (implicit schema) input format and the redeploy command are removed.
  • Added add_node, add_mcf_file, Source and Provenance nodes with licence, refresh and coverage metadata, multi-entity observations through custom:<name> dimensions, and constraint_properties on StatVars.
  • Minimum supported Python is 3.13.

v1.0.0a5 (2026-09-10)

  • Breaking: license_type and curator are now DCID references, written as licenseType: dcid:<token> instead of a quoted string. Bare tokens passed to add_source / add_provenance are minted; SourceNode / ProvenanceNode still require the dcid: prefix.
  • Breaking: renamed the ingestion settings to match the DCP Terraform outputs: LOAD_JOB_NAME -> INGESTION_PREP_JOB_NAME, LOAD_JOB_REGION -> GCP_REGION, and LOAD_JOB_SERVICE_ACCOUNT -> INGESTION_SERVICE_ACCOUNT. The old names are no longer accepted.
  • Breaking: renamed run_data_load to run_ingestion_workflow, and the dataload CLI command to ingest. imports is now keyword-only, so run_data_load(settings, "climateFinance") becomes run_ingestion_workflow(settings, imports="climateFinance").
  • Breaking: export_all now empties the target directory before writing.
  • Added license_type and license_attribution to add_source, and source_data_url and license_attribution to add_provenance.

v1.0.0a4 (2026-09-07)

  • add_provenance now warns instead of raising when source isn't registered in the bundle, so a Provenance can reference a Source that already exists in base Data Commons.

v1.0.0a3 (2026-08-19)

  • Added add_node(dcid=..., name=..., type_of=...) to CustomDataManager, for MCF nodes with a caller-supplied typeOf e.g. entity instances (e.g. an organisation) or constraint-value classes (e.g. Commitment) the typed builders don't cover.
  • gcs_input_folder_path (GCS_INPUT_FOLDER_PATH) is now optional and defaults to ingestion/input, the DCP Terraform default.
  • Fixed KGSettings rejecting its snake_case field names; KGSettings(local_path=...) now works alongside KGSettings(LOCAL_PATH=...).

v1.0.0a2 (2026-08-10)

  • Breaking: ProvenanceNode.source_link is renamed to source, so a provenance's parent source now serializes as source: dcid:source/<name> (was sourceLink:), the property name Data Commons ingestion preprocessing expects.
  • Added add_mcf_file(file_name, provenance=...) to declare an MCF file in config.json's inputFiles. add_mcf_file("*.mcf", provenance=...) covers them all in a single-provenance import; declare each file separately when an import has more than one provenance.
  • Added constraint_properties to StatVarNode and as a parameter on add_variable_to_mcf, for the DCID(s) of properties that constrain the StatisticalVariable. Bare tokens are minted to dcid:<token>.
  • Breaking: datacommons-admin was bumped to >=1.1.1 and now required a ingestion_workflow_name (INGESTION_WORKFLOW_NAME) setting.
  • Fixed run_data_load and the dataload/pipeline commands not loading every import when none are named.

v1.0.0a1 (2026-07-27)

  • First alpha of the dcp-tools package, the first release under the new name.
  • Renamed the package from bblocks-datacommons-tools to dcp-tools. The import path is now dcp_tools (was bblocks.datacommons_tools). Installing the old bblocks-datacommons-tools distribution now pulls in dcp-tools and re-exports it with a DeprecationWarning; update imports to dcp_tools at your convenience.
  • Minimum supported Python is now 3.13 (was 3.11).
  • Packaging now follows the bblocks-projects copier template (ruff lint preset, ty type checking, pre-commit hooks, PyPI trusted publishing).
  • Breaking: removed support for the second, implicit import mechanism — add_implicit_schema_file, add_variable_to_config, and the ImplicitSchemaFile / Variable model classes are gone. ObservationProperties is retained but repurposed: it's no longer nested under a variable definition, and now carries the file-level constant observation properties on InputFile; it also accepts custom keys (extra="allow", was extra="forbid"), and the four standard fields are unchanged. Loading a legacy variablePerColumn config now raises ValueError. Migrate by using add_input_file with column_mappings; see Data Commons custom data docs.
  • Breaking: with a single import format left, the input-file API drops the "explicit" qualifier: ExplicitSchemaFile is now InputFile and add_explicit_schema_file is now add_input_file. The arguments are the same ones (in the snake_case spelling below) and the generated config.json is unchanged.
  • Breaking: Config.input_files is now list[InputFile] (was a dict keyed by file name). InputFile also rejects unknown keys now (extra="forbid").
  • Breaking: export_mfc_file is now spelled export_mcf_file, and csv_metadata_to_mfc_file is now csv_metadata_to_mcf_file.
  • Breaking: removed the entity key on ColumnMappings. Use observationAbout for single-entity data or custom:<name> for multi-entity dimensions.
  • Added support for multi-entity observations using custom dimensions: declare each dimension with custom:<name> in ColumnMappings and a matching dcid:<name> in the StatVar's observation_properties.
  • Single-entity StatVars no longer emit observationProperties by default.
  • Fixed rename_variable, which left the Nodes lookup index keyed by the old name, so a following variable removal raised "not found" for the renamed node and still resolved the old name. The rename now goes through a new Nodes.rename, which keeps the index in step.
  • Fixed export_data, export_mcf_file, and export_vertical_specs raising OSError when the target file name nested in a subdirectory that did not yet exist (for example an add_input_file name such as "sub/gdp.csv"). Each now creates the parent directory before writing.
  • Fixed add_variable_to_mcf, which did not normalize a bare dcid to dcid:<token> like the schema-node builders do, so a bare token raised a ValidationError there while working everywhere else. Its population_type, measured_property, measurement_qualifier and measurement_denominator arguments had the same gap and now accept bare tokens too.
  • Fixed DcidOrListDcid, which used a PlainValidator that replaced the wrapped Dcid schema rather than running before it, so the dcid: prefix check never ran. Any string, prefixed or not, passed through and landed in the MCF verbatim. This affected type_of on every node, relevant_variable, observation_properties and member on StatVar nodes, and included_in, sub_class_of, domain_includes, range_includes and sub_property_of on the schema-node builders. These fields now normalize a bare token to dcid:<token> and reject anything empty or whitespace-bearing, and a non-string value now raises rather than being accepted silently. Breaking for anyone passing a bare token to one of these fields today and relying on it staying bare.
  • Breaking: member_of on a StatVar now has to be a real group dcid. It had the same bypass, so any string reached the MCF verbatim; it now requires a g/ segment, so one/g/economy is minted to dcid:one/g/economy and dcid:economy is rejected. Check the values you pass to add_variable_to_mcf and the memberOf column of your StatVar CSVs.
  • Breaking: relevant_variable on a Topic node is now plain DcidOrListDcid. Its type used to combine the fixed variant with two unfixed ones, so a value the fixed one rejected still got through the others. It accepts the same StatVar, group and topic dcids as before, and now rejects the malformed values that used to slip past.
  • Breaking: nodes now validate on assignment, not only on construction. Setting a field to an invalid value, for example node.member_of = "garbage" or renaming a node to a token with no dcid: prefix, raises instead of quietly writing it to the MCF file.
  • Breaking: build_stat_var_groups_from_strings is replaced by resolve_group_paths. It used member_of to hold an unresolved group path such as "Economic/Employment" until it was overwritten, which is what blocked validating the field. Group paths are now resolved before the nodes are built. If you call it directly, use csv_metadata_to_nodes(..., parse_groups=True, group_namespace=...) instead. add_variables_to_mcf_from_csv is unchanged.
  • Fixed add_variables_to_mcf_from_csv(parse_groups=True) raising AttributeError when the CSV had no memberOf column or a row left it blank. The missing column now raises a clear ValueError, and a blank value leaves that node's member_of unset. A group path holding a whitespace-only segment, such as a stray tab between two slashes, also used to mint a group with an empty name; those segments are now dropped.
  • Breaking: a Topic node's dcid now has to carry the dcid: prefix. The old check looked for a topic/ segment anywhere in the string and never required the prefix, so topic/x validated and was written to the MCF unprefixed. Add dcid: to the Node column of any topic CSV. Data Commons rejects unprefixed node ids on load, so those files were not loading correctly to begin with.
  • Breaking: re-pointed the data load flow at the DCP prep job, using the IngestionJobClient. run_data_load now triggers the prep job. The redeploy CLI command and the redeploy_service and redeploy_cloud_run_service functions are removed; the service restart is now owned by the ingestion workflow. Load-job settings are renamed: CLOUD_RUN_JOB_NAME → LOAD_JOB_NAME, CLOUD_JOB_REGION → LOAD_JOB_REGION, plus a new optional LOAD_JOB_SERVICE_ACCOUNT that sets which service account the load job impersonates (when unset, the caller's credentials are used). The unused Cloud SQL and Cloud Run service settings are removed too: CLOUD_SQL_DB_NAME, CLOUD_SQL_REGION, CLOUD_SERVICE_REGION, CLOUD_RUN_SERVICE_NAME, and DATACOMMONS_SERVICE_IMAGE.
  • Added an imports argument on run_data_load and a matching --imports flag on the dataload CLI command, to trigger a load of specific imports rather than all imports.
  • Breaking: renamed the node model classes to drop the MCF qualifier — MCFNode → Node, MCFNodes → Nodes, and every subclass. MCF is now treated as one serialization of a Data Commons graph node.
  • Breaking: a node's identifier is now a single dcid property — the duplicate Node/dcid pair is collapsed. Access it via node.dcid; it still serializes to the Node: line in MCF.
  • Breaking: Node.mcf is now the Node.to_mcf() method.
  • Breaking: the Python API is snake_case throughout — node/Config attributes, builder keyword arguments and the set_* methods. Serialisation still uses camelCase using pydantic's alias_generator=to_camel.
  • Breaking: export_all writes and overwrites the complete bundle, and no longer takes mcf_file_names or override. It writes config.json, the data CSVs, vertical_specs.json (when specs were added), and every MCF file you've added nodes to, overwriting what's already there. Pass nothing for the full bundle; use export_mcf_file for a single file.
  • Breaking: override is renamed to overwrite on export_mcf_file, Nodes.export_to_mcf_file and csv_metadata_to_mcf_file, and now defaults to overwriting.
  • Breaking: csv2mcf replaces --override with --append. It overwrites the output file by default; pass --append to add to an existing file instead.
  • Breaking: add_source and add_provenance take separate dcid (identifier, minted with the slug prefix), optional name (label), and description parameters. Previously name was the identifier and no name property was set — pass the identifier as dcid= now.
  • Breaking: CustomDataManager.remove_indicator is renamed to remove_variable (parameter indicator_id → dcid), matching the variable terminology used elsewhere.

v0.1.1 (2026-02-19)

  • Added a --sync flag to the upload and pipeline CLI commands, which deletes remote blobs that no longer have a local counterpart after uploading.
  • Added sync_directory_to_gcs in storage.py, which composes upload and stale blob cleanup.
  • Added a sync parameter on upload_to_cloud_storage.

v0.1.0 (2026-02-13)

  • Initial release of the dcp-tools package for external preview and testing

v0.0.9 (2025-09-17)

  • Added new configuration options, including set_customIdNamespace, set_customSvgPrefix, set_defaultCustomRootStatVarGroupName and set_svHierarchyPropsBlocklist.

v0.0.8 (2025-09-03)

  • Removed white space between quoted items to defend against a bug with data loading on the DC side.

v0.0.7 (2025-08-27)

  • Handle linebreaks and trailing spaces by removing them. This prevents errors when serialising to MCF which could (quietly) break the data loading job.

v0.0.6 (2025-08-14)

  • Node name is now an optional attribute. This enables easily appending data to existing Base DC Nodes.

v0.0.5 (2025-08-14)

  • Nodes can now contain a single dcid or a list of dcids

v0.0.4 (2025-07-22)

  • Improved how groups strings are transformed to camelCase by dealing with additional special characters
  • Removed option to override input and output folders on the data load job.

v0.0.3 (2025-07-18)

  • Fixes two bugs related to MCF files. It now enforces the dcid: prefix for Node and automatically trims spaces between dcid: and the start of the id string.

v0.0.2 (2025-07-07)

  • Minor update to documentation and release to PyPI

v0.0.1 (2025-07-07)

  • Initial release of the dcp-tools

View as markdown