Build with the DFO / API

Get a dataset

Use POST /api/v1/datasets to create CSV, XLSX, or Parquet data from a resolved spec.

Before you start

Start with a definite spec from How to resolve a question. Set the development API URL.

export DFO_API=https://developmentfinance.org

This page stores the Search response in resolve.json, the response to the create request in response.json, and the completed manifest in manifest.json.

Steps

Resolve the question and save the result.

curl -fsS -X POST "$DFO_API/api/v1/search/resolve" \
  -H "Content-Type: application/json" \
  -d '{"input":{"kind":"raw_text","query":"ODA disbursements from Germany to Nigeria for basic health care"}}' \
  > resolve.json

jq -e '.kind == "resolution" and .parts[0].status == "definite"' resolve.json > /dev/null || {
  echo "Search did not return a definite interpretation"
  exit 1
}

Build the request body from the Search response's spec_id and definition, and save it as mint.json.
```
jq '{
  schema_version: 1,
  input: {
    kind: "spec_id",
    spec_id: .parts[0].spec.spec_id,
    definition: .parts[0].spec.definition
  },
  options: {format: "parquet", bundle: true}
}' resolve.json > mint.json

curl -fsS -X POST "$DFO_API/api/v1/datasets" \
  -H "Content-Type: application/json" \
  -d @mint.json > response.json
```
schema_version is 1. Supported formats are csv, xlsx, and parquet. Check GET /api/v1/datasets/capabilities for the active formats. bundle defaults to true and includes the data file, README.md, SOURCES.md, CITATION.cff, and datapackage.json. data/codes.csv is present when the data contains codes.

Supply definition when the Datasets API has not registered the spec_id. Send the Search definition unchanged so the API can confirm that it reproduces the identifier.
Check whether the request returned a manifest or an operation.
```
jq 'if .operation then
  {operation: .operation}
else
  {
    content_version,
    coverage,
    artifacts: [.artifacts[] | {rendering, media_type, url}]
  }
end' response.json
```
A request within the synchronous budget returns 200 and a manifest. A larger request returns 202 and an operation object. The initial operation response contains operation.poll and retry_after_ms.

Save a completed manifest from the response.

Run this Python block after steps 1 to 3. It writes manifest.json for the remaining steps.

import json
import os
import time
from pathlib import Path

import httpx

result = json.loads(Path("response.json").read_text())
with httpx.Client(base_url=os.environ["DFO_API"], timeout=120) as client:
    if "operation" in result:
        poll_url = result["operation"]["poll"]
        wait_seconds = result["operation"].get("retry_after_ms", 1500) / 1000
        while True:
            time.sleep(wait_seconds)
            response = client.get(poll_url)
            response.raise_for_status()
            result = response.json()
            if result["operation"]["status"] == "done":
                manifest = result["manifest"]
                break
    else:
        manifest = result
Path("manifest.json").write_text(json.dumps(manifest))

Operations can be pending, running, done, or failed. Poll responses for pending and running work omit retry_after_ms. A done response omits poll and places the manifest beside operation. A failed poll returns 500 operation_failed, so raise_for_status() stops the script.

Select the data artifact by rendering, then download it from its link.
```
import json
import os
from pathlib import Path

import httpx

manifest = json.loads(Path("manifest.json").read_text())

artifact = next(
    item
    for item in manifest["artifacts"]
    if item["rendering"].get("format") == "parquet"
    and item["rendering"].get("bundle") is True
)
response = httpx.get(os.environ["DFO_API"] + artifact["url"], timeout=120, follow_redirects=True)
response.raise_for_status()
Path("bundle.zip").write_bytes(response.content)
```
Each url is a download link relative to the API host, /api/datasets/{spec_id}/{content_version}/files/{artifact_digest}. The path carries no version prefix and works as returned. The address of the link is stable. Once the file behind it passes retention, the link renews it where it can and otherwise answers 410 artifact_expired (see Retention and capacity). A request to it answers 302 with a storage URL signed for five minutes, so follow redirects (curl -L, or follow_redirects=True in httpx). Once the version is taken down, the link answers 410 snapshot_revoked.

A manifest can include several artifacts for one content_version, including a Parquet series-statistics file. Select the requested format and bundle fields inside rendering. Bundle renderings also carry a rendition token that can change when the archive layout changes. The statistics artifact has one row per series and is separate from the observation table.

datapackage.json lists each data resource and its path inside a bundle. Read it before opening a data file.

Retrieve a stored snapshot to list a version's files again.

spec_id="$(jq -r '.definition.spec_id' manifest.json)"
content_version="$(jq -r '.content_version' manifest.json)"
curl -fsS "$DFO_API/api/v1/datasets/$spec_id/$content_version" > snapshot.json
jq '{content_version, artifacts: [.artifacts[] | {rendering, url}]}' snapshot.json

A snapshot reads the stored version and lists the same download links while its artifacts remain servable.

Verify it worked

A manifest has a content_version, coverage, and an artifact whose rendering matches the requested format. The selected link downloads a file whose SHA-256 digest equals its artifact_digest.

Retention and capacity

Generated artifacts are working-tier by default and expire after the retention period published by GET /api/v1/datasets/capabilities (30 days), measured from stored_at. A pinned artifact is kept indefinitely. After a working-tier artifact expires, its download link renders the file again from the version's stored frame while that frame is present, and otherwise answers 410 artifact_expired. A snapshot can return 410 artifact_expired when no listable artifact location is servable, or 410 snapshot_revoked for a revoked version.

Terminal operations are guaranteed pollable for the retention period in the Datasets capabilities response. An older operation can still be present. 404 operation_not_found means the API does not know that operation.

GET /api/v1/datasets/capabilities publishes the synchronous budgets and background ceilings. Requests over either synchronous budget can run in the background if they fit within both background ceilings. A request beyond a background ceiling returns 422 over_ceiling.

curl -fsS "$DFO_API/api/v1/datasets/capabilities" \
  | jq '.caps | {sync_budget_rows, sync_budget_requests, async_max_rows, async_max_requests}'

Choose a source, method, or unit

Bind a facet in input.scope.facets when the data requires a particular provenance, measurement method, or unit. How the APIs fit together explains facet selection and unit conversion.

Add this scope object to input in mint.json. Keep kind, spec_id, and definition alongside it.

{
  "scope": {
    "facets": {
      "provenance": {"ref": "provenance/OECDCreditorReportingSystem"},
      "unit": {"ref": "CurrencyUSD"}
    }
  }
}

Request a bare frame

Use layout: "frame" for a bare Parquet file in the frame layout. The request needs format: "parquet" and bundle: false. Frame layout cannot be combined with options.pivot, which reshapes the table from long to wide with one index column and one columns column.

{
  "options": {"format": "parquet", "bundle": false, "layout": "frame"}
}

How the APIs fit together explains versions, rows, series, and facets. API conventions documents response envelopes and problem details.

View as markdown

Documentation