Using the API from Python¶

Everything the portal does goes through the API, so a script can do the same: build a recipe from a template, run it, wait for the job and download the dataset. This page shows how, with Python and the requests library. The complete list of operations is in REST API and, on the service, at https://qumasc.psnc.pl/api/v1/docs.

API keys¶

A script signs in with an API key, not with your password. A key acts with your rights.

In the portal open Account → API keys.
Under New key give the key a Name, for example the script that uses it. Valid until is optional; without a date the key stays valid until you revoke it.
Choose Create the key and copy the key now: it is shown only once.
Send it in the header X-API-Key of every request.

The list Your API keys shows the first characters of each key, when it was made and last used, and until when it is valid. Revoke stops a key at once. Keep keys secret: do not put them in code you share. Changing your password does not affect your keys; deleting your account revokes them (see Your account).

Before the first job¶

The rules of the portal apply to scripts too:

Store your provider keys for the sources of the recipe, for example the CDS key for ERA5-Land (The web portal). You can do it in the portal or with PUT /me/credentials/{connector} (see Your own credentials for data sources).
Accept the source licences the recipe needs (in the portal, or with POST /me/licence-acceptances).
Regular users may run only recipes built from a template. Advanced users may submit any valid recipe.

Example: ERA5-Land per region¶

The script below runs the template era5land-to-regions for the NUTS 2 and NUTS 3 regions of Małopolskie (PL21), January to March 2024, and downloads the files. Install requests first (pip install requests) and put your key in the environment variable QUMASC_API_KEY.

"""Run the template era5land-to-regions and download the dataset."""
import os
import time
from pathlib import Path

import requests

API = "https://qumasc.psnc.pl/api/v1"
session = requests.Session()
session.headers["X-API-Key"] = os.environ["QUMASC_API_KEY"]


def call(method, path, **kwargs):
    """Call the API and return the JSON answer; stop with the service's message on an error."""
    response = session.request(method, API + path, timeout=120, **kwargs)
    if not response.ok:
        is_json = response.headers.get("Content-Type", "").endswith("json")
        problem = response.json() if is_json else {"detail": response.text}
        raise SystemExit(f"{response.status_code} {problem.get('title')}: {problem.get('detail')}"
                         f"\n{problem.get('errors') or ''}")
    return response.json()


# 1. The templates you may run.
for template in call("GET", "/recipes/templates")["items"]:
    print(template["id"], "-", template["title"])

# 2. Build the recipe from the parameters of the form, with the estimate.
#    Fields you leave out get their defaults.
parameters = {
    "area": {"code_list": "NUTS", "version": "2024", "levels": [2, 3], "codes": ["PL21"]},
    "period": {"start": "2024-01", "end": "2024-03"},
    "variables": ["t2m", "tp"],
    "statistics": ["mean"],
    "output": {"format": "parquet", "boundaries": True},
}
built = call("POST", "/recipes/templates/era5land-to-regions/recipe",
             params={"estimate": "true"}, json={"parameters": parameters})

missing = [item["connector"] for item in built["credentials_required"]
           if item["required"] and not item["stored"]]
if missing:
    raise SystemExit(f"Store your provider keys first: {missing}")
to_accept = [item["source"] for item in built["licence_acceptances"]
             if item["required"] and not item["accepted"]]
if to_accept:
    raise SystemExit(f"Accept the source licences first: {to_accept}")
for warning in built["warnings"]:
    print("warning:", warning["message"])
estimate = built["estimate"]
print("download:", estimate.get("download_bytes"), "bytes;",
      "result:", estimate.get("output_bytes"), "bytes;",
      "core hours:", estimate.get("core_hours"))

# 3. Submit the recipe as a job.
job = call("POST", "/jobs", json={"recipe": built["recipe"]})
print("job", job["id"], job["status"])

# 4. Wait until the job has ended.
while job["status"] not in ("succeeded", "failed", "cancelled"):
    time.sleep(30)
    job = call("GET", f"/jobs/{job['id']}")
    progress = job.get("progress") or {}
    print(job["status"], progress.get("fraction"), progress.get("current_step"))
if job["status"] != "succeeded":
    raise SystemExit(f"The job ended as {job['status']}: {job.get('error')}")

# 5. The datasets of the job, and their files.
results = call("GET", f"/jobs/{job['id']}/results")
for dataset in results["datasets"]:
    print("dataset", dataset["id"], dataset["title"])
    downloads = call("GET", f"/datasets/{dataset['id']}/downloads")
    for link in downloads["links"]:
        # The links are signed and valid until downloads["expires_at"];
        # they need no API key.
        target = Path(dataset["id"]) / link["asset"]
        target.parent.mkdir(exist_ok=True)
        with requests.get(link["href"], stream=True, timeout=600) as answer:
            answer.raise_for_status()
            with open(target, "wb") as file:
                for chunk in answer.iter_content(chunk_size=1 << 20):
                    file.write(chunk)
        print("saved", target, link["bytes"], "bytes")

The asset keys name the files without an extension (for example values); the format of each file is in the dataset record (GET /datasets/{id}, member assets). Downloads count against your monthly download quota.

What the answers contain¶

Request	Answer
`GET /recipes/templates`	`items`: the templates with `id`, `title`, `description`, `parameters_schema` (the form as JSON Schema), `parameters_defaults`, `connectors`, `credentials_required`; `links.next` names the next page, if any.
`POST /recipes/templates/{id}/recipe`	Body `{"parameters": {...}}`; with `?estimate=true` also `estimate`. Answer: `template`, `parameters` (with the defaults filled in), `recipe`, `recipe_hash`, `warnings`, `credentials_required`, `credentials_link`, `licence_acceptances`, `estimate`.
`POST /jobs`	Body `{"recipe": {...}}`. Answer `201`: the job with `id` and `status`.
`GET /jobs/{id}`	`status` (`queued`, `planning`, `running`, `writing`, `succeeded`, `failed`, `cancelling`, `cancelled`), `progress` (`fraction`, `current_step`, `steps_done`, `steps_total`), `usage`, `warnings`, `error`. A job that waits for a provider (for example in the queue of the Climate Data Store) is `running`; `current_step` names the step.
`GET /jobs/{id}/logs`	The log of the job, page by page.
`GET /jobs/{id}/results`	`job_id` and `datasets` (`id`, `title`, `licence`, `expires_at`, …); `409` while the job has not succeeded.
`GET /datasets/{id}/downloads`	`expires_at` and `links` (`asset`, `href`, `bytes`); `?packaging=zip` gives one link to a zip archive with README, licence, recipe and provenance.
`POST /jobs/{id}/cancel`	Cancels the job.

The parameters of each template are described by its parameters_schema. For era5land-to-regions they are area (code_list, version, levels, codes), period (start, end as YYYY-MM), variables (t2m, tp, d2m, swvl1, ws10), statistics (mean, min, max, sd, median), output (format: parquet or csv; boundaries) and an optional title. The fields of the form are explained in Making a dataset.

Errors¶

Errors are problem details (application/problem+json) with title, status, code and a detail written for people. The ones a script meets most:

Status and code	Meaning	What to do
`422 invalid_parameters`	A parameter is wrong. `errors[]` give a `pointer` into the parameters (for example `/period/end`) and a `message`.	Correct the parameter.
`409 credentials_required`	A provider key is missing. `connectors` names the sources; `link` the portal page for the keys.	Store the key.
`409 licence_not_accepted`	A source licence is not accepted.	Accept it in the portal or with `POST /me/licence-acceptances` (`source`, `licence`).
`409 quota_exceeded`	A quota would be exceeded; `quota` names it.	Delete datasets or ask for a larger quota.
`429`	Too many requests or queued jobs.	Wait the seconds of the header `Retry-After`.

To repeat a submission safely after a network error, send the header Idempotency-Key with a value of your choice on POST /jobs. A second submission with the same key returns the job of the first one instead of making a new job.