DataPress logo
API Reference

Exporting Metadata

Version:  4.0
Updated:  14th July 2024

Use these endpoints to export all datasets from the site.

Warning
Do not rely on screen-scraping or crawling techniques, which may be automatically blocked.

Exporting as JSON

This JSON structure is easy to parse and process if you need a full-site export.

  • It is similar to the Dataset editing API, with additions:
    • The resources and links are sorted into lists.
    • The topics, masthead and team fields are enriched with additional metadata.
GET
https://data.website/api/datasets/export.json
HTTP Headers
AuthorizationOptional API key will export the public catalogue and your private datasets.stringOptional
/api/datasets/export.json
[
  {
    // The public URL of the dataset
    "webpage": "https://data.website/dataset/ab123",
    // The ID for use through the API
    "id": "ab123",
    // The title of the dataset:
    "title": "Freedom of Information Requests",
    // The body of the dataset page:
    "description": "<p>Page content <img src=\"...\" /></p>",
    // The masthead image at the top of the page:
    "masthead": {
      "id": "london-borough-of-barnet",
      "title": "London Borough of Barnet"
    },
    // Tags are used by the search engine:
    "tags": ["FOI Requests", "FOI", "Freedom of Information"],
    // Topics are pre-defined in settings.
    // They create designated landing pages for similar datasets:
    "topics": [
      {
        "id": "community-information",
        "title": "Community Information"
      },
      {
        "id": "council-and-democracy",
        "title": "Council & Democracy"
      }
    ],

    // -- [SIDEBAR CONTENT]
    // The next review date:
    "nextReviewDate": "2025-01-15",
    // The licence:
    "licence": {
      "url": "https://.../doc/open-government-licence/version/3/",
      "title": "UK Open Government Licence (OGL v3)"
    },
    // The clickable "contact" link in the sidebar:
    "contact": {
      "email": "[email protected]",
      "title": "Barnet Open Data Team"
    },
    // Custom fields are pre-defined by the site administrators.
    // These can be string or string[]
    "custom": {
      "unit": "Assurance",
      "service": "Information Management Team",
      "retention": "3 months",
      "geography": ["London", "Barnet"]
    },


    // -- [FILES]
    // The list of files in the dataset:
    "resources": [
      {
        // Resource ID is unique within the dataset
        // (Not guaranteed to be unique across the site)
        "id": "k93",
        // File title:
        "title": "FOI requests received 2020-25",
        // Link to download the file
        "url": "https://data.website/download/[...]", // [immutable]
        // Filename:
        "filename": "25-01-21 FOI requests received.csv",
        // Optional description, plain text:
        "description": "This file contains [...]",
        // Optional Timeframe covered by the file, as ISO month strings:
        "timeFrameFrom": "2020-01",
        "timeFrameTo": "2024-11",
        // File size in bytes:
        "size": 1337303,
        // MD5 of the file:
        "hash": "2cdf3a47738b93492906e3ad7e415d1a",
        // Timestamp of the upload:
        "timestamp": "2025-01-21T16:10:02",
        // Backlink to the parent dataset
        "dataset": "ab123",
        "webpage": "https://data.website/dataset/ab123"
      }
    ],

    "links": [
      {
        // Link ID is unique within the dataset
        // (Not guaranteed to be unique across the site)
        "id": "el3",
        // The title of the link:
        "title": "TfL Open Data",
        // The URL to the link:
        "url": "https://tfl.gov.uk/info-for/open-data-users/our-open-data",
        // Optional description, plain text:
        "description": "Link to TfL Open Data site.",
        // QA last saw this HTTP status code:
        "httpStatus": 200,
        // QA detected this Content-Length, if available:
        "size": null,
        // QA fetches the Mimetype header, if available:
        "mimetype": "text/html; charset=utf-8",
        // QA fetches the remote ETag or other hash, if available:
        "hash": null,
        // Most recent QA timestamp:
        "qaTimestamp": "2025-01-28T11:17:08.531Z"
        // Backlink to the parent dataset
        "dataset": "v8wmv",
        "webpage": "https://open.barnet.gov.uk/dataset/v8wmv"
      }
    ],

    // -- [ACCESS CONTROL]
    // Sharing: "public" or "private".
    // Public datasets are visible to all users:
    "sharing": "public",
    // The owner team. All members can edit the dataset:
    "team": {
      "id": "vrv65",
      "title": "London Borough of Barnet"
    },

    // -- [METADATA]
    // Timestamp metadata:
    "createdAt": "2019-02-05T16:34:18.000Z",
    "updatedAt": "2025-01-21T16:10:03.385Z",
  },
]

Exporting as CSV

1. A table listing all datasets:

GET
https://data.website/api/datasets/export.datasets.csv
HTTP Headers
AuthorizationOptional API key will export the public catalogue and your private datasets.stringOptional

2. A table listing all file uploads:

GET
https://data.website/api/datasets/export.resources.csv
HTTP Headers
AuthorizationOptional API key will export the public catalogue and your private datasets.stringOptional

Exporting as Excel

The export workbook is a single Excel file with multiple sheets:

  • datasets - A table listing all datasets in the catalogue.
  • resources - A table listing all file uploads in the catalogue.
  • links - A table listing all external links in the catalogue.
GET
https://data.website/api/datasets/export.xlsx
HTTP Headers
AuthorizationOptional API key will export the public catalogue and your private datasets.stringOptional

Exporting as CKAN

1. current_package_list_with_resources

This endpoint is maintained for compatibility with legacy CKAN clients expecting this JSON response format.

Deprecated
This is not a complete representation of the catalogue, and will be removed in the future.
Do not use this endpoint for new integrations.
GET
https://data.website/api/action/current_package_list_with_resources
HTTP Headers
AuthorizationOptional API key will export the public catalogue and your private datasets.stringOptional
current_package_list_with_resources
{
  "success": true,
  "result": [
    {
      "id": "ab123",
      "name": "ab123",
      "title": "Freedom of Information Requests",
      "resources": [
        // ...
      ]
    },
    // ...
  ]
}

This endpoint is maintained for compatibility with legacy CKAN clients expecting this JSON response format.

Deprecated
This is not a complete representation of the catalogue, and will be removed in the future.
Do not use this endpoint for new integrations.
GET
https://data.website/api/action/package_search
HTTP Headers
AuthorizationOptional API key will export the public catalogue and your private datasets.stringOptional
package_search
{
  "success": true,
  "result": {
    "count": 123,
    "sort": "score desc, metadata_modified desc",
    "result": [
      {
        "id": "ab123",
        "name": "ab123",
        "title": "Freedom of Information Requests",
        "resources": [
          // ...
        ]
      }
      // ...
    ]
  }
}