The Detail Data Portal is a data catalogue: a listing of various datasets relevant to Northern Ireland, including databases, pre-formatted tables, statistical reports and other resources. It is maintained by NICVA as part of the BIG Lottery funded Detail Data project.
Users can access, view and download data from the datasets available from the Portal.
Why was the Detail Data Portal created?
We want to see the community and voluntary sector develop its capacity in using data to provide and improve services, undertake research, and use evidence to support communities and bring about change.
The Portal's primary focus is on the needs of the local voluntary and community sector, but this in no way limits the range of datasets that can be hosted.
What is Open Data?
Open Data is data that is available for anyone to to access, use and share for any purpose (in some cases, with requirements to attribute the original authors).
Open Data increases transparency, unleashes social and economic value and unlocks civic participation and engagement. (Open Data Institute)
What is my role in all of this?
That depends! You could be a data user, a contributor, or both. You can access, download, reuse and share a wide range of open datasets to suit your needs. In future, you will also be able to contribute by uploading your own datasets to the portal. In the meantime, please contact us if there is some data you would like to share on the Portal.
What is a Data Portal?
A Data Portal (otherwise known as a Data Catalogue) is a 'wrapper' containing individual Datasets, each of which contain a number of Resources alongside the Metadata used to describe both.
In addition, the wrapper allows us to add other useful features for data users such as guides, blogs and featured case studies.
How is the Detail Data Portal structured?
The Portal (otherwise known as a Data Catalog) is a 'wrapper' containing individual Datasets, each of which contain a number of Resources alongside the Metadata used to describe both. In addition, the wrapper allows us to add other useful features such as guides, blogs, case studies and a user community.
- Datasets contain individual Resources as well as the Metadata for the Dataset, giving users information about the data and where it comes from.
- Resources are the actual files that together make up a dataset (there might be only one single file, or many). This might include Excel (.xls) or Word (.doc) files, as well as .csv .kml .xml .pdf .odf and many others supported by the Portal system. Resources also contain Metadata.
- Metadata is essentially data about data. It contains the who, what, where, when and why of a dataset and resource to give users a bit more information other than the data itself, including the licence and how it can be reused.
A dataset, such as NICVA’s Individual Giving Survey, therefore has a number of resources contained within it, such as data files for every part of the survey, as well as the metadata that accurately describes both the dataset and each resource.
What are acceptable formats for the data files (resources)?
The types of files used and how the data is presented is very important. We aim to make any open data provided through the Portal at least 3 stars on the 5★ open data ladder. We will also certify datasets through the ODI's Open Data Certificates.
It is therefore important how the data that we upload is formatted. Here is a very quick guide to making open data really useful.
What are the licensing requirements?
When data is uploaded to the portal you will be required to choose an open data licence to instruct others their right to use, modify and share the data.
Readily-available and useable licences, in line with the open definition, are available from Open Knowledge.
We recommend using one of these open licences if you are uploading datasets, specifically the Open Data Commons Attribution License (ODC-BY) as you will retain the moral right to be recognised as the author.
Where are your Terms and Conditions?
Why use a Portal and why structure data like this?
Because the Portal is structured in such a way, it can “talk” with other data catalogues on the Web. This means that it can pull in interesting datasets from other catalogues with ease, giving you many more resources than could otherwise be provided, but with the same level of functionality as the Detail Data Portal.
Also, because the Data Portal is based on an open source web-based format, resources uploaded in CSV or XML format can be “parsed” automatically, meaning that that data is stored on the platform in a way that adds an extra level of functionality (known as the Datastore), such as the geocoder to allow interactive map-based data visualisations, and a public API to allow external linking queries for web application developers (see below).
Using accurate metadata alongside a standard model for data interchange harnesses the ability of the Web to link data together so that it can be easily found, merged and shared. The standardised framework that the Portal uses to describe its datasets through the metadata is called the Data Catalog Vocabulary (DCAT).
Are there other resources I can use alongside this data?
There's many, too many to list in one place, but check out these data tools for a starter.
Technical and API
What is the platform behind the portal?
The portal was created on an open source cataloging platform called DKAN.
DKAN has been created specifically for storing, publishing and accessing data online, and is modelled on the popular CKAN platform (used, for example, by data.gov.uk).
Integrated with the Drupal content management system, DKAN has a number of useful features such as the ability to preview and visualise data, link datasets to those in other data catalogues, and exposing metadata to the Web. We are also able to respond to the need to add or adjust features in future.
What is the Dataset API?
The API (application program interface) is a protocol that provides the building blocks for programmatic access to the list of datasets stored in the Detail Data Portal, with their associated metadata. It exposes the dataset metadata to the Web.
The API can query metadata for specific datasets and also retrieve datasets. Developers can use this API to access catalog information to support their own applications.
If you have used the CKAN API, you will be comfortable with the DKAN API as it has been designed with interoperability in mind.
What is the Datastore API?
In contrast to the Dataset API (which provides access to information about the datasets on the Detail Data Portal) the Datastore API provides access to the data itself in a structured and machine-readable format.
Additional parameters (filters, limit, full text query, join, sort etc) allow for extra functionality and ease-of-use.
The most straighforward way to access the Datastore API for an individual resource is to click the 'Data API' button on that resource where you will get an example.
The default return format is XML, but JSON and JSONP can be specified (e.g. http://data.nicva.org/api/action/datastore/search.json)
Not all resources have been added to the Datastore, as they are not always suitable or practical to do so. If there you find some data that would like to access through the Datastore API but hasn't been added, please ask if this can be done.
Do I need to register to get an API key?
No. There is no API key required to use either the Dataset or Datastore APIs. Just write good code that won't burden the server.
Where can I get more information?
There are numerous sources on the Web to aid the open data community, from using and applying open data, making your own organisation's data open and campaigning for more open data.
+ Socrata’s Open Data Field Guide.
+ Open Data Institute (ODI) has a wealth of information including a number of guidebooks. NICVA acts as the ODI Node for Northern Ireland.
+ Open Data Commons focuses on licencing and legal tools.
+ Nesta (National Endowment for Science Technology and the Arts) have a large range of resources on open data, particularly relating to how it can be used for the public good.
+ Data.gov.uk The UK Government’s open data portal.
+ Data.gov.ie The Irish Government's open data portal.