FAQ's Data Quality Index Consultation

FAQ's Data Quality Index Consultation

Based on the first DQI-Webinar (23 February) check out Frequently Asked Questions on Timeliness and Validation.

On 15 February the IATI Secretariat launched the second phase of the Data Quality Index (DQI) Consultation, which is divided into three parts: 

(1) Timeliness and Validation => Read the Summary

(2) Data Completeness => Read the Summary

(3) Weights and Measures => POSTPONED

At the start of each Consultation part,  the Secretariat hosts a Webinar to explain the process, measures, and how you can engage. On Wednesday 23 February the first DQI Webinar on Timeliness and Validation was held, to watch the recordings click here.

Check out the FAQ's below (also accessible in other languages, please click 'Select your language' on the top right).

In case of any questions or comments related to this FAQ, let us know via connect@iatistandard.org

What is the DQI? Is it a dashboard, a tool, a report, a ranking (an Index)?

The Data Quality Index is a set of measures that will assess the quality of an organisation’s data. These measures have been designed in consultation with publishers and data users to ensure that they drive the types of quality improvements that improve the usability of the data (i.e. they are rooted in data use needs). Performance on each measure and the measures on aggregate will be displayed in an improved Dashboard that is updated daily. The composition of the aggregate performance (what measures are included and what weights they are given) will be consulted during the third phase of the Data Quality Index Consultation. For a refresher on the background on the rationale behind the Index, you can visit the Background Paper from the first consultation held in 2021.

If the goal is to improve data quality, how will the DQI achieve this, and how will it be better than existing approaches for improving data quality?

By collectively agreeing on what good data quality looks like – and establishing measures to assess whether an organisation is achieving these quality standards, we can then begin to prioritise and work with publishers to systematically improve the quality of their data. The goal is not just to create a set of measures, but to use this set of measures to engage with publishers to make improvements to their data. It will enable a proactive rather than reactive approach to addressing data quality, which is a key priority as part of ongoing work towards achieving our strategic objectives. We will then assess at least annually (via the IATI Strategic Plan Results Framework) whether progress is being made.

How do you avoid any Index becoming a "league table"? Is that the purpose of this?

We will carefully consider how performance on the measures is presented to avoid the pitfalls of a “league table.” A discussion on this will be held during Phase 3 of the consultation on Weights and Measures.

Could we get a common vocabulary together?

Providing definitions for each of the DQI terms and ensuring that they are integrated into the future Dashboard is something the Secretariat can commit to working on. Please provide suggestions for this in the consultation on Connect.

Why the focus (here and in the other timeliness sections) on transactions? Do you expect those to be updated most often or is it considered the most important data?

This was discussed in the first round of the consultation. In IATI data, transactions are updated the most frequently and represent a change in the data.

For future improvements to the DQI, other timeliness measures could be explored. More work would be needed to define what a “substantial” update includes, e.g. data has been changed or added to, rather than the last-updated-date being amended.

Timeliness - tying this to transactions rewards transaction-heavy publishers. Results, documents, partners and other elements could also be updated. Don't forget that baked into the standard are "generated-datetime" and "last-updated-datetime"

This was discussed in the first round of the consultation. In IATI data, transactions are updated the most frequently and represent a change in the data.

For future improvements to the DQI, other timeliness measures could be explored. More work would be needed to define what a “substantial” update includes, e.g. data has been changed or added to, rather than the last-updated-date being amended.

Can we clarify what does “active activities” mean?

“Active activities” refer to activities that have an actual-start-date in the past and an actual-end-date (if published) in the future. If the actual start and end dates are not present, the planned start and end dates will be used (added into the DQI Consultation text);

Do you expect publishers to make these updates in real-time?

We don't expect this. It’s proposed that the DQI tracks how often a publisher updates its data. Where a publisher is updating in real-time this is useful information to data users. The weights and measures given to each frequency are to be discussed in the third section of this consultation. It is our view that we want to encourage timely reporting but this does not have to mean daily.

Could the error count be not just raw count but also a % of activities or some other denominator?

Yes, it could. Please leave any specific (track) changes on the methodology via this DQI Live-Editing-Page. Click on the Collaboration tab, and leave your comments directly into the document.

Why have duplicative timelag elements? all transactions vs spend transactions vs spend updates. That's 3, not 2 different spends…

The most common use case of IATI data is to assess spending – e.g. to a recipient country, in a particular sector, etc. As such, the goal here was to provide an additional measure assessing whether an organisation is publishing regular spend data.

Aside from changing the indicators, weights, and measures, how will the dashboard be different from the current one in terms of its advocacy purpose in the way that it will try and encourage change?

Under the guidance of the IATI Governing Board, the IATI Secretariat will utilise the DQI to actively communicate performance on improving data quality in regards to individuals and groups of IATI publishers, with the end goal of enacting positive change for overall IATI data quality.

On timeliness: is it important for there to be a "category" to express higher velocity - daily; weekly - or even "realtime"?

Timeliness assesses updates to transactions. See the methodology here. Adding categories for daily or weekly could be an option. Please add what you would recommend and the rationale to the consultation page on Connect.

Is it possible to separate sovereign and non-sovereign data for "the assessment"?

The sovereign vs non-sovereign classification is not currently captured in IATI. This should form part of the next IATI upgrade and then be included in the DQI.

What if there were no spend transactions during the month? How do you distinguish between whether the spend transaction is missing for a valid reason (no transactions to publish) or because the publisher did not update?

This is valid and why it will matter how performance on this measure is assessed. Just because we assess whether spend transactions happen in each month (which is useful information for data users) doesn’t mean that an organisation would be assessed / categorised negatively for not having spent every month. For example, perhaps it would make sense to assess whether quarterly spending is provided as most organisations would have spent each quarter. This will be discussed in Phase 3 of the consultation on Weights and Measures.

"Mandatory elements" may need to be redefined as they are different depending upon where in the aid chain a publisher fits in.

There are very few mandatory elements in the IATI schema. What elements are captured in the DQI comes under the next phase of the consultation when we look at Data Completeness, and then again when we assign weights and measures to these metrics.