IATI Data Quality Index Consultation - Part 2 (Data Completeness) Summary

IATI Data Quality Index Consultation - Part 2 

Data Completeness 

Summary

Background 

As part of the IATI Strategic Plan (2020-2025), IATI is committed to supporting organisations to publish high-quality IATI data. To help achieve this, and in response to a request from IATI members, a set of revised data quality measures is being developed, to underscore “what good quality data looks like” for IATI’s more than 1,400 publishers. These measures will form the basis of a “Data Quality Index” (DQI), which will allow publishers to better understand their successes and challenges along the IATI data quality journey, and help ensure their data can be used by intended beneficiaries at the country level.

The DQI Consultation

In order to ensure a methodology that is transparent, fair, and reflects the diversity of IATI publishers and meets the needs of data users, an online consultation with the IATI community is being held to receive feedback and refine the DQI methodology in advance of its launch in late 2023. The first part of this consultation, on “Timeliness, Frequency, and Validation” was open from 15 February - 28 March 2022.  The second part of this consultation, on “Data Completeness” was open from 28 March -  6 May 2022.

Many questions were also received from the community in regards to the format of the DQI and specific questions on how it will be weighted and used to promote better IATI data quality. As these questions are relevant to the entirety of the DQI consultation, answers on these issues have been included by the Secretariat in the consultation FAQ document.

High-Level Summary of Comments

The following is a high-level summary of the comments received during the consultation period. The consultation pages will be left open for additional comments following the official close of the consultation, however consultation pages will not be actively monitored or moderated. 

Find out more on the main DQI Consultation -page on IATI Connect

Please find below for each measure the highlighted input by the Community on the DQI Consultation Part 2 (Data Completeness)

Mandatory and Recommended (2.1.1 & 2 .1.2)

• There was agreement amongst commentators that the mandatory and recommended IATI elements should be included in the DQI assessment.
• It was generally agreed that the methodology should follow all of the rules and guidance contained within the IATI schema. This includes the logic of how activity dates are arranged, for example publishers should not get credit for activity actual end dates that are before activity actual start dates.
• There was discussion about how the quality of reporting org names, titles and descriptions could be assessed. One proposal was to add a minimum character limit for each narrative; this could work as an automatic check to see if substantial content has been added.

Location (2.2)

• There was a good level of agreement that the methodology was including checks on the necessary sub-elements within the location element, with the one addition that Gazetteer Reference should be included.
• There was discussion about what types of activities and publishers could publish meaningful locations. Examples provided included activities which were to do with general budget support and administrative costs, these activities often cannot be linked to sub-national location details.

Classifications (2.3.1 - 2.3.5)

• There was discussion on what types of organisations flow type classifications were suitable for. Often non OECD DAC reporters do not know if the funding they are receiving are ODA or not, and often should not be counted as ODA further down the chain. There was a comment that the Dutch reporting guidance now does not include flow-type as the Dutch considered it to bemore useful to governments and multilaterals than downstream partners. But there were concerns raised that excluding this data may inhibit its usability by data users.
• General agreement that if a default is not present, it should be present for every transaction. However, further discussion took place  on what types of transactions finance-type relates to. There was a proposal that finance-type does not relate to expenditure transactions and these should be removed from the assessment.
• Wording around aid types, and specifying that any vocabulary can be used was added to the methodology.
• One commenter noted that not all activities need to contain a policy-marker. A request was made for further guidance on what is good practice in relation to policy-markers and what is expected of publishers.

Sustainable Development Goals (2.5)

• Not all activities will follow the Sustainable Development Goals (SDGs).
• There was general agreement that Sustainable Development Goals should be included in the DQI.
• However, there was disagreement about whether SDG reporting using sectors should also be included. It was noted that this is primarily a concern for UN reporting. Other commenters wanted to follow the IATI guidance, and agreed that assigning percentages to SDGs is not accurate and should not be encouraged.
• Some commenters also mentioned the relationship between SDGs and results. At present, there is little evidence of IATI publishers including SDGs within a result element and it is not currently a proposed measure within the DQI.

Identifiers and traceability (2.6.1 - 2.6.4)

• Commentators agreed that the reporting-org within an activity file should match the reporter’s IATI Org ID on the IATI Registry.
• There was discussion over what the term ‘valid’ means in relation to organisation and activity identifiers. The word ‘valid’ has been removed from the methodology.
• Following a community proposal we have adjusted the check on participating organisation refs to check that they either match an IATI Org ID on the IATI Registry or start with a prefix on the Org-id.guide list. There was also an ask for there to be a scraping of IATI data to find all of the participating organisation details and create a consolidated list of ‘valid’ ones.
• Commentators noted that publishers should only be expected to know organisation identifiers for funders.
• There is an outstanding question of if the above logic should also be applied to transaction provider and receiver org identifiers.

Financial (2.7.1 - 2.7.3)

• There was general agreement with the methodology for the financial measures proposed and that zero valued transactions should be removed from the assessments.
• Commentators proposed that organisations who were donors should be removed from the assessment looking at incoming funds. This could be done by using the participating-org role attribute, removing activities where the reporting-org is labeled as the only funder.
• There was discussion on removing end-of-the-chain organisations from the assessment of outgoing commitments. To enable this, the methodology could check if there are no external organizations listed as implementers in participating org.
• There was no comment about which level hierarchy the transaction provider and receiver organisation checks should be applied to.
• Commentators agreed that with the methodology that budgets should span the entire duration of activity. However, there was discussion of how this would work if an activity was delayed and therefore there was no budget or spend for a time. There was recognition that 0 valued budgets are allowed and would account for this.

Humanitarian (2.8.1 & 2.8.2)

• Discussions noted that the proportion of humanitarian activities within a publishers’ data was misleading as no proportion was ‘better’ or ‘worse’ than another. Rather, a count of the number of humanitarian activities should be added.
• There was disagreement on how humanitarian scope should be assessed in the DQI. In the current Standard it can only be applied per activity, when applied to partially humanitarian activities it becomes misleading as not all of the activity’s finances will be contributing toward the particular scope/s.
• One commenter also noted that humanitarian clusters (sector code 10) don't work when clusters are combined within a particular country. The question was also raised on how to assign percentages for when an activity is only partially humanitarian. There is no option for selecting ‘none’ within sector vocabulary 10, reflecting the parts of an activity which cannot be mapped to a humanitarian cluster. These are issues within the IATI Standard and require further input from the humanitarian community to address them.

Results (2.9.1 - 2.9.4)

• There was general agreement that results relate the most to closed activities, rather than active ones. To accommodate this there was a proposal made for the methodology to include both active and recently-closed (closed within the last 12 months) activities.
• It was also noted that results are not relevant to all activities, particularly not those focusing on general budget support or administrative costs. It was suggested that the aid-type codelist could be used to filter these activities out of the assessment.
• There was disagreement on what sub-elements within a result should be included in the DQI. Some commenters wanted to drop the checks on baseline targets and indicators, others wanted to see further checks added in e.g., on aggregation status, type and measure.
• Discussion took place on how baseline, target and actual measures worked for qualitative indicators and if each value was necessary. The IATI Standard is built to accommodate each of these for qualitative measures, asking for a narrative comment rather than a value.

Documentation Links (2.10.1 & 2.10.2)

• There was general agreement of the methodology for checking document links in activity files. It was noted that a future enhanced quality check should not only look for the presence of a document link but check the validity of the URL provided.
• Commentators were unsure on how document links in organisation files are related to data quality. It was suggested that this measure should be removed.
• One commentator was asking why the methodology penalized organisations who had more than one organisation file and also why this is not recommended by the Standard.

Next Steps

The consultation pages for Timeliness, Frequency and Validation as well as Data Completeness will be left open for additional comments following the official close of the consultation, however consultation pages will not be actively monitored or moderated.

The next phase of the consultation on Weights and Measures is currently postponed whilst the IATI Secretariat reflects on how the initial methodology should be updated to meet the needs expressed across our diverse publishers and data users. Specifically there will be careful consideration on how to resolve issues on divergent views on the methodology.