IATI Data Quality Index Consultation - Part 1 

Timeliness, Frequency, and Validation 

Summary


Background

As part of the IATI Strategic Plan (2020-2025), IATI is committed to supporting organizations to publish high-quality IATI data. To help achieve this, and in response to a request from IATI members, a set of revised data quality measures is being developed, to underscore “what good quality data looks like” for IATI’s more than 1,300 publishers. These measures will form the basis of a “Data Quality Index” (DQI), which will allow publishers to better understand their successes and challenges along the IATI data quality journey, and help ensure their data can be used by intended beneficiaries at the country level.

The DQI Consultation

In order to ensure a methodology that is transparent, fair, and reflects the diversity of IATI publishers, an online consultation with the IATI community is being held to receive feedback and refine the DQI methodology in advance of its launch in late 2023. The first part of this consultation, on “Timeliness, Frequency, and Validation” was open from 15 February - 28 March 2022. 

Many questions were also received from the community in regards to the format of the DQI and specific questions on how it will be weighted and used to promote better IATI data quality. As these questions are relevant to the entirety of the DQI consultation, answers on these issues have been included by the Secretariat in the consultation FAQ document.

High-Level Summary of Comments

The following is a high-level summary of the comments received during the consultation period. The consultation pages will be left open for additional comments following the official close of the consultation, however consultation pages will not be actively monitored or moderated. 

Find out more on the main DQI Consultation-page on IATI Connect.


Please find below for each measure the highlighted input by the Community on the DQI Consultation Part 1 (Timeliness, Frequency, and Validation) 


Timeliness (frequency) (1.1)

  • There was some disagreement amongst commentators on the frequency categories for assessing timeliness (e.g. weekly, monthly, quarterly, etc.). One commenter felt that the categories proposed were sufficient and in line with previous  assessments, but others advocated for reducing the number of categories, noting that the “six-monthly" seems overly complicated and that the category "less than annual" should become "inactive" if IATI is serious about timeliness. There was some level of agreement that frequency categories should be: “weekly, monthly, quarterly, annually, inactive.”
  • It was generally agreed that publishers should not get credit for publishing transactions with a transaction value of 0; and that future transaction dates should be ignored from the assessment (i.e., a publisher will not get credit for updating their transactions if they are for future dates).

Timelag (all transactions) (1.2.1)

  • There was some level of agreement on calling this measure “Timelag,” and no additional suggestions were made.
  • It was noted that the model implicit in the proposed DQI is that programs are implemented in a fully predictable and / or linear fashion and therefore data should be available at predictable and evenly spaced moments, but that many donors / types of funding do not conform with this model.
  • There was a suggestion to improve the definition of “active activity” as often activities will have a planned and actual start date and only a planned end date: *Active activities refer to activities which have an actual-start-date in the past and a planned or actual-end-date in the future. If actual start and / or end dates are not present, the planned start and end dates will be used.
  • Following several questions / comments around why this measure focuses on the activity level, a point of clarity was made: during the last round of consultation there was a debate around using transaction updates as a proxy to see if an organization has updated their data. However, it was also the view that Timelag and Frequency were particularly pertinent to transaction data, and these measures are useful to data users. Hence, these two measures look at the latest transaction dates.
  • It was suggested that the Timelag measure could be visualized in a range plot.

Timeliness (spend transactions) (1.2.2)

  • There was a good level of agreement that the proposed Timelag measures are based on an implicit business model of activities (programs, projects) with multiple transactions, which is not always the case, especially in more mature organizations: some publishers may have only a few major grants, with one payment per year (which should be recognized in finalizing the methodology for this measure).
  • Commenters also noted that the proposed measures are potentially duplicative: Timelag in all transactions, then Timelag in spend transactions, then also in spend updates. There was a suggestion that this could possibly be adjusted for in the weighting portion of the consultation (part three).
  • Important to note that this Spend Transactions measure, like the Spend Updates measure (1.3), are primarily helping to assess the timeliness and predictability of the provision of spend data. How the measure is actually weighted will determine how publishers are assessed on the measure. This will be part of phase 3 of the DQI Consultation.
  • One commenter advocated that zero values should be allowed in order to focus on recency of reporting rather than how quickly an organization is disbursing / spending money, but noted that there will need to be guidance / allowance for zero values when a publisher does not spend during a given period. They suggested that zero values are only applied for expenditures, not for disbursement, as the former are much more logically aggregated than the latter.
  • There was also a recognition that there is a high potential for manipulation of this measure, and an open question posed around which measures are to be put in place to prevent this.

Timeliness (spend updates) (1.3) and Timeliness (active publishers) (1.4)

  • There was some discussion between commenters around the rationale for including spend updates, noting that less disbursement can be (and is often) a reflection of higher quality program implementation and that more transactions can often mean piecemeal funding of implementing partners. One commenter viewed the spend updates measure as quite problematic and felt that it should be removed.

Validation (summary) (3.1) and Validation (activities) (3.3.)

  • Commenters generally agreed with the inclusion of validation and the proposed measurement, however, there was a comment that perhaps the results could be visualized in a way that made this indicator easier to understand and digest.

Validation (progress) (3.2)

  • With limited comments from the community, the sample graph chart seems sufficient;  organized by month, but with weekly detail also included.
  • There was also a suggestion that percent change would allow for greater visual consistency across publishers as the temptation will be to compare publishers’ charts, and different scales would be misleading.
  • There was also a question around how “credit” will be given in a consistent and fair manner for those organizations with less errors.

Next Steps

The consultation pages for Timeliness, Frequency, and Validation will be left open for additional comments following the official close of the consultation, however consultation pages will not be actively monitored or moderated.

The next phase of the consultation is on Data Completeness and is open via IATI Connect from 28 March - 29 April 2022.

Files

Be the first one to comment


Please log in or sign up to comment.