Data Quality Index Consultation- sub-section 1.1 Timeliness - Frequency


Instructions for submitting your feedback

1. Read through the proposed methodology for this measure and / or download the attached PDF at the bottom of this page; 

2. Share your feedback through the comment box below, consider the guiding questions in your comments and include the question number in your response;

3. And finally you can suggest track-changes or add comments directly on the specifics of the methodology of the Timeliness and Validation - go to this DQI Live-Editing-Page


Proposed Measures - 1.1 Timeliness - FREQUENCY

Please find below the proposed methodology for this measure. Only active activities* will be assessed in the Timeliness measures. 

*Active activities refer to activities which have an actual-start-date in the past, and no actual-end date or a planned-end-date in the future. If actual start dates are not present, the planned start date will be used

DEFINITION assess how often a publisher's data is updated, using transaction-date elements. 

OBJECTIVE / EXPLANATION 

  • Data quality objective: frequent and predictable updates.
  • Based on the methodology, this measures whether a publisher has updated one transaction of any type. As such, it is not an assessment of whether substantive updates have been made, but rather whether any more recent transactions have been added. 
  • As such, this will motivate publishers to publish at least one transaction with a more recent date every time they update their data.
  • Bigger picture, the goal is to motivate more frequent and predictable updates.

OUTPUT

Categorisation of frequency:

  • Weekly;
  • Monthly;
  • Quarterly;
  • Six-monthly;
  • Annual;
  • Less than annual.

 

METHODOLOGY - calculate frequency assessment
For publishers of 1 year or more
  • Weekly = average of 4 updates a month across the past 12 months;
  • Monthly = 10 out of 12 of the past 12 full months;
  • Quarterly = all of the past 4 full quarters; 
  • Six-monthly = both of the last 6 month periods; 
  • Annual = 1 of the last 12 months.
For publishers of 6 months or more
  • Weekly = average of 4 updates a month across the past 6 months;
  • Monthly = 5 out of 6 of the past 6 full months;
  • Quarterly = 2 of the past 2 full quarters;
  • Six-Monthly = 1 of the last 6 months.
For publishers of 3 months or more
  • Weekly = average of 4 updates a month across the past 3 months;
  • Monthly = 3 out of the past 3 full months;
  • Quarterly = 1 out of the last 3 months.
For publishers of less than 3 months
  • Weekly = average of 4 updates a month across the past 1 month;
  • Monthly = 1 out of the past 1 month.

Note: future transaction dates will be discounted from the measure as they contradict IATI rules.


Please find below a visualisation for this proposed measure. Do note that this has been created to help participants picture what the DQI could look like. It is not final, nor part of the proposed methodology:


Guiding questions - please refer to the question number when you respond via the comment box below!

1. How many and which frequency categories should be used?

  • Are the ones listed still useful or should they be reduced, or others be added? Would it be useful to have a category for 'every two years' and 'less than every two years'?

2. How many updates should be required to qualify for each category?

  • For data users, predictability is key, so taking an average provides a misleading assessment of the frequency of publication.
  • Should a publisher get credit for publishing weekly if they have published 4 updates a month but all in one week? Or should they be required to update their data 1 time per week?
  • Should a publisher get credit for publishing monthly if they have only published in 10 out of the past 12 months (or 5 out of the past 6 months) or should they be required to publish in all 12 months?

3. Should there be 4 categories used to calculate the frequency assessment as presented, or should this be reduced?

  • For example, should there only be 2 categories, to include 'For publishers of 1 year or more' and 'For publishers of less than 1 year'? Would this make it easier to understand how each type of publisher is being assessed?

4. Do you agree that future transaction dates should be ignored from the assessment?

  • In other words, a publisher will not get credit for updating their transactions if they are for future dates.

5. Should publishers get credit for publishing transactions with a transaction value of 0?


Webinar

For each discussion, the IATI Secretariat will organise a webinar to explain the proposed methodology, answer questions and further explain how to engage.

  • Please find an overview of the most frequently asked questions of the Timeliness and Validation webinar here.
  • Missed the DQI Webinar on Data Completeness held on March 30? Watch the recordings here or read the summary here!

BACK TO MAIN DQI-PAGE

Files

Comments (11)

Anna Whitson - IATI Secretariat Moderator

Dear members of the IATI community,

As moderator of this consultation on the forthcoming IATI Data Quality Index, a warm welcome to you all! Thank you in advance for your inputs, which will no doubt provide invaluable as we work toward a DQI that supports our publishers to better understand and improve the quality of the data they publish. On behalf of the Secretariat, again, welcome!

-Anna Whitson; Outreach, Partnerships and Engagement Specialist, IATI Secretariat

Yohanna Loucheur

Q1 & 3: we should reduce the number of categories. The "Six-Monthly" one seems overkill. There is a commitment to publish frequently, so creating a different way to assess publishers "every two years" doesn't make much sense. The category "Less than annual" should become "Inactive" if we're serious about timeliness.

Q2: no evidence is provided to support the statement that "predictability is key" for data users. What kind of predictability are we talking about: predictability in publishing data (of whatever data is available), at fixed intervals, or predictability in project implementation (thus delaying publication if needed so that the data is included in the file)? The model implicit in the proposed DQI (in the timelag measures) is that programs are implemented in a fully predictable and/or linear fashion and therefore data should be available at predictable and evenly spaced moments. Many donors/types of funding do not conform with this model. 

Q4: yes, since these do not align with the standard

Q5: intuitively, no, but willing to consider counter-arguments.

Anna Whitson - IATI Secretariat Moderator

Hi Yohanna - apologies again for delayed response due to technical error on my part. In reference to your question on why predictability is key for data users, predictable and timely publishing is especially important for data users from partner country governments - predictable and timely information ensures that they can be confident about the amounts and the timing of disbursements, especially in regards to using this information for their own planning and budgeting purposes.

Yohanna Loucheur

[~582] 

Hi Anna

Thank you for your response.  Yes, I understand that predictability of funding is important for partners. However, predictability of *publishing* isn't quite as crucial, at least not enough to design a whole indicator around it. 

Regarding "active activity", I cannot believe that I have to say it again - I'm sorry but this is getting ridiculous: an actual-end-date cannot be in the FUTURE, by its very definition. This is from the 2.03 IATI standard: "11.1.5: The actual end date of the activity must not be in the future."

Yohanna Loucheur

What does "active activity" mean? I checked the methodology paper (the editing live page) but the term "active activity" does not appear on it. Since this determines the universe for the assessment, and since it is not a term from the IATI Standard, it would be important to define it clearly at the start.  

Anna Whitson - IATI Secretariat Moderator

Hi Yohanna - thanks for your comments! Apologies for the lateness of this comment as there was a glitch (or operator error!) which left this comment unpublished. I would point you to the FAQ section of the consultation: https://iaticonnect.org/FAQ-DQI-Timeliness-Validation

Here you will find the definition for “active activities,” which refers to activities that have an actual-start-date in the past and an actual-end-date (if published) in the future. If the actual start and end dates are not present, the planned start and end dates will be used.

leo stolk

Why only looking at updated transaction?  Adding a planning or actual on a result indicator is also an update, so is improved or updated description, participating organisations etc.
Q1  suggest to reduce to four categories only:  Weekly; Monthly; Quarterly; less than quaterly. 
Q2  suggest to be straight and simple; only value met criteria in full. 
Q3  understand the four categories;  do not think reducing categories will increase understanding
Q4 Agreed
Q5 Do not see the value of '0' transaction value, so No.

Sarah Scholz

Apologies for our lateness:

Q 1&3: Tend to agree with Yohanna on 6-monthly and using "inactive" for less than annual updates. Suggest weekly, monthly, quarterly, annually, inactive.

Q2: That said, will there be some compromise or flex dates incorporated? For example, if a publisher updates 366 days after the previous update (missing the "annual" by one day), they'll become listed as inactive for a full calendar year unless the next update is within 363 days? Similarly, while it doesn't seem that publishing 4 updates in one week each month should count as "weekly", how strict will that week/month/quarter/year be?

Q4: agree 

Agree with Leo that updating results might count, but I'm not sure if improving data quality of the other fields is really a substantive update to the data. I recognize you're not considering substantive-ness, but rather whether a single update was made, so I'm not sure where this line should be drawn, but improving fields seems different.

 

Anna Whitson - IATI Secretariat Moderator

Thanks, Sarah, for your comments! On your Q2, I'm asking [~494] and [~566] to come in and provide more information to you on how flexible this will be.

Amy Silcock

Thank you for your comments Sarah. On Q2 this is something we'd like to hear from the community on. What level of flex and aggregation is appropriate? 

For the 'active publishers' assessment, the proposal is that any organisation who hasn't updated any part of any file in a 363 day time period will be marked as 'inactive'. As soon as they do update a file they'll revert back to active. So, if an organisation misses an update by one day they'll only be marked as inactive for one day (not a whole year).

As for the other assessments, weekly, quarterly etc. There's a much greater room for flexibility and the risk that organisations will be assessed as having a lower frequency of publishing for longer. Would be good to hear further thoughts on how this should be handled. At the moment we're not proposing to capture in which week an organisation updates their data, but it would be helpful to distinguish publishers who update their data more often (presumably they wouldn't score any higher on the summary section for this, as it's a nice to have but not expected).

 


Please log in or sign up to comment.