Data Quality Index Consultation - sub-section 1.2.1 Timeliness - Timelag (all transactions)


Instructions for submitting your feedback 

1. Read through the proposed methodology for this measure and / or download the PDF at the bottom of this page;  

2. Share your feedback through the comment box below, consider the guiding questions in your comments and include the question number in your response;

3. And finally you can suggest track-changes or add comments directly on the specifics of the methodology of the Timeliness and Validation measures - go to this DQI Live-Editing-Page


Proposed Measures - 1.2.1 Timeliness - TIMELAG - ALL TRANSACTIONS

Please find below the proposed methodology for this measure. Only active activities* will be assessed in the Timeliness measures.  

*Active activities refer to activities which have an actual-start-date in the past and an actual-end-date in the future. If actual start and end dates are not present, the planned start and end dates will be used.

DEFINITIONassess how recent a publisher's transaction data is at the point it is published.

For instance, a publisher may refresh their data monthly, but the refreshed data is in fact three months old. Alternatively a publisher may refresh their data only once a year, but when they do it contains current data that is less than one month out of date.

OBJECTIVE / EXPLANATION

  • Data quality objective: When data is updated, it contains recent data.
  • Based on the methodology, this is measuring whether the data a publisher has published contains transactions (of any type) that have occurred recently.
  • As such, this will motivate publishers to ensure that their updated data contains transactions with recent transaction dates.
  • Bigger picture, the goal is to motivate publishers to ensure that when they update their data, they are including data that is as recent as possible.

OUTPUT

Categorisation of how many months in areas the data is:

  • One month in arrears;
  • One quarter in arrears;
  • Six months in arrears;
  • One year in arrears;
  • More than one year in arrears.

Note: visualisation to show the month for which the most recent data was last updated in

 

METHODOLOGY

Count the number of transactions that took place in the last 12 months.

Collect the last update date.

Calculate arrears assessment by comparing last update date to transaction dates:

  • One month in arrears = transactions with transaction dates for at least 2 of the last 3 months;
  • One quarter in arrears = transactions with transaction dates for at least 1 of the last 3 months;
  • Six months in arrears = transactions with transaction dates for at least 1 of the last 6 months;
  • One year in arrears = transactions with transaction dates for at least 1 of the last 12 months;
  • More than one year in arrears = no transactions with transaction dates in the last 12 months.

 Note: future transaction dates will be discounted from the measure as they contradict IATI rules.


Please find below a visualisation for this proposed measure. Do note that this has been created to help participants picture what the DQI could look like. It is not final, nor part of the proposed methodology:


Guiding questions - please refer to the question number in your response via the comment box below!

1. Do you have any suggestions for a more intuitive name for this measure instead of Timelag? Perhaps 'Recency'?

2. How many and which arrears categories should be used?

  • Would it be useful to capture arrears of more than 1 year, e.g., more than two years in arrears?

3. How many updates should be required to qualify for each category?

  • Do you agree with the number of updates needed per category? If not, how would you change them?

  • Should a publisher get credit for having data one month in arrears when they have achieved this in only 2 of the past 3 months or should 3 months be required?

4. Do you have any suggestions on how this measure could be visualised?

5. Should publishers get credit for publishing transactions with a transaction value of 0?


Webinar

For each discussion, the IATI Secretariat will organise a webinar to explain the proposed methodology, answer questions and further explain how to engage.

  • Please find an overview of the most frequently asked questions of the Timeliness and Validation webinar here.
  • Missed the DQI Webinar on Data Completeness held on March 30? Watch the recordings here or read the summary here!

BACK TO MAIN DQI-PAGE


Files

Comments (6)

Anna Whitson - IATI Secretariat Moderator

Dear members of the IATI community,

As moderator of this consultation on the forthcoming IATI Data Quality Index, a warm welcome to you all! Thank you in advance for your inputs, which will no doubt provide invaluable as we work toward a DQI that supports our publishers to better understand and improve the quality of the data they publish. On behalf of the Secretariat, again, welcome!

-Anna Whitson; Outreach, Partnerships and Engagement Specialist, IATI Secretariat

leo stolk

Suggest to improve definition of active activity: often activities will have a planned and actual start date and only a planned end date.  So:
*Active activities refer to activities which have an actual-start-date in the past and a planned or actual-end-date in the future. If actual start and/or end dates are not present, the planned start and end dates will be used.

leo stolk

I do not understand the choice of looking only at the transactions. updating result element or other elements should be seen as an update too. 
This approach is too focused on financial content of an activity.

Thea Schepers

I haven't had the time to give as much detailed feedback as I had wanted but the biggest problem I have with this aspect of data quality index is the focus on transactions. I asked this in the webinar and the reply had to do with technical restrictions, but it is not difficult to look at all data and consider any update an actual update, and consider them all equally. If someone updated their description of the activity, they have actually updated the activity.

Only looking at transactions limits the validity of the index itself. It will say more of the type of activity and the type of organisation than of data quality. This is such a fundamental flaw that it actually has me concerned about the usefulness of the whole index.

I didn't know this at the time of the webinar, but colleagues have since told me that this has been raised in the first phase of this consultation as well. It would really help if the input from people who give their time to this project, actually leads to improvements, especially if it comes from a wide range of people. If at all possible, I would really look at how to change course here.

Amy Silcock

As a point of clarity, during the last round of consultation there was a debate as this thread mentions that transactions are not the only way to assess if an organisation has updated their data. 

It was also the view that Timelag and Frequency were particularly pertinent to transaction data, and these measures are useful to data users. Hence, these two measures look at the latest transaction dates.

An additional assessment was proposed, and has been included in this round of consultation to mark publishers as 'active' or 'inactive'. This one looks at the last-updated-date, which captures ANY change in an organisation's IATI reporting.

Sarah Scholz

Q1: Time lag makes sense

Q1-3: No strong feelings

Q4: Perhaps a range plot

Q5: If a new transaction, yes. If simply improving fields but not providing additional substantive informtion, I'm not sure. Either way, this should be consistent with other indicators.


Please log in or sign up to comment.