Discussion

Data Quality Index - Section: Timeliness

IATI Secretariat
IATI Secretariat • 1 September 2021
Data Quality Index Consultation - sub-section Timeliness

Instructions for submitting your feedback
  1. Access the Timeliness-section by clicking on the button below, or download the attached full report.
  2. Share your feedback through the comment-box below, and consider the guiding questions in your comments.

3.1 Timeliness

  • Do you agree that  frequency and time lag are good assessments of timeliness and should be retained?
  • Do you agree with the additional measures of spend data updates and consistency as a measure of timeliness? 
  • Do you agree with the additional disaggregation assessing any updates in IATI activity, splitting publishers into active and inactive?
  • Do you have any other suggestions?

BACK TO MAIN DQI-PAGE

Files

Comments (15)

Amy Silcock
Amy Silcock

Feedback on timeliness so far TL:DR

  • Support for a wider range of elements counting as an ‘update’ to an activity. This includes updates to results, new activities being added, provider/receiver/participating org information, title and narrative changes. Only active activities will be monitored.
  • The proposed consistency of spend measure is to check if spend (disbursements and expenditures) have been published each quarter and whether this is consistent for all activities. For instance, do active projects have spend data dated at the end of each week/month/quarter and is this done on the same dates in the month. It will be good to hear from publishers on their practices. This is also what we mean by alignment of date.
  • Support for marking publishers, rather than files, as active or inactive. Option for this to be added as a highlight on the DQI visualisation.
Herman van Loon
Herman van Loon
  • Do you agree that  frequency and time lag are good assessments of timeliness and should be retained?

    Yes
     
  • Do you agree with the additional measures of spend data updates and consistency as a measure of timeliness?

    Possibly. Could you give some examples? Usually only a subset of all activities will be changed. Finished activities will not be changed anymore.
     
  • Do you agree with the additional disaggregation assessing any updates in IATI activity, splitting publishers into active and inactive?

    The current proposal has i.m.o. some problems. The text seems to suggest you propose to do this on a XML activity file by file basis. In the Netherlands case, IATI XML activity files are split on a two yearly basis. In other words we split activity files across  time instead of countries (to avoid duplication of activities in multiple XML files). The older files will not change any more since most activities will be finished. This would mean that the Netherlands data might be flagged as inactive, where they are not. Only the finished activities will be inactive. 

    So I suggest to do this test on a publisher by publisher instead of a file by file basis or skip the test at all.
     
  • Do you have any other suggestions?

    Consistently assess publishers activity data as a whole instead of on a file by file basis. The splitting of data into multiple XML activity files is a  measure to avoid technical problems when processing data. It does not really represent something functional. 
Petya Kangalova
Petya Kangalova Moderator

@herman  to your specific questions.

On the specific measure for spend data updated consistently will be checking if spend (disbursements and expenditures) is published each quarter and whether that is consistent for all activities. For instance, for your active projects do you add spend data (disbursement and expenditure) at the end of each week/month/quarter and is that done on the same dates in the month. It will be good to hear from publishers on their practices. This is also what we mean by alignment of date. 

The timeliness measure will only look at active activities. Organisations won’t be penalised for having ‘closed’ activities

 

On the split of activities into active and inactive, the suggestion is to do the assessment on a publisher by publisher basis, looking at all activities included in all publisher files. It won’t be the file being flagged as inactive, but the publisher.

 

leo stolk
leo stolk
  • Do you agree that  frequency and time lag are good assessments of timeliness and should be retained?

Yes but only look at transactions, frequency and time lag should also look at updates in result entries, updates in recipient activity ID, provider activity ID, title and description narrative changes.  improving these aspects of an activity may be as important as updating a transaction.

  • Do you agree with the additional measures of spend data updates and consistency as a measure of timeliness? 

possibly yes, again update of non spending elements may be as important and should be looked at in my opinion

  • Do you agree with the additional disaggregation assessing any updates in IATI activity, splitting publishers into active and inactive?

Agree, suggest turning the active publishers in shades of green, and leave the less actives without colour.

  • Do you have any other suggestions?

see above my plea to look at more than transactions alone. 

Petya Kangalova
Petya Kangalova Moderator

Thank you for the suggestion. We would welcome feedback from the community, specifically around checking updates not only from transactions but also the other elements as suggested above from you!

Yohanna  Loucheur
Yohanna Loucheur

Agree with others that we should not look only at changes in financial data - updates in other aspects of activity information can be just as important. Why not use the time-stamp on the IATI file, this provide the last-updated information?  

Also agree with the Netherlands about closed activities. These files are no longer updated but are still very useful, they should not be classified as inactive. Let's make sure the tests are properly designed (e.g. check the status of the activities in the file to determine whether updates are expected). 

On frequency: a "weekly" category could be added to recognize publishers updating their data more frequently (several now do it daily).

Additional measures: it is not clear to me what the new measures aim to capture in addition to what is already included. A consistent update to important data such as spending - isn't this the same as disbursement? What would be expected?  Similarly, what are we looking for regarding transaction dates alignment? 

Petya Kangalova
Petya Kangalova Moderator

Yohanna Loucheur  thanks for the suggestions about adding potential "weekly" category. We will take that into consideration during the methodology phase. In the IATI community the last-updated-datetime was also discussed but was not seen as a reliable measure as it is not correctly updated in all publisher files.

Also noted about organisations not being penalised for having ‘closed’ activities. 

On the consistency of updates the proposal is to check if spend (disbursements and expenditures) is published each quarter and whether that is consistent for all activities. For instance, for your active projects do you add spend data (disbursement and expenditure) at the end of each week/month/quarter and is that done on the same dates in the month. It will be good to hear from publishers on their practices. This is also what we mean by alignment of date. 

 

Marie-Line Simon
Marie-Line Simon
  • Do you agree that  frequency and time lag are good assessments of timeliness and should be retained?

Also agree with the remarks above: any update is important, not only financial. An interesting indication would be that any data change is dated and highlighted.

  • Do you agree with the additional measures of spend data updates and consistency as a measure of timeliness? 

Agree with the Netherlands about this: only a part of the information will and can get updated. An absence of change does not mean the information is outdated.

  • Do you agree with the additional disaggregation assessing any updates in IATI activity, splitting publishers into active and inactive?

As for us, Belgian ngo's, registering data in IATI has become mandatory for all Development cooperation orgs receiving state donations as from 2017. We work on a 5-year basis, which allows us to plan all activities and expenses over 5 years. The (legal) minimum frequency has been set to once a year (April) for the past activity year, which means we usually publish data only once it is definitive and approved. Therefore:

1- our follow up of our IATI register is in line with our planning and reporting processes, and its timeliness is limited by the one of our reporting (only once a year) - we cannot register real-time information. And we do not expect to record accurate data more than once a year.

2- the process of filling IATI in is generally in charge of someone who does not have the time and posibility to centralize such data more than once a year.

3- What is being published will supposedly not change, as we register only planned and reported data. So labelling it as 'inactive' is not correct.

  • Do you have any other suggestions?

Donors (especially official/state donors) should be better informed of what is expected from IATI in terms of timeliness.

A distinction should be made between the types of reporting organizations, as there is a large set of capacities, expectations, willingness, etc.

Anna de Vries
Anna de Vries

Do you agree that  frequency and time lag are good assessments of timeliness and should be retained?

  • Frequency is a good assessment if focused on changed data for e.g. transactions, results added, or newly added activities (last update date is not always a good measurement since some publishers have daily automated uploads without any new data).
  • Timelag is unclear, does only count if the transaction dates are all different? In case of expenditures it is often advised to do a cumulative figure instead of reporting too much detail, which will improve clarity but seem like it is lower data quality in terms of time lag..

Do you agree with the additional measures of spend data updates and consistency as a measure of timeliness? 

  • Spend data updates: yes, good addition
  • Consistency: does this take into account that it should only measure for Active Projects/ activities with activity status 1 or 2?

Do you agree with the additional disaggregation assessing any updates in IATI activity, splitting publishers into active and inactive?

  • Agree on principal but need to decide what to do for publishers who no longer report since all their project activities have closed, since this does not necessarily mean their historical data is unreliable

Do you have any other suggestions?

  • More thought is needed in how to treat historical data, a publisher can also improve their overall quality by adding additional track record information rather than only adding more recent data. 
  • Would there be a way to assess the data quality based on the project cycle as well, e.g. if activities have been properly ‘closed’ and if so, if all project years include the expected data (transactions, budgets)
Evgenia Tyurina
Evgenia Tyurina

Hello everyone. Here is the ILO's feedback:

Do you agree that frequency and time lag are good assessments of timeliness and should be retained?

Agree

Do you agree with the additional measures of spend data updates and consistency as a measure of timeliness?

Regarding the consistency of updates measure the document indicates that it “will be assessing whether the updates to transactions are made at a consistent interval”. Further clarification would be appreciated on this point. The transactions of certain types (e.g. Incoming Funds, Disbursements ) may not happen very often. In cases where there are no new transactions under certain activities during the interval (reporting period) and, therefore, no need for an update, how this will be taken into account for the assessment of the consistency of the interval? The document also indicates that the consistency of updates measure “will check if transactions (disbursement and expenditure) are updated consistently for all activities and whether the transaction dates align”. It is not very clear what does this  mean in practical terms. For transactions that are published one by one (not grouped) and have different transaction dates what would the “alignment” of transaction dates mean?

For spending data, further clarification would be helpful. Would an update of spending data for at least one of the activities in the file be enough or will it be monitored activity by activity? If the latter, there might be cases where no spending is recorded under an activity during the reporting period, which is a valid reason not to update the spend data. How will these cases be taken into account for the assessment?

Do you agree with the additional disaggregation assessing any updates in IATI activity, splitting publishers into active and inactive?

Generally yes. However, it should be acceptable not to have any updates for activities with “closed” status.

Do you have any other suggestions?

To add the last update date at the top of each activity page on d-portal.

Question: how often the timeliness measures will be monitored?

Athira Lonappan
Athira Lonappan

Do you agree that  frequency and time lag are good assessments of timeliness and should be retained?

Maybe, only frequency can be retained. It would be worth reviewing if people actually check the time lag statistics tab. A combination of frequency and last updated date would do the trick.

Do you agree with the additional measures of spend data updates and consistency as a measure of timeliness? 

If in case an activity is closed would the spend data still be updated? Would this according to IATI be marked as an inactive user? 

Do you agree with the additional disaggregation assessing any updates in IATI activity, splitting publishers into active and inactive?

Instead of splitting publishers into active and inactive, I would rather recommend having activity files which have not been updated to be flagged as inactive, the reason being, if publishers have multiple activity files under their account and if only few files are regularly updated, it would be better to conclude that few files are inactive and not the user.

Do you have any other suggestions?

Can publishers be notified prior to the recurring date on which data has to be updated depending upon the frequency? If the publishers fail to do so then automatically mark the file as not updated (red flag). 

 

Otto Reichner
Otto Reichner
  • Do you agree that frequency and time lag are good assessments of timeliness and should be retained?

Yes, we do agree that we already have good measures for timeliness and frequency.

  • Do you agree with the additional measures of spend data updates and consistency as a measure of timeliness? 

Yes, we agree that this could be value add, but this will naturally drive all publishers to ONLY publish active/ongoing activities. WFP currently publishes also closed/finalized activities for several years.

  • Do you agree with the additional disaggregation assessing any updates in IATI activity, splitting publishers into active and inactive?

Like above, this makes sense if intention is to drive publication of ONLY active ongoing activities.

  • Do you have any other suggestions?
  • We suggest to always aim at monthly updates as frequency targets.
Marie Maasbol
Marie Maasbol

Thank you for this consultation. Please find below the feedback from the Commission (FPI, DG NEAR, ECHO and INTPA).

Do you agree that frequency and time lag are good assessments of timeliness and should be retained?

  • The Commission agrees that frequency and time lag should be retained as assessments for timeliness and encourages the emphasis on timely and up-to-date information.

 Do you agree with the additional measures of spend data updates and consistency as a measure of timeliness?

  • The Commission agrees with this proposal on the condition that such assessments can only be executed for activities that are on-going. Closed activities will not be updated and an organisation should, therefore, not be penalised on these grounds.
  • Additionally, the Commission would like to flag that updates to files should be done when necessary, and not for the purpose of scoring well on this assessment. We would, therefore, like to know more information on what is meant by consistency/consistent interval?

Do you agree with the additional disaggregation assessing any updates in IATI activity, splitting publishers into active and inactive?

  • The Commission agrees with this proposal, but would like to re-emphasise the point that closed activities should not be considered for this assessment, as no updates will be executed for closed activities. 
Alex Tilley
Alex Tilley
  • Do you agree that frequency and time lag are good assessments of timeliness and should be retained?

Yes, agree

  • Do you agree with the additional measures of spend data updates and consistency as a measure of timeliness?

Yes, agree

  • Do you agree with the additional disaggregation assessing any updates in IATI activity, splitting publishers into active and inactive?

Yes, agree

  • Do you have any other suggestions?

These suggestions are all good and will help improve the Timeliness of IATI data. I would hope to use them in the Timeliness element of the Aid Transparency Index.

Pelle Aardema
Pelle Aardema

3.1 - Timeliness

  • Do you agree that  frequency and time lag are good assessments of timeliness and should be retained?

Yes, agree with the suggestions above that the measure should take into account other data elements as well.

  • Do you agree with the additional measures of spend data updates and consistency as a measure of timeliness? 

Updates of non-spending elements may be as important.

  • Do you agree with the additional disaggregation assessing any updates in IATI activity, splitting publishers into active and inactive?

It may be useful to use more 'shades of green' - as Leo proposed

  • Do you have any other suggestions?

Take into account multiple elements, not only transactions.

I think the last-updated-date is a very good indicator to show if anyone has looked at the data recently. An absence of change does not mean the data is outdated.


Please log in or sign up to comment.