Data Quality Index Consultation - sub-section New Measures

Instructions for submitting your feedback
  1. Access each section by clicking on the buttons below, or download the attached full report.
  2. Share your feedback on the New Measures sub-sections of your choice through the comment-box below.
  3. Consider the guiding questions for each part, and please refer to the indicated numbers in your comments.

4.1 Basic validation measure

  • Do you agree that a publishing statistic for the Validator should be included as a dimension in the DQI index?

4.2 Data Complementarity 

  • What additional measures would you suggest adding here?


4.3 Trust in Data

  • Are there any measures that you would suggest adding that would help to assess data quality issues that impact trust in the data? 

BACK TO MAIN DQI-PAGE
​​​​​

Files

Comments (14)

Amy Silcock

Feedback on new measures so far TL:DR

  • Support to use validator metrics in the DQI
  • Further discussion needed on how data complementarity should be measured. Further ideas are welcome
  • Hesitancy to use trust in the DQI, it would be useful but super hard to measure. Variety of ways this could be assessed, further suggestions welcome.
Herman van Loon

4.1 Basic validation measure

  • Do you agree that a publishing statistic for the Validator should be included as a dimension in the DQI index?

    Yes. What is missing though in the Data Validator is the check on the  existence of the IATI identifiers being used. Since the existence of the IATI identifiers is critical for the quality of traceability data, this should i.m.o. be fixed as soon as possible.     
Herman van Loon

4.2 Data complementarity

Ability to assess predictability:
is an important measure not only to see if an organization follows through on their commitments, but also to monitor the progress of an activity.

Usefulness of budgets to inform budget planning:
please take for this metric also into account that many activities will have a lifespan shorter than 3 years.

Ability to assess funds flowing into and out of an organisation based on the data they report: 
please take into account that the bilateral donors at the beginning of the funding chain will have no incoming transactions, since the funding comes from the tax payer.

 

Herman van Loon

4.3 Trust in data

  • Are there any measures that you would suggest adding that would help to assess data quality issues that impact trust in the data? 

    Assess whether the publishers data can successfully be used by third party IATI applications.  
Petya Kangalova - IATI Secretariat Moderator

@herman can you clarify what you mean here? Do you mean at least IATI schema valid files so that tools can pick up the data or something else?

Herman van Loon

I am not referring to an automated check here. An automated check can check only check so much when it comes to the usability of the data. The real test i.m.o. is whether or nor the data can be used in real applications. I am not sure how to measure this easily, but many IATI data pilots have been done, many highlighting  room for improvement of IATI data quality for specific publishers.  It would be nice if these insights could also be taken into account in the dashboard.

leo stolk

4.1 Basic validation measure

  • Do you agree that a publishing statistic for the Validator should be included as a dimension in the DQI index?

Yes!

4.2 Data Complementarity 

  • What additional measures would you suggest adding here?

this is a complex proposition. agree on assessment of predicatability,
I have problems with the multiyear availability for three years as criteria. Specially for INGO's funding local NGO actors. Accuracy of budget previsions depend on strengths of partnerships, strength of alliances, proposal quality and donor assessment and verdicts and their funding availability, monitoring and evaluations. All in all a too complex mix of factors. Even for bi-laterals budget planning is fluid as it more and more depends on domestic politics (see DFID FCDO) and politics in recipient country (See Mozambique 2.2 billion illegal debt scandal). 
agree on  assessing incoming and outgoing flows

4.3 Trust in Data

  • Are there any measures that you would suggest adding that would help to assess data quality issues that impact trust in the data? 

Agree with suggestions.  Possible addition is assess how organizations react to feedback from data users. Important if a feedback mechanism gets integrated in future tooling.

Evgenia Tyurina

Hello, everyone. Here is the ILO's feedback:

4.1 Basic validation measure

Do you agree that a publishing statistic for the Validator should be included as a dimension in the DQI index?

Agree

4.2 Data Complementarity 

What additional measures would you suggest adding here?

We do not have any additional proposals, but we would like to comment on the “usefulness of budgets to inform budget planning” measure proposed under this section. As already mentioned under “Availability – financial” section, the planning in the ILO is done on a biennial basis, therefore publishing of “budgets for next three years” is not a realistic measure for us. Additionally, it is obviously not possible to provide “budgets for the following 3 years” for activities that are shorter than 3 years and, therefore, this measure should take into account the duration of the activity.

4.3 Trust in Data

Are there any measures that you would suggest adding that would help to assess data quality issues that impact trust in the data? 

We do not think that trust in data is something that can be measured by a signal indicator or a set of specific indicators. In our view, whether to trust the organization’s data or not can be decided by looking at the whole set of other measures that are part of the IATI Data Quality Index.

Anna de Vries

4.1 Basic validation measure

Do you agree that a publishing statistic for the Validator should be included as a dimension in the DQI index?

  • Yes I agree, would also suggest sending a report of these statistics to the organisations contact email on a more regular basis since many organisations are not yet familiar with the validator.

4.2 Data Complementarity

What additional measures would you suggest adding here?

  • With predictability it might also be interesting to know if there is a large difference between planned and actual start/end dates, and whether result targets have been met.
  • For complimentary data, assigning budget percentages to SDG goals would be a good addition
  • For results data, having both output, outcome & impact indicators is an additional layer of depth
  • Measuring whether internal related activities make sense and whether the organisations data fits within the data model & structure of IATI (besides only traceability)

4.3 Trust in Data

Are there any measures that you would suggest adding that would help to assess data quality issues that impact trust in the data? 

  • Measuring whether or not improvements are made should also be based on how many improvements are needed in the first place and the severity of the issues (if an organisation is already doing well then fewer improvements are needed which should not count against them).
  • Measuring clarity of data would also be good, for example not using complex internal codes, unreadable explanations, or descriptions that are too long, reporting their transactions in a way that makes sense in terms of dates & level of detail. And having measurable results (numerical and percentages) instead of only qualitative.
  • With trust, do you mean trustworthiness or how much it is already trusted? Because in the second case you could also see if it can be measured how much this data is being used by stakeholders on the main portals.
Athira Lonappan

Basic Validation Measure

Do you agree that a publishing statistic for the Validator should be included as a dimension in the DQI index?

Yes, I think it should be included. Also, instead of having IATI Validator as a separate tool, it could be integrated in the process of uploading the file. So a publisher when uploading the data has to first validate the info through the tool and once the IATI Validator shows 0 errors (warnings can be kept optional) can proceed to upload the data. 

Data Complementary 

What additional measures would you suggest adding here?

A reference to already existing data which would match the sector code and the geographic location for a new user would be helpful. Example : If Fields Data wish to publish data of Mbale, a city in Uganda contributing in sector 11220 (Primary Education) , then it would be good to have reference to an existing file with a similar sector-geography combination.

Trust in data

Are there any measures that you would suggest adding that would help to assess data quality issues that impact trust in the data? 

Peer-reviews - if two or more organizations are mentioned in one activity, then the others could validate the information.

 

Otto Reichner

4.1. Basic validation measure

  • Do you agree that a publishing statistic for the Validator should be included as a dimension in the DQI index?

Yes, we agree in principle, as generally any IATI submission should be compliant with IATI validator. Please do provide more details on the methodology and calculation to enable decision making.

4.3. Trust in data

  • Are there any measures that you would suggest adding that would help to assess data quality issues that impact trust in the data?

Honestly none of the proposals do represent fair and measurable KPIs for trust.

Unfortunately, we also have not come up with a good KPI proposal for this measure, so we recommend to not include it for now.

Marie Maasbol

Please see below the feedback from the Commission (FPI, DG NEAR, ECHO and INTPA)

4.1 BIG VALIDATION MEASURE

Do you agree that a publishing statistic for the Validator should be included as a dimension in the DQI index?

  • The Commission would like to know more about what is meant to include the Validator as part of the Index? It is the understanding of the Commission that the focus of the Index is “what” data should be published, whereas the purpose of the Validator is to allow users to check the quality of what is published. Therefore, the Commission would propose that the Validator is a connected tool for the Index, in which it is used to support publishers in improving the data quality assessed in the Index.

4.2 DATA COMPLEMENTARITY

What additional measures would you suggest adding here?

  • The Commission would like to re-emphasise that the Index should take into account the budgetary time-frames of each publisher while assessing how forward-looking the published information is.

4.3 TRUST IN DATA

Are there any measures that you would suggest adding that would help to assess data quality issues that impact trust in the data?

  • With an increased focus on the use of data rather than on a simple transparency objective, the Commission would like to advocate for a measure that can assess whether the publishers’ data can successfully be used by third party IATI applications.
Alex Tilley
  • Do you agree that a publishing statistic for the Validator should be included as a dimension in the DQI index?

Seems reasonable

  • What additional measures would you suggest adding here?

 

  • Are there any measures that you would suggest adding that would help to assess data quality issues that impact trust in the data?

Validation by country offices/missions/embassies for major donors.

Pelle Aardema

4.1 - Basic validation

  • Do you agree that a publishing statistic for the Validator should be included as a dimension in the DQI index?

Yes! With the addition that the validator should check the validity of identifiers.

4.2 - Data complementarity

  • What additional measures would you suggest adding here?

Predictability and the assessment of incoming/outgoing flows sound very useful. The budget assessment should take into account the different practices of different kinds of organisations, thus appears to become really complex.

Assessment of the presence of baselines, targets and actual values - and if they're all of the same type (numeric, ...) - for a single indicator, taking into account the indicator period.

4.3 - Trust

  • Are there any measures that you would suggest adding that would help to assess data quality issues that impact trust in the data? 

The proposed measures appear to measure the good intentions of the publisher. Not sure if that says anything about the trustworthiness of the data.


Please log in or sign up to comment.