Data Quality Index Consultation

DATA COMPLETENESS

Sub-sections 2.3.1 - 2.3.5 Classification and 2.5 Sustainable Development Goals


Instructions for submitting your feedback

1. Read through the proposed methodology for this measure and / or download the attached PDF at the bottom of this page; 

2. Share your feedback through the comment box below, consider the guiding questions in your comments and include the question number in your response;


Proposed Measures - 2.3.1 CLASSIFICATION

Please find below the proposed methodology for this measure. 

DEFINITION Assess if activities include an OECD DAC sector classification. Note: other sector vocabularies are not counted in this measure.

OUTPUT

  • Percentage

 

METHODOLOGY 

Count the number of active activities that include a valid sector classification, using an OECD DAC Sector codelist (sector vocabulary 1 or 2). Divide by total number of active activities.

Valid sector classification rules:

  • Sector classification must be present at either activity or transaction level, not both
  • At activity level, the percentage for sector vocabularies 1 and 2 must each sum to 100
  • At transaction level, each transaction must have one sector code from either vocabulary 1 or 2


Proposed Measures - 2.3.2 CLASSIFICATION

Please find below the proposed methodology for this measure. 

DEFINITION Assess if activities include an aid-type classification.

OUTPUT

  • Percentage

 

METHODOLOGY 

Count the number of active activities that include either or both:

  • At least one valid default-aid-type element
  • All transactions have a valid aid-type element

Divide by the total number of active activities.

Valid aid-type rules:

  • At transaction level, no more than one aid-type code from each vocabulary must be used

Guiding question - please refer to the index number and question number when you respond via the comment box below!

1. The measure looks for the presence of the default-aid-type element, if not present should an aid-type element be present for every transaction?



Proposed Measures - 2.3.3 CLASSIFICATION

Please find below the proposed methodology for this measure. 

DEFINITION Assess if activities include a policy-marker classification.

OUTPUT

  • Percentage

 

METHODOLOGY 

Count the number of active activities that include at least one valid policy marker code from the OECD DAC policy marker codelist (vocabulary 1). Divide by the total number of active activities.

Valid policy-marker rules:

  • Policy marker elements using the OECD DAC policy marker codelist (vocabulary 1) must also contain a significance.


Proposed Measures - 2.3.4 CLASSIFICATION

Please find below the proposed methodology for this measure. 

DEFINITION Assess if activities include a finance-type classification.

OUTPUT

  • Percentage

 

METHODOLOGY 

Count the number of active activities that include either or both:

  • at least one valid default-finance-type element
  • all transactions have a valid finance-type element

Divide by the total number of active activities.


Guiding question - please refer to the index number and question number when you respond via the comment box below!

2. The measure looks for the presence of the default-finance-type element, if not present should a finance-type element be present for every transaction?



Proposed Measures - 2.3.5 CLASSIFICATION

Please find below the proposed methodology for this measure. 

DEFINITION Assess if activities include a flow-type classification.

OUTPUT

  • Percentage

 

METHODOLOGY 

Count the number of active activities that include either or both:

  • at least one valid default-flow-type element
  • all transactions have a valid flow-type element

Divide by the total number of active activities.


Guiding question - please refer to the index number and question number when you respond via the comment box below!

3. The measure looks for the presence of the default-flow-type element, if not present should a flow-type element be present for every transaction?

 


Proposed Measures - 2.5 SUSTAINABLE DEVELOPMENT GOALS

Please find below the proposed methodology for this measure. 

DEFINITION Assess if activities are mapped to the sustainable development goals or targets.

OUTPUT

  • Percentage
  • Percentage

 

METHODOLOGY 

Count the number of activities that are mapped to a sustainable development goal or target.

Check for the presence of:

  • a tag element using either the UN Sustainable Development Goals or Targets codelist (vocabulary 2 or 3), not both

Divide by the total number of activities.


Guiding question - please refer to the index number and question number when you respond via the comment box below!

4. Should both tag and sector SDG classifications be included? SDG guidance states that tag should always be used.


GO BACK TO MAIN DQI-PAGE 

Webinar

For each discussion, the IATI Secretariat will organise a webinar to explain the proposed methodology, answer questions and further explain how to engage.

  • Please find an overview of the most frequently asked questions of the Timeliness and Validation webinar here.
  • Missed the DQI Webinar on Data Completeness held on March 30? Watch the recordings here or read the summaryhere!
Files

Comments (13)

Michelle Levesque
Michelle Levesque

2.3.5 Flow type - I always struggle with this one because the flow type may not be relevant or known by the downstream recipients/implementers so to require it across the board would be unfair or misleading. I believe this may be one of those elements that either has to be nuanced depending upon the publisher type (discussed previously) or these classification have to be explicit in the contracts which is a very hard thing to require on a consistent basis. I've had a goverment donor explicitly tell us that the grant was not ODA related and thus shouldn't be published to IATI which made me question what I had understood to be that all grants would be ODA from our perspective.  We now have an algorithm that looks at where the project is and who gives us the money to classify it is ODA (code 10) or non-export OOF (21 becuase plain OOF - 20 has been withdrawn)  and I still don't know if we really have it right. As a non-OECD reporter,  I'm not really sure my understanding of their guidance is correct.  

 

The other elements suggested here seem reasonable across the board for all publishers.

Anna Whitson
Anna Whitson

Thanks so much for kicking off the consultation, Michelle Levesque ! I'm sure Amy Silcock would have some comments from the technical side!

Sarah McDuff
Sarah McDuff

Michelle Levesque  We definitely understand the concerns around flow type and while it is important for some data users, it is less used / important than finance type (which is critical) and aid type. It is logical that downstream implementers would likely not be familiar with this categorization and that asking them to report it could result in accurate reporting. We will take this into consideration as we finalize these measures -- thanks for the inputs!

Elma Jenkins
Elma Jenkins

We have some general comments on this phase which can be found in the numbered points in the document below :  

https://docs.google.com/document/d/1BpEjcypxpHJ_SNCnscy2HZe7otEZZIYcjE7HPFdDPfI/edit?usp=sharing

As well as some indicator specific comments :

2.3.1 - 2.3.5 Classification  

2.3.3 Policy Marker classification: 

  • Not all activities need to include a policy marker. There is no requirement for background documents explaining the rationale for the policy marker and score so there are no checks that the significance was screened. Policy marker use generally is not standardised across all types of donors. Giving publishers a score on % these may not have much meaning since it may depend on whether they are screening activities for the policy markers. 

2.5 Sustainable Development Goals 

  • Not all activities/publishers will follow the SDGs - many may not, particularly non-official organisations. SDGs are a UN initiative that delegates implementation to international agencies and governments but orgs. outside this system may not report on SDGs. Giving publishers a score on % these won’t be an objective way of measuring publishers. 

Amy Silcock
Amy Silcock

I agree that further guidance is needed on how organisations use policy markers. There's currently a great deal of variation with some organisations only publishing those which are relevant and others publishing all but including their level of significance with many being 'not significant'.

I see the encouragement for all organisations to use policy markers and SDGs as similar to asking all publishers to use the OECD 5 digit sector classifications. These classifications help guide users to find potentially relevant activities to their research question. These activities still need to be assessed for suitability by the user.

Evgenia Tyurina
Evgenia Tyurina

2.3.2 Question: The measure looks for the presence of the default-aid-type element, if not present should an aid-type element be present for every transaction?

Yes

General comment: It is not clear from the methodology description if at least one of the aid type vocabularies needs to be used for the data to be considered “good quality” or all vocabularies. In ILO’s view publishing aid type using at least one of the vocabularies should be enough. You may consider reformulating for more clarity.

2.3.4 Question: The measure looks for the presence of the default-finance-type element, if not present should a finance-type element be present for every transaction?

When finance type is published at the transaction level the “Expenditure” transactions should be excluded from the review because none of the values in the IATI Finance Type codelist reflects the nature of expenditure transactions – the purchase of goods and services from commercial suppliers on market terms.

General comment: 2.3.4 is repeated twice

2.5. Question: Should both tag and sector SDG classifications be included? SDG guidance states that tag should always be used.

Since the IATI standard allows using both the “tag” and the “sector” for publishing SDG information, ILO considers that the publisher should be able to choose under which of these elements to publish SDG data. If sector is published using SDG vocabulary it should be considered as “good quality data” even if the SDG tag is not published. If there is a valid reason for not using “sector” for publishing SDG info, then such a possibility should be excluded from the standard. Until then, in our view, publishers should not be punished for choosing one of the options proposed by the standard and not the other.

Michelle Levesque
Michelle Levesque

A comment I made during the discussion and will reiterate here as it relates to the SDGs.  Because we know using the tag is more universal and the recommended way to include SDGs in the data, yet the UN agencies will use sector because of our concurrent UN reporting requirements, if we include both sector and tags we start to get redundant in our data files.  This is the same case in other elements and a concern we have to consider is the huge file sizes that may get unwieldy.  In testing our new files, we stressed CoVE to the point that was unworkable.  This is just something to think about as/when we start thinking of IATI 3.0 or whatever the next upgrade would be called and as we continue to provide recommendations and guidance on how to publish data.  I understand making it easy for users is key but we can't make it so difficult for publishers either.  Just something to think about. 

Yohanna  Loucheur
Yohanna Loucheur

2.3.1 is redundant with 2.1.1

2.3.2; 2.3.4; 2.3.5 Others mentioned in the discussion that some of the classifications are only used by donors - these elements do come for the DAC CRS after all. While they can be very relevant in the context of partner country public accounts (eg whether an activity is a loan or a grant), only flows originating from eg bilateral donors and MDBs would be characterised - and accounted - using these categories. To assess and potentially shame publishers for not using categories that don't really apply to their activities does not add value, and could be counter-productive. We should seriously consider an Index with sub-sections for specific groups of publishers.

We agree with the inclusion of policy markers and SDG. 

Thea Schepers
Thea Schepers

We consider flow type more an indicator of type of publisher than of data quality in our opinion. We have just scrapped it in the guidelines we are updating because it means nothing to the NGOs we work with and it seems only relevant to government and multilateral donors to us. In the meeting it was mentioned that this leads to issues in data use, I'd love to know more about that but I have little hope that it is used correctly by publishers.

Sarah Scholz
Sarah Scholz

Q1. - Q4. These make sense to us.

Q5.    The U.S. supports the policy marker related to the SDGs to the extent that it is voluntary.  As a voluntary marker, we believe it may be unsuitable to assess its use as a quality indicator. (This relates a larger issue / theme throughout this DQI feedback: is the purpose of this index to "score" publishers on measures, which implies a best practice, or to provide neutral data points? You mentioned elsewhere that it’s only to provide data points, but there must be recognition that any such index allows and encourages cross-comparison and implies best practices where they may not be universally applicable.)

Back to the technicals of assessing this measure: The cross-cutting nature of applying result oriented targets to activity data makes it extremely difficult to apply precise or meaningful percentages to SDG classifications.  If SDG reporting is measured, it should be only as a tag, not a sector classification.  

Is this measure going to differentiate between SDG assigned to an activity or the results of an activity?  

Additionally, there are certain modalities that SDGs cannot be applied, and should not be included in a measure.  For example, there is no reason to apply SDG’s to general budget support to governments or multialterals, or administrative costs among others.

Amy Silcock
Amy Silcock

Thank you for your comments and feedback. 

For the SDG measures we propose to only look at SDGs included at the activity level. There is the discussion in this thread of if SDGs included within a sector should also be included, noting that this is more applicable to UN agencies who have a mandate for doing so outside of IATI. So far, only two IATI publishers have specified SDGs at result level. Given that I'm not sure of the value of adding a SDG result check to the DQI.

As for the other comments on which measures are relevant to different subsets of publishers, these will be noted and picked up in the next round of consultation.


Please log in or sign up to comment.