Data Quality Index Consultation

DATA COMPLETENESS

Sub-sections 2.6.1 - 2.6.4 Identifiers and Traceability


Instructions for submitting your feedback

1. Read through the proposed methodology for this measure and / or download the attached PDF at the bottom of this page; 

2. Share your feedback through the comment box below, consider the guiding questions in your comments and include the question number in your response;


Proposed Measures - 2.6.1 IDENTIFIERS AND TRACEABILITY

Please find below the proposed methodology for this measure. 

DEFINITION Asses if activities contain a Reporting Organisation Identifier that matches the IATI Registry

OUTPUT

  • Percentage

 

METHODOLOGY 

Count the number of active activities which have a reporting_org_ref that matches their IATI Org ID on the IATI Registry.

Divide by total number of active activities.



Proposed Measures - 2.6.2 IDENTIFIERS AND TRACEABILITY

Please find below the proposed methodology for this measure. 

DEFINITION Assess if participating organisation contain organisation identifiers.

OUTPUT

  • Percentage (participating_org_refs that match the IATI Registry)
  • Percentage (participating_org_refs that match the IATI Registry or are on org-id.guide)

 

METHODOLOGY 

  • Count the number of participating organisations which have a participating_org_ref that matches an IATI Org ID on the IATI Registry.
  • Divide by total number of participating organisations.

----------------------------------------------------------------------------

  • Count the number of participating organisations which have a participating_org_ref starting with an organisation registration agency listed on org-id.guide or that matches an IATI Org ID on the IATI Registry.
  • Divide by total number of participating organisations.


Proposed Measures - 2.6.3 IDENTIFIERS AND TRACEABILITY

Please find below the proposed methodology for this measure. 

DEFINITION Assess if participating organisations contain activity ids

OUTPUT

  • Percentage

 

METHODOLOGY 

For active activities:

  • Count the number of participating-orgs which have an activity-id.
  • Divide by total number of participating-orgs.


Proposed Measures - 2.6.4 IDENTIFIERS AND TRACEABILITY

Please find below the proposed methodology for this measure. 

DEFINITION Assess if transactions contain provider and receiver organisation identifiers

OUTPUT

  • Percentage (provider-org)
  • Percentage (receiver-org)

 

METHODOLOGY 

For active activities:

  • Count the number of transactions with provider-org elements, with an IATI Org ID or activity ID that have been included for transaction types 1, 9, 11, 13.
  • Divide by total number of transactions with types 1, 9, 11, 13.

For active activities:

  • Count the number of transactions with receiver-org elements, with names or IATI Org ID or activity IDs, that have been included for transaction types 2, 3, 5, 6, 7, 8, 10, 12.
  • Divide by the total number of transactions with types 2, 3, 5, 6, 7, 8, 10, 12.

Guiding question - please refer to the index number when you respond via the comment box below!
  • Which transaction types should be captured?

GO BACK TO MAIN DQI-PAGE 

Webinar

For each discussion, the IATI Secretariat will organise a webinar to explain the proposed methodology, answer questions and further explain how to engage.

  • Please find an overview of the most frequently asked questions of the Timeliness and Validation webinar here.
  • Missed the DQI Webinar on Data Completeness held on March 30? Watch the recordings here or read the summaryhere!
Files

Comments (15)

Michelle Levesque
Michelle Levesque

Only looking for or-ids that match the publishers list seems short sighted.  While the reference for a participating org (PO) when that PO is a publisher should tie to the reference used by the publisher (I agree with that part of the validation), why would one want to exclude where there are references for non-publishers?  Yes, I get that they are hard to come by but that doesn't mean they aren't valuable.  No they can't be validated against a publicly available code list as none exists but they could be evaluated for structure using the guidance that the first field should be the country of registration and the next represents the registration organization. We could get closer to consistency and use. (i.e. US-EIN-xxxxxxx or CH-UID-xxxxxx)  This is why I'm always harping on the fact that the IATI issued codes shouldn't start with XI because no one needs to know where the registration organization is based but rather the country in which the organization itself is based.  At least that way users could possibly get a list of PO who are in a given country.  

Yohanna  Loucheur
Yohanna Loucheur

Michelle, agreeing with you in general, but could you clarify which specific Index element you are commenting on? I'm only seeing a reference to the IATI Registry for 2.6.1 (and 2.6.2, but that seems to be a mistake). 

Michelle Levesque
Michelle Levesque

Yohanna Loucheur - Thanks for asking because now that I re-read the proposal I think I was reading it incorrectly.  Of course a reporting org should have its correct identifier from the registry.  For some reason I read it that participating org id's had to all come from the list of reporting org IDs in 2.6.2 and 2.6.4 and my point was simply that many participating organizations (excluding the reporting orgs which by definition are publishers) are not IATI publishers so they wouldn't have IDs that match and a publisher shouldn't have a lower percentage because of it.  

What I would propose is that if a participating org is not a publisher then the structure of the identifier used should start with the 2-digit ISO country code in which the organization is headquartered.  I don't know if the system can check for that so it may not be feasible.

 

Yohanna  Loucheur
Yohanna Loucheur

Michelle Levesque yeah, I think the best that we can aim for is that a "valid org ID" would be one that aligns with org-ID.guide guidance/lists (not sure how to formulate, but that would be the general idea). In most cases, this would indeed mean an org ID that starts with a country code. 

Amy Silcock
Amy Silcock

Yohanna Loucheur and Michelle Levesque on the discussion between valid vs well structure Org refs we're trying to capture this in 2.6.2, having two outputs one that checks if the org ref matches one on the IATI Registry. The other looks to see if any org ref has been included, to the second we could expand the scope of the check to see if it matches a prefix in org-id.

I can see the benefit of this but I'm also wary as in IATI data we see examples of organisations adding additional variables onto the end of their own reporting-orgs. Either way the check can be 'gamed', would be good to hear your thoughts if checking against the Registry + checking against Org-id is a reasonable solution.

Michelle Levesque
Michelle Levesque

Amy Silcock I see the pros and cons.  Yes publishers could game the system by simply putting in the first part of the org id structure (e.g. US-EIN or GB-COH) and nothing else and that isn't a real org ID.  However, comparing to Org-ID prefixes does provide a minimum of quality that is missing today which I could see as a benefit.  As I've mentioned a few other times in the context of IATI issued publishing numbers, ensuring the first two characters are based on where the org is and not where the registration organization is I believe adds quality and context to the IDs even if they aren't complete with real registration numbers.  Checking that the first two characters actually represent a valid ISO 2-digit code is a positive check in my opinion.  

 

I would propose that IATI technical scrape (I think the techies call it that) the registry for anywhere org ids are listed.  Either they, a working group, or a temp consultant, could analyze the data to see where we have multiple IDs for same org, figure out which if any are actually correct and provide feedback to the publishers to see about correcting where necessary.  And that same scraped data could be provided so that publishers would have it as a reference to use rather than have every publisher have to construct org ids from scratch for non publishers.  This is a project outside the specific scope of the data quality index but I do believe would aid in improving data quality on this key element.

Yohanna  Loucheur
Yohanna Loucheur

Amy Silcock 

Amy, I'm afraid I don't understand what you mean by "in IATI data we see examples of organisations adding additional variables onto the end of their own reporting-orgs. Either way the check can be 'gamed' Since 2.6.2 is not about reporting-org, I don't see how this would play in.

In any case, the fact that we still don't have the actual methodology being proposed for 2.6.2 is the main problem. It's very difficult to comment on something we cannot see, there's a great risk of error.

At this point I am tempted to err towards caution and suggest to drop 2.6.2 until a methodology can be properly consulted. What is absolutely clear is that the IATI Registry cannot be the only source of "valid organisation ID", given that many participating orgs are not publishers.  For instance, we just agreed on a way to refer to partner country government bodies; these should be considered valid Org-IDs, but would not be on the IATI Registry. 

Elma Jenkins
Elma Jenkins

We have some general comments on this phase which can be found in the numbered points in the document below :  

https://docs.google.com/document/d/1BpEjcypxpHJ_SNCnscy2HZe7otEZZIYcjE7HPFdDPfI/edit?usp=sharing

As well as some indicator specific comments :

2.6.1 - 2.6.4 Identifiers and Traceability 

2.6.2 - participating organisations references  

  • Seems to be a repetition of the Reporting Organisation references test and is not specific to participating orgs. What code lists for participating orgs recognised references will be used?  

2.6.3 participating organisations activity IDs 

  • - No guidance given? 

Evgenia Tyurina
Evgenia Tyurina

General comment/question on 2.6.2: the definition of this measure is “Assess if participating organisation contain organisation identifiers” but the methodology talks about Count the number of active activities which have a reporting_org_ref that matches their IATI Org ID on the IATI Registry.” Should it be participating_org_ref instead? How will the DQI assess the cases where the participating organization is not registered in IATI and, therefore, has no org ID on the IATI Registry?

General comment on 2.6.3: Collecting, storing and publishing partner organisations activity IDs is a very time-consuming and cumbersome exercise. At the same time, if one of the organisations in the finance chain publishes the activity ID of the partner it is enough to establish traceability between those activities e.g. if ILO publishes the activity ID of the donor it is enough to connect the donor activity with the ILO activity and, therefore, the donor does not need to collect, store and publish ILO’s activity ID on their side. For this reason, ILO agrees with the proposal that has been discussed during the Traceability Guidance consultation that it should be the responsibility of the recipient organisation to seek activity IDs from the donors but not the other way around. This makes the ambition of ensuring traceability in IATI more realistic. Considering the above, the ILO proposes to only include participating organizations with “funding” role for the assessment under this measure.

General comment/question on 2.6.4: It is not clear from the methodology what is meant by the “valid IATI Org ID” for provider and receiver organisations. If it means that Org ID matches an Org ID on the IATI Registry (as for 2.6.2) then how will the DQI assess the cases where the provider or receiver organization is not registered in IATI and, therefore, has no org ID on the IATI Registry?

The title of 2.6.4 does not correspond to the content of the methodology that, in addition to provider and receiver organization identifiers, also covers provider and receiver organisation activity IDs and receiver org name (btw why not the provider org name?). 2.6.4 assess the availability of the same data as 2.6.2 and 2.6.3 but published in a different place (with an exception for the names of receiver orgs) and, therefore, the ILO proposes to remove 2.6.4 from the DQI.

Yohanna  Loucheur
Yohanna Loucheur

Anna Whitson - IATI Secretariat , mistakes and omissions in the methodology were signaled in the comments above, over a week ago. We cannot provide informed feedback without complete and accurate information on the methodology. Could you ask the Tech Team to provide an updated paper as quickly as possible please?  

Amy Silcock
Amy Silcock

Anna Whitson - IATI Secretariat and Sander Hees - IATI Secretariat there are indeed some changes to be made. I'm happy to amend the page if given access, or let me know how to provide the changes.

Amy Silcock
Amy Silcock

I've updated the methodology above (thanks Sander Hees - IATI Secretariat for your help).

 

Key changes are:

- methodology now match the proposed measures (apologies for the copy and paste errors)

- 2.6.2 is now has one measure to check if participating-org-refs match an IATI Org-ID on the IATI Registry. The second checks against the Registry and Org-ID.guide

- 2.6.4 I've removed the word 'valid'. So the methodology now looks for the presence of any ref or activity-id

Evgenia Tyurina
Evgenia Tyurina

Dear Amy Silcock, dear colleagues, 

We see no added value in checking for "any ref or activity-id". Publishing activity-ids that are incorrect (do not exist in IATI) or participating-org-ref that is not recognized as valid by the community does not improve the quality of data nor enhance traceability.

We agree with Michelle Levesque  proposal of a “technical scrape” for org-refs that would provide a source of truth so that “publishers would have it as a reference to use rather than have every publisher have to construct org ids from scratch for non-publishers”. We think this project should be implemented before the relevant measure is included in the data quality index.

Sarah Scholz
Sarah Scholz

Q1.  US agrees that all of these transactions should have a receiver organization and therefore can include a receiver org ID.

When assessing org  IDs, the methodology should include valid org IDs for non IATI publishers.  These organizations are still the majority of implementing organizations (role 4 participating organizations).  

There are many organizations, such as bilateral donors, that will rarely if ever report a provider org ID.  Again, the Index may imply an unintentional and untrue best practice.


Please log in or sign up to comment.