Dear DUWG Colleagues:  I'm polling for your thoughts on a matter the U.S. Transparency teams are working through regarding how we publish our data to IATI. We think this might be relevant to a few other publishers as well, and appreciate your feedback, thoughts, and ideas:

The U.S. is currently considering the best way to consolidate our reporting without losing granularity by agency. Currently, U.S. foreign assistance data is published under three different publishers: USAID, MCC, and USG (the latter of which combines around 20 other U.S. agencies' data). By reporting under more than one "publisher", we're concerned that it's difficult for users to see all U.S. foreign assistance data unless they're aware of this peculiarity (does the common user in -say- Indonesia really know that MCC is also a U.S. Government agency?) Ideally, the standard would have a designated space for an "umbrella" publisher and also sub-department or agency (think UN and its bodies, Oxfam and its country branches, governments and their departments/agencies, etc.) However, the highest-level is currently "publisher", which for many, is already disaggregated. We understand adding a designated umbrella field to the standard to be a significant undertaking that may affect other, non-umbrella publishers. 

Instead, we are advised to consider options within the existing standard that may partially resolve the issue. Some of these we're already doing. Options include: Leaving the publisher field disaggregated, and identifying the umbrella org within the Activity ID prefix (e.g. US-USAGOV-MCC-XXX); Aggregating the publisher to publish under 1 USG, and identifying agency (sub-"publishers") within either (a) the reporting-org narrative field, (b) the participating org field with type: "extending", or (c) in an other-identifier element, using an internal codelist of departments. 

(1) Are there other publishers facing the same constraint? Perhaps UK, Germany, Canada, Spain, SIDA, UN Oxfam.

(2) How are you handling it? Are we all handling this consistently? If not, how will users know what to filter by?

(4) USERS: Are you aware of this issue when searching for data? How do you aggregate / disaggregate the above publishers' data? Is that process easy and clear to you and other users?

(3) Any recommendations for the US, IATI Secretariat, or all publishers with umbrellas and sub-entities?


Thanks, all! We're really looking forward to this discussion.

Comments (8)

leo stolk

1) Oxfam currently publishes per individual legal entity, so each Oxfam publishes their data set. This is likely to change in the near future when more and more operational liability and responsibility shifts to Stichting Oxfam International (NL-KVK-41159611) and all Oxfam teams and offices use one Grant/program/project management system. Publishing in IATI may then happen by Oxfam International from the common platform.
2) currently we report separately as separate legal entities.
3) as user I harvest individual data sets all clearly recognizable by Oxfam XXXX in publisher name.
4) harmonize the publisher names is the easiest step US GOV MCC, rather than Millennium Challenge Corporation. You could also harmonize part of the publisher ID US-USGOV-MCC etc.

Sarah McDuff - IATI Secretariat

Very interested to see others inputs here but just want to share that this is a concern that has also been raised by a number of other publishers, including quite recently by the UN Secretariat, as they also have multiple entities within the Secretariat. From a data use perspective, as you note, consistency across publishers is absolutely key. However, it should also be the case that our data access tools do this work for users and we have definitely noted this need for a consolidated view of all entities within a Government -- as part of our review of user requirements for the new data portal. Currently it is very difficult for users to know which organizations they should be pulling together -- even with the current situation of USG + USAID + MCC, and I suspect that most non-expert users don't know to do this or if they realize it needs to happen, struggle to do so.

Steven Flower

Really interesting - thanks [~461] and team for raising

Before looking into the IATI XML and such, I investigated the software that is used for the IATI Registry (CKAN). Thanks to [~574] at the Centre for Humanitarian Data, I learnt that there's an "extension" to CKAN to enable publishers to be organised into hierarchies (parent > child > child etc etc):

Hence, in theory, it could be possible to enable this extension in the IATI Registry, and then start to (re)organise publishers accordingly. For example, there could be a USG "organisation", that then has the children for the various offices/agencies within. Such a setup might also work in the case of [~401] and Oxfam (although there would always need to be a "parent" -- which might not actually publish specific data...), too.

Dan and I were not too sure how user permissions work out within this - but a well-planned trial would help us know more.

I haven't seen this function in action, but perhaps the [~453] & CKAN partners have? It'd be good to know more.

Whilst this does not solve all the issues in terms of nested organisations publishing data, it could go some way to addressing the disparate nature of "publishers" in the IATI Registry. Of course, there's more to say around this in terms of reporting-org & participating-org - which I will do later - but wanted to share this CKAN news first of all..!


Not sure if I am missing something here, but in my experience solving this via the transactions, not via the publisher field (which due to secondary publishers, duplicate publishers etc, I see as a purely administrative field) may be beneficial. The charts I make to show who are the big contributors in country X are not organised by 'publisher', rather they work on the basis of aggregating on 'funding org' or similar. So could this be handled by fixing the problem in the transactions? If all US activities had their initial funding transaction or similar coming from a US umbrella org (handing the funds onto USAID, USG, MCC etc) then I could chose to either show the US umbrella org in the charts, or show the individual agencies as the funding source when that was more helpful.

I am also worried how the various datastores would handle this - as unless the information is included in the activity files (or the datastores are retroactively populating it), many IATI using systems will not be able to access it and discover the intended relationships - few systems can even mine the org files, so hosting vital data use metadata in the (hard for end users to access) registry (which already happens) seems like a backward step to me.

Mark Brough

HI [~461] and colleagues, many thanks for raising this! I have an additional suggestion - to prefix the narrative within the reporting organisation with "US - ", so that a user would see e.g. "US - MCC", "US - USAID", etc. "Reporting organisation" is a really useful unit of analysis, and work on the CDFD tool suggests that this is much more consistently high-quality data then either provider/receiver organisations or participating organisations. So I think it is really important to continue to state the name of the specific agency in the reporting organisation field.

I think there might also be some value in considering having some kind of "umbrella" or "grouping" field, either in the schema or in the Registry. This could then be used by IATI's data access tools e.g. when data is made available through simplified spreadsheets, such as the CDFD tool.

Sarah Scholz

Thank you so much, all, for these thoughtful responses and ideas. We are very grateful for the discussion, which I'm sure will be really useful as we move forward exploring our options. I'm also very happy to see that the discussion could lead to more consistent use of the Standard by publishers in the same boat as the U.S., and relieved to hear that we're not the only ones! We'll take all of the "actionable" (on the publisher side) ideas back to capital and in the meantime, will be interested to see what further input the Tech team and others may have.
Thank you!

Amy Silcock

Really interesting discussions. thank you [~433] for the heads up about CKAN extensions to allow for hierarchical groupings. We'll take a look.

I agree with the comments that consistency in naming both on the IATI Registry and in your data would be a good and simple step forwards. It makes it clearer to data users that the three publishers are part of the US Government, whilst not being too much work as the changes don't need to be logged within existing data.

Standardising IATI Org IDs whilst being 'ideal' is much more complicated. Particularly given that previous IATI Org ID changes are already captured within some of the US's activities under 'other-identifier'. Org ID's can be changed but this requires more effort.

As [~350] mentioned, having all three publishers list 'US Gov' as the funder for (first order) activities would help to further tie the data together to see the US's total funding as published in IATI. I think this is already happening in some instances.

Please log in or sign up to comment.