Data Quality Index Consultation - sub-section Data Availability

Instructions for submitting your feedback
  1. Access each section by clicking on the buttons below, or download the attached full report.
  2. Share your feedback on the Data Availability sub-sections of your choice through the comment-box below.
  3. Consider the guiding questions for each part, and please refer to the indicated numbers in your comments.

3.2 Comprehensiveness - change to Data Availability

  • Do you agree with the proposed additions and changes to the measure of what comprehensive data looks like? 
  • Do you agree with the proposed change to call this measure Data Availability? Can you think of a better naming if you disagree?
  • Do you agree with the categorisations? If not, what would you propose? Have we missed a measure of data availability that is crucial for measuring data quality?

3.3 Data Availability - Mandatory

  • Mandatory: Do you agree that mandatory and recommended elements should be assessed under this category?

3.4 Data Availability - Location

  • Location: Do you agree with the proposed assessment of country/region/ recipient language?

3.5 Data Availability - Classification

  • Classification: Do you agree with all the proposed measures and elements under this category? Should they all remain or are there some you think are not relevant for all users? Do you agree that under policy markers only gender should be assessed?

3.6 Data Availability - SDGs

  • Do you agree that SDGs should be included as a separate measure?

3.7 Data Availability - identifiers and traceability

  • Do you think that we have captured the relevant traceability measures? Are there any that are missing?

3.8 Data Availability - Financial

  • Do you agree that both transactions and budgets should fall under financial? Previously budgets were assessed separately under forward looking?
  • Do you agree with the transaction types we want to look at and assess? 
  • Do you agree we should also include total budgets in the organisation file? Is assessing total budgets for next three years a realistic measure?

3.9 Data Availability - Humanitarian Data

  • Do you agree with the assessment areas included under humanitarian?

3.10 Data Availability - Results


3.11 Data Availability - Documents

  • Do you agree that a separate assessment  for results and documents should be included?

3.13 Data Availability - Coverage

  • Do you agree that total expenditure should be used?

BACK TO THE MAIN DQI-PAGE

Files

Comments (32)

Amy Silcock
Amy Silcock

Feedback on data availability so far TL:DR

  • Suggestion and support to change the category name to Data Completeness
  • Overall support for the new categories of assessed data, nothing missing
  • Need to ensure that the same metric doesn’t appear twice in the index
  • Need to ensure data can be anonymised/sanitised without publishers being penalised
  • Need to decide which metrics are suitable for which groups of publishers e.g. DAC publishers, humanitarian, NGOs

Further discussions to be held at the upcoming VCE:

  • How to use and measure ‘location’ data. Which elements should be looked at, what should be expected?
  • Should ‘recipient language’ be measured? It’s currently included in the IATI Dashboard
  • Which policy-markers should be included in the DQI?
  • How to assess inclusion of partner organisations' iati-identifiers, useful but depends on how external organisation’s are publishing.
Herman van Loon
Herman van Loon

3.2 Comprehensiviness

  • Do you agree with the proposed additions and changes to the measure of what comprehensive data looks like?

    Partly. I.m.o. the role the publisher has in the network should be taken into account. E.g. for locations, a bilateral donor channeling funds through iNGO's and multilaterals, is not likely to know where the activity will take place and so will be unable to publish a lot of location data. For an NGO, it is unlikely that the aid-type or flow-type is very relevant, since these are DAC elements primarily focused on bilateral donors. They should not be penalized for not publishing those elements.
     
  • Do you agree with the proposed change to call this measure Data Availability? Can you think of a better naming if you disagree?

    No, it is a bit confusing because it can also be read as coverage. Basically it is about the 'Completeness of published fields'. Coverage is a separate item. 
     
  • Do you agree with the categorisations? If not, what would you propose? Have we missed a measure of data availability that is crucial for measuring data quality?

    Yes
Herman van Loon
Herman van Loon

3.3 Data availability: mandatory

  • Mandatory: Do you agree that mandatory and recommended elements should be assessed under this category?

    Maybe. Shouldn't a distinction be made between mandatory and recommended elements? Otherwise you could call the recommended elements also mandatory, which is not the case. Of course we could also decide that a recommended element should be mandatory. That would require an integer update to the standard though.
Herman van Loon
Herman van Loon

3.4 Data availability - location

  • Location: Do you agree with the proposed assessment of country/region/ recipient language?

    No, since there is no distinction being made between types of publishers. Especially for location this is problematic. E.g. most (aprox. 80%) of the Netherlands aid, as a bilateral donor, is channeled through multilaterals and iNGO's or donor country  based NGO's. No specific locations where the aid will flow to. That is up to decide by the multilaterals and NGO's. So location information (geo-points) can not be published.

    On the other hand, if you are a local NGO you almost certainly will know where the activity takes place and than you can publish location information (provided there are no security concerns of course).
Herman van Loon
Herman van Loon

3.5 Data availability - classifications

  • Classification: Do you agree with all the proposed measures and elements under this category? Should they all remain or are there some you think are not relevant for all users? Do you agree that under policy markers only gender should be assessed?

    Partly.

    Again the type of publisher you are, should be taken into consideration. E.g. Aid-type, Finance-type, Flow-type and collaboration type are very OECD/DAC centric classifications, mostly relevant for bilateral donors and multilaterals, but nor relevant at all for NGO's, recipient governments and private sector. 

    Sector and policy markers can be relevant for all publishers.

    The disbursement channel also looks only relevant for bilateral donor's.
Herman van Loon
Herman van Loon

3.7 - Identifiers and traceability 

  • Do you think that we have captured the relevant traceability measures? Are there any that are missing?

    Partly.

    It is important when referring to the funding activity-id in an incoming fund or incoming commitment transaction, that also the validity of that activity id is checked (it must be an existing activity-id).

    When referring to an recipient organization in an outgoing commitment or disbursement transaction, it is often not possible to publish the activity-id of the recipient, because that activity might not yet exist in the activity administration of the recipient. 

    Also the type of publisher should be taken into account: for a bilateral donor there are no incoming funds or disbursements (because it is tax payers money being spend), with the exception of delegated cooperation.

    For all other publishers incoming funds or disbursements are relevant to publish, with the exception of the organizations responsible for actually doing the work on the ground. They will only have expenditures.
Herman van Loon
Herman van Loon

3.8 Data availability - Financial

  • Do you agree that both transactions and budgets should fall under financial? Previously budgets were assessed separately under forward looking?

    No, budgets are typically forward looking and it is unclear for me what problem is being solved by lumping budget together with financial.
     
  • Do you agree with the transaction types we want to look at and assess?
    In the text I see that you want to distinguish between incoming and outgoing transactions. That is ok. I do dot see any further references to specific transaction types.
     
  • Do you agree we should also include total budgets in the organization file? Is assessing total budgets for next three years a realistic measure?

    I am not sure on the organization level since multi year budgets are only indicative. Usually reliable budgets are only defined one year ahead.
Herman van Loon
Herman van Loon

3.9 Data availability - Humanitarian

  • Do you agree with the assessment areas included under humanitarian?

    Yes, with the exception of localization. When channeling humanitarian funds through multilaterals, iNGO's or donor country based NGO's, often the local actors are not known yet and so it is impossible to publish this information. The solution here is to make use of the traceability of activities between publishers: further down the funding chain, it is far more likely that local actors will be known. 

 

leo stolk
leo stolk

3.2 Comprehensiveness - change to Data availability

  • Do you agree with the proposed additions and changes to the measure of what comprehensive data looks like? 

in general I agree, scoring/weight could be adjusted per role of publisher in a trail or network. Risk is to make this too complex, and for instance distinguish between two type of roles, predominantly funding, or predominantly implementing/extending

  • Do you agree with the proposed change to call this measure Data Availability? Can you think of a better naming if you disagree?

Suggest Data Completeness rather than availability. Completeness of content per activity/record enables optimalisation of its data use. 

  • Do you agree with the categorisations? If not, what would you propose? Have we missed a measure of data availability that is crucial for measuring data quality?

Suggest to make the inclusion of accurate granular information of implementing actor(s) (national/local etc) as participating organization, an area of assessment. The last mile, is often missing or hidden, even in no-risk contexts. Excluding those who actually achieve the change. 

leo stolk
leo stolk

3.3 Mandatory: Do you agree that mandatory and recommended elements should be assessed under this category?

- Yes, but using the element is not sufficient to be complete and compliant to mandatory use. Check should be how those mandatory elements are used. For instance a publisher can declare itself as accountable participating org, so used the element, but not disclosing details of funding or implementing participating org. 

leo stolk
leo stolk

3.4 Location: Do you agree with the proposed assessment of country/region/ recipient language?

i agree with country region and sub national location details. Have serious doubts on the check of recipient country language,
Which larger publishers will be able to publish quality information in all official languages of the countries they work in?  For instance Oxfam would need to be able to publish in Spanish, French, Pashto, Arabic, Dari, Bahasa indonesia, Portugueese, Norwegian, Japanese etc etc. I expect that this will likely affect data quality and thus data use. 
Suggest instead to check each publisher's consistent use of its chosen working languages in its data set, that will probably contribute to data quality. 
Also this proposal is not clear will it be checked at each activity language <-> recipient country?    

leo stolk
leo stolk

3.5 Classification: Do you agree with all the proposed measures and elements under this category? Should they all remain or are there some you think are not relevant for all users?
Yes they should remain, but main weight should be given to SECTOR, additional points for the use of other classifications

Do you agree that under policy markers only gender should be assessed?

Not really,  i think it is important to check the use of gender policy marker, but again other policy marker like climate change mitigation and adaptation are growingly important too. 

leo stolk
leo stolk

3.6 Do you agree that SDGs should be included as a separate measure?

Yes at least at goal level.  below goal level, targets and indicators is way more problematic and poorly fitting the NGO sphere. 

leo stolk
leo stolk

3.7 Do you think that we have captured the relevant traceability measures? Are there any that are missing?

I disagree with the receiver org activity ID, as very often they are not available when starting publishing upstream. Also often receiver organisations of INGOs do not publish

leo stolk
leo stolk

3.8 data availability financial 

  • Do you agree that both transactions and budgets should fall under financial? Previously budgets were assessed separately under forward looking?

No problem, not all organization can declare next years budgets that easy. Longer term funding mechanisms are often the exception. 

  • Do you agree with the transaction types we want to look at and assess? 

yes

  • Do you agree we should also include total budgets in the organisation file? Is assessing total budgets for next three years a realistic measure?

yes, most organisations have a multiyear planning and budget allbeit indicative and not tight to recipient countries and regions. 

leo stolk
leo stolk

3.10 results
agree with proposed simple measure

3.11 Documents
 

  • Do you agree that a separate assessment  for results and documents should be included?

Yes.  

3.13 Coverage
Suggest to use total expenditure on mission goals.  INGOs also spend on fund raising and domestic campaigns, suggest to exclude when defining total expenditure.

Yohanna  Loucheur
Yohanna Loucheur

This is a loooong list of questions - perhaps it should have been further divided into sections? 

Agree with others re Data Completeness rather than Data Availability.

Agree that list of elements under Classification is too long and not adapted/reflective of all publishers. Index should use principle of parsimony and focus on core elements. Suggest removing from *Index* (not Standard!!): Finance Type, Flow Type, Collaboration type, Disbursement Channel. 

Also suggest to remove "Localisation" under Humanitarian Data, as it is duplicative with the Identifiers and Traceability section. 

Agree to include Policy Markers (especially GE) and SDGs (as long as use of SDG Indicator is not mandatory).

Yohanna  Loucheur
Yohanna Loucheur

3.4 Location

Agree with others that bilateral donors often lack details regarding the specific geographic location of projects. At the very least, this assessment should exclude certain types of aid/receiving organisations (e.g. multilaterals). 

Regarding sub-national locations, we should not be expecting both geographic coordinates AND location details - it would be duplicative, especially where a gazeteer ID is used (which should be the preferred option).

Recipient language: putting this under Location may cause confusion. Suggest rather to move it to the "Mandatory" section (which already contains non-mandatory elements), as it relates to titles and descriptions. 

3.7 Identifiers and Traceability

Agree with the importance of using correct organisation identifiers. However, the IATI Registry is definitely *not* the right reference list for participating organisations - many implementing organisations are not on the Registry. We've been talking about this for years and, as if I'm not mistaken, the best approach we've come to so far is to check that Org Identifiers are well formed, using org-id.guide as a reference. 

Agree with others that an activity identifier can be provided for incoming funds but rarely for outgoing funds - this should be reflected in the index.

Want to flag that Participating Organisation is already in the Mandatory section of the Index; perhaps it would make sense to have the role and type there as well? 

 

3.8 Financial

Unsure... I don't quite understand what is proposed by splitting transaction types and assessing them, and assessing whether organisations have reported spend. 

Also, the *number* of activities with budgets for the next tow years compared with the *number* the activities at the start of each year - what does that tell us? How does it relate to the total budget for the next three years? 

Marie-Line Simon
Marie-Line Simon

On location, financial, comprehensiveness

Publishing data concerning the organizations’ actions, and especially on IATI, should not give rise to risks for the people contributing to these actions or for these organizations’ leaders and workers. As we support human-rights-based interventions, we are particularly keen to ensure the protection of data based on the concrete experience of partner organizations struggling to defend socio-economic and human rights in their countries and at international level. These are thus regularly considered as obstacles by the economic and political actors, and are therefore exposed, in their lives and work, to risks of being persecuted, threatened or blocked in their functioning. Above all, the publication of specific budgets in relation to specific organizations represents a risk. This
information could allow the authorities to financially dispossess these social movements by blocking their bank accounts or resources (see the situation in Romania several years ago) or by completely preventing them from functioning (a real risk, for example, for trade unions in Cambodia).
Location is part of what we consider sensitive information. So I would recommend taking this into account in your reflection: anonymised data is not necessarily out of lack of accuracy.

Anna de Vries
Anna de Vries

3.2 Comprehensiveness - change to Data Availability

Do you agree with the proposed additions and changes to the measure of what comprehensive data looks like? 

  • Yes, overall I agree

Do you agree with the proposed change to call this measure Data Availability? Can you think of a better naming if you disagree?

  • Name change is fine

Do you agree with the categorisations? If not, what would you propose? Have we missed a measure of data availability that is crucial for measuring data quality?

  • Perhaps it should be assessed separately whether a publisher has both an organisation and activity file since this is often not the case. The proposal implies that it is and also uses items such as organisation budget for measuring quality. This can be of added value, but is there a fall-back of measuring the quality of the activity file information if there is no organisation file?
  • Besides mandatory there might be some other important data quality fields that have value for quality such as background or target group. This could be a measurement of having more detailed information (including subnational locations) since there is already a country/region reported as a mandatory field.

3.3 Data Availability - Mandatory

Mandatory: Do you agree that mandatory and recommended elements should be assessed under this category?

  • Agreed but for some elements there is also a separate category, are these then counted twice?

3.4 Data Availability - Location

Location: Do you agree with the proposed assessment of country/region/ recipient language?

  • Subnational level would be interesting to measure however it would be good to only have it ‘count’ if e.g. both the coordinates and name are given to prevent just pinning the geolocation of the country which does not add anything.
  • How do you propose to measure recipient language?

3.5 Data Availability - Classification

Classification: Do you agree with all the proposed measures and elements under this category? Should they all remain or are there some you think are not relevant for all users? Do you agree that under policy markers only gender should be assessed?

  • The added value of these elements is questionable since the defaults (aid, flow, collaboration, finance type & disbursement channel) are almost always the same for many of the publishers, and can be set as a default value for all activities. These elements are also unclear to publishers how they should use them, so more guidance on this would be needed if it is to say something.
  • Not sure why only gender is assessed, for me it does not say the ‘data quality’ is better/worse either way.

3.6 Data Availability - SDGs

Do you agree that SDGs should be included as a separate measure?

  • Yes, would be a good addition to stimulate use of this.

3.7 Data Availability - identifiers and traceability

Do you think that we have captured the relevant traceability measures? Are there any that are missing?

  • I think this is an important addition but thought needs to go into what this will mean for ‘end of the line’ publishers in terms of data quality (if they themselves are implementing and there are no partners), or for donors who don’t have any provider orgs. Additionally if you do have partners but those are not publishing, should that then affect your data quality per se?

3.8 Data Availability - Financial

Do you agree that both transactions and budgets should fall under financial? Previously budgets were assessed separately under forward looking?

  • Can be here as well since it would also show budget depletion in a current year for example 
  • For budgets it might be valuable to consider whether they report budgets annually or multi-annually.

Do you agree with the transaction types we want to look at and assess? 

  • The document does not specify this, merely incoming & outgoing. I would see having both the (incoming) commitment and the actual incoming fund or disbursement as an indication of data quality since you are both reported on expected and actual which increases transparency.
  • Also on transaction dates I am unsure why the value date is a good measurement to include since using the same date in iati does not necessarily mean that is the date they use in their own accounting.

Do you agree we should also include total budgets in the organisation file? Is assessing total budgets for next three years a realistic measure?

  • While a good idea this does not yet seem realistic since very few publishers use the organisation file elements of IATI.

3.9 Data Availability - Humanitarian Data

Do you agree with the assessment areas included under humanitarian?

  • Humanitarian data does not apply to all types of activities and therefore difficult to measure as a standard for data quality..

3.10 Data Availability - Results

  • No question here, but I think that it is important to be able to measure progress by having both baselines&targets included and not only actuals. So that would be a bonus for data quality.

3.11 Data Availability - Documents

Do you agree that a separate assessment  for results and documents should be included?

  • Documents could fall under ‘more detailed information’ instead of existing as a separate assessment.

3.13 Data Availability - Coverage

Do you agree that total expenditure should be used?

  • This would be valuable but is currently limited due to the fact that organisation budget often is not reported.
Evgenia Tyurina
Evgenia Tyurina

Hello, everyone. Here is the ILO's feedback:

3.2 Comprehensiveness - change to data availability

Do you agree with the proposed additions and changes to the measure of what comprehensive data looks like? 

Agree

Do you agree with the proposed change to call this measure Data Availability? Can you think of a better naming if you disagree?

Agree

Do you agree with the categorisations? If not, what would you propose? Have we missed a measure of data availability that is crucial for measuring data quality?

Agree

 

3.3 Data Availability - Mandatory

Mandatory: Do you agree that mandatory and recommended elements should be assessed under this category?

Agree. Propose to change the name to “Mandatory and recommended minimum”

 

3.4 Data Availability - Location

Location: Do you agree with the proposed assessment of country/region/ recipient language?

Agree with the assessment of country/region.

As for the sub-national location, this element may not be relevant for activities that are delivered at the national government level (e.g. policy advice, support to legislative reforms etc.). How is IATI going to assess whether the data under this element is missing for a valid reason or not?

As for the recipient language, the formulation “title and description elements written in at least one of the official languages spoken in the reported recipient country” sounds unrealistic for multilateral organisations that operate in many different countries.

 

3.5 Data Availability - Classification

Do you agree with all the proposed measures and elements under this category?

Yes. In addition, we propose to include tied-status element. Whether the procurement channels are limited by the donor or not is one of the indications of the quality of funding.

In addition to the CRS purpose codes sector vocabulary 1, the document talks about other sector vocabularies 3, 4, 5, 6, 12, 99, 98 that will be looked at. For those vocabularies will it be an “and” or an “or” assessment?

Re Aid Type, will all four Aid Type Vocabularies be looked at? Will the publication for at least one of them be enough?

Should they all remain or are there some you think are not relevant for all users?

All are relevant

Do you agree that under policy markers only gender should be assessed?

At this stage, yes

 

3.6 Data Availability - SDGs

Do you agree that SDGs should be included as a separate measure?

Yes.

The SDG publishing guidance that the document refers to indicates that it is not recommended to publish SDG data under the sector. If so, why dies the SDG sector vocabularies exist at all (vocabulary 7-9)? Are the SDG publishing guidelines outdated? We think that SDG sector vocabularies should also be looked at while assessing the availability of SDG data

 

3.7 Data Availability - identifiers and traceability

Do you think that we have captured the relevant traceability measures? Are there any that are missing?

It is not clear from the documents whether the organisations that are not registered in IATI will be excluded from the traceability assessment of participating org. information, since they do not have an IATI identifier, or they will be included by looking at their name and type elements.

Regarding the assessment of the availability of activity identifiers, we would like to highlight two points:

  • It was discussed during the traceability guidelines consultation that in order to avoid double work it should be the responsibility of the recipient to seek the activity IDs from the donor and not the other way around. Therefore, in our view, the assessment should only focus on the availability of provider org. activity IDs.
  • Based on our experience, there might be cases where donors are not capable to provide their activity IDs when asked by the recipient. There might be different reasons for that (activity not included in their publication, technically not possible (yet) to map activities to those of the recipient etc.). In this case, the responsibility for not providing activity ID should remain with the donor and not with the recipient. The recipient’s Data Quality Index should not “suffer” in this case.

 

3.8 Data Availability - Financial

Do you agree that both transactions and budgets should fall under financial? Previously budgets were assessed separately under forward looking?

Yes

Do you agree with the transaction types we want to look at and assess? 

We think that the assessment should only focus on transaction types 1,2,3,4 and 11. Other transaction types may not be relevant for all publishers.

Do you agree we should also include total budgets in the organisation file? Is assessing total budgets for next three years a realistic measure?

Planning in the ILO is done on a biennial basis, so the “budgets for next three years” is not a realistic measure for us.

Re the forward-looking budgets, the document says that it will include “assessing the number of activities with budgets reported for the next two years”. This measure is not relevant for activates with less than two years duration. In our view, the activity budgets should be assessed in connection to the activity duration: if the yearly budget is provided for the whole duration of the activity then the budget should be considered “forward-looking”. Looking at the share of such activities in the total number of activities may be an additional measure.

 

3.9 Data Availability - Humanitarian Data

Do you agree with the assessment areas included under humanitarian?

Yes, with an exception for how the “localisation” is proposed to be measured. ILO does not use the participating organisation type 24 - Partner Country based NGO in its internal classifications of implementing partners, but uses type 22 - National NGO instead. We think this type should also be looked at while assessing the localisation.

 

3.11 Data Availability - Documents

Do you agree that a separate assessment for results and documents should be included?

Yes

 

3.13 Data Availability - Coverage

Do you agree that total expenditure should be used?

Yes

Athira Lonappan
Athira Lonappan

Do you agree with the proposed additions and changes to the measure of what comprehensive data looks like? 

In our experience, we have seen that sometimes the more you ask, the less you get. I wonder if adding more requirements would reduce the amount of data that complies and gets submitted.

On a slightly different note, could you look at it from a different lens - considering removing/reducing the current number of columns. e.g. asking for just the district/province level information and then reverse engineering to map to state and country level information.

Do you agree with the proposed change to call this measure Data Availability? Can you think of a better naming if you disagree?

The proposed name sounds appropriate.

Data Availability - Mandatory

Mandatory: Do you agree that mandatory and recommended elements should be assessed under this category?

If elements are marked as recommended then in my opinion it should be optional or else all should be marked mandatory.

Data Availability - Location

Location: Do you agree with the proposed assessment of country/region/ recipient language?

After downloading datasets, what I observed is that there are a total of 14 columns dedicated to location of which many values are missing, which might suggest that people would like to fill less but crisp information rather than having to fill too many values. 

Data Availability - SDGs

Do you agree that SDGs should be included as a separate measure?

Yes

Data Availability - identifiers & traceability

Do you think that we have captured the relevant traceability measures? Are there any that are missing?

Say suppose a user wants to trace the flow of funds, knowing which sector it has been sent to is also important, so having valid sector codes is something I think should also be cross checked.

Data Availability - Humanitarian Data

Do you agree with the assessment areas included under humanitarian?

Regarding localization, it is difficult for smaller organizations with small team size to contribute their data to IATI considering that such organizations might find it difficult to provide all the required information and also be skeptical of paying for the publishing tools and services. 

 

 

 

 

 

Otto Reichner
Otto Reichner

3.2. Comprehensiveness - change to Data Availability

  • Do you agree with the proposed additions and changes to the measure of what comprehensive data looks like? 

We do agree with many, but also have reservations (see comments to subsections)

  • Do you agree with the proposed change to call this measure Data Availability? Can you think of a better naming if you disagree?

Not really,  comprehensiveness was a better term for us.

  • Do you agree with the categorizations? If not, what would you propose? Have we missed a measure of data availability that is crucial for measuring data quality?

In principle yes, but with some detailed reservations (see comments to subsections)

Otto Reichner
Otto Reichner

3.3. Data Availability - Mandatory

  • Mandatory: Do you agree that mandatory and recommended elements should be assessed under this category?

We agree that also the recommended elements should become ‘mandatory’. Once that change is agreed, both can be included in this category.

3.4. Data Availability - Location

  • Location: Do you agree with the proposed assessment of country/region/ recipient language?

We don’t agree with Sub-national location disaggregation and recipient language.

3.6. Data Availability - SDGs

  • Do you agree that SDGs should be included as a separate measure?

Yes, we do agree that due to its importance, the SDG should be evaluated as a separate measure.

3.7. Data Availability - Identifiers and traceability

  • Do you think that we have captured the relevant traceability measures? Are there any that are missing?

We agree that in principle the correct use of organization identifiers should be checked, in particular for reporting organization and for those participating organizations published on the IATI registry.

The feasibility for checking organization identifiers and activity identifiers in transaction provider section needs to be further analyzed. Publishing organizations should not be penalized for missing information provided by other partners.

3.8. Data Availability - Financial

  • Do you agree that both transactions and budgets should fall under financial? Previously budgets were assessed separately under forward looking?

We do agree to include budget under financials.

  • Do you agree with the transaction types we want to look at and assess? 

We agree that it is very important to evaluate the amount spent (disbursed or expenditures). It should be further explained how the transaction types are intended to be used to distinguish between incoming and outgoing. It would be important to clarify the definition of ‘commitments’ and its relevance only for organization’s extended aid (e.g. donors).

  • Do you agree we should also include total budgets in the organization file? Is assessing total budgets for next three years a realistic measure?

We agree to include the total budget with the clarification that also not formally approved budgets can be reported.

3.9. Data Availability - Humanitarian data

  • Do you agree with the assessment areas included under humanitarian?

No, only the humanitarian flag is agreed.

Inclusion of other areas would penalize most organizations as budgetary and financial frameworks are not aligned to Humanitarian Response Plans (agreed at country level), which makes it nearly impossible to report correct financials by the intended dimensions.

Localisation should also not be included as it would penalize organisations that do implement mostly directly, i.e. using NGOs and others as pure service provider.

3.11. Data Availability - Documents

  • Do you agree that a separate assessment for results and documents should be included?

Would you please provide the proposed detailed methodology and new KPIs.

We would prefer to have documents and results evaluated separately.

3.13. Data Availability - Coverage

  • Do you agree that total expenditure should be used?

Yes, we agree that the total expenditure is the correct figure to evaluate coverage and should be used.

 

Amy Silcock
Amy Silcock

Feedback on data availability so far TL:DR

  • Suggestion and support to change the category name to Data Completeness
  • Overall support for the new categories of assessed data, nothing missing
  • Need to ensure that the same metric doesn’t appear twice in the index
  • Need to ensure data can be anonymised/sanitised without publishers being penalised
  • Need to decide which metrics are suitable for which groups of publishers e.g. DAC publishers, humanitarian, NGOs#

Further discussions to be held at the upcoming VCE:

  • How to use and measure ‘location’ data. Which elements should be looked at, what should be expected?
  • Should ‘recipient language’ be measured? It’s currently included in the IATI Dashboard
  • Which policy-markers should be included in the DQI?
  • How to assess inclusion of partner organisations' iati-identifiers, useful but depends on how external organisation’s are publishing.
Marie Maasbol
Marie Maasbol

Please see below the feedback from the Commission (FPI, DG NEAR, ECHO and INTPA)

3.2 COMPREHENSIVENESS

Do you agree with the proposed additions and changes to the measure of what comprehensive data looks like?

  • The Commission agrees with this proposal.

Do you agree with the proposed change to call this measure Data Availability? Can you think of a better naming if you disagree?

  • The Commission agrees with the other name proposal suggested by colleagues in this field – Data Completeness – to make it more distinguishable from coverage.

Do you agree with the categorisations? If not, what would you propose? Have we missed a measure of data availability that is crucial for measuring data quality?

  • The Commission agrees and believes the proposed measures are comprehensive.

 3.3 MANDATORY DATA

Do you agree that mandatory and recommended elements should be assessed under this category?

  • The Commission would like to propose that both mandatory and recommended elements are to be assessed on the condition that recommended elements are not included in the scoring of the Index. This would be an opportunity to alert publishers to make positive changes also in the recommended elements. Without a clear distinction between mandatory and recommended elements, one might wrongly understand that all elements are mandatory.

3.4 LOCATION

Do you agree with the proposed assessment of country/region/ recipient language?

  • The Commission appreciates this proposal, but would like to flag that it is important that a distinction is made between the type of publisher, as we recognise that major structural elements of organisations affect the information available to publish depending on the operation. As such, the Commission advocates that parts of the assessment could be excluded when not appropriate to the organisational processes. It suggests in particular that the location indicator should be a flexible assessment, as:
    • Information on geolocation is not available on all ODA spending. For example, a publisher will not be aware of the geo-location, when resources are allocated to core funding in an International Organisation or International NGO.
    • In addition, it is vital to consider that humanitarian context when discussing project and geolocations, as the number of attacks on humanitarian workers is increasing. As such, information on geolocations cannot always be shared due to such security reasons.
  • The Commission would like to stress that providing the official language of the countries a project is taking place in will be very difficult to achieve for larger publishers operating globally. Additionally, this will have implications for communication, data use at the regional, global level and would as well increase costs.

3.5 CLASSIFICATION 

Do you agree with all the proposed measures and elements under this category?

  • The Commission agrees with this proposal and supports the alignment with the OECD-DAC.

3.6 SDGs

Do you agree that SDGs should be included as a separate measure?

  • The Commission agrees with this proposal under the understanding that such a measure will apply to new activities, as requiring retroactive changes to thousands of contracts prior to 2019 (the year IATI published guidance on how to report on SDGs) would not be feasible for large donors like the Commission.

3.7 IDENTIFIERS AND TRACEABILITY

Do you think that we have captured the relevant traceability measures? Are there any that are missing?

  • The Commission recognises the benefits of such an assessment. It is, however, important to note that such a measure should apply to new activities only, as requiring retroactive changes to thousands of contracts would not be feasible for large donors like the Commission.
  • Moreover, the Commission has many implementing partners that are not IATI publishers. For example, some EU MS are not publishing to IATI. We are raising awareness of the importance of IATI and encouraging development partners to publish. However, if the information is not available in the IATI registry, a publisher should not be penalised.  
  • Additionally, this measure should consider the type of publisher, as for a donor, like the Commission, incoming funds is not applicable.

3.8 FINANCIAL

Do you agree that both transactions and budgets should fall under financial? Previously budgets were assessed separately under forward-looking?

  • The Commission would like to flag that a budget is an estimate that can be reviewed and modified, whereas financial transactions are committed. This distinction should be reflected in the proposal, and would therefore suggest to keep budget separate from financial.

Do you agree with the transaction types we want to look at and assess?

  • The Commission agrees to the proposal of assessing incoming and outgoing transactions. We would like to note that some organisations, like the Commission, does not have many incoming transactions.

Do you agree we should also include total budgets in the organisation file? Is assessing total budgets for next three years a realistic measure?

  • The Commission would like to refer to the above answer on budget estimation and would suggest that this will be considered in the Index, if a publisher is to include the total budget in the organisation file.
  • The Commission proposes that the Index will take into account the budgetary time-frames of each donor while assessing how forward-looking the published information is. At the beginning of the Multi-annual Financial Framework (the EU long-term budget) the Commission provides up to 7 years forward looking information but at its end, such time-span is not available.
  • A suggestion in this regard would be for the Index to assess whether an organisation has a legal framework and foundation with a long-term (i.e. three years forward-looking) budget. This could encourage organisations to establish more long-term budgets and not penalise publishers at the end of a long-term budget cycle.
  • Additionally, a distinction should be made for the Humanitarian sector, as disaggregated budgets are also not always available due to the short-term nature of projects, where funds are allocated based on needs.

3.9 HUMANITARIAN DATA

  • The Commission proposes that the Index considers the security concerns that are associated with the humanitarian context. This will in some instances also have consequences for providing the information assessed under localisation.
  • Regarding the following point: “Humanitarian scope - assessing whether the activity refer to an appeal or emergency and whether it references the Humanitarian response Plans or Global Emergency Identifier (GLIDE)” - the Commission would like to ensure that humanitarian organisations will not be penalised if part of its activities do not fall under an appeal. For example, the Commission does not decide if an activity is part of an appeal or not. As it stands, it is OCHA that takes such decisions. Therefore, humanitarian data is reporting under "emergency" in IATI. We do not find it straightforward the way OCHA define an activity as part of an appeal or not and would require time and discussions with OCHA to further understand the decision-making process in this regard.

3.10 RESULTS

Do you agree with the assessment areas included under results?

  • The Commission agrees with the proposal and supports that, with an increased focus on the use of data rather than on a simple transparency objective, results reporting becomes more important.

3.11 DOCUMENTS

Do you agree that a separate assessment for results and documents should be included?

  • The Commission agrees with this proposal.

3.12 COVERAGE

Do you agree that total expenditure should be used?

  • The Commission welcomes this proposal, as coverage should be a key element of any transparency assessment.

 

Alex Tilley
Alex Tilley
  • Do you agree with the proposed additions and changes to the measure of what comprehensive data looks like? 

Some yes and others no.

  • Do you agree with the proposed change to call this measure Data Availability? Can you think of a better naming if you disagree?

Comprehensiveness or completeness would maybe be better.

  • Do you agree with the categorisations? If not, what would you propose? Have we missed a measure of data availability that is crucial for measuring data quality?

Yes, agree.

  • Mandatory: Do you agree that mandatory and recommended elements should be assessed under this category?

Yes, agree

  • Location: Do you agree with the proposed assessment of country/region/ recipient language?

Recipient language could be challenging, though the intention is good.

  • Classification: Do you agree with all the proposed measures and elements under this category? Should they all remain or are there some you think are not relevant for all users? Do you agree that under policy markers only gender should be assessed?

What is the rationale for only assessing the gender policy marker? This seems arbitrary – in terms of use and importance, the Rio Markers for climate change and the environment are widely used and important for tracking this type of funding, and why leave out disability, nutrition and others?

  • Do you agree that SDGs should be included as a separate measure?

Possibly – I would disagree that the SDGs are universally recognized and accepted, however. They are generally accepted by governments, official aid agencies and multilaterals since they were drawn up and agreed by these organisations, however, there may be more diverging views from CSOs and NGOs that also report to IATI.

  • Do you think that we have captured the relevant traceability measures? Are there any that are missing?

This is similar to the Networked Data indicator we are introducing in the Aid Transparency Index. On transaction provider and receiver information, including activity IDs – this would be good if people start doing it, but we found that it was not being done consistently or by many publishers.

  • Do you agree that both transactions and budgets should fall under financial? Previously budgets were assessed separately under forward looking?

No view on this. It is both.

  • Do you agree with the transaction types we want to look at and assess?

I couldn’t see commitments included. These are important.

  • Do you agree we should also include total budgets in the organisation file? Is assessing total budgets for next three years a realistic measure?

This is the approach we follow in the Aid Transparency Index.

  • Do you agree with the assessment areas included under humanitarian?

Yes, these look good.

  • Do you agree that a separate assessment  for results and documents should be included?

Yes, agree. The documents assessment seems quite superficial, however, given the various types of document that can be included in activity and organization files.

  • Do you agree that total expenditure should be used?

We looked at this as a possibility for the Aid Transparency Index and found two main problems with it. First it was only being used by a very small number of publishers, and was being used infrequently by them. Second was that there was no way of validating the total spend figure – in some cases the total spend number was generated by adding up all of the spend transactions recorded in the publisher’s activity data. This would make it ineffective as a measure of coverage.

Pelle Aardema
Pelle Aardema

3.2

  • Do you agree with the proposed additions and changes to the measure of what comprehensive data looks like? 

Agree with previous comments that not all elements make sense for all publishers. "OECD magic fields" don't make sense for NGOs, bilateral donors can't show income and often don't know exact countries or locations.

  • Do you agree with the proposed change to call this measure Data Availability? Can you think of a better naming if you disagree?

Agree with the suggestion to call it Data Completeness

  • Do you agree with the categorisations? If not, what would you propose? Have we missed a measure of data availability that is crucial for measuring data quality?

Categorisations are fine.

 

3.3

  • Mandatory: Do you agree that mandatory and recommended elements should be assessed under this category?

If this category contains both mandatory and recommended elements, that should be reflected in the name of the category

 

3.4

  • Location: Do you agree with the proposed assessment of country/region/ recipient language?

The methodology needs to take into account the role of the publisher. Not only for donors who provide unearmarked regional or global funding, but also for intermediairy NGOs who might hire a local partner that speaks the recipient language.

3.5

  • Classification: Do you agree with all the proposed measures and elements under this category? Should they all remain or are there some you think are not relevant for all users? Do you agree that under policy markers only gender should be assessed?

The methodology needs to take into account the role of the publisher. OECD classifications are not relevant for NGOs.

I couldn't find the reasoning for only assessing the gender marker.

 

3.6

  • Do you agree that SDGs should be included as a separate measure?

Agree

3.7

  • Do you think that we have captured the relevant traceability measures? Are there any that are missing?

If we want to be able to link activities, then the provider-activity-id is vital. As others have already mentioned, registering and maintaining receiver-activity-ids is often impossible.

 

3.8

  • Do you agree that both transactions and budgets should fall under financial? Previously budgets were assessed separately under forward looking?

No objection

  • Do you agree with the transaction types we want to look at and assess? 

yes

  • Do you agree we should also include total budgets in the organisation file? Is assessing total budgets for next three years a realistic measure?

This might be an nice incentive for broader use of the organisation file

3.9

  • Do you agree with the assessment areas included under humanitarian?

There are a few issues that may make it hard to operationalise this measure:

- the role of the publisher should be taken into consideration. For unearmarked or softly earmarked contributions, donors will often be unable to indicate HRPs, GLIDE, Localisation, etc.

- there are several actors who don't operate under the (UN) humanitarian clusters or HRPs

- the proposed measures for localisation may miss the point: a Kenyan humanitarian organisation will probably self-define as a 'National NGO', and likely also be classified by its international counterparts as such. (again a Red Cross example: the Netherlands Red Cross would probably identify the Kenyan Red Cross as a National NGO, not a Partner Country Based NGO).

3.10 - Results

3.11 - Documents

  • Do you agree that a separate assessment  for results and documents should be included?

Yes!

Note about the document: the Results element already existed in IATI versions 1.0x. It is not that new, just heavily underused. Making it part of the quality assessment could promote its use.

3.13

  • Do you agree that total expenditure should be used?

Yes, this could be a nice incentive to promote better use of the organisation file.


Please log in or sign up to comment.