Continuing the discussion from Bright ideas for the TAG:
I agree it is extremely important that aid agencies report their activity results, and that everyone can access these reports through IATI. I am in awe of the analytical grasp that Herb_Caudill has brought to bear on this. But it seems over-ambitious in the forseeable future to think in terms of a way in which indicators can be rigorously defined machine-readably for all activities.
A key point is that different activities - and within those, different objectives - are differently amenable to standardization. It may not be too difficult to count the number of children to whom your project has helped give a basic education, and it is certainly useful to be able to aggregate that number with others in local and national statistics. But not all objectives are like that.
Let’s say your objective is to build an all-weather road from Amule to Buba. A simple way of reporting the achievement of that objective in an aggregatable way is to say you built 100km of road. But the validity of any aggregation along those lines will be dubious because a road from Amule to Buba is a unique matter, entirely different in character from a road between Calakal and Dor. Measuring things like volume of traffic or value of merchandise does not absolve you from that because any assessment of the impact of the project has to take into account a host of other costs and benefits. Unless you follow the path of calculating everything into a single number using a full social cost-benefit analysis, you are left with a hopeless number of interpretable dimensions. Social cost-benefit analysis is less popular than it once was, and for good reason. In reality it hides many doubtful and politically controversial questions behind a facade of economic expertise.
Or take another example: one which you use in your exposition, Herb: training journalists. What matters in the result is not merely the number of journalists trained, and how they may be disaggregated by gender, ethnic group and so on. It is important to know how thorough the training was (a day or a year? Full-time or part-time etc), and about the features of the curriculum being used. It would be extremely complex and difficult to resolve these matters into machine-readable indicators, and the effort to do so would have unwanted side-effects. Not only would it increase the number of headaches in the publishing agency and deter it from using IATI at all; it might even end up forcing them to design their projects to fit in with the standardized indicators (including packages of standardized curricula) on offer. (I know you will say they can define their own indicators, Herb, but suspect they might feel it easier - or be pressurized - to take indicators off the peg.)
In the end, why do we want to aggregate results? Mainly for purposes of centralized planning and policy discussions. In some cases (like the provision of basic education) it is quite legitimate. In others - like roads - the planners and policy-makers really should not be shielding themselves from the social and political complexities behind pseudo-objective numbers. And in others again - like training journalists - it may be better not to involve central planners at all.
So what would my recommendations be for the TAG?
Remember that the essence of IATI is not necessarily to provide aggregatable or machine-analyzable data. The more fundamental purpose is to provide accessible information. For citizens, this may mean being able to find, understand, and follow up on the individual activities which are going to impact them, or which have impacted them, or which their government has funded or allowed to happen in their country.
There may be a case for providing some facility for using standardized indicators, but this should not be seen as something generalizable to every reported activity.
Being new to this space, I do not understand how the IATI standard has come to support a family of tags called “result” rather than “objective”. The IATI standard was supposed to be forward-looking (presumably in contrast to the OECD CRS). “Result” seems very backward-looking. Citizens and recipient-country governments need an easier way of knowing what aid agencies are planning to do and what they are in the middle of doing. Recording obejctives at the beginning also strengthens post-hoc accountability by allowing clear comparison of promises with, well, claimed results.
I do not think it should be stipulated that a “result” (or an “objective”) needs to be measurable. This job is done by indicators.
Apart from allowing and encouraging the reporting of pre-completion objectives, do not make the standard more complex. The priority should be to make donors and other publishers actually use those tag fields for reporting objectives, results and - above all - links to useful documents.
Hi Herb. Your last message - like the previous one, and discussions in the Google Group - is very constructive. But I do protest against the suggestion that I’m saying ‘there’s no way to measure the results of development work’ or that IATI should not attempt to record results.
One of the things I’m saying is that IATI can usefully record results (claimed and/or corroborated) even if it does not do so in a way which is aggregatable or machine-analyzable. Many will be interested in this information activity-by-activity. A lack of standardized codes should be no excuse for agencies not to use the results tag.
I do acknowledge that there is a strong case for supporting standardized indicators in many instances, although I have strong misgivings about their use in many instances too. I agree that the Accra Agenda points strongly in that direction, although I would point out that this is very much in the context of harmonizing performance assessment frameworks within recipient country systems. The US Foreign Assistance Framework with its 500+ standard indicators looks rather unilateral. What happens for a US-funded activity in a country whose standard indicators are different from the US ones?
As an information-facilitator, it is probably not up to IATI to solve problems like that. But IATI should provide room and facilities for solutions. So I would not oppose the creation of a way in which publishers can report objectives and performance in terms of standardized indicators. But I feel strongly that any use of any standardized codes here should be optional rather than mandatory within the IATI standard. Donors or recipient countries may nevertheless insist that their implementing agencies use them, but this should not bind other agencies in other relationships.
Thanks, Michael. I think we’re entirely in agreement so I won’t belabor this thread much further. Just to clarify:
The state of the world today is that (a) some organizations have created standard lists of indicators, and (b) some organizations use these indicators - some by choice, some as a condition for funding. Whether that’s a good thing or a bad thing is an interesting and important question, but is a topic for another day.
What the IATI schema lacks today is a way for an organization to say that they’re reporting on indicator XYZ from repository ABC.
This is glaring shortcoming in the schema. Without this simple element, the only quantitative analysis anyone can make of IATI data involves the amount of money that went from A to B. And while that’s important information, it’s the crudest indicator there is.
Fortunately this is a shortcoming that’s easily remedied. I’m looking forward to continuing the conversation in Ottawa.
Hi Herb. I’m glad we found our core of agreement, and that a session on improving the results element has been scheduled in the Ottawa meeting.
But a further thought occurs to me, which you may not like but I think is important:
In view of the very definite emphasis in the Paris, Accra and Busan agreements on harmonization and alignment according to recipient-country systems, shouldn’t the presumption be that vocabularies of standard indicators valid in the IATI standard would not include unilateral donor vocabularies like that of the US Foreign Assistance Framework?
This is an interesting thread and I want to highlight something important that Michael Medley said above:
MichaelMedley:The IATI Standard has actually moved somewhat away from this in the 2.01 version by making the Indicator element mandatory within the Results element. This makes it impossible to report Results in a narrative way (thus not aggregatable or machine-analyzable).
We had not noticed this important change in the 2.01 upgrade process, as it was not listed among the elements that were becoming mandatory. We plan to bring it up in Ottawa as something perhaps worth revisiting, given its impact on the availability of information on results (and objectives, which we are publishing in this element as “expected results”).
Thanks, Yohanna. I had not gone carefully through the 2.01 rules to see how the Results element enables the pre- or mid- project reporting of objectives. I’ve now done so (using http://iatistandard.org/201/activity-standard/).
I wonder if the hierarchy of tags involved (Result -> Indicator -> Period -> Target) might be obscuring the capability of reporting on pre- or mid- project objectives. I’d be interested in learning from publishers whether the complexity here does in fact significantly deter them from recording objectives at an early stage, and whether improved reference materials, software or training is needed to help.
On the earlier point about the fact that if you use the Results element you must also use the Indicator element: I don’t see how it makes narrative reporting of results impossible within the system. The only mandatory items within the Indicator element seem to be a narrative title, and an attribute called “Measure” which makes you specify whether you are measuring percentages or absolute quantities. The Period element, which would contain any hard quantitative information, does not seem to be mandatory (although then what is the point of “Measure” being mandatory?).
If you take the view that any specification of an indicator intrinsically takes you beyond narrative reporting, I would ask why you want to avoid that. I think indicators are useful even if sometime trivial. What I am very cautious about is the standardization of indicators necessary for aggregation and machine analysis.
It is true that, if you want to go into the Period element and use the Target and Actual elements, then for any Target or Actual you must specify a quantitative Value. I would agree that this is not always going to be appropriate. Indicators can be qualitative, and although qualitative indicators are frequently converted to pseudo-quantitative ones through a process of grading by experts, this raises questions which I think it is too early to grapple with.
No, and here’s why:
My primary takeaway from Ottawa was that IATI has huge potential as an intermediate exchange format from all along the chain of accountability: Not just for the donor to report to the world, but for the INGO to report to the donor, and for the small local grantee to report to the INGO.
In fact, it’s not realistic to expect the donors to report reliable machine-readable data to the world if there hasn’t been reliable machine-readable data all the way up the chain.
The fact of the matter is that the donors who are committing to report results data via IATI are still struggling to figure out how to assemble that data from disparate sources. This is where IATI can be very useful, assuming that it allows more precision in the ways I’ve proposed. If all US-funded actors are required to report to their US government donor agencies using IATI, using the indicator codes defined in the Foreign Assistance Framework, then the US has a chance of making good on its promises. Otherwise it doesn’t have a prayer.
The problem of cross-referencing indicators still remains, so that results data from different donors can be compared and aggregated where appropriate. Here I think the solution is not a global standard, but a crowd-sourced crosswalk API that can tell you which US indicator corresponds to which Mozambique indicator, or which UN SDG indicator corresponds to which World Bank indicator. More thoughts on that idea are here - please let me know if you’re interested in pursuing this idea.
I think this has a political dimension that we need to acknowledge. Our judgements about what is realistic for donors to implement and what is not realistic may be correct. Or they may not be. Bureaucrats sometimes need (and sometimes want) to be pushed to do things which do not seem realistic. I don’t think it is good in this case for technocrats to pre-judge and pre-empt.
Unless the TAG has guidelines/mandate which clearly covers this kind of issue, I think the question of whether to allow any standardized set of indicators to be used, or whether to wait for a common standard, or whether not to try to standardize indicators at all - ought to be referred politically upwards. The work done in showing that the first option is technically feasible is useful to inform such a decision.
The vision of donors and other aid agencies using IATI (not just IATI-consistent categories) as the medium for managing their own reporting chains - rather than relying mainly on internal processes - is an intriguing one. It could have far-reaching implications. It is attractive in many ways, but I’d have thought it would meet with strong resistance in many aid chains. My impression is that many donors - USG agencies not least - often like to intervene with advice and directives to implementers along the way. Sometimes this is because things have not gone as previously planned; sometimes it a cause of things not going as planned. Goalposts get changed, compromises are negotiated. Surely it can suit both the donor and the implementer to work things out in private: feel their way towards a common story rather than present their different versions as a public commentary going along. Not that it would be a bad thing if they could be made to do so. And I’m not saying we should block or pre-empt the choice of any agency to do so. Just that it is rather surprising to be told that if USG agencies can’t do things this way then they can’t meet their aid transparency promises at all.
Herb Caudill , you have a much closer knowledge of how USG agencies operate than I do. I genuinely think it would be of great interest if you could explain more about why it is so hard for USG agencies to adapt their internal reporting systems to IATI forms that they would actually find it more workable to use a public information channel.
Sorry, I wasn’t clear. When I say that NGOs should “report to their US government donor agencies using IATI”, what I mean is that they should report using the IATI schema, not via the public IATI registry. USG agencies “adapting their internal reporting system to IATI forms” is precisely what I’m talking about.
Here’s how things often work today:
All I want is for the PDF in step 3 to be replaced (or at least supplemented) with an XML document following the IATI schema, so that step 4 can be done by software. Right now that’s not feasible because the schema has no place for the codes 1.2.3 etc.
I don’t see this as a political matter, as technocrats vs bureaucrats, or as anything that should be remotely controversial. It’s just a matter of creating the possibility for structured, machine-readable results reporting, which is currently not an option because this element is inexplicably missing from the schema.
Sorry I misunderstood! But it is good to have the clarification: a helpful example.
However, what’s to stop USAID internally requiring the XML with the element as you recommend even if IATI does not adopt it? Wouldn’t it be a relatively simple matter for USAID to run their internal XMLs through a programe which converted them into IATI standard? Such a programme would, for example, see the code “1.2.3”, look up the corresponding verbal definition - “# communities served with potable water” - and insert the latter into
iati-activities/iati-activity/result/indicator/description/narrative
You might then still think the IATI data is not as useful as it could be. But the unchanged IATI schema would not seem to be seriously getting in the way of USAID improving its own practices.
Why would that be preferable?
I’m a little mystified by the fact that there’s any resistance at all to this change. The status quo is equivalent to having a budget element with no currency code element. This should be a complete no-brainer.
If this turns out to be unfixable within the primary IATI schema, I’ll submit it as an extension. Or, as you suggest, we’ll just extend the schema internally and clean it later so it validates for external publication. But I’m really scratching my head here to understand why this is controversial at all.
Standardizing and aggregating data is a powerful practice and a dominant technology in modernity, but it is powerfully dangerous as well as useful. The complexities of reality are chopped, stretched and squeezed into forms that the machine can ingest. What comes out are strong statements. These can be used as a basis for discussing inaccuracies, variations, biasses. But often they are used crudely to justify insensitive top-down policies and planning, and to browbeat stakeholders with facts and figures whose validity and relevance is dubious. There is also the question of whether some standards drive out others and - in that case - what political interests are at stake, and whose power is prevailing.
Describes all of human invention throughout history.
There is always the risk that people will use IATI in ill-advised ways. Not a strong argument for hobbling the standard.