Dear IATI Community,
Our IATI Datastore at https://datastore.iatistandard.org/ lets you search, browse and download activities - but did you know we also have budgets and transactions data available?
Datastore Functions
Most people query activities via the Datastore website or API. When you do this you can get the original XML, a JSON representation of the XML and a representation of all the fields on the activity, including budgets and transactions.
But a few people will also be querying budgets or transactions directly. You can do this from the Datastore website by selecting that option in the download box, or via the API by changing the “collection” parameter. This will return data with one object per budget or transaction. When you do this via the API you can filter on any attribute of the budget or transaction or its associated activity.
Details of the bug’s scope
However, recently we discovered issues when querying budgets and transactions directly. These existing bugs cause an incorrect number of budgets or transactions records, or values for budgets or transactions to be reported on the wrong budget or transaction within an activity. This specific release will fix these bugs.
Note that these bugs only affect people that query budgets and transactions directly in the Datastore. These bugs do not affect budget or transaction data in activity records returned from querying activities.
Changes we will make
To fix these bugs, we will be replacing part of the code that prepares data to be inserted into the Datastore. This code is also used for preparing Activity data, and will result in minimal changes to how Activities are stored. Because of this, we will be testing this change very carefully before releasing it.
The changes people will notice when querying activity data concerns empty or blank values - sometimes, for a field you would get a list of values that included some empty values. In the future, these empty values will not be there, and only values with actual data will be there. We don’t know of any use case that relies on these empty values, so we don’t think this will cause any problems. (If you have a use case that relies on these empty values, your code probably has the same bugs we have been working on and you should contact us to discuss that).
When querying budget and transaction data directly people will notice the same removal of empty or blank values, and fixes to the bugs we have identified.
Any questions? Contact the IATI Secretariat!
These changes are now in testing. We’ll update this post as we release them. Tip: click 'Follow Content' at the top of this post to receive a notification whenever we update this post.
If you think you may be affected by this or have any questions, let us know here or via our support email address : support@iatistandard.org.
Hello,
Thanks for also reporting this at https://github.com/IATI/datastore-search/issues/665
I guess there is a practical limit on how much data can be downloaded directly from the web UI in one go. We should highlight that in the web UI and point to alternative options instead.
Talking of which .... is our API suitable for this? This is described at https://developer.iatistandard.org/ and allows paging.
However if the main need is just to get all the data in one download then other tools may be more suitable - if you or anyone else reading this wants help please do contact support https://iatistandard.org/en/guidance/get-support/