Dear all,

As you may have seen, the IATI Datastore is currently unavailable. The Secretariat expects that this will be resolved by Q4 2021. There is an urgent need to provide continuity of service for existing and potential users of the IATI Datastore.

Datastore Classic is used to support systems in Liberia and Somalia. It is also used by a number of other users. It currently runs quite happily on a €15 a month server, though concurrent requests are a bit slow.

As Chair of the Data Use Working Group, I propose that we use funds from the DUWG to do the following:

a) increase the server capacity of Datastore Classic;
b) pay a supplier (I propose Open Data Services, given that several staff there were intimately involved with the development of what became Datastore Classic) 2 days a month for maintenance;
c) manage this as a community project, with input from relevant users, perhaps as a sub-group of the DUWG;
d) redirect all requests from iatidatastore.iatistandard.org to Datastore Classic (IATI tech team would just have to point this domain at a server that the community manages). Datastore Classic would then try to forward as many of these requests as possible, and would log unhandled requests.

I would not receive any income through this proposal. I would continue to help manage and provide further development to Datastore Classic as part of my own work supporting the Government of Liberia.

Look forward to hearing everyone's thoughts!

All the best

Mark

Comments (20)

leo stolk
leo stolk

Hi Mark, this reads like a sound proposition. strenghtening the server capacity and support of the datastore classic. I use the datastore classic for my use case regularly. Suggest you forward this proposal to the tech team and the secretariate.

leo

matmaxgeds
matmaxgeds

Hi Mark - given Somalia are currently depending on Datastore Classic I would be much in favour of making sure it has the resources to cope!

Anders Hofstee
Anders Hofstee

Hi Mark,  I agree that is best to avoid the breakage of existing workflows and systems like those in Liberia and Somalia.   This looks like a pragmatic solution to protect this pre-existing investments in country-level systems.
 

Wendy  Thomas
Wendy Thomas

Hi everyone, and thanks Mark for sharing this. Just a quick update to let you know the Secretariat is discussing how best to put in place some support for Datastore Classic as one of the options for users during this period of transition. Thanks!

Alex Tilley
Alex Tilley

Hi Mark - this sounds like a very helpful proposal. We're using downloads of the IATI data a lot at the moment and will use the datastore classic query builder for this. Will be good if the DUWG or IATI Secretariat can support it.

Best, Alex

Herman van Loon
Herman van Loon

Hi Mark, sounds like a good and practical approach. It is very important for the continuity of the datastore service and since it is already working it seems like the quickest way to restore this service to the community. 

Herman van Loon
Herman van Loon

+1 also for the performance of the Datastore Classic. I just ran a query to retrieve all activities in the implementation phase with the word 'water' in the title. Asking for the retrieval of 50 rows the response time was about 8 seconds. Asking for all IATI data ever meeting this selection criteria the response time was 14 seconds. Pretty impressive imho. 

Rory Scott
Rory Scott

+1 This is a very sound proposal, and I don't see many other options. A few considerations:

This is an opportunity to double down on fundamentals and achieve some consistency and stability for IATI data use. Datastore Classic is clearly a very resilient and well designed piece of software. Although the work done my Mark and others to redeploy it was undeniably excellent, the fact is that only moderate tweaking, the underlying code is able to serve most use cases (minus aggregation) for less than the cost of a coffee a week. And clearly there are a lot of people in the community who have already decided to use it. Why not take what is working enable it with funding and institutional support?

Furthermore, I think that redirects are important. The fact that there has been unplanned and uncaught downtime in the primary data access point is already disruptive. Now users trying to visit previous queries will be met with an html landing page. (Note that documentation on IATI.cloud still points to the official IATI Datastore v2 URL).

We can't be sure of the scale of the impact, as from what I know there isn't detailed telemetry of datastore usage, but I think the principle of cushioning users from disruption is key to reassuring IATI stakeholders.

 

 

Mark Brough
Mark Brough

Hi all,

Thanks for all the positive comments above!

My understanding is that we now have consensus to proceed with something along the lines of the above proposal. IATI Secretariat , please can you confirm that my understanding is correct?

I will bring the above proposal forward for discussion and approval in the DUWG meeting on Tuesday 13th July. I'd encourage everyone to provide further feedback on the proposal in advance of the meeting. Please do so preferably in this thread. Alternatively you can also send me your thoughts via email: mark@brough.io

Mark Brough
Mark Brough

Dear all,

As agreed in today's meeting, please find here the link to the Terms of Reference for work on Datastore Classic over the coming months. You will need to request access which I will grant ASAP to everyone who requests it. I would very much welcome feedback by 12pm CEST (Berlin / Paris / Amsterdam) on Thursday 15th July.

PLEASE NOTE: if you are considering submitting a bid, please don't request access to the document. To avoid conflicts of interest, you must not engage in this process. If you are considering submitting a bid, please let me know!

Many thanks, and I look forward to receiving everyone's inputs!

Mark Brough
Mark Brough

leo stolk  Thanks Leo for your comments! I have accepted and incorporated everything. Other DUWG members -- please do request access and it will be granted ASAP.

Mark Brough
Mark Brough

Hi all - we now have a final version of the ToR. If you have any concerns with this, please raise them immediately now!

Again, please request access and I will give you permissions ASAP. If you are a vendor considering considering submitting a bid, please don't request access to the document. However, if you are considering submitting a bid, please let me know!

Annelise Parr - IATI Secretariat
Annelise Parr - IATI Secretariat

Hi everyone, I'm really happy to say that based on the Terms of Reference collectively drafted by the group and with agreement of the IATI Governing Board, UNDP has put a contract in place today with Open Data Services for a five month period of support to Datastore Classic using resources from the Data Use Fund. The group can discuss working arrangements in next week's Data Use Working Group meeting.

Mark Brough
Mark Brough

Many thanks for the update, Annelise! This is great news - thanks to all involved for the hard work in getting this in place so quickly.

Many thanks also to the IATI Technical Team for helping get redirects in place already!

We'll discuss working arrangements and forming a sub-group of the Data Use Working Group to support managing this work in next Tuesday's DUWG meeting - so please do join!

https://iaticonnect.org/group/25/event/data-use-working-group-meeting-2…


Please log in or sign up to comment.