A response to “A week in the life of an IATI developer”

Moderators

No moderators on this discussion

Imogen Kutz ’s ‘A week in the life of an IATI developer’ describes the IATI Technical Team ’s work on the forthcoming IATI python library – now known as pyIATI – and how it relates to other tools that the IATI Technical Team maintain. It’s great to have insights into the work of the technical team and sharing an overview of how different tasks are prioritised is much appreciated.

The blog describes the team following a number of software development best practices, some of which hint at an Agile approach. But none of them are explicit about delivering software early and often (i.e. iterative development), or customer collaboration (i.e. feedback from users).

I’m pleased to know that the PyIATI library is open source. And I understand that the tech team need to spend time building solid foundations. But I’m concerned about working for A Long Time on something without demonstrable output. As the UK Government Digital Service says: “Show The Thing!” Delivering early and often mitigates risk – and in this case, this risk is felt by the whole IATI community.

With finite resources, work on this library comes at the expense of work on other IATI tools and services such as the IATI Datastore, the query builder, d-portal, the IATI registry etc. Many of us here attended sessions on IATI tools at TAG, and suggested features and improvements. Having contributed to a number of IATI codebases, I have some familiarity with how they work. But I’m not really sure I understand how pyIATI will help address major issues with these existing tools.

I’ve made these points elsewhere, and Mark Brough has made similar points here. But I thought it worth reiterating. I appreciate the answer will probably be “have patience” (indeed, I’m not sure there’s another response at this point) but I hope that in the future we’ll be able to avoid long periods where there are no discernable improvements to IATI tools for end users.

Comments (3)

Hayden Field • 6 years ago

With regards to the points about working on pyIATI for an extended period without a clear released product…

As mentioned previously, the initial work on pyIATI has been to implement a reference implementation of multiple versions of ‘The Standard’. This is an essential single piece of work that is not particularly useful until complete - we could ‘deliver’ a copy of this that works with only v2.02 data, though the practical use of this is extremely limited. This has been shown by small one-off scripts and programs we’ve built upon pyIATI to fulfil internal needs (which additionally help to demonstrate that the functionality will be super useful once complete).

In terms of this phase taking as long as it has, we have spent around 2 months actively working on implementing ‘The Standard’, earlier work primarily being prototyping and proof-of-concept phases. A major part of this has been actively working to deal with the various oddities of IATI. Our work has uncovered a range of issues with ‘The Standard’ that had been missed previously. In terms of the scale of these problems, we currently have a list of 67 non-trivial issues with the Standard, 22 of which are categorised as ‘Critical’ - Dale Potter is leading on how to take these issues forward. With the limited capacity that we have, 2-3 calendar months is not an unreasonable amount of time to implement a tested reference implementation of ‘The Standard’ that is as to-the-letter as possible.

We’d love to spend our time working on other things, though more importantly we want to make sure that what we’re working on is based on a correct understanding of IATI. Without this foundational work (as required by 3A in the workplan agreed by the MA in 2016), other tools are guaranteed to either only partially implement ‘The Standard’, or replicate effort in some manner.

Imogen Kutz • 6 years ago

The blog was written to be accessible by the wider IATI community, so avoids many of the technical details of how we use specific agile processes. For example, we use daily standups, sprint boards and more to manage our work. These processes are under continual review and we aim to be as flexible as possible while maintaining a high standard of work.

In terms of pyIATI, it seems there is some confusion as to what it is. Rather than a specific tool the core focus has, as mentioned previously, been to create a software library to act as a reference implementation of ‘The Standard’. Other tools may then be built upon the library without having to worry about the complexities and issues introduced in historic Standard versions (each of which greatly increases implementation difficulty). It should also be noted that pyIATI will by no means be feature-complete by the end of our current set of sprints.

The project, as required in the workplan section 3A, is based on input from a range of people. This includes feedback from the TAG; the business analysts in the tech team; our past and present experiences of the many codebases we have to maintain; and general feedback about the future of IATI. This input was analysed to come up with the range of user stories as mentioned in the blog post. In total we came up with 50 stories that we ranked using the MoSCoW method, leaving 7 essential features based on 27 user stories. These were added to an internal Trello board and have been used to guide the direction of development.

During development of pyIATI, we’re working to engineer good code through use of test-driven development, documentation, consistent linting, human-readable variable names, SOLID principles and more. Each of these are frequently missing from existing IATI code for a number of historical reasons. With the requirements of the present and future we need to look at how to provide, not for 50 or 500 publishers but for 5000 or 50,000.

Take the Dashboard, for example: it is built with an architecture that is unable to scale much further without having to sacrifice key features such as daily updates - even now it can take over 48 hours for it to update after publishing new data. pyIATI is designed to streamline processes and act as a new architecture that will be able to scale for the future in a way that current tools cannot.

Hopefully this gives a bit more information than could be reasonably conveyed in the blog post. There is due to be another post in the near future providing even more detail as to why pyIATI is so vitally important for the continued success of IATI. This will go into more detail than has been provided here - if there are any more questions we could look to answer them there.

Andy Lulham • 6 years ago

Thanks, both.

Imogen_Kutz:

In total we came up with 50 stories that we ranked using the MoSCoW method, leaving 7 essential features based on 27 user stories.

Imogen Kutz : Would it be possible to share these user stories / features?

Please log in or sign up to comment.