The IATI Technical Team is currently working to review the large range of products it provides. This will ensure we are best able to provide the features and capabilities that are useful to publishers and users of IATI data. This review is being undertaken with an overarching long-term plan to migrate core IATI products to a new Architecture - as described in a separate post.
To help ensure core IATI products are useful to the widest possible audience, the Technical Team is making their product roadmaps public as it becomes reasonable to do so. It is desired that feedback from the wider community will provided and integrated into roadmaps as appropriate.
Background
At present the IATI Technical Team maintains a large number of distinct products. To best ensure that the products provided are useful to a wide range of relevant persons, it is important to have information about the current, intended and future status of each product. This may involve creating new products, maintaining existing ones, or closing down products that are no longer useful.
As part of a wider series of projects, personas based on the various IATI user groups are to be developed. Once finalised, these will form the basis of ongoing roadmap and feature discussions. The short-term roadmap until the personas are developed is designed to provide a strong architectural foundation for a sustainable future.
Product Roadmap
This Google sheet contains the high-level IATI Technical Team product roadmap as of late-February 2017.
Product Roadmap Table
-
The Product description and Product purpose columns define what a product should do and why.
-
Technology Stack and Current Status define the current technical status of products
-
The Headline long-term development status column gives the summary of our plan for each product in the longer term - see below for more details.
-
The Long and Short-term development plans columns offer more detail on the intended future for products over varying timescales
Headline long-term development statuses
There are several states that a product may be in.
-
New product: There are plans to provide some new functionality that is not provided by the IATI Technical Team at present. In the roadmap this applies to:
-
Python library
-
Conversion Tools
-
Improve: The base functionality that exists is either useful in its current state, or there is a clear path to make it useful. It is likely that the improvements will be based around the new architecture. In the roadmap this applies to:
-
Standard Single Source of Truth (SSOT)
-
Website
-
Datastore
-
Datastore Query Tool
-
d-Portal
-
Validator
-
Dashboard
-
3rd party products: Some 3rd party products, such as a discussion forum, are used to provide IATI functionality. New versions of these products will be upgraded to as they become available. In the roadmap this applies to:
-
Registry (CKAN)
-
Community Forum (Discuss)
-
IATI Support platform (Zendesk)
-
Maintenance only: No new features will be added, though if it breaks in some manner the Technical Team will endeavour to make it work again. In the roadmap this applies to:
-
Data Tickets
-
Publisher Guidance and Wizards
-
Close down: The end result will be that the product is shut down. For some products that contain written content, the copy will be migrated to the main IATI website before the product is closed. Where appropriate, clear timescales will be announced to ensure there are opportunities to migrate to utilising alternative products. In the roadmap this applies to:
-
Preview Tool
-
Aid Info Labs CSV Converter
-
CSV2IATI Platform
-
IATI Wiki
-
Google Group
-
Open Development Toolkit
Short-term plans are more concrete at this point in time than those longer-term. Up to the end of 2017 Q2, we plan to allocate time for:
-
Maintenance: All currently supported IATI products must keep working until they are closed or replaced.
-
Time-sensitive work: The IATI Technical Team is involved in a number of projects that are to be completed short-term. These include managing a upgrade to create v2.03 of the IATI Standard, and (independently-funded work) to add Grand Bargain monitoring on the IATI Dashboard.
-
Architecture design and development: Create a solid foundation for future development to build upon. This includes development of the initial stages of an IATI python library, which will offer functionality to deliver common IATI tasks (e.g. fetching and loading files, data validation, extracting data, etc). This library will be open-source and reusable by 3rd party tool builders and developers.
-
Validator development: This will be used to demonstrate that the aspects of a new architecture relating to SSOT representation and communication between services works. It will provide capabilities beyond that of the existing public validator, notably a public API.
-
Datastore development: Improvements to the official IATI Datastore are essential to better support data use. Specific use cases will begin to be scoped at the forthcoming TAG and we will then scope product options. Depending on results, this may include a thorough overhaul of the current Datastore, or formal adoption of a currently third-party product.
Ongoing Roadmap Development
Once a cast of user personas for IATI have been developed and validated, with the support UX consultants, a more thorough look at user needs and how to meet them through the provision of technical products will be undertaken. It is expected that this will occur towards the end of Q2 2017.
Feedback from the wider IATI community will also be utilised to help inform the direction of technical product roadmaps. We invite comment on these roadmaps in this thread as well as during the session at the forthcoming TAG.
Hi [~379] Andy Lulham - just catching up with this post, apologies for the long delay.
We will also be posting an update to the initial roadmaps in the next couple weeks too. In the meantime, here’s our write ups from each of the sessions that we ran at the TAG. Hopefully these are useful.
Consulting the community: IATI core technical product roadmaps
Session description (added to agenda before the session)
This session is designed to communicate how the IATI Technical Team plans to best offer useful functionality to publishers and users of IATI data.
The first part of this session will outline the long and short term plans for each of the 15+ public tools and websites that we currently maintain. This will be based on our recent post on the IATI Discussion forums.
Thereafter, we will facilitate discussion and feedback on the direction of future tools, looking at:
What functionality should we definitely keep?
What should be improved?
What should be added?
Please note that whilst the official IATI Datastore will be covered within this session, there will be a separate session to look at this product in depth: The future of the official IATI Datastore: Gathering user needs
Session format
Activity 1: Which IATi websites/tools do you use?
An A4 poster containing a screenshot for each of the 15 IATI websites/tools were affixed to the walls around the room. On arrival, each participant was asked to go to each of these and tick the sheet if they had:
Used the tool
Heard of the tool (but not used it)
Results: Based on the completed wall posters, the most widely used websites/tools are:
Registry
Documentation website
Discuss
Dashboard
Presentation:
The headlines of the Roadmaps post to IATI Discuss were presented.
Attendees were asked to write at the top of the participant questionnaires if they would like an email to the URL of the post.
These were sent w/b 20 March.
No further comments received as yet
Activity 2: User requirements for new IATI tools and services
In groups of 2 or 3, participants were asked to complete user requirements worksheets.
The resulting sheets are located here: Workshop Ideas.pdf
Feeding back
Key themes from participant worksheets
Accessing data (featured in 2 sheets)
Easy way to get the details of the project to create reports
by country
by donor
by sector
by location
by volume of interventions
As a data superuser/programmer, I want to be able to [have an] easy way to get the details of the project from the datastore so that I can get the details of the project
Finding the various IATI tools that are available - for publishing and data use (featured in 1 sheet)
Publishing CSV data to IATI (featured in 1 sheet)
Measuring data quality (featured in 1 sheet)
With grading of a file
To be able to spot potential data gaps
RESTful service
Better documentation (featured in 1 sheet)
Key themes from informal conversations
Key quotes
[From memory, prompted by the worksheets]
‘At the moment, it is a complicated process to generate a report of IATI data’
‘The mapping process for converting from CSVs to IATI needs to be simplified’
‘It’s hard to work with IATI data because there are often data quality issues‘
Consulting the community: The future of the official IATI Datastore: Gathering user needs
Session description (added to agenda before the session)
The IATI Datastore was launched in 2012 although has never progressed beyond alpha status. Whilst this tool experiences some use, it frequently suffers import problems and is difficult to debug and maintain from a technical point of view. Informal usability research has shown that significant improvements are needed to make this tool useful for wider data use, particularly for non-technical users.
The IATI Technical Team are committed to providing a more usable, robust and maintainable iteration of the Datastore. In this session we will begin to gather user requirements for a future data store. Outputs will form part of the detailed scoping resulting in technology choices that will lead to an improved product.
Session format
Activity 1: Which features of the IATI Datastore do you use?
An A4 poster containing a screenshot for each of the main sections/features of the IATI datastore were affixed to the walls around the room. On arrival, each participant was asked to go to each of these and tick the sheet if they had:
Used the tool
Heard of the tool (but not used it)
Results: Based on the completed wall posters, the most widely used features are:
CSV outputs
XML outputs
API queries built manually, assisted by documentation (i.e. as opposed to using the Query Builder, which provides a frontend ‘wizard’ for building queries)
Presentation:
A quick overview of the history of the IATI Datastore was given
The place of IATI Datastore within the public roadmaps for IATI products (i.e. improve the Datastore!) was presented.
Activity 2: User requirements for the new IATI Datastore
3 groups of up to 5 participants were formed, arranged according to the main interest groups that were present in the room
Developers
API users (i.e. tool builders – technical users)
Data analysts (non-technical users)
Participants were asked to complete user requirements worksheets.
The resulting sheets are located here: Workshop Ideas: Completed Worksheets.pdf
The ‘API users’ group wrote some notes instead - summarised below
Feeding back
Key user needs
Current datastore: Things to keep doing
Returning the raw XML blob is useful for debugging
Ordering of words by alphabetical is not at all useful in some cases
In general, JSON outputs might be generally favoured by developers
Current datastore: Things to improve
Regarding the Datastore query builder
Add ability to save queries for reuse
Make queries sharable between users - perhaps provide short URLs
Preview data to be returned
Clearer language should be used - e.g. tooltips for clarification
Make it easier to select multiple sector codes
There should be ability to export more data in CSV format
‘CSV is good because people understand the format’
Check CSV imports for all versions/languages of Excel as problems have been found with the use delimiters for non-English operating systems
Current datastore: Things to change
Add ability to query/filter on any aspect of the data, or even give a given XPath as an input
Mix of versions in the data that is returned, so the data is hard to use
Using the current datastore in end products means that the tools need updating when some aspects of the publisher’s dataset change.
Standardise versions of the output IATI data (to improve usability of the data)
Keep the service stable
Don’t make people change their tools when
Have a way of dealing with organisation identifiers that have changed: I.e. if searching for DFID data, the Datastore would return data for GB-1 and GB-GOV-1
Regarding internationalisation:
Need the Datastore to handle this in UI plus in data
Need to handle this all over IATI too
Reduce barriers to use
General points not specifically related to Datastore functionality
There is a lack of clarity on the difference between the Registry & Datastore
There should be a greater focus on validation to identify issues with published data. This new validation too should have:
An API
Error reporting
Feedback to publishers could by email notifications
Could include a ‘data health’ tool, with a traffic light system:
Red light: Data is not well-formed XML
Yellow light: Data valid but with issues - e.g. not using good data practices
What can you do with this data?
Key themes from informal conversations
Is there duplicated functionality between the Datastore and d-Portal?
Given there is some confusion about the differences between the Registry, Datastore and d-Portal, some participants questioned why are we adding extra confusion for users by sending them to different sites. IATI is complex to get your head round as it is!
Key quotes
[From memory, prompted by the worksheets and notes]
‘The Datastore is critical for enabling people to use IATI data.’
‘I want to be able to visualise data for specific queries/filters’
‘Data analysts and researchers are looking for an answer to a specific question’
‘CSV is good because people understand the format’
d-portal: Usability testing and requirements
Session description (added to agenda before the session)
D-Portal is website which offers basic search and visualisation of IATI data. It is developed and maintained by the IATI Technical Team.
This session seeks to further understand how D-Portal is used, and how improvements could be made to increase its effectiveness.
Session format
Activity 1: Which features of d-Portal do you use?
An A4 poster containing a screenshot for each of the main sections/features of d-Portal were affixed to the walls around the room. On arrival, each participant was asked to go to each of these and tick the sheet if they had:
Used the tool
Heard of the tool (but not used it)
Results: Based on the completed wall posters, the most widely used features are:
Search filters
Quick search
Activity view
Results view
**Presentation: **
A quick overview of the history of the d-Portal was given
The place of d-Portal within the public roadmaps for IATI products (i.e. improve d-Portal!) was presented.
No questions received
We asked attendees to write at the top of the participant questionnaires if they would like an email to the URL of the post.
These were sent w/b 20 March.
No further comments received as yet
Activity 2: User requirements for d-Portal
4 groups of up to 4-6 participants were formed
2 groups were formed of users who had used d-Portal before
2 groups were formed of users who had not used d-Portal before
Participants were asked to complete user requirements worksheets.
The resulting sheets are located here: Workshop Ideas: Completed Worksheets.pdf
Some groups wrote some notes instead - summarised below.
Feeding back
Key user needs
Attempted to list in order of popularity based on the session notes and worksheets.
Data should be downloadable in CSV/Excel
There are issues with the current CSV export functionality
Should be able to export implementing agency, plus start and end dates
There was an idea to link d-Portal and a report builder, so that you can move between them
Include definitions of fields (as tooltips?)
Add maps with ability to drill down to projects
‘Subject-based’ visualisations
There needs to be a greater focus on usability
Feedback mechanism
A data user reports a mistake with the data
The system then sends an email alert to the publisher
When the issue is fixed, the system sends an email to the person who originally reported the issue
The site should be multilingual
Add ability to search for activity by donor fund/activity ID
A data quality rating should be integrated onto the project page
Add a feedback feature to the website
CRS comparisons - e.g. smaller organisations involved with USAID are not reporting to IATI
Are there any measures to prevent double-counting?
Integrating content from the ‘Generators’ part of the site as part of a corporate site
The use of iframes is not considered good practice in 2017
Use a JS script instead?
Add a better explanation for where it could be used, and who owns the widget?
How do you prevent users from drawing the wrong conclusions about the data (i.e. where access to the data is greater than the understanding of the data)
Turn d-portal into more of a toolkit
Add a data dictionary
Select a geographic region to understand the funding landscape for that region
There should be a better way to select multiple countries/donors when searching
The site should use vector graphics where possible
The site should run under HTTPS
Key themes from informal conversations
It seems some users see d-Portal as the ‘go to’ place for data, even though d-portal is an opinionated version of data published to the IATI Standard
Could this add weight to the thoughts in the Datastore session, and perhaps there should be some consolidation of IATI websites
Key quotes
[From memory, prompted by the worksheets and notes]
‘How do I get the data that I need?’
‘I want to be able to visualise data for specific queries/filters’
‘Should d-Portal be an IATI data visualisation toolkit, rather than trying to be all things to all types of data user?’
Bumping this. Would be great to know where best to direct efforts. Thanks!
Dale Potter could you share with us all the notes / outcomes from the many Q&A sessions held at Dar? I’d like to focus on the sustainability of Datastore in particular. Thanks!