Datastore Search is launched!

Thank you and congratulations to the team! 

Thank you team and congratulations.
Can IATI please tweet and post this news as news on IATISTANDARD.ORG as it is news that should be shared proudly beyond IATI connect. 

theo.sande  

Hi leo stolk, here are all links to comms published on Datastore Search:

Website post: https://iatistandard.org/en/news/datastore-search-is-launched/
Twitter: https://twitter.com/IATI_aid/status/1501577029807972361?s=20&t=I5kaUmNlL2Z0J_6gLPyFWw
LinkedIn: https://www.linkedin.com/posts/international-aid-transparency-initiative_iati-datastore-search-is-launched-activity-6907343446479409152-chS7
Facebook: https://www.facebook.com/IATIaid/posts/399471458679426

thanks Rohini Simbodyal - IATI Secretariat  , appreciated !

Thanks

> Stay updated with the Technical Team’s work by engaging with IATI’s Technical Community of Practice on IATI Connect.

Please can you point me towards where this Datastore Search project is discussed in the CoP?  I can't find anything, but may not be looking in the right place

It would also be helpful to understand the roadmap for this and other products, and how best we can feed in. 

> Users are encouraged to contact the IATI Technical Team with suggestions for additional features and guidance. To gain support to use Datastore Search or for any other data use queries contact the IATI Technical Teaml: support@iatistandard.org.

As evidenced elsewhere (over the years) I am personally not a fan of pushing such conversations into private emails.  Surely if we want to use and learn with such tools, we should take advantage of public fora.  Is there a commitment to feed back (perhaps to the aforementioned CoP) the common queries and requests from people?

It's great that d-portal is mentioned here.  Just today, I was (with the fantastic support of shi) looking at a complex query whilst using the dquery backend.  It'd be great to understand how such searches could be met in this new tool, and how users can contribute their (increasingly complex) needs.

I really like the simple search box, and also the detailed interface exposing all the elements - good envelope pushing, but the simplicity seems to have come at a high cost e.g. the search box does not return what many users will expect, and the detailed interface means you need to have a second screen open to read the standard to know what you want. The user story previously targetted by the QB doesn't seem served by either of the replacements........can the user story of the new interfaces be shared?

As an example, I tried to use it to get projects in 'Sudan' today, the simple interface also returns 'south sudan' projects, so no use there.  So then I have to a) go to 'advanced', b) have to guess the right xml element (just about OK for me), c) scroll for the country code (from a list I have to google EDIT: dropdowns are available - thanks Amy!)).....this was previously 3 seconds with a dropdown in the old QB, which also performed a really good guiding role to likely important filters.

I also searched for FCDO in the search box.....I don't think it returned any FCDO published projects, just ones that mentioned FCDO in the narratives - is that really the expected behaviour?

I guess you have exposed that cool and simple interfaces need a fair amount of smarts behind them to make sense e.g. hints 'were you looking for country/publisher X', and e.g. when selecting an element in the advanced interface, showing the applicable vocabs with narratives (EDIT: solved).

RE D-portal v2 vs datastore v3.....presumably the single search box will be moved to D-portal v2 (which specialises in non-technical users).....and the advanced interface remains as the way for non-api users to generate complex queries?

It would also be nice if the query url was generated e.g. to save for later attempts, is that possible? Also if the links in the top right to advanced/simple were consistent between the two pages i.e. always showing both options.

Thanks for sharing this - it's clear that a lot of hard work has gone into this, and it's a technically interesting approach, so congratulations to the team.

It would be great to hear more about who the intended users are for this. I read here that it's been designed with journalists in mind, but it's unclear to me how they would use the search functionality. I also struggle to see how the output formats could be used by journalists without very extensive technical skills.

In recent DUWGs, we discussed three broad sets of users that Datastores have tended to cater to over the years:

(1) bulk export / download, e.g. for powering country systems such as AIMS
(2) aggregation / summary, e.g. for powering simple visualisations or dashboards
(3) analysts, e.g. for running analysis using something like pivot tables in Excel

Of course, this is not a complete typology, but perhaps it's useful for thinking through the objectives of these kinds of tools.

Look forward to discussing further and gathering feedback from everyone at the next DUWG. I'd also echo others' comments, that it would be great to understand more about plans for consultation and development going forward.

Hi everyone,

Not sure why another version of Datastore has been built as https://IATI.cloud does all of the above + so much more. It also allows user to search on https://aida.tools/ where actual use cases for search have been developed with a very clearly defined output into a set of categories (including proper downloads etc.)

Without that it looks as 'Datastore Search' is just an input form in HTML interfacing with Apache Solr w/o any proper specs as to what is expected to be shown to user. Disappointing but probably as expected. Looks like it was delivered like 'hey this queries the Solr index, great!' w/o any consultation with the community / CPO what have you. Good luck with that.

As this 'new' Datastore has not been built according to the actual specifications Datastore V2 (https://IATI.cloud ) was mandated to build against it feels like this attempt at a build for just another Datastore w/o proper specs and looks clunky by the time it was delivered into the field. Like an old T-72

Sorry to say this all looks very IATI 2014 and not IATI 2022....

We (Open Data Services Cooperative & 360Giving) first launched a search tool on top of 360Giving data (called GrantNav) in 2016. 

As the initiative has grown, alongside the complexity of data needs of the community, there have been several improvements, including a bunch in late 2021.

All these were coupled with extensive and ongoing user research and testing - including efforts to support those that really wanted to just use freetext search (which can be very hard to get what you need)

GrantNav also illustrates another principle that was discovered: search and then filter (or facet).  Hence, people make a broad search, and then use the available data points to refine down their results.  People also stressed how important it was to download the data in a common and useful format.

It'd be great to see some evidence of this in efforts from IATI. The 360Giving team would be very happy to help,I'm sure.

Thank you all for your feedback so far on Datastore Search. Our aim was to develop and launch the MVP, as per the IATI Governing Board’s steer. We are now gathering comments and feedback to be considered for development as we plan a roadmap for the tool. As matmaxgeds  has mentioned the tool offers a larger scope than previous IATI query builders but does need refinement and further features to help users utilise that scope.

As for collecting feedback, the DUWG are doing a great job of listing out features that different user groups would require. Steven Flower  happy to do similar in the technical CoP. The IATI tech team can present a summary of feature requests already collected as a basis. Aware that a large number of Datastore users aren’t on IATI Connect so parallel collection mechanisms are good. Steven Flower  if the 360Giving team are happy to share their findings on best ways to implement free-text search, and your method for returning results by relevance it would be great to learn from them.