[REPLAY] Product Talk: Using AI to enhance the data marketplace search experience

Watch the replay
Product

How AI is transforming our data portal solution and client data projects

AI

Over the past months Opendatasoft has been working to transform its data portal solution by enriching it with AI, helping clients to save time, improve the experience for their users, and reduce the risk of errors within processes.

VP of Marketing , Opendatasoft
More articles

Progress in Artificial Intelligence (AI) is accelerating rapidly, with the pace of adoption increasing significantly in 2024 with the launch of new applications and technologies based on AI models.

When it comes to data management, which involves many manual, time-consuming tasks, AI is also able to revolutionize processes and tools.

Consequently, over the past six months, Opendatasoft has been working to transform its data portal solution by enriching it with AI. The aim is to help our clients to save time, improve the experience for their users, and reduce the risk of errors within processes.

Copy to clipboard

We are committed to accelerating data democratization for our clients by applying AI within our solution.

An intelligent AI-based search engine

Since December 2023, Opendatasoft portals can harness a new, semantic search based search engine. This goes beyond literal, keyword matches to understand the meaning of terms and their context.

The new search engine provides users on client portals with real-time result suggestions and access to:

  • Datasets corresponding to the searched theme
  • Data assets corresponding to the searched theme
  • Data assets that include the searched keyword or a similar keyword

This new search method delivers more relevant results, faster. It greatly enhances data discovery, providing a smoother and more efficient user experience. For portal administrators, this also means easier maintenance, as there is no longer a need to add an exhaustive list of keywords to metadata — AI handles it all.

Similar dataset recommendations

To improve data discoverability and streamline the experience on our clients’ data portals, we also offer an AI-based automatic data recommendation feature.

Through this all other most relevant datasets are displayed on data asset pages viewed by users.

This approach aims to simplify the data discovery experience for all users, drawing inspiration from the best practices of e-commerce marketplaces. It aligns perfectly with Opendatasoft’s vision of making data access and use as simple and intuitive as possible for a broad audience, from data experts to novices. Each user can easily navigate through all data assets on a portal to find exactly what they need.

Copy to clipboard

Our choice of migrating to semantic search stems from a continuous desire to improve the user experience within our data portals. Search is a crucial pillar of this experience, enabling users to easily discover the data they are most interested in. With keyword search, users are often frustrated and receive irrelevant or incorrect results  if they do not enter exact keywords used to describe data assets. Instead, semantic search offers greater flexibility, generating relevant results even for approximate or imprecise queries or those in other languages.

The semantic (or vector) approach eliminates potential user frustration and significantly improves their search experience. Implementing AI was a no-brainer once we evaluated and compared several AI models, the best of which provided a 64% gain in relevance compared to a keyword search engine.
Damien Garaud
Backend Software Engineer, Opendatasoft

Enhancing the relevance of our AI-based search engine

Our journey of integrating AI into our search engine has been iterative. It began with a vector approach, which managed approximate queries with spelling errors or synonyms, with variable result relevance. We then adopted a hybrid approach, combining keyword and vector methods, ranking results by semantic similarity and popularity, thus improving their relevance. Finally, we returned to a solely vector approach, enriched with a relevance algorithm to present only the most pertinent results and display nothing if results are insufficient. This latest improvement ensures optimal result quality, considering the approximation or imprecision of user queries.

Our unique and innovative algorithm effectively identifies the best ratio between the number of suggested results and their relevance. Our evolution from a vector approach to a hybrid one, and ultimately back to semantic search supported by this new algorithm, reflects a process of continuous evolution where we re-evaluate previous choices while enriching them with new knowledge. It is very much a scientific approach, where continuous exploration and the integration of new data is essential to delivering ever more effective solutions.
Damien Garaud
Backend Software Engineer, Opendatasoft

Continuous iteration delivers improving results

To measure the success of our search engine, we use a relevance index that scores results on a scale ranging from 0 to 1. Thanks to our work on different approaches, this index has risen from 0.52 to 0.87 in just a few months.

Although this relevance index is crucial, it is not the only factor we consider to ensure search engine quality. As a solution provider, we have to meet the different and unique needs of all of our clients.

Our challenge is to find the right balance between maximizing the number of relevant results and minimizing less suitable results in order to offer a satisfying experience to all of our users. For example, some clients prioritize the quantity of relevant results, while others place more importance on reducing irrelevant ones.
Damien Garaud
Backend Software Engineer, Opendatasoft
Copy to clipboard

Our beta-tester clients have played a central role in the continuous improvement of our search engine. Their feedback throughout the development process, along with the continuous learning of our AI algorithms has helped shape a more user-centric search tool.

The new AI-powered search engine makes it much easier to access and discover data. It suggests relevant data even when approximate keywords are entered, reducing the risk of missing out on the data users are actually searching for. It also enriches the user experience by highlighting data that is interesting but wasn’t directly searched for, creating serendipitous results. It also has a major impact on the quality of our metadata, ensuring we focus even more on its quality and completeness. As beta testers, we found the whole experience useful thanks to the openness of Opendatasoft’s teams when it came to listening to our suggestions and feedback. We’re very interested in seeing the next steps in AI-powered search, exploring the solution's forthcoming AI features, and collecting feedback from our users to better understand how these innovations help them find data on the portal.
Camille Liégeois
Head of Open Data & Digital City Projects, Issy les Moulineaux

Copy to clipboard

Our use of AI isn’t limited to search. We now have a range of new AI-based features under development or being tested with our clients, which will soon be available.

Exploring data with AI

Soon, users of Opendatasoft data portals will be able to rely on AI chatbots to explore data. By querying a virtual assistant using natural language, they will receive instant answers in specific visualization formats, such as maps, figures, or graphics.

Building pages with AI

Portal administrators will also soon be able to use AI to create and integrate complete pages on their portal. Whether generating dynamic visualizations, building pages, or arranging elements, a virtual assistant will automatically respond to their prompts.

If you want to learn more about the upcoming AI innovations that are enriching Opendatasoft’s solution, contact us to speak to one of our experts.

Articles on the same topic : Features Data Intelligence Data democratization Data portal Company news Artificial intelligence
Learn more
What are the benefits to using your data portal to feed AI models? Digital transformation
What are the benefits to using your data portal to feed AI models?

Learn how data portals enhance the training and effectiveness of artificial intelligence models by providing reliable, high-quality and trustworthy data, which is essential to ethically deploy AI and harness its benefits.

Product News: AI enables intelligent semantic search and accelerates the use of large-scale data Product
Product News: AI enables intelligent semantic search and accelerates the use of large-scale data

Opendatasoft is launching a new AI-based feature: semantic search. This is based on a vector model for easier, enriched discovery of an organization's data assets on a data portal. To find out more, we interviewed Emmanuel Daubricourt, VP Product at Opendatasoft.

Opendatasoft Hackathon: exploring the potential of Artificial Intelligence in data management tools Product
Opendatasoft Hackathon: exploring the potential of Artificial Intelligence in data management tools

What benefits could AI bring to users of SaaS-based data management software? To find out, Opendatasoft organized an internal hackathon that brought together over 30 developers in order to test ideas for improving our platform using AI. Read all about the results of these two days of intense work in this article.