How AI is transforming our data portal solution and client data projects

Progress in Artificial Intelligence (AI) is accelerating rapidly, with the pace of adoption increasing significantly in 2024 with the launch of new applications and technologies based on AI models.

When it comes to data management, which involves many manual, time-consuming tasks, AI is also able to revolutionize processes and tools.

Consequently, over the past six months, Opendatasoft has been working to transform its data portal solution by enriching it with AI. The aim is to help our clients to save time, improve the experience for their users, and reduce the risk of errors within processes.

We are committed to accelerating data democratization for our clients by applying AI within our solution.

An intelligent AI-based search engine

Since December 2023, Opendatasoft portals can harness a new, semantic search based search engine. This goes beyond literal, keyword matches to understand the meaning of terms and their context.

The new search engine provides users on client portals with real-time result suggestions and access to:

• Datasets corresponding to the searched theme
• Data assets corresponding to the searched theme
• Data assets that include the searched keyword or a similar keyword

This new search method delivers more relevant results, faster. It greatly enhances data discovery, providing a smoother and more efficient user experience. For portal administrators, this also means easier maintenance, as there is no longer a need to add an exhaustive list of keywords to metadata — AI handles it all.

Similar dataset recommendations

To improve data discoverability and streamline the experience on our clients’ data portals, we also offer an AI-based automatic data recommendation feature.

Through this all other most relevant datasets are displayed on data asset pages viewed by users.

This approach aims to simplify the data discovery experience for all users, drawing inspiration from the best practices of e-commerce marketplaces. It aligns perfectly with Opendatasoft’s vision of making data access and use as simple and intuitive as possible for a broad audience, from data experts to novices. Each user can easily navigate through all data assets on a portal to find exactly what they need.

Our choice of migrating to semantic search stems from a continuous desire to improve the user experience within our data portals. Search is a crucial pillar of this experience, enabling users to easily discover the data they are most interested in. With keyword search, users are often frustrated and receive irrelevant or incorrect results  if they do not enter exact keywords used to describe data assets. Instead, semantic search offers greater flexibility, generating relevant results even for approximate or imprecise queries or those in other languages.

Enhancing the relevance of our AI-based search engine

Our journey of integrating AI into our search engine has been iterative. It began with a vector approach, which managed approximate queries with spelling errors or synonyms, with variable result relevance. We then adopted a hybrid approach, combining keyword and vector methods, ranking results by semantic similarity and popularity, thus improving their relevance. Finally, we returned to a solely vector approach, enriched with a relevance algorithm to present only the most pertinent results and display nothing if results are insufficient. This latest improvement ensures optimal result quality, considering the approximation or imprecision of user queries.

Continuous iteration delivers improving results

To measure the success of our search engine, we use a relevance index that scores results on a scale ranging from 0 to 1. Thanks to our work on different approaches, this index has risen from 0.52 to 0.87 in just a few months.

Although this relevance index is crucial, it is not the only factor we consider to ensure search engine quality. As a solution provider, we have to meet the different and unique needs of all of our clients.

Our beta-tester clients have played a central role in the continuous improvement of our search engine. Their feedback throughout the development process, along with the continuous learning of our AI algorithms has helped shape a more user-centric search tool.

Our use of AI isn’t limited to search. We now have a range of new AI-based features under development or being tested with our clients, which will soon be available.

Exploring data with AI

Soon, users of Opendatasoft data portals will be able to rely on AI chatbots to explore data. By querying a virtual assistant using natural language, they will receive instant answers in specific visualization formats, such as maps, figures, or graphics.

Building pages with AI

Portal administrators will also soon be able to use AI to create and integrate complete pages on their portal. Whether generating dynamic visualizations, building pages, or arranging elements, a virtual assistant will automatically respond to their prompts.

If you want to learn more about the upcoming AI innovations that are enriching Opendatasoft’s solution, contact us to speak to one of our experts.