Product News: AI enables intelligent semantic search and accelerates the use of large-scale data

Artificial Intelligence (AI) opens up new ways to improve the performance of data management tools, particularly around improving the efficiency and speed of completing different tasks.

That’s why Opendatasoft is launching a new AI-based feature: semantic search. This is based on a vector model for easier, enriched discovery of an organization’s data assets on a data portal. To find out more, we interviewed Emmanuel Daubricourt, VP Product at Opendatasoft.

Access to data is a major challenge for organizations, and data portals represent the right answer for data leaders looking for a fast, large-scale and easily deployable solution. At the same time, given the variety and diversity of data available on data portals, it is essential to constantly improve search performance and relevance.

Search is at the heart of a successful data portal experience. Users need to be able to confidently search and find information just as easily as they would be able to when searching the web. To drive data democratization and increase the use of data, search must make it easier not just to find what users are looking for, but also enable them to discover data which they may not be aware of, but which is relevant for their purpose.

Artificial Intelligence offers an unrivaled level ability to deliver this type of deep, scalable functionality. That’s why it seemed obvious to us to develop a new search engine to enhance the experience on our customers’ data portals.

Opendatasoft portals now feature semantic search, based on a vector model. This means that the search engine goes beyond literal matches of the search word and its variants, to understand the intent and contextual meaning of search terms. For example, a query on the word “gasoline” may return results with terms such as “fuel”, thanks to this understanding of meaning.

Furthermore, our search engine takes into account the terms present in the metadata, which further increases the possibility of finding relevant results for each query.

Compared to traditional keyword search, semantic search returns more relevant results and runs faster, speeding access to the right data.

As well as offering more relevant, deeper search results, our new search engine improves the user experience by suggesting results in real-time, as users type, just like Amazon or other e-commerce sites. This enables users to gain more in-depth results and identify at a glance the data assets that meet their needs.

Semantic search delivers numerous advantages. Users benefit from faster, more efficient search, which increases their productivity. They get more relevant search results using simpler queries and minimize the number of searches they need to make to find truly relevant data. The aim is to reproduce the experience found on an e-commerce site, so that data can be searched just as easily as an everyday consumer product.

Secondly, discovering data assets is much easier, regardless of the volume of data available on the portal, and the technical terminologies used to define them. The experience is smoother with a significant reduction in search time.

Portal administrators also benefit. They can see increases in data usage and more frequent interactions with their data portals, building loyalty among users through greater confidence in data.

It also enables data producers to contribute metadata more simply, by describing the dataset naturally, just as if you were having a conversation, rather than by adding as many synonym keywords as possible to improve dataset visibility. As well as being time-consuming, the synonym approach risks overwhelming – and losing – portal visitors as it provides them with large amounts of redundant or irrelevant information.

This new feature is part of our mission to democratize data for everyone. It makes data even more accessible to all kinds of audiences, and leads to increased adoption and use of data portals.

Yes, this feature is just the first of many. From a user experience point of view, AI helps us to deliver impressive results that will radically change the way data is consumed in the near future.

We identified a number of avenues during the hackathon we ran with our teams last July and we are now building on this initial work. We are also continuing to explore a full range of possibilities, with the aim of developing features that will have a significant impact and enable our customers to be more productive.

In particular, we are focusing on four areas. We are aiming to provide recommendations for datasets which are similar to those being looked for, as well as offering extended search across all data formats and types of assets. To increase productivity we are also looking to provide tools that assist in the creation of data visualizations, and make translation of portal content faster and more automated.

If you’d like to find out more, our team will be happy to give you an update during a demo of our solution.