Internal data marketplace

What is an internal data marketplace?

An internal data marketplace (IDM) effectively and securely shares data through self-service with all employees across the entire organization. It centralizes all types of data asset (including data products, visualizations, and tabular data), and makes it available in an intuitive, easily discoverable and consumable way, without requiring technical training or support. This scales data sharing, eliminates data silos, and maximizes data value, all while preserving data governance and security.

Delivering the same user experience as an e-commerce website, the internal data marketplace simplifies the sharing and consumption of data, building trust among users, increasing collaboration, reducing administration, and enabling data owners to easily create and publish data assets such as data products.

How are internal data marketplaces different from data warehouses, data lakes and data catalogs?

The growth of big data means that organizations are generating and collecting ever-growing volumes of data. At a technical level this is being stored and managed through data warehouses and data lakes, and inventoried through data catalogs. These are used and run by technical and data experts.

Internal data marketplaces do not replace these technical tools, and often integrate closely with them. The difference is, that rather than being a solution for technical specialists, they provide a user-friendly, centralized space where any employee can easily access the data they need in their working lives, without any training or support. They provide a digital storefront to an organization’s data, enabling it to be effectively re-used at scale. 

Who uses an internal data marketplace?

An internal data marketplace brings together three groups:

• Data owners who create and manage data assets, such as data products, that are published on the marketplace 
• Marketplace administrators, responsible for managing and monitoring the marketplace
• Employees who discover, access and consume relevant data assets 

Internal data marketplaces enable these three groups to collaborate and drive forward data sharing and data democratization.

What are the benefits of an internal data marketplace?

Internal data marketplaces deliver key benefits to organizations by scaling data use:

• Improved data accessibility, breaking down internal data silos and enhancing collaboration and innovation
• Better informed decision-making through access to complete data
• Increased operational efficiency by automating data sharing
• Greater ROI on data investments by maximizing data consumption and value
• Streamlined data governance through centralized, comprehensive control over data use, ensuring regulatory compliance 

What are the challenges to creating a successful internal data marketplace?

As with any marketplace, an internal data marketplace has to meet the requirements of both sellers (data owners) and buyers (employees consuming data), as well as the platform’s administrators.

There are three key challenges to building a successful internal data marketplace:

• Ensuring usability – it has to be easy for users to quickly find the data they need through self-service, in formats that they can consume without requiring expert support
• Guaranteeing data quality – data has to be reliable, high-quality and trusted by users
• Maintaining data governance and security – it must support organizational data governance policies and incorporate robust security, particularly around who can access specific data 

What are the components of an internal data marketplace?

A successful internal data marketplaces needs to be built on key components and capabilities:

• An intuitive, self-service interface modeled on that of an e-commerce site
• The ability to connect to all of an organization’s data, whatever system or storage space it resides in
• Powerful data discovery features that help users quickly and confidently find relevant data 
• Full lifecycle management of data assets, from their creation via publication to ongoing maintenance and data lineage
• Collaboration features, such as ratings and feedback tools that enable data owners and users to work together
• The ability to provide data in multiple ways, such as through APIs and data visualizations 
• Identity and access management to control and manage user access to the platform and specific data assets