What is data mesh and why is it vital to data democratization?

Data mesh is an enterprise data architecture designed to underpin data democratization and increased use of data across the enterprise. Originally proposed by Zhamak Dehghani of Thoughtworks in 2019 it focuses on creating a distributed, decentralized way of dealing with data, making it faster to share and create data products..

Given the growing interest in data mesh we asked Opendatasoft’s CTO and co-founder David Thoumas about how it can help organizations become more data-driven and enable data democratization.

To start with it is important to understand that data mesh is not about specific tools or technology. Instead it defines how you govern, work with, and share data within your organization. Unlike a lot of previous data architectures it focuses on the organization itself. It looks to decentralize responsibilities for particular data to those that are closest to them, but backed up by agreed, company-wide governance and metadata standards to ensure interoperability, with the architecture enabled by a shared self-service data infrastructure.

Essentially, it is a federated model, a bit like the United States of America, with some central government but with power and responsibility in the hands of those closest to citizens (data owners).

Data mesh provides a framework and guidelines to help you work with data in the most optimized way focused around these building blocks:

• Distributed data products oriented around domains, owned by independent cross-functional teams which can contain both embedded data engineers and data product owners
• Centralized governance to ensure interoperability, consistency and security
• A common data infrastructure to host, enhance and share data

Previously the overriding concept was that data needed to be centralized, whether in a data warehouse or a data lake. Architecture choices were driven by technology choices, rather than necessarily the needs of the business. And this concept continues to prevail in most organizations worldwide.

In contrast data mesh is not linked to any specific tools – essentially all existing tools are data mesh compatible. So for example you can build a data mesh on top of your data warehouse. This means you can define the problems you need to solve first, rather than focus on technology. Data mesh therefore helps you use the right tools for the right problems, reducing the risk of projects failing, taking too long to implement or not delivering enough results.

Whatever the sector, every organization is looking to better leverage their data to drive greater efficiency, transparency, innovation and new revenues. That’s why more and more are launching internal self-service portals to share data with employees and are also creating external data products to share with their wider ecosystem.

Data mesh helps deliver this vision through six key benefits:

1. Increased innovation as everything is in place to help employees develop data products for (internal and external) reuse

2. It removes the bottleneck of everything having to go through a central data team – so helps drive a shorter time to market and better allocation of resources, using the skills within teams

3. Greater reusability – sharing data products fosters reuse of what has been done by other teams delivering faster results

4. Better communication across the business on how you handle data through a shared vocabulary and language through terms such as data domains, data owners and data products

5. Only governance, maintaining the self-service architecture and the rules covering metadata are centralized. That means teams can use whichever tool they want/is the best fit for their needs. This gives teams more ownership and buy-in, which drives engagement and participation. It also lowers costs for tools in terms of licensing and training. This is because the needs are more precisely defined and as a consequence, organizations can buy licenses tailored to their exact needs.

6. Empowers teams and helps with collaboration. Individuals are trusted as domain experts, with their expertise valued. They feel part of the data team.

Adopting the data mesh concept is a transformation – it doesn’t happen overnight. Like any change project it requires a great deal of investment of both people and time. You need to have the resources in place to work on building a data culture, focused on rules around governance, literacy, and quality.

It requires knowledge and alignment between departments to work, breaking down silos to encourage data re-use. This is backed up by a need for data maturity – organizations need to identify problems first, before thinking of solutions. In many ways this can seem more daunting than simply buying a data tool and implementing it, but there are now many consultancies around that can help put the framework into action.

At Opendatasoft we believe in data democratization. Everyone benefits when it is easy to share and use data-driven information in our professional and personal lives. So we’re passionate about making it easy for all business users to work with data. This closely aligns with the vision behind the data mesh concept.

As I’ve explained, data mesh is not about choosing specific technology or tools. However, many of the features our all-in-one data experience platform offers makes building and operating a data mesh simpler. For example:

• If data is stored in multiple, different systems across the organization, you require a tool that can seamlessly connect and integrate between all of these solutions. That’s something our platform does well through its wide range of third-party connectors.
• We’ve always been focused on making things simple, so that data can be accessed and used by non-experts – exactly the sort of business users data mesh encourages to become data owners.
• We have built-in governance features that make enhancing quality, user management and security straightforward across the organization.
• The platform is designed to make it easy to create and share data products, and our rights management capabilities enable sharing with other teams and outside your organization through data access control.
• Our vision is built on the same concept of federated data management as data mesh. You can give business units their own domain through our workspace feature, while all being part of the same license. It is easy for data owners to share all or part of data products between domains.

At a company level ODS’s Customer Success Management approach focuses on mapping out audiences and stakeholders, understanding their issues and then explaining how data can solve them. This focus on identifying problems first, before looking at technology is very similar to data mesh.

Summing up, it is really positive to see the growing traction behind data mesh. It recognizes that data sharing is vital, but that success is about more than just technology and that you need to adopt a hybrid, agile approach that mixes central governance with local data ownership. It paves the way to deliver more data value, faster and for everyone.