Data Mesh

What is data mesh?

Data mesh is an enterprise data architecture based on a distributed, decentralized approach to managing and sharing data.

It is designed to increase the use of data across the organization, enabling companies to become more data-driven by making it faster to scale, share and create data products. Data mesh therefore supports strategies that aim to deliver data democratization.

The concept was originally proposed by Zhamak Dehghani of consultancy Thoughtworks in 2019, and has since been developed and adopted by multiple organizations.

Unlike a lot of previous data architectures it focuses on the organization itself, rather than technology. It looks to decentralize responsibilities for particular data to those that are closest to them, but backed up by agreed, company-wide governance and metadata standards to ensure interoperability, with the architecture enabled by a shared self-service data infrastructure.

Essentially, it is a federated model, a bit like the United States of America, with some central government but with power and responsibility in the hands of the states, those closest to citizens (data owners).

What are the principles of data mesh?

Unlike concepts such as data warehouse or data lake, data mesh is not a specific tool or technology. Instead it is a set of principles that define how companies govern, work with, and share data within the organization.

It is based on four key principles:

1. Domain ownership – rather than a central team, data is owned by those that are closest to it, such as those that create it. These independent, distributed teams are responsible for ensuring data is available, discoverable, addressable, trustworthy, reliable, interoperable, and understandable by all.
2. Data as a product – data is seen as a product supplied by teams to their customers across the rest of the business, meaning it must be designed to meet their specific needs.
3. Self-serve data platform – data has to be available to all through self-service, so it can be accessed easily without requiring additional support.
4. Federated computational governance – data governance and metadata standards are agreed and managed centrally to ensure interoperability, consistency and security.

What are the benefits of data mesh?

Organizations want to be able to share data internally to improve decision-making, increase transparency, and drive innovation. Data mesh underpins this through:

• Faster access to understandable data products by the whole business, making data part of everyone’s role.
• Simpler and quicker development of data products as independent data teams are responsible for their own data, rather than everything having to be controlled by a central data team.
• Greater reusability – teams share processes and learn from each other, delivering faster results with less resources.
• A common language and vocabulary around data that is shared by the whole business, ensuring consistency and common understanding.
• Centralized governance ensures common standards around security and metadata, meaning that regulatory compliance needs are met.
• Flexibility in terms of tools. Teams can use whichever tool is the best fit for their needs, giving them independence and increasing their buy-in to the program.
• Empowered teams. Individuals within independent data teams are trusted as domain experts, with their expertise valued, further driving engagement and maximizing use of resources.

How is data mesh different from other data management methodologies?

There are two main differences between data mesh and other data management methodologies:

• Data is not centralized (as in a data warehouse or a data lake). The federated nature of data mesh means that data owners are distributed across the organization, supported by centralized governance.
• Data mesh is not a specific technology or tool. Essentially all existing tools are data mesh compatible, meaning companies can start with their strategy and aims, and then deploy the right tools to meet their needs. This reduces the risk of projects failing, taking too long to implement or not delivering enough results.

What are the challenges to successfully implementing data mesh?

Unlike traditional data projects, data mesh relies less on technology and more on implementing a data-driven approach across the organization. This can lead to three main challenges:

• Adopting data mesh requires internal transformation to build a common data culture. This means breaking down silos and extensive change management across the organization. This takes investment of time and resources.
• Common rules around data governance need to be agreed, put in place and then followed across departments. This requires engagement and buy-in across teams.
• Organizations need to be able to take a strategic approach that identifies problems to solve first, rather than simply adopting technology. This requires data maturity, built on a data culture.