Standardized Data

What is standardized data?

Standardized data is data from different sources that has been transformed into a consistent, standards-based format. The data standardization process involves harmonizing data so that all entries in different datasets that relate to the same terms all follow the same format, allowing them to be compared meaningfully.

Examples of the types of data formats that require standardization include:

• How addresses are recorded and displayed
• Capitalization (or not) of job titles
• Data formats (for example choosing between DD/MM/YY and MM/DD/YY)
• Time formats and timezones used
• How email addresses are recorded
• How website addresses are recorded (such as including/not including https://)
• How phone numbers are recorded and displayed (such as with/without country codes)
• How state names are recorded (either in full or abbreviated)

As well as common terms such as addresses and phone numbers, different industries may also have their own standards. These common data models are designed to increase the interoperability of datasets in the sector through standardized formats. For example, in healthcare, how data is formatted may vary widely between different healthcare providers within their internal systems. By applying an industry-standard common data model this data can then be shared confidently between providers, and also with regulators and governments, such as during the COVID-19 pandemic.

Data standardization is different to:

• Data cleaning/data cleansing, which involves identifying and fixing incorrect, incomplete, duplicate, unneeded, or otherwise erroneous data in a data set. Data standardization does not fix incorrect data, just formats it consistently.
• Data transformation when data is enriched with additional information and datasets, such as by adding geographical information. Data standardization does not involve adding additional information, just applying a standard format to what is there already.

Why is standardized data important?

Harmonizing data formatting through data standardization is essential to:

• Ensuring data quality and consistency, making standardization an essential element of data governance.
• Enabling the interoperability of datasets from different sources, particularly from departments within an organization or from external sources.
• Being able to make accurate comparisons between datasets. Essentially you are able to compare “apples with apples”.
• Creating trust in data and thus increasing usage across the organization and beyond
• Data democratization. Without confidence that data is standardized and consistent, employees, citizens and partners will not rely on or use data sources, holding back data democratization.
• Better, more informed decision-making based on accurate, standardized data.
• Being able to run cross-functional analytics comparing different datasets in a meaningful way.
• Removing the cost and inefficiencies of having to manually update or compare different datasets and work around format differences.
• Successfully applying AI and machine learning algorithms and gaining meaningful results

How do you standardize your data?

Achieving standardized data is a multi-stage process, that follows these key steps:

• Audit all data sources and understand the information they contain. This includes the data type, frequency, importance, size and whether internal or external. Research and understand the needs of all data users within the organization/ecosystem.
• Define and agree standard formats for data across the organization, such as how dates, addresses, and phone numbers will be recorded. There are a range of standards that can be adopted, such as ISO 8601 for date and time data formats. Ensure everyone understands – and uses – these standards.
• Import data from your internal and external sources into your data platform.
• Apply processors to the data source to correct any formatting differences and to standardize data.
• Validate that changes have been made successfully by testing data fields
• Once successfully standardized, data can then be published, shared and visualized.