Issues and risks that slowly and unnoticed can make your data unusable

Please note

This document is still a work in progress.

Introduction

Businesses have become completely data-dependent. However, an ever increasing number of rules and regulations causes a lot of erosion.

In nature, a cape forms when everything else falls victim to erosion. Because our goal is to create a rock-solid foundation immune to erosion, we chose this term for both our architecture and our company.

From a data architecture perspective, the Data Cape’s strategic location between land and water is ideal to integrate with data lakes and provide a rock-solid foundation for data lake houses and data warehouses.

From a legal perspective, the Data Cape’s solidity stands for protection. Businesses don’t have to worry about fines and lawsuits, and users can be sure that there are no privacy violations. How well they are doing, can be measured using our Solidity Quadrant.

Governance, risk, compliance AND completeness

The Data Cape incorporates the functions of a traditional GRC program but has one very important additional component that makes it different: Its primary goal is to produce data of maximum completeness. It’s a unique in that it is a highly integrated solution.

From a data perspective, GRC programs are often seen as a nuisance across business units that need data to operate. That’s what makes the Data Cape a perfect addition, because it provides as much data as possible and treats non-compliance and data erosion as equally important problems.

Completeness: Preventing data erosion

Data erosion can happen at any stage of the data lifecycle:

Data creation

User-related data is usually created on the user’s devices, e.g. in a browser on a desktop computer, tablet, or phone, in an app on a mobile device, etc. However, user-facing systems create user-related data as well, e.g. a customer support system or a CRM. This is the stage at which most issues are introduced into data. It is also the stage where companies have to ask for a user’s consent to comply with privacy regulations.

Data transmission

When data is created on the device of a user, it needs to be transmitted to the company’s data pipelines, securely and reliably.

Data processing

Because the receiving endpoints are usually unprotected, a lot of unusable data can potentially be received. There can also be a desire to filter out bot traffic, for example.

Data storage

With an increasing number of storage technologies, storing even the largest amounts of data has become a relatively easy task. However, a lot of mistakes can be made when modeling the data.

Data usage and downstream processes

Data can usually not be fixed downstream, so you need to get it right at the creation stage. While this is logical and should be common sense, the desire to get things done and show results quickly often leads to non-compliant and/or unnecessarily incomplete data.

Preventing data erosion

Please note that this document is still a work in progress. This section will explain in detail how to prevent data erosion at the creation stage.

Completeness: Through granular compliance

When ensuring compliance, this is often done at a very high level. For example, when a user doesn’t consent to the collection of PII, many companies don’t collect any data. But why not at least gather non-PII data?

When looking at individual data points and not an entire data object as a whole, there are many data points that are not tied to an individual but contain information that still can be helpful to the business.

The main priority always has to be compliance. But almost equally important is maximizing the amount of legal data, which can only be achieved with a data-point-level approach.

Completeness: Synthetic data for legacy tools

Most tools working with user-related data were created long before recent privacy and data regulations. That’s why the entire ecosystem relies heavily on user-related data, even if user-level data would not be required.

The issues with “consented” data

In order to stay compliant with regulations, a lot of companies choose to work exclusively with data of users that have given their consent. This approach minimizes legal risks and ensures compatibility with consuming tools requiring user-level data.

However, any analysis, insight, recommendation derived from partial data has to be adjusted accordingly, creating additional work and subpar results. Somebody creating reports and dashboards may have to add or subtract 30% to the numbers to reflect what actually occured, for example.

How synthetic data can overcome these issues

In order to deliver on its promises of data compliance and completeness, the Data Cape must anonymize data that lacks user consent and replace user-related data points with synthetic data that mimics user-related data without being user-related.

This approach maximizes compatibility with the existing tool landscape. 3rd-party tools consume the actual user-related data and the synthetic user-related data. The result are analysis, insights, reports, dashboards, etc. of maximum quality.

Compliance: For different types of data

Another common mistake is that all data is treated equal. However, different types of data need different treatment to ensure compliance and completeness. In order to keep this document relatively concise, there are more specific documents for different types of data:

Compliance: Across a multitude of tools

Compliance is not easy and should therefore be managed centrally for different tools, not separately. In practice, however, centralized compliance becomes less likely as the size of a company increases, although the risk of fines increases significantly with company size.

The legal departments of these companies often do not know enough about data architectures to demand a centralized approach. In addition, different departments often have different risk tolerances.

Because regulators do not distinguish between departments, an organization’s overall legal exposure typically corresponds to the exposure of the least risk-averse department, with potentially significant consequences for other departments.

Compliance: Across different regulations

Creating data that is compliant with one specific regulation is challenging. It becomes even more challenging if more than one jurisdiction is involved, especially because a lot of laws are kept rather vague and broad.

However, there is also some good news: Privacy and data rules and regulations around the world greatly overlap because they lawmakers tend to prefer not to reinvent the wheel and cooperate internationally. This is another reason to solve compliance in a central place.

Risk: Fines and lawsuits

Whenever organizations do not gather or process user-related data in accordance with rules and regulations, they risk fines from regulators and lawsuits from individuals and activist groups.

Risk: Reputational damage

In addition to fines and lawsuits as discussed above (scroll up), reputational damage can be even more costly, although it is not easy to quantify. Once customers or investors have lost their trust in a brand or organization, it is often very hard to gain it back.

Conclusion

Please note that this document is still a work in progress. This paragraph will later give a conclusion.

Scroll back to top