Setting up a Data Governance Framework/Ecosystem on Public Cloud Stack e.g. GCP, Azure, AWS

Data Governance Framework with Azure Stack as Example

Overview

Most of the BI and analytics projects fail to generate substantial or recognizable value because of issues across acceptable data quality, needed scale and speed. Also most of the BI projects also find it difficult to have substantial organization wide engagement.
So a Data Governance frameworks should be able to mitigate all the above

Data Governance should not start at gates of BI function but should be a horizontal across all organization, It should be part of over arching Data Strategy

Some Key Concepts across Data Governance

1. What is data governance?
   Data governance is the practice of managing and organizing data and processes to enable collaboration and compliant access to data. Data governance allows users to create value from data assets even under constraints for security and privacy.
   Maybe you shudder when you hear data governance because you imagine people saying “no-”
   No, you can’t use the data for that
   No, we won’t support you
   No, you don’t have rights
   No, no, no.
   But data governance is much more than a nasty librarian with rules that thwart you at every turn. Modern data governance is freedom. It helps you grow the business while ensuring that internal and external policies are followed. Easily. Invisibly. Frictionless.
2. Data governance in detail
   Data governance is the best way to help your organization become data centric. Knowledge of the ins and outs is powerful and people who can drive adoption become valuable leaders in the organization.
   Data governance is an organizational discipline. It is supported by technology, but not a technology in itself. True data governance puts the rules in place and aligns the organization so data is not a potential liability. In an overly data-rich world where the winners and losers are based on how well they react and respond (especially in a downturn), data governance helps organizations easily understand those mounds of data. It fuels growth and digital transformation.
   Today, data governance technology is viewed as an important part of many organizations. Organizations leverage it as a business discipline in support of business outcomes, such as revenue growth or regulatory compliance.
   Governance is the foundational underpinning of Data Intelligence. Because the disciplines of data management are splintered and misunderstood, Data Intelligence provides the umbrella term to help organizations find, understand and fully leverage all those governed data assets.
   Data governance, data catalog, privacy solutions and more get simplified when an organization seeks to be intelligent about their data rather than implementing point solutions. Piecemeal parts such as data quality and master data management feed into a Data Intelligence strategy that provides strong business benefits.
3. Benefits of data governance
   Good data governance first and foremost ensures the massive amounts of data in an organization can be harnessed for business value. Ungoverned data is messy, rule-less and cramps productivity.
   Data governance means an organization can trust their data to answer important questions, like which products in which markets are likely to deliver the highest revenue. Bad data means bad answers, which lead to bad decisions, which can hurt an organization fiercely.
   As an example, take a company with data inconsistencies. Say, customer names are listed differently in the systems where it resides, such as customer service, sales, or logistics systems. The lack of data governance means these issues aren’t resolved and the intelligence about customers will either be limited or plain wrong. A scary thought.
   Another example is regulatory compliance. Privacy laws, such as GDPR from the European Union, or CCPA from California, must be adhered to or regulators levy costly fines. Without data governance, an organization has no mechanism for seeing where an individual’s information exists and therefore has no way to remove or edit it to meet these regulations.
   Data governance helps with both defensive (avoiding fines and low productivity) and offensive (growth) use cases. Every sophisticated organization knows they need data governance to achieve the highest data benefits.
4. Specific areas of benefit
   Management ensures that data assets are managed so they can be fully leveraged
   Sales grow when customer data is well managed and used to target opportunities
   Procurement achieves utmost cost reduction by using governed data to optimize purchasing processes and supply chain
   Compliance/Legal depends on data governance to meet requirements.
   Finance has well-governed data to report on business activities accurately
5. Goals of data governance
   -Increasing the organization’s data maturity so that all decisions are data driven
   -Removing information silos across the organization Knowing where data is and how it relates to each other
   -Implementing policies that meet compliance regulations
   -Ensuring that data is consistent and of high quality
   -Having confidence data is trustworthy Knowing data won’t be misused Establishing security measures
   -Ensuring all internal and external/3rd party data is safe Avoiding duplication and eliminating the time needed to massage data or create reports Bottom line?
   Governed data eliminates heartburn and drives the utmost use of this asset. If data doesn’t help the organization grow or save costs, it sits untapped. Data governance helps achieve results and avoid gut decisions.
6. Building a data governance framework
   A good data governance framework includes policies, procedures, processes, rules along with the right organizational structure and the technology to make it all happen. Believing a technology solution alone will magically result in data governance can be a fool’s errand. Any initiative is composed of people and processes, along with the supporting technology and data governance is no different. The framework must also involve the executives in the organization. Data governance has to be understood and appreciated in the boardroom or it will never survive at the troop level. Resources, in the form of both money and people, need to be allocated. Empowerment must happen. Leadership needs to lead and achieve alignment that ensures adoption. Two office workers collaborating The first step of building the framework is to articulate the objectives in a Mission Statement and define the KPIs that will be used to determine progress. Expressing this early ensures that your effort is aligned with the vision. Then the framework must include all aspects of how the organization will govern its data. What are the rules? The policies? The processes and procedures? The business glossary? The map of all data assets? This step will consume significant energy, data experience, and knowledge of the organization. Next, determine accountability. Who will be responsible for each part of the program, and who sits with overall responsibility and decision making? Lastly, once the data governance framework is laid out, the committee will turn to technology and choose a platform that best supports the vision. A good technology solution will gather metadata from a variety of systems, manage a business glossary, enforce policies and procedures, tie to a technical data dictionary, and more. The ease of an integrated platform ensures these capabilities come together easily.
7. Data governance principles
   When building a data governance program, the committee should consider this set of principles:
   Transparency
   Transparency involves easy understanding of where data resides, what data is available and how people use the data. Data transparency is crucial for issues of privacy and security, especially considering growing scrutiny from regulators (i.e. the state of California and CCPA; the EU and GDPR) and awareness among the public (i.e. Equifax’s 2017 breach). Accountability
   Organizations need to endorse a culture of accountability around its data. Policies should make it clear that employees are fully answerable for how they handle data. Some ways to entrench accountability into the organization are: Assigning roles and responsibilities across the data lifecycle (i.e. who is responsible for creating data) Restricting access to sensitive data, so only those who need it to perform their jobs can access it Administering processes and procedures for Documenting how people use data, where it resides, and how it has moved or changed over time Engagement
   Data governance only makes an impact if all users adopt data governance practices. It is essential to drive engagement from all stakeholders. In order to do this, the committee must understand data users’ needs and challenges and demonstrate why embedding data governance daily activities alleviates their problems. If users understand that data governance will actually make their lives easier, they will be more likely to support and adopt data governance. However, it is not enough just to tell them the benefits of data governance; the committee needs to implement an education and training program, so the users understand how to embed data governance into their daily activities.
8. Best Practices
   A data governance program can feel overwhelming, but these best practices make it something every organization can achieve:
   Focus on the operating model Identify data domains Identify critical data elements within the data domains Define control measurements
   Data governance is the lifeblood for any organization that depends on data to make important decisions. And what organization doesn’t? Businesses must have good, well-governed data to ensure they derive the most benefit from this precious asset while maintaining important policies.
   The best, healthiest organizations in today’s environment have effective data governance programs. We will see them survive and thrive while others falter. Make sure your organization is on the winning side There is no absolute HUB or SPOKE model of governance its always a Hybrid of both

The 11 building blocks of a tech enabled data governance framework

This article is becoming too big, this I will discuss in my next BlogPost