Mindful Data Governance Initiative

Mindful Data Governance Initiative

Before I go into what Mindful Data Governance Initiative is, I do what to say that I truly believe that every business out there can get real value from implementing Data Governance. If Data Governance is not on the roadmap for your business to implement I would seriously start thinking on how you could get it on there with key messaging to the executives. In my opinion Data Governance was nice to have back many many years ago but now it has become a must have, especially with the amount of data that is being collected by businesses today and the importance of analytics.

What is the Mindful Data Governance Initiative?
Why did I developed this methodology that I call Mindful Data Governance Initiative, I wanted a methodology in implementing Data Governance for my clients that would not hinder much of my client’s employees day to day activities. I wanted a methodology that would not break my client’s budget and allow my clients to see the importance and value that Data Governance can bring to their business. Businesses are paying money for custom software, SaaS, and data feeds that are bringing important transactional data into their business: Why not make sure that data is of the highest quality? Why not make sure the data is not being duplicated? Why not make sure the data is linked and being shared? Why not make sure that data is showing them the right picture on how their business is doing?

Let me just say this again implementing a Data Governance initiative does not have to take a large amount of dollars and does not have to be disruptive to employees day to day activities. This is why the methodology of Mindful Data Governance Initiative can impact businesses in a very powerful way, so businesses can start to reap the rewards of Data Governance and not have the worry about the impact of implementing a Data Governance solution could have on their business, their departments and/or their employees. The Mindful Data Governance Initiative is an approach that is done in phases with little impact to the business and starts to only address the nonstandard Data Governance items that the business is already conducting. We will go into what I define as a nonstandard data governance item very soon.

Why did I choose the word Mindful?
Merriam Websters definitions of Mindful are bearing in mind and inclined to be aware (conscious or aware of something). You can implement Data Governance within your business without creating a disruptive environment and impacting employees day to day activities by being mindful of the approach you take in Data Governance. Most business have some sort of nonstandard data governance that they are conducting and could implement standards around. Looking at implementing Data Governance this way (starting with the non-standards that you know exist) businesses can start their Mindful Data Governance Initiative easily and cost effectively. With this methodology, businesses can see the benefits of creating documentation and policies around those nonstandard tasks that have been identified and ensure the 4 principles are being applied to them availability, usability, integrity, and security.

Why did I choose the word initiative?
Lets look at the definition of initiative from Merriam Webster, 1. An introductory step. 2. Energy or aptitude displayed in initiation of action. Synonyms: action, drive, and ambition. I think when most businesses executives and sponsors think of Data Governance they do no think of it as an initiative, they think of it more as a directive that will disrupt and hinder their business’ day to day operations and could cost them a significant amount of money. They think of policies that are time consuming and costly to write and enforce in their business. That simple is not true with Mindful Data Governance Initiative. This approach can be very unobtrusive and budget sensitive especially because it can be done in phases.

Lets look more in-depth at the definition of initiative and why I choose that word to use in my methodology. The first definition of initiative is an introductory step. When implementing a Mindful Data Governance Initiative you look at everything as phases and do not worry about implementing the entire data governance all at once, as you do not have to implement everything to see the benefits and obtain a return on investment (ROI) of this initiative. When taking this introductory step (the first phase) the business and the executive sponsors will not only see a quick win but they will see why data governance is important. Before I tell you what I think should be the introductory step of your Mindful Data Governance Initiative let me explain that I think most business implement some type of nonstandard data governance in sections of their business’ and they just do not realize it. Here is an example of what I think is an nonstandard data governance practice: It is known by certain employees in the Customer Service department that when entering new products into their inventory systems that the name of the grouping of those products must match exactly the name inside their CRM system. Thats it and I believe most business will find a lot of these tasks undocumented within their organization, and the impact of not documenting these processes is that if these employees leave the business so does the knowledge. This should be your first step, identifying these areas and asking these questions: Where does the data come from? Where is the data being stored? Where is the data being used? How accurate is the data? Are these data points already being collected in other parts of the business? What rule(s) does the data have to follow? Who is collecting the data? These are just some of the questions and answers that should be documented, so business risk goes down and financial benefits increase.

The entire reason why I have developed the Mindful Data Governance Initiative approach is because I believe that Data Governance does not have to be a big impact on the business and can increase the value of a business’ data, decrease cost, increase the business’ revenue as a whole, ensures compliance and regulation needs are met, and promotes transparency through out the business. In my next blog I am going to talk about what are the different levels in the Mindful Data Governance Initiative and how you can find out what level does your business fall in. If you are interested in implementing the Mindful Data Governance inside your business, let me know I would love to help.

Data Governance 4 Your Business

What is Data Governance?
Data governance refers to the overall management of the availability, usability, integrity, and security of the data deployed in your business. A healthy data governance program includes a governing group, a defined set of procedures (that are repeatable), and a well-designed plan to execute those procedures (documented).

I always think of the number four when I think of Data Governance. Here is why: Some attributes of the number four are hard work, security, practicality, productivity, appreciation, tradition, solid foundations, security-consciousness, self-control, loyalty, conscientiousness, high morals and ethics. The essence of the number four is security, diligent work and strong foundations. All those attributes and the essence of number four is exactly how your data governance is to be implemented and treated. One of the most important attributes of a Data Governance is a solid strong foundation. Remember DATA GOVERNANCE 4 YOUR BUSINESS

Your Data Governance should consist of a four-way structure incorporating availability, usability, integrity, and security.

Why is Data Governance important?
Data Governance is important because it ensures that the data assets are formally, proactively, properly, and efficiently managed throughout the organization to secure its trust and accountability.

Data Governance comprises the collecting of data, revising and standardizing it, and making to ready for use. It makes the data consistent. Data Governance ensures that critical data is available at the right time to the right person, in a standardized and reliable form. This helps the business and its operations to be better organized. Adopting and implementing Data Governance can overall help improve the efficiency and productivity of an organization.

What are some of the methodologies of Data Governance?
Data is king and is so very important to every business no matter the size. You can implement Data Governance in phases but the implementation must always be across the entire business in order to be successful. Also it is very important to remember that once a phase has been implemented the governance body will have to continue monitor, maintain and review those implemented processes and the data, this is critical. Other success factors that can help you implement a winning Data Governance are: Look and prioritize areas of improvement (phase approach); Create roles, responsibilities, and rules from the processes people use in working with the data; establish an accountability infrastructure; convert your business culture to a master data management system. The way to start this highly structured and monitored Data Management strategy is to standardize the use of terminology across business units and enforce consistency of use. The ultimate goal of Data Governance is to make sure it is possible to consolidate your data and create a consistent view of that data across the business for advanced Business Intelligence activities. Literally you are turning data into a “Single Source of Truth” that the entire business is looking at.

Business Intelligence without Data Governance
Can you implement Business Intelligence without Data Governance? Of course you can. I believe that the two must go hand and hand. A sound Data Governance can significantly increase the returns of a company wide Business Intelligence investment. When starting a BI initiative with clients I seldom hear them talking about Data Governance but that does not take away the reality that it has to exist. Without governing your data in this data driven world business businesses will never realize the full potential of the data they are collecting. Data governance used to be a nice to have, but due to the increasing focus and importance of data and analytics, it’s becoming a necessity that helps to drive data management across the business.

For Example
Take a financial business I worked with that had very poor, inconsistent customer data. All of the customers with first, middle and last names had multiple differences, and addresses were inconsistent. This type of situation makes it very difficult to do any type of customer analytics, from identifying cross-sell opportunities to tracking and understanding customer experience. Data Governance can be a first step in identifying the issues, defining standards, and implementing changes in the business to align with these standards.


AWS Layers: How to include dependencies with your AWS Lambda Python Function

I wanted to write a quick how to blog in reference to show someone how to include dependencies with their python AWS Serverless function.

The Use Case

You have been assigned a project and you need to create a Lambda Serverless function in python that needs to execute a stored procedure or a SQL statement against a RDS Postgres Database. You need this function to be ran on a time schedule and it has to interact with the Postgres database.

What is AWS Layers

AWS has implemented some nice functionality called layers for us to easily include dependencies that our python script needs. Layers allow you to configure your Lambda function to pull in additional code and content in the form of layers. A layer is a ZIP archive that contains libraries, a custom runtime, or other dependencies. With layers, you can use libraries in your function without needing to include them in your deployment package. This allows your deployment package to be smaller.

Prerequisite: You have gone through the steps of writing and saving your Lambda function written in python you now want to add the psycopg2 library for your code to use.

Step one: Download the psycopg2 from https://github.com/jkehler/awslambda-psycopg2

Step two: Create a directory name python and put the psycopg2 folder inside the newly created python folder

Step three: Zip up the python directory that you created in step two

Step four: Save your function and then go to the Lambda home screen and click on “Layers”

Step five: Click the “Create Layer” button located at the top right part of the AWS console

Step six: Layer Configuration. Give your Layer a unique name. Enter a description of your layer. Click the “Upload” button and browse to your python zip file and select. Choose the Runtime version that you want. Click the “Create” button.

Step seven: Now that your layer is created time to setup in your function. Go back to the Lambda home screen and select functions. Choose the function that you created and has the dependency for psycopg2.

Step eight: Select the Layers box right under the name of your function

Step nine: Select the “Add Layer” button

Step ten: Select the Layer your created from the “layer” dropdown. Select the version from the “Version” dropdown. Click the “Add” Button.

You have now added a layer to your Lambda python function and you can go ahead and test it. Hope this helps.


Getting ready to start a BI Solution?

Before building out the plan for a bi solution project, I always like to think about what other projects that I have done or know about that are similar to the one I am going to embark on. I always ask myself what issues or obstacles did I encounter in those projects? There are so many reasons why projects get dragged out or just do not succeed, either from the start or they dwindle out if the project exceeds its expected delivery date. Lets just review some and see how many you came across in your professional career on why a project has failed.


  • No Executive sponsorship. There is no real buy in from the senior management team, which can lead to many different problems for the success of the project. You have to be persistent here and spend time presenting and socializing how important this project is to them and the company.


  • Battle between departments. Believe it or not I have run into this issue so many times when I have investigated why so many companies have failed in implementing a business intelligence solution. So what I mean here is that each department is acting like a ten year old child that does not want to share their data or their process because they believe it is their own secret sauce or they have something to hide.


  • No real project plan in place. This is were a company decides they would like to shoot from the hip on this project. They have no real kick off, no steps, no definition of what success is for the project, no anything. They fail to realize that this is a company wide project and the need for a project plan not only holds everyone accountable but ensures the right steps are completed before the next step are taken. It is this mentality that will kill any project almost from the start,


  • The IT department wants to build everything from scratch. This can be a problem in a lot of companies were the IT department wants to build everything custom. This can be not only very time extensive but also very costly. I always tell my clients to take the build, buy, or align approach when looking at the different areas of a project. There are several reasons when you should consider building a custom application over buying or aligning: Off the shelf products cannot meet every need, off the shelf products are to rigid, off the shelf products may not be compatible with your existing applications. Now let me give you several reason when you should be thinking of buying an off the shelf product: budget is limited, lack of time, lack of technical proficiency, and technology would not give you a competitive edge.


Above are just a few examples why your project could take longer than expected to be completed or just fail. I am sure that you can think of many more or even have experienced many more. I know in my long career as an IT professional I have come across reasons why projects have failed that would make people laugh and then some other reason that would make people cry. I will save those stories maybe for another blog. I think the main reason why I wanted to get you thinking about project failure is so that history doesn’t repeat itself and I am a firm believer that if you can learn from all failures, you will have a more successful future.

Building out a Business Intelligence Solution? Make sure you document your data collection process

The first step of building out a business intelligence solution is that you have identify all the data sources within the business. It is right after you identify the datasources is that you want to develop  and document all the data collection processes. Why develop a data collection process? Not only will creating a data collection process standardize the way you collect data for all the groups from within your business but it will ensure the integrity of your data is kept very high. Processes will ensure that your actions are repeatable, reproducible, accurate, and stable. Think about it if a business had an employee that was collecting critical data for the business and the business had no idea how it was being collected and that employee left, well I think that would have some impact on the business. The business most likely will be able to figure out how the employee was doing this but after how long? At what cost to the business? Could there be repercussions? I think you are understanding the point I am trying to make.  Ensuring processes are in place for your data collection will improve the likelihood that the data is secure, available, clean and the measurements derived from that data will help the business in the now and in the future.

There are many reasons why every business should be documenting the data collection process. If you are documenting, your processes become transparent and your data becomes comprehensible in the future for yourself and others. Your documentation of each of the data collection process should include:

  • Give a good description behind the data that is being collected.
  • Provide all the answers the who, what, why, where and how of the data.
  • Provide any conditions of the use of the data as well as the confidentiality.
  • Provide any history around the project for collecting this data
  • What are the measures and/or reports that will be built from this data

Even if you currently have a business intelligence solution in place but do not have all your data collection processes documented, go back and start getting them documented as soon as possible.