Your Single Source of Truth

Purchase The Kindle Edition
Purchase The Paperback Edition
Your Single Source of Truth is a quick-read for busy business and IT professionals struggling to create a Business Intelligence solution. Packed with advice, proven methods, and real-world uses cases, this book provides the knowledge to get you not only started but to keep your Business Intelligence solution going.

This book is intended to help you understand how a business can deal with their epidemic data problems and see a bigger clearer picture from the data they are collecting. There are mountains of data being collected in many different departments each with their own transactional system (silos). And each silo is not being joined to give a bigger and clearer picture to the business. This is a data centric world and businesses are collecting and saving data at an enormous rate but most are doing nothing with that data. They are not learning from the data and not making actionable and informed decisions from the data.

Business Intelligence and silos of data is not just a small business issue — it’s an issue that all different size businesses are facing and are having problems getting their arms around. Whether it is lack of resources, low priority, or a lack of understanding that there is a problem. I believe if I can explain the issue, analyze it and point companies in the direction in solving their Business Intelligence issues then I would get to see many businesses grow and flourish. I want to help businesses answer those questions that I believe every business wants to answer: How is my business doing right now? How is my business doing compared to how it did in the past? Are all my areas of my business performing well? Which areas can have better efficiencies? What are my customers thinking and how can I better serve them? This is just a very small sample of questions that I know a business intelligence solution can help businesses answer and this book will help get you started.

Seeing the picture

Join Goggle and marketing data to get big picture of your prospects. The why and how

The Why:

When you look at each silo of data individually you do not get a complete picture of what a visitor or prospect is interested in, you only see what that one piece of the puzzle is showing you. You can get good information from your Google Analytics data (silo) when you look at it by itself and you can get good information from your marketing data (silo) when you look at that data by itself. What I am writing about in this blog is bringing those silos of data together to give you more insight into a potential customers journey, like when you put pieces of a puzzle together you see the picture clearer. Lets say that your business is utilizing Pardot as its marketing tool to not only manage email campaigns that are guiding potential customers to landing pages that are hosted on your web site but you are also gather customer information from forms that are being embedded within your business’s website. Wouldn’t it be great to see what content that prospect was looking at prior to and after they filled out a form on your website or where did they navigate to after getting to that landing page from one of your email campaigns? You have all the information in Pardot first name, last name, email address, etc and now you would be able to gather additional intelligence about them by what they are viewing within your business’s web site. You could even use that information to send out more direct campaigns based on the new knowledge that you have obtained from joining the data. For instance the prospect was on your website for 5 minutes and they visited the integration, business analytics, and warehouse pages. I know what I want to send to that prospect now.

 

The How: 

You have to join that data somewhere so lets bring that data into a warehouse. I used the CopperHill AIR Platform to run all the jobs that I created with Talend to move the data from Google and Pardot into the AIR Warehouse. Lets start with the steps I took with Google first. I opened Google Query Explorer which is a tool that allows you to play with the Core Reporting API by building queries to get data from your Google Analytics views. This tool allows you to set different metrics, dimensions, sorts, filters and segments so that you can see the end results before applying them into your data integration tool. Once I got the queries working just as I wanted them to and the data looked good, I move those exact settings inside my Talend tool. With Talend I had to install the Google Analytics Component as it is not part of the base install. The component is free and very easy to install. Once I setup the component and deployed it to the AIR Platform I was immediately streaming data. Below is just an example of one call to Google’s Core Reporting API within the job. I repeated the same steps for each call I wanted to make to Google Analytics API and all the data transfers were handled inside that one job. Channel Data

The next step was to get Pardot data flowing. Pardot does have a great API that you can access and read your marketing data from. Again I used the AIR Platform to run the job that I created in Talend to move this data out of Pardot and into the AIR warehouse. Let me say this, you do have to have knowledge of the Pardot API or creating this job can be a little tricky. What I did was setup a REST call to each of the Pardot objects that I wanted to receive data from and store within the AIR warehouse. I have attached a portion of the job I configured to retrieve data from the Pardot campaign object. As you can see the setup and configuration is a little more complex then the call to the Google API but the data is flowing just as fast.

Now that all the data from the two sources Google and Pardot are streaming into the AIR warehouse we can create the reports showing exactly what we stated in The Why section. This thought process and build out can be applied to really any marketing tool that a business is using and join that to your Google Analytics data. If you want to gather great actionable information around your prospects then joining google and marketing data is definitely a great idea for you and your business. Finish the data puzzle so you can see the big picture and just do not look at the pieces by themselves.

Middleware versus custom code

I have been asked many times in my career about my thoughts around using middleware versus writing custom code to integrate systems that need to communicate with each other. My thoughts have changed over the years as my experience has grown significantly. There are  a lot of challenges that need to be thought about and considered when you are making a decision between middleware or writing custom code and what is best solution for your situation.

With whatever solution you go with, the solution will have to govern or handle the following areas.

Internal Applications or SaaS that have changing API’s: A lot of the applications that you might be using have APIs available to allow other applications to integrate with them. These APIs can change with new releases of the application as a company enhances their software to accommodate new functionality and features. If you decide to write custom code to handle the integration you will need to get skilled programmers who will then have to get a firm understanding on how the API works (transaction rules, capabilities, schema, authentication, etc) for each application, after each new revision or deployment. If you are using middleware the configuration of the integration process can be changed easily and quickly when APIs change.

Handling business process changes: As a company grows business process will most likely change. These changes can include new or different business rules, data transformations, and business logic. With constant changes to the business and if you choose to write custom code, that code can grow to become unmanageable. With the use of middleware all changes are done through a graphical user interface which includes connections, transformations, business rules, flow logic, and data mappings. Most if not all these changes are done without writing any code.

Monitoring: More complex integrations require the integration between the systems to be monitored. If you have choose to do the system integration with custom code the developers will need to write all the monitoring and error handling code which can take a significant time to write possibly pushing your project longer then expected. With most middleware solutions monitoring, error handling, and logging is implemented with some point and clicks and an update to the configuration file.

Optimizing Processes: Most API’s have stipulations within them to help manage the integrity and performance of the application. These stipulations can add to the complexity of the integrations by restricting the amount that can be sent or received in each call, required authentication, how many times you can make the call to just name a few. With just some of the stipulations that I have name, the complexity of the custom code goes way up and a significant amount of code must be written and tested to handle them. Middleware usually contains functionality (tested functionality) to handle these stipulations and makes sure all the processes are optimized and the data is delivered faster.

My Conclusion: I have built many middleware solutions in my career and the codebase at times was very large depending on the complexity of the integrations. Within each project the business owner always thought that integration should be easy by saying “Hey you are just moving data from one point to another point”. I can tell you that is usually not the case at all. Based on my experience using a middleware solution is usually the most cost effective solution to go with in delivering a speedy ROI back to the business. Most if not all the functionality is already been developed and tested thoroughly so there is no need to reinvent functionality that has already existed.