The power of integrating your systems

The first thing that I always hear from businesses is that they do not have integrated systems because of COST and the the effect of resource consumption. The biggest resource consumption that you can save on is your employees time. To be able to put an end to swivel chair technology in your business will not only save you time and money, but will ensure that one of your business most important asset (your business data) is kept valid and clean. Your business should be driving to get all your systems integrated as much as possible to leverage the existing processes, people, technology and information that you have. Security is another another important factor why system integration is important to businesses. If your business uses many systems in its day to day operations then integrating those systems should be at the top of your IT’s implementation goals.

What integrating your business systems can mean to you:

  • Better Management of Information: If your systems are not integrated it is impossible to get a complete picture of how your business is doing. When you have several systems that you are inputting data, some of those systems will be out of sync for longer periods of time especially during high traffic times of sales or as your business starts really to scale up. So what that means different managers in different departments that are using different systems are not seeing the same picture.
  • Higher Productivity: Dealing with many different standalone systems can consume time and with employees manually entering data into these systems (swivel chair tech) that is time be wasted when those employees can be redirected to higher priorities. Not only is this type of process time consuming but has the potential of creating bad data as employee can enter different information in each system. The scalability and expansion that system integration can give your business for growth can be achieved without adding more employees, so what does that mean. Well that means as your business increase so does your profitability.
  • Cost Savings: The return on investment (ROI) that you will see not having to manage the data input for multiple systems will be seen extremely fast, what you will also notice is that you data will be cleaner and all the different different departments will be seeing the same picture at the same time.
  • Greater Customer Satisfaction: Having multiple systems that are not integrated can mean that it can take extra time to fulfill your clients orders as well as respond to their enquiries or complaints. Nothing can increase customer satisfaction more like real time data flow between your systems.
  • Improved Security: Most businesses deal with some type of sensitive information that requires protection. By integrating your systems, you can easily build in the security tools necessary to prevent access by unauthorized users. Integrated systems will also help your business keep access to other systems to only the employees that need it, lowering the potential of issues and security threats.
  • If you are one of those businesses that is thinking can I really afford to integrate my systems, think about it this way can you really afford not to integrate. Integrating systems does not have to be done all at once, it can be a phased approach keeping cost a little more spread out while seeing the benefits and savings sooner.

    Your Single Source of Truth

    Purchase The Kindle Edition
    Purchase The Paperback Edition
    Your Single Source of Truth is a quick-read for busy business and IT professionals struggling to create a Business Intelligence solution. Packed with advice, proven methods, and real-world uses cases, this book provides the knowledge to get you not only started but to keep your Business Intelligence solution going.

    This book is intended to help you understand how a business can deal with their epidemic data problems and see a bigger clearer picture from the data they are collecting. There are mountains of data being collected in many different departments each with their own transactional system (silos). And each silo is not being joined to give a bigger and clearer picture to the business. This is a data centric world and businesses are collecting and saving data at an enormous rate but most are doing nothing with that data. They are not learning from the data and not making actionable and informed decisions from the data.

    Business Intelligence and silos of data is not just a small business issue — it’s an issue that all different size businesses are facing and are having problems getting their arms around. Whether it is lack of resources, low priority, or a lack of understanding that there is a problem. I believe if I can explain the issue, analyze it and point companies in the direction in solving their Business Intelligence issues then I would get to see many businesses grow and flourish. I want to help businesses answer those questions that I believe every business wants to answer: How is my business doing right now? How is my business doing compared to how it did in the past? Are all my areas of my business performing well? Which areas can have better efficiencies? What are my customers thinking and how can I better serve them? This is just a very small sample of questions that I know a business intelligence solution can help businesses answer and this book will help get you started.

    Seeing the picture

    Join Goggle and marketing data to get big picture of your prospects. The why and how

    The Why:

    When you look at each silo of data individually you do not get a complete picture of what a visitor or prospect is interested in, you only see what that one piece of the puzzle is showing you. You can get good information from your Google Analytics data (silo) when you look at it by itself and you can get good information from your marketing data (silo) when you look at that data by itself. What I am writing about in this blog is bringing those silos of data together to give you more insight into a potential customers journey, like when you put pieces of a puzzle together you see the picture clearer. Lets say that your business is utilizing Pardot as its marketing tool to not only manage email campaigns that are guiding potential customers to landing pages that are hosted on your web site but you are also gather customer information from forms that are being embedded within your business’s website. Wouldn’t it be great to see what content that prospect was looking at prior to and after they filled out a form on your website or where did they navigate to after getting to that landing page from one of your email campaigns? You have all the information in Pardot first name, last name, email address, etc and now you would be able to gather additional intelligence about them by what they are viewing within your business’s web site. You could even use that information to send out more direct campaigns based on the new knowledge that you have obtained from joining the data. For instance the prospect was on your website for 5 minutes and they visited the integration, business analytics, and warehouse pages. I know what I want to send to that prospect now.

     

    The How: 

    You have to join that data somewhere so lets bring that data into a warehouse. I used the CopperHill AIR Platform to run all the jobs that I created with Talend to move the data from Google and Pardot into the AIR Warehouse. Lets start with the steps I took with Google first. I opened Google Query Explorer which is a tool that allows you to play with the Core Reporting API by building queries to get data from your Google Analytics views. This tool allows you to set different metrics, dimensions, sorts, filters and segments so that you can see the end results before applying them into your data integration tool. Once I got the queries working just as I wanted them to and the data looked good, I move those exact settings inside my Talend tool. With Talend I had to install the Google Analytics Component as it is not part of the base install. The component is free and very easy to install. Once I setup the component and deployed it to the AIR Platform I was immediately streaming data. Below is just an example of one call to Google’s Core Reporting API within the job. I repeated the same steps for each call I wanted to make to Google Analytics API and all the data transfers were handled inside that one job. Channel Data

    The next step was to get Pardot data flowing. Pardot does have a great API that you can access and read your marketing data from. Again I used the AIR Platform to run the job that I created in Talend to move this data out of Pardot and into the AIR warehouse. Let me say this, you do have to have knowledge of the Pardot API or creating this job can be a little tricky. What I did was setup a REST call to each of the Pardot objects that I wanted to receive data from and store within the AIR warehouse. I have attached a portion of the job I configured to retrieve data from the Pardot campaign object. As you can see the setup and configuration is a little more complex then the call to the Google API but the data is flowing just as fast.

    Now that all the data from the two sources Google and Pardot are streaming into the AIR warehouse we can create the reports showing exactly what we stated in The Why section. This thought process and build out can be applied to really any marketing tool that a business is using and join that to your Google Analytics data. If you want to gather great actionable information around your prospects then joining google and marketing data is definitely a great idea for you and your business. Finish the data puzzle so you can see the big picture and just do not look at the pieces by themselves.

    Middleware versus custom code

    I have been asked many times in my career about my thoughts around using middleware versus writing custom code to integrate systems that need to communicate with each other. My thoughts have changed over the years as my experience has grown significantly. There are  a lot of challenges that need to be thought about and considered when you are making a decision between middleware or writing custom code and what is best solution for your situation.

    With whatever solution you go with, the solution will have to govern or handle the following areas.

    Internal Applications or SaaS that have changing API’s: A lot of the applications that you might be using have APIs available to allow other applications to integrate with them. These APIs can change with new releases of the application as a company enhances their software to accommodate new functionality and features. If you decide to write custom code to handle the integration you will need to get skilled programmers who will then have to get a firm understanding on how the API works (transaction rules, capabilities, schema, authentication, etc) for each application, after each new revision or deployment. If you are using middleware the configuration of the integration process can be changed easily and quickly when APIs change.

    Handling business process changes: As a company grows business process will most likely change. These changes can include new or different business rules, data transformations, and business logic. With constant changes to the business and if you choose to write custom code, that code can grow to become unmanageable. With the use of middleware all changes are done through a graphical user interface which includes connections, transformations, business rules, flow logic, and data mappings. Most if not all these changes are done without writing any code.

    Monitoring: More complex integrations require the integration between the systems to be monitored. If you have choose to do the system integration with custom code the developers will need to write all the monitoring and error handling code which can take a significant time to write possibly pushing your project longer then expected. With most middleware solutions monitoring, error handling, and logging is implemented with some point and clicks and an update to the configuration file.

    Optimizing Processes: Most API’s have stipulations within them to help manage the integrity and performance of the application. These stipulations can add to the complexity of the integrations by restricting the amount that can be sent or received in each call, required authentication, how many times you can make the call to just name a few. With just some of the stipulations that I have name, the complexity of the custom code goes way up and a significant amount of code must be written and tested to handle them. Middleware usually contains functionality (tested functionality) to handle these stipulations and makes sure all the processes are optimized and the data is delivered faster.

    My Conclusion: I have built many middleware solutions in my career and the codebase at times was very large depending on the complexity of the integrations. Within each project the business owner always thought that integration should be easy by saying “Hey you are just moving data from one point to another point”. I can tell you that is usually not the case at all. Based on my experience using a middleware solution is usually the most cost effective solution to go with in delivering a speedy ROI back to the business. Most if not all the functionality is already been developed and tested thoroughly so there is no need to reinvent functionality that has already existed.