4 minutes reading time (882 words)

Things I've learned from Data Integration

Often, when we look back on our career, we don’t think of ourselves as knowing nearly as much as we do. If we sit back and really reflect on the years we have taken to get where we are now, we are surprised to see just how much we have been through, how many mistakes we have recovered from and just how much we know now that we didn’t know way back then.

We know ourselves to be experts in our field, but how did we get there? How long did it take to accomplish what we have in our niche? 

Roughly twenty years back, I started my career as a trainee in an IT company with a rich engineering background supporting my first pharmaceutical client who wants to generate annual revenue reports. When I try to learn the process of generating reports from my customer,  I was astonished with the effort that they put to generate reports manually.  Finally it looks like they were tired of generating reports manually and wanted to handover to our company. My head was spinning and unable to tie the data. Believe me, it was tough one for a trainee even though i had a bunch of friends were working on this project. Now I realized why they didn't want to do it manually and seeking professional assistance.

If I'm not mistaken, it was around thirty different reports that they wanted to match and merge. There were no planning, no design process followed and no architectural flow of data.  So I was confused where to start :) . I was very quiet and thinking only me had the problem. Without any hesitance, I asked my team mates who she had five years of working experience already.  She didn't say anything but took me to my manager and told him that I didn't understand the project. At the time, I realized that I seeked assistance from a wrong girl because my manager was a ruthless and very impatient to explain the process to me but he gave a 200 pages requirement document to go through.  My adrenaline was sky rocketed when I went through the document because it was pretty lengthy and they were all talking about more business processes.  Parallely I was invited to attend several project discussions. I remember that most of the time, we spent on meetings and writing minutes of meetings.  Finally I decided that there is no soft way of learning both business process and design fundamentals of the data integration. So I decided (no other go) to spend extra hours and work on the weekends. Finally came up with a draft data flow diagram for my better understanding like below but it looks like a lifecycle of the hairball.  

Hairball Architecture

Even though the first diagram looks so clumsy but it gave me a fair understanding of the data flow, business process and how to design the Data Integration architecture.  Given the engineering background really helps to build such a flawless architecture. It was a team effort but we came up with a solid foundation for our future.  Once we build an Data Integration architecture, you can easily deliver the reports to the customer at any given time. In RCTECH we follow the below listed best practices (list of all items that we have learned through the years in our niche) for delivering an outstanding Data Integration solutions for our customers. They are

  1. Identify the list of desperate source systems that you need to derive the data integration reports.  Data could be from internal and/or external to the organization.
  2. Plot a decent data model that helps to understand the relationships between business process and business rules within the data.
  3. Total volume of transactions that the data integration could handle and how long would customer need to store these information in your new environment.
  4. Expected performance (turn around time or query time) while generating both scheduled or adhoc reports from the data integration environment?
  5. List out different formats that the data reside in the source systems.
  6. Design Architecture style that supports either Realtime or Batch.


Often reporting needs are defined for the summary data such as Annual revenue reports, Quarterly, Monthly etc., While summary data meets their business requirements, it is always recommended to bring the detailed data into your new data integration layer.  This avoids the problem due to data aggregation, mathematical calculation errors, etc., Most importanly summary data will not help to do a drill down analysis(I will discuss about drill down analysis) in my next blog.  Also it is important to engage your business users or business owners during the course of data integration development phase. It will help you to a greater extent to address any business related questions that may be missed during the requirements gathering phase.  Also it helps your business owners to gain trust about your process and data integration strategy.  I would say trust is an important factor here. If the trust is lost, it is extremely difficult to regain. In RCTECH,we consciously engage our customers through out the project life cycle and all of them are our HAPPY customers.  Good thing is they are all our repeatable customers!  Good Luck for your Data Integration Solutions!

For more information about Data Integration Click Here

What is the direct link between Quality of your co...

Related Posts



No comments made yet. Be the first to submit a comment
Already Registered? Login Here
Saturday, 08 August 2020
If you'd like to register, please fill in the username, password and name fields.

Captcha Image