The aphorism “the only constant is change” is never truer than when applied to the Enterprise and technology. Customer needs evolve constantly; technology correspondingly evolves. This interaction applies to data integration. Data is an essential element and component of EPM systems for everything from building successful business plans to launching and monitoring an effective customer engagement campaign.
The Progressive Nature of Integration
Before getting into the technical elements of the evolution of data integration and why and how they have proven effective in the process of data management, it is necessary to briefly explain the need to evolve in the area of data integration. Over the past 20 years, it has become evident that data plays an immense role in a business’s drive to become and remain competitive in its market. Long before the term "big data" emerged, data was essential. The need to compile large amounts of data from multiple sources, the need to extract and categorize data became more relevant, and the need to analyze that data requires data extraction, transformation, and loading (ETL).
Introducing Hyperion Application Link (HAL)
From 2000 to 2006, the ETL package offered by Hyperion was Hyperion Application Link (HAL) ETL tool. HAL was a rebranded Vignette product and was included in all of the Hyperion products. While this product allowed the extraction and categorical organization of data from a number of platforms, there was a need to expand the range of the tool while also improving its functionality. The tool provided no templates, thus requiring each implementation to be completely customized. . Additionally, the graphical nature of the tool was such that maintenance was an issue. Performance was also poor.
Let’s Give Data Integration Management (DIM) a Quick Wave
Just before Oracle acquired Hyperion, along came DIM, an OEM data integration tool based on Informatica’s PowerCenter. DIM required a very high level of technical expertise to stand up and maintain a solution. DIM’s life in the EPM stack was short lived because shortly after the acquisition, Oracle replaced DIM with Oracle Data Integrator (ODI).
Another Quick Wave to FDM, (yes, it is still around, but I would like to get to the good stuff)
Coincident with the DIM product launch, Hyperion acquired Upstream Software and their data integration product, Weblink. Weblink offered exactly the type of end user tool that HFM customers wanted. Weblink was rebranded as Financial Data Quality Management (FDM).
Financial Data Quality Management is a web based financial data integration platform focused on both the end-user and Finance. FDM has a friendly user interface, with plenty of out of the box features including Sarbanes-Oxley required audit/ process controls and visibility into all the aspects of data integration. However, it is not as powerful as a true ETL tool when it comes to large volumes of records.
Goodbye HAL, Hello ODI
ODI, formerly known as Sunopsis, was acquired by Oracle in 2006. One of the challenges associated with this evolution in the data integration message to Oracle EPM customers is that ODI and HAL are two completely different platforms performing similar functions. HAL is an EAI tool and ODI is an ETL/ELT platform. These differences meant there could be no direct migration path between the two. Primarily, the challenge with the lack of a migration path from HAL to ODI was engaged by using the channel of similar features offered through the ODI tool.
ODI is distinct from HAL, DIM and, FDM in a number of different ways, including the type of target and source technologies that integrate effectively with ODI and the presence of automatable features. In Chapter 2 of Developing Essbase Applications, Advanced Techniques for Finance and IT Professionals, Cameron Lackpour, Veteran Developer and ODTUG Secretary, dedicates a portion of his chapter on data quality to his preferred data integration platform -- ODI. I can hear Cameron bubbling over with geeky enthusiasm when he writes, “ODI isn’t just the choice by a process of elimination, it has compelling features” I can imagine that he would add “many” or “a lot” with an emphasis on either.
With Cameron’s permission, I have included an image from the book, which contains a list of ODI’s most compelling features according to CL.
Yep, he likes it.
Oracle's Impact on the Evolution of Data Integration
Over the years, the manner in which data is used has evolved in use and frequency and Oracle has worked to remain ahead of the data integration curve
When Oracle introduced ODI, they also introduced the cloud-based computing mechanism that would ultimately revolutionize how data integration solutions would be administered and managed by the service provider and the client. Not only did the introduction of web-based features improve flexibility and functionality, it also reduced the need to replace hardware as changes and upgrades were made providing a more cost-efficient integration solution.
Oracle offered another boost to data integration through Data Relationship Management (DRM) and its ability to integrate directly with EPMA 11.1.2.2 as well as the other EPM tools via files and tables. DRM also integrates with Data Relationship Governance (DRG) to improve data integration workflow. In 2013, the company successfully combined the robust data extraction and data movement engine of the ODI tool with the FDM classic extensibility and usability now known as FDMEE.
Finally, the next era has already been launched with the introduction of quantitative management and reporting, and the escape from the Microsoft dependence to expand the flexibility of the program even further. The emergence of big data has solidified the need and the demand for immensely powerful and highly effective data integration tools, and therefore, the evolution continues.
ETL and ELT, ODIEE, Big Data OH MY!
In the last several months I’ve become a Big Data addict be it through blogs, magazine articles, or Twitter. I am fascinated by Big Data’s evolution from a curiosity in Computer Science departments to big business. A little over a month ago, as I was waiting in line at the market, I overhear a woman saying to her friend or relative “Oh wow, how cool is this? I was just browsing for these RayBans and there is an ad on facebook showing them at a discounted price.” She mentioned the website but at this point, I was just smiling and thinking to myself, that is so not a coincidence but instead the impact of Big Data.
Big data and data mining allows Google and other search engines to collect data based on your daily activity and then and the search engine acts a source of information for our mobile applications such as Facebook, the data however flows both ways. This form of data mining has been around since the 70’s however, it began on computers and personal laptops and started to penetrate the mobile market about a decade ago. When regulators such as the FCC mandated that cell phones are equipped with GPS for emergency services, this was revolutionary for big data. Out comes applications allowing you to enable and share your location, allowing Companies to market to you based on location. The list goes on! Just the other day I received a message from I-maps letting me know that Orange Theory had a class starting in 60 minutes 20 minutes away from my Hotel in Livonia, MI. I’ve never attended an OTF class in Michigan but I am a member and I do book my classes online. Thank you Big Data! Big Data is much more than the above but it gives you a flavor of what it can do. Oracle plays in this space as well.
Enough of that, WELCOME ODIEE
Oracle’s data integration portfolio continues to expand and includes Oracle Data Integrator, GoldenGate, Data Service Integrator, Metadata Management, and others. Just recently, Oracle added Oracle Big Data Preparation Cloud Service.
“Oracle Data Integrator Enterprise Edition Big Data Option offers customers enterprise scale big data Integration. With the advanced option, Oracle Data Integrator extends big data heterogeneity to include multiple big data standards. Through its decoupling of logical design and physical implementation, Oracle Data Integrator lets customers choose between multiple underlying big data platforms that best suits the customer requirement. Customers can now future proof their Hadoop investment and increase Hadoop platform inter-operability. Oracle Data Integrator decreases big data projects’ time to value by offering out of the box code templates which increases developer productivity, streamlines the development process and improves performance." Oracle Data Sheet for Oracle Data Integrator Enterprise Edition Big Data Option
ODIEE offers one complete platform with a familiar interface that our ETL and data Developers are already familiar with. ODIEE as Denis Gray puts it in this Oracle’s webcast series “Introducing Oracle Data Integrator for Big Data” is a “game changer for Hadoop and big data integration!” Why you ask? ODIEE for big data unlike other big data platforms does not work side by side with Hadoop but natively within Hadoop, offering the best in class ELT architecture.
ODIEE for Big Data allows data ingestion via custom scripting, and can bring in data from any source at any latency. ODIEE also allows the use of outside sources to bring in data such as streaming with Kafka or using Flume for log files. Ingested data can include structured or unstructured data as well as log files, all done in real time, streaming, or batch. In turn, ODIEE allows data transformation using custom scripting in Apache Hive, Pig, or Spark. The names are cool (or geeky, but I think they are cool) and so is the ability to deploy the data in either Hive, Spark, or Pig in different environments. Did I mention that you can also write custom scripting in the same interface to move data outside of ODIEE and into your business intelligence platforms or other platforms for further analysis?
Can you tell I like ODIEE? There are many other cool features within ODIEE for Big Data. To read more and find out Oracle ODIEE and related products compare to the competition make sure you check out the Oracle blogs at blogs.oracle.com/dataintegration and webcasts at www.oracle.com/technetwork/middleware/data-integrator/overview/index.html.
The evolution of ETL from HAL to ODIEE has driven immense changes in purpose, functionality, and development. Oracle is playing to win in this space and we’ll see more to come. I can hardly wait.
