Bank on ODI for high volume data transactions
Integration Technologies are quickly and vastly moving into an age where there is much more user interaction with data of different flavors by using different interfaces. Even if we find an application which fits our needs, we still crave for bringing data together and making sense of it becomes increasingly difficult. Most of the crucial applications database valor run on Oracle Database, other databases and platforms supplied by different vendors. Applications themselves may intercommunicate by using technologies such as SOA, Web services, and applications oriented communication and data may be hosted remotely as well as managed by in central data repository.
Working on a requirement for loading High volumes in our current assignment, we were astonished by the features of ODI and its support in various forms using ETL arch to FMW. We would like to throw some light on its capabilities and motivation for its basic functionalities.
Oracle Data Integrator addresses data integration needs in these increasingly heterogeneous environments. It is a Java-based application that uses the database to perform set-based data integration tasks also extends this ability to extract from and provide transformed data through Web services and messages and to create integration processes that respond to and create events in your service-oriented architecture.
Oracle Data Integrator addresses data integration needs in these increasingly heterogeneous environments. It is a Java-based application that uses the database to perform set-based data integration tasks also extends this ability to extract from and provide transformed data through Web services and messages and to create integration processes that respond to and create events in your service-oriented architecture.
High Level Functional Data Services –flexible data services it supports:
Batch Data Services – these are data services that provide bulk data movement and transformation services. Typically, a batch data service would expose a Web Service API for SOA-based applications to invoke these bulk data/ETL style jobs from the SOA layer. Several known implementations incorporate these batch data services as sub-processes to a transactional BPEL or ESB process – so that the point of control for the ETL jobs is at the SOA layer, but the delegation of efficient bulk data handling occurs at the most appropriate architecture tier.
* Goad: ERP, Data Warehouses, Business Intelligence and Performance Management Applications require bulk data movement.
Data Access Services – these are data services that provide direct access, through a managed (synthetic or physical) view, to the resident location of the data. Data access services may be as simple as a Web Service for fetching data from database. Data access services may also be as complex as issuing queries to synthetic data views and having the service federate data source queries in real-time with aggregated data result sets.
* Goad: Present a simplified query interface to consuming applications. Usually by combining a shared abstraction (Canonical Model) with instance virtualization (Data Mash up).
Data Quality Services – these data services use algorithms and pre-defined business rules to clean up, reformat, and de-duplicate messy business data. Typically these services are used inline with other data services (for example: using a data quality service inline with bulk data/ETL services) or statically on a data source (for example: cleaning up a legacy database). But more recent applications show that hosting a data quality service within a SOA can provide much needed cleansing and standardization services to SOA messages and data.
* Goad: Automatically improve the quality of bad data so that legacy data resources become more valuable and usable.
Data Transformation Services – these are the classic data services, simply waiting to take one format in, and provide another format out. In a SOA-only world, these would have been deployed as XSLT libraries, where a consuming application service would send in some data, choose a corresponding XSLT, and receive the data in a new format. In a more mature SOA, transformation services may also include ETL like services that specialize in efficient transformation of bulk data (10-100’s of MB) payloads.
* Goad: Present a reusable service for WSDL-driven data transformation – generally supporting multiple types of transformation (such as: RDB-to-RDB, XML-to-RDB, XML-to-XML, Flat-to-XML, Flat-to-RDB…)
Data Event Services – these are data services that monitor, correlate, and propagate events that happen on business data. Data events may occur at the middleware messaging, data integration, and database tiers of the infrastructure. In a mature SOA implementation, data events can be subscribed to regardless of whether the events are occurring in the database, middleware or elsewhere.
* Goad: Every part of the data environment must be capable of trapping actions, checking policies and taking action based on those policies
A big part of the Data Service challenge is to provide a controlled, but flexible infrastructure that will allow different organizations to build, modify and publish their own services within a shared framework. Instead of arbitrarily assuming that every piece of data must be converted to XML at some point – that assumption could quadruple the size of payloads and decimate performance levels – instead, be willing to work on the data in its source formats.
In addition to these ODI provides Changed Data Capture (CDC) capability that identifies and captures data that has been inserted, updated, or deleted from a source, and it makes this data available for integration processes. Changed Data Capture uses publish-and subscribe model. An integration Scenario can subscribe to the changes that happen on a source. ODI provides two methods for tracking changes from source datastores to the CDC framework: database triggers and RDBMS log mining. The CDC framework is generic and open, so the change-tracking method can be customized. CDC also includes the capability to manage a “consistent set” of source data.
What we have mentioned above may not give the complete picture, but ODI can be used in most of the data services. It sounds trite, but the simple advice for people who are interested/deals in Data Services is to always use the right tool for the job.
Author(s):
Vamsi S.K.N,Prasanthi Sailaja Nalla
AIA Center of Excellence,Infosys.
Please feel free to send in your comments..
Batch Data Services – these are data services that provide bulk data movement and transformation services. Typically, a batch data service would expose a Web Service API for SOA-based applications to invoke these bulk data/ETL style jobs from the SOA layer. Several known implementations incorporate these batch data services as sub-processes to a transactional BPEL or ESB process – so that the point of control for the ETL jobs is at the SOA layer, but the delegation of efficient bulk data handling occurs at the most appropriate architecture tier.
* Goad: ERP, Data Warehouses, Business Intelligence and Performance Management Applications require bulk data movement.
Data Access Services – these are data services that provide direct access, through a managed (synthetic or physical) view, to the resident location of the data. Data access services may be as simple as a Web Service for fetching data from database. Data access services may also be as complex as issuing queries to synthetic data views and having the service federate data source queries in real-time with aggregated data result sets.
* Goad: Present a simplified query interface to consuming applications. Usually by combining a shared abstraction (Canonical Model) with instance virtualization (Data Mash up).
Data Quality Services – these data services use algorithms and pre-defined business rules to clean up, reformat, and de-duplicate messy business data. Typically these services are used inline with other data services (for example: using a data quality service inline with bulk data/ETL services) or statically on a data source (for example: cleaning up a legacy database). But more recent applications show that hosting a data quality service within a SOA can provide much needed cleansing and standardization services to SOA messages and data.
* Goad: Automatically improve the quality of bad data so that legacy data resources become more valuable and usable.
Data Transformation Services – these are the classic data services, simply waiting to take one format in, and provide another format out. In a SOA-only world, these would have been deployed as XSLT libraries, where a consuming application service would send in some data, choose a corresponding XSLT, and receive the data in a new format. In a more mature SOA, transformation services may also include ETL like services that specialize in efficient transformation of bulk data (10-100’s of MB) payloads.
* Goad: Present a reusable service for WSDL-driven data transformation – generally supporting multiple types of transformation (such as: RDB-to-RDB, XML-to-RDB, XML-to-XML, Flat-to-XML, Flat-to-RDB…)
Data Event Services – these are data services that monitor, correlate, and propagate events that happen on business data. Data events may occur at the middleware messaging, data integration, and database tiers of the infrastructure. In a mature SOA implementation, data events can be subscribed to regardless of whether the events are occurring in the database, middleware or elsewhere.
* Goad: Every part of the data environment must be capable of trapping actions, checking policies and taking action based on those policies
A big part of the Data Service challenge is to provide a controlled, but flexible infrastructure that will allow different organizations to build, modify and publish their own services within a shared framework. Instead of arbitrarily assuming that every piece of data must be converted to XML at some point – that assumption could quadruple the size of payloads and decimate performance levels – instead, be willing to work on the data in its source formats.
In addition to these ODI provides Changed Data Capture (CDC) capability that identifies and captures data that has been inserted, updated, or deleted from a source, and it makes this data available for integration processes. Changed Data Capture uses publish-and subscribe model. An integration Scenario can subscribe to the changes that happen on a source. ODI provides two methods for tracking changes from source datastores to the CDC framework: database triggers and RDBMS log mining. The CDC framework is generic and open, so the change-tracking method can be customized. CDC also includes the capability to manage a “consistent set” of source data.
What we have mentioned above may not give the complete picture, but ODI can be used in most of the data services. It sounds trite, but the simple advice for people who are interested/deals in Data Services is to always use the right tool for the job.
Author(s):
Vamsi S.K.N,Prasanthi Sailaja Nalla
AIA Center of Excellence,Infosys.
Please feel free to send in your comments..
