This blog is for Mainframe Modernization professionals to discuss and share perspectives, point of views and best practices around key trending topics. Discover the ART and science of mainframe modernization.

« Better Managing Your Modernization Process | Main | Application Portfolio Analysis - The first step to Modernization »

Migration of Mainframe Batch Workloads to ETL Platforms

"It's only after you've stepped outside your comfort zone that you begin to change, grow, and transform.
                                                                                                                                          ~ Roy T. Bennett

During the late nineties, technology pundits predicted the imminent downfall of the mainframes. These ginormous computing machines from an earlier age seemed to have outlived their usefulness. Now, two decades later, the Big Iron continues to rule the roost running inside some of the world's biggest companies including banks, retailers, insurance companies and airlines. 92 of the top 100 banks worldwide, 70% of the world's largest retailers, 10 out of 10 of the world's largest insurers and 23 of the world's 25 largest airlines run on mainframes. To paraphrase Mark Twain, news of the mainframe's demise was greatly exaggerated.

The Second Law of Thermodynamics states that the total entropy of an isolated system always increases over time. An IT system is no different. Most of the mainframes systems have been around for several decades and one of the most taxing challenges these applications face today is the muddle which has developed over years of operation. Bit by bit the complexity of the systems increased to such an extent that maintainability has become a huge challenge. Consequently, mainframes have become expensive to maintain and run.

IT organizations have a plethora of options where mainframe modernization is concerned. Every mainframe landscape is unique and the ability to create a foolproof modernization roadmap lies in the ability to understand the context of the mainframe landscape. This includes consideration of the application complexity, current spend, integration points, technology landscape, variety of processes as well as its adaptability to the proposed target framework. Owing to the similarity in their concepts, processing and terminologies, the ETL platform has evolved as a good alternative for running existing batch workloads migrated from a mainframe environment.

Migration to ETL involves extraction of business rules from the existing application and using them to create specifications or code for the new system. This is an excellent way of taking advantage of existing business logic within the mainframe system while introducing modern technologies and IT concepts. The rewritten IT systems can quickly adapt to changing markets, shifting customer needs and new business opportunities. But at the same time, the migration process can be time consuming and adds project risk. Following few strategies can help ensure a smoother transition:
  • Phased migration approach with clear fallback strategy is vital to ensure there is no adverse impacts to consumers of the IT system as part of this migration.
  • Outbound extracts can be the first to be targeted for migration. While data continues to be consumed from the legacy platform, the files generated on the ETL platform can be vetted against the files on the legacy platform using a generic comparator framework and flows cut-over once all certification activities are complete.
  • Inbound feeds are generally tricky as it involves maintenance of the database tables and any discrepancies can result in data corruption. For ensuring the processed data is unchanged in the new platform, the load ready files can be compared over a period of time before the flows are actually cut-over to the new platform.
  • Internal processes like housekeeping, purge, etc. can be the next to be rewritten on the ETL platform.
  • In many cases, the database continues to remain on mainframe. When feasible, moving the database to a distributed database can be considered as the next logical step.

Batch to ETL.png

Some of the key challenges faced during typical ETL migration projects are listed below:
  • Mainframe application complexity due to the extensions and additions done to the underlying architecture over a span of several decades. Mergers and acquisitions further add to the confusion with disparate systems, standards and practices.
  • Meeting existing SLAs is a key parameter for any successful ETL migration. While mainframe load utilities working at partition level can easily load several hundred million records on a daily basis, it may be a challenge for ETL processes.
  • Loss of subject matter expertise in the application is another common concern as mainframe experts move to other areas in the enterprise with mainframe stack being replaced by the ETL suite.

In a mission critical software life cycle, complexity is a given and quality is a requirement. ETL combines data preparation, data integration and data transformation into a single integrated platform to transform how IT and business can turn data into insight. ETL processes inherently lend themselves to automation and this brings a swarm of benefits to these processes including end to end data lineage, faster delivery times, improved productivity, reduced cost, decreased risk of error, and higher levels of data quality among others. With improved integration of ETL and Big Data technologies, it's now possible to augment data with Hadoop-based analytics. Features like data connectivity, transformation, cleansing, enhancement and data delivery can be run within a YARN-managed Hadoop platform. 

Infosys has a proven track record in collaborating with clients to migrate their mainframe workloads to ETL platforms. For example, Infosys has partnered with a leading US bank to ensure non-disruptive migration of their batch processing across several operational and analytical applications to an ETL based Data Integration platform featuring reusable frameworks such as generation of Gold Standard files, Common Maintenance frameworks, and Data Quality and Governance frameworks.

Is your mainframe systems and workflows limiting your agility? Consider migration to ETL platform. It might just be the right solution for you.

Blog authored by 
Ramakant Pradhan, Ramkumar Nottath and Arunshankar Arjunana

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

Please key in the two words you see in the box to validate your identity as an authentic user and reduce spam.