Infosys’ blog on industry solutions, trends, business process transformation and global implementation in Oracle.

« March 2020 | Main | May 2020 »

April 24, 2020

Oracle Data Lake

Oracle Data Lake

Evolution of data

In today's world, the quantity of data produced in a day is exponentially growing which is about 2.5 quintillion bytes of data being generated. The reason for the epidemic creation of data includes several platforms like Internet (information at our fingertips, web searches), Social media (fuels data creation with Facebook, Instagram, twitter, snapchat), Communication (from sending texts to email, GIFs, emoji's, skype calls), Digital photos (YouTube, voice search), Services like weather channel, Uber rides, transactions. As the data keeps growing, data handling comes to stake for most of the enterprise. The importance of these data rely on how the data is stored and how to extract value out of it effectively. The traditional method of storing data, such as relational database and data warehouses have their own limitations of storage capacity, type of data stored (Unstructured /semi- structured), storage cost, non-scalable.

Why Data Lake?

In order to overcome the limitations of traditional storage methods, Data Lake is provided by many service providers like amazon, snowflake, Microsoft, Oracle etc. for large storage with structured data, semi - structured data, unstructured data and binary data. It is a single root of all enterprise data including raw data from source system and transformed data used for activities such as visualization, reporting, prediction, advanced analytics and machine learning.


cloud.png

Here is how data lake differs from traditional data warehouse.

Data Lake

Data Warehouse

No Structured Data model and Retains all the data irrespective of any models

Highly structured data model which have specific data which answers the necessary questions

Data Lake stores all the data types including web server logs and sensor data

Data warehouse does  not supports datatypes such as web server logs, social network activity, sensor data

Data Lake stores Raw Data and Data is available all time, to go back in time and do an analysis.

Data warehouse stores Processed Data and significant time is spent on analyzing various data sources

Schema is defined after data is stored, efforts at the end of the process

Schema is defined before data is stored, efforts at the start of the process

Can store unlimited data forever

Expensive to store large amount of data

Adaptive, Highly accessible and quick to update

More complicated and costly to make changes

Used by data scientist for predictive analysis and machine learning, in-depth analysis

Used by business professionals for structured view of data and operational view of data

Uses ELT (Extract, Load, Transform) process, it empowers users to access data prior to the process of transformed and structured.

Uses ETL (Extract,Tranform,Load) process, it provide insights into pre-defined questions

 

Data Lake- Oracle Cloud Architecture:

archi.png

Data Lake mainly constitutes of:

  • Sources
  • Landing zone
  • Standardization zone and
  • Analytics Sandbox

·      

Key Components:

Oracle Data Integration Platform Cloud(ODI)

Oracle Data Integration Platform Cloud is affiliated platform for real-time data replication, data quality, data transformation, data governance, cleanse, integrate and analyze data. ODI encompass:

  • Migrate data without any down time
  • Integrate Big Data
  • Data health monitoring
  • Automate Data Mart generation
  • Profile and validate data
  • Synchronizing data
  • Support redundancy

Oracle Autonomous Data warehouse:

Oracle Autonomous Data Warehouse provides a fully autonomous database that does not require data administration for scalability and provides fast query performance. Deployment features includes either dedicated private cloud in public cloud service or a shared simple elastic choice. Database is capable of self-patching, self-tuning and upgrading by itself. The key features are:

  • Elasticity
  • Autonomous
  • Database migration utility
  • Cloud-based data loading
  • Enterprise grade security
  • Concurrent workloads
  • High performance

·        Oracle Stream Analytics

Oracle Stream Analytics is a tool for real-time analytic computing on streaming big data. OSA executes in a scalable and highly available clustered Big Data environment. It significantly enables users to explore real-time data like sensor data, social media, Banking etc. through live charts, maps, visualizations. Oracle Stream Analytics includes 30+ visualization charts, which are user friendly with respect to interface, based on Apache Superset. It is developed and made available to all the users without the need of any technical background.

Key features:

  • Location-based analytics using built-in spatial patterns
  • Machine learning to predict upcoming events
  • Ad hoc queries on processed data
  • Detecting real time fraud

·        Oracle Cloud Infrastructure

Oracle Cloud Infrastructure is a cloud service, which enables you to build and run a broad space of applications in a highly available environment with control improvements related to on premise data centers, subject to cost savings and the elasticity of the public cloud. Oracle provides technologies that entrust enterprises to solve critical business problems. Oracle Cloud Infrastructure is cloud purpose-built to allow enterprises to run business-critical production workloads. Key Features includes:

  • High availability - deployment against multiple regions, availability domains (AD) and faulty domain configuration
  • Scalability - ability to scale resources automatically up and down w.r.t changing business needs so you pay for only what you use
  • Performance - High performance computing instances (HPC)
  • Price - low and enhanced price performance compared to other cloud services

·        Oracle Identity Cloud Service OICS

Oracle Identity Cloud Service provides single-sign-on SSO, identity management and identity governance for the applications, which is in the mobile, cloud and on premise application. It is fully integrated service delivering the core identity and access management activity with a multi-tenant cloud platform. Anyone can use the application any time anywhere on a device in secure manner. Oracle Identity Cloud Service will directly integrate with the existing directories and identity management which in turn easier for the users to access the applications. The benefits includes

  • Better user productivity and experience
  • Reduced cost
  • Improved business responsiveness
  • Hybrid identity

OAC - BI Reporting & Visualization:

BI helps in decision-making driven by data. BI encompasses the generation of data and analysis, eventually visualization of data so that business analysts and business leaders make the most needed decisions about products, strategies, market timing, and other mission-critical factors.

  • Oracle Analytics Cloud allows you to take data from any source, and explore and collaborate with real-time data
  • OAC helps you ask any question from your data with mobile-friendly features in OAC
  • OAC includes Self-service Visualization, Data preparation, Advanced Analytics, Enterprise Reporting
  • OAC is cloud-based analytics solution within the Oracle Analytics or Business Intelligence space 

Advantages of Data Lake

  • Data Lake stores data in original form and the advanced analytics depends on the actual raw data, used by data scientists and analyst to experiment with data and advanced analytical support
  • A data lake handles structured, semi structured or unstructured data such as streaming data, logs, equipment readings, telemetry data and able to derive value regardless of data type
  • For high-speed data streaming in huge volumes, Data Lake makes use of tools such as Kafka, Flume, Scribe, and Chukwa to acquire high-velocity data, which is in the form of Tweets, WhatsApp messages, Instagram or it could be sensor data from the machine
  • Offers cost-effective scalability and flexibility, we can store all types of data inexpensively hang on to it for some future analysis for getting value out of it anytime needed
  • Collects and stores huge data sets, visualize telemetry and customer data, detect anomalies and ensure security
  • Data Lake can be the data source for a front-end application providing application support
  • In Data Lake, we can define the structure of data or schema, transformations at the time of its use, which is called schema on reading and also it allows schema free unlike traditional data warehouse
  • Data Lake supports more languages other than SQL such as to analyze the data flow, PIG can be used and Spark MLIB for machine learning. Tools like Hive allow us to run multiple parallel sql queries thereby reducing the query access time   

Industrial Applications of Data Lake

Oil and gas industry analytical requirements include minimized unplanned downtime, optimized directional drilling, lowered lease operating   expenses, improved safety and adherence to regulatory matters as collection of data for prediction analysis

Smart city initiatives includes tracking the vehicle pattern, speed, waterways, tolls, highways, bridges, usage timings which can be used to   manage traffic signals, control traffic, prevent congestion 

Life science study, which includes storing data on heart rate, blood pressure, white blood and red blood cell counts, temperature, height,   weight, enzymes and analysis, may help in predicting the increase in human life expectancy by data analyst

Marketing and Customer data platform creates a database for every customer that incurs data from multiple sources like mobile and web   preferences, profile data, browsing history, behavioral and transaction data, brick and mortar system, loyalty program which leads to   personalized marketing program

Banking industry stores customer account data, credit and debit card transactional data, wireless payment data, general ledger data   including   purchase information, trading data for many years now, improving the data agility.

·   

 

April 23, 2020

Hyperion Data Integration automation in a hybrid environment

This article explains the data load automation solution in a hybrid Hyperion environment. Firstly, let us understand what is actually meant by a hybrid integration environment. In the Hyperion world, data integration i.e. import, transform and load of data from the source system to the target system is achieved by either on premise FDMEE or cloud Data Management.

A module called Data Management is available along with the cloud Hyperion applications like EPBCS, FCCS, PCMCS etc to meet the data integration requirements for the cloud applications. Using Data Management, only cloud ERP applications like Oracle Financials Cloud can be directly connected using the inbuilt adapter.

The on-premise tool for data integration is FDMEE. Using FDMEE, on premise ERP applications like Oracle EBS, Peoplesoft, SAP etc can be directly connected and the target Hyperion applications are also on premise. 

A hybrid environment is where the source ERP applications are on premise applications, the target systems are cloud based applications and FDMEE is the integration tool. In this case, the data loads will be configured per the standard process but if end to end automation is required where some rules need to be triggered in the target application post data export, this cannot be achieved directly in FDMEE as cloud rules/scripts cannot be directly called from FDMEE.

As an example of this scenario, consider the source system as on premise Oracle EBS GL, target application as cloud EPBCS and FDMEE as integration tool. Such a scenario is possible only when the client organization is still using on premise ERP applications, a mix of on premise and cloud Hyperion applications and also has license for FDMEE from the past.

In such cases automation will be achieved in 3 steps -

1.       Create the regular data load rule in FDMEE to import, validate, export data from on premise EBS GL to cloud EPBCS application.
2.       Write an event script in FDMEE - AftExport - in which you call a windows batch and pass the scenario and period as arguments from FDMEE.
3.       In the windows batch, use EPM Automate commands to trigger the post load calculations or business rules of EPBCS application.

A prototype of the AftExport event script is explained below:

# Import required libraries

if fdmContext["LOCNAME"] == (" << EPBCS location name >> "):

# If period name is in the format of Jan-20, split to get the month and year. Convert year to FY20

  period_split = fdmContext["PERIODNAME"].split('-')

  period = period_split[0]

  year = "FY" + period_split[1]

  scenario = fdmContext["CATNAME"] 

  version = "Final"

# Call the windows batch placed in the FDMEE inbox directory and pass the year, version and scenario as arguments

  os.chdir(fdmContext["INBOXDIR"] + "/Batch_Files")

  command = fdmContext["INBOXDIR"] + "/Batch_Files/AftLoadBatch.bat " + year + " " + version + " " + scenario

  p = subprocess.Popen(command, shell=False)

  retcode = p.wait()

 

Similarly, you can also have BefLoad script to call a windows batch to execute any business rules before loading the data like clearing data for the period before load.

Below is the prototype for the windows batch:

REM Login command

CALL C:\Oracle\EPM_Automate\bin\epmautomate login <<username>> <<password>> https://planning-test-XXX.pbcs.us6.oraclecloud.com XXX >&1

REM Run business rule for Currency Translation

CALL C:\Oracle\EPM_Automate\bin\epmautomate runBusinessRule USD_Translate Scenario=%3 Version=%2 Years=%1 >&1

REM Run business rule for Aggregation

CALL C:\Oracle\EPM_Automate\bin\epmautomate runBusinessRule AggregateData Scenario=%3 Version=%2 Years=%1 >&1

 

Pre-requisites for this automation:

  • 1.       EPM Automate should be installed on the FDMEE server preferably at the path C:\Oracle\EPM_Automate. Ensure that the full path has no spaces in between else the EPM command throws an error in the windows batch.
  • 2.       If using a password encryption file for the EPM Automate login command (which is the recommended best practice), ensure that it is placed at the path C:\Oracle\EPM_Automate\bin.
  • 3.       Create batches in FDMEE and schedule them. The FDMEE batch will be triggered at the scheduled date and time which will trigger the associated data load rule. As soon as the data export is completed, the AftLoad script will be triggered to call the windows batch to execute the EPBCS business rules, thus achieving complete end to end automation.

April 21, 2020

Synopsis of EPM Cloud Updates - April 2020

ENTERPRISE PERFORMANCE MANAGEMENT CLOUD - APR '20 UPDATE


EPM CLOUD COMMON:


NEW EPM CLOUD INTERFACE:

There is an upgraded GUI included in this update, with better navigation and new themes like the popular Sky Blue.


REST API TO REVIEW REJECTED METADATA AFTER IMPORT

When using a REST API to import metadata, you can now define an error file that collects unimportant metadata records for each dimension. If an error file is defined, each dimension will be generated with a separate error file. Using Inbox / Outbox Explorer or applications like EPM Automate or REST APIs, the error files are then zipped together and the zip file is saved in the outbox where you can access the file.

Note: This relates to Planning, Financial Consolidation and Close, and Tax Reporting


REST API FOR PARTIAL CLEAR

REST APIs can now be used to do partial clearing for an ASO block. You can clear the cube, members, valid MDX query, with information, comments, and attachments to help. On Essbase you can also do a physical clearing. This allows for greater versatility and granularity when clearing the cube.

Note: This relates to Planning, Financial Consolidation and Close, and Tax Reporting.


ACTION MENUS NOW SUPPORT LAUNCHING A FORM IN MEMBER CONTEXT

You can launch a form in the member context of the original form via an action menu, either on the web or in "Smart View". EPM Cloud users were previously only able to launch forms in the dimension context of the original form. This feature relates to these EPM Cloud business processes: Planning, Financial Consolidation and Close, and Tax Reporting.


NEW FLEX FORMS IN "SMART VIEW"

Flex forms are a new type of form offering versatile row management in "Smart View". Flex forms retain the properties and characteristics of the regular form, but you can now rearrange the row members and sort or transfer rows. In ad hoc mode, you can also open a flex form and use ad hoc analysis to adjust the grid layout and submit data.


SUPPORT FOR MICROSOFT EDGE BROWSER

From this update, it support Microsoft Edge browser version 80 +.


ACCOUNT RECONCILIATION:


IMPORTTMPREMAPPEDTRANSACTIONS EPM AUTOMATE COMMAND CHANGE IN BEHAVIOR


The EPM Automate ImportPreMappedTransactions command, which imports pre-mapped transactions from a CSV file, displays the import status and a log file. The log file can be accessed from your tax reporting system using the update command File.


FINANCIAL CONSOLIDATION AND CLOSE


TOP 5 CONSOLIDATION AND TRANSLATION JOBS BY DURATION IN THE ACTIVITY REPORT

The Activity Report was expanded with a table showing the top 5 consolidation and translation jobs by length. It includes information in consolidation performance diagnostics logs to help solve consolidation problems.


PERFORMANCE SUBSTITUTION VARIABLES

The below Substitution Variables can be checked to enhance performance:

·         "EnableSimpleAggregation"

·         "OptimizeDBRefresh"

·         "OptimizeYTDCalculation"

·         "OptimizeConcurrency"

NOTE: "The degree of performance improvement varies widely across different applications, as it is powered solely by application design and data distribution."


TASK MANAGER ALERTS

Alerts feature has been improved to add inst, questions, attributes, and expanded workflow. Alerts Types can be characterized as a stored procedure capturing critical information and assigning key personnel for resolution of issues. In Task Manager, notifications can now be linked with more artifacts such as schedules and reporting times. Many events may be associated with a single Alerts. The updated Warnings List allows for unified management of all warnings.


CONFIGURABLE CALCULATION RULES

If the Ownership Management function is allowed for an application, the following Configurable Calculation Insertion Regulations require you to write to a Proportion member.

·          "FCCS_50_After_Opening Balance Carry Forward_Consolidated  (for multi-currency applications)"

·         "FCCS_60_Final Calculations_Consolidated (for multi-currency applications)"

·         "FCCS_130_After_Opening Balance Carry Forward_Consolidated (for single currency applications)"

·         "FCCS_140_Final Calculations_Consolidated (for single currency applications)"


APPLYING THE SOLVE ORDER METADATA PROPERTY

A new Configuration function helps you to quickly apply the Solve Order property for all Dynamic Calc members of a dimension in Extended Dimension applications. The metadata property of the Solve Order determines the order of the member evaluation and the order in which calculations are to be solved.


CONFIGURABLE CONSOLIDATION RULE-SETS

The below features apply to Configurable Consolidation Rule-Sets:

·         You will also use the effects of the current entity's Opening Balance Carry Forward, Proportionalization, Standard Eliminations and OpeningBalance Ownership Change framework rules when you create a set of rules for configurable consolidations, for source data.

·         When constructing a set of rules for Configurable Consolidations, you can now pick a Data-set View aspect member. The default member view selection is "FCCS Periodic," but it can be changed to "FCCS QTD," "FCCS HYTD" and "FCCS YTD" if enabled.

      NEW CONFIGURABLE CALCULATION RULE IN TRANSLATION SEQUENCE

      For the consolidation phase a new Configurable Calculation rule called "Foreign Exchange (FX) Calculations" is now available. The rule executes after translations, but before calculations on the Foreign Exchange / Cumulative Translation Adjustment (CTA). This allows you to build rules that alter previous system estimates, but are still subject to the Foreign Exchange and CTA system's "balancing" effects.

      SEEDED NET INCOME OWNERSHIP RULES

      In the seeded Ownership rule-sets "Net Income (Subsidiary)" and "Net Income (Equity)," the movement element was added to the rule-set framework, with the member "ILvl0Descendants(FCCS ClosingBalance)" and excluding: "FCCS OpeningBalance."

TASK MANAGER GLOBAL INTEGRATION UPDATE

The tab System Attributes has been dropped and Global Connection Tokens is now part of the Settings system.

TASK MANAGER REASSIGNMENT NOTIFICATIONS

When tasks are reassigned, e-mail updates are now sent to reassigned users immediately.

ABILITY TO USE AMOUNT OR RATE OVERRIDES IN ON-DEMAND RULES

You can now define "FCCS Amount Override," "FCCS Rate Override" or "Entity Input" in On-Demand rules as legitimate Consolidation members for run-time prompts.

NEW CALCULATE MOVEMENTS SYSTEM RULE FROM CLOSING BALANCE INPUT

Now Financial Consolidation and Close provides data entry for "Closing Balance Data." Any measurement of movement amounts based on the Closing Balance Input entry, however, required user-created Insertion Point or Regulations on request. A new machine rule has been introduced with that release: "Calculate Movements." This rule will measure movements from Closing Balance Input based on new attributes of the Account metadata and the Movement dimension. When metadata attributes are modified and the system rule is allowed on the Consolidation's Local Currency tab: System screen, any "Closing Balance Input" entry will produce a determined amount of movement that will be posted to the specified movement. You can select a global default movement for all level 0 accounts, and you can also select specific movements for each individual level 0 account.

NOTE: At present, members of the seeded movement cannot be used as appointed part of the movement. A subsequent release addresses this shortcoming.

TASK SCREEN ENHANCEMENTS

The Task screen was improved to include a new Overview ribbon within the task header. This summary bar tells the user easily how much time remains on the mission, its priority and the number of new comments, the questions and attributes needed, and the notifications opened.


"NARRATIVE REPORTING"


UPDATED NARRATIVE REPORTING EXTENSION FOR "SMART VIEW"

The enhanced Oracle "Smart View" Narrative Reporting Extension is now available for Office. This update contains changes to the general functionality and bug fixes. Download and install the new "Smart View" Narrative Reporting extension to get some improvements.

UPDATED NARRATIVE REPORTING EXTENSION FOR DISCLOSURE MANAGEMENT

An enhanced Narrative Reporting Extension is now available for the Oracle Disclosure Management. This update contains bug fixes and general enhancements. To get those changes, download and install the new Disclosure Management Narrative Reporting extension.


"PLANNING"


CAPITAL SUPPORTS IFRS 16 FOR LEASE ASSETS

Capital also supports International Financial Reporting Standards 16 (IFRS16) for lease properties, which adjust the manner in which lease properties are managed on the P&L and on the balance sheet, for the lessee's perspective.

NEW STATUS INDICATOR ON FORMS

A new type status indicator now shows icons to show the server operation "busy" and "idle"

REVIEWING REJECTED METADATA AFTER IMPORT

While determining the import job of the Planning metadata, you now have the option to define an error file that collects the records of metadata not imported for each dimension.

UI OPTION FOR CREATING FREE FORM APPLICATION

Free Form Planning applications can now be built using a UI-based wizard.

IMPORT DATA FROM A FLAT FILE INTO A STRATEGIC MODELING MODEL

You can now import data into a Strategic Modeling model from a flat file. Only files in .csv format are supported.

CALCULATION MANAGER OPTION TO DISABLE FLASH


"TAX REPORTING"


CHANGES TO MOVEMENT DIMENSION

When tax reporting facilitates ownership management, a number of reserved system members are generated upon upgrade and should not be used for data input.

TASK MANAGER REASSIGNMENT NOTIFICATIONS

In Task Manager, email updates are now sent directly to the reassigned users when tasks are reassigned.

             

TASK MANAGER GLOBAL INTEGRATION UPDATE

The Program Attributes tab has been removed in Task Manager, and Global Connection Tokens now forms part of device settings.

TASK MANAGER ALERTS

The Notifications function has been expanded in Task Manager to include orders, queries, attributes and extended workflow. Alerts Types can be characterized as a stored process collecting critical information and assigning key personnel to solve issues. Alerts can now be correlated with more Task Manager artifacts, such as schedules and reporting times. Many events may be associated with a single Alerts. The new Warnings List allows unified control of all alerts.

OWNERSHIP MANAGEMENT

Ownership management consists of controlling global consolidation settings and applying those consolidation settings on a scenario-by-scenario, year-by-year, and period-by-period basis to each entity's hierarchies. Ownership settings refer to each combination of parent and child for each Case, Year and Time.

UPDATES TO TAR AUTOMATION

The TAR Automation form has been modified for Administrators and Power Users to allow you to set multiple Accounts and single selections for Scenario, Year and Time using the Member Selector. You may also choose to reveal the member name or alias.

TASK SCREEN ENHANCEMENTS

The Task screen was improved to include a new Overview ribbon within the task header. This summary bar instructs the user quickly about how much time remains on the mission, its priority and the number of new comments, the questions and attributes needed and the notifications opened.

April 20, 2020

EPM Agent Integration

Introduction:


The EPM integration agent is used for extracting data from on-premises and loading it into Oracle EPM Cloud applications. We can connect the on-premises data sources using SQL queries.


Download EPM Agent:


Login into PBCS application, goto Application and click on Data Exchange:

                    

A1.png


Click on Actions under Data Integration and choose Download Agent from the drop down list.


           
A2.png


Update parameter.ini file:

Extract the Zip folder and update the .ini file which is located under windows folder.

           
A3.png

        
A4.png


Extract the Zip folder and update the agentparams.ini file which is located under windows folder:


A5.png


Password Encryption

Use the encryptpassword.bat job to encrypt the password and update it in the agentparams.ini file:


           
A6.png

Create App folder:

Run the CreateAppFolder.bat file to create app folder.

A7.png


Download the SSL certificate


Access the Cloud service URL and Download the SSL certificate

Place the downloaded the certificate into cert folder.


A8.png

A9.png

Start the EPM Agent:

Set the EPM Agent home directory in cmd and run the EPMAgent.bat file


A10.png



Define SQL query in EPM Cloud:



Goto Actions and click on Query

Create new query with SQL


A11.png


A12.png



Creating SQL query using REST API


Launch SoapUI


A13.png

A14.png

REST API Project

Create new REST API project with EPM

Cloud Service URL

A15.png

SQL Query Parameter

Set the SQL query


A16.png

Update Authorization details

A17.png


A18.png

Create Target Application in  

Data Management


Create new target application with Data source option

Select On-Premise Data base and upload the file template.

Create Location and Import formats.

A19.png
A20.png


Create Data Integration
 Define Source and Target application in the data integration

  Define the mapping and save it.

  Run the data integration


A21.png


Trigger the data load rule in DM:


Execute data load rule

A22.png

Data Extraction from Database Source:

The data extracted successfully.


Picture23.png

April 9, 2020

Oracle Stream Analytics

What is Data Stream:

Speed is one characteristic that drives the world now a days, whether it is downloading a big file, movie or working from home etc. Merely increasing speed is not sufficient, storage increase demand is also continuously rising. A solution was offered by services like Netflix/Spotify to consume content directly into handheld devices without downloading with exceptional speed. These services made it possible to send and receive billions of bytes of data. Due to continuous flow of data like jet of water, these services are called as streaming services. Today data streaming exists in many forms; audio, video, media streaming is just one part of it. Humongous growth in data and advancements in engineering processes led to different ways of gathering, analyzing and processing the data. Due to this it was possible to provide instantaneous analysis of the streamed data.

Why Stream analytics:

Streaming analytics or Real-Time analytics is an emerging type of analytics that sources data in real time, performs simple operations or calculations real time in order to provide business insight of fast moving data.  It is quite different from traditional Warehousing ETL techniques, in traditional techniques business calculations are performed on a batch of data overnight, however in real-time analytics operations like filtering aggregation, grouping etc. are performed on a stream of continuous flowing data. Huge amount of data is flowing from one system to another system every minute. It is observed that organizations which can act on the stream of data are able to improve their operational efficiency. A wide range of industries can take advantages by issuing real time alerts with the help of real time data stream analytics. These alerts can be different type including promotional alerts, fraud detection alerts or informative alerts etc. Data stream analytics is highly scalable, low cost, high throughput and reliable solution. Data Stream analytics is cloud based service, making it as low cost solution in which organizations pay as per usage. Streaming analytics is primarily a cloud solution provided by multiple vendors like Microsoft, Oracle, Amazon etc.

Oracle Stream Analytics (OSA):

Oracle Stream Analytics is a big data based real time tool which uses in-memory engine technologies for real time stream data analytics. Data streams can source data from applications from different areas like sensing equipment, Banking Point of Sales, ATMs, Twitter or any other social media, Traditional Databases or Data Warehouses etc.  OSA offers a web-based, user-friendly streaming analytics for business users. Users can dynamically develop, design and implement instant analytical solutions which give insight of streaming real time data. One of the best advantages of the tool is that it allows user to explore the data with different advanced visualizations like charts, maps, geographic markings etc.  OSA uses Apache Kafka and Apache Streams integrated with Oracle's engine in order to address the real time requirements and analytical challenges of the users.

Components:

Stream: As the name suggest stream specifies the source of flowing data (not static, continuously changing). The data can be sourced from stock market, JMS Server, REST APIs, Twitter etc. This data or stream of data changes with every passing second and is fed to Oracle Stream analytics for processing.

Reference: Reference is the source of data which is referred for fetching some information about the event data. It can provide contextual information about the flowing data in stream. It can be static database tables or static excel or csv files. In this release of OSA only oracle tables are supported for reference.

Exploration: Business rules or set of criteria defined for exploring and managing the event data. Exploration applies filters on data, group data by different groups, provide summary of the event etc. An already configured data can be added or attached to an exploration.

Topology Viewer: Topology viewer provides a graphical representation that showcase the dependencies amongst different entities. Immediate Family and Extended family are the two contexts supported by OSA topologies. Immediate family identifies the dependencies between parent and child, however Extended family identifies the dependencies in full context.

Pattern: Based on common business scenarios a simple way to explore event streams is referred as Pattern.

Map: Geo fencing collection is referred as Maps, it is used to locate the geographic coordinates specified from different sources like GPS.

Shape: Shape is the representation of event data in different forms like charts, pie graphs etc.

OSA Architecture:

First step in OSA is to ingest data from applications, golden gate change data capture method from Kafka. After that examining and analyzing is performed on the sourced stream by using data pipelines. In Data pipeline data is queried, business or conditional logics are applied, patterns are identified on the data streams. All these operations are performed when data is flowing and not stored anywhere. Continuous Query language(CQL) is used for querying data. CQL is similar to SQL, it contains additional constructs for pattern matching and recognition. OSA generate the query and spark stream automatically. Once the analysis is complete using data pipeline data can be fed in data lake for deeper insight analysis or any other integration trigger/alerts can be sent immediately.  A high level architecture is as shown below: 


arch.png


Few reasons why OSA should be used against its competitors are:

Simplicity: It is simple to use web-based tool which doesn't require much technical skills. It can also generate and validate some of the most powerful data pipelines automatically.

Apache Spark: OSA is built on apache spark which give the flexibility to attach itself to any compliant yarn cluster. It is the first tool in market to introduce event by event processing on a spark streaming.

Enterprise Grade: OSA can scale out horizontally and highly available (24*7) for critical workload pipelines makes it an enterprise level tool. In-built governance ensures no data loss at any point in time.

Industry Advantages:

Risk and Fraud Management - Financial industry uses stream analytics to detect the fraud on the PoS or online by analyzing the data streams.

Transportation and Logistics - OSA can help in managing fleet, tracking assets, and help in improving supply chain efficiencies.

Customer Experience and Consumer Analytics - Knowing the sentiment of the customers is the key in releasing offers, analyzing trends etc. OSA can play a crucial role in analyzing the customer trends.

Telecommunications - OSA can help in proactively monitoring the networks. It can also predict network failures and help achieving high availability.

Retail -- Instant shopping trends, shelf arrangements for benefits, customer cart utilization response can be achieved with OSA to increase the sales in retail industry.

 

References: https://docs.oracle.com/middleware/1221/osa/using-streamanalytics/GUID-81454309-164B-4933-B972-A9FEF06159D0.htm#OEPSX127

 


Subscribe to this blog's feed

Follow us on

Blogger Profiles

Infosys on Twitter


Archives