Infosys’ blog on industry solutions, trends, business process transformation and global implementation in Oracle.

« December 2020 | Main | February 2021 »

January 17, 2021

Working Model of Stock Price Prediction using Natural Language Processing

Natural language processing, widely known as NLP, is a subfield of artificial intelligence. This is used to create a link between human and machine. NLP helps machine to understand human language by educating or train them based on rules or data.

NLP is mainly used in Speech Technology, OCR, Machine learning etc. Most common example we can consider of NLP is email or text filter, predictive words in email or text, digital assistant, data analysis etc.

Now researchers have taken NLP into a next level where a machine is trained to understand financial market's up and down. Using its data analysis capability, a machine can now predict how a specific stock price will behave in future.

nlp1.jpg

Now this is important to understand why we needed such an AI powered system which will tell what stock to buy or not. This can be also done by a human also. But here machine beats human by it's computational power. It can analyze large number of historical and current financial data, data from social media and news and then analyse it and finally provide the prediction.

Investment is one of the most difficult decisions which may result in huge profit or loss according to the investors' analysis. It is very crucial that the extent of human errors in these pressure situations is reduced so that the profit can be maximized. The technical analysts believe that the future price can be forecasted using the past price movements.

Sentiment analysis uses text mining, natural language processing and computational techniques to automatically extract sentiments from a text. It aims to classify the polarity of a given text at the sentence level or class level, whether it reflects a positive, negative, or neutral view. In stock market prediction task, two important sources of the text are used either social media or online financial news article and historical stock prices. Sentiment analysis decreases the risk factor by informing the investors about the intricacies of the decision they are about to make. The stock closing prices for some future date could be predicted by training the machine learning models by providing the stock prices for previous dates. When sentiment analysis is applied on stocks in news from moneycontrol.com regarding the public sentiment or opinion on that stock. Then, it becomes evident that whether to invest in that stock or not.

Block diagram representation:

F1.png

Above diagram shows how data can be fed to a machine and then rules will be applied to those data to make the prediction. The more data machine consumes to train the more accurate result can be seen.

To show how this prediction model works I have created various case studies and tested with Amazon, Facebook and Netflix stock prices.

Note: These codes are written in Python using google colab.

Programs are created for this article are very simple and it shows how to train the machine with dataset and predict future stock prices. I have used SVM, LR and Decision tree model. These programs uses downloaded stock files from financial site (Yahoo Finance) as input data. Also, these programs can be more enhanced which can read from all type of social media news and multiple financial files.

SVM model:

Support Vector Machine is a supervised machine learning algorithm which can be used for both classification or regression challenges. However, it is mostly used in classification problems.

LR model:

Linear regression was introduced in statistics as a model to understand the relationship between input and output numerical variables. But later this is used in natural language processing. It is both a statistical algorithm and a machine learning algorithm.


Below are the steps used for Decision Tree Classifier with Amazon, Facebook and Netflix stock files.

Case Study 1: With Facebook Stock

We are loading the stock file here.

fb1.PNG

Next step is making 'Date' field as indexed field.

fb2.PNG















Here we are using identifier 1 or 0 to understand when stock prices gone up or down. We are monitoring 'Close' field for this purpose. So, if the price is up next day it will show as 1 and if the price is down then it will show 0. Please refer 'Price_Up' column.

fb3.PNG


Next step is manipulating this dataset for further activities:

fb4.PNG

 

 Now we are creating training model; 80% of total stock data will be used as training data and 20% as testing data

fb5.PNG


Above score is predicted by Decision Tree Classifier.

Below is the comparison of actual and prediction data:

fb6.PNG

Case Study 2: With Amazon Stock

We are using same program here with different stock file.

amzn2.PNG


amzn5.PNG

Prediction as below:

amzn6.PNG

 

So far, we have seen Decision tree model, the prediction score is not high enough. Therefore, Support Vector Machine Model or Liner Regression Model comes into picture. Below studies have been done with both SVM and LR.

Case Study 3: This has been performed with Wiki data and Facebook stock price which is available in quandl

First, we will install the required packages as below. Also this is a new program created. Please follow below series of steps.

quan1.PNG

quan2.PNG

 

quan3.PNG

 

quan4.PNGquan3.PNG

 

quan5.PNG

 

quan6.PNG


quan7.PNG


SVM model score:

quan8.PNG

Linear Regression model score:

quan9.PNG

quan91.PNG

 

quan92.PNG

LR vs SVM prediction:

quan93.PNG


quan94.PNG

Above case studies help us to understand how an AI based prediction system can be built. Also, the prediction score will depend on multiple factors like complex logic, training dataset etc. This process is not just simply trying to predict a value but it works on every stock related sentiment and risk analysis.


Now we will perform another case study which will show the graphical representation of prediction.


Case Study 4: Graphical representation of Amazon stock price prediction

To showcase this below steps/codes were built. 

gm1.PNG

 

gm2.PNG

 

gm3.PNG


gm4.PNG

 

gm5.PNG

 

gm6.PNG


gm7.PNG

 

gm8.PNG

 

gm9.PNG

gm91.PNG


gm92.PNG


gm93.PNGgm94.PNG

 

Graphical representation of Original and Predictive values in Tree model:

gm95.PNG

gm96.PNG

 

Graphical representation of Original and Predictive values in LR model

gm97.PNG

 

gm98.PNG


As mentioned earlier, this machine can perform more accurately if it is built to handle massive load as training data equipped with better hardware or processor etc. Also, this can be interfaced with any number of language feed. The machine will translate any feed to a common machine language and then perform its analysis

 

Reference:

1. Wikipedia

2. Yahoo Finance

3. Forbes

4. Google Images

 


January 8, 2021

Artificial Intelligence is the future of Finance

Artificial intelligence was founded as an academic discipline in 1955. It was created on the assumption that human intelligence "can be so precisely described that a machine can be made to simulate it". 
mb1.png
Human has limited brain power and limited time. On the contrast AI has lot more resources like computational power. AI uses different type of algorithms which are capable of self-learning from data pattern or features in the data; They can enhance themselves by learning new strategies. Also, these algorithms can be so powerful that it can write a new algorithm based on changing situation.
With a changing technology AI is reshaping the financial sector by its own way. Every day we can see new features, new technology in all kind of Digital Assistant apps. This is making AI as a strong competitor to any technology. We can see now a better customer care which uses a self-help VR system which is nothing but an intelligent natural language processing technique with mix of high-end speech technology.
AI has multiple benefit in finance industry starting from credit decision, personalize banking, risk management etc. Also, it helps to automate middle-office support by providing 24/7 customer interactions, reducing the repetitive work etc. According to an article from businessinsider, automating middle-office tasks with AI has the potential to save North American banks $70 billion by 2025. Further, the aggregate potential cost savings for banks from AI applications is estimated at $447 billion by 2023, with the front and middle office accounting for $416 billion of that total.


Credit decisions:

mb2.png
From data to DECISION; AI driven credit decisions help bank and credit lender make smart choice by analysing large amount of data and multiple factors like current income, employment, credit history and ability to earn in addition to older credit history. It provides faster, non-biased and more accurate assessment. Credit Scoring technique is nothing but an intelligent AI system which is running on complex rules. Those rules which is running in backgrounds distinguish between a high risk and low risk applicant based on analysing all the credit history in past. AI can also adapt to new problems, like credit card churners, who might have a high credit score, but are not likely to be profitable for the card issuer. Or an applicant with a stable job income but might not be a good candidate for high amount of loan based on all his previous dealings.
Based on advance rule AI can help customer who has good credit risk but getting denied based on manual/rule-based history check. One negative aspect can be, accessing all kind of previous history or analysing a large amount of personal data can lead to a privacy concern.


Risk Management:

mb3.png
AI model has improved the analytical capabilities in risk management by its processing power of huge amounts of structured and unstructured data in a short period of time which cannot be done by a human. Algorithms analyse the past details for risks and identify what can be the potential issues. This allows risk managers to identify any risks associated with any action and that gives the idea how to prevent it.
With old risk management tool, it was very difficult and time consuming to analyse the real time activities and identify the potential risks. New tools with AI is helping to analyse all kind of risks and come up with probable solution,  although it has some technical challenges of developing AI apps for banking, such as building correct and relevant algorithms, there are also challenges related to the regulatory field and data access rights.


Fraud Protection:

mb4.png
AI is a perfect match for the rapid escalation of nuanced, highly sophisticated fraud attempts. AI approach to fraud detection has received lot of publicity in recent times. Also, this has successfully shifted industry's old rule-based approach to ML based solution. Using rule-based approach it was almost difficult to find hidden and implicit correlations in data. ML based approaches provides the features like automatic fraud detection technique based on data and this is possible in real time. New approach provides fraud analysts with real-time risk scores and greater insight into where best to set threshold scores to maximize sales and minimize fraud losses. 
Due to rise of mobile payment, all banks have introduced various verification stages so that it can handle modern frauds and scams. AI algorithm can help in paper based and bank data reconciliation which eliminates human error. Intelligent AI algorithm can help a bank to understand or send a notification based on transaction done away from customer's location. Another common scam is when scammers use someone's personal information. AI can help to prevent such case to and inform individual for e potential data theft.
AI can be used for anti-money laundering. Different countries had different guidelines while operating bank can be same. So banks or investment firms has to deal with different regulations to identify any suspicious activity. Rule-based approach sometime fails to identify the risk. 


Trading:

mb5.png
AI has brought a significant change in trading. Electronic trades account for almost half of the total revenues from cash equity trading. Most companies such as hedge funds, use AI-powered analysis to get investment ideas and build portfolios. This kind of trading is spreading quickly across the globe.
Trading system which is built on advanced AI can monitor any kind of structured data like DB or sheets and unstructured data source like news or social media etc. Based on market news or social media follow up or simply analysing large data it can give the right direction to any trading platform.
AI Trading gives benefit over Algo trading. Algo trading is whereby a computer program follows a set of instructions set to execute a trade. AI trading, on the other hand, is whereby machine learning is used to observe, study and analyze market conditions, trading patterns, and data, then predict what will happen.


Personalized Banking:

mb6.png
Banks and financial service providers were challenged to provide a best customer service in digital world. New technology is helping customer to do most of the banking task virtually. AI driven chatbots are helping customer with their queries which has helped to reduce call center workload. Now a virtual agent can help and guide you if you want to open a bank account or pay a bill.
AI helps to understand customer behavior which helps the bank to customize the services or production by adding customized features. Based on all the transaction done by customer with bank, AI can suggest bank what can be the best reward program for a customer. Reward program will help bank to retain customer and based on good services received, one customer can help bank get new member by referring them. Sometime financial institutes are using automated virtual system for any market research related work, as per market survey, people are more comfortable and honest while interacting with non-human entities like chatbot.


Process Automation:

mb7.png
Automation is one of the key aspects in Banking. By adapting robotic automation process many industries have cut down their operation costs. Automation of digital and physical task helped to boost the productivity also.
RPA is least expensive and easiest to implement which serves better than old business process. Example: it can read thousands of emails, letters or agreement or any legal document and identify the right keyword and stores in database for further business process. It helps to avoid any human error which is very important for any banking sector. An article from Forbes says, more than 65% cost reduction reported by Ernst & Young by adapting automation and this significant milestone has been identified as as "Gateway Drug To Digital Transformation". 

AI and Remittance industry:

The global payments landscape is going though a massive change worldwide. Every person in world has a smart phone along with a bank account or wallet which is helping customer to transfer the money across globe from anywhere and anytime. Now for this customer needs a seamless framework along with better rate and high security. Here AI is helping the leading remittance companies in market to provide better exchange rate, handle the risks or provide a secure transaction channel to every customer in world. There are tools based on AI, which can tell a customer about comparison of exchange rates provided by different remittance company. 
Now the next question comes in mind, what if the money transferred using new technology goes into wrong hands. Answer to this question is identify the right pattern to protect the money to prevent any organized crime. In earlier days any financial institute used to use the hard-coded rules to identify any suspicious behavior. But there was no process to identify any number of patterns. Example: if banking software finds a large number of amounts was sent from person A to B then it used to raise flag. But what about small amounts sent multiple times to one account or multiple accounts across many countries. Banks are increasingly turning to machine learning to mine vast quantities of bank data and find anomalies in accounts and transactions that might otherwise have gone unnoticed. 
Most automated transaction monitoring systems can identify transactions that are related to terrorism financing by using watch lists. To identify the "unknown" financier of terrorism financial institutes are using a different search strategy for detection. For example, using other relevant information from other customer channels combined with data could help an institution to better identify suspicious behavior.


Conclusion:

mb8.png
AI has the potential to become more intelligent than any human. The same technology which helped human race to build self-driving car or intelligent natural language processing system can be used for destructive work like creating viruses or scam mails etc. 

"The development of full artificial intelligence could spell the end of the human race. Once humans develop artificial intelligence, it will take off on its own and redesign itself at an ever-increasing rate. Humans, who are limited by slow biological evolution, couldn't compete and would be superseded."- Stephen Hawking

We cannot ignore the benefit of AI, but people always raise the question whether use of AI is good or bad. We cannot run away forever from technological progress and not facing it now may cost more in the long run. 


Reference:

1. Image Source: Google
2. Forbes.com
3. Wikipedia
4. BusinessInsider.com

January 7, 2021

Kick Start Hybrid Cloud with Oracle EBS using Oracle Supply Chain Planning Cloud

Overview

As part of the cloud strategy, organizations are investing and advancing the use of cloud services across business functions. Organizations plan to build the cloud strategy which delivers quick results with agility, business value and low risk. In this blog, you will learn how to kick start the Oracle cloud journey with on-premise Oracle EBS and public cloud Oracle supply planning cloud. The quick win is assured due to the fact the integration is standard feature provided by Oracle across on-premise EBS and supply planning cloud but there are choices to make which 

Seeing is believing - Proof of concept

For the SMB segments, seeing is believing even more when their "Cradle to Grave" cycle lacks the traceability and involves manual offline processes. Most of the demand planning processes to derive customer forecast and measuring the accuracy based on the actual sales orders are not tied back and the loop is not closed. All the historical data and its analysis are done on data warehousing tools which are manual people based and lack industry benchmark standards

 A quick proof of concept to show and reflect the changes needed to move towards industry benchmark standards and system driven forecasting process with built in analytics both in demand and supply planning helps the organizations on embracing the Oracle cloud journey.

SCP eBS POC Image 1.jpg

The above process flow illustrates the how the various supply chain planning cloud services provide different business function capabilities. Below is the summary of steps to perform quick proof of concept and test the applicable planning capabilities after configuring the planning system for the organization.

1.       Prepare the data for the various planning entities based on the organization requirements to reflect the business data for consumption downstream

2.       Upload the files on to the UCM folder against planning cloud

3.       Load the planning data files in the Net change mode

4.       Review the collected data and use the additional planning attributes using simulation set

5.       Configure and run the demand plan, load the historical data when needed

6.       Configure and run the supply plan with simulation set

Review plan output and performance indicators - Proof of concept

1.       Prepare the test scenarios to be validated

2.       Review the pre-build demand plan views

3.       Review demand plan output and analyze against the exist ones when available

4.       Review supply plan output and analyze against the exist ones when available

5.       Create additional demand/supply plan views based on business need

6.       Enrich the table views with available analytics and SCP cloud capabilities


Problem Description

The key challenges in the supply chain planning is the orchestration of the changing demand and supply scenarios. As the business planning organization embraces the SCP cloud journey, those challenges and short comings in the existing system landscape needs to be identified and business process need to be developed as per the industry standards. In any planning organization, there are top 5 main challenges faced as noted below:

1.       How does the organization validate the sales/customer forecast for the future and subsequently adjust against the inflow of the customer orders?

2.       How to plan to avoid excess inventory across the supply chain and improve the inventory turns?

3.       How does the organization not only manage their backlog of orders but also reprioritize based on customer request?

4.       How do we automate the planning process and capture the notes during various approvals?

5.       How do we improve productivity using analytics for making business decisions?


Shifting to cloud - choices to make

Apart from the business decisions and workflow around planning attributes enrichment and master data maintenance, there are decisions on product and customer hierarchies which need to be finalized before the SCP modelling so that business objectives can be realized and met in the future. Few of the other important decision on choices are listed below which greatly influence the change management across the organization

SCP eBS Choices to make Image 2.jpg

Solution  Details ( Process flows, System architecture)

The below illustration reflects the solution point of view and understanding on how the demand in the form of customer/sales forecast and actual customer orders drive the consensus business plan based on the supply and constraints (material and resources).

SCP eBS System Landscape Image 3.jpg

Data Collection from EBS

Ø  Target Refresh - The extract data for Oracle supply chain planning cloud is scheduled to run every day once in the Target mode which refreshed the collected data in planning central using the batch jobs scheduled from OIC.

Ø  Net Change - The extract data for Oracle supply chain planning cloud is scheduled to run twice or once in 2 hours in the net change mode to collect the changed supply and demand to improve the GOP results

Ø  Custom Measures - The custom measures are refreshed daily based on the on-premise EBS data and loaded to supply chain planning cloud as business metric. Few of the custom measures are sales order by customer request date, customer schedule ship date and customer price list


Demand Planning Process

After the Sales/Market forecast is uploaded into demand plan, the same is compared with the statistical forecast generated by the forecasting algorithm and the custom exceptions are raised for the demand planner to review. The quarterly forecasting review process is iterative, consensus is captured in the system using notes which are finally made available for management review. Once the approval process is completed after review using aggregation and disaggregation based on the customer and product hierarchies, the final shipment forecast is updated and approved.

The approved quarterly forecast is further managed on weekly basis based on actual flow of customer orders and netted using custom formula to manage the supply planning


Simulation Sets

The planning attributes are enriched and used in supply planning as part of the business requirement. Few of the planning attributes are given below apart from the time fences, lead times, order modifiers and safety stock which drive supply planning along with decision rules

Ø  Component Substitution logic - Use primary before using substitute/Exhaust all existing supplier using primary supplies first/ Exhaust all existing supplier using substitute supplies first

Ø  Enforce Purchasing Lead Time - Yes/No

Ø  Use common supplies - Yes/No

Ø  Hard pegging level - None/Project/Project and Task/Project Group


Supply Planning Process

The approved final shipment forecast is one of the demand streams apart from the actual customer sales orders which drive the demand after the consumption process during supply planning. Below are the few highlights of supply plan.

Constraint Plan

Constrained planning respects the material lead time as a constraint and overloads the resource. Such overloads are available as exception which are managed by the production planners. Demand prioritization rule for customer orders and customer/sales forecast

ü  Customer Orders by SSD

ü  Customer Orders by CRD

ü  Customer Orders by Order Value

ü  Customer/sales forecast by Forecast date

ü  Customer/sales forecast by Order Value

Decision Rules - Optimize business goals

SCP Decision Rules Image 4.jpg

Build Plan - Comprehensive view for analysis

Build plan is one of the standard view available as offering with supply plan whose layout can be configured and the filterable criteria applied for better analysis and decision making.

SCP Build Plan Image 5.jpg

Sales and Operation Planning Process

The approved quarterly forecast from demand plan is published to Sales and Operation plan where the analytical views are built on aggregation and disaggregation based on the customer and product hierarchies for the business users to create scenario simulation and what if analysis. The approved consensus forecast is validated for capacity planning based on supplier and resource modelling.

Further changes are made to the consensus forecast based on supplier capacity shared by supplier using supplier collaboration and rough-cut capacity shared by the production planners in the organization using BOR - bill of resources

The S&OP process has the various approval available in the SCP cloud for future reference and finally made available for executive review. The iterative S&OP process leads to the business plan which is tracked against actual for any course correction.


Benefits and takeaways

To stay relevant, organizations must develop cloud strategy which delivers quick results with agility, business value and low risk so they can transform processes to become resilient and adapt smartly to future requirements. Few of the SCP cloud driven solution/use cases and business benefits are listed below  

SCP Results Image 6.jpg

Subscribe to this blog's feed

Follow us on

Blogger Profiles

Infosys on Twitter


Archives