Open Source has revolutionized IT sector as it harnesses the intelligence and efforts of the community to develop and maintain software faster. Our experts discuss the latest happenings and give their view points on how open source can be leveraged for your organization’s transformation.

December 27, 2018

Couchbase Mobile Field Service Solution for a CPG Enterprise

CPG & Retail industries are the most disruptive and amongst an early adopter of technology due to the nature of the industry - they are consumer focused and highly competitive. As a part of the supply chain life cycle - the handshake between the CPG and Retail firms is extremely critical. The sale of products by the CPG firms to the Retail firms is through the on field sales representatives. My experience was around developing a mobile strategy for a CPG giant in Midwest.

The field sales agents typically work on collecting orders from the retail stores / warehouses which are deemed as their consumers. In the digital world, these orders are collected on their handheld devices (mobiles/ tablets). Each sales agent would service multiple customers in a specific geography and the products that they are entitled to sell in the geography. Product pricing (base price and any custom) and promotions are also essential for them to conduct the sale.

Figure 1. Data Entities for Order Collection

The enterprises have some basic requirements for the hand held applications. The mobile applications don't just integrate real time with the enterprises for data but they also need an ability to operate offline (with no internet connectivity). There is probably a notion that with the connectivity penetration across the geographies - why is this so important a feature. But for a field service sales agent  its very important to maintain the order collection experience and brand value. In order to operate offline - the data has to be available on the device.

Figure 2. Required Sync Features

This presents a need for a data synchronization from the enterprise data center to the handheld devices and vice versa. The retail enterprises look for basic requirements from the data sync and the handheld applications and these are illustrated in Figure 2. There is a functional need is to support one way (Most Customer, Products and Pricing data) and two way sync (Order and Order Status), full refresh (initial setup) and incremental sync, support for structured (product, customer and pricing data) and unstructured data (product images). Also each sales representative would need selective data (for example the customers they service in a geography) - this mandates the sync framework to support the selective routing of data based on routing rules.

Figure 3. Reference Architecture

A typical technical architecture for the data sync between and enterprise and devices is depicted in Figure 3. Instead of realizing this complex solution grounds up, enterprises should explore an out of the box solution and I am going to discuss one such solution - Couchbase Mobile field service solution.

Couchbase Mobile extends the robust Couchbase Data Platform to the edge, securely storing and synching data from any cloud to edge devices or directly between edge devices. I am not going to talk about the merits of Couchbase data platform and features it provides in this blog - but will focus on the mobile data sync solution and how it delivers features that any enterprise expects from a mobile data sync solution.

Figure 4. Couchbase Mobile - Components

The Couchbase mobile solution has the 3 components - the Couchbase server on the enterprise data platform, which is the highly scalable & available document based database server, the Sync Gateway which is a secure web gateway which provides synchronization capabilities between the server and handheld device and Couchbase lite embedded database which is a wrapper around the SQL-Lite database on the device.

The Sync GW provides a way to define the data routing rules for selective data sync, also exposes a set of APIs for data access off the data server. The Couchbase GW supports SSL for secure data transfer over the network. Couchbase server and Couchbase lite support encryption of data at Rest and the platform integrates with Enterprise LDAP for Authentication - fine grained data access control through Authorization controlled by Couchbase Sync server rules.

The Couchbase lite is a light weight database and the platform does provide programming support for IOS/ Android application development with compatible SDK's for C#/Java/NodeJS development. The SDK provides a way to support a continuous replication or one time replication. The sync is managed by hand held device and is managed over the web socket connection established with the Sync GW and very much bandwidth efficient!

There is a need for multiple applications sharing the same embedded mobile database, since the same customers & products are required by the multiple applications on the device (each serving a different purpose) for the sales representative and the data platform supports this requirement - but mandates that the sync be managed by one mobile application at a time.

All in all, the Couchbase mobile field service solution is an excellent solution option which provides for all the needed components, fully pre-engineered for deployment and operations which supports all the basic requirements of a CPG /retail field service mobile application, out of the box, thus improving the time to market and improving the overall customer experience. 

Continue reading "Couchbase Mobile Field Service Solution for a CPG Enterprise" »

March 28, 2018

2018 Trends in Open Source

Open Source in years has been evolving from being Free software alternatives to being innovative in many areas. In the recent couple of years Open Source projects are leading the innovation and even defining standards for the future. For example, Cloud engineering and solutions is been led by Open Source projects. Similarly another example Machine Learning and AI is also been led by Open Source projects. My experience working with Open Source technologies, here are the few trends that will define 2018. 

Progressive Web Apps
Experience layer from last several years has been critical aspect of digital transformation. Omni channel including Web, Mobile, other channels, have become important as the critical capabilities for successful digital experience delivery.

AngularJS, React, React Mobile, Vue.js and related technologies are becoming more stable and improving. This trend will continue in 2018. More Native based mobile frameworks like React Mobile and similar frameworks will keep increasing to deliver value for the user experience.

Cloud Native Engineering & PAAS
Platform as a Service has taken a good shape and how its been positioned with in the Cloud Strategy for any enterprise. Standards based products like OpenShift, PCF, Kubernetes, Tectonic etc have moved from Container as a Service to Platform as Service. Cloud Native Computing Foundation is driving and bringing the open source innovative projects together to make the cloud strategy and migration easier for adoption.

OpenShift, PCF and other Kubernetes based products are graduating from being Container as Service(CAAS) to Platform as Service (PAAS). One trend that is seen is enterprises are adopting product based implementations like OpenShift, PCF, Tektonic etc. Other trend is enterprises building their own PAAS using the best breed open source projects together. Enterprises are looking for PAAS can give single window for the service catalogue not just on Private cloud but also native services with in the public cloud providers. Multi cloud support as key differentiator will drive the PAAS adoption.

Anything as a Service (XAAS)
Anything as a Service as an acronym that represents any service should be available over internet than on-premises. Earlier years saw SAAS, IAAS are adopted by the big enterprises, Network as a service, Monitoring as Service, Communication as a service, Storage as Service are becoming very popular and will continue to hog limelight.

Given the high speed networks, stable virtualization and container technologies combined together is giving the power for this concept to shape and will increase in adoption in 2018.

Open Source Databases adoption on the raise
Open Source RDBMS adoption will increase in 2018 as its been going on last few years. There are massive parallel processing products on top of the open source RDBMS are seeing a role in the use cases where horizontal scalability is critical.

EDB PostgreSQL, MariaDB keep enhancing the features on the horizontal scaling, cloud migration, support for database as service on a cloud etc. Migration from the proprietary database to Open Source traction will continue in 2018 like seen in last couple of years. Citus, Vitess kind of products are making ways into main stream with support for horizontal scaling, NoSQL kind of capabilities like Map reduce, Replication, Sharding with good cost effectiveness.

NoSQL adoption is at a raise as the cloud migrations and adoption increases. This will continue in 2018. Explosion in data ingestion from different parts of the enterprise and models with multi-structure, unstructured, flexi-structure increasing, NoSQL adoption will continue. Engagement Databases like Couchbase, Aerospike with memory first architecture providing faster access to data to the experience layer is seeing a great traction. MongoDB releasing the ACID capability in its new version this year will fuel in a great differentiator in the market. 

 Machine Learning and AI
Gartner predicts that Machine learning and AI scope will be increased more in 2018. Several green field areas will see a surge around ML and AI. Technology around the adoption of Machine learning with in enterprises is increasing with platform builds, new user development environment adoption and move from on-prem to a cloud based compute executions for the leaning systems.

New Open Source Intelligent Solutions are set to change the way systems and people will interact with each other. Conversational AI with question and answer paradigm, Chabots will become a medium of interaction. Autonomous vehicles and drones are set to be used more in the business. Glasses technology to become another user experience channel for making things easier for the users.

Real-time Integration and Data Analytics
Modern data analytics platform moving away from old fashioned ETL and integration. Real-time data streaming to modern ELT/ ETL programming models like Nifi with Kafka is becoming main stream and enterprises are ready to adopt.

Confluent with its new avatar as the company behind Kafka will give a great push to this technology. Going beyond messaging into more reactive architecture/design is increasing in the adoption with large enterprises. Nifi with its large file processing especially with Data Integration like ETL style on streams has increased real time data transformation a big possibility. 2018 could be the year for Nifi adoption at large scale. 

It will be an exciting year 2018 for Open Source!

October 30, 2017

Query-driven data modeling methodology for Apache Cassandra

This blog explains how to use query-driven data modeling in Apache Cassandra. NoSQL data modeling is a process that identifies entities as well as the relationships between them. It can be used to determine patterns when accessing data as well as the types of queries to be performed. In doing so, it reveals how data is organized and structured along with how database tables are designed and created. It is important to note that indexing the data can degrade the performance of queries. Hence, understanding indexing is essential in the data modeling process.

Data modeling in Cassandra focuses on the query-driven approach whereby specific queries are the key to organizing data. Let me first quickly explain these terms: Queries retrieve data from tables and schema defines how the data is arranged in the table. Thus, a query-driven database design facilitates faster reading and writing of data, i.e., the better the model design, the more rapid data is written and read.

Now, first, we must create a conceptual data model that will define all known entities, relationships, attribute types, keys, cardinality, and other constraints. This data model should be created in collaboration with business stakeholders and analysts. For example, a conceptual data model could be presented as ER-diagram.     

The next step is logical data modeling. Here, the conceptual data model is mapped to a logical data model based on queries that are defined in an application workflow. The logical data model corresponds to a keyspace where table schemas define columns as well as primary, partition and clustering keys. Thus, the query-driven approach provides a logical data model using data modeling principles, mapping rules, and mapping patterns.

Here are some rules for query predicates that ensure stability and efficiency:

    Only primary key columns should be used in the query predicate

    All partition key columns in the query predicate must have distinct values

    Clustering columns may be omitted in the query predicate

    All partition key(s) must be used in the predicate

Besides these query predicate rules, there are additional data modeling principles to map to logical data models. It is important to note that violating these principles and rules will affect the ability to support query requirements and may lead to loss of data and performance degradation.

Here are the fundamental principals of logical data modeling:

1.    Know your data, particularly entity and relationship type keys that are needed to be preserved and relied on to organize the data properly

2.    Know your queries such that all columns are preserved at the logical level

3.    Enable data nesting to merge multiple entities together based on a known criterion

4.    Minimize data duplication to ensure space and time efficiency

5.    Use equality search attributes to map to the prefix columns of the primary key

6.    Use inequality search attributes to map to the table clustering key column

7.    Use ordering attributes to map to clustering key columns with ascending or descending clustering order

8.    Use key attribute types to map to primary key columns and uniquely identify table rows

Finally, we must analyze and optimize this logical data model to create the physical data model. The above-mentioned modeling principles, mapping rules, and mapping patterns ensure correct and efficient logical schema. However, efficiency can still be impacted by database engine constraints or finite cluster resources such as typical table partition sizes and data duplication factors. There are some standard optimization techniques that can be used, including partition splitting, inverted indexes, data aggregation, and concurrent data access optimization. These methods can be used for optimizing the physical data model, although we will not be covering this topic in detail in this particular blog entry.

This is the way to go about enabling query-driven data modeling in Apache Cassandra.

Continue reading "Query-driven data modeling methodology for Apache Cassandra" »

October 27, 2017

The Rise of Open Source


"[Open source] gives customers control over the technologies they use instead of enabling the vendors to control their customers through restricting access to the code behind the technologies."

- Eric S Raymond, The Cathedral and the Bazaar


Open source adoption began long ago, in the 1990s, with the introduction of the Linux Kernel. Then, there were only about 100 developers who contributed code to Linux. Since then, the Linux community has exploded. Today, over 8000 developers and 800 companies contribute code to Linux. What has steered such rapid development and adoption of Open Source technologies ? I believe there are three key reasons for this. Open Source helps businesses:


  • Improve agility and accelerate business innovation
  • Gain greater flexibility and scalability
  • Realize higher cost savings


We are witnessing open source adoption across all the enterprise layers like experience, application, business, integration ,database and infrastructure layer. The adoption begins with the experience layer which is primarily driven by the need to quickly enhance customer experience. The adoption of open source JavaScript frameworks like Angular and React has made it easier and faster to develop responsive web designs. While increasing the customer experience, it has also helped lower maintenance needs. Open Source has also provided freedom of choice and helped move to a polyglot environment. It is no longer necessary to stick to a single technology vendor to realize the benefits of economies of scale. Easy access and reduced cost of usage has made it possible to leverage an ecosystem of technologies across the layers based on the use case and the application attributes.


Today, more than ever, open source adoption is critical for business success and many organizations are realizing how open source helps them stay ahead of the curve. For instance, a global retail and wholesale giant replaced their Siebel CRM with a new microservices-based application developed on an open source stack. The adoption of open source helped them accelerate their transformation towards an environment which is more agile and adaptive. While the architectural principles were decided and enforced centrally, this organization gave the individual LOBs the freedom to choose their technology stack as long as it is open source


Despite its advantages, open source also has its own challenges. Customers struggle to identify the right open source technology. Scaling adoption across the organization is difficult considering the dearth and high cost of talent. Finally, large transformational programs means managing a partner ecosystem to get the best of breed solution.


Infosys open source offerings helps clients address these challenges for seamless open source adoption.




More importantly, our expertise in delivering such transformation has been gained from vast experience across organizations with varied maturity levels. For example, while some of our clients want ad-hoc open source implementations to prove a concept or a technology, others want it only occasionally for selective lines of businesses. Then, there are those who consistently and strategically leverage open source across all lines of business to differentiate themselves and gain a clearer advantage.


While achieving 100% open source adoption may not always be feasible, we will see a confluence of hosted, proprietary and open source technologies. Open source will gain more traction and play a major part within enterprise architecture even as it co-exists with non-open source technologies. It is recommended to choose an enterprise-supported version of open source to ensure enterprise-grade security, reduce risk of failure, and guarantee response time.


We are already seeing the adoption of open source across industries including financial service organizations that were deemed the slowest to adopt open source. I believe that, in future, the demand for rapid digitization and cloud adoption will only accelerate the demand for and adoption of open source. Open source adoption is going to accelerate and organizations should assess how and where they can leverage it for competitive advantage.


Subscribe to this blog's feed

Follow us on

Blogger Profiles

Infosys on Twitter