ESP – Product Engineering – Platform and Software Products

In one of the previous blogs (MapReduce), I discussed MapReduce framework, which can be used for efficient processing of  large data. Let’s assume that you have a need to process web scale data and you need a cluster of machines with MapReduce library installed on them. You don’t have to invest in buying a new set of hardware and setting up a cluster. Amazon’s Elastic Compute Cloud (EC2), supports Hadoop’s MapReduce implementation, as Amazon Elastic MapReduce web service (ElasticMapReduceDeveloerGuide). Like all other Amazon Web Services, this service is available on a pay-as-you-go basis. In this case Amazon’s Simple Storage Service (S3) is used as a data store instead of HDFS. Though HDFS is the default file system supported by Hadoop, it also provides interfaces to various other file systems including Amazon’s S3. To get started, you need to load your data and Mapper, Reducer executables into S3 and request Amazon Elastic MapReduce to start a new job flow. Elastic MapReduce starts a new EC2 cluster, which runs MapReduce on the data set uploaded by you and stores the resulting data in S3.

Continue reading "MapReduce in the Cloud" »

Posted by Manjunath Ballur at 10:57 AM | Permalink | Comments (0) | TrackBacks (0)

The pitch on cloud computing seems to be getting louder and louder. Smelling the business advantages of moving over to the cloud, enterprises are looking out for opportunities to reduce both capital and operational costs by embracing opportunities in the cloud. A fundamental objective of cloud computing is to allow enterprises to focus their energies on their core competencies by handling over non-regular but intensive operations like storage, infrastructure management etc. to the cloud. With cloud computing, the idea is to disconnect price from the cost of operating and acquiring technology and in turn, connect it to the core value offered as a service.

Continue reading "Hovering clouds in IT today" »

Posted by Suraj Nair at 3:48 AM | Permalink | Comments (0) | TrackBacks (0)

While contemplating about the importance of virtualization in achieving green computing standards especially in organizations hosting data centers, I came across an interesting article here on how energy emissions from data centers can be used to warm homes in Scandinavian countries.

Continue reading "Green Computing and Virtualization" »

Posted by Suraj Nair at 11:55 AM | Permalink | Comments (4) | TrackBacks (0)

Parallelism has until recently been a term associated with the world of high performance computing. Though humans have been endowed naturally with the ability to 'parallelize' worldly activities (one dangerous manifestation of which is the tendency to talk on the cell while driving your car), designing systems to embrace parallelism has always required that extra bit of mental effort.

Continue reading "Embrace Parallelism with Virtual Machines" »

Posted by Suraj Nair at 2:21 PM | Permalink | Comments (0) | TrackBacks (0)

When I first heard about Server Virtualization, my first thought was, “Why did nobody think about this earlier”. Of course, I later learnt that the idea had been around for more than 4 decades. But then, why did it go into hibernation?

IBM introduced the concept of virtualization at a time when computers didn’t mean anything other than mainframes. It was much later that x86 and the PC revolutionized the world. So while the computer became smaller, cheaper, faster and more accessible, the very need for Virtualization disappeared. After all, why have Virtual machines when you can have ‘real’ ones?

But the exponential growth in computing capabilities (hardware processing power) has today resulted in heavily underutilized machines. Couple this with the concerns over rising carbon emissions (power consumption and cooling needs) and the increasing complexity in managing numerous servers in the data center; …and the stage was set for Virtualized machines to make a comeback.

There was but one small problem.

Continue reading "Server Virtualization: Beyond Hypervisors." »

Posted by Randhir I B at 7:43 AM | Permalink | Comments (0) | TrackBacks (0)

The formal interpretation of Moore's law - as regards transistor divisions on a silicon chip - has been reaching its stretched limits. However, the advent of multicore based processor design has opened up a horizon of opportunities for those applications which are designed to exploit available hardware parallelism. However, existing applications are also expected to run faster as they get near dedicated attention on individual cores as against the case on a single processor single core machine where threads had to context-switch based on slicing.

Continue reading "The Multicore and Virtualization Mix" »

Posted by Suraj Nair at 10:53 AM | Permalink | Comments (0) | TrackBacks (0)

Is it just me, or is there a direct correlation between a technology’s position on the hype curve at any given point of time and the prevalent confusion around the terms that abound describing it? Virtualization, as IBM defined it in the 1960’s, seemed rather simple, but today it can mean a lot of things.

To someone who’s relatively new to the world of virtualization; some of the terms that have come to describe the various approaches to server virtualization can be pretty daunting if not outright confusing.

How is ‘Hosted Virtualization’ different from ‘Bare-metal Virtualization’? What then is ‘Paravirtualization’? What is a hypervisor? Is ‘Hardware Emulation’ some kind of virtualization technique? And which of these (if at all) can be considered as ‘Full Virtualization’?
Wait! Did I hear you say ‘hardware-assisted virtualization’ and ‘OS-assisted virtualization’?

So, are there really so many different approaches to server virtualization? 

Continue reading "Server Virtualization: Just how many types are there?" »

Posted by Randhir I B at 4:37 AM | Permalink | Comments (0) | TrackBacks (0)

The last blog on Virtualization provided what you could term a 'laundry listing' of a number of Virtualization types to reinforce the fact that  -'Virtualization means a lot of different things ...'. Another growing Virtualization concept is called 'Application Virtualization' and Microsoft's Application Virtualization Solution (App-V) is an interesting advancement of this virtualization type.

It is often the case that applications are designed (unknowingly) to cause a change in the state of the machine on which they run. Installation of applications often cause existing files to be overwritten, cause changes in common locations like the Program Files directory or affect the settings in registry locations like the COM registration database for example. The net result of such an application installation is that other applications that had been running well and fine start to complain. This happens because a new application installation mingles it's state with the state of the machine thus affecting existing applications. Despite you trying to install an application in a specific folder (during the process of installation), there is usually something that the application installation ends up doing in common locations like C:\Windows\System32 etc. (driver files etc.) which changes system state by overwriting/replacing configurations or files. This might be a necessity for the application to execute as per it's design.

Continue reading "Knowing Application Virtualization" »

Posted by Suraj Nair at 10:43 AM | Permalink | Comments (0) | TrackBacks (0)

The first time I ever heard of Virtualization as a computing technology term was when I was part of a team hosting a visit by a senior delegation of a reputed Japanese IT Fortune 500 company. During the course of the meeting which focused on a server side product to be launched in the global market, one of the guests lamented the complexity and costs involved in testing the product across varied combinations of operating systems, component software etc. Spontaneously, one of our colleagues shouted out -  ‘Virtualize your environment!’. This suggestion though was shot down politely as ‘this was not the real thing – and hence cannot be guaranteed to be as good as the real thing’. The concept of Virtualization has come a long way since then and today the same company prides itself in being pioneers in developing storage virtualization products. Virtualization in the industry itself means a lot of different things. Ideally, the term Virtualization would imply using a physical resource such as a server and dividing it up into virtual resources called Virtual Machines thus ensuring the maximum use of the available resources on the physical machine. But, over time, the term Virtualization has come to cover a lot many different concepts which finally happen to have the same aim – that is to consolidate physical resources, simplify deployment and administration and reduce data center costs. However, it would be nice to understand the various commonly prevalent types of Virtualization briefly in this post.

Continue reading "Virtualization means a lot of different things" »

Posted by Suraj Nair at 7:22 AM | Permalink | Comments (0) | TrackBacks (0)