ESP – Product Engineering – Platform and Software Products

True Integration with VSTS 2010

2010-02-18T05:11:16Z

Microsoft (India) conducted the "VSTS 2010 Launch" session at the Infosys Bangalore campus this week to familiarize the software community with VSTS 2010 before the official launch (planned in April, 2010). The session included a keynote followed by an intensive hands-on demonstration of the various facilities that are on offer in the latest edition of Visual Studio Team System Suite. There were a number of features on demo, but what was of special importance was the support for the architect community in this release.

]]> The Visual Studio environment, in all its various avatars over the years, has always been known to be an IDE (Integrated Development Environment) - but as a tool the focus has been more on its use as a 'Development Environment' used primarily by the developer community. The term 'Integrated' seemed to be a misfit with regard to how Visual Studio was generally used. However, with the VSTS 2005 suite Microsoft had signaled its intentions of developing Visual Studio into a complete Application Lifecycle Management solution providing tools for everyone involved in the lifecycle - be it the project manager, the architect, the developer or the tester. The Team Foundation Server along with its assorted client side solutions helped achieve this to a large extent. Microsoft’s new VSTS 2010 release builds on the same platform and provides a lot more. A lot of what VSTS 2010 promises has to do with aligning to current technology trends (APIs for multicore programming, cloud computing etc.) and business needs (cutting costs by introducing more avenues for automation). Though you may argue that this has been true for the various Visual Studio releases over the decade, with VSTS 2010 this seem to have been a foremost thought considering that it was a ‘work under progress’ during the year of the 'Great IT Depression' - 2009.

There are a number of invaluable features built into VSTS 2010, but what is heartening is the importance that has been accorded to the architect community in this release. Microsoft has realized the need to support UML within VSTS acknowledging that UML is a hugely popular modeling language. As a result, one will no longer have to look at using other detached tools like Microsoft’s Visio or tools from Rational (Rational Software Architect for example) to model architecture and class diagrams - though the support for DSL (Domain Specific Language) is still retained in VSTS 2010.Significantly, VSTS 2010 also provides facilities to generate source code from UML diagrams and vice versa - a feature that has been considered an important USP of other like tools. VSTS 2010 allows a ‘work item’ or task (from the project management perspective) to be automatically generated from a use case diagram - for example, and assigned to a particular developer – an example of the integration of an ‘architect’ responsibility with a ‘project management’ responsibility.

With VSTS 2010, a lot of importance has been accorded to support maintenance/reverse engineering. Other than the ability to generate UML diagrams (sequence diagrams etc.) from code, VSTS2010 also provides a facility to generate dependency graphs based on assemblies, namespaces, classes etc. Dependency graphs help understand the dependencies between assemblies. In addition, it goes beyond to show details of the exact dependency in terms of the function call being made, the class object being instantiated across dependencies etc. This is of great importance today with a number of customers wanting to re-engineer legacy applications to modern technologies but having absolutely no relevant documentation in place to guide in the initial understanding. A recent project involved the need to re-engineer a native VB/C++ based application to .NET with nothing but the source code made available. In a world where time-to-market is all important, such projects do not involve a complete migration of the application. Rather, complex algorithmic engines are maintained ‘as-is’ in the native code base with the remaining parts of the application migrated to latest technologies to provide a more modern perspective to the end user. (Such native implementations are typically invoked using facilities available in the platform -JNI for Java, interop for .NET) The availability of dependency graphs helps understand the various components and the interfaces to those components without spending time on code reading.

Another neat feature provides the facility to allow the architect to monitor if the source code being developed is on the lines of the architectural decisions that had been written down. For example, in a 3-tiered architecture, the normal rule is for the presentation layer to access the data layer only through the business layer APIs, but during development it is quite normal that such rules are ignored under pressure or as a result of bug fixes resulting in code which works but defies architectural decisions. Such deviations which break architectural rules will result in compile time violations in the VSTS 2010 Error List, thus allowing corrections early in the development life cycles without the reviewer having to point this out. According to Somasegar in his blog here - "The Layer designer enables you to define the logical layers and valid communication paths between layers of your project. Once you have associated assemblies, namespaces, and classes with layers in the Layer diagram, you can validate existing or new code against the layering constraints."

While wholesome support for the Architect community is a novel addition, there are a number of features supporting configuration management and the build process. In addition, Microsoft has deeply focused on testing by providing a suite of facilities to bridge gaps between the developer and the tester. A bug "that manifests on the testers machine but is not reproducible on the developer's machine" is a common incident that repeats in projects across companies. Using VSTS 2010, a tester can now save the environment in which the bug had manifested as a virtual machine instance (using virtualization facilities provided on the OS platform) for the developer to reproduce and investigate better.

To a critic, the sheer number of features available might seem be overkill besides of course, the enormous demands on system RAM requirements when installing and using the complete suite. Microsoft has been making videos of various features available online, in addition to detailed documentation already being made available. It is necessary to install only those components of VSTS2010 that one would really be using to overcome system RAM demands.

VSTS 2010 is truly meant to be an "Integrated Development Environment”. So, go ahead – irrespective of what role you play in the project team - and get your hands dirty using VSTS 2010!

Globalization and the Japanese Software Industry

2010-02-03T06:27:51Z

Japanese products in the manufacturing sector have most often than not withstood intense competition from competitors all around the world. Except for the recent glitch, Japanese cars have been the most sought after the world over for various engineering attributes like performance, reliability, design etc. But, unfortunately, the same cannot be said about the Japanese software industry – though there are efforts currently underway to change the structure of this industry.]]> Tatsuo Tanaka in his paper “The Competitiveness of Japan’s Software Industry” attempts to clearly bring out reasons why Japan has not been able to lead the pack in the manner it does in the manufacturing industry. Tanaka’s research indicates that Japan excels in producing custom and embedded software, but lags when dealing with packaged business and online software.
Japan’s area of strength has always been hardware and considering that embedded software is incorporated in hardware is an obvious reason for its quality to match the hardware. Japan’s focus on custom software can be attributed to the reason that most Japanese company’s handover software development to subsidiaries to develop software for in-house use, and that ends up being customized with specific extensions. Besides, the tendency to form long term business partnerships with their customers result in software being developed customized to that particular customer. If there were indeed short term relationships, companies would try to generalize on the product that they developed so that the initial investment is not wasted and moulded for newer and newer customers, besides inculcating the use of standardized third-party software in the products that they develop. As the global market is more inclined towards business/general purpose software, companies in Japan which concentrate more on custom software are not in a position to compete in the software export market.

Software products offered by organizations in Japan are developed in bits and pieces (modules) by various subsidiaries who may in turn request the services of other first tier contractors who in turn may forward that to other contractors. Despite intense quality focus (as is the norm in Japan), the end product would contain software developed using varied technologies, processes and standards with pieces involving very less generalization.
The need is to develop a process model which is going to help make Japanese products viable globally – the technical aspect of which involves the strategies for internationalization and subsequent localization. To quote from a communication with an Infosys employee who has spent a major part of his career interfacing with Japanese customers – “The American and European markets have products which are widely used. The product companies in Japan too have their own products which are in the same space. However, they lack competiveness in terms of features. The problem is that most software products are mangled to meet Japan specific requirements like specific encoding types (SJIS, EUC etc), specific hardware platforms, specific cultural expectations etc. What is required is a methodology which can be used to guide these Japan specific products to compete globally with established products and at the same time maintain their very Japan characteristics- and this includes aspects like product positioning, differentiation etc in addition to internationalization and localization”

Considering the effort and investments that have already gone into making some of the products, and the fact that these products have done exceedingly well in the domestic market, Japanese customers typically would expect the effort to ‘globalize’ their software to be minute and there lies the challenge. Though such projects are often driven by a set of initial requirements, it is important to look beyond just what has been specified. The base requirement of "internationalization" would mean a lot of proactive contribution based on experience of how products work across continents, countries and cultures and there is a possibility that the current product would need a "re-architecting" to achieve this flexibility. With years of experience in developing customized software for the domestic market, it is important that a good case study is prepared in a way that could convince the customer of the need to adapt to the change. One related challenge is to get customers to adapt to new technology. In a product that we were expected to globalize a couple of years ago, it required a lot of effort to convince the customer of the need to use the latest technologies available with .NET. The customer insisted on using C++/MFC and Visual Basic simply because most of the company's resources worked on these technologies in supporting their other legacy customized products. Their C++ based products having been highly successful in the domestic market, it required a lot of convincing to explain the advantages of using .NET technologies in the context of this product and the global market.

There seems to be a wind of change blowing across the Japanese software industry and a realization that custom products for the domestic market will not provide a wide market share in the face of alternatives available with software from companies out of Japan, whose products have modern features. Such companies are able to access a wider market across geographies because I18N is an inherent characteristic of their software thus allowing them to build on their initial investment. In the face of the changing world it has become inevitable that the Japanese software industry is now retrospecting on how it has to position itself in the global market. We have often interacted with Japanese Fortune 500 companies who want us to contribute to help place their software in the global market and such requests are increasing ever year.

Internationalization and Performance considerations

2010-01-28T06:54:44Z

Almost always, during the design discussion of any Internationalization project, one of the questions asked by the client is, “So, will Internationalization have any impact on the performance of the application?”. No matter what you think, there is no denying the fact that Internationalization does have a performance impact on the application, whether it is big or small. There may be situations where the business benefits of Internationalization will outweigh the performance criteria and in such situations it makes sense to go ahead with Internationalization even at the cost of some amount of performance degradation. However a good design can help you in avoiding severe performance hits.

]]> In order to come up with a good design to minimize the performance impact, it is important to first understand the areas which contribute most towards performance degradation. Some of the areas are listed below,

Character set conversions:

During Internationalization, the application is designed to support a particular encoding which may be UTF-8, UTF-16 or any other encoding typically used. However this does not guarantee that the incoming data will be encoded using the same character set. In order to process the incoming data, the application has to first convert the data to support the same encoding as the application; else the string processing within the application will go haywire. In case of a networking application where huge amounts of data might reach the application from various kinds of devices, encoding conversions at the applications’ interfaces will definitely degrade performance. This will also happen if the application is expected to write its output to files having a different encoding scheme or pass data to third party applications which support a different encoding. In general, character set conversions- whether done at the application’s interfaces or within the application- contribute towards performance degradation.

Choosing an appropriate encoding for the application can minimize the impact due to character set conversions. If majority of the incoming data is in a particular encoding, it makes more sense for the application to support the same encoding in order to minimize the character set conversions at the interfaces. E.g. if the incoming data is UTF-8 or US-ASCII encoded, the application should internally support UTF-8 for better performance. If majority of the input data is in a native encoding like Shift-JIS or EUC, it would probably be better to support native encoding within the application, even though supporting Unicode might be the ideal choice. The choice of encoding also depends on many other factors, so a tradeoff is often required.

Memory requirements:

The memory requirements of an application generally change after it is internationalized. This impacts both secondary and primary memory. E.g. keeping all the message strings in separate properties files increases space usage on the hard disk as well as increases the access time for the messages. Also since all encodings have different memory requirements, the application may use more RAM considering the data structures will need more memory to store the internationalized data. E.g. for English and Western European languages, choosing UTF-16 over UTF-8 generally doubles the memory requirement while choosing UTF-32 uses 4 times more memory. While UTF-16 and UTF-32 make string processing easier to some extent, it is also important to consider the performance hit due to the added memory requirements. The tradeoff is generally between memory and ease of processing.

Choice of an appropriate encoding depends on the locales which are to be supported. It is observed that UTF-8 generally takes 50% less space than UTF-16 for English or Western European languages, but it might take 50% more space than UTF-16 for some Asian scripts like Chinese. So if the majority of the data is in a Western European language, choosing UTF-8 would be a better option. Moreover since UTF-8 is backward compatible with ASCII, it is easier to make internationalization changes. Similarly for far eastern locales, choosing UTF-16 would give a better performance since it will use almost 50% less space than UTF-8. By choosing the appropriate encoding, the memory requirements can be optimized which leads to better performance.

Message catalogs:

Typically during the process of Internationalization, all user visible strings are moved into a message catalog or resource file. The application has to retrieve the required strings from the message catalog using the corresponding string IDs. This loading of strings can happen during application start, which increases the loading time of the application or it might happen when the application is running, which decreases the response time of the application. In an application having a huge number of strings, this can contribute to performance degradation.

While creating message catalogs is an essential part of Internationalization, the application can be designed in such a way so as to minimize the performance impact. Instead of loading the entire message catalog when the application starts, the application can be designed to load only the essential strings at load time and load the rest as and when the need arises. Along with this a message caching mechanism can be implemented to enable faster access to frequently used message strings.

Sorting multilingual strings:

Sorting of strings in an application is a very common feature and does have a performance impact in a Unicode environment due to the difference between Unicode sorting rules and Non-Unicode sorting rules.

Though this impact is not very significant in most cases, the use of proper Collations and Collation keys can improve the performance of the application. This can be done at the application level (as in Java) or even at database level. The collation you choose can significantly impact the performance of queries in the database. Collation also impacts substring matching in queries. A collation can be chosen for the quickest possible performance or for the most accurate results. Both have their pros and cons. If you want accuracy, you can choose to go with the Unicode Collation Algorithm, but it will have some performance overhead.

In general reducing the amount of encoding conversions and string formatting can help in minimizing the performance impact due to Internationalization. The choice of an appropriate encoding also plays a very important role. In the end it is always a tradeoff between the performance and ease of processing. What are your views on this subject and how does your design team deal with it? It would be good to share some best practices for handling these issues.

Handling Data in Enterprise Mashups

2010-01-27T14:12:23Z

Mashups are always ever-green, hence gets the attention from all the stakeholders, be it a creator of the mashup or the user of the mashup. Thanks to Google Maps which has taken the popularity to next level. A Typical mashup application comprises of a web application that combines data or functionality from two or more external sources to create a new service. The term Mashup implies easy, fast integration, frequently using open APIs and data sources. An example of a mashup is the use of cartographic data to add location information to real estate data, thereby creating a new and distinct web API that was not originally provided by either source. These mashups have also got its foot into enterprise business and the termed coined is “Enterprise Mashups”. Here in addition to just data the process also comes into picture. If the enterprise is SOA enabled then we can directly use the BPM engine for process orchestration. Enterprise Mash up consists of:

Web services
RSS Feeds
Platform services in a cloud
Data
Client Application

In this blog, I will quickly touch upon on Data part of the mash-ups. Data in Enterprise Mashups can be in the form of:

XML data residing in RSS feeds or in webservices.
DB data
Unstructured data
JSON

In Mashups the processing of data is a dynamic activity hence the time taken to process the data may increase the overall execution of the mashup application. To tackle this problem distributed computing can be applied on different kinds of data as mentioned above.

For XML and JSON data, the parallel parsers can be used to create the Mash up. This could be multithreaded or use Multicore architecture of Intel chip at hardware level http://www.intel.com/cd/software/products/asmo-na/eng/406212.htm. On other hand we can use hadoop’s HDFS and MapReduce for un-structured data.
Hadoop is a framework based on java that supports distributed computing scale very well for data intensive applications. Hadoop Distributed File System (HDFS) is the primary storage system used by Hadoop applications. HDFS creates multiple replicas of data blocks and distributes them on compute nodes throughout a cluster to enable reliable, extremely rapid computations. MapReduce is a programming model and software framework for writing applications that rapidly process vast amounts of data in parallel on large clusters of compute nodes http://wiki.apache.org/hadoop/. One good example of an enterprise mashup is “CRM-gadget” http://www.programmableweb.com/tag/enterprise , which searches new account or validate accounts on oracle on demand over Google local search. This mashup can tap the potential of Hadoop HDFS and Mapreduce and reduce the time to search the accounts.

To conclude, we need to build POCs and see the dynamic dissection/split of data on parallel/distributed nodes to achieve almost linear speed-up. This will in-turn reduce the total time of executing an Enterprise Mashup application.

Google File System

2010-01-24T10:29:49Z

The Google File System (GFS) is a scalable distributed file system designed and developed by Google for distributed data intensive applications. GFS was born out of the need to meet the rapidly growing data processing needs of Google. The design of the GFS shared many of the same goals (e.g. concurrency, scalability, availability and reliability) as previous distributed file systems, but differed from earlier file systems to meet the demands of application workloads and technological environment at Google. Almost a decade later, most of Google’s applications rely on GFS to store and process data. Although Google has not published the GFS code, the design of GFS is discussed in detail, in a paper (titled “The Google File System”) published by Google engineers. To explore more about the design of GFS, one needs to read the original paper present at http://labs.google.com/papers/gfs.html.

]]> During early days, engineers at Google felt that the existing distributed file systems could not satisfy the demands of their applications. Hence they decided to design a new file system. Several key assumptions guided the architecture of GFS. Some of the assumptions were:

Google uses thousands of storage machines built from cheap commodity hardware (typical Linux machines) and any of these machines could fail and never recover from failures. Hence the file system had to incorporate monitoring, error detection and recovery mechanisms.
Google’s Web applications generate and consume files of sizes varying from few hundred megabytes to several terabytes. Files with small size were almost non-existent. For e.g. web crawlers employed by Google’s Search Engine, continuously scan internet and store information related to millions of web pages. Hence they needed a file system which could handle huge blocks of data.
Most of operations on the files involved either large streaming reads (very few random reads) or large sequential appends to the end of file. Random writes within the file were almost non-existent. In large streaming reads, clients typically read 1MB or more of data. Overall they needed a file system optimized for reading and writing huge chunks of data in streaming mode.
The files were often used as producer-consumer queues, with multiple producers writing into the same file. Hence the file system had to provide APIs to support concurrent appends to files, with minimum synchronization overhead.
During early days, Google had only search engine, for which huge amount of data was processed in the background (e.g. generating inverted indexes for web pages). At that time, they did not have any user facing web applications like Gmail or Youtube which are sensitive to latency (i.e. require low latency). Hence they needed a file system tailored for batch oriented operations (higher throughput) than for latency oriented operations (lower latency)

With the assumptions listed above, they designed and developed a file system which consisted of mainly 3 components:

A master server for maintaining the file system metadata. There would be one active master per cluster.
Chunk servers for storing the actual chunks of data. Each file is divided into several chunks and each chunk is of size 64 MB. And each chunk is replicated at least 3 times. For e.g. let’s assume that a file “a.txt” has chunks c1, c2 and c3. Each of these chunks will have at least 2 more replicas, e.g. c1’ and c1’’, c2’ and c2’’, c3’ and c3’’. These chunks are usually placed on different machines in order to ensure availability in case of machine failures. Users can override this default replication factor of 3 and specify their own replication factor for each of the files.
GFS client, which will be used by applications for reading, writing or deleting data. GFS client provides APIs like create, delete, open, close, read and write. Apart from these standard APIs, GFS provides snapshot API for creating replicas and record append API for concurrent appends to a same file.

Master maintains the file system metadata. For e.g. file namespaces, mapping between file names to chunk locations. Chunk servers send regular heartbeat messages to the master indicating their health and changes in the chunk status (if any). For e.g. a chunk could get corrupted (this is determined using a checksum) or could have outdated data (outdated chunk is determined using chunk version number). Whenever a chunk server dies, all the chunks present on that chunk server need to be re-replicated. Master comes to know about the death of a chunk server if it does not receive a heartbeat message within a configured interval. Master places chunks in such a way that the data is distributed evenly across all the machines within a cluster.

Certain metadata, e.g. file namespaces and file to chunk mapping, is kept in persistent state on the master’s disk. In case of a crash, master recovers by reading the metadata stored on the disk. This data is also replicated to shadow masters at regular intervals of time. If master machine itself crashes and it is not possible to restart the master, then one of the shadow masters takes over.

GFS implements lazy garbage collection mechanism for removing the deleted data. Deleted files are not removed immediately. They are garbage collected at a later point of time. This helps in undoing accidental deletes, which could be costly considering the size of data.

Leasing mechanism is used to maintain data consistently across all the chunks. The GFS client has to obtain a lease on a chunk to do any data mutation on that chunk. Till the lease on that chunk expires, other clients cannot access that chunk for any data mutation. Any mutation to a chunk, is replicated to all the chunk replicas and the mutations are applied in a consistent order to all the replicas. For e.g. if data blocks A, B and C are written to primary chunk c1, then secondary chunks c1’ and c1’’ also get the data in the same order, i.e. A, B and C. This ensures data consistency on all the chunks.

Application code is linked with GFS client library. For any operation, client first contacts the master for getting the chunk location and lease on that chunk (in case of mutations). Once the chunk location is obtained, the client directly contacts the chunk servers to read, write or delete the data (by bypassing the master).

Google’s publications on GFS and MapReduce (a programming model for distributed data processing) have inspired an open source project named Hadoop (http://wiki.apache.org/hadoop/HDFS?action=show&redirect=DFS). If you want to explore Hadoop, check: http://hadoop.apache.org/.

Exponential growth of internet and proportionate growth in data has exposed some of the drawbacks of GFS. This has prompted Google to rethink on some of the initial design decisions. Some of the drawbacks of earlier system are:

It was designed mainly for batch centric applications, i.e. the applications which need to process huge amount of data in batch mode and are not sensitive to latency. With Google Search Engine becoming immensely popular, Google added other applications like Gmail, Youtube etc, which are sensitive to latency. Hence if these applications were to use GFS, certain adjustments had to be made to the file system.
To simplify the design, GFS was implemented with a single master node, which maintains the file system metadata for entire cluster. By initial estimates, GFS was expected to handle few million files with sizes up to few terabytes. But the demands for data grew from terabytes to petabytes. This increased the size of metadata maintained by the single master. This in turn increased the processing time at master node and limited the number of client requests that a master can handle within a specified period of time.

Over the years, some of these drawbacks have been managed by tweaking the file system or tweaking the applications which used this file system. Engineers at Google have been working on a new distributed master system (as opposed to single master design) to solve some of the problems of GFS. If you are interested in knowing how the file system has evolved over the years, you can check this recently published ACM link: http://queue.acm.org/detail.cfm?id=1594206.

Is Big Bang the right approach to Internationalization?

2010-01-11T09:12:11Z

Over the years our project teams have matured in the way they handle the implementation of an Internationalization project, however things were not always so smooth. There were times when the project was tested and delivered to the client, but it refused to work on the client’s machines. The offshore team just couldn’t figure out the reason for this to happen. A lot of fire fighting effort was then required to get things back on track and take corrective actions. Most of the problems were due to wrong planning, lack of technical understanding and incorrect assumptions. Things are pretty much streamlined now with an i18n Center of Excellence (CoE), i18n frameworks, analysis tools, POC’s and best practices in place. Here I am going to recollect my earliest Internationalization experience and what we learnt from it.

]]> Almost a decade back, during one of our assignments we were engaged with a Japanese client. They had an English product which they wanted us to internationalize and subsequently localize to Japanese. Internationalization was a known concept around that time but we did not have adequate practical experience with such work. We had a team of people who were familiar with the concept of Localization and they were brought into the team. Some more relatively lesser experienced people were also added to the project. The product was written in Visual C++ and so the objective of choosing the team was to get people with adequate understanding of C++ and train them on Internationalization and Localization concepts.

The requirements were gathered, process documents were created and the team came up with the implementation and release plan. As with all Japanese projects, time to release was a critical factor and the offshore team did not have much time to ramp up their i18n skills. At the concept level the team had an understanding that anything that is shown in English on the UI must now be shown in Japanese. So the approach was to find all the hard coded strings in the source code and move them to an external resource file. Secondly since native C++ functions and data types do not have support for Unicode, they had to be replaced with their wide char equivalents. This means a ‘char’ variable should be replaced with ‘wchar_t’ and functions like ‘strcpy’ should be replaced with ‘wcscpy’. None of the team members had an understanding of the repercussions of making these changes and since time was ticking away like a bomb, it was decided to follow the Big Bang approach and do a find-replace on the entire source code since analyzing the data flow in the source code to find impacted areas would have taken too much time. Subsequently scripts were written to automate the whole process and substitute all the data types and functions with their wide char equivalents and substitute all hard coded strings with resource bundle calls. With the approach neatly lined up the team got busy making the substitutions and compiling the individual modules. Finally all the changes were complete and the source code was compiled. Since the Localization to Japanese was not yet done, the product was tested using the English resource files and everything worked as expected on all the offshore machines. The product was delivered right on time to the customer. It was now time to sit back and wait for the appreciation mails to flow in.

The customer installed the product on one of their Japanese machines and tried to launch it. The application crashed. No matter what combinations they tried the application refused to launch. The customer pressed the panic button. The offshore team could not figure out the reason for the crash. They got a machine with Japanese OS and tried running the application on it. It worked fine. After understanding the customer’s environment, it was decided to install the product in a folder having a Japanese name. The product failed to launch and crashed. The code was debugged and it was found that one of the replaced wide char functions was the culprit. Pointer arithmetic on data bytes was not modified to reflect the fact that a character could now be represented by multiple bytes; and so at some point this resulted in incorrect processing, corrupt data and eventually a crash. This happened as the team had followed the Big Bang approach and just replaced all the impacted functions with the wide char equivalents without analyzing the data processing logic. It is not just enough to use wide char functions; a thought has to be given to the usage as well. Subsequently an extension was sought and corrective measures were taken and the project was eventually delivered in perfect working condition for the Japanese environment. The initial approach had backfired and quite a few lessons were learnt from the experience,

Have the right team - Your team might comprise of people with 5+ years of experience, but when it comes to Internationalization, it is important to have a team which understands the concepts and technical aspects of Internationalization. It will shorten the development cycle and the end product will have lesser defects.
Have the right processes in place - A Big Bang approach is always dangerous to start off with. A more mature implementation methodology is required. Checklists must be in place to ensure that when a particular change is made; all other changes related to that change are also dealt with. Internationalization changes can have cascading effects on other areas of the code. Changes should be done module wise or feature wise so that defects are caught earlier and in a localized manner instead of taking the Big Bang approach and messing up the entire code.
Analysis is more important than development - It is very important to have a team of experts who will analyze the source code to find all areas which need to be modified to support Unicode. It is quite possible that some functions and data types need no change because they will not be handling any Unicode data. In such cases replacing them with their wide char equivalents is an overhead and could contribute to a performance hit. It is also important to understand the data flow in the application so that the required changes can be done in the code to handle encoding conversions etc in the functions or external interfaces. The memory usage of the application also increases when you support Unicode, hence the code must be analyzed to increase memory allocations only in the impacted areas. The Big Bang approach doesn’t check for all these things and it mostly leads to bloated code which uses more memory than desired and under-performs at runtime.
Use the right Tools - Using the right set of tools during development can speed up the development process. There are a lot of commercial tools available in the market which can help in static analysis of the source code. Infosys has developed a set of in-house tools for Internationalization and Localization. Among other features in the tool set, it helps reduction in analysis time by auto-detecting all areas in the source code where i18n changes are possibly required. It can also help later in assessing the i18n readiness of the product. However it should be kept in mind that tools are not a substitute for experienced people. While they can help increase productivity, the developers should still have an understanding of the i18n concepts in order to interpret the output of the tools correctly.
Do not make assumptions regarding the input data - In the scenario above, the team assumed that since the product was working with English inputs, it should also work with Japanese inputs. It is wrong to make such assumptions. A Japanese user can input filenames in Japanese or try saving a file in a folder with a Japanese name. The code should anticipate such use-cases.
Have the right test environment - Just because there is no language translation expert in the team, it is inadequate to test the product with English data. This will definitely spring some nasty surprises later when the product is deployed in a pure Japanese environment. You should either plan for localization at the time of testing or use alternate approaches like Pseudo-localization and make sure the product is tested with Japanese strings as well.

The Big Bang approach is similar to cooking a dish by mixing all the ingredients into the pan at the same time. The outcome is unpredictable and in most cases will not get you the desired result. It is better to follow a systematic approach which will guarantee success as well as allow you to take corrective actions as and when something appears to be going wrong, rather than waiting for disaster to happen and start cooking all over again.

Deciding Optimal Unicode Solution for Globalization Database

2009-12-31T10:31:25Z

The concept of Globalization and the estimation model has been explained very well by Aviraj Singh in his post Effort estimation for a Globalization project. Being a database person I always look at it from a different perspective, giving a bit extra weightage to database. There are lots of granular intricacies that one has to think of before deciding the solution for supporting Unicode data in databases. It can be achieved though Unicode database i.e. Upgrading database character set to one that supports UTF-8 encoded characters as SQL datatypes like CHAR/VARCHAR2 etc. Another option can be Unicode Datatype i.e. to support multilingual data only for certain columns by using Unicode national character set to store multilingual data in SQL NCHAR datatype attributes , without making any changes to database character set. The most confusing and key decision for Globalization project is whether one should opt for Unicode database or Unicode data types for supporting multiple languages in database. This is a key decision for the success of any Globalization project and will also have a considerable impact on effort estimations. An incorrect choice at this stage can lead to a lot of rework and end hour surprises.

]]> It is always better to clarify business and technical requirements especially as regards the need for globalization , details of languages/geographies to be supported , the size of database and data distribution , application downtime available for upgrade(for existing application) , application code language. In addition, one should also understand the future business growth plan and geographies to be supported. Once specific details are available, a team of experts should study the requirements closely and brainstorm to arrive at an optimal Unicode solution to be implemented. Following are few key things that should be considered for decision making:

Ø Business Requirements

o Do I need to provide multilingual support for existing database or need to create it from scratch? Which languages do I need to support?

o How large is the existing database? If an incremental upgrade is required, then Unicode data type may be the better option. In case it needs a complete overhaul in a big bang, then changing database character set may be the better option.

o Application/Database downtime and future business requirements may also be factors that influence the final decision. Examples could be as regards the languages that application may require to support in near future etc.

Ø Business Data

o Type of data

Does the application needs to support Asian, European etc languages; do these languages have supplementary characters? This will help in deciding optimal Unicode encoding. For example, UTF-16 provides more compact storage for Asian languages whereas European scripts are more efficient with UTF-8 encoding.

o Data distribution

If multilingual fields are distributed across all databases, it’s better to go for Unicode database than Unicode data types.

o Data volume and application downtime

If the data to be migrated is huge and the application downtime does not allow for a big bang migration, it’s always better to opt for an incremental upgrade with Unicode datatypes; especially for a existing DB where database character set (say WE8ISO8859P1) is not a subset of any UTF-8 encoding. In this case database character set upgrade will require additional overhead for converting data from existing database character set to Unicode character set.

o Binary data ( BLOBS and CLOBS)

If there is a requirement to store different types of multilingual documents and search their content in BLOB data types, one should go for a Unicode database. BLOB data is converted into the database character set before being indexed. Hence, if your database character set is non-Unicode then there will be data loss if the documents contain characters that cannot be converted to the database character set.

Ø Performance

o A Unicode database comes with a performance overhead due to the non-optimal use of data storage (depending on language and database/national character set compatibility) and conversion that may be required before storing data in database. If feasible, it may be better to store multilingual (Unicode) data in NCHAR/NVARCHAR data types, without changing the database character set.

Ø Application code

o Application code too plays an important role in deciding the suitable optimal Unicode solution. For example VC /VC++ applications on MS windows may perform better with Unicode datatypes as the data lengths of wchar_t buffer in VC/VC++ match the length of SQL NCHAR data types in database. This will make data comparison more efficient and may avoid buffer overflow in client applications.

One can also consider selecting combination of Unicode database and Unicode datatypes depending on project requirements. This can be a ideal situation where the database character set (US7ASCII) of the existing database is an exact sub-set of Unicode database character set (AL32UTF8) and you have Java application code running on Windows. Both Java and Windows being better compatible with NCHAR data types of UTF-16 encoding will give better performance and will be easy to manage so national character set may be set to AL16UTF16. Database character set upgrade to superset Unicode character set will also be quite easier and faster as this will not require any conversions.

Unicode solution has a huge impact on design, implementation approach and hence has an impact on the effort estimates of a globalization project. Though it is bit difficult to generalize the globalization effort estimation framework as scope and intensity of application/database code changes will be largely driven by business requirements, it will still be better to focus on areas discussed above in the initial stage and then review the project estimates and implementation strategy accordingly.

I have tried to cover most of key focus areas for driving Unicode solution for globalization database. Any other thoughts on this are most welcome.

Google Public DNS

2009-12-30T13:37:32Z

A month ago, Google announced the release of Google Public DNS (Domain Name System), which is a free DNS resolution service. DNS is used to translate human friendly computer names into IP addresses. When a user types the name of a website, the Domain Name Servers convert this name into an IP address, and this IP address is used by your machine to send requests. A DNS network contains a set of servers which maintain a cache of domain name to IP address mappings. Usually these Domain Name Servers are maintained by your Internet Service Providers (ISP). With Public DNS service, Google wants to provide an alternative to your ISP’s service. Public DNS leverages the existing infrastructure used by Google’s search engine, which uses crawlers to scan through millions of websites. The DNS information cached by these web crawlers is used by Public DNS. Already a company by name Open DNS offers a similar popular DNS resolution service.

]]> These DNS services claim to provide faster (by caching relevant DNS information and hence speeding up page retrieval) and safer (preventing spoofing and denial of service (DoS) attacks) service as compared to your ISPs.

Delay in loading a webpage could be caused by factors like geographical distance between the client and resolving servers (which could result in longer round trip time, or loss in packets due to network congestion etc.), cache misses (in this case, a resolving server does not have information about the requested domain name and needs to recursively query other servers to get the information) and heavy load on resolving servers due to under provisioning of servers or denial of service attacks (deliberate overloading of servers by malicious users, to deny service to genuine users). Public DNS claims to mitigate these delays with following approaches:

1. Adequate provisioning of servers to handle both the genuine requests and denial of service attacks.

2. Usually DNS lookup queries are load balanced amongst several name resolving servers. If there is over provisioning of resolving servers (as described in point 1, over provisioning is necessary to prevent DoS attacks) and if the load balancer randomly selects the servers, it could result in different servers having entirely different set of cached information (fragmented cache). This results in high percentage of cache misses and hence increased traffic between the servers, especially for popular domain names (remember that whenever a server cannot find the requested information in its cache, it has to query other servers). Public DNS handles this problem by splitting servers into 2 categories. One category of servers uses a global cache which contains popular domain names (e.g. Google.com). Since popular names are requested frequently, this global cache remains refreshed at all the times, resulting in quicker resolution. Other category of servers uses a local cache (i.e. each server maintains its own cache), which caches less popular domain names. Since these less popular domain names are requested infrequently, cache misses will not result in increased network traffic. But to service these less popular domain names as efficiently as popular domain names, Public DNS optimizes the request resolution by always forwarding requests for a domain name to the same server. For e.g. if the request is for www.indya.com, it is always forwarded to server A. If the request is for www.sify.com, it is always forwarded to server B. So, if user requests www.indya.com repeatedly, the cached information at server A would result in quicker resolution.

3. To ensure faster resolution of domain names, Public DNS pre-fetches and periodically refreshes the names irrespective of whether user requests these names. This is implemented using an offline component which periodically selects and ranks the domain names based on factors like popularity and hit rate (number of times it is requested). Another runtime component resolves these pre-fetched names and refreshes them based on their time to live value. This ensures that frequently requested domain names are served quickly (even if they are not universally popular domain names like www.google.com).

4. Google hosts Public DNS in its data centers across the world and routes the requests to the geographically closer mirror sites (e.g. google.co.in for requests from India), thus resulting in faster browsing experience.

Another consideration for a DNS service is security. DNS servers could become targets of spoofing (redirect users to malicious sites) and denial of service (DoS) attacks. Public DNS has implemented following approaches to prevent above mentioned security threats:

1. To prevent the DoS attacks:

a. Public DNS enforces rate control over the amount of traffic that could be directed to other name servers. Thus it will not be possible for attackers to flood name servers with high volume of malicious traffic. The rate control is also enforced on the responses that are sent back.

b. To prevent amplification attacks (amplification attacks exploit high response to request ratio of name servers. Attackers can inject large responses into name server’s cache, thus flooding the network with traffic), the response traffic is limited by applying “maximum average amplification factor” to each client IP.

If requests/responses exceed any of the above mentioned parameters, the error is returned. In some cases, no response is sent for such requests.

2. To prevent cache poisoning, basic validity checks, like rejecting the malformed responses or responses which don’t match the attributes of the requests (e.g. source IP, port), are enforced.

3. To make it difficult for the attackers to easily predict and match a combination of name servers, ports and query names, these attributes are randomized. For e.g. the requests are sent out on different port numbers and to different name servers (not to the nearest name server always) to add some unpredictability to the requests. Also, the cases in the queried domain names are changed to prevent forged responses. For e.g. wwW.gooGLE.com or WwW.gOoGlE.cOm.

4. To prevent attackers from injecting multiple duplicate requests for the same name resolution, Public DNS does not allow more than one request with same query attributes (port number, destination IP).

If you want to try out Public DNS, follow the instructions mentioned at: http://code.google.com/speed/public-dns/docs/using.html

To try out free basic version of Open DNS, check http://www.opendns.com/start/.

Don’t think local, think locale

2009-12-11T12:25:24Z

Imagine yourself going to Japan to open a restaurant. Your market research says that your burgers are going to sell like hot cakes there, so you have planned a major investment there and drawn up plans for expansions. You land at the Narita airport and are absolutely clueless on how to get out of there. You look around and find that all directions and signs in Japanese. You try to ask for directions but all you get is blank stares because no one understands English. Somehow you manage to find your way out and get busy with your work. After a lot of hard work, you finally open your restaurant but you don’t find many people walking in. Your business goes dry and it’s difficult to survive with so much local competition around. What is really going wrong? Didn’t your market research say that you are bound to succeed?

]]> This is a big dilemma for a lot of entrepreneurs when they try to enter emerging markets. You need to cross the language and cultural barrier in order to succeed beyond your neighborhood. If you open a restaurant in Japan, you have to ensure that your menu is customized for their tastes. You have to ensure that you have a menu in Japanese as well. All posters and signboards inside or outside your restaurant must be in Japanese, else how will the Japanese people know what you are trying to sell? Mc Donald’s sells their burgers in many countries, but they have customized their burgers according to their target market. While you may find a vegetarian burger in India, you will probably not find it in Japan. Instead they have a teriyaki chicken burger in Japan which they don’t sell in Canada. Over the years companies such a Mc Donald’s, Microsoft, Apple, IBM etc have realized the importance of customizing their offerings for the global markets.

Localization is critical when entering new markets. Localization is more than just translation of your user interfaces of help documents. It also takes into consideration the cultural, legal, regulation issues etc. It makes sense to invest in the global markets only when you foresee an ROI from the opportunity. So which are the emerging markets in 2009 and beyond? Which geographies should you target to increase your revenues? There are the most common questions which come up and firms like Forrester and Gartner have extensive market research data to answer all these questions. A research by Byte Level Research says that non-English speakers will represent 79% of all the internet users by 2010. So which language will dominate the internet in future? German is currently the most popular language on the internet, but Spanish is expected to overtake it and Chinese (simplified) is quickly gaining ground. According to the World Intellectual Property Organization (WIPO) and the International Telecommunications Union (ITU), Chinese will outrank English as the most-used language on the internet. Today the number of online users from China and Europe far exceed those from the United States.

The numbers don’t lie. All these statistics have been generated by collecting information and data from hundreds of small to medium to large corporations across the globe. Many companies are expanding their already established businesses, into other geographies. New players into the market are already making expansion plans into other geographies. According to a Byte Level Research done in 2007; on an average 80% of the companies interviewed, see their competitors taking their business global. It is imperative for them to be pro-active in such a scenario and make plans for going global themselves. It’s a case of ‘Go global or perish’. Intel generates around 70% of their revenues from outside the US. Microsoft makes around one third of their revenue from outside the US. Google had already reached the 50% mark by 2008. As I have mentioned in one of my previous blogs, it’s not enough being the best in your neighborhood anymore. Don’t think local, think locale…

Green Computing and Virtualization

2009-12-09T11:55:20Z

While contemplating about the importance of virtualization in achieving green computing standards especially in organizations hosting data centers, I came across an interesting article here on how energy emissions from data centers can be used to warm homes in Scandinavian countries.

]]> The article mentions that in a typical data center only 40-45% of the energy is used in actual computing while the remaining is used in powering agents to cool these servers. Besides, data centers run by a search giant already seem to be using around 1% of the word's energy and their demands seem to be rising fast every year.

This is interesting information in the context of the fact that the world has finally woken up to the need to put their heads together to resolve issues related to global warming at the UN Climate Summit at Copenhagen, Denmark.

There is a general feeling that the summit highlights the importance that the world is conscious that technological advancements have contributed to increased emissions and the need of the hour is to bring in another set of eco-friendly technology advancements. Which means 'Virtualization' is a word that is going to be part of every day conversation in enterprises as the advantages of 'virtualizing' the data center are well known. As a by-product, the focus is also going to be on development of tools that are going to make the process of virtualizing the infrastructure and managing it much easier.

Testing Cycles and Product Stability

2009-12-08T09:49:15Z

Years of experience in software development have not helped reduce anxiety levels whenever a project enters the 'Testing' phase of the Software Development Life Cycle. It feels the same as one would feel when parents accompanied you to school to collect your academic results at the end of term examinations. There is always the anxiety of whether the output of the design and coding phase will be able to successfully sustain the test case bombardment. Besides, you would also be anxious to know if there are enough test cases to traverse all paths of the software while testing functionality and QoS parameters - so as to be confident that all loose ends are covered. An even more difficult pill to swallow is a situation where you realize that a number of your tests are failing, and you will have to get back to the customer with the bad news and request an extension. But even after that, how do you gurantee that your product will be defect free ? How do you gurantee that you have not introduced defects unknowingly while fixing the known ones ? Experts would ofcourse recommend a 'thorough peer code review' - but even after that, you would still need a 'tested and passed' certificate before the software is passed to the customer for his acceptance tests.

]]> I have often felt the need to be able to estimate upfront the number of test cycles that should be executed during the 'Testing' phase. What projects often do is estimate for two cycles of test as an approximation. In Test-Cycle 1, all the test cases are run and all the defects in the sofware are assumed to have been detected. Fix them. Test-Cycle 2 is run to make sure that all the defects detected in Test-Cycle 1 are fixed. It is quite common that some defects are not completely fixed, especially when the number of defects detected is large. In addition, new defects may have also crept in as a by-product of the defect fixes. Naturally, this would call for another test cycle, once the Test-Cycle 2 defects are fixed ..and you end up squeezing in a Test-Cycle 3 with the hope that Test-Cycle 3 would be defect-free. (By-product defects are often seen in GUI intensive products where for example, regression defects are common while controlling state of change of UI elements.)

One can relate the above situation to what Bruce Powell Douglas mentioned during his July 2009 visit to the Bangalore campus. With regard to a different context Bruce had said that - "The number of defects in the software is proportional to the number of defects that you know about" which means testing and quality has to be a continuous process.

Every project in it’s estimation phase has to dwell deeper into the possible ways to counter deficiencies in development during the SDLC. One such problem is to estimate the number of defects that would need to be ironed out during the various phases of the project and the effort required to counter them so that the goal of a zero defect product is achieved before the delivery to the customer. The measure is a method of checking if the output of the particular phase has been reviewed and tested and that it will contribute to the final quality goal.

Most leading companies have metrics with regard to software quality management which are created as a result of observing trends in defect data across projects in similar technologies having similar complexities. One such metric would be for example, the number of defects per KLOC of code that are expected to detected during the complete life cycle (In today's world of function point based estimation, this might be a little primitive).

Let us assume that, as per the expected standards, the total number of defects per KLOC for both GUI and non-GUI applications (developed in C or C++) could range between 15-20 defects. Considering the pessimistic choice of 16 defects per KLOC, for a given application of an estimated size 25 KLOC, the total number of defects that could be expected in the complete life cycle of the project is (25 *16 ) 400 defects.

The total number of defects estimated to be detected during the SDLC is divided across various phases of the project. The confidence that justice has been done to the review of the product and that the product is measuring up to required quality can be obtained by measuring defects at the end of the phase. Assuming a general spread of defects detected to be 30%:40%:30% across the Requirements and Design, Coding and Testing phases (please check your company metrics for the spread if any), 30% of the estimated defects still exists when the software enters the Component/Integration Testing phase.

The Component Testing or the Integration Testing phase is high on the list of defect detection stages simply because this integrates a number of independently developed modules – and is more or less the environment which the user is expected to use the application. This is the last stage of validation and verification – before the application passes hands outside the development team.

Ideally, the CT testing should be divided into cycles of CT to make sure that defects creeping in because of defect fixes - are caught and cleaned.

It is quite common to see three cycles of CT testing being planned for certain products. The first cycle should aim at detecting 60-65% of the defects estimated in the CT stage. Assuming that your defect estimate calculations reveal that there are still 120 defects still lingering in the software as you begin your CT, you should be detecting 120 in the first cycle of CT testing and 68 defects in the second cycle of CT testing. The third cycle of CT testing should be a more confirmation cycle to confirm that there are no regression defects.

But, is this approach enough to gurantee good software ? Time pressed, you are assuming and hoping that you would have neatly fixed the 68-70 odd defects that are lingering in the software at the end of the second cycle of CT thus allowing CT-3 to be more of a confirmation cycle - which is quite a risk!

A better approach would probably be to consider the "divide by two" approach to determining the number of cycles of CT. In this approach, the number of defects estimated to be lurking in the software at the start of CT, may be recursively divided by 2 (until the defects are single digit) to determine the number of cycles.

Hence, in this case – 120, 60, 30, 15, 7 are the number of defects expected in each cycle with the sixth cycle being a confirmation cycle. However, in this approach care must be taken to change the test cases being executed – and also to focus more on problem areas (considering the 80-20 rule) so as to not make testing redundant and wasted by repeatedly testing defect free areas.

The "divide by two" principle might not be acceptable in all situations - and it is understandably difficult to convince about its need in a development process which contains strong design and code review phases. But, it is a good method to forsee upfront how much time and effort you would need during your testing phase to provide an efficient and defect free product.

P.S - External Testing is a phase where the development team completes their CT testing and provides the application to an external team within the unit for independent testing – which will surely comprise unbiased testing. This actually means that even on receiving an OK from the development team, there could still be defects lingering in the system –hidden from familiar eyes-which can only manifest in the eyes of an unrelated tester. This team should not be in anyway involved in the design/development of the product. The test cases developed by this team should be based completely on the FS/FD.

Internationalization and the development life cycle

2009-11-25T08:10:02Z

As a product company your team has come up with a brilliant concept which has tremendous marketing potential in your country. Your marketing survey shows that the concept will soon catch up with other countries across the globe and you can capture the overseas market too. The only catch is that the product will be required to be globalized before it is launched in the international markets. A global launch is still 5-6 months away, so what product strategy will you adopt? Develop the product in English and later when you have access to the global markets, think of internationalizing it or start developing an internationalized version of the product right from conceptualization stage so that you are ready to penetrate the global markets when the time comes? This is a question most product managers will face while developing a product.

]]> The ideal time to start thinking about Internationalization or Localization of your product is at the conceptual stage of the product development life cycle. The product management has to be clear about their vision for the product. If the product is meant for international audiences, it is a good idea to plan for it earlier than later since it will definitely be more expensive to do the same thing later and you may also lose out on market share due to a delayed launch. Internationalization is often more important than localization in the development stage since localization will normally not involve code re-engineering whenever it is done. Think about a product which is not internationalized and you want to introduce multilingual support to it. Changes will have to be made in multiple layers in order to achieve this. These changes will be costly in terms of the time taken, bugs introduced and additional testing required. However if the product architecture had made the same provision at design time, things would have been much simpler and all the development team had to do was get the string resources translated in order to support a new language.

What does it take to internationalize your product right from requirements to design to development and finally testing? The requirements gathering team must understand the typical i18n requirements and they must evaluate the product requirements from an i18n perspective too. The design team must understand the typical i18n aspects and ensure that the product design takes care of all i18n issues along with the intended architecture and design. The development team should be experienced with making i18n related code changes and it is important that they understand the i18n best practices in order to avoid rework later. The development of the core components and internationalization must go hand in hand. Pseudo-localization testing must be planned for the product to identify potential localization issues. If these practices are followed, it is more or less assured that it will be easy to adapt the product to different regions or countries as and when required with minimum cost and time to market.

Trading off with Design Patterns

2009-11-24T10:37:51Z

Over the last decade or so, any queries as regards designing object oriented software systems would lead to one being advised to read the Go4's book on Design Patterns (Design Patterns: Elements of Reusable Object-Oriented Software). It is without doubt a wonderfully written book and should be in the possession of most software designers involved in the world of object oriented design. But, what happens when an over-enthusiastic reader ends up seeing patterns in every software problem he encounters ?

]]> I had recently come across the a wonderfully hilarious but true write up on when software designers and architects go overboard (http://www.joelonsoftware.com/articles/fog0000000018.html) trying hard to resolve a future problem which may or may not exist - thus compromising on what needs to be resolved NOW. I have been a victim of the same phenomenon - the obsession with "what if tomorrow .....". No, this is not a criticism of the need to look ahead - but it is just to drive home the point that it is important that one does not lose focus on what needs to be solved right now.

The same applies to the choice of design patterns. Over-enthusiastic designers end up creating a problem to implement a design pattern which they may find savy and also as a result of the "what if tomorrow ...." phenomenon. A manifest of this was a case where a product was redesigned to enhance maintainability. The end result was beautifully maintainable code but seemingly not performant enough to be launched in the market. The designers had looked far ahead to the problems of tomorrow to actually think about it's current deployment requirement.It is thus extremely important to 'trade off' between what a customer wants today as an immediate market requirement as against what he is willing to compromise on at a future time.

One needs to be extremely conscious when trying to fit a design solution or pattern to resolve a problem. The Singleton pattern is a very commonly used pattern - and admittedly one of the easiest to understand. You will not have to look too hard into the problem that you are trying to solve to find reasons to implement the Singleton pattern and lot of enthusiastic new designers will jump at the prospect of a chance to implement that chapter in the Go4 book. But, you have got to be sure that there is no thread-safety requirement that the customer has passed on without anyone realizing - which might end up making the 'Singleton' choice look amateurish.

Bruce Powel Douglass spoke at a session at the Infosys campus in Bangalore in July this year where he stressed on how architects and designers need to make a thorough analysis of the choice of design patterns and the best fit necessitated in the system that was under built. He called it 'the selection of patterns using design trade-off analysis'where he explained how the choice of design patterns needed to be weighed against the design criteria that the product was expected to achieve. For example, it was important to weigh the following typical design criteria against the needs of the product.

1. Worst case performance
2. Time to market
3. Memory
4. Reusability
5. Simplicity
6. Safety and Reliability

Now, of a choice of multiple design patterns, each design pattern needed to be rated as against how much it would help achieve each of the above design criteria to help provide some direction on the final choices. A nice elucidation of a structured approach to correct choices in design.

Though Bruce with reference to the world of embedded systems, this approach holds true no matter what system you intend to design. In today's world of instant solutions, doing this might seem slightly unwieldly but it is no doubt absolutely worth the effort.

So, go ahead and master the design patterns as espoused in so many books available, but remember to use it only as a tool to solve a problem that exists. Use it to solve a potential future problem when you are sure that it is not going to compromise the current solution.

Effort estimation for a Globalization project

2009-11-11T08:19:04Z

Effort estimation is the first step to undertaking any software project and a Globalization project is no different. Effort estimation for a product or application which needs to be Globalized follows more or less the same estimation principles as regular maintenance projects, yet there are no defined methods specifically for estimating the amount of I18N or L10N changes required. While working on the proposal for a Globalization project for one of our clients we were faced with the dilemma of adopting standard methodologies like SMC based estimation, FP based estimation etc or trying to create a hybrid and come up with our own estimation model which follows the same estimation principles but is more tailored for globalization projects. Finally we came up with a raw estimation model which was fine tuned over time and gave us estimates which were statistically inline with the results from other maintenance projects.

]]>

The first step to estimation is to understand the underlying product. Embarking on a project without complete information generally leads to disaster later. In the initial meetings with the client it is important to understand the current scope of the product. It will be useful to know the target geographies where the product is going to be sold, the current degree of internationalization if any, the platforms which need to be supported, the product architecture etc. Each requirement throws in more challenges in terms of estimation. The technical people involved in the estimation should have prior Globalization experience and understand the various I18N impact points in the code. They should be able to isolate code which needs I18N related changes with the rest of the code. Off course this is a very daunting task when the code base is huge, which is the typical scenario with a product; so we need tools and utilities which can find out all the impact points in the code. There are static analysis tools available which can do this to a certain degree. They can help in finding out the number of hard coded strings in the product, the number of non-Unicode API's and data types used etc and come out with reports which can be further analysed and used while estimation. At Infosys we use our in-house developed Internationalization tool which is rule based and helps in analysing code based on the specific set of rules that we set. This way the reports contain very relevant information which can be directly used in the estimation model.

At the time of estimation, it is important for the architect to decide on the encoding which will be supported by the product. This decision has a direct binding to the impact points in the code. In case the application has to support UTF-16, most of the API's and data types in a C++ application have to be replaced with their wide char equivalent, while if the application has to support UTF-8, only a handful of string related API's are impacted. The decision to use a particular encoding can prove to be very important since deciding to use a different encoding later at the implementation stage can prove to be very expensive and introduces risks in the quality and schedule of the project. Every encoding has its pros and cons and it must be well debated before going ahead with the decision. If there is database support in the product, the database layer should be analysed so that data that flows in and out of the database is in the required encoding. All internal and external interfaces of the application must be analysed so that the data flowing between modules or applications has an encoding which the communication layer can understand. The tools which help in estimation have a limited scope and the rest depends on the expertise of the person analysing the code and design documents.

The software estimation process breaks down the requirements into sub requirements which are made as granular as possible. At a very granular level if we know the number of API's or data types we need to change, we can roughly estimate the effort required to make those changes. If we know the third party tools the application interfaces with, we can estimate the effort required to internationalize the external interfaces or upgrade the third party tools to their Unicode supported version. A simple requirement like Unicode support for the UI translates to creating resource files for all locales, getting the number of strings which need to be externalized into those resource files, creating a library for reading and writing to the resource files etc. In this way we estimate at the very granular levels always taking into account our past experiences while making similar changes and the organization wide PCB (Process Capability Baseline) metrics. This estimation model is based on the bottom-up approach where estimates at the very root level finally add up to give the total development effort. To this we add the usual project management and testing efforts and come up with a final estimate.

The key to the whole estimation process is understanding the product and coming up with an exhaustive list of I18N impact areas and breaking them down into measurable entities which can be analysed manually or using tools. Like any other estimation process, this may or may not be very accurate, but after applying this to several Globalization projects, the model gets more and more well defined and the estimates are much more accurate. I am sure there are other estimation models people have experimented with while estimating effort for Globalization projects. It will be interesting to discuss alternate models and understand the pros and cons of each.

Embrace Parallelism with Virtual Machines

2009-11-10T14:21:15Z

Parallelism has until recently been a term associated with the world of high performance computing. Though humans have been endowed naturally with the ability to 'parallelize' worldly activities (one dangerous manifestation of which is the tendency to talk on the cell while driving your car), designing systems to embrace parallelism has always required that extra bit of mental effort. Software architects and designers have been spoilt by the scale up in processor speeds over the years thanks to Moore's Law. But, that's all changing at a very fast pace today. Today's hardware designs (multicore based processors) require software to be designed with parallelism built into the system for performance scaling over the next generation hardware (with the promise of multiplying cores) . Herb Sutter has very beautifully explained the reasons for the need for change in software design strategy in his famous article "The Free Lunch is Over" - published sometime in 2005.

Let us consider software system that is running on a given server on modern multicore hardware. The system is considered to be well scaled if it is able to spawn enough threads to keep all the available cores busy and hence speeds up execution (for now, let's ignore specifics like overhead of thread management as against the benefit of parallelized execution etc.). Is there a limit to the ability to scale here ? Yes and it is the number of cores that are available on the server. Ideally, you would want to be able to dynamically make more and more cores available to be able to consume the tasks that are parallelizable but waiting for a core to be freed - but then you are limited by the number of cores available.

Moving the above problem to a wider canvas on a larger scale (the cloud computing environment for example), let's understand that software that can scale across multiple servers exhibits parallelism in it's own way. In this context, the processing element is a 'server' as against a 'core'. Ofcourse, making more and more server machines available to feed the software's hunger for parallelism is easier said that done (considering the high costs of server procurement, server management overhead, power management etc.) - unless you have considered virtualization.

Virtualization solutions to such problems would involve having the ability to to distribute incoming requests across a number of virtual machines - instead of physical servers. Such solutions have the ability to generate additional processing elements (virtual machines) based on increasing workload so that all available parallelizable tasks are being catered to. Besides, with reduced workloads, unused virtual machines can be made dormant thus saving on power costs etc . Notice, the ability to truly scale efficiently is not hindered by processor configurations (as seen in the standalone software system illustration above). Considerations along these lines are probably important considerations in architecting cloud computing solutions.

The concept of a 'processing element' in parallel programming patterns- which was usually a processing core - can now be extended to include virtual machines too . With that, it becomes easier to relate common parallel design patterns like Task Decomposition, Data Decomposition etc. to solving large problems using virtual machines. Embracing parallelism with virtual machines is a reality today.

]]>