Infosys’ blog on industry solutions, trends, business process transformation and global implementation in Oracle.

« Living IT transformation | Main | Innovation in Siebel »

Siebel Open UI with Dragon Speech Recognition

One of the key aspects of any leading organisation today is to create an equal and socially inclusive environment for their employees. So, when it comes to Assistive Technology (AT) users, especially in the public sector industry, providing them with the necessary support to carry out their day-to-day duties is of vital importance. This is where voice recognition software such as Dragon NaturallySpeaking comes into picture. It has been developed by Nuance Communications and runs on Microsoft Windows and macOS.

The Siebel CRM application, after migrating to Open UI, has provided businesses and developers with a much wider scope of customization. Since it's primarily built on jQuery, integrating with external application through open source code has become a possibility. In this blog post, I'll explain how the Siebel CRM application can be integrated with Dragon Speech Recognition software to enable execution of the various processes and activities within CRM through voice command and also how to create such custom commands within Siebel CRM through jQuery and also through Dragon macros.

The Requirement:

A client of ours who recently upgraded to Siebel IP15 had a requirement to have a voice enabled Siebel CRM application. They have an Assistive Technology user group and a certain section of it uses voice recognition software to perform their daily official activities. Since Siebel CRM is a core application for the client and with their long standing commitment towards equality and human rights, addressing this need is of vital importance. Although the client did try out the solution of enabling voice commands for the Siebel CRM application through Windows Speech Recognition (WSR), achieving seamless speech detection turned out to be a distant dream and it also was not able to reach the different processes within CRM as creating custom command in WSR for Siebel CRM had its own limitations.

The Challenge:

Dragon software doesn't work adequately with Siebel IP15 on it's out of box setup. This is because:

·         Dragon is created to tag/access a limited set of HTML elements through a very specific set of attributes.

·         Dragon then makes use of those HTML attributes to flag the elements through commands created using macros.

·         The Out-of-box Siebel application, however, has a structure which is built by making use of a vast set of HTML elements, designed to be accessed through keyboard and mouse, but not by voice commands.

·         So, by default, Siebel doesn't have most of the HTML attributes required by Dragon to flag elements and a lot of the elements in Siebel Open UI are not supported in Dragon.

The Solution:

In order to make Dragon work, we have to customize Siebel Open UI through JavaScript (.js) files, to support and allow Dragon macros to flag the different elements of Siebel. This would enable users to access all different functionalities of Siebel which are currently not supported in IP15.

Dragon provides the necessary guidelines to enable an HTML page to function well with its program. I will not get into the details in regards to how and why an HTML element needs to be constructed in dragon in a specific fashion, as you can find the detailed documentation regarding the same in Dragon's website. The basic idea is that Dragon commands can associate to specific HTML element types which are:

  • Anchor Elements
  • Image & Imagemap Links
  • Buttons
  • Edit/Input Controls
  • Text Areas
  • List Items or Select boxes
  • Label Elements
  • Frames

By default, the click/selection event on every control is triggered through a command preceded with the word 'Click', which can be disabled from the Commands tab under the Options dialog box of Dragon.

Some of the key points to keep in mind while working with Dragon are as follows:

1.     The first and the most important point to remember is that to make an element accessible by speech, their needs to be some text associated with it, which will be used as part of the voice command. For an element which has readable text in the forefront, say link or button, the associated command to access it through voice can be the intrinsic text itself. If no text for a control is displayed on the UI, they need to set to the ALT or TITLE attribute of the element. In case of input elements, the NAME attribute can also be used to set up the spoken text. In case of duplicate associated texts, all elements with the same text will be flagged, with numbers displayed on screen to differentiate the controls.

2.     The elements which do not belong to the stated list of permissible controls can be accessed by modifying the role attribute. Say, for example, we have a div element and we want to trigger a click event on it. We can convert the same to be identified by Dragon as a button/text/link by setting the role attribute for the same as button/text/link.

3.     It is also noteworthy to avoid special characters in the text associated to a control if it's supposed to form the voice command. For example, Dragon won't be able to recognise a text such as "Save & Close" on a button; instead it should be changed to "Save and Close".

With these aspects in mind, we will need to modify the Siebel Open UI application's HTML structure, to make it accessible by voice. Note that, the same cannot be applied to the HI application as it doesn't allow UI modification to set up custom voice commands. Also, since the HI application is based on ActiveX, the ActiveX controls in the application cannot be associated to voice commands as they technically do not lend themselves to speech recognition.

Some of the more generic elements won't require any modification, such as Screen/View tabs, OK/Go buttons, hyperlinks, etc. and can be accessed through the provided text. But these elements mostly constitute the navigation and querying aspects of the application but will not allow users to carry out complex functionalities.

Dragoon Architecture.png

Figure 1: Architecture of Siebel Open UI Integration with Dragon

Siebel Open UI constructs the UI layer through the physical renderer files. The primary file to control most of the HTML structure is "htmltmplmgr.js". So, to modify the controls, we would need to add "alt" or "title" attributes to the elements we want to control. Since this is a core OOTB file, and extending the same isn't possible, we need to be careful while modifying it. An ideal approach would be to separate out the code based on a flag, which can be set based on responsibility or a click event. This would avoid any potential conflicts with the existing functionalities which might occur. It must be taken into account that since we are modifying a vanilla file, any new patch or an upgrade to the current version would probably replace the file with a newer version. In such cases, we would need to merge the custom code to the new file.

Similarly, we can customise different elements by modifying the code from other vanilla files. This should be done by extending the vanilla file, wherever possible, such as the "physicalrender.js" file which lends itself for extension. There are certain elements or features which are problematic for speech recognition, such as the scroll bar, in which case we need to create a permitted custom element in the foreground and set up the click event to handle the desired effect programmatically.

In case of list applets, we have <th> and <td> tags which compose the headers and columns of the table respectively.  For obvious reasons, these elements cannot be replaced and hence, need to be modified by changing the role attribute from "gridcell" to "button" (as changing them to text or link would create conflicts with other elements displayed in the table). Similarly, there are <div> elements throughout the application whose role either needs to be added or changed to allow Dragon to access them through voice.

Siebel CRM with Dragon Command.png

Figure 2: Indicates all HTML elements which are links being flagged with Click Link command

Now, all elements which are permissible by Dragon for association can also be accessed by element name. For example, "Click Link", "Click Button", "Click Image" etc. will flag all such elements visible on the UI and number them accordingly, as shown in Figure 2, for users to choose the desired the element. So, it's recommended to set unique association text through alt/title/name for the distinct elements used to perform different functions in order to reduce the time taken by Dragon to process the HTML elements which it does every time a user starts speaking.

We also have several keyboard shortcuts in the Siebel CRM application. Siebel CRM allows us to configure new shortcuts for custom or vanilla functionalities through Accelerators of the Command object type, as well. We can create custom voice commands for these shortcuts through the Dragon tool itself. The software allows macros to be created from the front end which can be done from the Command Browser under Tools. These macros can be exported to a data (.dat) file which can then be deployed either as a patch for Dragon or be imported by the users themselves.

In Conclusion:

One of the primary features of Siebel Open UI is its flexibility towards customisation which enables us to design a more interactive UI and also to incorporate external applications such as Dragon to support the needs of Assistive Technology users. Since, it allows implementation of open source code, the solution provided in the above description can be approached in more than one way. We have had multiple customisations done on the CRM application since the initial development to assist AT users in accessing different features of Siebel and also to modify certain processes to better serve their needs.

Although, the above solution is provided for integrating Dragon with Siebel CRM, the same can be applied to any other HTML based application, which allows modification to the UI layer.

I hope to have covered most of the fundamental details in my explanation. However, if you would like to discuss further on the topic, feel free to comment below.


Reading Buddy Software is advanced, speech recognition reading software that listens, responds, and teaches as your child reads. It’s like having a tutor in your computer
Tool : Buddy Software

Reading Buddy Software is advanced, speech recognition reading software that listens, responds, and teaches as your child reads. It’s like having a tutor in your computer Tool : Buddy Software

Reading Buddy Software is advanced, speech recognition reading software that listens, responds, and teaches as your child reads. It’s like having a tutor in your computer Tool : Buddy Software

Reading Buddy Software is advanced, speech recognition reading software that listens, responds, and teaches as your child reads. It’s like having a tutor in your computer Tool : Buddy Software

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

Please key in the two words you see in the box to validate your identity as an authentic user and reduce spam.

Subscribe to this blog's feed

Follow us on

Blogger Profiles