Unstructured business processes - creating differential BPM strategy is the wise thing to do
Finally BPM has found some spicy angle to its existence in form of what is being called as managing ‘unstructured or ad hoc processes’. That’s a good sign that progressive discovery of new concepts/ideas related to real-life stuff is still happening.
Various sources on the net estimate that 60-70% of the business processes in an enterprise are unstructured or ad hoc unfortunately. Can’t really say how much true that is because no reliable study or research is available to make conclusive view on it. But at least there is fairly reliably observation that reasonable part of the business process today apparently is what we call unstructured. Hang on with me, I will come to elaborate on what an unstructured process is if you have not been initiated into this new fancy stuff so far. Do we really consider the segmentation of structured and unstructured processes in the BPM strategy today? Largely not and that’s because they way BPM programs are run, there is hardly any visibility in most of the organizations about all the business processes in totality. So default assumption is that processes are all well structured. That may be the reason when some time unstructured processes are picked up for BPM implementation that see great challenges meeting the requirement. Irrespective of who is finally blamed for the failure, just think about it…it something new, something different, something we really didn’t pay much attention to...
(I expect to hear “I did it 10 years ago” as usual but we are not talking about exceptional wizards here, those could excuse me with all due respect)….
So what really is this unstructured process stuff or alternatively called ad hoc process? Here are some characteristics that are being considered as the DNA of such a process:
- Most importantly, activities or steps in such processes are not pre-determined and may not be (or need not be) repeatable either
- Process steps are majorly controlled and governed by the sequence, nature and even the interpretation (that may not be static) of the events as received by the users
- New process steps might have to be defined on the fly by the user based on the event. Sometime defining such a step itself could be a step in the process which might take days to complete.
- Might involve certain judgmental decisions that are not based on the ‘codified policies’, instead based on the subject matter expertise/domain expertise applied on the situation/scenario at hand (you might like to read this blog - http://www.column2.com/2009/09/micro-workflow-gestural-analysis-bpm2009-bpms209/ )
- Set of users/actors involve (or who could be involved) is not formally defined. It is discovered as event/process scenario unfolds. Scenario dictates who should be involved and who is the right person to execute a particular step.
- Number of iterations of the processes steps is unpredictable and largely dependent on the scenario at hand.
- Collaborative interactions among the users typically is a major part of such unstructured processes. (you might like to read this article - http://www.softwaremag.com/L.cfm?Doc=1102-12/2007 )
- As the nature of the process suggest, it is difficult to tie in KPI of the process to the process design since there is no pre-defined view of the process. This is still an evolving area of research focusing on how to manage the performance of the unstructured process while tweaking all the variables involved.
General assessment is that unstructured processes are not managed adequately (since there is little handle on performance management of such processes) and cost a lot in terms of poor productivity of the users involved. There are emerging debates on if the management of the unstructured process is within the scope of BPM or not. While keeping product (that is BPMS) in mind, yes, we might debate if current BPMS should be tried for unstructured process but as far as BPM is concerned, I don’t see any reason why management of unstructured process needs to be outside the boundary of the BPM. BPM itself means management of the business processes, whether structured or unstructured. Current BPM approaches/strategies/methodologies might be effectively tuned for the structured processes but as we are progressing into advance BPM, even the management of the unstructured processes needs to be conquered within umbrella of BPM.
Interestingly enough, if you observe carefully, you will find that ‘exception management’ of the structured processes in many organization remains to be an unstructured process actually. It means that once exception is identified (which is an event that triggers the unstructured process), there will be someone who has responsibility to monitor the exception event and who will look at the exception and then do one or more of things (that are not clearly defined as process flow but could be bound by few formal or informal directives) in terms of a) Validating the exception b) figuring out what it means c) figuring out who needs to do what with it d) Forwarding the details and actionizing request to identified authority e) monitoring the state of exception as actions move from one person to other (on system or through email exchanges) and finally f) certifying the exception to be closed after all iterations are closed and exception is resolved. It’s not that there are no exception management software/tools/modules but due to the fact that exception management process itself is not clearly defined (or may not be defined) in the organization, there is very little that such tools/software can really help. So in a way, we really have had encounter with the unstructured processes in the form of exception management processes but I think the problem of unstructured process in the organization seems to be much larger than just the exception processes.
Engineering of the unstructured processes is far from being mastered (keep in mind that even the BPM for structured processes is far from being mastered across the globe) so of course approach toward it is slightly experimental as well as open to new best practices. There are couple of them that are slowly gaining importance based on whatever trials that have been done so far. I’m putting down what I have learnt from our own laboratory as well as gathered from advancements happening across the globe:
- When processes discovery/modeling is being done, classify them into structured (or pre-defined/static) and unstructured (dynamic/variable) right in the beginning so that we can take differential strategy of BPM (to be realistic) for each type of process portfolio.
- To start with, don’t over-engineer the unstructured processes to try to turn them into structured processes else they lose their edge of flexibility and responsiveness. Focus only on managing them better. This will involve aspects like identification of patterns/events, notification to users, provision of necessary information to make the decision, tracking the process foot print and recording the outcomes, providing tools/utilities that support the adhoc activity like email based interactions, ability to take actions directly from Microsoft documents using smart right click menu links etc. (you might like to read this blog - http://www.column2.com/2009/02/actionbase-adds-structure-to-email-based-processes/ ). As you manage such process better, you will get better handle on the behavior of variables involved and provide you better understanding of the improvements that can be brought.
- Methodically plan to transition the unstructured process to regularized structured process (even partially is fine) if the patterns of the process become repeatable over period of time. Recording of the process flows of the unstructured processes is important to discover the process patterns.
Actionbase is one of the emerging products that are trying to address the space of managing unstructured processes (read more about it in the article - http://www.bpm.com/actionbase-launches-version-60-to-manage-unstructured-processes-in-organizations.html ). There are other players like Itensil (http://www.itensil.com/) and Handysoft (http://www.handysoft.com/) that are proposing additional capabilities but as I see, space of unstructured processes is far from being clearly understood, profiled and domesticated. This space looks very exciting and brings new hopes of innovation.
I’m hunting for strong real-life examples/use cases of the unstructured processes. So if you got any in your organizations, it will be great if you could drop a note on that on this blog or to my email (rakeshmishra@infosys.com).
Comments
I've got some examples of semi-structured processes on my blog.
http://rvsoapbox.blogspot.com/2009/04/semi-structured-processes.html
Posted by: Richard Veryard | December 31, 2009 04:56 PM
Thanks for sharing the blog Richard. I went through the stuff and here is what my view on those examples is (as similar to what I have seen other similar examples quoted on internet). As I see, majority of these examples fall into category of a) Exception handling b) Unusual or unexpected scenarios like terrorism, natural disaster, major escalation. Now, these are not regular processes and very well understood that no one will ever has totally structured/formulated processes for these in business as such except for those for whom such unusual scenarios are the core processes to deal with (like anti-terror squad, disaster management units etc.). I'm actually looking regular day to day kind of mainstream business processes that are unstructured and are executed mostly based on unstructured flow as per the characteristics defined in my blog. Do let me know if you figure out some that are good exmaples/case studies to work upon.
Posted by: Rakesh Mishra (Author) | January 4, 2010 11:44 AM
Unstructured process is a paradox. If a set of activities cannot be defined as sequence, it cannot be called a process, it will just be a jumbled-up activities. There is nothing called unstructured process and that is the reason, we cant name even a single commonly used such so called process. BPM, which is primarily (practically entirely) managing processes, is meant for set of activities that can be defined in the form of process. It, still has,not gained maturity. Let the kid to grow and gain maturity for meeting set objectives. Ghost of unstructured process will not add any value but affect its natural growth.
Posted by: Rupesh | January 5, 2010 09:05 AM
Thanks for the view-point Rupesh. I can understand where you are coming from. I do not entirely agree with that (we can definitely agree to disagree while still introspecting our individual rationals), especially with "If a set of activities cannot be defined as sequence, it cannot be called a process". If I were to look at it closely the way I will put it, no process is entirely unpredictable, entirely random and entirely adhoc. In the context of the unstructured process that I'm talking about here, most of the time there is a broad idea/view on what type of activities might need to be performed to manage the outcome of the unstructured process but putting them down in well-defined sequence and policies may not be feasible (mind you I'm not saying its not possible). It is typically seen as unnecessary overhead to model such processes, create formalization and trying to manage them. As we move forward in this complicated world, I think time has come (and my hint is for paradigms like CEP that no one would have bothered few years ago)to deal with the unstructured side of the story, whether its information or the process.
You have rightly said that today's BPM is entirely about structured processes (unfortunately though) but that's only a reflection of progressive maturity and a milestone that we are able to reach so far. It doesn't mean it need to stop here or it will stop here. I'm very sure it is going to change, sooner than what you or I might think...But let me say, I'm still exploring so do not want to close the doors...:-)
Posted by: Rakesh Mishra (Author) | January 6, 2010 05:52 AM
Rupesh,
The reservations you express are not unusual for a BPM practitioner - but if you talk to the business side of an customer they seem to get it right away. They understand that there are many processes that don't fit the assembly line model of most BPM, or even the variability enabled by dynamic (rule-based) BPM. If you look at any process that involves knowledge work - it is always an ad-hoc, unstructured process that includes negotation, discussion and collaboration, and the flow is managed and changed by the participant themselves. I'll give two examples that are hopefully meaningful to you:
1. Creation of a BPMN model for a process. Gathering the information and building the model needed for a BPM implementation cannot be done in BPM - though I think most practitioners would consider it a process. That is why SAP used Google Wave to implement a modeling environment (the Gravity project) - not their own BPM suite (you can read more about it at http://blog.actionbase.com/can-google-wave-become-a-disruptive-good-enough-bpms
2. Responding to an RFP. That is an example of a mixed process - some of the work is routine and can be modeled and executed via a BPM suite, but there is also a large portion that is ad-hoc - dependent on the specific RFP - it changes and morphs throughout the process as participants start handling the unique requests and elements of the specific RFP, and take into account the enviroment at the time of response (including which skills are available etc.)
I have been discussing these issues in my blog at blog.actionbase.com.
Hope this helps you better appreciate ad-hoc processes and how they are used.
Posted by: Jacob Ukelson | January 11, 2010 06:51 AM
Thanks Jacob. RFP scenario is a fair sample of what I was looking for.
For the BPMN for BPM implementation, I think that is more of tooling and methodology issue. My realization was that BPM projects mostly always had been case of 'square peg into round hole', most of them in my view because IT always focused on BPM implementation as typical software development/implementation project and they straight away adopted the waterfall methods/tools/techniques for BPM. But in the process, IT folk forgot that BPM is least of a technology strategy in classical software development sense so it started failing. As focus is being on the core of business value achievement through BPM, we can see that industry is discovering newer methods (like agile in BPM) of delivering BPM programs. However, still, integration of IT planning/designing with business planning/designing for the same business goals is something that organizations still need to crack. In that sense, I found BPM to be a great innovation catalyst that is forcing industry to loosen the conventional methods and adopt newer ways.
Posted by: Rakesh Mishra (Author) | January 11, 2010 12:08 PM
Rakesh,
Hi. Sorry it took me so long to respond.
Yes - you are right, creating a model of a business process is a tooling and methodology issue – and that exactly is my point. It is a complex, collaborative, ad-hoc, human process that a standard BPM suites can’t handle. Some BPM vendors have seen the issue and are creating specialty, bespoke tooling (some call it social BPM) to address it – SAP Gravity, Software AG Alignspace.
My point is: “why?”. Since creating a model is a process – why didn’t they just use their own BPM suite and build something to handle modeling in a better way (a “eat their own dogfood” argument). The reason is they can’t - since creating a model for an existing process is a collaborative, ad-hoc, human process – something BPM suites are NOT suited for handling.
Posted by: Jacob Ukelson | January 16, 2010 05:45 AM
Hello,
I could create an analogy to Services in SOA. When one constructs a Service, it is accessed against the SOA Principles. Looks like we need principles for creating or certifying BPM processes as structured or enterprise business process. Anything that does not follow these principles are un-structured. If I understand the argument correctly, the 8 points of what Rakesh has mentioned are the good candidates for such principles.
Posted by: Bhupesh Naik | January 18, 2010 12:56 PM