·         Individuals and interactions over processes and tools

·         Working software over comprehensive documentation

·         Customer collaboration over contract negotiation

·         Responding to change over following a plan

In this article I would like to cover the failures that I have experienced in the Scrum development methodology at our customers development and what could be done to improve the process. Scrum is one of the Agile development methodologies that uses time boxed effort, known as Sprint cycles, to develop a working increment of a software. Sprint cycles (usually) last 3 weeks, in which requirement analysis, development, testing and documenting are all being performed. Which requirement are to be build in a Sprint are determined from a Product backlog (pool of user requirements) determined by the user community and moved into the Sprint backlog for planning. Each item on the Sprint backlog is called an Epic and can be narrowed down into separate pieces of coding called Stories. In order to determine the effort for bringing each Story live, Pokering takes place, determining how many Stories can be handled within a Sprint with the available resources. Every day team members gather in a Standup to discuss the work done yesterday, the planning for that day and possible impediments, in order to track the progress of each Story. After 3 weeks a review takes place of the work that was completed and not completed and a go/no go decision to Production is made. Finally in the Sprint retrospective all team members gather to discuss the past Sprint in a Retrospective, determining what went well and what the improvements are for the next Sprint.

A three week period between start and end of a project sounds like a dream to every customer. The thought of having a working product in such a short timeframe means that the end user does not have to wait months for there requirements to be delivered and the change list is not frozen from the start. In reality, a three week period is just sufficient for a small piece of software to work, sometimes as limited as a one line code. Major projects such as the European payment system (SEPA) or the New Market Model in the Netherlandsare comprised of multiple large changes, that relate to different systems and have a big bang go-live. Although the Scrum methodology foresees these scenarios by breaking down large changes in small Stories that are finally merged into one grant product, successful merging requires the same planning and management that is performed in the traditional Waterfall model. And because the changes are big and complicated, a three week period is not sufficient to design, build and test the whole solution.

Agile is about customer centricity and responding to change. One would think that understanding the customer is key to achieve these goals. Often the Agile team will therefore accept a request for change which is stated in high level description. "We want a page with a comprehensive customer overview" or "We want this page to run 50% quicker". Existing waterfall methodologies would not work if requirements are stated in such a vague manner. But as Agile prescribes the user representative to closely work with developer, the requirements become clearer and sharper along the way and the end product becomes exactly what the user had in mind in the first place. One can imagine that a lack of clarity from the start can lead to development going overtime and over budget. Or when a strict 3 week development cycle is kept, the software can be instable due to reduced testing time. What was expected to be a simple change could end up as a major unforseen effort.

One of the philosophies of Agile is working software over documentation. That sounds like music to the ears of a developer. Just build, build, build and we'll see about the documentation later. Unfortunately, documentation is best written when it is fresh in the mind. When documentation is written after go-live of the software, it is often seen as a hassled task and is often forgotten. For the maintenance team taking care of the software after go live, this is far from ideal. There is most of the time a lack of knowledge transfer between the developing Agile team and the maintenance team, causing discussions of what is and what isn't the task of the development team when it comes to the handover. As the Agile team is on a tight 3 week schedule of delivery, documentation can be of bad quality. Whilst Agile manifesto is saying that "Working software is more important than comprehensive documentation" it is actually trying to say that it "prefers" working software over (not abandon) comprehensive documentation. Try to create working software, because this is the only thing that adds value to the customer's business and not the extensive documentation. Still, useful documentation is to be delivered.

"Failing to plan is planning to fail" is often said. And with Agile's fundamental value of responding to change over following a plan, makes Agile an unstructured and short term thinking methodology. How does 3 week Agile delivery fit in a corporate strategy that is often months and years planned ahead? Requirements come from business users who only deal with day to day issues, which can only put a plaster on the wound but does not bring long term value to the organization. That is why product plans often comprise of change request that can not be legitimized by corporate goals and often have an exaggerated business case, saving thousands or even millions, but really will not.

Agile just cannot be seen as the magic trick that will solve all issues in software development. Its pitfalls are numerous and the criticism on its predecessors are not fair. We cannot leave out planning just because we want something as fast as possible. If we want to deliver long term value, two broad categories of planning should always happen[2][2]:

1.    Selecting which initiatives should be funded - doing the right work.

 2.    And once a project has passed the "should we do this?" filter we need to focus on doing the work right.

To achieve this we need to plan at many levels.

Innovation or problem: Every organisation needs a mechanism to get work into the pipeline - idea generation needs to be encouraged and problems identified. At the initial idea/problem generation stage there should be very little filtering, every member of the organisation should be able to point out something that is not working in the most effective way, identify a problem the organisation needs to address or propose a completely new thing that could be done.

Portfolio Planning: Portfolio planning is a governance level activity which is about selecting the right programs and projects which the organisation should fund at any given time. Deciding which work to undertake should be based on the organisations mission and goals - work should only be funded that delivers on the strategy set at the highest levels in the organisation.

Program Management: A program requires additional coordination to keep the interrelated streams of work synchronised. The program manager should be a dedicated role who has the strategic view of all the streams of work that must come together to deliver to overall program. The program manager provides the unifying vision for the separate project teams who will work on different streams of work.

Articulate the Vision: teams need to gain a clear understanding of the goals and objectives that have driven the selection of this project to be worked on. Team members often don't understand why they are developing what they are developing. Stewardship of the product vision is the responsibility of the product owner - this persons main responsibility to the team is to explain the why question.

We can conclude that to make Agile a success it cannot remain a single level project execution methodology. Putting the development team central will only work if we only aim for short term value. To succeed in an ever changing environment, corporate level planning is still required to ensure end-to-end impact. And that is why Agile is not just a 3 week development methodology, but is also a continuously adapting management process.