Empirical

Means that the information is collected by observing, experience or experimenting.

Empirical process
Is used for handling processes that are complex and not very well understood.
So let's look at how it's defined.
An empirical process is seen as a black box and you evaluated it's in and outputs. This is done by defining checkpoints that should occur at predefined points.
So what do we have in Scrum?
We have the sprint as the black box in the meaning that the work that goes on in the sprint should not be disrupted. We have backlog items that's not started as input and finished backlog items as outputs.
As checkpoints in Scrum we have daily scrums, sprint planning meeting, sprint review meeting and sprint retrospective. Since each sprint is time boxed, the checkpoints will always occur at a predefined time. Inspection is done at each checkpoint so you can adapt the process with the information collected through observation, experience and experimenting.

Another important aspect of an empirical process is that the process model (the work we do) is directly derived from the input and output data. So there should not be any possibility to change how the work is done in the process from the outside. That's why we don't add more backlog items to a sprint than the last sprint proven that we could do even if demands for this is coming from higher management. We need to be true to the truth and protect the Scrum process.

So now we know that Scrum is an empirical process but why is that so good? Why not use a defined process? It is much more predictable.

To answer this question we need to look a bit deeper into what we are facing when developing systems and what an defined process is.

Defined process

A defined process is derived from first principles, meaning in science that it needs to adhere to laws of nature and other fundamental and well defined laws.

Another more understandable description is made by Ken Schwaber in Agile Software Development with Scrum.

"The defined process control model requires that every piece of work be completely understood. Given a well-defined set of inputs, the same outputs are generated every time. A defined process can be started and allowed to run until completion, with the same results every time."

This is theoretical true but in the real world there will be a variance but held to a minimum. There is also no need for feedback because you will always know what to do next.
So here is a variance of Ken Schwaber definition.

"A defined process is an amount of tightly coupled steps where the output from one step is the input to the next step and where no observation or evaluation of the output is done to feedback to the process. A defined process when started will run to the end without any checkpoint. The output from a defined process should always be the same or with little variance given the same input to the process."
The process could be very complex but needs to be well understood before execution.

This is typical for a "waterfall" like process where everything needs to be defined in the beginning and then you follow a cookbook like recipe and you should get an expected result. So if system development should fit this process it should be very predictable, non creative and should not demand any thoughts during the execution phase.

This is not how I or many of my colleagues think about system development. For us it's a highly creative and intellectual process that demands a high degree of thinking. We need to expect the unexpected and be able to act on that in a complex environment.
The complexity in system development is often from 3 major areas, requirements, technology and people.
- Requirements are often not that well understood, badly communicated and missing.
- In a system development project we are at least working with 10 different technologies and that is in a small or medium sized project. For example: Java, JavaScript(and other scripting languages), XML, SQL, XSLT, JSP, Spring, web services, Schemas, Maven and a bunch of other frameworks we need to satisfy our requirements.
- And people that can change anything in the other two so that means that we have a very unpredictable complexity.

So a waterfall like process doesn't look that good for system development but what about all the other iterative process that are out there. Shouldn't they function as well as Scrum?

To answer that question, we need to look at each of these processes and see if it is an empirical process. Most agile processes use time boxing and contain feedback checkpoints. Others lack regular quality checkpoints that provide information to adapt the process. They also do not protect the process and will corrupt the data by outside involvement. Many of these processes are often more waterfall and/or defined process than an empirical process. Just because you use iterations doesn't mean that you have an empirical process.

Unit testing is often used to test complex units with one or more collaborators, usually with the help of mock objects. But the greatest value with unit testing, in my opinion, lies in testing the lowest level utility methods; the building blocks that simply have to be bullet-proof in order for the rest of the system to have a stable foundation.

Let's say that we have an Article domain object, storing article numbers from the database in the form "01234567":

public class Article {

    /**
     * 8 character numeric article number, eg "01234567".
     */
    private String articleNumber = "";

    public String getArticleNumber() {
        return articleNumber;
    }

    public void setArticleNumber(String articleNumber) {
        this.articleNumber = articleNumber;
    }
}

Business now demands that the article number should be formatted with dots here and there. They have added the story "User should see article numbers formatted as 123.456.78" to the Sprint Backlog.

We decide to write a test for this:

import static org.junit.Assert.assertEquals;
import org.junit.Test;

public class ArticleTest {

    @Test
    public void testGetFormattedArticleNumberNoZeros() {
        Article tested = new Article();
        tested.setArticleNumber("98765432");
        assertEquals("987.654.32", tested.getFormattedArticleNumber());
    }
}

We search around for a while, and find several nice tools for parsing formatted strings, but nothing useful for writing formatted strings that will help us in our scenario. Lacking ideas, we desperately try DecimalFormat. It's actually not that bad. If we divide by 100, we'll automatically get the right-most dot, don't we? We decide to give it a try:

import java.math.BigDecimal;
import java.text.DecimalFormat;
import java.text.DecimalFormatSymbols;
import java.text.NumberFormat;

public class Article {

    /**
     * 8 character numeric article number, eg "01234567".
     */
    private String articleNumber = "";

    public String getArticleNumber() {
        return articleNumber;
    }

    public void setArticleNumber(String articleNumber) {
        this.articleNumber = articleNumber;
    }

    /**
     * Retrieves the formatted article number.
     *
     * TODO I didn't find a nice generic formatting method, so I wrote this terrible hack.
     *
     * @return the formatted article number, eg 123.456.78
     */
    public String getFormattedArticleNumber() {
        NumberFormat nf = NumberFormat.getNumberInstance();
        DecimalFormat df = (DecimalFormat) nf;
        BigDecimal number = new BigDecimal(articleNumber);
        number = number.divide(new BigDecimal(100));
        String str = df.format(number);
        return str;
    }
}

We get a failure:

expected:<987[.654.]32> but was:<987[ 654,]32>

The decimal point is a comma (,) in the default locale, and the grouping separator is a blank. We can tweak that by calling the getDecimalFormatSymbols method, giving us a DecimalFormatSymbols that we can change:

DecimalFormatSymbols symbols = df.getDecimalFormatSymbols();
symbols.setDecimalSeparator('.');
symbols.setGroupingSeparator('.');
df.setDecimalFormatSymbols(symbols);

We also make sure the grouping size is always three by calling df.setGroupingSize(3). Here is the code so far:

public String getFormattedArticleNumber() {
    NumberFormat nf = NumberFormat.getNumberInstance();
    DecimalFormat df = (DecimalFormat) nf;
    df.setGroupingSize(3);
    DecimalFormatSymbols symbols = df.getDecimalFormatSymbols();
    symbols.setDecimalSeparator('.');
    symbols.setGroupingSeparator('.');
    df.setDecimalFormatSymbols(symbols);
    BigDecimal number = new BigDecimal(articleNumber);
    number = number.divide(new BigDecimal(100));
    String str = df.format(number);
    return str;
}

Success! Excellent. Now, we suspect that there might be problems with leading zeros, so we add another test:

    @Test
    public void testGetFormattedArticleNumberLeadingZero() {
        Article tested = new Article();
        tested.setArticleNumber("01234567");
        assertEquals("012.345.67", tested.getFormattedArticleNumber());
    }

As we suspected. We get a failure:

expected:<[0]12.345.67> but was:<[]12.345.67>

This can be fixed by setting the minimum integer size to 6:

df.setMinimumIntegerDigits(6);

We run the test suite again. Success! Great. We suspect that trailing zeros can also be troublesome. We add a few tests for that:

    @Test
    public void testGetFormattedArticleNumberSingleTrailingZero() {
        Article tested = new Article();
        tested.setArticleNumber("98765430");
        assertEquals("987.654.30", tested.getFormattedArticleNumber());
    }

    @Test
    public void testGetFormattedArticleNumberTrailingZeros() {
        Article tested = new Article();
        tested.setArticleNumber("98765400");
        assertEquals("987.654.00", tested.getFormattedArticleNumber());
    }

    @Test
    public void testGetFormattedArticleNumberAllZeros() {
        Article tested = new Article();
        tested.setArticleNumber("00000000");
        assertEquals("000.000.00", tested.getFormattedArticleNumber());
    }

We run the tests. The last three tests all fail, indicating that trailing zeros are lost. We fix that by setting the minimum fraction size to 2:

df.setMinimumFractionDigits(2);

Success! The complete method now looks like this:

/**
 * Retrieves the formatted article number.
 *
 * TODO I didn't find a nice generic formatting method, so I wrote this terrible hack.
 *
 * @return the formatted article number, eg 123.456.78
 */
public String getFormattedArticleNumber() {
    NumberFormat nf = NumberFormat.getNumberInstance();
    DecimalFormat df = (DecimalFormat) nf;
    df.setGroupingSize(3);
    df.setMinimumIntegerDigits(6);
    df.setMinimumFractionDigits(2);
    DecimalFormatSymbols symbols = df.getDecimalFormatSymbols();
    symbols.setDecimalSeparator('.');
    symbols.setGroupingSeparator('.');
    df.setDecimalFormatSymbols(symbols);
    BigDecimal number = new BigDecimal(articleNumber);
    number = number.divide(new BigDecimal(100));
    String str = df.format(number);
    return str;
}

It's not pretty, but it works. We check in the code, relax, and think for a while. Is there a simpler way? Of course there is:

/**
 * Retrieves the formatted article number.
 *
 * @return the formatted article number, eg 123.456.78
 */
public String getFormattedArticleNumber() {
    StringBuffer number = new StringBuffer(articleNumber);
    number.insert(3, '.');
    number.insert(7, '.');
    return number.toString();
}

Here, the power of unit testing shows. We don't even think twice about replacing the existing code. The tests still run, so we hook this code into the user interface, run the functional tests, check in, wait for a successful build, and finally ask the Product Owner to run the acceptance tests for the story.

What is required of an organization to be Scrum-ready and be able to reach hyper-productivity? What is required to start with Scrum?
- This is something I’ve talked a lot to Jeff Sutherland about. Jeff would focus on some of the practices and the technical infrastructure. That is something engineers listen to. Like automated checkin testing: a checkin should only be accepted if a test passes, otherwise it should be rejected. You cannot run Scrum without that. Jeff has also said that you will never run effective software development within Scrum without upfront architecture.

There are also the standard things: you have a product owner, the product owner is present and updates the backlog. On top of that I would like to emphasize that Scrum won’t give you excellence. When people start going into Scrum I always say “Wow are you going to make trouble”. But of course they’re not causing problems. What they are doing is surfacing problems that are already there.

To run Scrum you need connection with the customer. Scrum will not give you that, only make it obvious that you don’t have it. You also need some degree of unity of purpose: the team being a team. Scrum will not make a team into a team, only make it obvious that they are not working as a team. Some of the things people feel they should be getting from Scrum, such as a team and better customer communication, are actually preconditions to Scrum. What Scrum gives you is predictability.

You have said that Scrum can be described with organizational patterns. Is it a good idea to start with the patterns before going for a full Scrum? What kind of benefits will the separate patterns give a new Scrum team?
- I think it depends on the individual teams. If it is a small, hungry and fairly young at heart team there is a kind of hipness to Scrum that sets a wonderful rallying point. Jeff would say “go to Scrum overnight”. I don’t know if I would be quite that bold. You probably should do things in pieces. Start with a product backlog, then a product owner, then sprints and so on. You should do it incrementally.

In a large and trenched organization you really benefit from the organizational patterns approach to Scrum because the improvement is incremental and it reduces risks. In large and trenched organizations you want to manage risk very carefully. In a young organization, that is small and isolated, you are going to find a way to recover anyhow. But in a large system you need to be attentive to risks, and that’s where patterns really shine. Some of those things are not Scrum, they are preconditions to Scrum.

Is there not a risk that if you do things incrementally, you will never get to the actual Scrum?
- So who cares? The goal is not to do Scrum. The goal is to repeatedly deliver value to your stakeholders: your users, customers and employees. You might be following the practices but not delivering value to your customers. Then you are missing what it is all about. It is not to do Scrum, to do XP, to be part of a club or to follow a religion. It is about expecting and adapting within some framework that makes sense. With that said Scrum is a great starting point. For some teams XP or Crystal Clear is a good starting point. For more heavy weight agile things DSDM is going to be a great starting point. Once you get to the starting point and show that you can do the disciplines, you will evolve.

I have always thought of Scrum as being something that has to come from a grassroot level, that the developers have to drive Scrum forward. How do you introduce Scrum to an organization that has been fed with it from the top of the organization?
- There’s the ideal and there’s the pragmatics. Before either of those there are some principles in organizational change. It’s going to be very difficult to make any change if management is against it. You need at least the tacit support of upper management. One of the things I say is that managers can’t make substantial improvement in an organization, they can only prevent them. You also need buy-in from the troops. This top-down thing is falling into some challenges. I think you need both and I would prefer to have strong elements of self-organization. This gets into the principles. By the time you get to that level, the question of bottom-up vs. top-down does not make any sense. There is no bottom or top. It’s potentially one level. To me that is the idea.

Lately I have been talking to Jeff Sutherland quite a bit and I’ve been understanding Scrum much better. I am finding out more and more what the history is about and they say some weird things. One of the things in the Toyota-way approach is that it’s not really worthwhile to go in and just have self-organizing teams. So instead you have these things called cells. You get people together as close as possible so you don’t stockpile parts to move them down the assembly line. You just move them over the table. You are only working on one part at a time and you do not stockpile. This is a lean principle, just in time, no inventory. To make that work you need continuous process improvement, and everybody is a process engineer. You need hands-on involvement from managers.

If there is anything wrong in the plant management does not get a report up the line. They come down to the factory floor and become materially involved. And they do this on a daily basis. They can look at status boards on the factory floor. At any given time they can tell what the inventory status is. It is very visual. It is like constant feedback loops. It is important that upper management receives this feedback.

Jeff is really big on having the entire company involved in the Scrum framework and the Scrum value system. He would say that everybody is part of some Scrum. Management is part of their Scrum, let us call it a meta scrum. Teams are doing things in Scrum, and then you have a Scrum of Scrums that gets them together. In Jeff’s model you basically have a two level structure.

The point is not that they’re going through the framework and following a process, but that there is a company- wide sense of very close communication and continuous process improvement. That is neither top-down or bottom-up. Maybe it is a steady state result of one or the other, but by the time you get to that point, agile permeates every corner of the enterprise. The problem with bottom-up is that you will end up with a couple of teams that run Scrum, and a couple that don’t.

Management will tacitly support it with no real vision of changing the company. You may be able to get a 30-35 % improvement. We have to go beyond bottom-up vs. top-down thinking and realize there’s a whole enterprise. Now you are really going to see the benefits and being able to transform the way you do development, and get these hyper-productive teams. They are getting a factor of 10 or 100. That is exciting.

Are managers needed in self-organizing teams and what are their roles?
- I have an anecdote related to this. I had a client who had a whole bunch of project managers. They were all yuppies, enthusiastic about agile. They said they would make their team run agile. I told them that it is not about telling them what to do, it is about self-organization. They said “I’m going to self-organize my team”. If you look at the Scrum framework, the job of managers is to remove impediments. If the team cannot solve the problem themselves they give it to the ScrumMaster. If the ScrumMaster cannot solve it he takes it to management. That is one very important role of management. The ScrumMaster is being a servant leader.

Finding how he or she can move the team forward. Management have the same role but they have bigger hammers. If you look at the management’s training or wishes, they like working with people and growing people. To some degree ScrumMasters can do that but their main focus is getting products out of the team. Management has a more strategic view: to get funding and send people to conferences and training.

The ScrumMaster has a lot of responsibilities on a day-to-day basis. Sometimes conflicts will escalate to a situation where you need to bring in management. So it is a kind of leadership. But what exactly a manager does differs from culture to culture. I do not think we need a standard Scrum manager. I think individual cultures will have different roles for managers. Will they do the hiring and firing, or will the teams do that? Who does salary administration? Does the team divide a pile of money themselves, or does the manager give out rewards based on their involvement in programs inside the company? I strongly feel there’s a place for managers in a Scrum context. This is a place where Scrum doesn’t go. That said, there is no place for project managers.

Let’s say you are called in as a consultant by upper management. You enter a problematic organization. What would be the first thing you do?
- The first thing would be to listen and to find out what the problem really is. I have some skills, both technical and organizational, but I’m not a personal coach. I would probably take someone along with that kind of skillset and have a team of two or three people to just sit and listen. The next step would be to say that we really need to get the whole organization involved: let us sit down and do an analysis. There are a lot of ways this could go. As an external consultant you may find out that the problem with the corporation is that a higher manager is an alcoholic or some other single point of failure. Then you do what you need to do. There are a couple of organizations where I have gone and done exactly this. In fact it was management who called me in. It was the CTO. The old CTO had just been fired. The new CTO’s company had gone under so he was looking for a job. The company hired him in hopes he would clean up the mess and fire all the stupid developers and replace them with his old employees. He was smarter than that. He saw these organizational patterns, and he called me. What we did was we got everybody together on the engineering team, the whole engineering organization of about 50 people, in one room and we did a roleplay. This was the first time the engineering organization understood what they were doing. They where astounded. That is what I would do!

Again it is an interest of the Scrum principles of honesty, openness and visibility. Let’s make things visible, let us bring it in front of everyone. Let us be inclusive so that we get everyone’s perspective. Then we are in a position to do something. Also, in the interest of Scrum, I will not tell them what to do. I can not tell them what to do to as a consultant. I can suggest analogies that has worked in other companies in the past. I want to make models, and show them models of themselves. Given these models what do you think of yourselves? What do you think of these models of yourself? We make these social network models. They are usually very telling. It never ceases to amaze me.

The best story I have about this is a company I used to work with that was called Project Silver Bullet. I was invited to study them. I did my exercise, I made models and I handed them the models the next day. I delivered them to the manager because she was my client. She was chocked and angry. She said “who told you this”. I said “we just did a roleplay with your people, and I made models”. She said: “No, you were only there for four hours. Who told you this. Who’s trying to destroy me?” The model was so good she actually felt threatened by it. So they see themselves and they can see the problems right away. The data all comes from them, and I put them into this tool that makes three dimensional scaling of the data, and shows them what communication structure is. This is very telling.

You have been part of many different discussions at Øredev; everything from DDD to Agile. You have written several books and your webpage states that your interests are collecting carpets, Questing and genealogy. You seem to be a man with many interests and many opinions. If we turn the tables, what do you think would be a good question that you would like to have published in an interview like this?
- There are several actually. One is of the kind “what do you want to be when you grow up?” That is one of my standard questions. But a really good one is: “If I could make a lasting impression on the world, a lasting change. Without ever being credited or recognized for it. Or if I could have a broad change in my lifetime and be credited for it, but it doesn’t persist after my death. Which do I choose?”

And the answer?
- I’m in this to make the world a better place for everybody. At the Øredev opening dinner Michael Tiberg had me stand up and said “Cope helped connect me to a lot of you! And that’s why you are here”. I see myself as a yenta. It’s a yiddish word that means matchmaker. I connect people together. Part of it are the connections themselves. It’s wonderful to watch people come together. It’s wonderful to come together with people who have different opinions. Like King David said: “iron sharpens iron”. In the long term these connections will cause the kind of feedback that makes it possible for the world to become a better place.

I choose being a leader as a person rather than creating this leadership role. A mantle that is going to be handed off, from me to another person and then to another and then passed around very rapidly. The everyday play and game of doing what we do in our profession is fun. That is what is going to make the world a better place. I do not know about today, but in the long term. That is a lot more satisfying to me.

The first part of this series can be found here: Scrum Shock Therapy, Part 1.

The Management Recipe

The checklist above is only a part of what it takes to go through Shock Therapy. I often find myself at new companies and need to work with a bigger picture than focusing solely on the team. As is evident from the polls and surveys above, we need to address management as well. The word shock is perhaps more applicable here. If it has learnt anything from the past it is that you have to "drive" development - delegate, follow up, check, adjust, hand out, recheck...
Telling them to trust the very group that has not delivered before is asking for a leap of faith.

Setting up a recipe for this group is tricky. You have to adapt to management style, the corporate culture and so forth and the things you ask for are softer. These can be very hard indeed to change! In plain English; The management has to take some more real tough decisions for a Scrum to work efficiently. If you need to kill a document standard or process step then do so! If this is impossible (for now) then go for "barely enough".

The Management

• Attend the Scrum training session with everyone else
• Hands off during 3 iterations
• Attend:
• Some Daily Scrums - be quiet
• All Sprint Reviews
• Start to work on waste - now!
• Management by walking, asking and listening, i.e. practice facilitation.
• Make the first step easy for the team
• ...

When learning something new you have to do it several times just to make sure you are not totally confused. I have found that three times a charm. Try your best to get management to abide by this "rule". After the three iterations everyone has better picture to judge the value of the effort and Scrum.
Eliminating waste is vital on any type of project or endeavor. In a transition it makes even more sense, you have to make space to learn something new.

Finally, management has to ask the people that will perform these miracles what they need to do just this. Have they? Have you?

The Management Exit

• One successful team
• Removed impediments and studied results
• Change in perception at "ground level"
• A good agile reason to change the rule

Having one successful team is sort of self-evident. This takes patience when the ride is anything but smooth. Being a part of this process is important. Management will have the best ability to change ambition level, removing impediments etc. Get cracking and be that good example.

If there is a change at ground level on how they perceive management, then you're probably on to something.

Is this recipe the final version - probably not? Your mileage will vary, adapt to your management.

The Organization Recipe

Many of our customers are just starting out doing Scrum. Therefore we typically come into play as an external effect. It is important for us to understand how the agile effort has started and its nature within our client's organization. We can basically divide this efforts into to two major types - "bottom-up" and "top-down". The typical traits for a bottom-up style is one where the programmers et al have decided to go agile. The management might be aware of this and even letting it run. Then again this might be a stealth and an under the cover job.

I have found the bottom-up approach to be the most common case. Programmers usually like to work in an agile way. In these type of setups my work usually revolves around questions like "How do we fit this to a waterfall context?" etcetera. This can be tough work, but I find it to be the more easy of the two - the people who do the day to day grinding are already on board.

The top-down effort of introducing Scrum comes with its own set of problems. Any methodology or decision that involves change tend to meet resistance from the organization. We need both experience and sideways effort to the transition.

The Top-Down Scrum Organization

• Get a Scrum Sensei
• Inject a Agile Senior Programmer into the teams
• Make sure there is organizational transparency

The Scrum Sensei is an experienced Scrum Master who's been there before. Much in agile and Scrum is common sense, but there is no need in making the simple mistakes. With a battle-trained mentor the whole organization get a sounding board (Swedish: bollplank) helping out to implement Scrum. This part is important to the Shock Therapy where he or she will be the contract owner enforcing the startup and approving exit. In some sense this person could act as a "bad cop" in a good-bad ScrumMaster setup.

Or in the words of Nanny McPhee - a children's movie where a nanny answers the seven ne'er-do-well children on how long she will stay:
- When you need me, but do not want me, then I will stay. When you want me, but do not need me, then I have to go.
The Scrum Sensei needs to work in similar way to bootstrap the team and management.

A top-down effort always risks resistance from the organization. My experience tells me that a side way force is needed for translation on the factory floor. Moreover, such a person has the experience on a technical level to do the hands on tools and practices that is a vital part of any agile projects. A seasoned programmer with agile experience knows what works and how to translate the management vision into bits and bytes.

"Self-evident" sometimes reads "evident to myself and no one else". If we want to affect the organization we also have to make sure that practices, experiences and results are easy to come by. Walking down a corridor should yield interesting information about ongoing projects. If there is a pilot project doing Scrum then visibility is high on the wish list. Not only does it mean commitment for the Scrum team, but also a pull factor for other teams to get on board. Organizational transparency comes in many flavors. Apply liberally!

Summing it up

There is no denying that projects have problems getting started with Scrum. But more often than not I believe that we can do something about it. Why not try some nice Shock Therapy?
It might be the sweetest hard deal around!

/Björn Granvik, Jayway

MySpace Therapy:
http://jeffsutherland.com/scrum/2008/09/shock-therapy-bootstrapping.html

Scrum Poll on Nokia Test Practice:
http://tinyurl.com/68vapj

VersionOne Report: State of Agile Development Survey:
http://www.versionone.com/agilesurvey/

Situational Leadership:
http://en.wikipedia.org/wiki/Situational_leadership_theory

Nanny McPhee:
http://www.imdb.com/title/tt0396752/

I asked for comments/questions. As it turns out I got mostly questions on Scrum in general and not so much on my topic - how to bootstrap Scrum. I'm not sure what to make of this.
Anyway, here are my takes on those questions. I hope they're of some value.

Dear Björn,

How do you motivate a team to produce more without paying more for IT (salaries)?
Hourly salary seems retarded for agile teams.
Thanks!

Dear anonymous,

I'll first look at the non-monetary part, mostly Scrum, and later on salaries and bonuses.

Self organizing
In many organizations people enjoy very limited control over their own situation. Being able to self organize is important. We humans just tend to like this.

Pick your tasks and choose your implementation
Again, if I can use my competence and decide (as far as possible) on how to implement a certain goal, I will enjoy my work the more. Using this approach I've even been able to rehire people to very problematic situations. Very powerful.

Deliver
When I ask programmers what they are most proud of, I usually hear words like "idea, made by me, real users". It might not be the hardest and certainly not the biggest project they worked on. They used their competence, created something and delivered! In Scrum you get the delivery feeling at every sprint. If you break it down into activities no bigger than a day, then you will get that nice "flow" and people will be able to say "I did" every morning.

Good Colleagues
I've found that Agile makes sense if for no other reason than you can keep and hire the good people. SmartFriends(tm) is a great way to work!

I'm sure I could pick more facets of Scrum and Agile on how to motivate your team. But let's move on to the Money. This part of my answer doesn't have so much to do with Scrum or Agile, but rather my beliefs. So you have to be the judge here - what would work for you and your situation?

Powerful, way too powerful
Salaries, bonuses etc affect you - all the way to the bank and your holiday and...
In short: Once you start using money in different forms as your primary means of rewards, you will get a powerful ally in driving your people. Perhaps to powerful.

Take bonuses at a consultancy firm (a real example from life). You have all the good intentions so you set up a bonus system so that everyone will benefit if they have a client. More hours with customers, bigger bonus. Easy. This way you can lower your costs in bad times. Nice, if you're just counting beans.

Effect: No one wants to come in do monthly meetings if it's going to cost them money.
Counter effect: Introduce a threshold so that there is room for a bonus even though it doesn't mean 100% with customer.

Another effect: You just removed your ability to make strategic decisions like "put someone on the big corporate account" that pays less, but means more hours. Who wants to loose money working for that customer if they have to take a cut money-wise?
Counter effect: You introduce another "rule" specifying that a lower price per hour will not affect their bonus.

Yet another effect: Holidays suddenly never cross the monthly boundaries. That would put your people below the threshold for the bonus on two separate salaries.
Counter effect: ...haven't got the faintest here what they did here.

I hope my example makes sense in your situation. My point is that money matters. When used to drive people you will get side-effects.
For me this is not good enough. I don't want just to persuade people's wallets - I want their hearts and minds. It has to be fun and engaging.

Here's my short take: First make sure you have decent salaries etc. When this in place, aim for those things that make our working day worth while. Set interesting goals, get good colleagues, the right work to do, empower people and so on.

How to ensure creativity and quality in Scrum?
/Fredrik

Hi Fredrik,
I liked the way you wrote this on a green note (meaning "good" when voting at the conference). Thanks

Creativity
I believe there doesn't have to be a contradiction between setting a goal and using proper frames (time, resources etc) vs. creativity - at least as long as we have the right to work to our own judgement within these constraints. I often find that this last part is missing. You get all the obligations, but no mandate.

In Scrum, one part of the creativity is built right into the process: Understanding the different goals. In the product backlog you should enter business value. This way the team has direct contact with the user's intentions and can suggest alternative solutions that might be better, i.e. be creative.

Quality
If we're talking IT, I would like to use automatic testing, continuous integration and so on to ensure consistent quality. Add to this an annoying email whenever something breaks and you have a good start

Moreover, the tight feedback loop using sprint reviews and such should pick up on issues like usability etc. Basically make sure you have a product owner that can spend time on the project.

Also make sure that your "definition of done" reflects your quality goals. It has to be very clear what "done" means within your organisation.

Hur gör man för att bryta ner gigantiska projekt över flera plattformar i lagom stora bitar för att köra Scrum?
[How to break down into gigantic projects over several platforms into decent chunks so that Scrum can be used.]
Sven Nilsson, SAAB

Hi Sven,

Your question seems to revolve around two hefty issues: Breaking down the project into several teams that are likely to depend on each others and breaking down the actual work to fit into a sprint. Perhaps "platform" has a special meaning within your organization and how you work. I would have to know a bit more to answer you on that point.

Teams and dependencies
This is tough on several levels. First let's do the pure Scrum answer - Scrum of Scrums. This basically means that we organize our teams so that one person from each group "steps up" and forms a team that work across several teams. Dependencies etc can be resolved in this group, set up an encompassing backlog and work much in the same fashion as a Scrum Team. Jeff Sutherland, co-fonder of Scrum, has run this set up on 500+ persons. I would love to work on such an outfit.

However, and this is a big however, this is a major change for most companies. The ones I've met (medium size and up) just aren't geared to handle such an approach right now. The number of issues and obstacles with such a transformation can be huge. If you find yourself in such a situation then I suggest you adapt piece by piece and continuously strive to be ever more agile. This is hard work and will mean small improvements upon each other, but maybe no hyper productive state.

This is where I chicken out and suggest you should get someone with experience. Every company is different and needs their set of actions.
Hell, buy me lunch and maybe I can point you in a good direction

Breaking up the work into chunks
This takes creativity.

First: Try to slice the work so that you will touch base with most layers/parts. This way we can have a better understanding of what it means to deliver the whole system. Therefore make the slice as thin as you can while at the same time deliver something "valuable". This might mean you need to mock other components (code that isolates different components with dummy answers). This can be very beneficial since it becomes some sort of contract with other teams.

If the chunk still is to big then you might have to resort to cutting up the chunk into parts by level in your architecture. This of course has the drawback that we don't know what it will take to implement the parts we didn't do. Business value might be less, but we might be able to "prove" that we're on the right track. What is "proof" to your product owner?
If you can, stay away of doing tasks in your product backlog. In worst case people will be task-driven: "I've changed the registry, can I go home now?"

If we want to/have to promise the customer a delivery date. How can we do that when we don't analyse the whole project?
We are focusing on the next sprint?

How can we tell management how many resources we will need in one month?
/Patrik Johansson, Ericsson

Hi Patrik,

Several questions and perhaps the most common ones. Let's dive into them.

Analyze the whole project?
Why we can't analyze the whole project? Because, this is blatant lie for anything bigger than a trivial project. I apologize for my frankness, but I've never seen a project with a single version of the Gantt schema, a single time plan, with a known set of resources...
They always change. Always.

If you and your management can't agree that things change then you might have to go dualistic - outwards project manager old style, inwards ScrumMaster. It's not easy, but in time you will gain some victories, err deliveries, and you can move the agile thinking further up. Hard work, but it's better than doing waterfall all the way.

Your question is still valid. How do we commit on delivery if most things might change? The customer still needs to plan an ad campaign.
One trick is to reinterpret the "commit to delivery date" into agile terms. Get a stable backlog and learn the team's velocities. This way you can commit to the same "distance". Things will change, but (as always) it's down to making things fit into the available time.

For this work we need some iterations under our belt to know our velocity and hashing out what the product backlog should contain. Therefore, mix the project plan's you have do at the start with some "real work" like coding. Inspect and adapt. After while I think you will able to commit to management.

Focus on the next sprint?
You shouldn't just focus on the next sprint. The closest work (sprint) is fine grained and the most well understood. The further away (further down on the product backlog), the more coarse grained should the user stories be. No point in being specific, when they're far into the future.

Resources?
Which resources can you get? Go for full time members. Make sure can keep them. Look at the product backlog and calculate what you can get done with the people (velocity etc) that you got. The important part is to couple the resources with the goals. All too often your resources get slashed and the goal knocked into orbit - and not on the same day.
Get them in synch so that a change in one of them will affect the other.

I hope this helped.

Regards,
Björn

I'd love to get your feedback. See you in the halls and the Way Group booth.
Or you can just give your comments about the session here on this blog.

Developer's conference in Malmö, Sweden

Scrum and Agile development has started to rock the world within IT in various ways. A simple, but powerful process that seems to deliver. Developers and customer alike are happy. What's not to like about it? Still, there are signs of problems. Projects that claim to use Scrum do not reap the benefits, management looses patience with this new method and team members struggle to adapt an agile approach to their everyday work life. Why is this? And more importantly, what can we do about it?

The Therapy: Get off to a good start by directing the team with a careful set of good practices and a strict agreement that leaves little or no choice. The team can then, over a couple of iterations, gradually take command themselves.

Easy, but still so hard

Why then is Scrum so hard? Well, first of all it is fail-fast, meaning that any problems will be quickly apparent. There are no artifacts or process steps to hide behind. This is never easy. To this list we can add even more pitfalls that are just plain hard to deal with whatever method we are using.

Still, it does not explain why there seems to be so many simple mistakes with such a simple framework. One of the best examples is the failure to keep backlog that is prioritized and up to date.
- We're doing Scrum, but we're too busy to work on the product backlog!

That one has always short-circuited my brain. What can be more important than to decide what is important? This pattern of falling short and missing basic parts of Scrum even has a name - "ScrumButt". If you look in the sentence above you see why.

Nokia created a test to make sure the projects are "Scrum enough". These plain questions form a lithmus test that can be used to analyze most agile projects.

The Nokia Test
Iterative Development?

• Iterations - time boxed, less than 4 weeks.
• Software features - tested and working at the end of each iteration.
• Iteration must start before specification is complete.

Scrum (in Nokia's opinion)?

• You know who the product owner is.
• There is a product backlog prioritized by business value.
• The product backlog has estimates created by the team.
• The team generates burndown charts and knows their velocity.
• No project managers (or anyone else) disrupting the work of the team.

The questions seem to be on the easy side. It turns out that this might not be the case.
One online poll, Nokia Test by Practice , about the ability to achieve the Nokia Test level gave some interesting answers. In essence, only one out of four give themselves a full mark. Half of the groups report that management interferes, not directs but interferes.

Informal queries by Jeff Sutherland among his audiences around the world give an even worse number - something like 1 out of 10 pass the Nokia Test. Hand on your heart: Does your project live up to this?

A survey by VersionOne tries to dig deeper into the current state of the Agile affairs. Under the headline "Top barriers to adopting Agile" we read the top four answers:

1. Organisational culture, 45%
2. General resistance to change, 44%
3. Lack of people with experience, 42%
4. Lack of management support, 32%

Take a long look at these barriers. No technical stuff. Nothing specific to agile. Just people.

What do we know?

Learn to walk before you run. Greek Proverb.

We could be a bit closer to today's reality than these ancient words of wisdom. The Situational Leadership offers an approach to management that has been around since the 60's. These theories say that there is no single leadership style that is optimal for all people in every situation. Situational leadership talks about four different leadership styles that need to be matched with the follower. Basically, when leading someone who is passionate, but have no experience, you should adapt and give clear directives. As the follower evolves his or her ability to execute, so should the leader using less directives and more encouragement. In the end the follower is able to perform with only a stated goal and little or no directives.

If this holds true for people, could it not be true for teams as well?

The Shock

So we have problems making Scrum and Agile efforts to run. What is the conclusion? Could it be that we adapt our leadership style to the group? Make them walk before running?

This is counter intuitive, at least to me: To go agile, we first need to control. We should not let our teams and managements start off without some kind of directives. Give them a good deal, some practices that make Scrum work. Otherwise, we are just repeating the same old mistakes as the next-door agile project. Set up an agreement with both management and team so that they can evolve and experience Scrum for real. Just reading a book (or article) or listening to some good speaker alone does not help you out there in real life. You have to experience it.

Shocking as it may feel: We need to exercise some control over the Scrum project start so that it can evolve and self organize! But, and this is a big but, this has to be done with compassion and genuine involvement, otherwise we stand the risk of destroying what we are trying to build.

The Therapy

So what can we do? We need to work the team, management and often the whole organization.

The Team Recipe

So how do we get started? Here's a recipe for the team as it used at MySpace:
The Team

• Scrum training session for everyone
• Sprint 1 week long
• Definition of Done:
• Feature Complete
• Code Complete
• No known defects
• Approved by the Product Owner
• Production Ready
• Story Points
• Physical Task Board
• All-in-one Sprint planning meeting.
• No Multi-tasking, work in priority order.

Scott Downey, Agile Coach at MySpace

The idea of this recipe is to set a good practice right from the start. Instead of debating the details on how to work in an agile way, we can focus on getting going.

Everyone has to have a common Scrum understanding to start from, hence the training session. One-week iteration sounds really tight for any reasonably sized projects. The thinking behind this is to get as quick iterations as possible - the faster that the group gets going the better. Short sprints help in this aspect. We learn faster with more turnarounds.
This in turn affects the meetings where MySpace has elected to combine them all, except the morning meeting, into one single occasion. The time is just too short to in a weekly sprint.

MySpace's definition of done is a good and basic list. However, as a programmer I lack for instance continuous integration, pair programming and so on. These are not mandatory to have, but I see them as enablers for Scrum and supporting your agile initiative.

The next two on the list - Story Points and Physical Task Board - are really effective. If you haven't tried them out then please do. They work. Finally we have something that is a no-brainer to me even though I have great problems avoiding it, "No Multi-tasking, work in priority order". Context-switching is expensive, just avoid it. The priorities are in order, you just have to follow them. In the Reference box you find a link to Scott Downey's description of their therapy.

When the team matures, they should evolve and handle things more on their own and no longer need directives. At MySpace the group can "prove" that they are ready for prime time by reaching the following goals.

The Team Exit

• Hyper-Productive (>240%)
• Three successful Sprints consecutively
• Good business reason to change the rule

The Result?

So how does this work out? At MySpace all groups achieved exit. All, but one, improved after that. One group even achieved a whopping 1,650% improvement after just four months (16 sprints). At Jayway one of the teams used such a bootstrap technique, primarily on the technical side, and reached 800% after 3 months.

The second part of this blog can be found here: Scrum Shock Therapy, Part 2 where I will look into how to handle management and organisation with some recipes.

/Björn Granvik, Jayway

After several CTO assignments at various companies, Jeff Sutherland found himself at Easel corporation where he created Scrum together with Ken Schwaber. Ken later presented this methodology at the OOPSLA conference in 1995.

A dozen years later I find myself in search for a good spot where I can concentrate on my interview. Jeff and I sit down at a café at the Øredev conference. My computer has been acting up going blue screen on me twice that day. Time is short and I wonder if it's an omen.

Photo of Dr Jeff Sutherland.

Q: I hope I've done my reading. As far as I understand you've been a fighter pilot in Vietnam, an assistant professor in math and medicine and the CTO at nine different high tech companies. Moreover, you travel the world to teach Scrum. Does it take a resumé like this to invent a process like Scrum?

The key part of what makes Scrum work is "complex adaptive systems" and you need some understanding of this. Also you need to understand systems theory and how to optimise a system as a whole.

You do not have to be smart, but you have to thoughtful.

Q: How did Scrum start and from where does it draw its heritage?
Scrum and Lean both derive from complex adaptive system theory. Constraint theory is a narrow subset of complex adaptive systems thinking. Lean implementations of these theories have been created at companies like Toyota and Honda. Honda was actually more aggressive at this in the nineties, but then Toyota was more successful in the market. They also were more formalised and better known.

Q: Why does it work?
The first reason for all of this is important - super intelligent performance out of ordinary developers. In order to do that we need we need to set up a simple framework so that people will self-organize. The second reason is that there is no central point of control. The manager needs to back off. That is critical.

The third principle is behaviour that emerges in the team based on self-organizing in order to get max production.

The fourth reason is that the production is far higher for the group than any single developer. Groups actually achieve more than any lone bright developer.

These are the basic principles of a complex adaptive system. Any organization is a complex adaptive system. Software production is an organizational phenomenon. It is an empirical process that must absorb significant change in requirements, technologies, priorities, and people.

Q: You talk about managers, why is it important for them to back off?
A company needs to stop sub-optimizing locally. That is the thing that Toyota figured out, largely due to the efforts of Edward Deming and other Americans who trained the Japanese in WWII industry processes. Most managers just care about local performance. Managers have to work as a team to look across the whole spectrum. It is difficult for them. They're paid and given bonuses on a single level, not the company as a whole.

For instance, look at the General Motors manufacturing plant in Fremont, California. GM shut down that plant 1982 due to several problems, among them quality. Two years later it reopened as a joint venture between Toyota and GM. Toyota needed a plant in the US in order to avoid import restrictions. In an agreement with the labour union they offered reemployment to the former employees.

Two years later the NUMMI plant was producing over 200 000 cars per year and was twice as productive as the typical GM plant. The thinking they applied is the same as "lean production" where Toyota makes the team responsible to figure out the best way to solve the problems and achieve the goals. NUMMI later won several awards for quality.

"Your battery is running low", declares my machine solemnly. Jeff and I make a run for a power outlet and find one on the second floor. My power chord is too short so we stack up real close to the wall. It will do. More than that actually, it feels like we really are working together on this interview.

Q: Some parts of Scrum are frightening, like making the group totally responsible for an implementation. How come people in management buy this?

The best answer I can give you, is the discussion I had with my first CEO when I introduced the ideas of Scrum. We had a critical project with a hard deadline. It was a question of survival.

The CEO expected a Gantt chart [ed: a timeline specifying activities and their dependencies, showing the progress of a project].

• How many Gantt charts have you received that actually worked? I asked the CEO.
• None, was the short answer. In my 25 years I haven't received a single one that was correct. I asked him if a 100% failure rate wouldn't make it foolish of me to give him one.
• What are you going to give me, he asked?
• Working software. Decide for your self to if we're on track. In return I made him promise not to disrupt the team before the deadline. At this point he started to worry.

I said that at the end of the month he could change everything he wanted, but not during that month.

• If you do this, you will get significant more code with better quality.
• I can live with that. Let's do it.

Scrum was born in that minute.

It's a basic contract between management and team. The team delivers twice as much stuff with twice as good quality (minimum) and the management supports the team in their efforts to self-organize and self-manage. If either party does not fulfil this contract then it will fail.

Q: Doesn't Scrum fail?

Scrum itself is a set of rules that causes self-organization if certain things are put in place. It's like the rules of soccer. Does soccer fail?
With Scrum you will fail significantly less.

The Standish Group \[ed: a company that specializes in IT studies\] reports that project in the range 3-6 million dollars with a traditional project plan fail more than 85% of the time. Scrum does a lot better than that!

Q: What are your top three tips on using Scrum?

Bas Vodde was the Lead trainer for Scum Master at Nokia Siemens Networks. He has the shortest set of questions to identify a dysfunctional Scrum team.

You must have iterations of 30 days or less with working software tested at system level at end of each iteration. You need an agile specification, just enough for programmers to start for that iteration.

You need a product owner who orders activities by business value of the feature set. The team estimates the product backlog.

And lastly, the team needs to monitor progress with a burn down chart and know its velocity at the end of the iteration.

Q: Finally, what is in the future for Scrum?

Hyper productivity on an economic and marketing level where companies consistently out perform their competitors. Scrum is a disruptive technology that will alter markets. Companies doing Scrum well will grow rapidly in both size and profitability and dominate their waterfall competitors. There is already a brain drain going on where the better developers are fleeing from waterfall companies to Agile companies. This will hasten the demise of traditional projects.

Q: Is this a shameless plug?
Oh no, we have achieved it in several departments at my company \[PatientKeeper\]. It is doable. Other companies I work with right here in Scandinavia are doing it as well.

Going away from this fast paced interview, I still have the feeling that I could have done more with my life. If Jeff can do it, then so can I ...I hope

Originally published in JayView.