It’s often said that Scrum is an empirical process so the question is, is that true and if so what does that really mean?
Means that the information is collected by observing, experience or experimenting.
Is used for handling processes that are complex and not very well understood.
So let’s look at how it’s defined.
An empirical process is seen as a black box and you evaluated it’s in and outputs. This is done by defining checkpoints that should occur at predefined points.
So what do we have in Scrum?
We have the sprint as the black box in the meaning that the work that goes on in the sprint should not be disrupted. We have backlog items that’s not started as input and finished backlog items as outputs.
As checkpoints in Scrum we have daily scrums, sprint planning meeting, sprint review meeting and sprint retrospective. Since each sprint is time boxed, the checkpoints will always occur at a predefined time. Inspection is done at each checkpoint so you can adapt the process with the information collected through observation, experience and experimenting.
Another important aspect of an empirical process is that the process model (the work we do) is directly derived from the input and output data. So there should not be any possibility to change how the work is done in the process from the outside. That’s why we don’t add more backlog items to a sprint than the last sprint proven that we could do even if demands for this is coming from higher management. We need to be true to the truth and protect the Scrum process.
So now we know that Scrum is an empirical process but why is that so good? Why not use a defined process? It is much more predictable.
To answer this question we need to look a bit deeper into what we are facing when developing systems and what an defined process is.
A defined process is derived from first principles, meaning in science that it needs to adhere to laws of nature and other fundamental and well defined laws.
Another more understandable description is made by Ken Schwaber in Agile Software Development with Scrum.
“The defined process control model requires that every piece of work be completely understood. Given a well-defined set of inputs, the same outputs are generated every time. A defined process can be started and allowed to run until completion, with the same results every time.”
This is theoretical true but in the real world there will be a variance but held to a minimum. There is also no need for feedback because you will always know what to do next.
So here is a variance of Ken Schwaber definition.
“A defined process is an amount of tightly coupled steps where the output from one step is the input to the next step and where no observation or evaluation of the output is done to feedback to the process. A defined process when started will run to the end without any checkpoint. The output from a defined process should always be the same or with little variance given the same input to the process.”
The process could be very complex but needs to be well understood before execution.
This is typical for a “waterfall” like process where everything needs to be defined in the beginning and then you follow a cookbook like recipe and you should get an expected result. So if system development should fit this process it should be very predictable, non creative and should not demand any thoughts during the execution phase.
This is not how I or many of my colleagues think about system development. For us it’s a highly creative and intellectual process that demands a high degree of thinking. We need to expect the unexpected and be able to act on that in a complex environment.
The complexity in system development is often from 3 major areas, requirements, technology and people.
– Requirements are often not that well understood, badly communicated and missing.
– And people that can change anything in the other two so that means that we have a very unpredictable complexity.
So a waterfall like process doesn’t look that good for system development but what about all the other iterative process that are out there. Shouldn’t they function as well as Scrum?
To answer that question, we need to look at each of these processes and see if it is an empirical process. Most agile processes use time boxing and contain feedback checkpoints. Others lack regular quality checkpoints that provide information to adapt the process. They also do not protect the process and will corrupt the data by outside involvement. Many of these processes are often more waterfall and/or defined process than an empirical process. Just because you use iterations doesn’t mean that you have an empirical process.