Experimentation: A Pattern for Affecting Measured Change

Note: This post is based almost entirely on one of the things that I took away having attended a talk titled “Agile Coaching Tips” by Rachel Davies held at Skills Matter but perhaps overlooked at the time. As such, the originating ideas are not mine and instead, Rachel’s. This post is more something that I have constructed as a result of my own recent thinking, primarily driven by my interest in the kanban community and talking with people like Benjamin Mitchell which has led to me growing more appreciative of the value of qualitative and quantitative measures. This in turn has led me to revisit one of the subjects that Rachel introduced in the talk she gave, “Making Change As An Experiment” (See between 38:36 and 42:09 in the video I have linked to.)

Why Experiment?

Rachel proposed, and I agree with her, that talking about change with somebody can often result in them becoming wary whereas using nomenclature, in this case talking of experimenting with something different, instead of using the word change can make those people involved and or potentially affected feel more comfortable. In that respect, I like the idea of using experimentation as a means to effect change within what I do as primarily, it is considering the feelings of the people that are involved in that change up front. So often in my experience that not insignificant factor can be overlooked.

Secondly, when thinking about experiments that are run in a laboratory, it is accepted that you may fail to meet your objective; you’re questioning whether you can change some thing’s existing state. In relation to that. because you had a hypothesis that were looking to prove or disprove, you will have had to measure the outcome of any actions you performed and the conditions they were attempted under.

With those 2 points in mind, I’ve taken Rachel’s original idea and noted how I think it could be used as a pattern to bring about measured improvements.

Experimentation As A Pattern

1. Understand the Current State

The first step in my opinion, is to understand what it is you are attempting to achieve overall (your vision). From there, the next step is to understand the first step (perhaps the first impediment you want to remove or the first improvement you want to make otherwise) you want to take you towards your vision. Techniques such as Retrospectives, The 5 Whys and A3 Thinking amongst others, can help here but above all, the one tool that I keep coming back to currently is Force Field Analysis.

2. Hypothesise

The received wisdom is that hypothesising should be the process that is used to understand the goals of the experiment with which I would agree. Important to me here though is that the goal should be something that can be clearly articulated to anyone, i.e. in the case of a development team, those outside of the team from a non technical background too. It is possibly worth noting the goals too, this is not something that I have personally done but I do see that it might have brought value to previous change efforts.

It is particularly key here that if the experiment you wish to run is related to a practice within a team, everyone within the team understands why you are wanting to introduce a new practice or change an existing one. Furthermore, it is important in my opinion that everyone in the team has an opportunity to discuss how you might change. This is important so that everyone is in a position to question the validity of their own and that of their peers approach on a daily basis throughout the experiment. In my opinion, it is for all those involved in the experiment to ensure that it runs as defined.

3. Plan The Experiment / Decide On How To Measure Effect Of Change

The idea that any change in practices or process should be measured is, as I describe above, one of the most important aspects of why I think that experimentation is a useful way of looking to introduce change. Based on the findings from Understanding the Current State and the goals derived from the work done whilst Hypothesising you should be able to define either quantitative or qualitative set of measures. It would be ideal in most cases if you were able to identify measures that you already have in place so that you can compare and contrast, but not essential. In the absence of existing measures, I would suggest that it is worth extending the duration of the experiment so that you can observe change over time.

By way of a simple example, let’s imagine that you or the team you work within notice that features are batching when they get to the tester and perhaps even that in a lot of circumstances, the features end up having to be reworked. The overall affect of this is that the team is unable to concentrate on other work and that the features themselves take longer to be delivered unnecessarily. Some measures you might consider are: The amount of re-work, for example the number of times that a feature is passed between a developer and a tester; the number of bugs raised against a feature whilst it is being developed and finally the Cycle Time for features.

Finally, decide how long the experiment will run for. As I mention above, be mindful of how long you will need to gather data that will be statistically significant. If you already have a good set of data to compare any new metrics with then you can (though not necessarily should) use a shorter period of time and conversely, I would suggest that in the absence of any data to start with you should consider leaving the experiment to run for as long as possible.

4. Run the Experiment

As I’ve suggested in the sections above, I think the most important things to be concerned with whilst running the experiment are as follows: Have a clear vision for what you’re trying to achieve and be able to articulate a set of discreet measurable goals for the experiment. Ensure that, on a best endeavours basis (acknowledging that some people will always be difficult to be persuaded of the benefits of one practice over another. See Diffusion of Innovation, the Five Stages of the Adoption Process) everybody in the team is signed up to the experiment and know that they have a role to play within the experiment; time box the experiment so that those that aren’t necessarily persuaded of its virtues know that it is for a set time and furthermore to ensure that the amount of energy expended on an invalid hypothesis is minimised; take time at regular intervals to ensure that you remain true to the practices or process improvements that you are trying to bring about, without doing this it will invalidate the experiment.

Finally, at the end of the experiment take another opportunity to review what happened throughout the experiment, not just from the perspective of the measures that you have in place but from the perspective of those involved too; how did everyone involved feel the experiment went and what benefits did they feel it brought? In addition, my advice would be to be as transparent with all affected parties as possible with the measures that you are using and the results that you publish. I think this is important if for no other reason than it might generate further discussion, particularly outside of the team.

5. Iterate

It’s important to recognise that what is learnt from the experiment is the most important thing, as opposed to whether or not you succeed in meeting your goals. If you successfully met your hypothesis that’s great, but if you didn’t, ask yourself whether there is still value in what you are trying to achieve? If there is, can you change your previously defined practices to derive the value you are hoping to? Importantly too, are the measures that you defined still valid, are there ones that could be replaced with others, alternatively are there any that can be added to allow you to better understand performance against your goals?

In Conclusion

I am sure that the using experimentation as a pattern for change is something that others including Rachel have suggested before and so what I am proposing is nothing new, it is also without doubt that making change using the technique that I suggest adds overhead to the change process itself. I do firmly believe though that using experimentation as a technique to introduce change is beneficial owing to its focus on using measurement which allows those involved to be objective in their appraisal of any impact and that lastly, it considers the people involved up front thinking about the fact that when confronted with it, most people are wary of change.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s