Monthly Archives: July 2010

Experimentation: A Pattern for Affecting Measured Change

Note: This post is based almost entirely on one of the things that I took away having attended a talk titled “Agile Coaching Tips” by Rachel Davies held at Skills Matter but perhaps overlooked at the time. As such, the originating ideas are not mine and instead, Rachel’s. This post is more something that I have constructed as a result of my own recent thinking, primarily driven by my interest in the kanban community and talking with people like Benjamin Mitchell which has led to me growing more appreciative of the value of qualitative and quantitative measures. This in turn has led me to revisit one of the subjects that Rachel introduced in the talk she gave, “Making Change As An Experiment” (See between 38:36 and 42:09 in the video I have linked to.)

Why Experiment?

Rachel proposed, and I agree with her, that talking about change with somebody can often result in them becoming wary whereas using nomenclature, in this case talking of experimenting with something different, instead of using the word change can make those people involved and or potentially affected feel more comfortable. In that respect, I like the idea of using experimentation as a means to effect change within what I do as primarily, it is considering the feelings of the people that are involved in that change up front. So often in my experience that not insignificant factor can be overlooked.

Secondly, when thinking about experiments that are run in a laboratory, it is accepted that you may fail to meet your objective; you’re questioning whether you can change some thing’s existing state. In relation to that. because you had a hypothesis that were looking to prove or disprove, you will have had to measure the outcome of any actions you performed and the conditions they were attempted under.

With those 2 points in mind, I’ve taken Rachel’s original idea and noted how I think it could be used as a pattern to bring about measured improvements.

Experimentation As A Pattern

1. Understand the Current State

The first step in my opinion, is to understand what it is you are attempting to achieve overall (your vision). From there, the next step is to understand the first step (perhaps the first impediment you want to remove or the first improvement you want to make otherwise) you want to take you towards your vision. Techniques such as Retrospectives, The 5 Whys and A3 Thinking amongst others, can help here but above all, the one tool that I keep coming back to currently is Force Field Analysis.

2. Hypothesise

The received wisdom is that hypothesising should be the process that is used to understand the goals of the experiment with which I would agree. Important to me here though is that the goal should be something that can be clearly articulated to anyone, i.e. in the case of a development team, those outside of the team from a non technical background too. It is possibly worth noting the goals too, this is not something that I have personally done but I do see that it might have brought value to previous change efforts.

It is particularly key here that if the experiment you wish to run is related to a practice within a team, everyone within the team understands why you are wanting to introduce a new practice or change an existing one. Furthermore, it is important in my opinion that everyone in the team has an opportunity to discuss how you might change. This is important so that everyone is in a position to question the validity of their own and that of their peers approach on a daily basis throughout the experiment. In my opinion, it is for all those involved in the experiment to ensure that it runs as defined.

3. Plan The Experiment / Decide On How To Measure Effect Of Change

The idea that any change in practices or process should be measured is, as I describe above, one of the most important aspects of why I think that experimentation is a useful way of looking to introduce change. Based on the findings from Understanding the Current State and the goals derived from the work done whilst Hypothesising you should be able to define either quantitative or qualitative set of measures. It would be ideal in most cases if you were able to identify measures that you already have in place so that you can compare and contrast, but not essential. In the absence of existing measures, I would suggest that it is worth extending the duration of the experiment so that you can observe change over time.

By way of a simple example, let’s imagine that you or the team you work within notice that features are batching when they get to the tester and perhaps even that in a lot of circumstances, the features end up having to be reworked. The overall affect of this is that the team is unable to concentrate on other work and that the features themselves take longer to be delivered unnecessarily. Some measures you might consider are: The amount of re-work, for example the number of times that a feature is passed between a developer and a tester; the number of bugs raised against a feature whilst it is being developed and finally the Cycle Time for features.

Finally, decide how long the experiment will run for. As I mention above, be mindful of how long you will need to gather data that will be statistically significant. If you already have a good set of data to compare any new metrics with then you can (though not necessarily should) use a shorter period of time and conversely, I would suggest that in the absence of any data to start with you should consider leaving the experiment to run for as long as possible.

4. Run the Experiment

As I’ve suggested in the sections above, I think the most important things to be concerned with whilst running the experiment are as follows: Have a clear vision for what you’re trying to achieve and be able to articulate a set of discreet measurable goals for the experiment. Ensure that, on a best endeavours basis (acknowledging that some people will always be difficult to be persuaded of the benefits of one practice over another. See Diffusion of Innovation, the Five Stages of the Adoption Process) everybody in the team is signed up to the experiment and know that they have a role to play within the experiment; time box the experiment so that those that aren’t necessarily persuaded of its virtues know that it is for a set time and furthermore to ensure that the amount of energy expended on an invalid hypothesis is minimised; take time at regular intervals to ensure that you remain true to the practices or process improvements that you are trying to bring about, without doing this it will invalidate the experiment.

Finally, at the end of the experiment take another opportunity to review what happened throughout the experiment, not just from the perspective of the measures that you have in place but from the perspective of those involved too; how did everyone involved feel the experiment went and what benefits did they feel it brought? In addition, my advice would be to be as transparent with all affected parties as possible with the measures that you are using and the results that you publish. I think this is important if for no other reason than it might generate further discussion, particularly outside of the team.

5. Iterate

It’s important to recognise that what is learnt from the experiment is the most important thing, as opposed to whether or not you succeed in meeting your goals. If you successfully met your hypothesis that’s great, but if you didn’t, ask yourself whether there is still value in what you are trying to achieve? If there is, can you change your previously defined practices to derive the value you are hoping to? Importantly too, are the measures that you defined still valid, are there ones that could be replaced with others, alternatively are there any that can be added to allow you to better understand performance against your goals?

In Conclusion

I am sure that the using experimentation as a pattern for change is something that others including Rachel have suggested before and so what I am proposing is nothing new, it is also without doubt that making change using the technique that I suggest adds overhead to the change process itself. I do firmly believe though that using experimentation as a technique to introduce change is beneficial owing to its focus on using measurement which allows those involved to be objective in their appraisal of any impact and that lastly, it considers the people involved up front thinking about the fact that when confronted with it, most people are wary of change.


Output from the Session I Proposed at the UK Agile Coaches Gathering, 2010

At the recent UK Agile Coaches Gathering (held at the magnificent Bletchley Park), the theme for which was Helping People Grow,  I proposed a session titled “Leaving a Legacy, How Do You Leave an Environment in Which a Team Can Continue to Grow” for one of the slots during the Open Space. There follows a summary of the session and then the notes that I captured, augmented from memory where possible.

To start I thought that I should provide an explanation as to why I proposed the session. In recent months, I have heard an awful lot of stories about change efforts that have been fantastically successful but when one thing is changed, perhaps a person leaving an organisation or the structure of the team changed, the momentum from that effort goes too and in some circumstances then leads to those involved moving backwards to their previous state. Furthermore and whilst I have never held the title of Agile Coach, from personal experience there seems to be a ceiling within organisations which when reached by a team, change becomes more difficult; typically because they are needing to challenge hierarchies or the received wisdom within that organisation.

Prior to the session, I did some preparation and thought about how as a coach you ensure you leave a legacy I came up with the following:

As a coach, you need to help a team learn how to change but that at the same time, you need to help an organisation learn how to allow that team to change

I then specifically listed a couple of points for each of Team and Organisation that I thought would ensure a legacy of a culture of continual change. From a Team perspective, I noted that I thought primarily they need to be taught how to be introspective and secondly and linked to the the first point, you need to also provide them with a tool set to facilitate their continued change. Form an Organisation’s perspective, I noted that you would need to help them understand their purpose, to help them understand the value they want from that team and lastly, how to communicate both those things to the team.

Once I’d set the scene with the above, we moved in to the discussion the notes of which follow:

  • The team need to get to a point where they have the courage to stand up for what they believe in
  • A coach should not become too embedded within a team
    • Doing this reduces the team’s reliance on the coach
  • Internal learning within a team leads to a team encountering blockers outside the team
  • Teams can be too busy building stuff to have a vested interest in removing the impediments outside their team
    • The above is most apparent in Scrum when the Scrum Master is delivery focussed (e.g. a Project Manager or Lead Developer) as opposed to being focussed on Process Improvement
  • It is key that the team needs to feel as though they own the process
    • Process Smell: 1 person owning the process
  • A lot of the examples given by teh attendees centred around organisations that were hostile to any change in the first place
  • Even if you have somebody that really cares about improving the team, people, including those outside the team need to experience the benefit before they will actually assist in further improvement
  • (Team reaching a ceiling) Awful lot of organisations that can’t articulate their vision
    • Means that they struggle to say no to additional work load as a consequence meaning that the org is just busy, no slack
    • No tools to resolve conflict
  • It was noted that is is harder to help an organisation be able to define and communicate their values when they have existed for a long time
  • Is the answer, in terms of leaving a legacy that you leave a team that have a clear set of values that they believe in?
  • To test whether in a coach’s absence the team is still being successful the following might be considered to be good indicators:
    • Are they still working as a team
    • Are they changing things
    • Are they still focussing on quality, internal and external
    • Are they learning
    • Are they innovating
    • Can it still be said that they have confidence
    • It was noted that the tests above are valid at an organisational level too
      • e.g. Kanban boards spreading outside of IT departments
  • Differentiating factor? If teams or the organisation is stressed?
  • See image below, showing how trust of a team is derived by others, thanks to Rachel Davies
    • Reciprocal though is that the others need to know that you as a coach know what you’re doing
  • We should accept that change efforts are incredibly fragile. e.g. When a new leader enters an organisation they will always want to make their own change
    • Credibility comes from evidence: gut feel is not enough
  • Is transparency part of the answer?
  • Are the goals of the team and the organisation aligned? If not, changes in practices will never flourish
  • Is the thing that is missing conversation? In a hierarchical structure is the ability to hold conversations across all levels there?

Image: Trust Equation

Thanks to all the participants who came along to the session and made it what it was, I enjoyed it and learnt something too, hopefully the same can be said for yourselves.