21 May 2009

Failing the Iteration vs. Cutting Scope (Part I)

Recently, I had an interesting (and polite) debate with my friend Bob “Agile Bob” Hartman of Agile for All. It started when I heard him tell a team to "never cut scope from an iteration."

Something--probably my general principle about "never being an absolutist"--bothered me about the statement. We had a private discussion later, and we agreed that I could summarize here, and we can openly debate, if necessary, right here on my blog. And you are welcome to weigh in. It could turn into a hotly debated topic, or just a tempest in a teapot.

I'm also breaking this into multiple parts, because I think there are three layers to the decision:
  1. The choices available regarding iteration-wise scope-cutting.
  2. How to use process and practices to help you make the right choices.
  3. The process-coach's role in these choices.
A caveat: For teams who are just starting their transition to Agile, coaches who disagree or who simply contradict each other are confusing at best, and possibly counterproductive. When people pick up a new skill or practice, they want to know the right things to do to get good results (See Shu-Ha-Ri for some background on that).

A polite disagreement with a colleague with as many years of Agile-related experience was no surprise to me. I suspect that we may both learn something. If you’re new to Agile, well…you may learn something too, just have some antacid handy. And rest assured, I believe Agile Bob and I agree on the core principles involved.

“We’re Behind!”

Let me set up a common scene for you: The team is half-way through a two-week iteration. To make the math easy, let’s say the team’s velocity was previously measured at 20 points. The last story scheduled for this iteration is called “Plugh” and it’s been estimated at 4 points.

On that fateful 6th day of the iteration (we try not to count weekends!), the team sees that they’ve completed 8 story points. The burn-up graph clearly shows the trouble ahead: At this rate, the team would complete 16 points, with a shortfall of 4 points. Plugh would not get done.

What are the team’s options? Two possible options come immediately to mind:

A. We acknowledge this and keep going.

This course of action results in two possible outcomes: Perhaps Plugh fails to get completed, or the team works (perhaps heroically) to get all commitments finished. (This is perhaps the “default” Scrum approach. This is also Agile Bob’s preferred choice.)

B. We ask the Product Champion (Scrum "Product Owner" or XP "Customer") to drop Plugh (or split it, but we’ll avoid shades of gray for simplicity’s sake).

Again, two similar outcomes: The iteration completes without Plugh, or the team finishes early and pulls Plugh back in and completes it. (This may be the “default” XP approach.)

You’ll notice that both options end up having the same two possible outcomes, at least on the surface. Either Plugh is done, or it’s not. The reality of whether or not Plugh gets completed in this iteration is not obviously affected by our choice.

The choice could affect the outcome of the iteration indirectly: Will the team choosing Option A work effectively, keeping quality in mind, if they opt for a heroic effort? Will the team choosing Option B remember to ask for more work if they finish early?

The choice could also have longer-term effects on the quality of the release, and team morale and trust. To me, those are the more significant considerations for long-term team performance.

Agile Bob’s Argument

At the heart of Agile Bob’s argument for Option A is a legitimate concern over the message we (management and coaches) are sending the team, and what habits the team is forming. In the course of our brief conversation, Bob put forth two arguments, one in favor of Option A, and one against Option B:

In support of Option A, Bob explains that no one likes failure: Let the failure to implement Plugh be examined in the retrospective, and the team will adjust to avoid over-commitment in future iterations.

Cutting scope (Option B) relieves the team of responsibility for Plugh for two weeks. In the next iteration, they may complete Plugh, but the “Plover” story may suffer. If this continues unabated, scope is slowly sliding away from the release plan due to the habit of sheer laziness.

I agree with both of these as concerns, or considerations. The issue I have with this is the suggestion that one choice works as an absolute rule for all teams. Perhaps Option A is the right choice for his team at this time. But I see enough value in Option B to resist choosing one over the other prior to considering more context. I want to encourage the team to reflect honestly on their own habits and tendencies.

Both of Agile Bob’s arguments can, by my direct experience, be falsified. I’ve worked with (and on) teams who handled iteration-wise scope-cutting as a rare but useful technique, and I’ve worked with a number of teams who have made a bad habit of failure, at least until the real cause of dysfunction was identified and corrected.

“A Habit of Failure?”


First, let's not confuse this with a good habit of failure. The technique called "fail fast" is inherent in Test-Driven Development, and in Agile processes: If you're headed in the wrong direction, you want to know it as soon as possible, ergo "fail fast!"

Perhaps a more accurate description of what we're talking about here would be: Making a habit of over-promising and under-delivering, without any real learning or improvement.

Any habit of over-committing is a habit of failing the iteration.

Consider "velocity" (the number of story points completed per iteration). Anything that artificially increases velocity can become a habit of over-commitment.

I recall a conversation with a team after their first iteration. They had scheduled 20 points but completed 12. I suggested that they simply schedule 12 points for the next iteration.

"But that was our first iteration." In the next iteration, they scheduled 20 again, and completed 12.

"But the architect was out sick for half the iteration!" They scheduled about 20, and got 12.

"But the test-system was out for repair..."

The PM looked at me and asked, "How am I supposed to plan for all these [expletive deleted] surprises?" and I began with "Well, your velocity is 12..."

Any time a team makes a plan that assumes there will be no unpleasant surprises, or the team talks about "increasing the velocity," or tries to "get some points" for a partially-completed story, there is already a subtle habit of failure in place.

And that's when everyone is trying to do the right thing! Sometimes we're not so fortunate. Many teams in transition from a waterfall-flavored process to an Agile-flavored process will discover that they've developed a taste for failure. More accurately, a fear of success. A single fearful person can do some hefty damage to the organization, whether or not it's intentional.

Comfortably Numb

This unconscious sabotage arises from a small part of the team (perhaps one or two people) who have not yet reached a sufficient comfort-level with all of their“Agile Transition” responsibilities. People who are used to doing something their way (e.g., “I must see all the requirements before I can begin design”), and truly do not realize their negative impact on the team or the organization, and thus on themselves.

So these individuals think (or feel): “The sooner this 'Agile' stuff fails, the sooner I can get back into my comfort zone."

Using our example of iteration failure, we can add more detail: "If we let the iteration fail, we will then commit to less next time, and I can have more time to do my work using the [inefficient and cloaked] practices I’ve been using for years. What’s the hurry? We’re not really releasing in October. They’ll slip the date, just as they’ve always done…”

This same argument could be used with either of our options, A or B, but there’s a difference in perception between a tool for "sullying the opinion of Agile within the organization" (a possible perception of Option A), and a tool for "relieving some of the temporarily heightened pressure of an over-committed iteration" (a possible perception of Option B).

Perhaps it's just my brain's preference for bizarre analogies, but Option B strikes me as a "safety valve" and Option A looks more like a "toilet about to overflow." Either can be handled responsibly or otherwise, but as far as analogies go, I have my preferences...

Whatever their perceptions or analogies, the activities of these subtle subconscious saboteurs cannot hide in a highly-iterative agile environment. We will notice the effects in the stand-up meetings and the retrospectives.

"So...Fire Them?"

On one of my favorite XP teams, we had an intentional agile-saboteur: I have no doubt that he consciously knew what he was up to.

This gentleman was a senior developer who had been working on the legacy product for a decade. He was nearing retirement age, and would be able to pull in a comfortable pension. His expanded team was rewriting the system in Java using the XP process.

Can you imagine his feelings? He had poured his heart, soul, and career into "The Product" only to have a team of young know-it-alls come in and rewrite it in a language named after a beverage using some newfangled "Extreme Snowboarding" process (where no one would leave him alone long enough to think). He probably resented the fact that he was going to be forced to learn a whole set of new techniques that he would never use again. And if he didn't play along, he was worried they would find some reason to fire him, and he would lose his pension.

(He never expressed any of this. Did I mention that being an "Agile Coach" requires a small amount of empathy?)

More fear, rather than making transition easier, makes people do stupid things. Like sitting with his arms folded, literally rolling his eyes (and letting out an occasional loud sigh), but otherwise saying not a single word during a two-hour pair-programming session.

Right now, you think I exaggerate! Nope. I endured two hours of sighs and eye-rolls, I kid you not! I was trying to outlast him. (In hindsight, it was the wrong strategy.) At first I tried very hard to engage him in conversation, to bring out some of the wisdom locked away in his brain, and (frankly) to get him to like me, just a little. After that I just test-drove my code, and perhaps looked a little closer at the screen whenever I heard a sigh.

Unsalvageable relationship? Fire him? Actually, it was the PM who mentioned this option during a private conversation about the man, but only to dismiss it because he was very well liked and respected in the organization, and he knew the system better than she did.

And then the lightbulb appeared above our heads, hers and mine, simultaneously.

Legacy code, for all its faults, contains a lot of business knowledge (obscured and obfuscated, but it's in there somewhere). Giving this guy the "XP customer" role gave us access to that expertise, gave the PM more time to do her work, and gave him a heightened sense of participation and control over the robustness of the product.

Actually, he and the PM shared the customer role, because she would weed out frivolous stories and prioritize the rest. But he would frequently point out business rules that everyone else had forgotten, and we'd quickly write down another user-story. Sure, we had to put up with the occasional snarky "Well, obviously you've forgotten about this failure case..." (Have I mentioned that an Agile Coach has to be part diplomat?)

"Just Give Them Whatever They Want?"

We should try to give people the benefit of the doubt. When the organization is transitioning to the win-win-win game of Agile software development, it can take a while for people to see the benefits to themselves as employees and teammates (one of the “wins”). People aren’t trying to hurt themselves when they resist change. They’re mostly either confused, skeptical, or tired of changes that have generated no tangible improvement. We have to be clear about the potential benefits to employees, teams, and the organization; and we have to set reasonable expectations about our progress towards particular levels of agility.

We have to look for the right changes for the right person, at the right time. Find out what motivates negative behavior, and refocus that nervous, destructive energy towards positive results. This will result in a much happier and productive team.

I’m not one to promote coddling the team, or limiting constructive feedback, during a transition. Imagine if we chose Option B and the Product Champion simply said “Every iteration is a success! You completed less than last time, but you did a great job!” Ack! That's a watered-down definition of "success" that will lead to dysfunction. We don't congratulate the team for cutting iteration scope, but neither do we punish them by similarly overusing the word "failure." Let's save the word "failure" for real, impending failures.

Let's give the teams the tools they need, and help them decide when and how to use them. One of those tools is velocity. Velocity is measured to plan, and to diagnose, like body temperature. We neither scold nor reward our children for having a fever. Fever is a symptom, and “my kid is sick” is just today’s reality.

Besides, the team is not a collection of people playing the role of children; with managers, executives, and coaches playing parents. The Agile team is the whole team of professionals (including developers, testers, tech writers, analysts, managers, execs, coaches) who work together to identify problems early and to adjust accordingly.

Stay Tuned...

That's enough to start with. Part II (working title "Making the Lean Choice") is just a week or two away. Here's further related reading, from people I really trust:

Agile Bob's blog post mentioning iteration-wise scope-cutting as a form of mediocrity.

Jim Shore's blog post on the three forms of success. He talks about sabotage, as well. Jim despises mediocrity, too.

Esther Derby wrote an article for Better Software Magazine that talks about self-organizing teams. It seems relevant to this post.

05 May 2009

First Steps Towards Agile

Yesterday I received a note from a fellow NAU alumnus who found me on LinkedIn and asked where to start with a small-team transition to Agile.

Aside: I sent a reply; mostly an unedited stream-of-consciousness, but I thought it was good enough to post here. I'm not sure I'd give the same advice to everyone, but it seemed to make sense for his circumstances as I understand them. I got the sense that he's in need of a do-it-yourself Agile transition, perhaps due to his remote location. (Though I suspect he knows that I'd travel to his particular region of the country without hesitation. I give discounts for certain personally-nostalgic locales! :-)

Feel free to comment with other suggestions. What would be the first two or three steps you would take in his circumstances?

The Letter

I was wondering if you wouldn't mind helping me? I lead a team of developers, analysts and a webmaster...read a lot about agile and think it's potentially a great way to go. How do I even get started?

To date I started doing some small team-ups for projects we have. I have assigned tasks to different team members. I know this isn't agile but up until this time our team developed everything silo'd. They all had their own projects and never worked on someone else's work... So I figured it would be a good start to at least expose them to teaming...

From here I am really not sure what to do nor do I deeply understand the concepts in agile. I have read a ton on the internet but there is nothing like experience to really help.
Effectively, I suggested he start with a retrospective...

My Reply


I think I have a great answer for you, but I want to say a few words beforehand so you don't think I'm just handing you more to read.

I was glad to hear that you're getting them used to working together as tiny teams. I suggest that small organizations try to partition the teams by separate code-bases. Products tend to map one-to-one with a code repository, and teams work best when they map one-to-one with a product. People often try to have multiple teams with their hands in one code-base, and they often discover that they're inadvertently impacting each other. If someone can impact your team, that someone is effectively on your team. But I'm wandering into dangerous territory since I'm not familiar with your set-up.

When a team starts to do things differently, it is difficult to figure out what to do first, or to just jump in with both feet (or head-first, perhaps?). So keep that in mind: Ultimately, you have to decide what's the right first step for change.

It may sound odd, but the first thing that needs to be in place is a mechanism for getting anonymous feedback on how the process changes are being experienced, and what the team's biggest problems truly are. I would start with a "retrospective" meeting. I suggest that teams do this every two weeks. It doesn't have to take more than an hour:

1. Get everyone in a room together and have them write troubles on sticky (Post-It) notes and have them post these stickies randomly on a wall. This gives people the ability to anonymously voice what they see as the biggest pain points. No limit to the amount of troubles.

2. Then have them go up in small groups and rearrange the notes. Not in rows/columns, but in clusters by how closely related the troubles are to each other. Duplicates are very close, similar topics farther, and wild suggestions farther. Let the stickies clump organically, so to speak.

3. Then look for the biggest cluster. Help the team find a way to fix that.

If your team is too small to distinguish a cluster after step 2, replace step 3 by giving each person three stars or colored sticky dots to use to vote for their top three problem areas. We usually require that they cannot vote for the same problem more than once. You'll probably find that they all agree on one or two major problems.

That's one of the first things I would do with your team. But since I don't know what they would come up with as the biggest problem, I can't suggest further action except to go out and obtain a copy of The Art of Agile Development by James Shore and Shane Warden, and read it as though your career depends on it. I worked closely with Jim Shore in 2001 and he knows how to solve problems. Most of what they have recorded there is stuff that we were doing in 2001, and it's still some of the best "agile" advice, all collected in one place.

Lastly, be aware of the two primary reasons why agile projects fail, and how to prevent them:

1. The Product Champion (Scrum "Product Owner" or XP "Customer"), the person who decides what features go in and what features don't, and also chooses which order they go in, must be readily available, willing to work on creating "stories" with enough detail and acceptance criteria so the team knows what is needed, how to test it, and how complex it is (for estimation). And this person must be ready to decide what the top priority detailed stories should be done first. Without this, the project will probably fail. "Agile" is not just a developer's process, it tends to impact the way the organization builds software product.

2. Too few automated tests. I call this the "Agilist's Dilemma" because without tests it becomes obvious to the developers (whether they'll tell you or not) that it's getting harder and harder to change the code to accommodate the next set of stories. At the very least, encourage developers to try Test-Driven Development practices wholeheartedly, right away, for two 2-week iterations. They'll need to learn how to do it right, and with some discipline, to get the full value. Dave Astel's book is good (Test-Driven Development: A Practical Guide), Kent Beck's book is good (Test Driven Development: By Example).

My courses are even better, because you get hands-on experience plus team-specific guidance! I'm also available for coaching, of course.

Serendipity At StarEast

This same question (effectively "What are the first steps?") came up after Lisa Crispin's talk at StarEast today. I like her answer, too. She responded with these two (I'm paraphrasing):

1. Get some Agile training for the team.
2. Talk with managers about how their roles will change, and let them know that the team will need their support and confidence.

So perhaps I'd alter my list to be:

1. Get some Agile training and/or hire an external coach.
2. Run a retrospective on the current, pre-transition process. Refer to Art of Agile for potential solutions. Try the chosen solution for two weeks. Repeat.

...And include management in both!