Struggles of the New “Agile” Consultant

In the Beginning, There Was Chaos

I’ve always wanted to be someone who mentored and taught others how to live up to their potential. Naturally, I wanted to become a consultant. The path that lead me to where I am will have to be the subject of another post. For now, I’ll just summarize.

After college, I went in to retail. What else could you do with a Psychology degree? After about four years in retail management, I went back to college to get a Computer Science degree. I had various jobs where I performed the functions of systems engineer, software engineer, DBA, SEO, etc. Then, I graduated to development management where I managed a couple of teams at a large software company. Surprisingly, managing a software development team isn’t that much different that managing a retail team.

While managing these teams, it was hard to ignore the fact that they were in total chaos. My company was pushing a home-grown agile software development methodology. It sounded good because there was more focus on the customer, but it just didn’t work. So, I went on the journey of finding what agile development meant. Certainly it didn’t mean “no documentation” and “no design”, right? Well, it didn’t, but that’s how we were acting. The teams were working super long hours. The engineers had no idea how to estimate. The team really didn’t know what was supposed to be delivered.

Let There Be Scrum!

Then, slowly through several “iterations”, we implemented scrum. I played the Scrum Master role. The path we took could be mapped by books that were used. The books that paved the road for us were “Agile Estimating and Planning”, by Mike Cohn, then “User Stories Applied”, by Mike Cohn, and then finally “Agile Software Development with Scrum”, by Ken Schwaber. As we went through each book, we implemented these new things we learned incrementally.

So, we did it. We successfully implemented scrum. It was awesome. Why was it awesome? Here are some of the key reasons why:

  • Freedom – The team was free to do whatever it took to make software. The management team didn’t care if we used Scrum, XP, or Waterfall, they just wanted software. We were continually getting beat up for the crap we turned out. We finally got tired of it, and management let us do whatever it took.
  • Engaged Teams – I was blessed with a wonderful team. Was every single person on the team engaged and wonderful? NO. But, these things have a way of working themselves out. The team, with a little guidance, quickly became a high performing, self-organizing team.
  • Awesome Product Owner – Our product owner on our team was amazing. She was strong, had technical sense, and strived to play by the Scrum rules. She mastered the product backlog. Sure, maybe at times she was a bit “too” engaged throughout the sprint, which caused minor distractions at times, but that was to be expected and easily corrected.
  • Individuals and Interactions Over Processes and Tools – You probably recognize that this is part of the agile manifesto. I listed this explicitly because this was a big deal. Originally, we were trying to use all kinds of tools; QuickBase, TeamTrack, TestDirector, Excel, etc. It was becoming a mess. We were spending so much time making sure everything was up to date. In addition, we were spending time generating reports that no one was reading. In the end, we used the tool with the lowest possible impact to the team; the white board. The teams sprint backlog was documented on the white board, where we tracked the daily status.

    Okay, to be honest, we didn’t “just” use the white board. We also used a tool called “ExtremePlanner” to hold the entire product backlog, and help with sprint and release planning. This was basically the playground for the product owner, so there was no overhead experienced by the team.

Don’t get me wrong, the road wasn’t always smooth. There was a lot of history that we had to work through. There was a pile of crap that we called software that was released to the public, and we had to fix it. But, we did.

Enter the Agile Consultant

I really wanted to use my experiences over the last couple of years, both good and bad, to help others make the transition to common sense, i.e. Scrum. I know that most software development shops really didn’t know how to make software that the customer wants without smashing the deadline, scope, or quality. That’s exactly where we were.

Our company closed our office in Omaha, and I didn’t want to move to the home base in San Diego, CA. This was the perfect opportunity to step into an “agile” consultant role. By certain divine intervention, a local company was looking for a consultant who was an expert in agile software methodologies. That was my calling.

My first job as a consultant is working with a company that was (and still is) in trouble. Some promises were made by executives that are now long gone, and the current team has to make good on those promises. Below is a sample of some of the core problems at this division:

  1. Legacy software – Most companies who have been around for 5+ years have to deal with legacy software that is fragile and extremely time consuming to make any changes. This is just about the worst case that I’ve seen or even read about. The code is so poorly architected it is a significant challenge to make even the smallest of changes.
  2. Turnover – Approximately 95% of the original team left. Most of the people who left held knowledge that was not documented nor was it shared. Just about everyone working on this project is new (within 6 months).
  3. Poor development infrastructure – The team is using VSS, which performs file-locks when code is checked out. This is a serious bottle-neck. Due to the poor design of the code, there are only a few files that most programmers have to touch. In addition, there is no consistent build or test environments or process.
  4. Lack of core experience – There are some key personnel, such as BA, QA, Management, that have no experience in those disciplines. Not only do they have to learn the product, they have to learn how to do their job.
  5. Micro-Management – There is a desire to manage every last task the team has to do. This has had a very damaging effect on the team. Most of the team is paralyzed when they have to make any decision for themselves. In addition, the management style is militant and somewhat cold.

I knew that this would be a challenge going in. We changed our approach several times. Below are some things we tried.

  1. Implement Scrum on a small project and use the success to convince others to adopt. We would play roles on the Scrum team, product owner and Scrum Master.
  2. Educate everyone on Scrum, then let them try it themselves on any product they decided, with us helping and coaching only. We wouldn’t play any roles on the scrum teams.
  3. Implement Scrum on the flag-ship product, and then let the other smaller projects fall in line. We would stay very close and coach the new Scrum Master, product owner, and team.

We actually did all of these things in order, and are now on #3. Scrum is well underway.

We went through quite a few struggles that we weren’t expecting.

  • We assumed that everyone would welcome us with open arms and that there would
    be a high level of trust. Boy, was I wrong. Most people were quite resistant. Many have come around after seeing some of the small successes, but there are some that I see that will always be resistant.
  • We underestimated the learning curve of Scrum. Yes, Scrum is simple. But, as Ken Schwaber (one of the original creators of Scrum) frequently states, “Its incredibly difficult to implement”. For example, creating a list of prioritized work has proven to be incredibly difficult. The product owner believed (for a while) that everything had the same priority … HIGH! We are planning on conducting additional training sessions to accommodate.
  • We didn’t know how much trouble the project was really in. As we all know, not all projects can be saved. It doesn’t matter what process you use. Again, as I’ve heard Ken Schwaber say, with Scrum, you’ll know how screwed you are after 30 days. This couldn’t be more true.
  • There is no trust amongst the team. QA doesn’t trust development, development doesn’t trust QA, BA doesn’t trust anyone, etc. What has been incredibly helpful here is our daily scrums. Just some time for the entire team to get together every day has helped the team members come closer to a trust relationship.
  • People don’t understand what self-organizing and cross-functional mean. The concept that you would have people from different disciplines on the same team is crazy! What’s more crazy is that you would allow the team to work on what they want to. When we introduced this concept, it was like we just told them they should believe in elves. I was shocked. To me, these concepts are common sense for any good manager. As Scrum is working, and we are coaching everyone along the way, these concepts aren’t approached with dismay and confusion anymore. It will be a slow process, but I think that the team will come around.

I’ve learned a lot while implementing Scrum at my first gig as an agile consultant. It is very different being in the role of consultant vs. full-time employee. At this client, we are not empowered enough to affect change as much as we would like. I think we will successfully implement Scrum, and eventually the client will be happy. We’ll come out other end with some battle wounds, but it will be well worth it. Will the project succeed? I don’t know. After the first sprint, the velocity isn’t what executive management was expecting. Actually, not even close. But, its honest and realistic. Something they have never had.

Improving Scrum Roll-Out

I know that there can’t be a fine-grained plan to implement agile development on every team. Every team will have unique needs that will have to be addressed. However, our overall approach will be different.

  • We will spend a significant amount of time with the management team, getting to know their problems, their personalities, and where they want to be. Each one of them will have different perspectives that will eventually have to be addressed. We need to set expectations better than we have in the past.
  • We will work at building trust and teaching Scrum and agile development from the top-down AND bottom-up. With our current client, we used a bottom-up approach, and it just didn’t work. However, if we would have only pushed it from the top, the team members would have totally rebelled. We will have to carefully map out our approach at our next engagement.
  • We will spend significant time educating the team on agile development and Scrum. This will have to be continual. We did a “light” introduction, and then dove right in. The team really needed more than this.
  • We will make sure to not get sucked into putting out fires, which is what happened at our current client. Now, this is a tricky one. Most likely there will be fires that they will need help with. If so, we will have to introduce a separate “fire-fighting” team. While the process improvement team continues with improving the process and coaching the members of the Scrum team.
  • We will need to know exactly what level of empowerment we have and/or need. While management may say that we can do whatever it takes, we will have to make sure that everything is communicated clearly and frequently. And, we will get push-back, as we’ve learned at this current client. We will now be more prepared.

We have learned a ton. Implementing Scrum as a consultant is much different than implementing Scrum at a company that you work for full time. It was very naive of me to think there would be no difference.

I will add posts as this project progresses, and as I learn new things about Scrum. I apologize for the length of this post. I promise future posts won’t be like this. I do hope that this has helped someone out there who is starting their career as an agile consultant.


The Role of the Functional Manager in Scrum

One of the surprises that I have encountered is the resistance to implementing Scrum from the functional managers. Just to clarify, functional managers refer to the QA manager, software development manager, BA manager, etc.

One of the things that Scrum does is provide a very simple framework with very few well defined roles. The only roles that are clearly defined are Scrum Master, Product Owner, and Team. Click here to find out more about these roles.

Nowhere does it define a “QA manager”, or “Software Development Manager”. I think I understand why. In traditional, “waterfall-ish” environments, everything is separated by function. There are functional silos where the manager will tell their team what to do. The QA manager will manage the work for the QA team, the software manager will manage the work for the software team, etc. Scrum emphasized cross-functional and self-organizing teams. This is a complete contradiction to how the functional manager has been trained to work.

When I was confronted by one of the functional managers, I was surprised at first. He felt that there was no place for his position, and that his job was in jeopardy. I assured him that this wasn’t the case, that his position was important and that it just needs to change. Then, I thought about this more. What IS the role of the functional manager? They don’t manage work. They (usually) don’t DO the work. They aren’t really chickens or pigs.

I searched for some time for some clear direction. After quite a bit of searching, I found out that this was one of those things where the answer was “it depends”. There was no clear-cut answer.

Depending on the needs of the team, there may be no place for functional managers. If the team is small and self-managing, the scrum roles are being filled competently and impediments are being removed without help from the functional manager, then I don’t believe there is a need for functional managers. However, there aren’t many organizations that will achieve this state of bliss.

The functional manager can serve very important roles within organizations that have implemented Scrum. Below are some ways the functional manager can add value.

  • Ensure each team member’s skills are kept up to date. This is a big one, and could consume a large portion of the functional manager’s time.
  • Help remove impediments. I have experienced situations in which there were just too many impediments for one Scrum Master.
  • Play the Scrum Master role on projects as necessary.
  • Help with resource allocation. Notice that I said “help with”, not “do”. This is more important with teams that have not attained the “self-managing” phase yet. Many times the functional group has to serve on many different projects. The functional manager can work with the team, Scrum Master and product owner on project assignments.
  • Handle personnel issues, paperwork, etc.
  • Participate in the hiring process. In a truly self-managing organization, the team would do the hiring. Until self-managing is fully realized, the functional manager can facilitate this.

Implementing Scrum does not automatically eliminate the functional manager role. However, it absolutely changes it. The functional manager will have to be willing to “release” some (most?) of their authority to the team. They will have to become a servant-leader.