As I returned from dropping my car for its annual service, my phone rang. I looked down to see who was calling. Recognizing the name on my caller ID, I predicted that this was not a “good” call. Here's how the conversation went:
“Rod there's no way this application can go out to the users,” our fearless project manager told me.
“Uh oh! Can you tell me what's up?” I responded.
“It looks like a lot of the business rules that we need haven't been implemented properly or at all. We're finding way too many issues to turn this over,” she declared.
“Let me call you back when I get to the office,” I replied as I made my way from the auto dealer.
After reviewing the issue logs generated by our testing team, I got on the phone with the project managers. They were right; we had a problem on our hands and it needed to be addressed forthwith. We needed to get this project back on track and that process began in earnest right after that call.
No Need for the Blame Game
I'm going to open with one of the saving graces that helped salvage this project. What's the saving grace? It's simple: WE DIDN'T ASSIGN BLAME! We approach problems like this using a Zen-like philosophy: “How do we fix it?” Blaming people doesn't help solve the problems. Getting everyone on the same page does. Moving away from the problem toward the solution is the fastest way to get things solved.
The First Question Is: What Happened?
While we didn't assign blame, we did question how this happened. There's a principle in process improvement called the Five Whys. The Five Whys examine a problem with the ultimate goal of finding a root cause. This is done by asking deeper and deeper questions related to the real source of a problem. By solving a problem at the root cause level, you make proper action items that solve it correctly with the goal of preventing its reoccurrence.
Before we started fixing specific technical issues, I spent time with my lead developer to discover what had happened. It turns out that the root cause of our situation was a communications failure, which is a recurring theme on software projects in general. During development, we used a set of specifications that had been created over many years. I asked him to make sure to look over the documentation in its entirety; I also emphasized certain parts of the documentation for closer review. He reviewed the sections I recommended but didn't look closely at items that he felt were already completed. But these items were completed for another project with slightly different requirements. Also, during the triage for this project, we learned that some of the documentation was out of date and/or incomplete. Once this project is complete, we'll be performing more of a postmortem to further investigate what happened and why.
Communications is Key
As soon as we knew we had a problem, we turned our communication level to “11.” Numerous hours were spent on the phone, screen-sharing using GotoMeeting, documenting issues in email, reviewing in the issue log. Pretty much every modern convenience was used sans sending messages via carrier pigeon. Some of the issues we faced were trickier to solve than others and it took every trick in the book to get to proper solutions. One takeaway: You cannot communicate too much.
One area of communication that was very important to solving this problem was to improve the documentation we were using. Numerous team members stepped up to the plate and rewrote entire sections of documentation. Other stepped up and created new documentation from scratch for the more confusing sections of business rules. Another takeaway: We need to review and update our documentation early in the development process.
You Only Learn From Mistakes
It has been a Herculean effort over the last three weeks (I am writing this in real time) to get this project in shape. Just today, this project got turned over to our users for a final round of testing. One major benefit of these road bumps was a steady increase in the project's quality. Every fix was analyzed closely and each step of the way, we made sure we didn't regress.
No development process is perfect. Over the years, we've been fairly successful in building software without too many large-scale issues. This troubled project was an outlier in my opinion. Upon reflection, learning about the holes in our process ensures that corrective actions will be taken to improve our process going forward.
