The software industry has a poor success record. In fact, all of IT does. Throughout phases of various types of formal leaders, including CIOs, project managers, and scrummasters, the project success rate struggles to get to 20%. In this article, you'll explore the multi-decade history of the software industry, some of the attempts at progress, and the results. You'll also explore what it takes to be a leader in software. At the end of this article, you will be able to:

  • Identify the main eras of the software industry
  • Understand the average level of success and failure over time
  • List the responsibilities of a software leader
  • Decide whether the calling of software leadership is for you

Custom Software: A History of Failure

For as many technical advances as the software (and IT) industries have seen over the decades, the rate of failure, outages, business disruptions, and lost investment is staggering. To illustrate this, consider the Standish Group. This organization has been publishing studies for decades, beginning with the original CHAOS report of 1994. In 1994, the CHAOS report contends that only 16.2% of software projects are deemed a success. Success means that the initial promise of the system was delivered on time, on budget, and with the needed level of stability while running the system. The Standish Group has continued to catalog projects and publish reports, roughly every five years, and the 2020 report shows an alarming result. The success rate overall is still only 16.2%! Although the report studies a wide range of IT projects, this is alarming for our industry. For readers interested in the source reports, I encourage the reading of the Standish Group material.

Even after 25 years since the original CHAOS report, software project success, on average, struggles to achieve 20%.

Let's now take a look at some major trends in the software industry according to the Standish Group.

1960-1980: The Wild West

Our industry has now topped 60 years since inception. Arguably, the late 1950s saw the first business software systems, but the cases to study are few. Fred Brooks is one of the early computer programmers (before the more common term, software developer or software engineer came about). In his book, The Mythical Man-Month, he writes a series of essays in which he reasons about some of the natural forces at play in software projects.

This twenty-year period from 1960 to 1989 was the early part of Mr. Brooks' career, and throughout the book, he shares the nature of early software systems and computers. These computers didn't come with a general-purpose operating system. The programmers loaded every bit of code the computer was to run. Software applications weren't a thing. We just called them “programs.” Operating systems were still in development. Moreover, every programmer did as he saw fit. There were no norms. No methodologies. No standard processes. No large technology vendors suggesting a “normal” way to proceed. The name of the game was to figure out how to make the computer do something.

1980-2000: The Waterfall Period

The Standish Group has labeled the period from 1980 to around 2000 as the Waterfall Period. This period saw the rise of software processes. Universities began computer science curriculums. CASE (computer-aided software engineering) tools were developed and taught as being the wave of the future where software programs would be programmed automatically by visual specifications of the required behavior. These two decades saw the right of the IT project manager and the establishment of the PMBOK, PMI's Project Management Body of Knowledge with the first report in 1987 and then the official first version in 1996.

During this period, the mainframe computer introduced segregation into businesses and gave rise to the IT department. After all, if a business was going to invest in one large expensive computer that multiple people could share, then someone had to be responsible for keeping it available and in good repair. At this point, programmers could concentrate on writing their program and then running it through a remote terminal. But because any program had the capability to crash the shared mainframe server, IT administrators began developing restrictions to protect the many from the few. In larger companies, this resulted in IT departments that separated the programmer from the server administrator. This segregation would take the next few decades to undo.

As these departments grew, IT project managers rose as a common role in the industry and PMI served as an educating and certifying body to give credibility to the role. The processes espoused by the PMI's PMBOK are now known as Waterfall. Essentially the concept of “gathering requirements” because the root cause for long project phases and rigidity is downstream. In this period, the Standish Group published its first CHAOS reports showing the world that the cumulative efforts by all involved were yielding a 16.2% success rate.

2000-2020: The Agile Period

Even though “The Agile Manifesto” wasn't authored until 2001, the late nineties saw the establishment of what would become the new way of thinking in software. Kent Beck's 1999 book, “Extreme Programming Explained,” suggested software team practices that, in my view, raised the work of computer programming into an engineering discipline. Many of the practices of Extreme Programming are now mainstays of agile and DevOps teams.

The Agile Manifesto and the accompanying rise of scrum training and scrummaster certifications broke long software phases into smaller phases. Initially suggested at month-long phases, called “sprints,” teams that understood the forces at play started to drop the notion of any kind of phase or iteration. When programmable virtual servers became widely available, the last barrier to iteration-less software development stepped aside.

This period is also the period that shifted from the server being a unit of hardware to being a slice of hardware. In concept, the multi-user mainframe server was reborn at the data center level when large hypervisor clusters running VMWare or Hyper-V were offered by hosting companies as easy VM Servers. AWS perfected the virtualized data center and Microsoft's Azure bet the farm on this model and took it further with specific runtimes so that software teams wouldn't even have to deal with the concept of a server operating system. Azure has copied from AWS and AWS has copied from Azure in a data center battle that has propelled the available technical infrastructure that can be used to run server-based software.

DevOps came along in 2010 as a long-needed protest to the segregated IT departments created in the 90s. DevOps, powered by virtualization and cloud technology, suggested that no divide should exist between those writing software and those running it. Many organizations have adopted that approach, including large enterprises such as Microsoft. In this period, the role of the software tester saw its demise as well. At the beginning of this period, some organizations maintained even ratios of testers to developers, but now, only specialized organizations maintain the role of dedicated software testers.

This period has also been the source of explosive growth. Stack Overflow publishes a developer survey that shows that every six to seven years in this period, the number of software developers has doubled, and the growth rate isn't slowing down.

2020+: The Infinite Flow Period

The Standish Group's CHAOS report epilogue predicts that we are in the next period, and they classify it as the Infinite Flow Period that will see the abandonment of the concept of a software project as well as the elimination of the formal role of technical project manager. We can already see that this has happened in organizations that earn revenue from operating complex software systems. The 2020 CHAOS report sheds a pretty damning light on the way the industry has chosen to manage itself. Still with only a 16.2% success rate, the industry is in great peril. With the growth in the number of developers in the field, we're seeing that the industry, as a whole, is getting bigger, but it isn't getting better. The industry needs a reboot. With the latest edition of the CHAOS report showing that the role of a project manager or the absence thereof makes no difference in the likelihood of success, you have to speculate about what can be done to operate a software team that's better than the industry's 16.2%.

Software project managers don't make a difference in the success rate of projects. Projects fail, even those that include a project manager.

Mixed Results of Modern Software Initiatives

The last two decades of advances in the software industry have led to some fantastic successes. No one can argue that new companies are not being formed on the backs of successful software projects. Individuals benefit from highly capable software applications in the palm of their hands. Many people shop at grocery stores where the checkout lines are software programs rather than store clerks. Software has replaced the paper map as a common item in a car glove box. More and more homes have televisions running software that streams television rather than receiving it over the airwaves or coaxial cable. With all the advances, you've got to wonder why some companies see a return on their investment but a disproportionate number of companies do not.

Digital Transformations Mostly Fail

The Boston Consulting Group (BCP) publishes research on the IT industry. They find that 80% of businesses have plans to move forward with some form of digital transformation. To translate that away from executive-speak, this means that more and more companies are going to upgrade software systems, modernize data centers, develop net-new systems, and generally revamp their technology. With email, file servers, and ERP systems becoming very mature and generally available, the technology initiatives of most companies involve custom programming. So, although these projects will likely include server and networking infrastructure, the world is becoming more and more of a software world.

BCG's 2020 study is interesting because they find that 70% of digital transformation projects fail. Although this is a little better than the Standish Group found, we're still on the wrong end of the 80/20 rule. (Readers who'd like to read more can refer to the BCG's writeup of the report: Flipping the Odds of Digital Transformation Success | BCG.)

Bad Software Costs Billions

In 2020, CISQ, the Consortium for Information & Software Quality, performed a study that yielded depressing results. The group studies the costs of poor-quality software. Their study can be found here (https://www.it-cisq.org/the-cost-of-poor-software-quality-in-the-us-a-2020-report/) and can be summarized into three cost types

  • $260B: the cost of unsuccessful software projects
  • $520B: the cost of poor quality in legacy systems
  • $1.56T: the cost of operational software failures

These figures are for 2020 alone and are for just the United States. If the technology industry as a whole is around $1.5T, these figures suggest a very upside-down return on investment equation.

Anecdotally, I've been involved in many conversations over 25 years in the profession. These conversations involve software developers lamenting that they spend too much time fixing bugs or investigating production issues. Others lament that the IT group is seen just as a cost center, only to have hiring freezes and layoffs in a downturn. When you look at these statistics, and if they correlate to similar things, you might conclude that overall, in the United States, the IT industry as a whole is, indeed, a cost center. Now, I haven't cited statistics on revenues produced by software and IT investments. Those are hard to come by. Because companies are still aggressively investing in technology, you can conclude that even with all the mess, it's still worth it. But from within the industry, we must do better.

$1.5T is lost in the US every year to operational software failures.

Teams That Beat the Averages

This article so far has focused on the poor results, on average, that our industry produces. But there are teams that beat the averages. There are teams that flip the scales, that rarely fail. Several organizations are performing ongoing research into the practices that lead to good software delivery results. DevOps Research and Assessment (DORA) is one such organization. DORA's Accelerate State of DevOps Report analyzes the results of software teams and the practices they use. The report can show a correlation between team behavior and team results. DORA was acquired by Google in 2019 and operates from this website: https://cloud.google.com/devops/.

A key finding of DORA's State of DevOps report was that elite performers take full advantage of automation. From builds to testing to deployments and even security configuration changes, elite performers have a seven times lower change failure rate and over 2,000 times faster time to recover from incidents. DORA also provides some metrics that elite teams can track to ensure they remain among elite performers.

Everything Rises and Falls on Leadership

John Maxwell is the author of over 70 leadership books. People in every industry have used his books to identify and grow leadership. His books and seminars frequently include the now-famous quote: “Everything rises and falls on leadership.”

John Maxell, author of 70+ books on leadership says: “Everything rises or falls on leadership.”

It's not hard to spot a leader. Others around the leader are better because of the presence or influence of the leader. In times of trial, the leader takes the responsibility, the blame, and the stumbles. In times of triumph, the leader steps aside and shines the light and rewards on the team.

Over the 60+ years of our software industry, much has been established. Many processes and paradigms have been tried. Many have succeeded. Most of them have failed at least once. The knowledge of how to succeed is there. Unfortunately, that knowledge isn't universally applied. I have a hypothesis. It's because of uneven leadership within organizations producing and running custom software.

What Software Leaders Actually Do

Some people reading this article are currently software leaders. Some aspire to be. Some work under the guidance of one. Some wish their organization had one.

It might be convenient to consider common job titles that suggest leadership:

  • Development manager
  • Lead engineer
  • Chief architect
  • Software manager
  • Project manager

I've seen people occupying each of these titles yet not filling the boots of leadership. I've seen those in roles with lowly titles demonstrating strong leadership for a software team. Let's shy away from job titles for a moment, as there isn't a clear correlation between job titles and leadership. In a perfect world, titles that confer authority or responsibility would reliably be reserved for those with the leadership skills to fill them. But there are too many exceptions to ignore.

Responsibilities of the Software Leader

Any analogy of leadership outside of technology will both apply and fall short. You can imagine an army captain organizing a company of soldiers for a mission. Done well, the soldiers have every resource they need, are trained in every skill required on the mission, and they operate together flawlessly to get the job done. Once successful, the captain passes on the rewards and accolades to the soldiers. If not successful, the captain doesn't pass the blame to any soldier in the company. Instead, he accepts the blame and accountability.

Another analogy that both applies as well as falls short is that of a shepherd. The shepherd has a flock of sheep. The shepherd provides what the sheep need, including protection, food, water, and safety. When successful, the sheep yield an abundance of wool and many healthy new lambs. If any danger threatens the flock, the shepherd faces it head-on and directly to save the sheep.

Software developers are neither soldiers going to war nor sheep producing wool, but in both analogies, the leader provides for, cares for, protects, and is with those under his leadership. If you apply this directly to a software team, here are some of the high-level responsibilities of a software leader:

  • Form your team. Whether you inherited a team or are forming it from scratch, you must make sure it has the capabilities required for the job.
  • Equip your team. From computers to tools to processes, you must make sure your team has everything necessary to succeed.
  • Design the working environment. The team needs an environment in which to work. This includes where raw materials are located, where work activities take place, and where finished products flow. Virtual spaces are still part of the environment.
  • Set vision and targets. Your people need a clear shared vision. And they need targets that are close enough to see.
  • Prioritize the work. The team needs the work to be in a clear sequence. Research has shown that 2/3 of prioritized work normally is of little value to software users. Picking the right 1/3 makes a difference.
  • Measure the team. “A players” love being measured. They intrinsically work for the grade. “C players” don't want to be measured. “A players” will course-correct and press on to achieve the targets set for them.
  • Monitor and adjust. Over time, the metrics collected will yield areas that need to be adjusted.
  • Strengthen your team. Every team member needs a professional development path. Don't expect each team member to automatically do that for themselves.

Software leaders are a rare breed. Leaders don't take the credit when things go well, and they absorb all the blame when they go poorly.

Results of a Strong Software Leader

The DORA organization's leaders have published a book that every aspiring software leader should read. It's called “Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations” by Nicole Forsgren, PhD, et al. This book applies available research to some practices and measurements that are correlated with high-performing software teams. This is one example of knowledge that strong software leaders bring into their organizations. At this point in our industry, a software leader must do more assembly of knowledge than independent research of it. The DORA metrics of high-performing software teams are as follows:

  • Deployment frequency. Elite teams are able to perform multiple deployments per day.
  • Mean lead time for changes. Elite teams are able to deploy code into production within one hour of making a commit.
  • Change failure rate. Elite teams have less than 15% of deployment resulting in the need for an immediate fix.
  • Time to restore service or Mean Time to Recovery (MTTR). Elite teams can fix a customer impact within one hour.

There's a lot behind each of these metrics. Engineering practices like test-driven development, continuous integration, and automated deployments, and others are required in order to make these metric thresholds possible. A strong software leader is educated and practiced in all of the needed processes to achieve these results. A strong leader has developed the people and communication skills to cast vision and align everyone in the same direction. A strong leader is able to identify and remove distractions so that focus is simplified down to just the essentials. Ultimately, what a business wants out of a software team is for it to:

  • Move fast
  • Deliver quality
  • Run stable software in production

When these are achieved, the team gains confidence in themselves and pushes for even higher levels of performance. When these are achieved, business executives realize what an internal asset the software organization has become. When these are achieved, your team will be an outlier in the Standish Group research, beating the averages by an enormous margin.

Strong software leaders empower their teams to move fast, deliver quality, and run their systems with confidence.

Conclusion

I hope you're enlightened, a bit worried, but also inspired by this article. Over the decades, our industry has seen macro trends come and go. It has seen great technological advances, but what it hasn't seen, on average, is much improvement in success. Rather than new technology or innovation, our industry needs leaders. It needs you. The knowledge is out there. The innovations are out there. The industry needs leaders like you to take the lead, accept responsibility, take authority for future success, and make a change. Software leadership is not a job title. You can do it. If you feel the calling in your heart, it doesn't matter your age or level of experience. Take the lead. Do the work. Make the change.