This excerpt is written By Sam Guckenheimer, Neno Loje, Published Sep 11, 2012 by Addison-Wesley Professional. Part of theMicrosoft Windows Development Series series. Copyright 2013, ISBN-10: 0-321-86487-5, ISBN-13: 978-0-321-86487-1. To purchase this book click here: http://www.informit.com/store/visual-studio-team-foundation-server-2012-adopting-9780321864871

This chapter covers the characteristics of software engineering and management practices, the “situationally specific” contexts to consider, and the examples that you can apply in Visual Studio (VS). In this chapter, you learn about the mechanisms that VS (primarily Team Foundation Server [TFS]) provides to support the team enacting the process.

One methodology cannot possibly be the “right” one, but...there is an appropriate, different way of working for each project and project team

-Alistair Cockburn

The preceding chapter discussed the Agile Consensus of the past decade. That chapter distinguished between complicated projects, with well-controlled business or technical risk, and complex ones, where the technology and business risks are greater. Most new software projects are complex; otherwise, the software would not be worth building.

This chapter covers the next level of detail-the characteristics of software engineering and management practices, the “situationally specific” contexts to consider, and the examples that you can apply in Visual Studio (VS). In this chapter, you learn about the mechanisms that VS (primarily Team Foundation Server [TFS]) provides to support the team enacting the process. Whereas Chapter 1, “The Agile Consensus,” gave an outside-in view of what a team needs, this chapter provides an inside-out overview of the tooling that makes the enactment possible.

Visual Studio and Process Enactment

Through three classes of mechanisms, VS helps the team follow a defined software process:

1. As illustrated in Chapter 1, TFS captures backlogs, workflow, status, and metrics. Together, these keep the work transparent and guide the users to the next appropriate actions. TFS also helps ensure the “doneness” of work so that the team cannot accrue technical debt without warning and visibility.
2. Each team project tracked by TFS starts off with a process template that defines the standard workflows, reports, roles, and artifacts for the process. These are often changed later during the course of the team project as the team learns and tunes its process, but their initial defaults are set according to the chosen process template.
3. On the IDE clients (VS or Eclipse), there are user experiences that interact with the server to ensure that the policies are followed and that any warnings from policy violations are obvious.

Process Templates

The process template supports the workflow of the team by setting the default work item types, reports, queries, roles (i.e., security groups), team portal, and artifacts. Work item types are the most visible of these because they determine the database schema that team members use to manage the backlog, select work, and record status as it is done. When a team member creates a team project, the Project Creation Wizard asks for a choice of process template, as shown in Figure 2.2.

Microsoft provides three process templates as standard:

1. Microsoft Visual Studio Scrum: This process template directly supports Scrum, and was developed in collaboration with Ken Schwaber based on the Scrum Guide.  The Scrum process template defines work item types for Product Backlog Item, Bug, Task, Impediment, Test Case, Shared (Test) Steps, Code Review Request/Response, and Feedback Request/Response. The reports are Backlog Overview, Release Burndown, Sprint Burndown, and Velocity.
2. MSF for Agile Software Development: MSF Agile is also built around an agile base but incorporates a broader set of artifacts than the Scrum process template. In MSF Agile, product backlog items (PBIs) are called user stories and impediments are called issues. The report shown in Figure 1.4 in Chapter 1 is taken from MSF Agile.
3. MSF for CMMI Process Improvement: This process template is also designed for iterative work practices, but with more formality than the other templates. This one is designed to facilitate a team’s practice of Capability Maturity Model Integration (CMMI) Level 3 as defined by the Software Engineering Institute.  Accordingly, it extends MSF Agile with more formal planning, more documentation and work products, more sign-off gates, and more time tracking. Notably, this process template adds Change Request and Risk work item types and uses a Requirement work item type that is more elaborate than the user stories of MSF Agile.

Other companies provide their own process templates and can have these certified by Microsoft. For example, Sogeti has released a version of its Test Management Approach (TMap) methodology as a certified process template, downloadable from the Visual Studio Developer Center athttp://msdn.microsoft.com/vstudio/aa718795.aspx.

When you create a team project with TFS, you choose the process template to apply, as shown in Figure 2.2.

Teams

Processes tend to prescribe team structure. Scrum, for example, has three roles. The Product Owner is responsible for the external definition of the product, captured in the product backlog, and the management of the stakeholders and customers. The Team of Developers is responsible for the implementation. And the Scrum Master is responsible for ensuring that the Scrum process is followed.

In Scrum, the team has three to nine dedicated members. Lots of evidence indicates that this is the size that works best for close communication. Often, one of the developers doubles as the Scrum Master. If work is larger than can be handled by one team, it should be split across multiple teams, and the Scrum Masters can coordinate in a scrum of scrums. A Product Owner can serve across multiple scrum teams but should not double as a Scrum Master.

In TFS, each team has a home page with data from the current sprint of its project, like an up-to-date burndown chart and the remaining, incomplete PBIs, as shown in Figure 2.3. Additionally, the team gets its own product backlog and a task board-both are available using the Web browser. To support larger projects with multiple teams, TFS enables the concept of master backlogs that consolidate each team’s product backlog into a single view.

In most cases, it is bad Scrum to use tooling to enforce permissions rather than to rely on the team to manage itself. Instead, it is generally better to assume trust, following the principle that “responsibility cannot be assigned; it can only be accepted.” TFS always captures the history of every work item change, thereby making it easy to trace any unexpected changes and reverse any errors.

Nonetheless, sometimes permissions are important (perhaps because of regulatory or contractual circumstances, for example). Accordingly, you can enforce permissions in a team project in four ways:

1. By role
2. By work item type down to the field and value
3. By component of the system (through the area path hierarchy of work items and the folder and branch hierarchy of source control)
4. By builds, reports, and team site

For example, you can set a rule on the PBI work item type that only a Product Owner can update PBIs. In practice, this is rarely done.

Process Cycles and TFS

A core concept of the convergent evolution discussed in Chapter 1 is iterative and incremental development. Scrum stresses the basis of iteration in empirical process control because through rapid iteration the team reduces uncertainty, learns by doing, inspects and adapts based on its progress, and improves as it goes.  Accordingly, Scrum provides the most common representation of the main macro cycles in a software project: the release and the sprint (a synonym for iteration), as shown in Figure 2.4. Scrum provides some simple rules for managing these.

Release

The release is the path from vision to delivered software. As Ken Schwaber and Jeff Sutherland explain in the Scrum Guide:

Release planning answers the questions, “How can we turn the vision into a winning product in the best possible way? How can we meet or exceed the desired customer satisfaction and Return on Investment?” The release plan establishes the goal of the release, the highest priority Product Backlog, the major risks, and the overall features and functionality that the release will contain. It also establishes a probable delivery date and cost that should hold if nothing changes.

The release definition is contained in the product backlog, which consists of requirements, unsurprisingly named product backlog items, as shown in Figure 2.5. Throughout the release, the Product Owner keeps the PBIs stack ranked to remove ambiguity about what to do next. As DeMarco and Lister have put it:

Rank-ordering for all functions and features is the cure for two ugly project maladies: The first is the assumption that all parts of the product are equally important. This fiction is preserved on many projects because it assures that no one has to confront the stakeholders who have added their favorite bells and whistles as a price for their cooperation. The same fiction facilitates the second malady, piling on, in which features are added with the intention of overloading the project and making it fail, a favorite tactic of those who oppose the project in the first place but find it convenient to present themselves as enthusiastic project champions rather than as project adversaries.

A common and useful practice is stating the PBIs, especially the functional requirements, as user stories. User stories take the form As a <target customer persona>, I can <achieve result> in order to <realize value>. Chapter 3, “Product Ownership,” goes into more detail about user stories and other forms of requirements.

Sprint

In a Scrum project, every sprint has the same duration, typically two to four weeks. Prior to the sprint, the team helps the Product Owner groom the product backlog, estimating a rough order of magnitude for the top PBIs. This estimation has to include all costs associated with completing the PBI according to the team’s agreed-upon definition of done. The rough estimation method most widely favored these days is Planning Poker, adapted by Mike Cohn as a simple, fast application of what had been described by Barry Boehm as the Wideband Delphi Method. Planning Poker is easy and fast, making it possible with minimal effort to provide estimates that are generally as good as those derived from much longer analysis. Estimates from Planning Poker get entered as story points in the PBI work item. Planning Poker is discussed further in Chapter 4, “Running the Sprint.”

Another great practice is to define at least one acceptance test for each PBI. These are captured in TFS as test cases, a standard work item type. Defining acceptance tests early has three benefits:

1. They clarify the intent of the PBI.
2. They provide a done criterion for PBI completion.
3. They help inform the estimate of PBI size.

At the beginning of the sprint, the team commits to delivering a potentially shippable increment of software realizing some of the top-ranked product backlogs. The commitment factors the cumulative estimate of the PBIs, the team’s capacity, and the need to deliver customer value in the potentially shippable increment. Then, only the PBIs committed for the current sprint are broken down by the team into tasks. These tasks are collectively called the sprint backlog (see Figure 2.6).

Don’t Confuse Product Backlog and Sprint Backlog

In our experience, the most common confusion around Scrum terminology is the use of the word backlog in two different instances. To some extent, the confusion is a holdover from earlier project management techniques. The product backlog holds only requirements and bugs deferred to future sprints and is the interface between the Product Owner, representing customers and other stakeholders, and the team. PBIs are assessed in story points only.

The sprint backlog consists of implementation tasks, test cases, bugs of the current sprint, and impediments and is for the implementation team. When working on a task, a team member updates the remaining hours on these tasks, but typically does not touch the PBI, except to mark it as ready for test or completed. Stakeholders should not be concerned with the sprint backlog, only with the PBIs.

Using tasks, the team lays out an initial plan for how to transform the selected PBIs into working software. Estimating each task’s remaining hours helps the team verify the effort is not exceeding their capacity, as shown in Figure 2.7.

Handling Bugs

Bugs should be managed according to context. Different teams view bugs differently. Product teams tend to think of anything that detracts from customer value as a bug, whereas contractors stick to a much tighter definition.

In either case, do not consider a PBI done if there are outstanding bugs because doing so would create technical debt. Accordingly, treat bugs that are found in PBIs of the current sprint as simply undone work and manage them in the current iteration backlog.

In addition, you often discover bugs unrelated to the current PBIs, and these can be added to the product backlog, unless you have spare capacity in the current sprint. (The committed work of the sprint should normally take precedence, unless the bug found is an impediment to achieving the sprint goal.) This can create a small nuisance for grooming the product backlog, in that individual bugs are usually too fine-grained and numerous to be stack ranked against the heftier PBIs. In such a case, create a PBI as a container or allotment for a selection of the bugs, make it a “parent” of them in TFS, and rank the container PBI against its peers (see Figure 2.8).

Avoiding Analysis Paralysis

A great discipline of Scrum is the strict timeboxing of the sprint planning meeting, used for commitment of the product backlog (the “what”) and for initial task breakdown of the sprint backlog (the “how”). For a one-month sprint, the sprint planning meeting is limited to a day before work begins on the sprint. For shorter sprints, the meeting should take a proportionally shorter length of time.

Note that this does not mean that all tasks are known on the first day of the sprint. On the contrary, tasks may be added to the sprint backlog whenever necessary. Rather, timeboxing sprint planning means that the team needs to understand the committed PBIs well enough to start work. In this way, only 5% of the sprint time is consumed by planning before work begins. (Another 5% of the calendar, the last day of a monthly sprint, is devoted to review and retrospective.) In this way, 90% of the sprint is devoted to working through the sprint backlog.

Bottom-Up Cycles

In addition to the two macro cycles of release and sprint, TFS uses the two finer-grained cycles of check-in and test to collect data and trigger automation. In this way, with no overhead for the users, TFS can provide mechanisms to support both automating definitions of done and transparently collecting project metrics.

Personal Development Preparation

As discussed in Chapter 6, “Development,” VS provides continuous feedback to the developer to practice test-driven development; correct syntax suggestions with IntelliSense; and check for errors with local builds, tests, and check-in policy reviews. These are private activities, in the sense that VS makes no attempt to persist any data from these activities before the developer decides to check in.

Check-In Cycle

The finest-grained coding cycle at which TFS collects data and applies workflow is the check-in (that is, any delivery of code by the developer from a private workspace to a shared branch). This cycle provides the first opportunity to measure done on working code. The most common Agile practice for the check-in cycle is continuous integration, in which every check-in triggers a team build from a TFS build definition. The team build gets the latest versions of all checked-in source from all contributors, provisions a build server, and runs the defined build workflow, including any code analysis, lab deployment, or build verification tests that have been defined in the build. (See Chapter 7, “Build and Lab,” for more information.)

Continuous integration is a great practice, if build breaks are rare. In that case, it is a great way to keep a clean, running drop of the code at all times. The larger the project, however, the more frequent build breaks can become. For example, imagine a source base with 100 contributors. Suppose that they are all extraordinary developers, who make an average of only one build break per three months. With continuous integration, their build would be broken every day.

To avoid the frequent broken builds, TFS offers a form of continuous integration called gated check-in. Gated check-in extends the continuous integration workflow, in that it provisions a server and runs the team build before check-in. Only if the full build passes, then the server accepts the code as checked in.

Otherwise, the check-in is returned to the developer as a shelveset with a warning detailing the errors. Chapter 9, “Lessons Learned at Microsoft Developer Division,” describes how we use this at Microsoft.

In addition, prior to the server mechanisms of continuous integration or gated check-in, TFS runs check-in policies on the clients. These are the earliest and fastest automated warnings for the developer. They can validate whether unit tests and code analysis have been run locally, work items associated, check-in notes completed, and other “doneness” criteria met before the code goes to the server for either continuous integration or gated check-in.

Test Cycle

Completed PBIs need to be tested, as do bug fixes. Typically, team members check in code in small increments many times before completing a PBI. However, when a PBI is completed, a test cycle may start. In addition, many PBIs and bug fixes are often completed in rapid succession, and these can be combined into a single test cycle. Accordingly, a simple way to handle test cycles is to make them daily.

TFS allows for multiple team build definitions, and a good practice is to have a daily build in addition to the continuous integration or gated check-in build. When you do this, every daily “build details” page shows the increment in functionality delivered since the previous daily build, as shown in Figure 2.9.

In addition, Microsoft Test Manager (MTM, part of the VS product line) enables you to compare the current build against the last one tested to see the most important tests to run based on both backlog changes and new or churned code, as shown in Figure 2.10. (See Chapter 8, “Test,” for more information.)

Daily Cycle

The Scrum process specifies a daily scrum, often called a “daily stand-up meeting,” to inspect progress and adapt to the situation. Daily scrums should last no more than 15 minutes. As the Scrum Guide explains, during the meeting, each team member explains the following:

1. What has the team member accomplished since the last meeting?
2. What will the team member accomplish before the next meeting?
3. What obstacles currently impede the team member?

Daily scrums improve communications, eliminate other meetings, identify and remove impediments to development, highlight and promote quick decision making, and improve everyone’s level of project knowledge.

Although TFS does not require daily builds, and the process rules do not mandate combining the daily and testing cycles, treating the daily cycle and test cycle as the same is certainly convenient. TFS helps considerably with preparation for the Scrum questions:

• As Figures 2.9 and 2.10 show, the automated release note of the “build details” page and the test recommendations of MTM help resolve any discrepancies in assumptions for question 1.
• The task board, shown in Figure 2.11, should align with question 2.

• The Open Impediments or Open Issues query, shown in Figure 2.12, should match question 3.

These tools don’t replace the daily scrum, but they remove any dispute about the data of record. In this way, the team members can focus the meeting on crucial interpersonal communication rather than on questions about whose data to trust.

Definition of Done at Every Cycle

For each of these cycles-check-in, test, release, and sprint-the team should have a common definition of done and treat it as a social contract. The entire team should be able to see the status of done transparently at all times. Without this social contract, it is impossible to assess technical debt and, accordingly, impossible to ship increments of software predictably.

With Scrum and TFS working together, every cycle has a done mechanism. Check-in has its policies and the build workflows, test has the test plans for the cycle, and sprint and release have work items to capture their done lists.

Inspect and Adapt

In addition to the daily 15 minutes, Scrum prescribes that the team have two meetings at the end of the sprint to inspect progress (the sprint review) and identify opportunities for process improvement (the sprint retrospective). Together, these should take about 5% of the sprint, or one day for a monthly sprint. Alistair Cockburn has described the goal of the retrospective well: “Can we deliver more easily or better?”  Retrospectives force the team to reflect on opportunities for improvement while the experience is fresh.

Based on the retrospective, the sprint end is a good boundary at which to make process changes. You can tune based on experience, and you can adjust for context. For example, you might increase the check-in requirements for code review as your project approaches production and use TFS check-in policies, check-in notes, and build workflow to enforce these requirements.

Task Boards

Scrum uses the sprint cadence as a common cycle to coordinate prioritization of the product backlog and implementation of the iteration backlog. The team manages its capacity by determining how much product backlog to take into the coming sprint, usually based on the story points delivered in prior sprints. This is an effective model for running an empirical process in complex contexts, as defined in Figure 1.3 in Chapter 1.

TFS includes an automated task board that visualizes the sprint backlog, as shown in Figure 2.11. It provides a graphical way to interact with TFS work items and an instant visual indicator of sprint status.

Automated task boards are especially useful for geographically distributed teams and scrums. You can hang large touch screens in meeting areas at multiple sites, and other participants can see the same images on their laptops. Because they all connect to the same TFS database, they are all current and visible. At Microsoft, we use these to coordinate Scrum teams across Redmond, Raleigh, Hyderabad, and many smaller sites.

Kanban

The history of task boards is an interesting study in idea diffusion. For Agile teams, they were modeled after the so-called Kanban (Japanese for “signboard”) that Taiichi Ohno of Toyota had pioneered for just-in-time manufacturing. Ohno created his own model after observing how American supermarkets stocked their shelves in the 1950s.  Ohno observed that supermarket shelves were stocked not by store employees, but by distributors, and that the card at the back of the cans of soup, for example, was the signal to put more soup on the shelf. Ohno introduced this to the factory, where the card became the signal for the component supplier to bring a new bin of parts.

Surprisingly, only in the past few years have software teams discovered the value of the visual and tactile metaphor of the task board. And Toyota only recently looked to bring Agile methods into its software practices, based not on its manufacturing but on its observation again of Western work practices.  So, we’ve seen an idea move from American supermarkets to Japanese factories to American software teams back to Japanese software teams, over a period of 50 years.

In software practices, Kanban has become the name of more than the task board; it is also the name of an alternative process, most closely associated with David J. Anderson, who has been its primary proponent. Where Scrum uses the team’s commitments for the sprint to regulate capacity, Kanban uses work-in-progress (WIP) limits. Kanban models workflow more deterministically with finer state transitions on PBIs, such as Analysis Ready, Dev Ready, Test Ready, Release Ready, and so on. The PBIs in each such state are treated as a queue, and each queue is governed by a WIP limit. When a queue is above the WIP limit, no more work may be pulled from earlier states, and when it falls below, new work is pulled.

Kanban is more prescriptive than Scrum in managing queues. The Kanban control mechanism allows for continuous adjustment, in contrast to Scrum, which relies on the team commitment, reviewed at sprint boundaries. Kanban can readily be used inside the sprint boundaries to keep WIP low.

Fit the Process to the Project

Based on your project context and your retrospectives, you may choose to customize your process template. Ideally, this is a team decision, but certain stakeholders may have special influence. Even then, every team member should understand the rationale of the choice and the value of any practice that the process prescribes. If the value cannot be identified, it is unlikely that it can be realized. Sometimes the purpose might not be intuitive (certain legal requirements for example), but if understood can still be achieved.

As Barry Boehm and Richard Turner have described, it is best to start small:

• Build Your Method Up, Don’t Tailor It Down
• Plan-driven methods have had a tradition of developing all-inclusive methods that can be tailored down to fit a particular situation. Experts can do this, but nonexperts tend to play it safe and use the whole thing, often at considerable unnecessary expense. Agilists offer a better approach of starting with relatively minimal sets of practices and only adding extras where they can be clearly justified by cost-benefit

Fortunately, TFS assumes that a team will “stretch the process to fit”-that is, take a small core of values and practices and add more as necessary (see Figure 2.13).

One of the tenets of the Agile Consensus is to keep overhead to a minimum. Extra process is waste unless it has a clear purpose whose return justifies the cost. Three common factors might lead to more steps or done criteria in the process than others: geographic distribution; required documentation; and governance, risk management, and compliance.

Geographic Distribution

Most organizations are now geographically distributed. Individual Scrum teams of seven are best collocated, but multiple Scrum teams across multiple locations often need to coordinate work. For example, on VS, we are running scrums of scrums and coordinating sprint reviews and planning across Redmond, Raleigh, and Hyderabad, and several smaller sites, a spread of 12 time zones. In addition to TFS with large screens, we use Microsoft Lync for the video and screen sharing, and we record meetings and sprint review demos so that not everyone needs to be awake at weird hours to see others’ work.

Tacit Knowledge or Required Documentation

When you have a geographically distributed team, it is harder to have spontaneous conversations than when you’re all in one place, although instant messaging and video chat help a lot. When you’re spread out, you cannot rely just on tacit knowledge. You can also use internal documentation to record contract, consensus, architecture, maintainability, or approval for future audit. Whatever the purpose, write the documentation for its audience and to its purpose and then stop writing. Once the documentation serves its purpose, more effort on it is waste. Wherever possible, use TFS work items as the official record so that there is a “single source of truth.” Third-party products such as Ekobit TeamCompanion, shown in Chapter 4, can help by converting email into TFS work items for a visible and auditable record.

Governance, Risk Management, and Compliance

Governance, risk management, and compliance (GRC) are closely related terms that are usually considered together since the passage of the Sarbanes-Oxley Act of 2002 (SOX) in the United States. For public and otherwise regulated companies, GRC policies specify how management maintains its accountability for IT. GRC policies may require more formality in documentation or in the fields and states of TFS work items than a team would otherwise capture.

One Project at a Time Versus Many Projects at Once

One of the most valuable planning actions is to ensure that your team members can focus on the project at hand without other commitments that drain their time and attention. Gerald Weinberg once proposed a rule of thumb to compute the waste caused by project switching, shown in Table 2.1

That was 20 years ago, without suitable tooling. In many organizations today, it is a fact of life that individuals have to work on multiple projects, and VS is much easier to handle now than it was when Weinberg wrote. In Chapter 6, I discuss how VS is continuing to help you stay in the groove despite context switching, but it is still a cognitive challenge.

Summary

As discussed in Chapter 1, in the decade since the Agile Manifesto, the industry has largely reached consensus on software process. Scrum is at its core, complemented with Agile engineering practices, and based on Lean principles. This convergent evolution is the basis for the practices supported by VS.

This chapter addressed how VS, and TFS in particular, enacts process. Microsoft provides three process templates with TFS: Scrum, MSF for Agile Software Development, and MSF for CMMI Process Improvement. All are Agile processes, relying on iterative development, iterative prioritization, continuous improvement, constituency-based risk management, and situationally specific adaptation of the process to the project. Microsoft partners provide more process templates and you can customize your own.

Core to all the processes is the idea of work in nested cycles: check-in, test, sprint, and release. Each cycle has its own definition of done, reinforced with tooling in TFS. The definitions of done by cycle are the best guards against the accumulation of technical debt and, thus, are the best aids in maintaining the flow of potentially shippable software in every sprint.

Consistent with Scrum, it is important to inspect and adapt not just the software but also the process itself. TFS provides a Process Template Editor to adapt the process to the needs of the project. The process design should reflect meaningful business circumstances and what the team learns as it matures from sprint to sprint.

Finally, inspect and adapt. Plan on investing in process and tooling early to improve the economics of the project over its life span. By following an Agile approach, you can achieve considerable long-term benefits, such as the development of high-quality and modifiable software without a long tail of technical debt. However, such an approach, and its attendant benefits, requires conscious investment.

The next chapter pulls back to the context around the sprint and discusses product ownership and the many cycles for collecting and acting on feedback. That chapter covers the requirements in their many forms and the techniques for eliciting them and keeping them current in the backlog.