Closed-loop feedback control with Scrum

Scrum is a simple framework to manage complex challenges. Software delivery is a complex challenge. Software delivery encompasses a multitude of complex activities to create and evolve complex products in complex circumstances. Scrum embraces and emphasizes the complexity of software delivery by implementing the only process type that fits its complexity, empirical process control.

Complexity

There are many variables that have an impact on delivering software; requirements, skills, experience, people, teams, technology, integrations, market conditions, company strategies, budgets, regulations and dependencies, to name just a few. Not all known variables are controlled by the people doing the work, although they have to incorporate the impact on the work. For some known variables too much detail is needed to fully comprehend them. Even if a variable is known, its behavior -now or in the future- may be unknown, or different from what is anticipated.

Products are created by people with cognitive capacities. People are not robots, or replaceable pieces of machinery. The outcomes are hard to describe in an exact and detailed way; before, at the beginning or even as the work advances. Every product created is unique and what is appreciated can only be established when released. The activities to be undertaken aren’t predictable with any degree of high precision. They have not been performed before, in the same way, under the same conditions, several times. The environment evolves constantly. And, most annoying, not all variables are known. There is a high degree of unpredictability.

The degree of dynamism of a challenge requires the right process to be in place in order to have control:

Open-loop systems

An open-loop system is designed for execution of a series of preempted steps to result in a defined outcome in a single run. Such a system assumes a near-perfect predictability of the variables that influence the process as well as of the process activities themselves.

For larger problems, typically a chain of open-loop subsystems is created. The output of a subsystem is the input to the next subsystem. Although theoretically risks should be confined to the smaller subsystems, in practice the whole system’s vulnerability increases exponentially. In situations of turbulence and change, deviations and variances accumulate across the various subsystems, e.g. in timing and quality. It is not uncommon for the accumulated problems to surface only at the end of the final subsystem.

Predictive plans and hand-overs of work between separate functional groups are implementations of open-loop thinking. Predictive plans can only include known variables, their known details and their anticipated behavior, creating the illusion that no unknowns exist. Open-loop thinking invites lengthy upfront consideration of all elements of the plan, and ultimately attempt to foresee the unforeseeable.

An open-loop system is -by design- unable to cope with the amount of disruptions and unknowns typical for complex challenges like software delivery.

Closed-loop systems

Closed-loop systems implement frequent opportunities for inspection so adaptations can be implemented. The actual outcome of the system is compared in a timely fashion against a desired outcome. Desires may change. Variances or undesired results are eliminated or corrected in the next or in future runs. Not all variables and parameters need to be known precisely and in detail, as the process is self-correcting. The system requires and creates transparency. Reality is inspected, and exposed, so that appropriate adaptations are undertaken. The people performing the inspections have clear and agreed standards in place to inspect-and-adapt against. Inspection for reporting and status purposes is pointless. Inspection without adaptation is pointless.

Closed-loop feedback control with Scrum

Scrum embraces and stresses the complexity of software delivery by implementing empirical process control. Scrum replaces the open loops of traditional, phase-gate, staged or similar processes with closed-loop feedback. Scrum defines regular opportunities to inspect and adapt. Scrum enables players to halt the traditional rat race. The players are enabled to stop, reflect, learn from inspections, gather feedback over the output and change course, re-organize, update priorities, improve, adapt. Scrum brings reality back in the game. Scrum brings transparency. That is pleasant when all is progressing well. That is crucial when all is not progressing as hoped for. It allows re-positioning and correction.

In Scrum all work is organised in Sprints. Sprints are time-boxed to assure timely inspection and adaptation. A Sprint forms an ‘inspect and adapt’ cycle of a few weeks at most that wraps the 24-hours ‘inspect and adapt’ cycle of the Daily Scrum:

  • At the Daily Scrum the people doing the work inspect their progress, and identify their most important work to do next within the container of the Sprint. They use the Sprint Backlog, the Sprint Goal and a progress visualization to self-organize within the Sprint. The daily cadence assures they never get out of sync for more than 24 hours.
  • A Sprint is a cycle that starts with identifying and interiorizing the most important ideas instantiated on Product Backlog. Sprints end with an inspection of the product Increment that was actually released or could be released, as well as how it was built, the process, the interactions, the technology at play. As with all inspections, they are forward-looking. They serve the purpose of adapting.

All events of Scrum set a frequency for the inspection and adaptation process, where the artifacts contain the information to be inspected and adapted. Scrum describes the accountabilities needed to perform the inspections and adaptations.

Organizing work in Sprints allows people to take a breath, to break with the traditional rat races, and work at a sustainable pace. Explicit reflection moments are introduced that are crucial for humans to thrive in cognitive, creative work of high complexity, like software delivery is. Within a Sprint, additional feedback loops are created, e.g. through agreed work and development standards.

Note: above text is adapted from my book “Scrum – A Pocket Guide (A smart travel companion)”. If you have more time to spend, consider reading it.

Done is a crucial part of Scrum, actually

If Scrum was to be reduced to one purpose, and one purpose only, that is the creation of a Done Increment in a Sprint. The typical term in Scrum to describe the state of software being releasable is “Done”. All that this state of releasability encompasses is captured in the “definition of Done”.

Done Increments are THE way to achieve agility through the empiricism of Scrum. 

Empiricism

The empiricism of Scrum, the process of regular inspection and adaptation, only functions well upon transparency. Transparency is having insights into reality but is is additionally upheld through standards and agreements, against which inspection and adaptation happens. The definition of Done is such a standard. The definition of Done is part of professional Scrum development. Other standards, like development and engineering standards, might even be derived from the definition of Done.

The frequency of the inspection and adaptation should be high enough to be able to act in the moment yet not too high to preserve the ability to get considerable amounts of work done.

The definition of Done serves empiricism

The definition of Done should be shared, explicit, clear and concise.

A Development Team will use the definition of Done to consider the amount of work to be pulled into a Sprint during Sprint Planning.

The evolution of an Increment is managed via collaborative inspection and adaption of the actual development work against the forecasted Product Backlog and the Sprint Goal; at least on a daily base, possibly sooner. A Daily Scrum assures that the people accountable for the actual development optimize their work plan against new insights and achievements. The definition of Done supports identification of work remaining to get the software to “Done”.

No later than at the Sprint Review, the Increment is collaboratively inspected and adapted with the stakeholders. This inspection opens up the opportunity of incorporating feedback from these stakeholders to identify what is most important to do next. Purpose is the open identification of what is important to do next, not hindered by unknown, unpalatable, unestimatable remaining work.

Releasing the software closes the feedback loop to the market and the users of the software. Releasing sooner is better in order to remain in line with external expectations and experiences. It is the only way to ultimately validate all assumptions (of functionality, and others) built into the product. Not being able to release an Increment at the end of a Sprint, or sooner, impedes agility. The decision of releasing an Increment by the end of a Sprint is a Product Owner decision, as the sole representative of users and stakeholders on the Scrum Team. The Product Owner’s shipping decision should not be constrained by ‘development’ work.

Undone software is best not released. There might be situations in which undone software is consciously released. An extreme crisis maybe? The least to do is make the undone work transparent via Product Backlog, knowing and understanding that any estimate of such corrective work is probably totally off and the nature of the work unplannable. This ‘registration’ does not render an undone release any more professional, and probably the crisis you are hoping to solve with the unrelease, will re-appear because an unrelease will not fundamentally solve it. Unreleases backfire. Probably better to Scrumble.

At the Sprint Retrospective, the Development Team might inspect and revise its definition of Done; incorporating new insights, new expectations, higher standards. Ownership over the Definition of Done lies with the Development Team. It is their accountability to develop software that lives up to the definition of Done. In many organizations the definition of Done is likely to be derived from organizational standards for development quality. A Development Team will enact them, expand them. If “done” for an increment is not a convention of the development organization, the Development Team will create their definition of Done, appropriate for the product.

Regardless, the definition of Done provides transparency over the state of an Increment at the Sprint Review, where this state optimally reflects ‘releasable’.

Done is a crucial part of Scrum, actually

Although no official artifact, the definition of Done is a crucial part of Scrum in upholding transparency over the state of releasability of the software created. No transparency means no meaningful inspections, and no meaningful adaptations of Product Backlog through stakeholder feedback upon review or through user feedback upon release.

In the last updates to the Scrum Guide (most recent: July 2013) the definition of Done was given considerably more attention. Rightfully, as “Done” is absolutely crucial in Scrum.

Here’s how I stressed the importance of Done in my book, “Scrum – A Pocket Guide“:

The empiricism of Scrum only functions well with transparency. Transparency requires common standards to work against and to inspect upon. The definition of done sets the standard for releasable.

 and

The definition of done is essential to fully understand the work needed to create a releasable Increment and for the inspection of that Increment at the Sprint Review. The definition of done serves the transparency required in Scrum in terms of the work to be done and the work actually done.

Scrum, actually

Scrum, actually, has been around for a while. Scrum emerged in the early 1990’s through the work of Jeff Sutherland and Ken Schwaber. They packaged their practices into a cohesive set of rules and roles and named the entirety „Scrum“. The term, actually, was inherited from the ground-breaking 1986 paper The New New Product Development Game. The reference to the game of rugby reflects the importance of team engagement.

Scrum, actually, has had a stable core since its first public presentation in 1995. The essential definition of Scrum was codified in the Scrum Guide in 2010. This definite body of knowledge describes all parts of Scrum, and the rules that tie them together. Scrum is defined as intended and designed, i.e. a cohesive set of rules and roles implementing empiricism for complex product development. The rules and roles described in the Scrum Guide gain full clarity when read as an expression of the Agile values and principles.

Scrum, actually, is intentionally kept low prescriptive. Scrum sets the frame for people, teams and organizations to create, maintain and sustain complex products. Scrum does not replace people’s intelligence and creativity, merely guides the work. Scrum’s basic rules are immutable. Flexibility comes from the zillion variations to apply the rules, selected and tuned to context and circumstances. Hacking the basic rules of the framework breaks the cohesion, and disregards one or more principles and foundations upon which Scrum is founded.

Hacked versions and implementations of Scrum are possible. Isolated use of Scrum’s terminology or practices is possible. They might work. They might be fun. They are not Scrum.

Scaled implementations of Scrum don’t change the fundamental rules and roles of Scrum, nor the underlying principles. They only require different tactics. Instances that change the core of Scrum are not Scrum.

Scrum, actually, in itself is not the purpose. Scrum is a tool. Scrum enables people to live the art of the possible, to make the most out of every single day constrained by their means, to maximize the value of their work in the face of uncertainty. Scrum can wrap many development and organizational practices, tools and techniques. Scrum creates the capability of continuous adaptation in a environment of constant change, regardless whether that change is caused by our own will or by external turbulence. Scrum turns complexity from a threat into an asset.

Scrum, actually, propels agility through releasable Increments of software. A releasable Increment is available by the end of a Sprint or sooner, not later. A Sprint takes no more than 30 days, and is often shorter. Frequently an updated, improved version of software can be made available to its users and consumers. Feedback from actual use can be gathered to drive changes, improvements, enhancements. Assumptions are turned into learnings, ultimately into a pivot if required.

Scrum, actually… is a means to an end, a tool designed for a purpose: people, agility, value.

Meetings in Scrum, actually

Scrum has no meetings, actually. What we call ‚meetings‘, even in the Scrum Guide, are planned occasions at which people meet, where meeting (the activity) takes place.

Scrum’s meetings are not about reporting, status, bureaucracy, spilling ink, documenting the past. Scrum’s meetings have a purpose. Scrum’s meetings are about collaboration, discovery, opportunities, conversation, ideas, constructive disagreement, looking forward to the (near) future. It’s why Scrum offers opportunistic events more than obliges (what we generally know as) ‘meetings’.

Scrum’s events provide people with an opportunity to incorporate change into the daily work, instead of locking it out. The old notion of ‚change’ dissipates. Change becomes natural, the regular way of doing business, even a welcome source of ideas and innovation. Change is used in a team’s or an organisation’s advantage.

Scrum’s events serve the empiricism that Scrum brings to software development. Empiricism thrives on inspection & adaptation. Inspection & adaptation happens at a frequency, in regular intervals. Adaptation only makes sense when inspection is done against reality, when the actual situation is made transparent.

  • Scrum’s events define the frequency at which inspection and adaptation takes place.
  • Scrum’s artifacts hold the primary information to inspect and adapt.
  • Scrum’s teams are the inspectors, the people accountable for performing the inspections and adaptations.

Empiricism in Scrum

In Scrum all work is organized in Sprints. Sprints deliver releasable Increments of software. A Sprint is a time-boxed feedback loop in itself, a container event containing the above Scrum events.

Deciding over Sprint length is a different decision from the perspective of inspection and adaptation. The Sprint length determines the frequency at which stakeholder input is formally gathered and shared with the full Scrum Team. It’s the minimal frequency at which organizational or market changes can be incorporated, the last possible moment to decide on releasing software to collect feedback so it can be adapted to, to decide what the most valuable work is to work on next. When Sprints are too long, important opportunities that require adaptation may be missed. When Sprints are too short, the ability to get significant work done might be lost.

The time-boxes for all events, as set by Scrum, provide focus. It avoids the creation of waste. It focuses people’s minds on collaboration and importance.

Scrum frames the creativity of people. Scrum provides boundaries that re-inforce self-organization. Scrum says not how to run the events. Scrum defines the input to the meetings, the expected outcome and a timeframe.

In Scrum, actually… meetings are opportunities where people meet to change their mind.

Team size in Scrum, actually

Self-organization is an essential management principle of Scrum. Yet, its importance and potential are only seeping through slowly. Despite the wide adoption of Scrum.

The most basic form of self-organization in Scrum holds that Development Teams organize and manage their own work within a Sprint, autonomously, against a forecast and a Sprint Goal. Where acceptance of this practice grows, few organisations take it a step further. Few teams are supported to figure out their own team size in order to best collaborate towards the creation of a releasable Increment of product in a Sprint. Understanding that the foundations for great work are commitment and motivation, Development Teams should be able to also create and re-create their structure and composition across time.

Collaboration is key. From collaboration performance emerges. Teams have the highest cohesion, the deepest trust and the most effective interconnections when the size of the team is around seven. Scrum used to have the rule known as 7 +/- 2, meaning a Development Team was expected to have at least 5 people, and 9 at most. The Scrum Guide has evolved this guidance to 3-9 people. This is confusing when looking for academic exactness, less confusing if this is seen as guidance against the goal of being “small enough to remain nimble and large enough to complete significant work within a Sprint“ (quote from the Scrum Guide).

Although the Scrum Guide sets an expectation for the size of a Development Team, there’s no formal process needed to really enforce this if self-organization is enacted. Through self-organization a team will adjust its size autonomously for optimal performance. Rather than instructing a team on their mandatory size, help a team discover what works best for them, including what team size maximizes the communication bandwidth. No external body can do this better. No external body can assess the combined effects of team dynamics, being co-located or not, availability of people and resources (tools, infrastructure), and all other parameters better than the people actually doing the work.

Try something you believe might work for you. Inspect it, adapt to your findings. Repeat. When heavily constrained in doing this, sticking to the guidance of having 3-9 people in a Development Team is a good idea.

In Scrum, actually… team size is a team decision.

Velocity in Scrum, actually

In Scrum it is considered a good idea for a self-managing Development Team to know about the progress it has been making:

The input to this meeting (note: Sprint Planning) is the Product Backlog, the latest product Increment, projected capacity of the Development Team during the Sprint, and past performance of the Development Team. (Source: The Scrum Guide)

Teams express this ‘past performance’ often as ‘Velocity‘. Although not mandatory, it is a good tactic to apply in Scrum.

Velocity: an optional, but often used, indication of the average amount of Product Backlog turned into an Increment of product during a Sprint by a Scrum Team, tracked by the Development Team for use within the Scrum Team. (source: the Scrum Glossary)

Many organisations adopt Scrum to be more agile, to increase their agility. Agility with Scrum is achieved through the creation and frequent release of Increments of software.

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. (source: Principles of Agile Software Development)

In the face of the purpose of increased agility through Scrum, the best definition of Velocity in Scrum actually is a measure of a team’s ability to produce releasable software in a Sprint. Having no releasable software by the end of a Sprint is… zero velocity.

Working software is the primary measure of progress. (source: Principles of Agile Software Development)

In the face of the purpose of increased agility through Scrum, it doesn’t add much value to discuss Velocity when no releasable software is created throughout a Sprint. There are probably more serious problems to resolve first. Discussing the standardization, normalization, industrialization, equalization of Velocity is futile, at best highly sub-optimal, in the face of striving for agility with Scrum. In the absence of the capability to produce releasable software every 1-4 weeks, such discussions do no more than distract from more serious problems to solve first.

You can obviously measure the Velocity of creating undone software, and be more predictable in making progress creating undone software. Keep in mind that it may be an obfuscating indicator, not a measurement that increases transparency.

Velocity in Scrum actually is an indicator of productivity, an indicator of how much software, preferably releasable software, a team has produced in a Sprint. That in turn is not a promise, nor a contract for the future. Predictions are fragile. Empirical process control has the potential of antifragility. We embrace complexity.

In complex environments, what will happen is unknown. (Source: The Scrum Guide)

In Scrum, actually… velocity makes most sense if a measure of a team’s capability to create releasable software.

The importance of Done in Scrum

In the last Scrum Guide (July 2011) the definition of Done was given considerably more attention. Rightfully, as “Done” is crucial in Scrum.

The Importance Of Done

The definition of Done is essential to fully understand the Increment that is being inspected at the Sprint Review with the stakeholders. The definition of Done implements the expectation of an Increment to be ‘releasable‘, so ideally it is comprised of all activities, tasks, qualities and work that allow releasing an Increment in production. The addition ‘potentially‘ to releasable refers to the Product Owner’s accountability to decide over the actual release of an Increment; a decision that will likely be based upon business cohesion and functional usefulness. But the Product Owner’s shipping decision should not be constrained by ‘development’ work.

The definition of Done should be clear and concise for the Scrum Team as it will determine how much work a Development Team can reasonably take in into a Sprint during the Sprint Planning meeting.

The empiricism of Scrum only functions well upon transparency. That includes the definition of Done. Transparency means not only visible, but also understandable. Besides being available, the content of the definition of Done should be clear by just reading it.

A New Scrum Artefact?

Should we make the Definition of Done an official Scrum artefact?

It would seem like adapting Scrum to reality, a mere formalization of an existing practice; because it is extremely important to put quality even more at the heart of what we do; because we want to clear out that little grey zone in the base Scrum framework allowing some people to doubt the formal need of a definition of Done. With regards to the latter, it would provide additional guidance for people and organizations to improve, and investigate the next steps on the cobblestone path to Agility, although probably not the guarantee hoped for by making it a mandatory artefact.

All existing Scrum artefacts support the ‘inspect & adapt cycles of Scrum; they provide accurate, up to date and transparent information to be inspected and adapted at the rhythm of the Scrum events (or sooner). In that sense, Done is already an artefact; it is in the Increment, making the state of the Increment transparent.

I suggest to formally include inspection of the Definition of Done at the Sprint Review, along the inspection of the working Increment, which it is a characteristic of. This pair-wise inspection serves to get feedback and input from the stakeholders that goes beyond mere functionality and business requirements. It will invoke a collaborative conversation over quality, and requirements with regards to the quality of working software in the organization.

The Sprint Review is also the time to inspect the current state of Product Backlog, i.e. what is now Done, what was left undone in this Sprint, what was additionally turned Done. From this current state, including the latest Velocity measurement, the actual product progress is available to the stakeholders.

I suggest to lay ownership over the definition of Done with the Development Team. A definition of Done can’t be forced upon a Development Team. Neither can it be cut short by forces outside of the Development Team. The Development Team will obviously include functional quality expectations from the Product Owner. The Development Team will obviously include general, organizational expectations and compliance (e.g. from the development or engineering organization). But it’s up to the Development Team to process it in the definition of Done. Decisions over the definition of Done will depend on the presence of skills, authorizations and availability of external systems, services and interfaces. Probably a Development Team would include stubs and simulators for non-available systems, add this to their definition of Done and make the impact of these dependencies clear to the Product Owner for ordering the Product Backlog. The effect on the planning horizon will no longer only be clear to the stakeholders by inspection of the Product Backlog at the Sprint Review, but also via explicit inspection of the definition of Done accompanying the working Increment.

The inspection of the working Increment and the definition of Done at the Sprint Review, and the related collaboration of the Scrum Team with the stakeholders, will provide the Development Team with important information to sustain, evolve and grow the definition of Done. They will probably have a deeper conversation over it at the Sprint Retrospective. The self-organizing drive of the Development Team will include all that’s actually possible, dig deeper, taking into account the feedback from the stakeholders, and evolve it as part of their continuous improvement of quality.

This ownership is comparable to the Product Backlog ownership. The Product Owner has accountability over it. But it won’t withhold the Product Owner from taking into account the technical and development input from the team. It won’t withhold the Product Owner from taking into account dependencies, non-functionals and organizational expectations. And after all, the frequent inspection of a working Increment provides information on reality to all involved so they can incorporate that in their work via their accountability.

A Desirable Side-effect

Although the goal is not to promote the Definition of Done to a Scrum artefact (as shown that is not needed), I do see an optional side-effect in explicitly inspecting it at the Sprint Review: additional transparency to the overall agile adoption.

Obviously the definition of Done will not always immediately, from day 1 of the adoption of Scrum, hold every possible task, activity or requirement to render every Increment completely production-ready. There will be a gradual evolution in applying Scrum. This is good as it helps all players continuously exploit the possible to a maximum extent.

Promoting inspection of the definition of Done with the stakeholders will help identify improvement areas in capturing enterprise agility. The finding of what is/is not included provides an indication on involvement of the broader organization in agile product development, even of organizational impediments. And stakeholders, in consultation with Product Owner and Scrum Master, might want to act on these from a management change perspective.

In a transformational period, including the definition of Done as an explicit artefact in the Scrum framework will help people and organizations in the software industry to… improve from better and more realistic insights.