Product Backlog and the tea leaves effect (a message to Product Owners)

Product Backlog is a strategic instrument of high value; for stakeholders, product management, the organization, the Development Team(s), and the Product Owner. When employed well, Product Backlog reflects the vision, ambitions, roadmap, needs and wants for a product or a service, all in one place. Imagine how your competitors would love to have access to it!

Product Backlog has the potential, as a single artefact, to replace a multitude of other predictive plans and traditional documents. In Scrum, Product Backlog is all the plan you need. For Product Backlog to enhance simplicity and transparency, Product Owners maintain Product Backlog continuously. The Product Owner, actually owning the product, orders and manages it to reflect the actual needs, wants and ambitions.

Product Owners likely apply multiple hands-on strategies to keep Product Backlog tidy and accurate. It helps for many Product Owners to realize how the tea leaves on a Product Backlog impact transparency.

Having explained Product Backlog and the Tea Leaves Effect in the past, I have now summarized it in a short movie. It takes less than 2 minutes of your time. Enjoy!

Product Backlog Upholds Transparency

The software development industry has long suffered from a dogged obsession with identifying, checking, detailing, double-checking, describing and documenting perfect and complete sets of requirements before allowing ‘development’ (typically only the coding or programming part of it) to start. Typically such a requirements phase consumed vast amounts of valuable energy, time and budget, causing gigantic delays before the core work of software development even begun. Besides this trauma, such a requirements approach often caused what became known as ‘analysis paralysis’. The obstinate desire for perfect requirements completely grinded projects as an overwhelming amount of information was collected, along with the abunding amount of requirement interconnections. We lost overview, we certainly couldn’t document it anymore, let alone explain it and get a sign-off. The insurmountable information overload caused a hard stop. Basically, projects already got off track during this preparation phase, got beyond recovery and were canceled before the real work could start.

All energy was spent on what in the end is only an intermediate deliverable. The requirements turned into a goal in themselves. It was ignored that only during implementation the real discovery of what works and what not works happens, when functional options are validated against technology and turned into usable versions of product. And, ultimately, real progress of any software product development effort is in working software, versions that can be released to and be tested by users.

This is reflected in the value statement of the Agile Manifesto:

“Working software over comprehensive documentation.

as well as in one of the 12 Agile principles:

“Working software is the primary measure of progress.”

Scrum implements this through the demand to have Done Increments of software by the end of every Sprint, where a Sprint takes no more than 30 days, and often less.

This demand supports breaking away from the false illusion that there is a direct correlation between the perfection of a set of requirements and the value of the software produced. There is no such cause-effect linearity. The chance of success of a software development endeavor is not linearly related to the level of detail, documentation and completeness of its requirements. Any requirement, no matter how well documented and agreed upon can still become obsolete, its content may change, its appearance when coded may differ or its delivery may not result in the expected value and satisfaction. Regardless how well it was thought through in advance, documented and agreed upon.

Despite the adoption of Agile frameworks, and the widespread adoption of Scrum in particular, we have a long way to go.

In many environments, as part of what is called ‘Agile’, requirements are now captured as User Stories, sometimes even on index cards. However, the hidden desire for perfection hasn’t changed. The belief that more detailed requirements will lead to better software remained intact. The courage or insights to acknowledge that progress is not measured on intermediate artifacts is missing. The ‘analysis paralysis’ phenomenon is replaced by a ‘story card hell’ [1]. Efforts are undertaken to collect all stories that can possibly be thought of, and for all stories identified, attempts are made to have every detail documented, in a tool or on its index card.

The format has changed, the obsession with perfect requirements hasn’t.

In the system called ‘Scrum’ one, and only one, source of input is defined, Product Backlog.

Closed Loop Feedback (Scrum)

All work for a product is captured in such Product Backlog, regardless whether the work is performed as a ‘project’ or not.

Product Backlog management in the empirical process that Scrum implements, holds that items in Product Backlog are added and changed as more is learned from the actual work. There is no need, nor a demand to have a full Product Backlog in place before development can start. A Product Owner having sufficient trust, ideas and funding often even provides the best starting position for innovation and new product development.

Product Backlog is incrementally managed:

  • Ordered against time (for assumed value). Understanding that value remains an assumption until the work is released.
  • Gradually decomposed.
  • Regularly refined, cleansed and updated.

Product Backlog refinement

Scrum prescribes no specific format or tools for a Product Backlog, or the items on a Product Backlog. Scrum certainly has no obligation to use the format of User Stories for all items on the Product Backlog.

Product Backlog serves transparency.

Product Backlog is expected to hold all types of work required to deliver or enable delivery of releasable Increments of product by the end of every Sprint.

Although Product Backlog might hold references to other sources and deliverables, in itself it remains the single source of truth.

Scrum’s principles and rules may not create a requirements paradise, but as a framework it allows finding distinct ways to deal with requirements in a way that overcomes ‘analysis paralysis’ without ending up in a ‘story card hell’. Keeping Product Backlog minimal and removing items (the tea leaves effect) is probably the most overlooked Product Backlog management strategy, yet it will get you a long way. Progress is measured and visualized, Sprint by Sprint, upon Product Backlog converted into Done Increment and the new insights applied on open Product Backlog.

Still. I see the challenges.

We can do better a job providing inspiration to enact the simplicity of Scrum to better deal with complexity. We haven’t been clear enough in helping teams, organisations and Product Owners more effectively utilise Product Backlog, one Product Backlog.

The challenges in the more effective utilization of Product Backlog are to start using Product Backlog to decompose from goals, objectives, functions and domains to actionable work, certainly in scaled implementations of Scrum. Or to use Product Backlog to create value and impact, while managing dependencies. To use Product Backlog to forecast, create product roadmaps, align work with other products and report on progress, budget and value. To use Product Backlog at the program and portfolio level.

Product Backlog can serve these purposes. We could have made people be aware of it more.

Product Backlog is open to adding metadata and properties for and grouping notions of Product Backlog items. Such additional metadata allows creating different views into Product Backlog while upholding one Product Backlog as the single source of truth, while upholding total transparency. Transparency is not upheld through an immense inventory of requirements with an immense inventory of detailed descriptions, not even in one Product Backlog, let alone spread across separate whatever-type-of backlogs.

Many techniques and practices exist surrounding Product Backlog. They might require their own artifacts, yet they cannot replace Product Backlog.

Product Backlog serves to uphold maximal transparency in product development when employing Scrum. Product Backlog, in the end, is all the plan you need.

[1] The term ‘story card hell’ was first used by James Shore, in a 2005 presentation, http://www.jamesshore.com/Presentations/Beyond%20Story%20Cards.html

Product Backlog and the Tea Leaves Effect

The Tea Leaves effectDo you like drinking a cup (glass) of tea? Ever had a tea bag that was torn? Spreading tea leaves in your tasty drink?

Then you know that such leaves circulate wildly and chaotically throughout your tea when you stir it, add water, or drink it. Science is somewhat more precise about the behavior of such leaves, calling it the tea leaves paradox, but let’s just stick with our more direct observations.

Floating tea leaves are best removed from the tea. It prevents obscuring of the tea, and makes a better and tastier drink. Most consumers prefer tea without such leaves. If not for the look of the tea, then certainly because it’s no fun to drink tea leaves.

The tea leaves effect is a good metaphor to understand the need to refrain from collecting and listing requirements in a Product Backlog endlessly. The tea leaves of your Product Backlog are those requirements that go up in ordering every time the Product Backlog gets stirred. They float around, obscuring your Product Backlog and decreasing its transparency, and they always sink to the bottom as the environment settles again.

Keep your Product Backlog clear, clean and tasty. Remove those features that cloud your Product Backlog. Remove those features that always sink to the bottom of what you plan for your product, and never get implemented. If those features are really valuable, they will pop up again at some point in time.

It connects to ‘trimming the tail‘ as described by Alistair Cockburn. The goal is to not build low value ideas. It sounds simple, the gains are considerable, but it takes much discipline. But you don’t want to feed your customers tea leave requirements, right?

The Value of the Product Backlog

Grafx - Scrum Gameboard (non-branded)Scrum is a light-weight framework with a minimal set of rules. The rules help people to empirically make the most out of every single day of creating software products. Next to 3 roles and 5 events, Scrum requires no more than 3 artefacts:

  • Product Backlog
  • Sprint Backlog
  • Increment (Potentially Shippable –)

Product Backlog holds desirements

It is often said that the Product Backlog must capture all requirements. However, the Product Backlog is not a replacement for the old requirements list. This would limit it to a new name for an old habit. The value of the Product Backlog lies not in precision, in detail or in perfection, like the requirements lists pretended.

The Product Backlog is an ordered list of ideas, features and options to bring an envisioned software product to life or sustain and grow it. The list may include fixes, maintenance work, architectural work, or requirements for security, scalability, stability and performance. Each item on the Product Backlog seems at some point in time valuable for a customer. Every item on the Product Backlog holds just enough detail to make clear what it represents, but the item is also intentionally incomplete to invoke additional and explicit conversation over it. Even the definition of ‘just enough’ varies over time; items on the Product Backlog that are far away in time (ordered low) need even less detail than items that are nearby in time (ordered high).

Gradual Product Backlog Refinement

I like the term ‘desirement’ for a Product Backlog item. The level of description and detail of the item lies somewhere between what used to be a desire and what used to be a requirement; where a ‘desire’ is too fuzzy to work on and a requirement is over-specified and over-detailed. And over-specification impedes an optimal use of technology, blocks capitalizing on synergies between different functions and is a waste of money in situations of even minimal turbulence or change.

As life progresses and the earth keeps turning, Product Backlog in Scrum gets refined, adjusted and updated. The list is continuously sorted and re-sorted by the Product Owner, who thereby looks to balance the needs of all stakeholders. And by continuously keeping to ‘just enough’ descriptions and designs of the work, i.e. leaving out unnecessary details, no excessive money and time is wasted when the item in the end doesn’t get created or is implemented in a different fashion.

Product Backlog is all the plan you need

Desirements move upon their ordering from Product Backlog via Sprint Backlog into Increments of working software. To know their cost, each is assigned an idea of the effort to get it done. The cost is generally expressed as the relative size of the item. Based upon the empirical past that showed how much work on average could be transformed into a working Increment during a Sprint, an expectation can be created on when an item on the Product Backlog might become available in the evolving product. It gives predictability, yet not transgressing into predictions given that any such expectation is constrained by today’s knowledge and circumstances.

  • In a traditional plan all requirements are gathered, listed, described, analyzed, estimated and decomposed upfront. All tasks for all requirements are elaborated, sequenced and resource-wise assigned. This way the total time is determined it will take to build the plan, where ‘follow the plan’ determines the success. Dependencies are handled via the detailed tasks, their sequence and their mutual impact.
  • In the ‘just enough’ approach of Scrum however, there is no need to plan all tasks for all ‘requirements’ upfront. Desirements are gradually discovered and refined, and they are only converted into development work when included in a forecast for a particular Sprint. The forecast is based upon the relative size indication and the past progress.

Product Backlog is all the plan you need, its desirements hold all the information needed for predictability about scope and time (if that’s what you need).

There is value in value

Priorities in traditional approaches are, besides ease of work and loudmouths, mostly based upon effort and risk. This is a focus on the internal process.

An important principle of agile however is “to satisfy the customer through early and continuous delivery of valuable software.” While the ordering of Product Backlog items happens upon a complex combination of factors like dependencies, priority, cohesion and consistency, most interesting is the required addition of a notion of (Business) Value. Without it, a Product Owner has no idea on how much value a feature, an idea or a feature set presumably brings to the customer whom he/she represents to the Scrum Team. The value can be indirect, in it that not picking up a Product Backlog item might undercut the value of the system or even the organization, or produce negative value.

The notion of (Business) Value also helps Product Owners and their stakeholders move away from the (false idea of) perfection of a total product that must be completely built before releasing. Focus shifts to a minimal marketable product release, the minimal work it takes to bring as much value as possible as soon as possible to the marketplace. Techniques like leveling up can be used to group Product Backlog items into cohesive feature sets.

The value of the Product Backlog

Above all, the value of the Product Backlog lies in transparency, in making clear what work needs to be done in order to create a minimally viable and valuable product (or product Increment). The Product Backlog brings out in the open all work, development, compliances, and constraints that the team has to deal with to create releasable software.

However, too often (and especially in traditional software development environments) the sequence of implementation is based upon priorities like MoSCoW, thus leading to opaque, large clusters of requirements, loss of focus and heavy releases. Therefore, to build valuable stuff, there is value in defining, considering and adding value to Product Backlog items. A Product Backlog item needs the right attributes to be ordered, more than just prioritized.