The domination of software development by a paradigm of industrial views and beliefs, a copy-paste of old manufacturing routines and theories, got us in a crisis. The attempts to overcome this crisis by fortifying the industrial approach are without result. The flaws and problems are huge, known and well documented. The crisis largely continues to this day.
The seeds of a new world view were already sown in the 1990’s, and resulted in 2001 in the formal naming of ‘Agile’. A turning-point in the history of software development. A new paradigm for the software industry was born; a paradigm that thrives upon heuristics and creativity. A paradigm that restores the respect for the creative nature of software development and the intelligence of its practitioners.
Yet, many say that Agile is too radical. They keep propagating, to this day, a gradual -if any- introduction of Agile practices into existing, traditional processes. Having witnessed several of such attempts, I am extremely skeptical about such a ‘gradual’ evolution, a slow progression to the Agile paradigm. Because:
A gradual evolution only scratches the surface. New names are installed, new terms, tools and techniques imposed, but the fundamental thinking remains the same.
The mentioned paradigms consist of fundamentally different concepts and ideas, and many are mutually exclusive. No meaningful introduction of Agile practices in an industrial paradigm is possible.
A gradual shift is factually a status-quo. The industrial paradigm remains. A permanent crisis is called upon us.
It requires honesty to accept the mismatch of the old ways, and courageous leadership to embrace the new ways.
Abandoning the old thinking will take time. Scrum helps. Its distinct rules help in getting a grip on the new, Agile paradigm. The limited prescriptions allow immediate action. They allow developing new ways of working; through discovery, experimentation-based learning and collaboration. It is worthwhile the giant leap.
I wish you a 2016 of determination and improvement. I congratulate you with the hard work you will have performed in a year from now.
(Note: much of the above was copied, with some editing, from my book “Scrum – A Pocket Guide“, providing context to my description of the Scrum framework, its rules and roles, and their purpose)
Creative, artful thinking and craftsmanship are ESSENTIAL QUALITIES for any software developer. But shouldn’t software development be about more than an act of art, a craft in best case? Isn’t there a need to add professionalism to it? Shouldn’t software development be a profession?
It is tempting to believe that software development already is a profession. After all, many people are employed in developing software, making a living out of creating and sustaining the working code of software products and services, as a team or as individuals. Many more are employed in and make a living out of all the side-activities in the wider software development industry; like funding, exploiting, selling, promoting software products and services. This however is not sufficient to state that software development already is a profession.
Some take their dreams for real and call software development a profession. For others it is more a matter of pride, status or ego to say they are part of a profession, instead of an artisan discipline. That still is not enough to state that software development already is a profession.
A profession has regulations, codes, expectations and rules, all building upon a recognized body of knowledge. A profession sets explicit expectations on ethics, conduct and behavior to be demonstrated by any member of the profession. Official exams are in place for members to demonstrate a level of required skills and knowledge before receiving an attestation, and the permission to bear an official title that is protected and reserved. A healthy profession’s regulations and standards have a purpose, a purpose that goes beyond self-serving the profession and the profession’s institute. All regulations and standards should serve to assure the best possible execution of work in order to protect the people and organizations taking part in or being subject to the profession’s activities and its outcomes, and the decisions taken as part of it.
It is difficult to predict if and when software development might transform into a profession. It seems unlikely in a short term. There is the gigantic amount of software development work, there are the endless variations of what might be considered ‚software development,’ the absence of a widely accepted definition and understanding of ‚software development,‘ disagreement over the work activities that should or should not be part of the profession.
Reasonable doubts may be expressed on whether being a profession will augment the quality of service, avoid charlatanism, etc.
Furthermore a variety of people and instances may have personal, political, career, financial or other interests in software development, possibly even outside of the activity itself. Such people and instances might see regulation as a threat to their interests. Or might see it just as a limitation to their artistic freedom. Not to speak of the difficulty in growing a body, ethics, assessments, instructional guidelines and a code of conduct that would be widely accepted within the industry.
Where there are many elements that probably impede software development from growing into a regulated profession, there are also very valid reasons why software development would be better off if it transformed into a profession.
Not the least reason is the overwhelming and crucial presence of software in society and our daily lives. Software has become increasingly vital to society. A major shift is happening. Software products and software services are at the front and at the back of many public, industrial and consumer services and facilities. Software no longer is a ‚nice to have,’ for amusement purposes only. Software is at the heart of the economy. Software is core to many, often even critical, services in the private and in the public domain; financial services, taxes, energy, retail, health care, education, defense, telecommunication, transport, the automotive industry, just to name a few. Numerous organizations thrive on software. Even when they are not software companies, their services largely depend on software.
The omnipresence of software leads to a huge demand for talented and skilled software development people. The demand for professionalism that goes beyond talent and skills is less acknowledged but equally important in the light of the crucial functions of software in society. It would be comforting to know that the level of professionalism shown in the industry grew at the same rate as the growth of presence and criticality of software products and services itself, although even a higher rate is required.
Software development deserves professionalism in doing and in managing it. Such professionalism would benefit much from being structured in a profession.
This post is one of the inquiries I have planned into the values of professionalism in software development and the potential of the scrum framework to induce the formation of a software development profession. A matter of incremental writing. I hope you enjoy it, and the future follow-ups and iterations.
The aim to deliver valuable software is a great, core principle of the agile movement. The difficulty however is that ‘value’ in itself is hardly quantifiable. Yet, I do believe it is imperative to think in terms of value in software development and therefore overcome some fluffiness attached to ‘value’. If we don’t find actionable ways to deal with ‘value’ it might remain meaningless; another buzz word, another way of getting people to think it’s time to move on to the next hype. It is not only difficult to quantify value, it is not even necessary, maybe even undesirable. It is better to measure whether we are delivering value (effectively).
What we thought defined success
For many years we were tricked into believing that software development can be considered a success if we meet 3 criteria: deliver (1) all promised requirements (2) within the planned time (3) for the allocated budget. It is reflected in the famous iron triangle of software development.
That in turn tricked us into believing that, in order to be successful, we had to exhaustively analyze, detail and describe all requirements upfront, and get formal approval and sign off over them before the actual programming can be done. The underlying motivation is to secure the first element of what we were told defines success, the ‘requirements’.
This beforehand gathered information then becomes the input to careful and exhaustive calculations for the delivery part, often with the addition of some ‘contingency’. The underlying motivation is to set and fix the other two elements that we were told make out success, time and budget.
The result is poured into a plan, often complemented by risk mitigation strategies and other exception procedures. When this plan and all its components and clauses are approved and signed off, implementation can -finally- start. And then, obviously, it is just a matter of following the plan to be successful. Deviations from the plan, seemingly happening sometimes, are unlikely to cause any real problems, because it’s what ‘change request’ instances are for (meetings, documents, approvals, and sign offs).
Unfortunately, this was (and at many places still is) believed to be the only way we can ensure the ‘success’ of software development.
Yet, things aren’t that easy. The installments of the mentioned extra safety nets are already an indication that, despite our firmness, this approach isn’t really giving us the certainty it claims to have. And despite the safety nets, the detailed preparations and the seemingly perfect plans, many projects are plainly cancelled (29% according to the Chaos Report by the Standish Group of 2011) or in the end still fail in meeting the widely accepted success criteria of scope+time+budget (57%).
Some however are actually successful in delivering according to the three predictions (14%). But then, success doesn’t take into account that often quality was lowered to unacceptable levels in order to live up to the success criteria; reason why it’s good to add that as a fourth variable to the iron triangle (see figure). Success doesn’t take into account that the elapsed time might be according to plan, but may business-wise be extremely slow. Note that it is not unusual for projects to deliver software no sooner than 12-24 months after the start! Success doesn’t take into account that by the time the software actually becomes available for use, nobody really sees much ‘value’ in the features anymore; at all rendering them useless or in the functional way the features are implemented. More innovative ideas for them have emerged, or improved technological ways to implement them are discovered. It connects to the finding that 64% of features, produced in this way, are rarely or never used.
The three factors they told us defined ‘success’, fail in doing exactly that. And even if they are met, they still only reflect how successful the ‘delivery’ of the software to the market is, not how the software is being received or appreciated on the market place, let alone that the approach helps us addressing changes or new requests from the market place. The three factors fail in showing how valuable the software is; it only says that a version of software got out the door at certain predicted conditions.
What we say determines success
Agile overcomes the fallacy of traditional, upfront predictions and descriptions with collaboration, communication, incremental progress and continuous improvement. All risks taken are limited in time through time-boxed iterations. Time-boxing allows adaptation and adjustment within the process.
The agile movement stresses the importance of active business involvement as part of cross-functional development work. The agile movement stresses the need for frequent releases to the market place to learn what actually is appreciated. There is no better risk mitigation than regular feedback from actual users.
Scrum, as leading framework for agile software development, is designed upon the foundation that software development is too complex for a predictive approach. Scrum implements empiricism in software development, thriving on frequent inspections to decide on the best possible adaptations. Inspections however are pointless if the inspected information is not fully known, opaque instead of transparent, and if the inspection is not done against known and agreed standards and definitions.
The target of the adaptations is not only the software being produced, it is also about the tools, the engineering practices, the relationships, the process and the three elements they told us define success, i.e. budget, scope and time. Scrum does not care about the abused notion of ‘project’. The concept of ‘project’ generally only serves the idea that success is possible if software is delivered on time, within budget and has all the predicted features. Scrum focuses on the software product (having a role and an artefact for its owner and its backlog) and incremental sustenance of the product.
The foundational belief to this is that the success of software development is defined by the capability to satisfy and impress the consumers of the features, at a reasonable price, at a cost that has an effective return, with creation happening in a way that is sustainable for all people in the ecosystem of the product.
Yes, we can… measure
The success of an organization, in terms of survival, prosperity and leadership, increasingly depends on their ‘agility’, i.e. their overall capability to deliver value in a context of constant change, evolution, innovation, improvement and re-invention.
The difficulty in establishing the value of software or software requirements is that it depends on context, time, technology, market, business domain. It also cannot be reduced to one parameter only. Predicting value doesn’t make much sense either, since it can only be validated by the market place.
But if value defines success, how can we then measure whether we are successful?
Scrum.org has developed the Agility Path framework to help organisations increase their agility and move toward a value-driven existence.
Agility Path focuses not on the illusive goal of calculating and predicting value or similar magic. With Agility Path the outcome of an organization’s work is measured with metrics to help them think about the way that outcome is achieved. Organizations get a clear indication on whether they are delivering value or not. If from the metrics it seems they are not, they have many areas and domains to inspect further, and do adaptations by installing, removing or improving practices, tools and processes. It is how the Scrum framework is used to manage the change process, the transformation of an organisation required to increase their agility.
The metrics provide the transparency needed to identify the areas where adaptation has the most impact. Think in terms of business metrics like employee satisfaction and customer satisfaction, revenues, investment in agile, etc. Think in terms of delivery capability metrics like cycle time, release frequency, defects, etc.
All metrics are consolidated into an overall Agility Index for the organization. This Agility Index is an indication of how effective the organization is in delivering value. One Agility Index is bound to time, place and context. The exact figure is of no importance. The evolution is of importance. The underlying set of metrics is of importance. The insights gained from the metrics, as a source of improvement, are of importance.
No magic, just hard work
There is no magical formula to calculate value, and even when trying to calculate value to steer development priorities, it is a prediction, an assumption that remains to be validated (or contradicted) when you actually go to market. You can never be sure.
The best we can do is to capture the results of our actions with metrics, and do those measurements regularly. We detect trends and patterns (and prefer that over a naked, singular figure). We relate that back to how we work, change how we work, and measure the change in outcome of that assumed improvement. We do that endlessly and relentlessly. We continuously improve by learning and adapting. We get the most out of what we do.
This flexibility primarily helps organizations respond better and faster to change, it implements openness to change over ignoring or blocking it. Once the thinking gets ingrained and new ways of working are installed, organisations discover that there may be different options to respond, thereby embracing change as a source of information. And, finally, organisations start innovating, be ahead and cause change (for the others to follow). Thus an agile approach harnesses change for the customer’s competitive advantage, and the organization’s agility.
My daily business is software development. I live, breath and practice it from the evil and deviant perspective of Scrum. Scrum is a lightweight framework for complex product delivery. Although the focus is on a product being created, built, maintained, much software development work is organised in projects. There is -semantically- nothing wrong with a ‘project’ in itself. A ‘project’ can be described as a focused endeavour towards an objective. The real problem in the world of IT is the past, and at many places still current, interpretation and definition of a project as a ‘fixed price’ project, a project in which price, scope and time are fixed. A viable return on a project however can only be achieved by balancing 3 elements: time, scope and budget. If there’s no active balancing act (one of the elements is forced or twisted beyond some boundaries) the only element remaining that can vary is… Quality.
Sidenote: if there are no frequent, intermediate releases -see the time constraint- there is also no guarantee on the optimization of value, regardless whether the 3 expectations of end_time+budget+scope are met.
I used to illustrate this ‘fixed price’ mechanism by representing a project as a machine with 4 levers, where the lever for Quality is non-negotiably fixed at ‘High’. As the operator can only move 2 levers at the same time, the last, free, lever will automatically re-position. Trying to fix the last lever will break the machine.
Although I still like my machine metaphor, let me hereby revert to the more known iron triangle. And let me start by emphasizing that, although unfortunately often forgotten, there is a 4th element, the inner element, Quality! My triangle with its three project controls on the legs is a music triangle. If one of the legs is deformed, clinging the triangle will lead to an ugly, out of tune, non-qualitative sound. Fixing scope, time and budget will lead to a terrible sound.
Let me use the iron triangle representation in another area to explain where another ugly sound originates from.
In my software business there is a typical and deluding career path, i.e. the virtual ladder that forces people on the path of ‘growing’ from developer to analyst to project manager (and beyond). Scary, enforcing the Peter Principle like that. I fight this unfortunate hierarchical vision by always referring to ‘roles’, and never ‘positions’. And ‘roles’ have everything to do with talent and insights, and not with bossy importance.
My triangle of Valuation illustrates this vision upon the core idea that every person should look for a balance in (1) interests (the love for a ‘role’), (2) the talents to perform well and the ability to do it in an appropriate (3) (working) domain. The outcome of such balanced iron triangle will be proper Valuation, the fact that people feel and are respected. Respect should be interpreted in a broad sense, i.e. financially & in various non-material ways.
Does this model solve everything? Of course not. One’s talents might be big in a domain that the person doesn’t care about. Or a person may believe in having a certain talent, that aligns with his/her interests and make it a working domain. And still not be valued. Might be that your talent isn’t what you hoped it is. Might take time to get valued. Might never come. If you’re happy and having FUN, you might not care and keep believing.
An Agile approach to Human Resources policies does not only respect people (as individuals and as Teams) and promote sustainable pace (to name a few). But well performing in ‘Roles’ will be valued more than striving for positions, and the iron triangle of (work) Valuation is a better reference model to have a good (frequent) dialogue over the balance in it, job satisfaction, performance and… Valuation.
Frederick Brooks published his essay The Mythical Man-Month in 1975, as one of 15 essays in the same-titled book on managerial aspects of software development. I wanted to check on the actual value of this historical piece of work. I have read the 20th anniversary edition, to which the author added an essay with his updated opinion on his original statements. This edition also holds the even more known 1986 paper No Silver Bullet.
The book: modern myth or ancient history?
The 1975 cases are technically not appealing (IBM’s OS/360) and hardly relevant for today’s technological environments. But the organizational topics are striking, with the title essay on top (justly mythical by itself). Although the core statement (“Brooks’ Law”) “Adding manpower to a late software project makes it later” has again been overtaken by history. But blame the myth of the man-month for confusing effort and progress, for neglecting the need for communication, for ignoring repartitioning tasks and training, and decreasing team productivity. People (developers) are not ‘replaceable pieces of machinery ‘! Brooks also stresses the creative nature of software development.
I agree on the diagnosis, but the suggested remedies (a lot on estimating and roles) are somewhat vague and far-fetched.E.g. 1/6 = coding, surgical team and ‘directors’/‘producers’, ‘aristocratic’ architects, etc.
But then, there’s the author’s 1995 retrospective seeming to make the lecture vivid again. He formulates views that are greatly in line with… Agile. But then (bis), Brooks doesn’t seem to recognize it and sticks to his ancient solution, i.e. a sort of surgical designers. But since the ‘90s we hàve seen the definite rise of Agile (this common denominator was only established in 2001!).
Can we today overcome the curse of the mythical man-month?
I believe that at least Agile does show a way out! Of the problems highlighted by Brooks, but in a very different direction than surgical design. Because Agile has a fundamentally different approach on communication and the ‘people’ aspect of software development.
e.g. Estimating is done by the whole team and in Story Points, i.e. complexity. This is at the same time the base for measuring progress (i.e. velocity being the number of Story Points realized in a Sprint), and avoids the confusion with effort.
In his Agile bible, Agile Software Development, Alistair Cockburn has substantially stressed the fact that communication hàs a cost, increasing our awareness of the impact on progress and budget. His views on Information Radiators and Osmotic Communication have meanwhile been greatly adapted by Agile teams.
The domination of software development by a paradigm of industrial views and beliefs, a copy-paste of old manufacturing routines and theories, got us in a crisis. The attempts to overcome this crisis by fortifying the industrial approach are without result. The flaws and problems are huge, known and well documented. The crisis largely continues to this day.
The seeds of a new world view were already sown in the 1990’s, and resulted in 2001 in the formal naming of ‘Agile’. A turning-point in the history of software development. A new paradigm for the software industry was born; a paradigm that thrives upon heuristics and creativity. A paradigm that restores the respect for the creative nature of software development and the intelligence of its practitioners.
Although I still like my machine metaphor, let me hereby revert to the more known iron triangle. And let me start by emphasizing that, although unfortunately often forgotten, there is a 4th element, the inner element, Quality! My triangle with its three project controls on the legs is a 
