What is Software Design?

In 1992 an article by Jack W. Reeves appeared in The C++ Journal entitled What Is Software Design? I did not see it at the time, being only 16 years old, but the article was reproduced ten years later as an appendix to Robert Martin's book Agile Software Development. I think it is the most profound observation on software development I have ever heard. Reeves had been musing on whether or not software development counts as engineering, and he was reminded of something he had read which contended that the product of any engineering effort is some kind of document. It might be referred to as a design, a blueprint, a specification, or possibly some other name depending on the field, but in every case the document constitutes a set of instructions and plans sufficiently detailed that it can be handed to another group of people, with entirely different skills, who can proceed to construct or manufacture something from it. If the document has been produced correctly, the engineers who produced it need have no further involvement in the building of the final product.

Reeves concluded that only one kind of document in the software development process can fulfil that function, and that is the source code. That would also include all the scripts and configuration files that are necessary to build, package and deploy the software into production. Together, these all comprise the real design of the software; no other document that we tend to think of as a design document can be considered as a complete software design.

To this day there is a pervasive insistence in the IT industry on viewing coding as building software, distinct from designing software. But most real software, with today's technology, is a sequence of binary numbers stored somewhere that a machine can read which encode instructions the machine can execute. Humans don't build that. The act of building is actually performed by a machine, running a compiler. We even say doing a build to mean generating executable code from our source listings.

Now it is still arguable whether or not the source code listings constitute the real software design, but if we accept it as a premise then it sheds a lot of light on the software development process. For one thing, it recasts testing and debugging. Traditionally we view testing as part of quality assurance, and debugging as fixing mistakes in the product. But if we think of software development as designing then the purposes of testing and debugging are to validate the design. All engineers perform validation of their designs no matter what field they are working in. Indeed outside of the software field, engineers have rigorous methods and techniques for ensuring that their designs are going to work when the final product is built. It is the lack of such rigour that often leads software development to be characterised as an immature and haphazard process, and not 'real' engineering. These criticisms are not totally unfair: the discipline of software development is immature, and it is often haphazard, but these criticisms still overlook one crucial difference between software development and real-world construction and manufacturing: building software is cheap. Indeed Reeves characterised it as practically free, and that was in 1992. It is orders of magnitude cheaper 25 years later. This is sufficient to explain the lack of rigour when it comes to validating software designs: we don't particularly need it. Engineers in other disciplines don't have this luxury: a civil engineer can hardly build a bridge in order to see whether it will stand up. Instead they build scale models and perform computer simulations. In aviation, full-scale models are built and test flown to validate the design predictions. In manufacturing, prototypes are built before the design is finalised and the factories are tooled up for the production runs. In software development, we can simply do a build and test whether the software meets our expectations. Afterwards we can discard the build and completely reclaim the resources used.

There is another crucial difference between software and other types of engineering: software designs are comparatively easy to produce. Children can learn how to do it. I think one reason for this is that software designs are not subject to real-world constraints such as the laws of physics and properties of materials. Other engineers need to learn enough about these constraints to understand whether their designs will work. Software engineers do not need to worry about whether a loop will wear out because it iterated too many times or too fast; we do not fear that an if statement might collapse under the load we are putting on it. The only real constraints we face are machine resources and execution time. Otherwise, all software designs exist as points in an uncountably large space of possible designs: the programmer narrows that space down to a single design that does what they want, rather as a sculptor begins with a block of material and gradually uncovers the form buried within that they imagined at the outset. Because of the ease with which software designs can be produced and tested, they can and do grow very complex very quickly, and a large part of a programmer's job is to manage that complexity.

Even in modern agile environments there is a tendency to separate "design" activities from "coding" activities. Unfortunately, as Reeves puts it, there is no hierarchy of importance in the different aspects of software design: a mistake in a single line of code can be just as fatal, or more so, than a mistake in the architecture diagram. The small-scale design decisions affect the large-scale ones just as much as the other way around. It is sometimes said that a "software design" is worthless without being validated by code. In truth, what people think of as the software design is only part of the design, not the whole of it. If the larger-scale abstractions cannot be changed in response to unforeseen truths uncovered during the small-scale designing at the source code level, the design as a whole will become compromised because of this inflexibility. In practice we perceive this compromise as a gradual loss of the high-level software design; if unchecked, the software eventually degenerates into a big ball of mud.

If you view software development in this light, many of the practices of eXtreme Programming appear so obvious that they scarcely invite debate. Short iterations with frequent releases to get customer feedback can be seen as a gradual process of refining a design. Customer tests, test-driven development, pair programming and continuous integration add rigour to the process of validating the design, moving software development away from craft and towards becoming engineering. The practices of whole team, simple design, collective ownership, design improvement and coding standard become unarguable. If the real output of the team is a design document, then it should be the best design the team can produce, collectively owned by every member of the team, and consistent in style and standards such that that only by consulting the SCM logs could you tell which member of the team wrote any part of it. You would expect nothing less of any engineering design.

Finally, one of the aims of XP was to make software development more humane, and viewed as a design activity it becomes clear why this succeeds when other software development processes fail. Designing is an inherently human activity with a strong creative element. It is ill suited to being subjected to the weight of overbearing process, and attempting to fit it into that mould leads to frustration, burnout and lower quality. It is clear why industry wants to view software development as a manufacturing process: it is a metaphor readily understood by managers, and amenable to measurement for the purposes of reporting and hitting targets. However, just because it is convenient to see software development this way does not mean that is how it really is.

The software craftsmanship movement developed in reaction to the shift of agile from a set of ways to practice software development into a set of ways to manage software development, and was an attempt to reintroduce the practices of XP that got lost. If we begin to view software development as design, we may be able to move beyond craftsmanship and mature into a true engineering discipline.