Matt's Ground Truth

Why Is Software Development Different? (and hard)

Matt Peloquin — Wed, 30 Jul 2008 20:29:00 GMT

Our profession spends an amazing amount of energy year after year reinventing how we create software. While looking at a thread about Agile principles on an enjoyable blog (NOOP.NL), it struck me for umpteenth time how mastering effective software development practices feels like riding a merry-go-round. There have been massive strides in software technology over the last 50 years, but our sophistication in applying that horsepower to useful problems seems stuck on a treadmill. We keep rediscovering good ideas like iterating solutions, using feedback loops, managing complexity and knowledge, avoiding waste, motivating teams, etc. These "new" ideas get dressed up as the latest silver-bullet solution for effective software development. The "new" way gains popularity, becomes baroque through embellishment, and then falls out of favor, replaced by the next "new" thing.

A recurring theme in practice discussions is why is software different from building construction? Or electrical engineering? Or ship building? Or ______? Software folks often have a grass is greener mentality regarding other construction and engineering disciplines. I have friends in engineering and construction, I follow the news, and I like watching those "building the biggest ______ in the world" shows. Other professions that create things don't seem vastly ahead of software in terms of predictability, visibility, control, and effectiveness (was your nearest sports arena built on time and on budget?). However they do seem to spend less of their collective time navel gazing, rediscovering, and getting excited about the same fundamental principles over and over again.

Software has been problematic since its inception (e.g., Apollo program). Software systems tend toward complexity. Software is less tangible and less constrained than other building mediums. There are many context shifts between pushing bits around according to the laws of physics and the interactions with an irrational end-user. The sheer size of information in the world that might be captured, created, stored, and manipulated is mind boggling (the average usefulness of that information is another question). As for unique creation vs. manufacturing replication, most software development today is unique creation -- even if it seems like it shouldn't be. Let's take it as a given that software is hard -- if for no other reason Donald Knuth, the guy who wrote all those important CompSci books that no other human has ever completely assimilated, says so.

OK, so software is hard. Squeezing an extra 10% of efficiency out of a jet engine while keeping it reliable and affordable is hard too. What is different about the software profession that causes our application of technology to real-world problems to be caught in a recurring cycle of reinvention? Here is a unscientific list of possibilities:

The rapid increase in practitioners. Just 50 years ago there were only a handful of software developers in the world. Reliable figures are hard to come by, but 15 million is often bandied about as the current number (these uses seem to derive from surveys and modeling by IDC -- I didn't have $5K handy to dig into the details). Professional organizations like the PMI and IEEE have been overrun by software folks in the last 20 years. The line between software and other professions is often blurry, as many people create some form of software for their jobs these days. Most companies we talk to are trying to expand their number software developers; if they could only find qualified candidates. Unlike other technical disciplines that had the luxury of evolving more slowly, software development had to rapidly assimilate larger and larger numbers of people from all over the world. It's not surprising that transferring knowledge and experience (whether on the job, through school, or via professional groups/networks/literature/training) of what works well and what works less well in different situations has not been particularly efficient.

The evolving and flexible nature of the medium. This makes software powerful, but also drives the proliferating churn of effective practices. Customers and clients continually expect more from software magic. Practitioners expend much of their energy just trying to tread water in the technology ocean (one senior engineer I worked with had this figured out 20 years ago -- he just "rode every-other technology wave" instead of every wave). The usefulness of software leads to it being applied to just about everything in modern life. It can be fairly straightforward to identify, compare, and contrast activities and output in other professions (medicine, for instance). The bewildering array and rapidly changing nature of software makes it more difficult for overworked practitioners to realize that many of the fundamental principles governing software creation are similar across time and space, even if work from different times and places doesn't appear similar at first glance.

A low bar to get something working. The bar to get anything up and running is much higher in EE, ME, CE, ChemE, construction, etc. It's easy in software to throw "something cool" together. This creates a dynamic where a handy spreadsheet metastasizes over 10 years into 400 linked spreadsheets used to run a large business (true story). The evolution of software development tools has been empowering, but allows people to get in over their heads. You can start building something without the knowledge and experience transfer that occurs in other technical disciplines, but the complexity of a system can spiral out of control in slow motion. You come into work one morning and wonder "how did we get to this point?" Often through a set of many logical, iterative steps that made sense at the time. New groups of people are continually pulled into creating ad hoc solutions for small problems, which eventually turn into large problems that need more sophisticated solutions. This creates a fertile ground in which smart people attack similar development challenges and end up treading down paths well-worn by others.

So what's the problem? On one hand, the "everything old is new again" dynamic really helps those who stay in the industry more than 10 years; you can largely learn new labels instead of new stuff. On the other the waste and zealotry that accompany continual reinventing and proselytizing the "new" approaches to software development is inefficient and annoying. Maybe Agile\Lean\Scrum is the last cycle in our circular evolution as a profession, but I wouldn't bet on it. The software industry still seems to be in its big bang expansion phase. Ten years ago I figured the industry would start to cool down and more professional stratification would emerge; didn't really see that "new economy" thing coming. Who knows how long it will take for the software development expansion to stabilize. Until it does, I expect we'll continue to see a lot of reinvention and repackaging of good practices.

In the end, maybe software development isn't so different after all -- the fashion industry, business books, bridge building, and lots of other stuff seem prone to recurring fads. It's nice to think we're special though...

Ground Truth

Matt Peloquin — Fri, 20 Apr 2007 19:09:00 GMT

Have you maintained software whose code bears little resemblance to its design docs and comments? How many times have you seen a detailed and diligently maintained schedule, which is regularly presented to management with lots of green and red highlighting, but which no one in the trenches thinks has much to do with the work they are accomplishing? Have you had a release sail through testing with flying colors, only to be rejected by customers because it wasn't what they wanted?

These are all examples of a lack of "ground truth". Ground truth is a key to successful software development. What the heck is ground truth and why should you care?

Creating software is difficult because we are building complex, abstract systems using highly flexible and largely unbounded mediums (e.g., napkin sketches, screenshots and stories, designs, code, content, data, project schedules, etc.). To manage the complexity and abstract nature of building software, we create all sorts of models to help us move from conceptual idea to concrete software. Requirements model what the system will be. Designs model how we we will put it together. Project plans model how we will get the work done. Test plans model how we will test it. Code, data, content, although arguably concrete, are often themselves modeling an abstraction of the real world (e.g., a user account).

All this modeling is critical for allowing us lowly humans (who can't keep more than 7 things in our heads at one time) to build complex systems. However, we need to make sure that our models do not drift from the reality they are intended to represent. This idea that models and reality should stay in sync is captured succinctly by the term "ground truth".

Ground truth came from cartography and is also used in the military. In cartography, ground truth comes from the idea that a map is an abstract model of terrain, often focused on particular attributes (e.g., terrain type, political boundaries, elevation, etc.). Saying a map has ground truth means that although it doesn't show all the detail of the real world, what it does show accurately reflects the real world. This is nicely summarized by the following advice for navigating: "when the map and terrain disagree, believe the terrain".

In the military, ground truth is expanded to mean accurate situational awareness. A commander directing a battle from HQ has information available that creates a model of the details occurring in the battle. It is critical that the model created at HQ reasonably reflects the ground truth of the battlefield -- otherwise poor decisions will be made and lives may be lost. For software development, our situational awareness encompasses what we are building, how we're building and validating it, and how we are getting all of that work done. If we lack accurate situational awareness, we will make poor decisions (e.g., leaving out a critical feature) and lose time on useless tasks (e.g., updating a schedule that no one is using to coordinate their work).

The notion of ground truth is a great concept, especially for work that relies on abstract modeling as much as software development. When I encountered the term in the (highly recommended) book Corps Business: The 30 Management Principles of the US Marines, it immediately resonated with my software development experiences. There were so many times I had seem efficiency and effectiveness suffer because models had drifted too far from reality, whether it was a fairy-tale project schedule, requirements docs that no one understood, or an architecture that could never be implemented.

How do you ensure ground truth for your software development? Using good development practices helps significantly, but there is also art involved. Open your eyes and ears to as many sources of information as you can; don't rely on your models (requirements, designs, plans) as the sole source of information. Constantly cross check information in the models to ensure both internal consistency and consistency with the real world. If you are managing work don't rely on status reports, make sure you talk informally to people doing the work about how things are going -- if you hear grumbling, investigate. If you are doing requirements don't rely on document sign-offs, informally talk through tricky details with different stakeholders (end users, customers, developers, testers, etc.) to spot check understanding. If you are designing, create functional prototypes to prove ideas and make sure the people using the design understand it. If you are testing, don't just rely on equivalence cases for written requirements, understand the operational context and be creative about what might go wrong.

Software development ground truth ultimately means building what is needed without wasting effort and time on unnecessary activities.

Throw Away Your Gantt Charts!

Matt Peloquin — Fri, 20 Apr 2007 17:44:00 GMT

Well, maybe not all of them. Some are so pretty, that uniform cascading of well ordered tasks trickling downhill to perfectly coincide with the deadline. Get color prints made, frame them, and hang them as abstract art. Don’t use them to manage your software development.

Balance is an important part of life, so I plan to add some perspective to the inflammatory opening remarks (so I can keep my PMP certification). First though, let's bludgeon the hot air out of the Gantt chart culture that inflicts many software organizations. Included in this category are the so called “activity network” models (primarily Gantt and PERT) and the tools that propagate their widespread use. Unfortunately this is pretty much every project/work management tool out there from Microsoft Project and other scheduling tools to the many group-ware collaboration websites. No matter how many pretty pictures are created, relying on these techniques to manage software work just isn't very effective.

Where does this hostility come from? Was I abused by project managers as a child? No, that came later. And then the cycle of violence continued for a time as I inflicted Gantt charts on others. Eventually it became apparent that the model of work created in the scheduling tool, with its ubiquitous Gantt chart view, usually lacked ground truth. Energy would be invested by all creating, reviewing, presenting, updating, and wringing hands over the Microsoft Project file. Then people would go and get the work done without any regard to the Gantt chart. What to do to avoid this waste?

A valid approach is to adopt task-driven methodologies such as Scrum. I’ve never been a fan of adopting methodologies though. Although there are benefits in being told exactly how to do something, customizing a simple set of principles and practices often leads to a better optimized solution.

What is the purpose of Gantt charts? They provide visibility, predictability, coordination, and control of work. These seem like important things, at least to the pointy-haired managers. Why do scheduling tools often fail to provide good visibility, predictability, coordination, and control of software work? Because they focus on time-based management of the “critical path”, i.e., dependencies and ordering of work. Most software work runs into trouble not because the critical path was mismanaged, but because EFFORT was underestimated. If you do not have high confidence in your estimates, building elaborate Gantt charts based on those estimates will not increase your ability to manage work effectively.

What are some basic principles you can incorporate into your planning and management of work? Regardless of lifecycle and methodology all software work boils down to deliverables people need to create. Focus on these deliverables. Have individuals and teams define the deliverables for a release (sprint, milestone, whatever). Have them estimate effort ranges (not time or dates) for the deliverables. Use a simple spreadsheet to track effort expended on and completion of deliverables (and/or use work queues and burn-down charts). Let the team work out the fine-grain details and dependencies of the work on their own, whether formally with a methodology like Scrum or informally. Look at the big picture on a regular basis and see where adjustments may need to made.

Gantt chart views are important for managing a complex series of work, for instance, a team with lots of dependencies on other teams. They work best when focused on deliverable milestones rather than task details. Just as different views of a design (e.g., physical and logical decomposition, database, UI screens) increase the understanding of the software to be built, multiple views into software work increase the understanding of how that work is going.

Thus you may not need to throw your Gantt charts out, but you should only rely on them for what they are good at. And that is not for planning what task Tom and Susie will be working on at 2:30 pm this Wednesday.