Our profession spends an amazing amount of energy year after year reinventing how we create software. While looking at a thread about Agile principles on an enjoyable blog (NOOP.NL), it struck me for umpteenth time how mastering effective software development practices feels like riding a merry-go-round. There have been massive strides in software technology over the last 50 years, but our sophistication in applying that horsepower to useful problems seems stuck on a treadmill. We keep rediscovering good ideas like iterating solutions, using feedback loops, managing complexity and knowledge, avoiding waste, motivating teams, etc. These "new" ideas get dressed up as the latest silver-bullet solution for effective software development. The "new" way gains popularity, becomes baroque through embellishment, and then falls out of favor, replaced by the next "new" thing.

A recurring theme in practice discussions is why is software different from building construction? Or electrical engineering? Or ship building? Or ______? Software folks often have a grass is greener mentality regarding other construction and engineering disciplines. I have friends in engineering and construction, I follow the news, and I like watching those "building the biggest ______ in the world" shows. Other professions that create things don't seem vastly ahead of software in terms of predictability, visibility, control, and effectiveness (was your nearest sports arena built on time and on budget?). However they do seem to spend less of their collective time navel gazing, rediscovering, and getting excited about the same fundamental principles over and over again.

Software has been problematic since its inception (e.g., Apollo program). Software systems tend toward complexity. Software is less tangible and less constrained than other building mediums. There are many context shifts between pushing bits around according to the laws of physics and the interactions with an irrational end-user. The sheer size of information in the world that might be captured, created, stored, and manipulated is mind boggling (the average usefulness of that information is another question). As for unique creation vs. manufacturing replication, most software development today is unique creation -- even if it seems like it shouldn't be. Let's take it as a given that software is hard -- if for no other reason Donald Knuth, the guy who wrote all those important CompSci books that no other human has ever completely assimilated, says so.

OK, so software is hard. Squeezing an extra 10% of efficiency out of a jet engine while keeping it reliable and affordable is hard too. What is different about the software profession that causes our application of technology to real-world problems to be caught in a recurring cycle of reinvention? Here is a unscientific list of possibilities:

The rapid increase in practitioners. Just 50 years ago there were only a handful of software developers in the world. Reliable figures are hard to come by, but 15 million is often bandied about as the current number (these uses seem to derive from surveys and modeling by IDC -- I didn't have $5K handy to dig into the details). Professional organizations like the PMI and IEEE have been overrun by software folks in the last 20 years. The line between software and other professions is often blurry, as many people create some form of software for their jobs these days. Most companies we talk to are trying to expand their number software developers; if they could only find qualified candidates. Unlike other technical disciplines that had the luxury of evolving more slowly, software development had to rapidly assimilate larger and larger numbers of people from all over the world. It's not surprising that transferring knowledge and experience (whether on the job, through school, or via professional groups/networks/literature/training) of what works well and what works less well in different situations has not been particularly efficient.   

The evolving and flexible nature of the medium. This makes software powerful, but also drives the proliferating churn of effective practices. Customers and clients continually expect more from software magic. Practitioners expend much of their energy just trying to tread water in the technology ocean (one senior engineer I worked with had this figured out 20 years ago -- he just "rode every-other technology wave" instead of every wave). The usefulness of software leads to it being applied to just about everything in modern life. It can be fairly straightforward to identify, compare, and contrast activities and output in other professions (medicine, for instance). The bewildering array and rapidly changing nature of software makes it more difficult for overworked practitioners to realize that many of the fundamental principles governing software creation are similar across time and space, even if work from different times and places doesn't appear similar at first glance.

A low bar to get something working. The bar to get anything up and running is much higher in EE, ME, CE, ChemE, construction, etc. It's easy in software to throw "something cool" together. This creates a dynamic where a handy spreadsheet metastasizes over 10 years into 400 linked spreadsheets used to run a large business (true story). The evolution of software development tools has been empowering, but allows people to get in over their heads. You can start building something without the knowledge and experience transfer that occurs in other technical disciplines, but the complexity of a system can spiral out of control in slow motion. You come into work one morning and wonder "how did we get to this point?" Often through a set of many logical, iterative steps that made sense at the time. New groups of people are continually pulled into creating ad hoc solutions for small problems, which eventually turn into large problems that need more sophisticated solutions. This creates a fertile ground in which smart people attack similar development challenges and end up treading down paths well-worn by others. 

So what's the problem? On one hand, the "everything old is new again" dynamic really helps those who stay in the industry more than 10 years; you can largely learn new labels instead of new stuff. On the other the waste and zealotry that accompany continual reinventing and proselytizing the "new" approaches to software development is inefficient and annoying. Maybe Agile\Lean\Scrum is the last cycle in our circular evolution as a profession, but I wouldn't bet on it. The software industry still seems to be in its big bang expansion phase. Ten years ago I figured the industry would start to cool down and more professional stratification would emerge; didn't really see that "new economy" thing coming. Who knows how long it will take for the software development expansion to stabilize. Until it does, I expect we'll continue to see a lot of reinvention and repackaging of good practices.

In the end, maybe software development isn't so different after all -- the fashion industry, business books, bridge building, and lots of other stuff seem prone to recurring fads. It's nice to think we're special though...