Technical Debt

re: Technical Debt

Basil Vandegriend — Tue, 20 Nov 2007 05:02:13 GMT

Great article, Steve. I always liked the concept of technical debt, but never thought about it to the depth you went in this article. I agree with the first comment that descriptive labels are much better than numbers, and like the labels with the short examples you proposed in your replying comment.

One of the biggest issues I have with accumulating technical debt is the risk that management will not appreciate the hidden costs associated with that debt, if not be oblivious to the existence or possibility of technical debt in the first place. I liked the ideas you suggested of tracking debt as either defects or as part of the product work list (ala Scrum) in order to make technical debt more visible.

re: Technical Debt

Aaron Erickson — Mon, 12 Nov 2007 18:15:03 GMT

Link correction - sorry about that.

blog.magenic.com/.../On-Technical-Debt-_2D00_-Revisiting-the-Technical-Mortgage.aspx

re: Technical Debt

Aaron Erickson — Mon, 12 Nov 2007 17:47:45 GMT

I love the analogy ... though it is one I have been talking about for a long time (<a href="blog.magenic.com/.../On-Technical-Debt-_2D00_-Revisiting-the-Technical-Mortgage.aspx">Techincal Debt vs Technical Mortgage</a>).

Any simplistic idea about the binaryness of quality needs to go away. I am glad the discussion is starting to appear in more mainstream places.

re: Technical Debt

Travis Jensen — Fri, 09 Nov 2007 00:21:07 GMT

This was a great post, Steve, and something I've been trying to quantify for some time. On some things, calculating cost is easy: I have to keep paying person Y to do function X because we are using system Z. In other cases, it is harder.

I do want to think a bit more about the notion of debt and how it really plays in agile development, where doing things the simplest way is not a negative. Certainly, in that context, the most painful debt is failure to refactor. Many times, we just "hack it in" because we don't want to pay for the refactor, but then we've introduced a pain point that will only grow.

Interesting topic, and undoubtedly one I will write more on at my blog as well (shameless plug: http://cmssphere.blogspot.com :).

Travis

re: Technical Debt

Steve McConnell — Thu, 08 Nov 2007 19:49:35 GMT

Nathan, Great comments. I agree that estimating the interest payment on technical debt is challenging at best. I do think you can estimate the debt itself -- the "principal" on the debt, so to speak.

When you get to a point where you are debating taking on technical debt, there are always at least 2 possible paths, one of which is the "good but expensive" path and one of which is the "quick and dirty" path. You wouldn't take on the technical debt in the first place unless you perceived a difference in cost between the "good" path and the "quick" path. When you reach that decision point, you could estimate the good path and the quick path. That estimate will help inform which path you should choose at that moment.

Separately, you should also estimate how much it will cost to backfill the good path *after* you've already gone down the quick path. Backfilling the good path will typically be more expensive than just doing the good path in the first place because the work will include ripping out the quick code, making sure you didn't introduce any errors while doing that, then adding the good code and going through the normal test & QA processes. The "ripping out" part makes it cost more to implement the good path later than it would have cost to implement it in the first place. And of course you've already incurred the cost of the quick path, so the real cost is the sum of the quick path + the clean path + the cost to rip out the quick path.

If the code is really well designed the "ripping out" cost can be minimal, but I think that's the exception.

The "interest payment" is trickier and depends very much on the specific case. I don't think there always is an ongoing interest payment. Sometimes the "interest" is really just the cost of ripping out the quick code and of implementing the good code. Other times the quick and dirty approach does create ongoing interest payments by making other related work take longer.

I think this might be a case where *controlling* this aspect of the project is a better choice than *estimating* it. I might encourage teams looking at these decisions to try to identify the cost to implement a quick and dirty path that doesn't affect other parts of the system (i.e., that doesn't create any ongoing interest payments) and also identify the cost of a quick and dirty path that does affect other parts of the system. So the decision table could look something like this example:

Option 1

Immediate cost of Good Solution: 10 staff days
Deferred cost to retrofit Good Solution: 0 staff days

Option 1 cost now: $6,000
Option 1 cost later: $0
Option 1 total cost: $6,000

Option 2

Immediate cost of Quick & Dirty solution with no interest payment: 3 staff days
Deferred cost to retrofit Good Solution: 12 staff days

Option 2 cost now: $1,800
Option 2 cost later: $7,200
Option 2 total cost: $9,000

Option 3

Immediate cost of Quick & Dirty solution with possible interest payment: 2 staff days
Deferred cost to retrofit Good Solution: 12 staff days
Estimated cost of "interest payments": 1-3 staff days

Option 3 cost now: $1,200
Option 3 cost later: $7,800-$9,000
Option 3 total cost: $9,000-$10,200

re: Technical Debt

Nathan Henkel — Thu, 08 Nov 2007 05:29:19 GMT

Steve,

As I've often found to be the case, your analysis is detailed and well thought-out. I have an observation and a question.

Observation:

You mention that what I'd call "pure debt-reduction" phases rarely turn out well. I'd agree with that, and I think the reason why is that it's really difficult to tell if your improving software without some sort of validation of the improvement. I might make changes that make a section of the code _look_ a lot prettier to me, but if those changes don't make the code more flexible, I haven't gained anything, and may have made things worse. It's usually hard to tell if you are actually making your code more flexible in the absence of a flexor--a new requirement or other required change. I think usually technical debt reduction is not an activity in itself, but a way of approaching your normal activities. If debt-reduction in a particular area is desired, managers would encourage programmers working in that area to pursue the most flexible solution, even if it takes longer, to spend extra time refactoring, etc.

Question:

How do you go about estimating the principle and the interest rate of the debt you're taking on? These things are typically fairly explicit when taking on financial debt, but are not so easily determined when it comes to technical debt. It seems the principle would be the cost of undoing the hack and redoing it the right way, and the interest would be the anticipated cost of changing new code that is somehow bound to the hack. However, determining the interest seems pretty hard here. If the hack happens just before a release, I guess you don't need to estimate the interest IF you can make your first priority paying off the debt in the case where the interest turns out to be too high (i.e. many features in the new release will be hard to implement with the hack in place). Otherwise, you have to know what requirements are coming down the pipe, and the design implications of those requirements to accurately assess the interest. Assuming you can actually do this, you also have to accept that the interest rate is variable, meaning periodic re-estimation, the estimated cost of which must be rolled into the interest rate.

All this isn't to say that it's impossible to do this effectively, but it's clearly seems a lot harder to manage knowledgeably than financial debt. I think this may be one reason that programmers shy away from this kind of debt--they know, whether because they've thought about it or just experienced it, that this sort of debt is difficult to manage.

re: Technical Debt

Steve McConnell — Tue, 06 Nov 2007 22:23:27 GMT

Big Maybe, Some of our clients have used the "tax" metaphor. I've mostly heard it used to refer to overhead required to develop a component within a larger system. I.e., to create even a "hello world" program the tax burden to work with the other programs is 5 staff weeks--after that you can actually start adding functionality.

That makes sense to me. Beyond that, I'm not sure I see a lot more implications of that particular metaphor, but I'd be happy to have someone *else* do a blog posting on it ;-)

re: Technical Debt

Big Maybe — Tue, 06 Nov 2007 16:09:00 GMT

Chui Tey: Both points are excellent insights.

Another common software metaphor borrowed from the financial world: taxes. Some engineers use the term "taxes" to refer to code that is needed to support "regulatory" requirements, such as interoperability, accessibility, power management, and other features that are imposed on ALL software written in a similar "tax jurisdiction" (such as the same OS [country], or corporate network [county]).

A lot of debt is incurred just to meet a system's tax bill. Not to stretch these metaphors too far, in a similar situation in the real world, it is pretty bad sign when a business borrows to pay taxes.

I think that exploring the reaches of the software tax metaphor deserves a post of its own. Eh, Steve? How about it?

re: Technical Debt

vb — Tue, 06 Nov 2007 12:45:33 GMT

Hi,

It is interesting to what lengths people can go to describe what happens in companies which provide services.

Because it is software we are more alert and alive to the problem.

I feel that the same situation exists in all other skill oriented industries - the only difference is that it does not get written about..

re: Technical Debt

Chui Tey — Tue, 06 Nov 2007 11:12:17 GMT

I'd like to suggest the following class of technical debt

Low interest debt in high inflationary environment.

In high inflationary environment it is often good to borrow money to purchase assets rather than holding on to cash. Similarly, some classes of technical debt can be paid simply by Moore's Law of inflating processing power.

Case in point is how Microsoft products have been panned for being memory hogs in their early days, but in the long run, it became a strategic advantage as hardware became cheaper.

Similarly, using a high level language in a project despite early performance problems is akin to making technical bets on how fast the underlying technology can catch up.

Another kind of technical debt is the zero interest debt. This is where you make a hackish workaround that may never require to be fixed.

When thinking about technical debt, it is also necessary to talk about technical equity, and return on equity. I'm channeling Kent Beck on this one, where developers are prone to developing features which are not required. In which case, there is no return on the invested equity.

re: Technical Debt

Dave R — Mon, 05 Nov 2007 23:08:53 GMT

Interesting stuff as usual, Steve.

(Probably a multiple post, sorry....)

We are doing this at present, modifying database tables to change data types to accommodate future enhancements. It's repetitive grot work, the developers hate it, and there's no immediate benefit; as the risks of squeezing in small enhancements outweigh the political benefits.

The trade-off comes next low season when we have an infinitely more flexible base to provide potentially much bigger payoff.

... and now I have a new metaphor with which to dazzle the CEO ;)

re: Technical Debt

cjlotz — Mon, 05 Nov 2007 06:01:22 GMT

Thanks for a great discussion! In our team we created a wiki page of items that we view as technical debt. The idea is to have it public to remind ourselves of the items that we feel need to be addressed in the future. The items are mostly Type I and Type II.A. Team members are encouraged to take up some of these items whenever the opportunity arises and we also try and address a bigger chunk of these items in an iteration of technical debt work immediately following a big release.

Items completed are marked as completed on the wiki page. We find it quite encouraging to see the list of items completed together with the work that still needs to be done.

re: Technical Debt

Ward Cunningham — Sun, 04 Nov 2007 19:58:03 GMT

This analysis and discussion is consistent with and and adds to the debt metaphor I introduced in my 1992 experience report. In that report I used "consolidation" to refer to refactoring, a word that was not in common use at the time.

I would also sometimes explain refactoring to my management as follows: "We tried to add the feature to our application but found that there was no place for it. So we first made a place, and then added the feature there."

I now ask teams to produce new functionality and new opportunity with each iteration. This requires skills that were not valued until recently. It was simply too easy to "burn" opportunity to deliver features and running out of opportunity was assumed to be a natural conclusion to any project.

re: Technical Debt

Hitesh — Sat, 03 Nov 2007 07:24:02 GMT

Hi Steve, Great post.

Richard, I guess we are borrowing time! And payback in spending more time in the future.

re: Technical Debt

Robin Barooah — Sat, 03 Nov 2007 03:42:41 GMT

As a follow up to my last comment, and to echo something others have already said here:

There's no way to pay off the design debt 'for it's own sake'. After all, if the system works, there's no justification for changing it. Also, there's no clear way to identify what constitutes the debt, or when it is paid off.

As far as I can see, the only way to meaningfully pay off the debt is in the context of refactoring in order to allow new functionality to be built without having to implement workarounds or duplication.

re: Technical Debt

Robin Barooah — Sat, 03 Nov 2007 00:02:41 GMT

I think that one of the reasons why developers prefer to avoid technical debt is that technical debt is not fungible the way financial debts are.

Putting that another way - financial success of the product won't change the fact that the only way to pay off the technical debt is for the developers to do painstaking and laborious work - which hurts the developers personally, whereas a financial debt can simply paid off out of revenues without anyone's job being worse.

This might further explain why business people are prepared to accept technical debt - they aren't the ones who are going to have to pay it off. The developers suffer real consequences because they aren't learning or growing by having to rework code that they knew was being done to substandard quality in the first place.

It's also hard to ensure that all of the technical debt is paid off. There's no easy way to get a balance, and the interest compounds at a very high rate.

re: Technical Debt

Stephane Grenier — Fri, 02 Nov 2007 19:48:08 GMT

Great article! I recently wrote something similar: www.followsteph.com/.../developer-debt

One thing I think that really helps to mitigate this is the Agile process. It's by no means a Silver Bullet, but what it does do is help bring technical/development debt to the surface on an ongoing basis. You can't escape it.

A lot of people want to think Agile development is fly by the seat of your pants development, or that it's a Silver Bullet. Not at all. It's basically a strategy for bringing forward issues such as technical debt on an ongoing basis (as well as continually giving your client deliverables, etc.). It gives you the ability to continually decide what's worth adding to your credit card. And because of that, if used improperly it can be pretty easy to nickel and dime'ing yourself to debt. It must be used carefully, like all other development processes.

But because of the power the Agile process does have in bringing forth all these issues on an ongoing basis, making you face them rather than shoving them under the carpet, I believe it can really help many teams. At the very least, you're going to be forced to make the decisions of whether or not to take on the debt.

re: Technical Debt

Steve McConnell — Fri, 02 Nov 2007 18:33:06 GMT

Richard, No question that the analogy is going to break down at some point. The "zwang" is that the more debt you accumulate, the more your rate of progress (i.e., velocity) will slow down. Like business debt, there isn't really an expectation that you *ever* pay it off completely. The goal is to keep the debt service at a reasonable level compared to your other expenses & investments.

I wouldn't try to convince senior management to *pay off* the debt. I would try to convince them to *pay down* the debt (i.e., pay off part of it). There are several possible leverage points you can use with senior management:

* "We'll be able to get future releases out in 5 months instead of 6 [or whatever] if we can spend X-amount of time paying down our technical debt."

* "We can add functionality in area X, which currently we can't do, if we spend Y weeks working on the technical infrastructure. Once we build the infrastructure for X, it will also allow us to A, B, and C fairly easily, which we can't do now without that infrastructure."

* "Our response time on hot fixes is really slow because there are major sections of the code that have become so complex that no one knows how to make changes in those areas safely. We end up with long review, test, and debug cycles even for simple changes, which requires a lot of time to release even a simple hot fix. If we spend X time paying down the debt, we'll be able to reduce our average hot fix time by Y."

* "Some of the 'shortcuts' we've been taking are starting to become visible to the customer, and the number of customer-reported defects has been increasing. Until we pay off some of the debt, we're going to have an increasingly difficult time assuring the quality of our system prior to release."

These are just a few examples of the kinds of indirect costs you get when you have a high technical debt load. If your level of technical debt is such that you can't promise specific business benefits along these lines from paying off the debt, then it's always possible that the debt isn't worth paying off.

Another reaction I had to your comment is that I don't think I've ever seen a case where diverting a staff of 20 for 3 months to focus on debt reduction makes sense. I've seen companies attempt such initiatives many times, and they nearly always end up being unfocused boondoggles that don't produce enough business value to justify their time or cost. I would suggest instead breaking the debt reduction payments into much smaller pieces and then including some percentage of debt reduction work into the team's normal work flow. Software teams will often tell you that "It will be more efficient to do the debt reduction all at once," but it really isn't because doing it all at once removes the "business value grounding" from the work.

re: Technical Debt

Richard Quinn — Fri, 02 Nov 2007 18:05:10 GMT

Thanks to Big Maybe and Steve for the answers.

I had already understood that quality was being sacrificed in order to gain development time. The new concept introduced in the article is the idea that the quality is not really sacrificed, at some point later it will be added (with interest) - therefore it was just borrowed and not sacrificed. In any case, the premise of the article is to use a terminology familiar in a business setting. So here we are borrowing dollars which would have been spent on programming days now which should be paid back later. The thing that was purchased was the business benefit gained from using the delivered functionality n months sooner than otherwise.

In a financial setting, the questions would be: we borrowed money. What will happen if we default on the repayments? In the real world foreclosures will happen, and so the (monetary) debt really must be paid back to avoid foreclosure and other nasty events.

What is the "zwang", the necessity, to repay the quality debt? How does the threat of foreclosure transfer from the financial to the software engineering world?

Currently I am having difficulty justifying taking a development team of 20 core developers + ancillary, test and documentation staff away from programming (delivering business value) for three months in order to straighten out software which has been deteriorating for years. I have no bailiffs, sheriffs or jails with which I can threaten senior management with dire consequences for none-repayment of the quality debt. In previous years, and most probably int the near future these people have consistently chosen to borrow more instead of servicing debt.

Thanks for your time,

Richard

re: Technical Debt

Doug Sjoquist — Fri, 02 Nov 2007 17:03:29 GMT

Type I/Unintentional debt occurs when making an impulse or other poorly thought out decision without realizing you are incurring debt. It is similar to spending money from a checking account that isn't there, so perhaps a reasonable financial analogy for non-technical people would be "overdrafts".

Most of us have some overdraft protection on checking accounts to protect us against our own laziness or errors, but the tolerance for it is pretty low. Similarly, most organizations can tolerate a little of this type without too much pain, but the curve rises steeply at a fairly early point with this type debt.

re: Technical Debt

bk — Fri, 02 Nov 2007 16:33:26 GMT

For me, the most valuable take-aways here are the "How do You Make an Organization's Debt Load More Visible?" and "Retiring Debt" sections.

If you actually put pen to paper and record the compromises you are making during your software development cycles, and then you get to the point where you see this list and say "ok, our list is big enough, time to work on making it smaller," then that kind of hard-accountability can have a real positive effect. The challenge, as I see it, is figuring out what actually needs to go on that piece of paper to make it a valuable tool.

re: Technical Debt

Steve McConnell — Fri, 02 Nov 2007 15:27:30 GMT

Richard, I look at it about the same Big Maybe described. You're acquiring an asset (the released software) for less than it's full price. Thus you're financing the gap between what you paid (actual dev cost) vs. the full price (what the dev cost would have been without the shortcuts). That gap is the technical debt.

My experience is that business people get the concept right away. I haven't found that they asked questions like, "Who authorized the loan?" On the contrary--it puts a highly technical topic into a context that they understand, which makes them feel like they a better understanding of the issues. Of course they will ask lots of questions, but that's to be expected in any dialog about technical/business tradeoffs.

re: Technical Debt

Big Maybe — Fri, 02 Nov 2007 14:50:58 GMT

Richard,

You borrow Quality, in order to buy Time. Assuming 100% quality is the ideal, and requires no interest payments (in the form of future maintenance work that otherwise wouldn't have materialized), then every bit of quality sacrificed now is "borrowed" in the sense that you will eventually pay it back, with interest, using time that you assume you will have in the future.

re: Technical Debt

Richard Quinn — Fri, 02 Nov 2007 06:45:05 GMT

Thanks for the article Steve, it really does help conceptualize a problem.

However, I think the concept of debt in this setting is misleading. Usually, debt is incurred because you have borrowed to spend now, which must be paid off tomorrow. In a software engineering setting what has been borrowed? What has been spent? The only concept which is clearly explainable is the idea of having to pay back over the coming weeks or months.

I predict, that if I talk to business people about technical debt the first thing they will ask me is: "Who authorized the loan? What was the extra money (time/staff/feature) spent on?". There is - I think - a clear gap here between the kind of debt software engineers must service and the kind which finance people must service, this gap may well lead to more, not less, understanding of the issue amongst non-technical business deciders.

Perhaps I missed something, I'm looking forward to your reply.

re: Technical Debt

Alejandro Garcia — Fri, 02 Nov 2007 00:42:24 GMT

Type I is the: "health bills debt."

In some cases you could probably make something to prevent it (like diet and exercise or training and better selection), but also sometimes it is completely unexpected (like when you break your leg, or your server melt)

Type II.B

is the 'mortgage' debt. You do it voluntary, you know is big and it will take you a while to pay but, in the future you will have a big asset.