The Coding Wasteland

Lately it has become popular to justify any messy code written in haste as intentional technical debt: “we chose to sacrifice quality for speed.” I’m not sure this is a constructive trend.

In my own work I’ve begun making a distinction between technical debt and technical waste. Here’s how I define the two:

Technical Debt consists of liabilities incurred from decisions based on insufficient knowledge.

Technical Waste consists of liabilities incurred through negligence or lack of discipline.

Technical debt is often unavoidable. You couldn’t be expected to realize that your micro-CMS system really should have been architected as a messaging system. Sometimes it’s avoidable given sufficient time for research, but you make the business decision that you can’t afford to take that time. For instance,  you’re pretty sure  MongoDB is a better fit for your domain. But you opt to use a traditional SQL RDBMS anyway because your team already knows SQL really well. You’re concerned with how long it will take them to become conversant in NoSQL patterns and strategies.

That’s Technical Debt: decisions to defer “known unknowns” until later in the development process. Technical Waste is the stuff you do when you know better. For instance, you’ve chosen to go with a better-understood data storage technology instead of the latest and greatest NoSQL solution. You know that the best way to give yourself options later in the process is to build some kind of clear dividing line in your codebase between a small set of database-specific objects and a larger set of storage-agnostic objects. You know that if you establish the line early enough, and establish clear guidelines and examples for how to maintain the division, it won’t significantly impact the project schedule. But through laziness, or distraction, or perhaps because you don’t want to hurt the feelings of team members who are producing working-but-highly-database-coupled-code, you allow the line to be blurred.

Technical Waste is often found in the nitty-gritty of code construction. It’s the test you blow off writing. It’s the 21st line of code you add to a method instead of refactoring it into smaller methods first. Technical Waste typically seems harmless in isolation; but multiply it by thousands of lines of code, and one day you realize you have a serious problem on your hands. Technical Waste is death by a thousand slips.

UPDATE: It’s important to note that Technical Waste isn’t just about the things you fail to do. It’s just as much about doing too much. Early in my career, like many young programmers I created a lot of technical debt by over-designing and over-coding systems that didn’t really warrant it. I call this Debt, because at the time I didn’t know any better; my head was full of UML and Patterns and not much experience. I haven’t completely kicked the habit of overthinking things; but I know that it’s waste when I do it, and I make an effort to keep the impulse in check.

Waste is often easy to justify in terms of a short-term time savings. But it consists of shirking practices which, when practiced diligently and with judicious use of tools, can have a net-zero or even net-positive impact on project velocity. Conversely, they are also practices which become more arduous and time-consuming the longer they are left undone: for instance, when you fail to lay the groundwork for a new method with a test first, the next, more complex test is that much harder to write from scratch.

From Debt to Waste

There isn’t a fixed line dividing technical debt and technical waste. The longer you practice your craft, the more you make good decisions an intuitive part of your process; the more you become both fast and good. Ideally, you should always be pushing the line back so that more and more decisions fall into the the realm of “waste” instead of “debt”. That may sound odd at first; but what I mean is that the more you practice, the more you become able to feel when you are lapsing into bad habits and creating waste. Remember, Technical Debt is about unknowns: questions you don’t have time to answer. Waste is about knowns: when you have a pretty good idea what the right choice is, but you rationalize a reason not to make it anyway.

To make that a little more concrete, using an example close to my heart: for years, I created Technical Debt in the form of programs that had no cohesive guiding principle to the way they handled failure. Every time I created a new exception class heirarchy, or decided to classify a particular case as “exception-worthy”, I felt like I was taking a shot in the dark.

Finally, I’d had enough of the nagging feeling that error/failure-handling was a “known unknown” in my design repertoire. So I submitted a talk proposal about handling failure in Ruby. When it was accepted, I was forced to make a concerted study of exceptions and failure handling.

Several talks and a book later, I feel like I finally have some clear philosophical guideposts when structuring a failure-handling strategy for an application or library.  In my own coding, I have pushed failure to adequately address questions about error handling from the realm of “debt” into the realm of “waste”. I have a discipline to follow now; if I fail to follow it that’s just plain laxity on my part, not “taking on debt”.

Pairing is Caring

You don’t have to write a book just to expand your understanding of what is wasteful. Ways to improve your instincts and discipline as a programmer are myriad, and could easily take up a whole other article, talk, or even book. If I had to pick just one, though, it would probably be pair programming.

Pairing helps in two ways: first, it exposes you to how other people make coding decisions. Second, it keeps you honest; it’s a lot harder to fall into bad habits and cut corners when you know someone else is concurrently reviewing your code.

Like I said, I could probably list a dozen more ways. But this article is long enough as it is. What about you? What are your favorite ways to improve your discipline and level-up your software design decisions?

 

15 comments

  1. Wow, did I ever need to read this right now.

    The project I’m on uses a lot of frameworks and technologies that are new to the team. This isn’t unusual for us, we’ve done plenty of work with beta and alpha tools before. But this project took a lot longer to ramp-up than normal, and I’ve been trying to figure out why.

    I think it’s technical waste. I love the idea of differentiating between the two. We’re used to technical debt; that’s something we deal with all the time. It’s an accepted risk that we’ve accounted for in our budget. We’re not used to technical waste, and we were generating a whole bunch of it early on in the project.

    This was a very helpful and timely read. Thank you!

  2. I tried writing about this earlier, referring to the “Quality/Speed barrier” — below the barrier, faster means worse and better means slower, but above the barrier, faster means better. I never quite explained it well. Your distinction between debt and waste — I’d say “investment” and “expense” — comes closer to saying what I wanted to say than I ever did. Thank you for that.

      1. I appreciate the support. And no, I don’t think we’ve met yet. I don’t know whether we will have that chance soon. I’ve only attended one Ruby-focused conference: WindyCity Rails in 2009.

  3. I know what you mean, but…!  I think the beauty of the term “technical debt” is that debt doesn’t go away.  It’s an albatross that weighs you down and follows you and haunts your dreams until you deal with it. (Or ignore it and wait 7-1/2 years for it to fall off your credit report. Which may not a bad idea for a recent college grad who won’t buy a house for 10 years.  Unless it’s student loans, and then you’re screwed.  But I digress.)

    “Waste”, to me, implies a one-time loss.  You go to a bar with your buddies, blow $150 on booze, wake up on the lawn in your underwear, find your empty wallet in the dog-house and think, “what a waste”.  But after the hangover is gone, it’s back to business as usual.  There’s no long-term effect except to your liver.  Things pick up right where they left off and you’re free as a bird to do better next time, unencumbered by your past mistakes.

    The other thing I like about “technical debt”, is that it makes sense to stakeholders.  They “get it” when you say, “it can be done faster but we’ll incur a lot of technical debt that will have to be taken care of later.”  I’m not so sure they’d be able to wrap their heads around Technical Waste.  “Where’s the waste? I get my features faster. Sounds like a good deal, to me!”

    I really like the idea of technical waste, but I’m not sure I’d draw the line at known vs unknown.  Perhaps at things you can fix vs things you can’t fix or that aren’t worth going back and fixing.  Using your definition of waste vs debt, though, I really like the part about pushing back the line from debt to waste.

    1. $150 at a bar is one thing, but I think most people also understand the idea of permanently squandered potential. Think of it this way: at the start of a project, the super-fast velocity you have is like the $500,000 a startup has in the bank right after its first VC round. Technical Waste is the equivalent of that startup blowing money on posh Silicon Valley parties, superbowl ads, and an espresso machine at every desk.

      And it doesn’t make anything faster. Which is my biggest point in making the distinction: we, as developers, need to stop making excuses. It’s one thing to take on debt in the form of deferring architectural decisions that MAY hurt us down the road. But I’ve seen too many projects where the code construction was just plain lazy and slipshod, in ways that never sped anything up, short-term OR long-term.  We need to stop rationalizing this stuff as “debt” and admit that sometimes, we just fall into bad habits. We need to stop telling ourselves that certain disciplines will always take longer, and instead take the time (after hours, if necessary) to polish our skill and our tools until those disciplines don’t take longer.

      1. OK, I’m with you on that.  I used to play a bit of poker and we called those “leaks” in our game.  Anything you do consistently that loses you more money in the long run than it makes is a leak.  Sometimes they’re things you know you shouldn’t do but occasionally do it anyway, because you’re bored or ticked off, or just “in the mood the gamble”.  Even if you win the hand, statistically, that play is a waste of money.  So I guess we’re mainly in agreement.  I’m just going to think of them as “technical leaks” in my development game that need to be plugged.  Which I think works out better in my mind because it makes it clear that it’s an ongoing waste.

          1. As another (mostly former) poker player, I also like this analogy. The really insidious thing about leaks is that they are really, really easy to rationalize to yourself in the moment. 

  4. I like the concepts but I’m not with you on the terminology 100%. Debt is usually something that you knowingly accept (eg. a mortgage) while also accepting risk – you simply don’t have sufficient knowledge to determine if your house will be worth more or less next year. The key here is that debt can be repaid. Waste implies something that is forever squandered and is not really accurate in describing deferred test cases. 

    In the end, I’d use the terms:

      – technical debt (knowingly accrued, usually with the intention of payback), 
      – technical risk (what you call technical debt)
      – technical leaks (what you call technical waste, exclusive of technical debt). 

    1. If a developer told you that he could speed up getting the feature out the door by naming all his variables x, y, and z; or that he could save some time by not indenting his code, what would you think? I’d think that he’s either very new to development (as in, started programming a couple weeks ago), or that he’s incompetent. This is something I keep trying to get across: we have got to stop drawing arbitrary lines about which practices are just “part of the work”, and which are “extra work” that we can put aside to save time. If it saves you time not to do it, you’re either very new at it, not doing it right, or not using the right tools. All of which are true of of all of us, at some point in our acquisition of skills. But we move past it, just as we learned to name variables well without even thinking about it, and how to use an editor which indents our code for us.
      How is a deferred test case something you can pay back? By being lazy you’ve slowed down development. Putting the test case in will almost certainly take longer down the road. Add that to the time lost to a) not having the test to guide you to the simplest possible design; or b) not having the test to catch a regression that later sneaks in, and you’ve slowed the project down. There’s no way to magically “make that time up” by putting the test in later. It’s gone.

      Note, I’m talking about mainline feature development here. Spikes and exploratory coding have different rules. If during regular feature development your tests are something that, by deferring, you can speed up
      development… then there’s something wrong with your testing process.

  5. So “technical debt” really emerges over the lifecycle of a project while “technical waste” happens up front when you (often knowingly) make a bad decision? This is a great distinction, and one that I think makes a lot of sense. 

    It is a great tool for explaining to a project manager how code that we swore was great two years ago now needs to be rewritten: our goals have changed, our features have changed, and many of the connecting bits of code have changed. The assumptions we made two years ago do not hold anymore, and its time to go back through the codebase and correct those assumptions.

    And with the distinction between debt and waste, it’s easier to then also explain that the decision we made back then were not inherently bad. It’s simply that those decisions have not weathered the test of time.

    One final question I’d like to ask: The architect of a project is responsible (largely) for enforcing a policy of “no technical waste” — that much seems clear. But how is the architect to best address questions of technical debt?

  6. Avdi, say you are in a situation of vast amounts of technical waste (and debt).  Does your engineering team need a commitment from biz leaders that we will spend the time to reduce this debt (thru more training, taking more time to build features in a quality way, etc)?  Or if you have biz leaders that dont understand this tradeoff, do you do it anyway, at the risk of confrontation/disappointment w/ the speed of engineering?  

  7. Avdi, I really enjoyed the post – I find the distinction a worthwhile one.  Technical Debt accrues slowly almost unseen.  Technical waste happens as a result of lack of discipline.  When reviewing code, asking whether it arrived at this point through a process of technical waste or technical debt is a good question.  If the honest answer comes back that it was technical waste then you can have a constructive conversation as why the waste occurred and what could have been done to prevent it.

    An off topic question, but do you think your exceptional ruby book would be valuable to read for a non-rubyist ?

    @aad6d5f4b334047a34aa0d139b7829f6:disqus I am slowly coming to the conclusion that Business Stakeholders should not have much in the way of explicit input into addressing technical debtwaste.  There is not really a tradeoff off to be made when  developing software you either go Fast and Clean or Messy and Slow .  Fast and clean should solely be a developer concern, ie we should always be focusing on fast and clean – you may ask for resources that allow you to go fast and clean ie training etc and would need to justify what resources are for.   I understand that this view is a little hardline, but most Business Stakeholders just want fast, it is up to us to be good enough to give them that… and be prepared to live with constant disappointment.

Leave a Reply to Anonymous Cancel reply

Your email address will not be published. Required fields are marked *