The hacker sits down. It's been a long time since he worked on this project, so he figures he should probably make sure the tests are passing first.

Selection_141

Whoops. Right.

Selection_143

Darn it.

Selection_144

Crap.

The hacker edits the project's Gemfile to specify the exact patch version of Ruby 2.1 currently installed.

Selection_145

Hell.

Selection_157

Righto, now…

Selection_146

Damn it.

The hacker kicks off the next command and steps away for some coffee and perspective.

Selection_147

TL;DR.

Selection_148

Shit.

The hacker decides to run a spec file directly.

Selection_151

What the shit? I know I have Postgres on this machine.

The hacker realizes that the port number shown in the error message is wrong, and edits the project's .env file.

Selection_152

Goddammit.

The hacker realizes that the tests need to be run in the “test” environment.

Selection_153

Son of a…

The hacker edits the .env.test file to also have the correct port number.

Selection_149

Seriously?

Selection_150

And now…

Selection_154

Motherfucker.

Selection_155

Fuckity fuckity shit fuck goddammit.

Selection_156

I don't remember what I was here to fix anymore.


 

NOTE: Any comments nitpicking particular steps in this timeline will be summarily removed. The details will vary from project to project, but anyone who has returned to a project that has lain fallow for six months or more knows exactly what I'm talking about.

Published by Avdi Grimm

87 Comments

  1. … and now is the right time to write a setup script to automate all this process for the next time the hacker or someone else needs to work in the project …

    🙂

    Reply
    • Been there, done that, end up spending the same amount of time fixing bit rot in the setup script.

      Reply
      • Bit rot only happens when setup is imperative/non-deterministic rather than declarative/deterministic :).

        I highly recommend checking out the Nix package manager (which runs on Mac and Linux) and the NixOS Linux distro.

        (Full disclosure: I’m a committer on the Nix project.)

        Reply
        • Which is great until all the frozen dependencies become a swiss-cheese of discovered vulnerabilities and you get a cascading chain of upgrade hell (admittedly this happens even more in ruby than other software ecosystems due the culture of fast deprecation).

          Reply
          • I believe you’re mistaken.

            Packages are referred to by name, and their definitions are given by the current checkout of the Nixpkgs repository. Keeping up to date is trivial.

            The development workflow for all of my projects works this way. I specify the dependencies I have, and a particular checkout of Nixpkgs. I frequently bump the checkout, adjust my dependencies list (if necessary), and then check that in.

            I’m not sure what you could mean by “you get a cascading chain of upgrade hell” or “all the frozen dependencies become a swiss-cheese of discovered vulnerabilities”, as I have never experienced any of those things with Nix (I’ve used Nix in production for the past 1.5 years). The design of Nix lends itself to making upgrades easy, and making it trivial to apply security patches.

      • I stopped specifying Ruby versions in my Gemfile for this reason. I also stopped using .env files as I found them too troublesome to manage the true 12-factor way. I moved towards always keeping my database on the same machine as the project code, and using Postgres’ peer auth. Hard to see this as less secure than using a .env file. If it scales to the point where that won’t cut it anymore, then it’ll be worth setting up Chef / Ansible / Puppet and that’s a good problem to have.

        I thought once about a script that would pull all repos in a source directory, and then run a series of configurable maintenance commands on each one. Say, run bundle update, then run the tests. I could run the tool once a week as a easy way to keep everything updated. Something like CI for code maintenance.

        Reply
        • The problem with not specifying your ruby versions is that gems are often dependent on certain versions, and you will find yourself having people / or yourself submitting bugs that are related to what ruby version each person is using (Note some gems may have dependency warnings some will not). Also in terms of the .env file 12-factor app way check out this gem -> https://github.com/bkeepers/dotenv.

          Reply
    • Now you have n+1 problems.

      Reply
  2. Hire the person who can predict future stack changes and implement them in a bin/setup 😛

    Reply
  3. The only thing you’re missing, Avdi, is the step at the end where you realize you have not git pulled yet, and it overwrites all your changes and breaks everything again.

    Otherwise, dead to rights.

    Reply
  4. Python does this to me as well. Also csharp has had a habit of DLL upgrades running amok (and proprietary frameworks are ALWAYS happy to break your old app ). I think the language that gives me the least trouble long-term is go… but then that’s because I spend a whole day just getting it to build the first time. So… you pay up front with some compiled languages. Java. I. Don’t touch java. But I’ve heard it can be sane, assuming you don’t mind half the libs coming over http instead of https during compile time.

    I think this is a symptom of just how big software has become. And the developer’s workspace is… just hard to tame.

    Presently, I’ve used containers to control issues where ‘I ran this just fine a few months ago’. So far they have never failed, but there is a learning curve. Using something like docker, you can make a ruby environment that has all the right variables and ruby version. It used to be hard to do it, but now there are commands that let you waltz right in to the container and run commands, or run the container with new source code mapped to a particular directory.

    Not saying that solves everything (yet another tool to know about, amirite?). Just saying that if the filesystem doesn’t change in 6 months, there’s something wrong with turing machines if that doesn’t behave the same when you run it.

    Reply
  5. I’ve been able to avoid this lately, by creating Docker images for all of my projects.

    That way, the platform and environment in which the code runs remains static and consistent.

    I do development of all kinds, Ruby and Javascript included, as part of consulting practice. My approach helps me avoid bitrot on platforms of all stripes.

    Anyone interested in learning more about using Docker to manage your (development) workflow at your organization should feel free to reach out to me.

    Good luck, and happy coding!

    Reply
    • Yup! Then you don’t have the equivalent of old bit-rotted code sitting around!

      Instead, you have the equivalent of an old bit-rotted VM full of packages you can no longer find and install 🙂

      Reply
    • Yup – I was gonna say
      Vagrant

      Which is basically the same thing. Easier to manage, maybe?

      Reply
      • Nope, nope, nopity nope. I had a talk built around a vagrant image (the talk was essentially the amazing things vagrant makes possible) and when I went to install vagrant and plugins 7 months later(to re-use the talk) it was hours of stress I had assumed I wouldn’t have to deal with. Oh the vagrant version is different and now plugins are different and this plugin is not compatible with this one anymore and this thing has changed names, and….

        There are no silver bullets.

        Reply
        • Sure there are. Keep your head still for a few microseconds and they’ll hit you right between the eyes.

          Your application is an ever-growing collection of independently-moving (and therefore -changing and -breaking-until-proven-otherwise) parts. I look at any apparently successful “sit-down-and-it-Just-Works-again” exercise spanning more than a couple of weeks as highly suspicious. Or…I can probably use my old Gemfile.lock, but as soon as I rebundle, hilarity ensues.

          Reply
          • Which is actually one thing that makes bundler and Gemfile.lock so damn amazing — that, at least, at least, you CAN quite often use your old (checked into your repo) Gemfile.lock, and have things often work. At least! And then go from there, if needed, updating the dependencies a slice at a time every dependency change trackable in git.

            It’s still a pain. But was SO much worse before bundler — and we didn’t even have as complicated dependency trees as we often have now (perhaps we never could have without bundler… for better or worse).

      • Do you mean Vagrantfiles, Cheffiles, sharing ports, and wondering why the request from the host machine to that port is just simply not responding?

        :: Googles “why is vagrant running so slowly on my machine”… reads 5 posts, concludes because Vagrant, and my computer, hate me. ::

        Reply
    • Only now, you have static dependencies without security patches and critical bug fixes.

      Reply
  6. If you think that is fun, wait to see all those Ruby people who are mixing Ruby with the Node ecosystem (npm), and maybe sprinkle some Golang workers…

    One way I deal with that problem is having to script two extra machines: the production server with devops tools and a local machine running Jenkins with an independent user.

    Trying to recover a project you only prepared in your machine is helll. Atomating and fixing problems with other machines helps a lot.

    Reply
  7. Sorry to borrow a terrible blight from social media, but, in this specific case, I wish I could go through and hit the “Like” button on all of these thoughts.

    Reply
  8. This is awesome! Couldn’t stop laughing, been there!

    Reply
  9. When you specify “Ruby” project, are you implying the situation is better in some other programming language? Please reveal it!

    Reply
    • Check /r/webdev sometime next month. Then check /r/thenewhotness two years later to discover the newer hotness that’s come along to ease the most egregious pain points in the old new hotness. And then, some time afterwards…

      Whoever said “all this has happened before, and all this will happen again” was an experienced software developer. (J M Barrie of course lived half a century before the industry even started.)

      Reply
  10. If you just insert “…and then spend way too much time figuring out WTF is actually supposed to happen here and how to make it so.. ,” this is all too often what goes on when I want to contribute to an open source project. I have to have the bandwidth to first wade through and set up all of the configuration details before I even begin to get to the actual code. As much as I do love solving problems and figuring out puzzles, jumping through twisty, little dark unnecessary hoops is not on my list of preferred volunteer activities. (DOCUMENTATION, anyone?)

    Reply
  11. priceless — if you’ve never flipped off your laptop, you’re not programming, you’re just playing…

    Reply
  12. doing this just today, but in Java, with many tools that should help, but all need constant care and feeding. Using Jenkins is a Bear, as it involves lots of “magic” for GUI testing.

    Reply
  13. Don’t use system rubies, don’t even install them if you can avoid it. Take them out of your path if you can’t. That way lies madness.

    Use a virtualized ruby environment(rvm/rbenv/etc.) and you come back and everything just works. I go many months without touching projects and nothing is broken like this. Such is the life of the solo web dev.

    Reply
    • If you’d been following along, you would have noticed I use chruby 🙂

      Reply
      • The point was you had to use something that you had to remember to do after you ran into friction. I change to a project’s directory and it’s already got the right ruby selected with the right set of gems. You want to minimize that friction as much as possible.

        Reply
  14. Sometime I feel like a plain-old CodeIgniter can get the job done. It’s simplicity cannot be beaten. I only wish there’s a great community like RoR around that stack to spin up all those gems (matchsticks!).

    On a serious note, do you think CodeIgniter 3.0 is still a good option for a moderate, multi-tenant CMS? I’ve done a project with CI 2.0+ and felt very comfortable switching between my Mac and Windows machines and continue the development.

    With Rails getting things setup itself is quite hard and I can’t even fathom switching between a Mac and Windows machines during active development.

    Reply
    • I would say, try Laravel, its not as easy to setup as CI, but it shouldn’t take more than five minutes to have everything resetup as long as you have composer around.

      Reply
  15. I think that ‘hacker’ has a really bad setup.
    Why didn’t ‘the hacker’ lock your ruby dependencies for the project.

    Reply
  16. Ouch. That was basically my Friday morning.
    Only replace Postgres with an outdated Chef driver.

    Reply
  17. My experience every time I try to setup a python or ruby project for the first time. I would say that it must be the plague of dynamic scripting environments but php stuff usually just works. A good example is the WordPress 5 minute install.

    Reply
  18. And this is where working in a stodgy old enterprise company pays dividends: We’re not allowed to update anything without filling out a ridiculous number of forms — including legal — so nothing gets upgraded. Everyone is locked into the same pre-built set-up the IT department will give us. Nothing’s changed in 10 years. Our dev boxes are firewalled from the internet, so we couldn’t upgrade packages if we wanted to. (Well, we could copy the ZIP file over and install it ourselves, but we don’t have root access and couldn’t install parts of it, and then there’s dependency hell, so we’d have to repeat the same process a dozen times.)

    If we want a modern feature, we code it ourselves or live without it.

    As a bonus, it means I’ll never leave the company because my skills are with an unmarketable version of the language.

    Fuckity fuck.

    Reply
    • Pssh. Where I used to work, if they wanted something to always work they’d archive the physical computer system. Along with a supply of spare hardware in case they stopped being manufactured 🙂

      Reply
      • That’s awesome. I’m surprised nobody by me has thought of that

        No, wait, I’m sure they have, but they love paying huge support contracts. And those companies won’t support servers after ten years. (Really, we just ditched the 2003 servers that ran our SQL Server 2005 database. 😉

        Reply
  19. at least you haven’t found a yanked gem this time 😛

    Reply
  20. This is a great illustration why nobody hits the ground running on day 1. Almost every non-trivial project I’ve ever been on is super painful to get going . The most painful setup I can remember was a android project. One entire full day to set up all dependencies and that’s with the project lead who knew exactly how everything was supposed to be set up. Things can get much worse — but things could be sooo much better. Maybe Docker is the answer but I’m guessing there’s some bit rot waiting for you there too.

    Reply
  21. Don’t forget the tests that absolutely were passing 6 months ago but now fail, possibly due to different time/date conditions.

    Reply
  22. Ohhh, I loathe the “Your Ruby version is x.x.x, but your Gemfile specified y.y.y” error. I probably have the Ruby version specified in the Gemfile installed, but chruby, etc. don’t look at the Gemfile when auto-switching, so now my lazy self has to switch to it manually. Plus, if I do have a .ruby-version file, now I’m specifying my ruby version in two places. Ugh.

    You still have to fight all the other problems, but I’ve significantly reduced that pain by using ruby File.read(File.join(File.dirname(__FILE__), '.ruby-version')).strip in my Gemfile. It’s ugly, but it helps.

    Reply
  23. This sort of thing is a major blocker to open source contributions. There is a python project I would like to submit a pull request for but tests won’t run. I have not had the time to go through error by error and also follow the lengthy upgrade log.

    Reply
  24. Hm, let’s just spin this Ruby 1.9.3, Rails 2.0, Oracle site back up and — fuck, fuck, fuckity fuck fuck!

    Reply
  25. It is all stored in the image. Oh, you’re using ruby. nm.

    Reply
    • Oh, smalltalk! That beautiful but unappreciated thing. 😀

      Reply
    • Except for what isn’t… And let’s not pretend that the problems don’t exist INSIDE your image. (Professional Smalltalk developer with years of experience on commercial projects.)

      Reply
  26. Best. Thing. Ever.

    Reply
  27. Add a couple of outdated js frameworks to the pictures and you might as well copy/paste some of the ruby files into a new project and restart from scratch.

    Maybe what we need is good generators instead of tests and versioning systems, we’d go fully meta and never write a line of code directly but tweak the generators to do it instead…

    Reply
  28. Common Lisp…. It’s a language with a standard. I can run my code from 1985 unchanged. Use Make as your configuration control. Use Latex for documentation.

    I have tried to work in a dozen spiffy-new languages like Python only to find out that my Python 2.7 code won’t work with the Python 3.3 “upgraded” libraries. Java code that requires Java 2 and won’t run in my current Java install… such nonsense. It is a complete waste of time. The goal is to get the computer to do something, not to train me to chase rainbows. If my prior working Ruby code won’t run then the failure is with Ruby.

    If you’re doing real work for real customers who will call you up in 3 years to do some maintenance then you need a real language with a real, international standard. Oh, and it would be useful to you if you thought to write down “WHY” you wrote the code. There is rarely an issue with what the code does but you’ll have a hard time remembering that you needed that special hack so it worked on the Palm Pilot. I’d suggest you look at Literate Programming. See “Physically Based Rendering” by Pharr and Humphreys for the gold standard of quality code.

    Reply
  29. It is hard to do anything but laugh when reading this, because all I can think is “Thank God it’s him & not me!” I feel like this has happened to me 100X..

    Reply
  30. Uh huh, yeah, I like it.
    I always feel you grinning, even when you’re being serious.
    Nice post.

    Reply
  31. Painfully true.
    It’s double fun when some old project still uses therubyracer gem which I could never install from the first try.

    Reply
  32. I wonder if Jruby wouldn’t fix this situation. You can put the exact version of the jruby.jar in your directory and vendor all of your gems. A shell script to isolate your environment to the directory and you should have a standalone directory which is repeatable and transportable.

    Of course if you have your ENV all screwed up that’s all your fault and nobody can fix that except you.

    Reply
  33. Wait, your native extensions actually installed on the first attempt? No changes in versions for the compiler your using? GCC vs clang? New version adding more warnings and the gem trying to compile with -Wall and thus hitting new compiler “errors” on previously valid code? You didn’t even need to get imagemagick working? Must be bliss.

    Reply
  34. Wouldn’t running tests daily at least alert you earlier to this problem? And fix them as soon as you can. Because even without TDD, you will want to run tests on your code when you are done with the change. And your test environment is not working. So really you are just fixing your test environment before making your change. But you have to go through the whole thing regardless.

    At the very least keeping up with your test scripts will enable you to be faster in case you have an emergency.

    Reply
    • The tests may be running on the test server environment, but not on your local development… unless you are talking about running tests on all projects on your local machine… but then what about when you try to install a “new to you” project that your organization has been working on. This is all about getting your own environment to match the last known state used by the project.

      Reply
  35. All who relate to this post understand why engineering quality software is not cheap. It is a difficult, time-intensive process. Though we often find ways to save time here and there, programming is still difficult. We always find ways to move the ‘difficult’ from one place to the next in the process, but it (the ‘difficult’) always pops up somewhere else. It’s like whack-a-mole.

    Reply
  36. Yup, and then if anything depends on an older version of openssl… your in gem update hell. Or if the author has a bad gemspec and you have new bundler o.0

    But seriously, do remember with an old perl 5.8 application: Step One: Realize OSX will not run your application, and you can’t compile tinytds with anything newer then centos 5… but you can’t seem to find the documentation written ten years ago of the specific apache version custom compiled to run it…

    So appreciate that you just have to remember a bunch of rake tasks.

    Reply
  37. It’s nice to know that I’m not the only one who experiences this! It can sometimes get even worse when you are able to run only to get subtle runtime errors due to this issue.

    Reply
  38. I have a bunch of old Rails projects, dating back to 2008, that I try to spin back up every so often to update and I run into this. E.g. NewRelic or Heroku send me a nice-gram that I’m running an out-of-date version that needs security fixes, and then…yeah, fuckity fuck fuck for an hour or so!

    For what it’s worth, it happens with almost all projects, in all languages. Unless a project is VERY popular (for open source projects) or has someone dedicated to making setup easy (corporate projects) it’s always a pain in the rear to get everything up and running. Doubly so if the project hasn’t been touched by anyone in 6+ months.

    Very nicely captured the feeling and process, though, yeah… 🙂

    Reply
  39. I liked the note at the end of the post. I’ve been there, people coming and commenting shit out of context!

    Reply
  40. This. So true. This also happens when switching from work projects to side projects. The struggle is real. 🙂

    Reply
  41. […] seem to have lost their taste for stable software. After all, you’re not really living unless your dependencies are constantly shifting under you in backwards incompatible ways, […]

    Reply
  42. Yep recognisable. Imagine returning to a rails 2.3 project which did not yet use bundler. Aaaarrgghhh 🙂 🙂 By the way: I always use rake db:test:prepare to setup my test database.

    Reply
  43. Very true. To avoid those I tend to do the following:
    1) Always specify a clean database creation and migration from 0 as a pre-requisite to the test task. This way if anything if fishy with the DB setup, I will know quickly.
    2) I use strenv so that anything I try to take from ENV that is not there will fail loud at the start (ENV.fetch bids the same but I find typing STRICT_ENV[‘KEY’]) neater.
    3) Bundler should be setup to build Nokogiri against system, because you are not getting that time of your life back, ever.
    4) I try not to specify the Ruby version in the Gemfile because it turned out to be too problematic in many cases. But I do usually add .ruby-version if I can help it.
    5) Pessimize all dependencies at least to the nearest major (or keep a lockfile, because
    6) If you use your own libraries, have them on Travis or other CI against multiple Ruby versions so that any dependency rot can be excluded at least for the dependencies you wrote yourself.
    7) For the “top level” application – an application that you ship onto the servers, that is – commit the lockfile. Always.
    Dependency hell in one form of another exists everywhere. For me starting a Rails application I wrote in 2006 is fundamentally easier than compiling a .so that has to be loaded into a third-party proprietary application – there everything is one file at the end. But to make that file you have to literally move a mountan (after having dug a tunnel underneath it with your bare hands).

    But there are steps and best practices which you start adopting having encountered this pain a few times and that help it to be avoided.

    Also: I’ve recently spotted a pattern where gem authors would cause a Nokogiri dependency by proxy. Case in point: I am using Jeweler, and it uses a third-party Github API gem, and that gem depends on Nokogiri because it wants to use it to generate XML serialized responses – and that feature in that gem is not even tested, so I doubt anyone uses it in situ. It did however cause a snowball effect that all of my gems that used Jeweler had to be locked to an earlier version because I want to keep 1.8.7 compat whereas Nokogiri won’t build on it anymore.

    To avoid that, I try to watch my Gemfile closely, and when Nokogiri appears in the list of installed gems whereas it was not there previously I usually try to trace which gem pulled it in without my consent. Also, gem authors should be alerted that depending on Nokogori is not that necessary for many cases and introduces potential issues or slowdowns for any gem that depends on you.

    Reply
  44. You had me at the excellent pictorial f-bomb image, then just got me into “yup”, “mmmhm”, “yupity yup”, etc. as I scrolled along.

    I truly appreciate that you wrote this. When I read your books or watch your talks I am humbled into thinking you and your ilk are somehow immune, kind of super-hackers. It is greatly reassuring to us mere mortals for you to reveal your humanity. Cuz, misery loves company.

    Anyone smug enough to suggest that this bullshitery is avoidable in some current incarnation of the world is either a) high, or b) has not lived this pattern 20 or 30 times yet. I have tried every technique I can think of to head off these things. So far the most effective has been the assiduous maintenance of a file called README in the project root documenting each hoop through which I am aware of jumping at the time of the project. I may consider changing it to README.fuckityfuk.

    This README can often save me 12 to 14 minutes of the several joyful hours described here, in the 5% of cases in which dormant projects rise from the ashes, at the trivial cost of several minutes per day over the course of the several months I am working on the project.

    So, if 2 minutes per day over 40 days of a two month project saves me 14 minutes of cursing, that’s the hacker version of “a stitch in time saves 9”. Generalized, that’s “#{input_effort_time * effort_count} stitches in time saves #{output_effort_savings}”. For me, this produces “80 stitches in time saves 14”. Oh, a bug — savings needs to be multiplied by chance_of_phoenix_rising, which I asserted was 0.05. So, “80 stitches in time saves 0.7”. A bargain!

    A developer on a team I worked in spent perhaps 20% of his time writing Chef scripts whose goal was to lock down the environment and prevent this kind of issue. He wasn’t big on the whole documentation thing but assured us, before leaving for greener pastures, that we “just” needed to run the helpful script he had written and everything would “just” work. We attempted to resolve a problem in the config, written by the glorious chef script several months later.

    Not so much. The developer had forgotten that he was the only person who had the ssh key to the chef server. So another month passed, at which point I recalled that this developer might have used that old single shared key that we were never ever to use under any circumstances. What luck! So now we’re in, and ran the script. Did it work? (Rhetorical question).

    I have concluded from this post that the single most important hacking skill is the ability to recognize and remember what each error message actually means you need to do. Or, failing that to diligently up-vote and star the StackOverflow question and answer that I used the first time I enjoyed the problem, and when particularly gnarly, to edit or extend the SO question or answer if needed.

    Thanks for making us all feel better, Advi. You are a hero.

    Reply
  45. You didn’t have to createuser -s perkolator?

    Love how this applies to pretty much all programming environments, just with different error messages. (Well, mainstream environments anyway. Dead ones have different issues)

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *