TL;DR: I’m throwing a pop-up workshop on Investigative Debugging. It’s limited to two evenings and 12 participants. You’ll learn hands-on stop hypothesizing about how code is working (or not-working), and instead leverage surveillance tools like strace, mitmproxy and docker diff to understand software behavior with hard data . Scroll down to the bottom of this page to sign up!
Debugging: Elementary, my dear Watson?
“Evidently,” said I, “[the] assistant counts for a good deal in this mystery.”
“Not him.”
“What then?”
“The knees of his trousers.”
“And what did you see?”
“What I expected to see.”
Sir Arthur Conan Doyle, The Red-Headed League
Here’s a confession: I’ve been a Sherlock Holmes fan since I was a little kid. I’ve probably read all the Holmes books three times or more.
Sherlock Holmes solved mysteries by thinking very, very hard about the facts at hand, forming a few possible theories that fit all of them, and then performing the minimum of tests or investigation to verify which of those models was right.
Holmes’ approach to sleuthing was very effective, for two reasons:
- He had an unparalleled analytical mind, along with encyclopedic knowledge of all topics related to criminal enterprise in London; and
- He was a fictional protagonist.
Unfortunately, when I’m debugging my code or trying to understand how a library works, I too often fall into the trap of thinking that I’m Sherlock Holmes. I imagine that I can lean back in my armchair, steeple my fingers, and think very hard about the problem until I come up with a hypothesis which I can then quickly test.
I’ve wasted quite a lot of time this way.
Real detectives gather data
Real-world criminal investigation doesn’t look anything like this. Outside the pages of Victorian fiction, police investigators collect mountains of raw data, and sift through it. They also make use of their special privileges as government agents to dig into criminal databases, subpoena private records, and tap phones.
We can debug programs this way too. Investigative debugging is the practice of using the full diagnostic arsenal at our disposal, along with our special privileges as machine operators, to understand code by surveilling it. Not through cleverness, but through sheer overwhelming empirical evidence. We get to the bottom of the behavior of a dodgy program or library by reading its tax returns, rifling through its mail, and tracing its phone calls.
What kinds of tools enable us to do this? I’m talking about:
- Containerization to isolate a program’s file dependencies and communications
- Language tracing libraries
strace,ltrace, anddtraceto track system and library calls- LD_PRELOAD to inject your own fake functions into programs
- Browser dev tools to peer into JavaScript HTTP calls
- mitmproxy and Postman to intercept and interpret HTTP traffic
- Wireshark to examine network traffic at a lower level
lsofto track file and port usagelddandstringsfor peering into executables
Over the years I’ve collected a toolbox full of these analytical instruments. And lately I’ve been making a concerted attempt to use them to collect hard data instead of spending time theorizing about what might be happening.
I’m far from an expert in any of these tools. But I feel like I’ve reached the point where I’m ready to share what I know in an experimental pop-up workshop format.
Investigative Debugging: The Workshop
The plan: a two-session pop-up workshop limited to 12 participants. I’ll run the workshop if I get at least 6 registrations by Saturday, May 16.
Day 1, Tuesday May 19th at 19:00CDT: We’ll start with a mystery, and we’ll use as many of the tools above as we have time for to investigate. This won’t be a comprehensive tour of any of these tools. Rather, I’ll be aiming to cover just enough to contextualize the tool. I want you to have used it once, and have a sense for when it’s indicated.
Day 2, Thursday May 21st at 19:00CDT: undefined! We’ll stay flexible, and dig into whatever questions arise from Day 1.
Cost: $100. This is to cover my time, as well as to make sure that everyone involved is serious about attending and being an active participant. This is a workshop for curious, engaged practitioners who are ready to help each other learn!
I’m gauging interest in the topic of investigative debugging. If fewer than 6 people sign up by Saturday May 16, I won’t hold the workshop and I’ll refund the charges.
Prerequisites: this workshop is intended for experienced software developers. If you’ve never heard of any of the tools I listed above, it might not be for you. If you’ve heard of a few of them and felt like you really oughtta learn to use them someday… this is your opportunity to get hands-on with them!
This will be a hands-on workshop. You’ll get the most out of it if you follow along with the examples. I’ll be sending out specific instructions for getting set up to follow along, but you can expect to need:
- Git
- Docker (on any host OS)
- VS Code plus Remote Extensions (I’ll be demonstrating inside of VS Code’s container remoting)
- Zoom
- Discord (for between-meeting discussions)
This is an experimental format. I don’t know yet if this is compelling subject matter, or if this is too much or two little time for it. If you’re looking for polished courseware, this is not the workshop for you! But if you want to help me test material and figure out whether working programmers can benefit from deeper analytical debugging chops, come join me! Let’s learn about investigative debugging together.