An Investigative Debugging Toolbox

Last week I ran an experimental workshop on Investigative Debugging. It was a lot of fun and the attendees seemed to get a lot out of it. Afterwards I sent out some extra reference material, including this Linux-oriented list of investigative debugging tools!

The focus here is on “surveillance tools” – utilities that give you observability into exactly what programs are really doing at runtime. It’s far from complete, but it represents a ton of research and I hope you find it helpful!

BCC toolsA collection of over 70 specialized tracing and performance profiling tools built on BPF.
bpftraceA Linux tool analogous to DTrace, built on BPF. bpftrace enables users to compose “questions” at the command-line, using a concise AWK-inspired syntax, for which they would once have had to use (or create) a dedicated tracing tool. It is often possible to recreate BCC tools using bpftrace commands. But when a dedicated BCC tool exists for a given query, using it may be easier than hand-rolling a bpftrace command.
DockerA technology for isolating Linux-based services into containers, each with their own isolated filesystem, process table, and virtual network. Containers are useful from an investigative debugging standpoint because they enable us to isolate and observe how a process interacts with its environment.
docker diffGet a comprehensive list of all files changed within a Docker container.
ftraceA low-level, “kind of janky” (Julia Evans’ words) interface to Linux kprobe and uprobe tracing.
HTTP_PROXY, http_proxy, HTTPS_PROXYThese environment variables are sometimes respected by programs as a way to configure a SOCKS proxy.
LD_PRELOADAn environment variable that can be used to force a program to link to arbitrary libraries before running. This can be used to substitute user-provided functions that override default behavior with e.g. tracing instrumentation.
lddList all the libraries that an ELF executable links to.
lsofList open filehandles, including network connections.
LTTngA linux tracing framework. I believe this is now one of the inputs to BPF?
ltraceTrace library calls.
mitmproxyAn interactive HTTP proxy server that can log and inspect requests and responses. Can be configured as either a SOCKS proxy or a transparent proxy.
nmList all the names (e.g. function names) in an ELF executable/library.
PostmanA workspace for experimenting with and documenting HTTP calls, which can also act as a mitmproxy-like request inspector for other processes.
readelfA general tool for inspecting and dumping information about ELF executables and libraries.
straceTrace system calls.
stringsDump all the static strings found in an ELF executable or library.
tplistA BCC tool for listing all USDT tracepoints available in a process.
uflowA BCC tool for tracing method calls and returns in high-level languages like Python, Ruby, and Java. May be installed somewhere weird and not in the path, like /usr/sbin/lib/uflow
WireSharkThe preeminent tool for tapping into and dissecting IP network traffic at the packet level.

Leave a Reply

Your email address will not be published. Required fields are marked *