Once upon a time, I owned a beautiful big house in the mountains of Eastern Tennessee. I don’t anymore. Now I rent a modest little rancher in a suburb of St. Louis. But every month, I still send out a payment on a home-repair loan. Because in order to sell that house in Tennessee, I had to spend $22k to replace an aging HVAC system.

So much for homeownership. Let’s talk about owning software.

Services require support

Around 25 years ago, I became the local maintainer for my team’s CVS repositories. This was our way of breaking free from Microsoft SourceSafe. Ever since then I’ve been involved in decisions about, and support for, software services that support team work. Services for teams have evolved a lot since then, as have the business models around them. One thing that has not changed, however, is that all services require maintenance and support. What do we mean by support? Well, a few examples might be:

• Team members may need to be taught how to use the service well.
• The server OS has to be upgraded to the current LTS, which updates libraries, or changes the daemon management subsystem, and this has implications for the service.
• Or, the underlying database is upgraded for similar reasons.
• Links to another service, like Github or Front or Trello etc. stop working due to updates in those services, someone is responsible for dealing with it.
• The server runs out of space, and someone needs to manage the smooth migration to a larger server or storage volume.
• A CVE affecting the stack the service relies on. Someone must be responsible for:
  1. Noticing;
  2. Auditing the service for being affected; and
  3. Addressing the issue – by changing the code if necessary.
• Someone must maintain backups. And ideally, periodically test them.
• We need custom integrations with our other chosen services. We either need someone else to write them or we need examples and advice for best-practices when writing such extensions.
• It may be expected that we customize the software for our own workflows. Again, this can mean anything from the vendor writing the customizations to our spec, to us writing them with help and advice from them and their documentation.
• The service unexpectedly goes down (all software goes down at some point, even if it isn’t a “bug” per se), someone needs to stop any other ops or dev work and figure out what’s up.
• When certain operations stop being snappy five years later because no one had ever tested it with that volume of historical data, that has to be somebody‘s problem. Someone must research between archiving old data vs. fixing the queries, and then implement the solution.
• A mandate comes down that all software must be audited. For new regulatory reasons, for IPO reasons, etc. Someone has to be responsible for doing the audit on the service.
• We suddenly realize we needed to clean out customer PII pronto. Someone must audit and perform the removal, something the software may not have supports for out of box.
• We discover parts of the software weren’t ADA compliant, making it impossible to use for a disabled team member. This has to be somebody’s problem.
• We’ve been bought and now the service must either log in using the corporate SSO, or be dropped, no exceptions.
• The world changes, and most teams are moving from using Fleeble to SnipSnarb for their CI service. We need the service to interface with SnipSnarb now.
• The UI feels increasingly dated and clunky by modern standards, and the team increasingly hates using it, especially the new hires. It needs a UI refresh.

These are just a few scattered examples of reasons services may need maintenance and support. No service will need all of them; all services will require at least some. I didn’t even mention the most common reason for needing support, which is the software has a bug that affects our team.

Ownership once entrapped many organizations

Services that support teams are living systems, and living systems always require ongoing maintenance and support. There is no exception to this rule. What has evolved and proliferated are the models of how teams pay for the support, and how that support is divided between internal staff, vendor staff, commodity hosting staff, and the wider community.

One upon a time, you might have purchased a license for some software (like SourceSafe) that you would then host on a computer in your team’s server room. This is what we’ve come to call “on-premises”, or “on-prem” software. Typically, on top of the license for the version of the software you bought, you’d also purchase an annual support license of some kind for your team. Depending on the level of the license, you might get priority phone support, you might get personnel physically visiting your site from time to time, and you might even get access to the software’s source code (protected by NDAs) so that you can better understand it, and better write local customizations or integrations with it.

“Ownership” of a service in this form tended to follow a fairly predictable arc. If it was a useful, popular service, the organization would gradually and organically customize it, extend it, build processes around it, and integrate with it over time. Gradually, more and more of the organization’s IT budget would be spent on supporting these customizations and integrations. After years of this, like barnacles on a ship’s hull, the ongoing support for the service would amount to significant drag on the org’s overall budget and development velocity; all for a service that wasn’t even part of the company’s actual product. In the worst cases, a whole department might evolve to service-the-service.

Meanwhile, as often as not the vendor would move on to deprecate the product, or even go out of business. Organizations would be left with “zombie” services that they had tightly bound themselves to over time. Not only would they still be spend person-hours on supporting their customizations; they’d also have to make increasing investments in supporting obsolete software stacks and even hardware to keep the old service running. Like the mainframe-based time/budget-tracking system at my first software gig, which had to be accessed via a VT100 emulator.

The Open-Source revolution happened

Over the first decade or so of my software career, a sea-change occurred. More and more of these services were replaced with Open-Source software. Often first as quiet acts of internal rebellion, but increasingly supported by corporate buy-in.

Sometimes the Open-Source services were simply better. Sometimes they were notably more barebones, but they offered such compelling advantages in terms of total cost of ownership that they were worth the tradeoff:

• Most importantly, seeing the code wasn’t a maybe-available license addon for more $$$. It was a given. This was a huge boon to extension, customization, and to answering our own questions about “how is it supposed to work”.
• The code was often higher-quality and easier to follow, because of the differing incentives of open-source development. Not always, but often.
• We benefitted from wide-open community shared knowledge forums. There was no vendor with a vested interest in being the only subject expert.
• We had options for help. We could try to fix problems ourselves internally, contract a project maintainer to help, or contract an independent specialist.
• Migration to and from Open-Source systems was usually easier.
• OSS projects might be abandoned, but never in a way where all we were left with was an opaque binary that only ran on a specific version of Solaris.
• And, of course: we got to benefit from the improvements made by every other shop with an investment in the service.

The worst case scenario of gradual internal customization-creep might still lead to having an internal fork of the project for which we were responsible. But at least, in that eventuality, we could fully own it. For better and for worse.

One thing that stayed the same, or even increased: the cost of support and maintenance in terms of internal effort.

Quantifying in-house support

Let’s take a minute to quantify that cost more concretely.

For an example we can start with a scenario loosely based on my old CVS repository admin duties. My role as the sole support/maintainer for that off-the-shelf, open-source service included stuff like:

• Initial installation and figuring out user authentication
• Importing older repos from other systems like SourceSafe, RCS, and PVCS, as needed.
• Helping developers get started and sorting out their authentication issues.
• Maintaining a few small scripts that captured our workflows.
• Occasionally untangling file permission wierdness.

The details will differ from service to service, but this is a fairly typical of a low-maintenance on-prem service that one developer or ops person might be responsible for.

Let’s be conservative and say this takes up an average of an hour a week of this developer’s time. Some weeks none, some weeks more, but an average of an hour.

Let’s see, according to Glassdoor, average total compensation (including benefits) for a senior software engineer in the US as of January 2024 is $194k. That works out to an hourly of $97. 97 times 50 weeks is $4,850/year.

So, a total annual support and maintenance cost of about $5k/year. And a concomitant loss to… whatever other projects that developer might otherwise have put their time towards.

With that number in hand, let’s proceed with our story…

Software as a service changed the math

Time went by, the world became more ubiquitously networked, and web browsers matured as thin-client frontends for hosted services. This produced the next major evolution in software for teams: Software-as-a-Service (SaaS).

Some SaaSes were open-source software hosted on other people’s machines, with hardware and software upgrades, backups, etc performed for you. Others were built on proprietary code.

Some promised only to take infrastructure—hardware, operating system upgrades, networked storage, backups—off your hands. Others included substantial support with the service itself, including onboarding, migration, answering new-user questions, troubleshooting, and addressing bugs.

What they shared in common was a pricing model that took a portion of what would otherwise be in-house support and maintenance, and make it into a fixed “rent”. One that might turn out to be substantially lower than in-house resources, due to the economies of scale and the fact that their people could be experts in that service and nothing else, whose expertise was shared across many hosted accounts. Moreover, the costs were predictable: for a given year and # of users, you knew exactly what you’d be spending for that level of support. No surprises. No unexpected weeks of having a developer doing nothing but troubleshooting a failed migration on a team-essential service.

Let’s make SaaS costs concrete. Earlier we said that a modern-day equivalent to the local version-control hosting hat I once wore, would cost around $5k a year in current developer compensation. By comparison, in 2024, for 12 developers on Github, you’d spend either $541/year (for the starter-level offering) or $2,880/year (for an enterprise-level hosting contract). With no matching loss in developer-hours to our mainline projects.

SaaS also has a mostly-salutary side-effect: unlike those barnacle-encrusted in-house services we talked about earlier, SaaS exerts pressure against the organizational urge to customize a service into a unique and special snowflake. Instead, SaaS tends to pull teams in the direction of modifying their usage of the product in a more conventional direction. Which leads to less need for specialized support.

We have a lot of options now

In the present-day, for any given type of service a team might want, we have many different options for how the ongoing support and maintenance labor is broken-down between the local team, a vendor, or the wider community:

• Artisanal Homegrown Service: We’re responsible for everything, but at least we know the codebase back-to-front.
• Proprietary on-prem software, hosted on hardware our ops people maintain, with a support contract from a vendor.
• Open-source code, locally hosted, entirely supported by our own people.
  • A degenerate case is a single developer who “owns” the service as their baby, possibly sinks a lot of “hidden” time into it, and when they leave we have a problem.
• Open-source code, locally hosted, but with a support contract from an org that maintains the code, or from a third-party expert consultant.
• Open-source software hosted by someone else that we pay annually. We might be on the hook for customizations and integrations, but we pay them for infrastructure management.
• Proprietary SaaS

Some concrete examples:

Chat/conferencing/chatops: A team might host their own Mattermost server and do all of their own administration and support. Or, they might host it on-prem but with a support plan from the Mattermost company. Or, they might opt to go with cloud-hosted servers. Or, they might pay Slack instead.

Version control hosting: a team might host their own Gitlab instance, or pay for Gitlab as a service, or run Github on-prem with support from Github, or pay for the Github cloud-hosted SaaS.

For any given service, a lot of factors might go into choosing which of these approaches make sense for the team. For instance, special privacy or regulatory concerns might mandate software be hosted on-site rather than in a public cloud. In every case, though, it’s not a choice of whether to pay for the ongoing costs of infrastructure, maintenance, support, and administration. It’s a choice of how to break those costs down between company personnel, vendor costs, and relying on open-source community interaction.

One-off licenses are a notable exception

One model you generally don’t see is software services for teams offering purchase of one-off permanent licenses. The reason for this is straightforward: it’s a model that doesn’t work for either vendors or customers.

On the customer side, we know that the real costs of any service are in the support, not the initial investment. And for any type of service we can think of, there’s already a well-established Open Source offering for which the “purchase” price is… free. Paying someone for even a “shared-source” license has no advantages over self-hosting Open Source software, and plenty of disadvantages in terms of lack of community support and ongoing viability.

From the vendor side, if they make their money from one-off licenses, the economics just don’t pan out over the long term. The incentives become for them to spend all their energy on endlessly expanding market share so they can sell to new users, and/or churning out paid upgrades and add-ons to sell to their existing buyers. There is no financial incentive to work on code quality, bug fixes, or other sustainability considerations.

Side note: while most of the software I use for running my own content business falls roughly into the categories of hosting I listed above, in the past five years I did allow myself to be convinced to purchase one one-off license for a software service. And it remains one of the worst experiences I’ve ever had with software:

• It’s simply a shoddy product all around, with lots of outages and known bugs.
• It has some of the worst support in the sector, often with weeks-long waits for responses.
• Folks on their semi-public users forum know that if they say anything critical, their posts are likely to be deleted by the vendor, because the vendor has a vested interest in giving off the appearance of happy customers in order to make new one-off sales.
• A lot of the vendor’s time and money goes into cultivating affiliate sales for the product, flooding the internet with kickback-funded third-party “reviews”.
• Promised features languish undelivered for years, as they work on new add-on products they can sell to customers for more money.

These are all predictable outcomes of a one-off sale model. The incentives in such a model are inexorable and unavoidable.

No reputable shop caring about long-term sustainability of their offering would pursue such a model. But if one were to choose to go down that road, they might market it by playing upon people’s nostalgia for shrink-wrap consumer software. They might emphasize how good it feels to “own” your software, while ignoring cost-of-ownership. They might rely on the inexperience of younger technical decision-makers, ones who have never had to confront the budget in real dollars that goes into long-term in-house support for services that aren’t the teams’ actual product. They might offer a “shared-source” style proprietary license, while being careful not to promise any updates or even bugfixes. They might leave the door wide open to abandon the whole initiative once they’ve made a quick buck. They might frame the whole shenanigan in terms of a brash and high-minded manifesto, to generate excitement and override the obvious objections.

Hypothetically speaking, of course.

Ownership is expensive

Here’s the thing about ownership: it has a few meanings. I “owned” that house in the mountains, and it’s still costing me money to this day.

You can kick around purely metaphysical notions of “ownership” to your heart’s content. In that sense, maybe the NFT bros had a point: the only thing you can truly “own” is something wholly intangible. Although, they still manage to lose their monkeys more than I lose my TV remote.

The only practical meaning of “ownership” when it comes to software services is: who’s on pager duty for it? Who has to drop what they are doing when it’s not working as-needed? Who manages upgrades? Who migrates data into it and out of it?

Once you’ve been around the block a few times and seen things from your VP Eng’s point of view, the one thing you’ll know for sure about ownership is this: When it comes to owning services for developer teams, ownership is expensive. It feels a lot like renting, only less predictable.