Running a whole platform on a laptop from one binary

There was no way to run the whole platform on a laptop.

Each service had its own partial dev setup, one repo had a compose file that ran the infrastructure for itself and nothing else, another had a devcontainer that ran no services at all, so any work across services meant wiring the database, the queues, the cache, env files and hostnames by hand, every single time.

This is client work so I’m leaving the names and the languages out, but the shape is a backend service with its workers, an API, a smaller app, a site built once per brand and a handful of portals, with three brands all running at once from one database.

So I wanted a new developer to get the infrastructure, the backend and the API running with the following commands, with only Docker and git installed.

stack auth login   # once: GitHub, for updates and the private images
stack bootstrap    # clone the missing repos, generate secrets
stack up           # infra, then the platform
stack trust        # once: make browsers trust https://*.localhost
stack smoke        # check it works, down to an email arriving

I built something like this about two years ago, so this time I knew what I wanted, and the first working version was up the day after I opened the issue.

The HTTPS setup, the env problems and the API gateway each got a post of their own.

Why not just a compose file

Plain compose gets you most of the way, but it can’t do the following.

Each service has to run from an image, outside the stack from your checkout, or not at all, and you need to switch while everything else keeps running.
When a service moves out of the stack its hostname has to follow it to a port on the host, and show a page saying what to run when nothing listens there, without restarting the other containers.
We need to run some checks before compose does, the API refuses to boot unless its .env holds every key from .env.example, and its parser rejects some characters in values.

Override files and a Makefile could get you the first one, and none of the others.

When it’s worth it

If you have a couple of services in one repo you don’t need any of this, a compose file is fine, and a CLI on top is one more thing to look after.

It starts to pay for itself when the work crosses repos, when hostnames and cookies have to behave like the real thing, and when more than one person has to repeat the setup on different operating systems.

The cost isn’t small either, it’s close to 10,000 lines of Go, anything compose can do that the catalog, the YAML that describes each service, doesn’t model needs a new field before a service can use it, and a catalog change only reaches people with a new release.

Keep the output boring

The CLI doesn’t talk to Docker through an SDK, it renders plain compose projects, a Caddyfile and the env files into ~/.stack, and then it runs docker compose on them.

That was the best decision I made, because every action can be repeated by hand, and when something goes wrong you’re debugging a compose file, not my tool.

flowchart TD
  accTitle: How stack turns the catalog into running containers
  catalog["service.yaml and local.yaml, built into the binary"] --> stack["stack"]
  settings["your settings and service modes"] --> stack
  stack -->|"renders"| home["compose projects, Caddyfile and env files in ~/.stack"]
  home -->|"runs"| compose["docker compose"]

Let’s take a look at how a service is described, it’s two files, and that split is the part I’d copy.

projects/platform/api/service.yaml   how prod would see it: image, ports, health, env
projects/platform/api/local.yaml     what only a laptop needs: modes, hostname, mounts

A local-only field in service.yaml is an error that names the file, so laptop assumptions can’t end up in service.yaml.

The catalog is embedded in the binary, so people install one file for macOS, Linux or Windows, and running it doesn’t need Go or a clone of the repo.

You also want to see what a render is about to change, and I only added that later.

stack platform diff renders in memory and diffs against what’s in ~/.stack, and golden-file tests do the same job in review.

It should have been there from the start.

Taking one service out of the stack

If we need to work on the backend service, which is called auth here, we take it out of the stack, run it from the checkout, and put it back when we’re done, with the following commands.

stack platform detach auth --port 8080   # its hostname now goes to port 8080 on your machine
stack run auth                           # runs your checkout, with the stack's env
stack platform attach auth               # back in the stack

Nothing else restarts, the other containers keep running and the proxy just sends that hostname somewhere else.

flowchart TD
  accTitle: Where a hostname goes when a service is attached and when it is detached
  caddy["Caddy"] -->|"attached"| container["auth container"]
  caddy -->|"detached"| host["port 8080 on your machine"]
  container --> infra["database, cache and queues"]
  host --> infra

detach writes the env the service needs to ~/.stack/platform/external/<service>.env, with the database, cache and queues on 127.0.0.1, and stack run loads it, so if you want to run under a debugger you can point your IDE at that file instead.

The service runs on your machine now, so you need its toolchain installed.

Until the service answers, its hostname shows a page with the stack run command you need, which is a lot better than a bare 502.

The sites and portals have no image yet so they always run this way, and unless you bind their dev server to 127.0.0.1 it listens on IPv6 loopback only, which Docker can’t reach.

Three brands also means three dev servers from the same checkout, and by default it allows only one, so that has to be turned off.

Run what prod runs

Running your own checkout of the API took three attempts in one day.

First it ran straight on the host, which didn’t have what the app needs, then it built the checkout’s dev target, which worked but meant waiting for a build.

The version that stuck mounts the checkout and a dev config over the same image the stack already pulled, and switches on the debugger, which is already in the image.

It doesn’t need a build, and everybody runs the same image.

Fake servers don’t find Windows bugs

GitHub sign-in and self-update were tested against fake servers at first, and that was fine until it ran on real machines.

On Windows you can’t replace a running exe, so stack update moves it to stack.exe.old first, and if an older stack is still running from that .old file, the rename fails with Access is denied.

So now it moves to a unique .old-<n> when .old is locked, cleans up whatever is free on later runs, and warns you that an older stack is still running.

CI only vets the code for macOS and Windows, and even a Windows runner wouldn’t have caught this, that bug needs an old process hanging around, and you only get that from someone using it day to day.