Selected workCircuit 4 · Machines
Server provisioning
Servers set up from one file, safely
- My role
- Sole author
- When
- ABM Egypt · Nov 2025 – present
- Where it stands
- Built all nine servers
What I built
A provisioner and an unattended capacity controller that create the task platform’s production servers from a single declarative spec.
Why it mattered
Hand-built servers drift, and the server types we needed were often sold out. Getting them had to be repeatable, safe to leave running, and impossible to turn into a duplicate bill.
$ provision --spec platform.spec
Illustration running in your browser. It follows the real system’s rules in simplified form; names and numbers are invented, and nothing here touches the real system.
What made it difficult
Ambiguous failures
A create request can time out on our side and still succeed at the provider.
Scarce capacity
Some server types were out of stock for long stretches, so the tool had to keep trying without hitting the provider’s rate limit.
Crashes
It has to survive dying at any point without losing track of what it ordered.
How I solved it
Dry run by default
Nothing changes without --confirm. A quota and capacity check runs first, and a lock guarantees only one copy runs.
Never delete
Servers are replaced one for one, with identity checks. There is no delete path.
Write intent first
Before each create it records what it is about to do, then re-reads the provider’s state. A crash or a timeout can’t produce a duplicate.
Patient by design
systemd timers keep trying for scarce servers, paced to stay under the API rate limit.
What I delivered
It built the task platform’s nine-server production environment, and handles resizing and one-for-one replacement of those servers.
I also wrote the matching provisioning code in PHP inside the platform.
Built with
- Bash
- jq
- Hetzner Cloud API
- cloud-init
- systemd timers
- flock