August 19, 2026 · 4 min read
The Scale Tax: what auto-scaling doesn't tell you about your next invoice
Scaling an app is the fun problem. Paying for it after the traffic leaves is the one nobody schedules a meeting for. A field record of how we built a code-execution engine that scales to 500+ concurrent users — and the auto-scaling traps that nearly kept it billing us long after the users went home.
Jul 15 — sized for exam peak
$404/mo
0.4% avg core CPU
Aug 7 — right-sized + scheduled
$210–230/mo
45% cut, zero downtime
Aug 9 — hibernated
~$0/mo compute
two commands from full capacity
Why you scale at all
Every request that lands on your server consumes a slice of CPU, memory, and time. Scaling exists because the number of slices you can hand out at once is finite — and the moment demand for slices outruns supply, requests queue, then time out, then your users leave a review about it.
In a platform we shipped and load-tested, the concrete version of this problem is a judge — a code-execution engine that takes a student's submitted code, drops it into an isolated sandbox, compiles or interprets it, runs it against test cases, and reports back pass/fail. Every submission needs an executor: a worker process that owns a sandbox for the seconds it takes to run that one submission. One executor can only run one student's code at a time.
So the scaling question is never really "how many users" in the abstract. It's "how many executors do I need available at the exact second 500 students hit Submit within the same minute during an exam." That's a much more specific, much more measurable question — and measuring it is what everything else in this piece is built on.
We measured ours directly: on our worker instance type, throughput holds at roughly 0.85 executions per second, per vCPU, once the executor pool is warm. That single number is the difference between sizing a fleet on evidence and sizing it on a guess you'll pay for every hour it's wrong.
Vertical vs. horizontal — and the ceiling nobody mentions
There are exactly two directions to add capacity: make the machine you have bigger, or add more machines like it. Most teams live the difference the hard way before they can name it.
Vertical scaling means giving one instance more vCPU, RAM, or disk. It's the first lever every team reaches for, because it's a single dropdown in a console and nothing about your architecture has to change. It has a hard ceiling, though — the biggest instance type your cloud sells — and a quieter one: a single machine accumulates state forever. Logs, temp files, database rows, container layers. Nothing about "bigger" fixes that.
Horizontal scaling means running the same job on more machines behind a load balancer, and swapping instances in and out instead of enlarging any one of them. It removes the ceiling and the single point of failure, at the cost of real coordination: shared state has to move somewhere all the instances can see it, and every instance needs to be genuinely disposable.
Fig. 1 — The judge ran vertically first: one DigitalOcean droplet, everything on it. It failed the same way, repeatedly — not from traffic, but from a full disk (unbounded logs + a Postgres table that keeps every submission forever), which meant compiled languages like Java and C++ stopped working while Python, needing no disk artifact, kept running. The fix wasn’t a bigger droplet. It was making workers disposable.
That failure is worth sitting with, because it wasn't a traffic problem at all. Vertical scaling doesn't just cap your ceiling — it lets entropy accumulate somewhere you can't reset without downtime. Our current architecture keeps one always-on "core" instance for the database and API, and a horizontal, spot-backed fleet of stateless worker instances that do nothing but run sandboxes — disposable by design, replaced rather than repaired, scaling from 1 to 8 instances depending on load.
Auto-scaling is the right idea. Here's where it empties your wallet.
The pitch for auto-scaling is simple: stop paying for peak capacity around the clock, and let the fleet track real demand instead. The gap between that pitch and a live AWS bill is where most of the damage happens.
Fig. 2 — This is the shape of every over-provisioned fleet: one short, real spike, surrounded by hours of idle capacity billed at the same rate. Auto-scaling exists to shrink the shaded area — but only if the policy behind it is honest about how long a new instance takes to become useful.
That last part is where it goes wrong. Five specific ways, all of which we've paid tuition for:
Cold-start blindness
Our worker instances take 7–9 minutes to boot before they can run a single sandboxed execution. A policy that reacts to “CPU is high right now” adds capacity that lands eight minutes after the spike that triggered it — often after the spike, and the users, are gone.
The scale-from-zero death spiral
Target-tracking on average CPU, combined with a group minimum of zero, can read a still-booting instance’s low CPU as “not needed” and terminate it before it ever serves a request — so the group scales down the exact capacity it just scaled up, on a loop, without ever landing. We hit this directly. See Figure 3.
No explicit ceiling
Every Auto Scaling Group needs a deliberate, written max_size. Without one, a retry storm, a scraping bot, or an infinite-loop bug doesn’t hit a wall — it hits your card statement, a week later, in a currency you don’t check daily.
Scaling the wrong tier
If the real bottleneck is a saturated database connection pool downstream, adding CPU-triggered compute upstream doesn’t add capacity — it adds more things competing for the same starved pool, at a higher hourly rate.
Spot without a fallback plan
Spot capacity runs 60–70% cheaper and is the right call for stateless, disposable work — losing one mid-job is just a retry. Put anything stateful on spot for the same discount and the 70% saving stops being a saving.
Fig. 3 — The fix we shipped wasn’t a smarter metric. It was refusing to let target-tracking bootstrap itself from nothing — a known-good desired capacity, set explicitly on a clock for known-busy hours, instead of a feedback loop measuring capacity that hadn’t arrived yet.
The cost ledger: $404 → $210–230 → ~$0
None of this was a redesign. It was the same architecture, turned into a dial instead of a switch, and the dial got turned down in stages — each one driven by a real number from Cost Explorer, not a hunch.
Fig. 4 — The hatched sliver on the last bar is deliberate, not an artifact: the load balancer and its Elastic IPs were left running through hibernation, and that residual floor hadn’t yet been checked against real Cost Explorer billing at time of writing. An honest chart shows the number you haven’t confirmed, not just the one that makes the slide.
Cheapest sticker price and sustainable infrastructure are not the same claim. The droplet was cheaper on paper and cost us a 2 a.m. incident every few weeks.
The playbook, if you're building on a budget — or not
None of this is exotic. It's Auto Scaling Groups, a couple of CloudWatch schedules, spot instances, and someone who actually reads the bill instead of just paying it.
Know your throughput before you provision
A guess sized for imagined peak is a guess you pay for every hour it's wrong. Get one real number — ours was execs/sec per vCPU — before choosing an instance size.
Stateless scales differently than stateful
Disposable workers can be spot, scheduled, and scaled to zero. Anything holding data needs a steadier, more deliberate plan — never the same lever.
Predictable load gets a cron, not a policy
Reserve reactive auto-scaling for genuinely unpredictable traffic. Office hours, exam windows, launch days — you already know the shape. A schedule is boring, and that's the point.
Every ASG gets an explicit max, no exceptions
“We'll add a ceiling later” is how a retry storm becomes a bill you notice a week after the fact.
Rehearse the wake-up before the dead season, not during it
Keep the reversible path documented and tested. The worst time to learn your restore command has a typo is the morning of the next cohort.
Cost and security often point the same direction
Locking a database down to only the IPs that need it isn't just a security fix — it removes an entire class of “something is hammering our DB and the bill can't explain why” incidents.
Scale is not the hard problem. Surviving the six months after it is.
Every team obsesses over whether their architecture can handle scale. Fewer ask whether it can survive the quiet months between launches — the ones with no traffic, no urgency, and a fleet that's still billed at exam-day size because nobody scheduled the review.
If your infrastructure can't sustain itself financially, you don't get to find out if it can handle your first thousand users. You run out of runway before you get the chance.
The engineering here was never the hard part — Auto Scaling Groups and CloudWatch schedules are documented, boring, well-trodden tools. The discipline is: measure before you provision, write down the number that should trigger the next review, and treat "we can turn this off" as a feature you build and rehearse — not one you hope you remember under pressure.
Field notes from a shipped, production judge infrastructure — figures reflect real Cost Explorer data through August 2026.