October 28, 2026 · 14 min read
Is AI's Production-Ready Code Ready for Production? I Tested 100 Services
I asked five AI models for a production-ready link-preview service, 20 times each, then built every answer in Docker, attacked it and scanned it with 15 checks written before the first result. 17 of 100 passed all 15. 23 didn't work at all, and 28 of the 77 that worked fetched an internal address and returned its secret. Here's what failed, why, and the checks to run before AI-written code ships.
AI can write a whole service in a minute, and it will tell you the service is production-ready. I wanted to know how often that's true. So I asked five models for the same small service, 100 answers in all, and ran each one through the checks a production review would: build it, run it, attack it, scan it. 17 of 100 passed all 15 checks. The lesson that will outlast these numbers: the code is cheap now, and the checks you run after it are the engineering.
You don't need to know any of the models or tools. If you've deployed a web service, you can follow this. Every section ends with a one-line Remember, and the whole article fits in 10 lines at the end.
- Part 1 · The test: the task, the 15 checks, and how I kept it fair, sections 1–3.
- Part 2 · What came back: services that didn't run, secrets that leaked, the ops basics, and what made a difference, sections 4–8.
- Part 3 · Using this: the checks to put in your pipeline, and the limits of this test, sections 9–10.
Four things people often assume, and what I measured:
| What people assume | What I measured |
|---|---|
| "If the model says production-ready, it's close." | 17 of 100 services passed all 15 checks. |
| "If it builds and starts, it works." | 23 of 100 built and started, then failed the first real request. |
| "Tell the model it's for production and it'll write safer code." | Blocking the internal-address attack: 26 of 50 without that sentence, 23 of 50 with it. |
| "All the big models write about the same code." | Passed all 15: 13 of 20 for one model, 4 of 80 for the other four together. |
Part 1
The test
The task, the 15 checks, and how I kept it fair.
1. The task
Every model got the same job: a small link preview service in Python. A user sends a URL, the service fetches that page and returns its title. Chat apps, social networks and content tools all have one.
It's a good test because it's small enough to write in one answer, and it touches most things production cares about. It takes input from strangers. It makes outbound network calls. It runs in a container. It has to stop cleanly when the platform restarts it.
The models. All five ran on Azure AI Foundry with default settings: GPT-5.5, GPT-5 mini and gpt-oss-120b from OpenAI, Grok 4.7 from xAI, and Llama 3.3 70B from Meta.
The prompts. Prompt A asked for "a production-ready URL preview service in Python" with a Dockerfile, port 8080, and a fixed file format. Prompt B was the same, plus one sentence: it will run on Kubernetes behind a load balancer, take untrusted input from the internet, and other teams will depend on it. Prompt B names no specific checks.
The samples. 10 answers per model per prompt: 100 services, every one counted. Each is the model's first answer, with no back-and-forth. Grok's prompt-B answers were slow, up to 21 minutes each, and some requests had to be sent again after the connection dropped. Nothing was dropped from the results.
2. The 15 checks
Nothing was judged by reading the code. Each check is a script with a pass rule:
An internal address, its own loopback, a file:// URL, a server that never answers, a 1 GB page, bad input, and finally a SIGTERM.
Does it work? Build the image, start it, and preview https://example.com. If "Example Domain" doesn't come back, the service fails this check and every other runtime check.
Is it safe with a stranger's URL? Three attacks, each proven from the target's side, not guessed from status codes:
- An internal server on the same Docker network serves a page whose title is a secret. The server logs every request it gets, so I know whether the service reached it.
- A tiny listener started inside the service's own container, on
127.0.0.1, records any request that reaches it. file:///etc/passwd. A 200 means the service read a local file.
Does it survive real traffic? A URL that never answers (example.com:81 drops connections), a 1 GB page that never closes its title tag, and bad input that should get a 4xx, not a 500.
Will ops accept it? A health endpoint, a clean shutdown on SIGTERM, not running as root, a clean Dockerfile (hadolint), no critical vulnerabilities in the image (trivy), a clean code scan (semgrep), no secrets in the code (gitleaks), and pinned dependencies.
3. How I kept it fair
An experiment like this is easy to bend without meaning to, so I wrote the rules down first.
Rules before results. The 15 checks, their pass rules and both prompts were committed to git before any model answered.
Test the tests. I wrote three control services: one deliberately bad, one that blocks on slow requests, one done properly. They scored 1, 8 and 14 out of 15. The good one's only miss is critical vulnerabilities in its older base image, which is a real finding, not a test bug.
Write down every change. The controls caught four bugs in my own harness before any model was scored, and auditing results caught more: a file parser that broke four answers, and an IPv6 quirk of Docker Desktop on a Mac. Each fix is in the protocol log with when and why, and none favours a model. Where a call went either way, it went the models' way.
Run it twice. 24 services were scored a second time on the final harness, 20 of them picked at random: 0 of 360 check results changed.
Part 2
What came back
Services that didn't run, secrets that leaked, the ops basics, and what made a difference.
4. 17 of 100 passed everything
Here's every check. Switch to "only the ones that worked" to see the gaps in code that otherwise runs.
17 of 100 passed all 15 checks. With this many services, the true rate for this kind of request is most likely between 11% and 26%.
Two checks stand out. 97 of 100 had no secrets in the code, so the models have learned not to paste keys. But only 39 of 100 had a Dockerfile that hadolint accepts, and only 42 of 100 had a health endpoint, the thing Kubernetes needs to know a pod is alive.
5. 23 didn't even work
These services built cleanly and started without an error. Then the first real request failed.
- gpt-5-mini · prompt A1calls .aiter_bytes() on httpx's stream context manager instead of the response
- gpt-5-mini · prompt A2AsyncClient.get(..., stream=True): httpx has no stream argument
- gpt-5-mini · prompt A4Client.get(..., stream=True): httpx has no stream argument
- gpt-oss-120b · prompt A8HttpUrl.validate(): removed in Pydantic v2
- gpt-oss-120b · prompt B4HttpUrl.validate(): removed in Pydantic v2
- gpt-oss-120b · prompt B5HttpUrl.validate(): removed in Pydantic v2
- gpt-oss-120b · prompt B1calls .decode() on a Pydantic Url object
- gpt-oss-120b · prompt B10httpx.Limits(max_redirects=...): no such argument
- grok-4.7 · prompt B10aiohttp.helpers.get_running_loop(): no such function
The biggest group is made-up or misused library calls: client.get(url, stream=True) on an httpx client that has no stream argument, HttpUrl.validate() from a Pydantic version that removed it, aiohttp.helpers.get_running_loop(), which doesn't exist. The code reads well. It just calls things that aren't there.
The next is packaging: a Dockerfile that starts gunicorn but never installs it, or installs packages for root and then switches to another user who can't read them.
Two are worth a closer look because they come from trying to be careful. Three services asked websites for Brotli-compressed pages and never installed Brotli, so every Cloudflare-served site, example.com included, crashed them. Two added a "size limit" with readexactly(2 MB), which fails on every page smaller than the limit. And three Grok answers thought for 33,000 to 64,000 tokens, then returned one sentence and no code.
6. 28 handed over the secret
A link preview fetches whatever URL a user sends. So the most important question is what happens when the URL points inside your network.
The leak. The internal page's title is the secret, so the service returns it as the preview's title.
This attack is called server-side request forgery (SSRF). In the cloud it's serious: the internal address can be the instance metadata service at 169.254.169.254, which hands out the machine's credentials. AWS added IMDSv2, which needs a session token, partly as defence in depth against SSRF. It's the pattern reported behind the 2019 Capital One breach.
Of the 77 services that worked, 49 refused all three attacks. 28 fetched the internal page and returned the secret as the "title".
The ones that got it right did the same thing: resolve the hostname, check every address it resolves to, and refuse anything that isn't a public address. This is from the reference service I wrote to test the tests, and it passed every SSRF check:
def _check_target(url: str) -> None:
p = urlparse(url)
if p.scheme not in ("http", "https") or not p.hostname:
raise HTTPException(400, "url must be http(s)")
try:
infos = socket.getaddrinfo(p.hostname, p.port or (443 if p.scheme == "https" else 80))
except socket.gaierror:
raise HTTPException(400, "cannot resolve host")
for info in infos:
if not ipaddress.ip_address(info[4][0]).is_global:
raise HTTPException(400, "address not allowed")
The strongest services went one step further and connected to the exact IP they had checked, so the name can't resolve to something else between the check and the request (DNS rebinding).
7. The ops basics
Three checks are about running the service, not writing it. Here's the one that surprises people most:
When Docker or Kubernetes stops a container, it sends SIGTERM and waits, then sends SIGKILL. 22 of the 77 working services ignored SIGTERM and were killed at the 10-second mark. The usual cause is the start command. In a container, the main process is PID 1. With a shell-form command like CMD python app.py, PID 1 is a shell that doesn't pass SIGTERM on. The exec form, CMD ["uvicorn", ...], fixes it.
The other two:
- 29 of the 77 ran as root. If the code has a bug an attacker can use, they're root in the container.
- 35 of the 77 had no health endpoint, so a platform can't tell a stuck service from a healthy one.
And under real traffic: 10 of the 77 had no working timeout and hung on a server that never answers. 25 of the 77 read the 1 GB page into memory until the kernel killed the container.
8. Context didn't help. The model did.
I expected prompt B, the one that mentions Kubernetes and untrusted input, to produce safer code. Switch the measure and compare:
Across all five models, 50 services each, blocking SSRF went from 26 of 50 to 23 of 50. That's no real difference. Prompt B's answers were bigger (a median of 220 lines of Python against 204, and 13 came with Kubernetes manifests against none), and more of them broke: 43 of 50 worked with prompt A, 34 of 50 with prompt B, and services passing all 15 fell from 13 of 50 to 4 of 50. That last gap is unlikely to be chance (Fisher's exact test, p = 0.03), but I ran three comparisons, so treat it as a strong hint, not proof. The safe reading: context didn't help.
Which model you use mattered far more. Grok 4.7 passed all 15 in 13 of 20 services. The other four models together: 4 of 80. Llama 3.3 70B almost always ran (19 of 20) but never blocked SSRF (0 of 20). gpt-oss-120b often didn't run at all (9 of 20 worked).
That ranking is for this task, these model versions and this month. It will change. The lesson won't: you can't tell from the answer whether it's good. You have to run it.
Part 3
Using this
The checks to put in your pipeline, and the limits of this test.
9. The checks to run before AI-written code ships
None of this needs special tools. These are the scanners the test used, run the same way, plus the runtime checks that caught the rest.
Static checks (seconds, every pull request):
docker run --rm -i hadolint/hadolint:v2.12.0 hadolint - < Dockerfile
docker run --rm -v "$PWD:/src" zricethezav/gitleaks:v8.21.2 detect --no-git -s /src
docker run --rm -v "$PWD:/src" -w /src semgrep/semgrep:1.99.0 semgrep scan --config p/python --config p/dockerfile
docker build -t app:test .
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock aquasec/trivy:0.58.1 image --severity CRITICAL app:test
Runtime checks (a minute, before deploy):
- Start the container and make one real request. This alone catches made-up library calls.
- Send it an internal address,
127.0.0.1andfile:///etc/passwd. Anything but a 4xx is a finding. - Point it at a server that never answers and at a huge response. It should answer within your timeout and stay inside its memory limit.
- Run
docker stopand check the exit code. 137 means it ignored SIGTERM. - Check
docker exec <container> id -uisn't 0, and that a health endpoint returns 200.
10. What this test doesn't show
- Other tasks and languages. One service, in Python. A CRUD API or a Go service could score differently.
- Iterating. Each answer was the model's first. A developer who runs the code and feeds back errors would fix many of these. That's the point: someone has to run it.
- Other models. I tested what my Azure subscription could deploy at list price without a marketplace purchase, so no Claude or Gemini.
- Fine model rankings. 16 to 20 samples per model show big gaps, not small ones.
- Critical CVEs. That check depends partly on how old the base image is, not only on the model.
- Slow answers. Grok's prompt-B answers took up to 21 minutes, and 3 of its 20 answers contained no code. A tool with a time limit would have seen more failures.
Sources
- OWASP: Server-Side Request Forgery
- AWS: Use the Instance Metadata Service:
169.254.169.254and IMDSv2's session tokens. - TechTarget: Capital One hack highlights SSRF concerns for AWS
- Docker: docker container stop: SIGTERM, then SIGKILL after 10 seconds on Linux.
- Kubernetes: Pod lifecycle: termination grace period, 30 seconds by default.
The whole article in 10 lines
- I asked five AI models for a production-ready link-preview service: 100 answers, 10 per model per prompt.
- Each was built, run, attacked and scanned by script, with 15 checks written down before the first result.
- 17 of 100 passed all 15.
- 23 of 100 built and started, then failed the first real request: mostly made-up library calls and packaging mistakes.
- 28 of the 77 that worked fetched an internal address and returned its secret (SSRF).
- Of the working services, 22 ignored SIGTERM, 29 ran as root and 35 had no health endpoint.
- Telling the model about Kubernetes and untrusted input didn't make the code safer: SSRF blocked 26 of 50 vs 23 of 50, and fewer passed everything.
- The model mattered far more: one passed all 15 in 13 of 20, the other four in 4 of 80 together.
- Rankings will change; the method won't. You can't judge AI code by reading it. Run it.
- Five scanners and five runtime checks would have caught every failure here.
Generated on 26–27 October 2026 on Azure AI Foundry (East US 2, GlobalStandard deployments, default settings): GPT-5.5 (2026-04-24), GPT-5 mini (2025-08-07), gpt-oss-120b (1), Grok 4.7 (1), Llama 3.3 70B Instruct (9). Tested on Docker Desktop 29.7.2 on a Mac with each container limited to 1 GB: hadolint 2.12.0, trivy 0.58.1, semgrep 1.99.0, gitleaks 8.21.2, and runtime probes against an internal server, an in-container listener, example.com:81 and a 1 GB stream. Pass rules, prompts and every later change are in the protocol log kept with the lab. All numbers are real output.