Why an AI Sent 155 People Someone Else's Data: How AI Memory Gets Reused, and How to Check Yours

Between 27 August and 9 September 2026, an AI model on Cloudflare's Workers AI sent 155 answers that contained pieces of other customers' requests. Nobody attacked it. The model promised to accept prompts up to 1,048,576 tokens, its GPU memory could hold 681,280, and a bug let the too-big prompts through. This article explains how AI memory gets reused from one request to the next, why that reuse leaked, what I found when I tested cache sharing on my own Azure account, and what to check in your own setup.

You don't need to know how GPUs work. If you've called an AI API, you can follow this. Every section ends with a one-line Remember, and the whole article fits in 10 lines at the end.

• Part 1 · What happened: the incident and its timeline, sections 1–2.
• Part 2 · Why it can happen: working notes, reused pages and the one-number mistake, sections 3–6.
• Part 3 · Sharing by design: prompt caches, my test on Azure, and an older incident with the same shape, sections 7–9.
• Part 4 · What to do: fixes by who you are, and a canary check you can run today, sections 10–12.

Three things people often assume, and what's true:

1. 155 answers with someone else's data

The model was GLM-5.3 on Cloudflare Workers AI. Customers sent it prompts and got answers back, like any AI API.

For 13 days, some of those answers "contained data from other customers' requests made to the same model shortly before theirs," in Cloudflare's words. The leftover memory held "other users' prompts and partial generations." Cloudflare counted 155 affected responses, all to prompts longer than 681,280 tokens, and says only "a small number of customers" were affected.

Picture a print shop. You pick up your printouts and one page in the stack isn't yours. You didn't do anything wrong, and neither did the person whose page it was. The machine mixed them up.

2. The timeline

Two details matter for later. The bug had already been reported upstream in July, by someone else. And the fix that returned the model to service was two changes at once: a bigger memory pool, and an engine patch that rejects requests it can't hold.

3. The AI writes working notes

When a model answers, it works in two phases.

First it reads your whole prompt. For every token, it computes a small set of numbers that later steps will look up again and again. Storing them saves recomputing them for every word of the answer. These stored numbers are the model's working notes on your prompt. Engineers call them the KV cache (key and value vectors), and the reading phase is called prefill.

Then it writes the answer one token at a time, looking back at those notes each time. This is called decode.

The notes are big. A long prompt's notes can take gigabytes of GPU memory, which is the most expensive memory in the system.

4. The notes live in shared pages

Serving software doesn't give each request its own private block of memory. That would waste most of it. Instead, GPU memory is cut into small fixed-size pages, and a request gets as many pages as its prompt needs. When the request finishes, its pages go back to a shared pool for the next request. This design comes from vLLM's PagedAttention, and most modern servers, SGLang included, work this way.

Freed pages aren't wiped. Wiping every page after every request would cost time on the most expensive hardware you own. It's also unnecessary, as long as one rule holds: a request only reads pages it wrote first.

5. The mistake: one number that didn't match

Cloudflare ran GLM-5.3 with disaggregated serving: one set of GPUs does the prefill, another does the decode, and the notes are copied between them over a fast network link. It's a common way to run big models efficiently.

The model was configured to accept prompts up to 1,048,576 tokens. The memory pool on the prefill GPUs held 681,280.

A prompt between those two numbers fitted the promise but not the memory. The right response is an error: "too big". SGLang had a bug here. It marked the request "reject", but the mark went into a field the rest of the code didn't check, so the request kept going through the hand-off to the decode GPUs. Decode then read pages that had never been filled for that request. Whatever the previous user left there went into the answer.

The SGLang team found a second, related problem while fixing it. If a request is cancelled in the middle of the copy, its late writes can land in pages that already belong to the next request. That corrupts the other user's answer silently, with no error.

6. Why it wasn't caught

• It needed a rare input. Only prompts above 681,280 tokens triggered it. Almost no test suite sends prompts that long.
• It only happened in one setup. On a single machine, SGLang returned an error for the same request. The bug was in the hand-off between prefill and decode.
• It failed quietly. A wrong answer looks like an answer. Nothing crashed and nothing was logged.
• Cloudflare knew about the mismatch. Its post says the team assumed the engine would return an error for any prompt it couldn't process. The assumption was the weak point.

7. Prompt caching: reuse on purpose

Section 4 was about reusing free pages. Prompt caching goes further. If two requests start with the same text (the same system prompt, the same document), the server keeps the first request's notes and lets the second one read them directly. Nothing is recomputed. That's why providers charge less for cached tokens.

This is safe in one sense: identical text produces identical notes, so the second request reads exactly what it would have computed itself. No content moves between users.

But it leaks one fact: someone else recently sent this exact text. A cache hit comes back faster, and many APIs report it directly as cached_tokens. In 2025, Stanford researchers audited AI APIs for this and found caches shared across users at 7 providers, OpenAI among them (Gu et al., ICML 2025). Providers may have changed their setups since.

Azure's documentation states the boundary: "The system doesn't share prompt caches between Azure subscriptions." It says nothing either way about sharing inside one subscription, so I tested it.

8. My test: does the cache reach next door?

Setup. Two separate Azure AI resources in one subscription (A and B), both in East US 2, both running gpt-4.1-mini on Global Standard. Each trial used a fresh 25,006-token prompt that starts with a random code, so no earlier trial could have cached it.

Method. In each trial I first sent B a prompt nobody had sent (the control). Then I sent A a new prompt until A reported a cache hit, then sent B that same prompt. If B's very first request reports cached tokens, B is reading notes that A's requests created. In run 2 the two deployments had different names, to rule out the cache being keyed on the name.

When I ran it, B's first request reused A's cache in 13 of 17 trials. The control never hit, 0 of 17. In run 2 alone, a difference this big would happen by chance about 4 times in 10,000 (Fisher's exact test).

What this means:

• It's by design, not a bug. One subscription belongs to one customer, and Microsoft only promises separation between subscriptions.
• No text crossed over. B only learned that someone nearby had sent that exact 25,000-token prompt.
• It matters for one common setup. If you serve many customers from one subscription, giving each customer its own resource doesn't give them separate caches.

Two more numbers from the same runs. A cache hit covered 99.8% of the prompt (24,960 of 25,006 tokens), which cut the cost of a call from ₹0.96 to ₹0.24 at Azure's list prices. And the cache didn't kick in right away: the first repeat of a prompt never hit it (0 of 7), while the third repeat almost always did (7 of 8).

9. Not the first time

On 20 March 2023, OpenAI took ChatGPT offline. A bug in redis-py, the library ChatGPT used to talk to its Redis cache, let some users see titles from another active user's chat history. During a 9-hour window, about 1.2% of active ChatGPT Plus subscribers may have had billing details (name, email, address, the last four digits and expiry of a card) shown to someone else (OpenAI's postmortem).

The trigger there was a spike in cancelled requests. A cancelled request left a shared connection in a bad state, and the next user got the reply meant for someone else.

Different layer, same shape as section 5: a shared resource, reused for speed, plus a rare case (a cancelled or oversized request) that skips a step.

10. If you run the model yourself

This applies if you host models with vLLM, SGLang or a similar server.

• Never promise more than memory holds. Set the maximum prompt length to what the memory pool can actually hold, or make the pool big enough for the length you advertise. Check this whenever you change the model, the GPU count or the memory settings.
• Reject at the door. The request should get an error before it's admitted, not partway through. SGLang's fix (PR #38935, merged 16 September) does exactly this.
• Upgrade. Cloudflare recommends that everyone running SGLang update to v0.5.22 or later. The SGLang post lists the affected versions and the settings to use on older ones.
• Isolate caches per customer. vLLM accepts a cache_salt with each request. Requests only share cached notes when their salts match, which its docs describe as a defence against timing attacks. Give each customer, or each trust group, its own salt.
• Send your limits in tests. Add a test that sends a prompt at the maximum length you advertise, and one just above it, and checks you get an answer and an error.

11. If you build on an AI API

This is most people: you call Azure OpenAI, OpenAI, Anthropic, Google or another provider.

• Know your cache wall. On Azure, it's the subscription. Put customers whose data is sensitive (health, money, legal) in their own subscription.
• Don't expose cache signals. Don't pass cached_tokens or exact response timings through to your customers. Those are the signals the Stanford audit used.
• Keep secrets out of prompts. API keys, passwords and full card numbers don't belong in a prompt. Redact personal data you don't need for the task.
• Ask your provider two questions. Who shares my prompt cache? How will I hear about an incident like this one?

12. A canary check you can run today

A canary is a random word you plant in each request. If an answer ever contains another request's canary, something is mixing requests. I wrote a small script for this. It needs only Python's standard library.

It sends many requests at once. Each one has the same long shared prefix (so the provider's cache is in play) and its own canary, and asks the model to repeat its canary. Then it checks every answer for anyone else's.

# Azure OpenAI
AZURE_ENDPOINT=https://<resource>.cognitiveservices.azure.com AZURE_KEY=<key> DEPLOYMENT=<name> \
  python3 canary_check.py --n 300 --workers 20

# Any OpenAI-compatible server (vLLM, SGLang, others)
BASE_URL=http://<host>:8000/v1 API_KEY=<key> MODEL=<model> python3 canary_check.py

When I ran it on gpt-4.1-mini in Azure (300 requests, 20 at a time):

{
 "requests": 300,
 "answered": 298,
 "echoed_own_canary": 298,
 "answers_with_cache_hits": 279,
 "answers_with_foreign_canary": 0,
 "seconds": 53.1
}
No foreign canaries found. (A clean run is a smoke alarm, not a proof.)

279 of the answers used the shared cache, and none contained another request's canary. A clean run doesn't prove a provider can't leak. Cloudflare's bug needed prompts over 681,280 tokens, which this script doesn't send by default. Treat it as a smoke alarm: cheap enough to run on a schedule, and worth running after any change to how you serve models.

Get the script: canary_check.py.

Limits of my tests

• One region, one model, one subscription. I didn't test across subscriptions, so the claim that the wall is the subscription is Microsoft's, not mine.
• Small samples. 17 trials per arm in the neighbour test. The gap is large and unlikely to be chance, but the exact rates could differ elsewhere.
• No speed claim. I timed calls from my laptop in India to a US data centre, and the network noise was bigger than any cache speed-up. A fair speed test needs a machine in the same region.
• I didn't reproduce Cloudflare's bug. That needed their setup. Sections 3–6 come from Cloudflare's and SGLang's posts.
• Some calls failed on my home network (timeouts and DNS errors). They're excluded, not counted as misses.

Everything I ran is in the lab folder: cache_lab.py (the cache and neighbour tests), results.json (every call), canary_check.py, canary_results.json and prices_snapshot.json. The whole thing cost about ₹125 in API calls.

The whole guide in 10 lines

1. An AI model sent 155 answers with other customers' data, from ordinary long prompts.
2. While reading a prompt, the model writes working notes (the KV cache) into GPU memory.
3. That memory is cut into pages, reused between requests and not wiped, so a request must write before it reads.
4. The model accepted 1,048,576-token prompts, but memory held 681,280.
5. A rejected request kept going and read pages it never wrote, so another user's notes ended up in its answer.
6. It hid behind a rare input and a quiet failure.
7. Prompt caching shares notes on purpose, and a cache hit reveals that someone sent the same text.
8. On Azure, two resources in one subscription shared a cache 13 of 17 times in my test.
9. If you run the model: fit the promise to the memory, reject at the door, salt caches per customer.
10. If you use an API: know your cache wall, hide cache signals, keep secrets out of prompts, and run a canary check.

Checked against Cloudflare's incident post and SGLang's security update (26 October 2026), Microsoft Learn's prompt caching page (updated August 2026), vLLM's prefix caching docs, OpenAI's March 2023 postmortem and Gu et al. (ICML 2025), as of October 2026. Azure numbers are my own runs on 29 October 2026 (gpt-4.1-mini, Global Standard, East US 2) with prices from the Azure Retail Prices API in INR. The print-shop comparison and the step-through figure are illustrations; every number is measured or quoted from the sources.