Security

Zero alerts did not prove our leaked-key detector was running

Per-key spend detection, automatic throttling, a one-step revoke and GitHub secret-scanning intake, and why a silent detector had to prove it was alive.

On this page

In the public AI-infrastructure key leaks of October 2026, almost every abuse was found by the victims' bills, not by the platforms' telemetry. At the time Apertis had nothing that watched each customer key's spend for anomalies, only rate limits. In November we shipped a per-key spend detector, automatic throttling, a one-step revoke and an intake for GitHub secret-scanning reports. The first production read of the detector found zero: over the whole life of the production containers, it had written no anomaly line and throttled no key. That zero could not tell a quiet detector from a dead one.

The short version

  • Detect. Each key's recent spend is compared with that key's own history, off the request path.
  • Slow. An anomalous key is held to a reduced request rate for a limited time, and the hold ends by itself.
  • Revoke. One command revokes a key of any kind, evicts it from Redis and the in-process cache, and emails its owner. In one staging rehearsal, the key was refused 1.6 seconds later.
  • Stay revoked. Four subscription status writers could overwrite a revoked key's status. They now skip it.
  • Hear from GitHub. An endpoint verifies GitHub's signed reports and tells an operator, once our partner enrolment is done (it is not yet). It changes no key's status.
  • Stay visible. The detector now writes a periodic alive line, and a missing line pages.

We do not publish the detector's rules, thresholds or windows. The customer-facing summary of what flags a key is in changelog 2.2.122.

Keep the request path out of it

The existing per-key spend recorder skipped keys whose owner had set no spend limit. The detector had to see every key without slowing any request.

The settlement code calls NoteKeySpend, which takes one mutex and adds to a process-local map. It makes no Redis call and no database read, so the detector cannot block a settlement. A flusher on the API role later pipelines the pending amounts into per-key buckets in Redis and judges them there. The anomaly line carries integers and a closed set of rule names, so no key value, model name or request text reaches it.

Throttle automatically, never revoke automatically

Automation may throttle an anomalous key but never revoke it. A throttle is reversible and limits the damage while someone looks. A revoke is terminal and breaks the owner's integrations, so it stays a person's decision.

Containment writes a Redis marker that expires by itself. While it exists, the per-key rate-limit middleware gives that key a lower request rate.

The detector's first review found two bugs that it rated important. The marker that records when a key last spent expired too soon, so a key idle for longer than the marker lasted was never noticed. And a detector outage made every key look idle, because no spend was recorded during the outage. The fix stops a detector outage from being read as idleness. Four deliberately broken versions of the review fixes each failed a test, and the 28 detector tests passed on PostgreSQL with Redis.

A later check found a gap outside the code: no alert route carried the containment audit line, so a throttled key would have paged nobody. We fixed the alerting before containment turned on.

Why zero anomalies proved nothing

The detector reached production on 2026-11-05. On 2026-11-12 we read the production container logs with two read-only reads that returned only counts. They covered each container's whole life and found 0 anomaly lines, 0 containment lines, 0 "resumed after a gap" lines and 0 store errors. The same reads saw the security heartbeat lines, so the read that counts audit lines was seeing real lines.

That still proved nothing. The detector wrote nothing when it found nothing, and when it had no store it returned without a word. A dead detector and a quiet one produced the same log. We closed the gap in two steps.

  1. Prove the page. A drill injected one synthetic contained-key line into the alerting loop's input, never into the API. In that one drill, on 2026-11-12, the page opened, and it cleared by itself when the drill expired.
  2. Prove liveness. The flusher now writes an alive line at a fixed interval, and the line says whether its store answered. A store that is absent or failing is reported, never silent. The alerting pages when a container's alive line goes missing, or when it reports no store.

When the alive rule went on, on 2026-11-13, the first check met a container that had just started during a release. It produced a note and no page, as designed for a container that has just started. The next three checks were clean.

Revoke in one step

Before this work, revoking a leaked key was a hand edit. A database write could not evict the Redis key cache, so the key kept working after the write. The existing disable-token command refused org and subscription keys, and the owner's revoke routes need the owner's session.

The new command takes a token id, never a key, and is a dry run by default:

revoke-token --id N --reason TEXT [--execute] [--without-redis]
revoke-token: dry run (pass --execute to write) token_id=N user_id=N org_id=N kind=K status=S
audit action=emergency_revoke_token token_id=N user_id=N org_id=N kind=K changed=true reason="..."

With --execute:

  • It refuses to write without Redis, because the cached key would keep answering, unless the operator states that the serving containers have no Redis either.
  • It locks the row, sets status 5 (revoked) and the revoke time, and treats an update that does not change exactly one row as a failure.
  • After the commit, it evicts the key from Redis and the in-process cache, then checks that nothing in its reach still answers for the key.
  • It emails the key's user, and an org key's owners, naming the key by id and never by value.
  • A rerun evicts again and notifies nobody.

The reason is refused when it looks like a key, because a pasted key would otherwise land in the audit line.

Make revoked mean revoked

The personal and org key routes already treated status 5 as final; the subscription writers did not. With the previous code, TestRevokedSubscriptionKeyStaysRevoked fails. The Stripe webhook sync, the plan-status sync, the admin subscription status change and the refund resolver could each overwrite a revoked key's status with one that can be undone, enabled or disabled. Each write now excludes revoked rows, as in the plan-status sync:

return tx.Model(&Token{}).Where("id = ? AND status <> ?", *sub.DedicatedTokenId, TokenStatusRevoked).
	Update("status", targetTokenStatus).Error

Review found that this broke something else: the non-transactional writers now returned an error for a revoked key, so a replayed checkout or dispute webhook would fail and Stripe would retry it forever. They now treat a revoked key as already settled. Renewing a plan key also copied status 5 onto the replacement; that fix (#7183) merged first.

Rehearse it before an incident does

Nobody can try a revoke on production, so the staging deploy workflow has an opt-in rehearsal. It mints a throwaway key for a staging fixture user, and the seed command refuses to run unless the deployment declares staging. Before revoking, it shows that the key answers 200, that a wrong key gets 401, and that the owner's routes accept the same requests on the live key.

The first rehearsal failed in the seed, not in the product: the workflow minted a 48-character access token for a 32-character column. After that fix (#7301), the second run passed. The revoke ran at 23:23:42.29 UTC and the key was refused at 23:23:43.86, about 1.6 seconds later. That is one run.

Way back Who would use it Answer after the revoke
enable-token operator refused
PUT /api/token/?status_only=true owner 409
PUT /api/token/ owner 409
POST /api/token/:id/rotate owner 409
GET /v1/models with the key anyone 401

The subscription undo paths need Stripe state that a staging window does not have, so the PostgreSQL test covers them.

Take GitHub's reports

GitHub's secret-scanning partner program sends matched tokens to the issuer's endpoint. Once our partner enrolment is done, which it is not yet, GitHub will send ours to POST /api/secret-scanning/github. GitHub signs the raw body with ECDSA P-256 and SHA-256, and the endpoint verifies it against keys fetched from GitHub's fixed URL, never from the request.

The request body is a JSON array with one item per match. Each item has four string fields: token (the matched string, such as sk-apx_…), type, url (a https://github.com/… address) and source. The endpoint answers with one item per token:

[{"token_hash": "<sha256 of token>", "token_type": "<type>", "label": "true_positive"}]

A token that lacks our key shape or a valid checksum is labelled a false positive without a database read. For a live key, the intake notifies an operator, and a Redis marker named by the token id stops a repeat report from sending a second notice. It does not revoke: a report is a lead for a person.

Answer When
200 the feedback array, one label per reported token
400 the body is not a JSON array of reports
401 the signature is missing or wrong
413 the body is over the size cap
503 GitHub's keys cannot be read, the lookup failed, or a notice could not be delivered

The last 503 case is deliberate. GitHub sends one report per new leak location, so a notice that did not go out has to make GitHub send the report again. The intake releases its marker and answers 503. That rests on a premise we have not confirmed: GitHub's documentation does not say that it retries on a 5xx. If it does not, the notice is lost, although the audit line is still written.

The route is already deployed: an unsigned POST on 2026-11-05 got 401 from the handler.

What we would do differently

  • Ship the liveness signal with the detector. A detector that reports only when it finds something cannot be told apart from one that stopped.
  • Write terminal states as a WHERE clause, with a test that fails on the old code. That baseline failure was the evidence that the fix did something.
  • Prove a refusal against a success. The rehearsal's 401 means something only because the same requests succeeded on the live key just before; otherwise it could be a wrong header.

Still open

  • As of our read on 2026-11-12, no real anomaly had fired in production. We have no catch rate, and the outcome is recorded as unknown.
  • The GitHub enrolment is not done. Its first test delivery will check the retry premise and production access to GitHub's key list.

How each number was measured

  • The zero counts: two read-only, counts-only reads of the production container logs on 2026-11-12.
  • 1.6 seconds to refusal: the timestamps in the one passing staging rehearsal, on 2026-11-05.
  • 28 tests and four broken versions: the detector pull request's evidence, on PostgreSQL with Redis.