Performance

Our gateway spent its latency budget before calling the model

Serial database and Redis round trips on our relay hot path, three changes that removed many of them, and what the measurements do and do not show.

On this page

Apertis runs one API in front of the major model providers, so every request passes through our gateway before an upstream model sees it. In July we found that a large share of a request's latency was spent before the gateway called the upstream at all: serial round trips to our own database and to Redis. Three changes over the next three months removed many of them. This post covers what was on the path, what each change did, the numbers, and what those numbers do and do not show.

The short version

  • Before calling the upstream, a request walked a chain: key, then user, then subscription, then billing, then channel. Each step waited for the one before it, and most steps were a synchronous call.
  • Our database is a managed PostgreSQL about 75 ms away from the API servers. At that distance, a handful of statements costs hundreds of milliseconds.
  • Change 1 (July) made the API key lookup cache-first.
  • Change 2 (October) cut the statements per successful request from 20.24 to 12.79 at 300 requests a minute, on a load rig with the same 75 ms distance.
  • Change 3 (October) made channel selection read an in-process snapshot, so it went from 15 Redis round trips to none for ten candidate channels when no recovery probe is due.

Where the time went

From the production API host, the TCP round trip to the database was 74 to 85 ms over five samples, and the application answered a local status check in 0.6 to 1.1 ms. The code itself was not the slow part.

The clearest number came from two endpoints measured in the same window over warm keep-alive connections, from a client outside our network, through our CDN:

  • /api/status, which does not run API key authentication: 118 to 122 ms to the first header.
  • /v1/models, which does: 192 to 198 ms.

The difference, about 75 ms, is one database query. Key authentication went to the database first and used its cache only as a fallback.

A code audit then traced the rest of the chain. A pay-as-you-go request made two or three serial database calls and about six Redis calls before it reached the upstream. A subscription key added three or four more database calls and two more Redis calls. Adding up the database calls at 75 ms each gave about 150 ms for a pay-as-you-go request and 375 to 450 ms for a subscription key on a model that also needs a compliance check. Those two totals are arithmetic from the code and the measured round-trip time, not end-to-end measurements.

The calls were serial for a plain reason: each step needed the previous step's answer. You cannot look up a user's subscription before you know which user the key belongs to.

Change 1: the key lookup goes to the cache first

  • The gateway keeps a mapping from the key's hash to the token, in process memory and in Redis. A cache hit still checks the presented key against the stored hash, so a rotated key is not revived by a stale entry.
  • The request timeout is read from the token that authentication already loaded. Before, it was read again from the database on every retry attempt.
  • A per-request compliance lookup on some models got a short-lived cache.

One part did not ship. We also planned to read a subscription user's quota from cache. Review found that a top-up or an admin edit did not refresh that cache entry. We kept the fresh database read and gave back about 75 ms on that path, because a correct balance matters more than 75 ms.

We have expected savings for this change, about 70 ms on /v1/models and about 150 ms for a pay-as-you-go chat request. We have no post-deploy measurement of it on record, so we present those as estimates.

Change 2: count statements, not milliseconds

For the second change we stopped measuring milliseconds and counted statements. We built a load rig with a PostgreSQL database behind 75 ms of added latency and a pool of 10 connections, and drove it at 300 and 600 requests a minute. A successful request sent about 20 statements at 300 requests a minute. Most of them did not need to exist. Some of these run during settlement after the response, so on the rig they cost connection-pool time, which is what delayed other requests' first tokens.

  • Inserting a usage log row took three statements (BEGIN, INSERT, COMMIT), because our ORM wraps every write in a transaction by default. Inserting a routing event row did the same.
  • The channel's row was read by id three times per request, although the routing cache already held it.
  • A monthly spending alert ran seven statements, including a row lock, even when there was nothing to claim.
  • The relay read the API key's settings from the database again, although authentication had already loaded them.

Each fix was small:

  • Single-row inserts skip the default transaction.
  • Channel metadata comes from the routing cache.
  • The key's settings travel in the request context.
  • The alert checks without a lock first and takes the lock only when there is something to claim.

An independent verification, not by the agent that wrote the change, ran one clean before-and-after pair at each load level:

  • Statements per successful request: 20.24 to 12.79 at 300 requests a minute, and 15.90 to 8.37 at 600.
  • Heavy requests that succeeded at 600 requests a minute: 2,849 of 2,993 before, all 2,993 after.
  • Time to first token for heavy requests, p50: 5,756 ms to 516 ms at 300 requests a minute, and 22,631 ms to 724 ms at 600.

Read the last line with care. The rig's pool is deliberately small, so it runs close to saturation. Taking about seven and a half statements off every request drains a queue that had been growing, and that is where most of the seconds went. Production is not starved like that, and our own release note said only that latency should drop slightly. The number we trust for production is statements per request. The clean comparison is one pair per load level, not a series, so these are single runs.

Change 3: channel selection stops asking Redis

Choosing which channel serves a request reads shared state: whether each candidate's circuit breaker is open, its recent health, and its latency window. Before this change, each candidate's circuit check was its own Redis script call, and the window metrics were read twice. Selecting among 3, 10 and 30 candidates took 8, 15 and 35 serial Redis round trips. The database was not involved: zero SQL statements over 6,000 selections.

Round trips set a floor that faster code cannot lower. Even at 0.1 ms each, 35 round trips cost 3.5 ms. When Redis was slow, each call could also wait for its own timeout, one after another.

Now each process keeps a snapshot of the routing state for every model and channel pair it can route. It refreshes the snapshot from Redis every 250 ms in one pipelined pass, and updates it with the outcomes of its own requests. Selection reads only the snapshot. When a broken channel is due for a recovery probe, one Redis call still claims that probe, so only one instance sends it.

The trade-off is staleness. A circuit opened by another instance can still be chosen here until its next refresh completes: about one interval plus the time the refresh takes. A test measured 117.7 ms at a 100 ms interval.

These numbers come from an in-process benchmark on an Apple M4, with Redis running locally (PING p50 65 µs). Each pair is p50 / p99:

  • 3 candidates: 682 µs / 3.24 ms before, 3.0 µs / 13 µs after; 8 Redis round trips before, none after.
  • 10 candidates: 1.28 ms / 2.17 ms before, 9.8 µs / 59 µs after; 15 round trips before, none after.
  • 30 candidates: 3.17 ms / 6.60 ms before, 31 µs / 138 µs after; 35 round trips before, none after.

The benchmark also checks behavior, not only speed. A channel with an open circuit was picked 0 times at every size. The control case, with the circuit closed, picked it 197 times in 2,000. A rerun after the merge measured 98.8 µs p99 for 10 candidates, higher than the 59 µs above, because runs vary. Both are far below a millisecond.

Two limits apply. The "before" column used a Redis on the same machine; over a network, "before" would have been worse, but we did not measure that. And the benchmark covers channel selection only, not a whole request.

What we would do again

  • Measure the difference between two endpoints in the same window. Absolute latency moves with the network from hour to hour. The gap between an endpoint with authentication and one without isolated a single query.
  • Count statements per request on a rig with realistic distance. On a laptop with a local database, 20 statements cost almost nothing, and the problem stays invisible.
  • Ask what staleness you can accept. Accepting state about 250 ms old removed the Redis round trips from channel selection in the common case; a due recovery probe still makes one.
  • Write down what you did not measure. Every number in this post comes with its conditions, because the same number means different things on a saturated rig, a local benchmark and production.

Still open

We have not published the gateway's own overhead at p50 and p99 in production, end to end. That measurement belongs to a separate piece of work, and we will update this post when it is in.

How each number was measured

  • Change 1: a probe script that times warm keep-alive requests and splits connection, headers, first data and total, run from a client outside our network; plus the round-trip time to the database, measured from the production host.
  • Change 2: the settlement load rig with statement profiling, at 75 ms database latency and a pool of 10, one clean before-and-after pair at 300 and 600 requests a minute, run by an independent verifier.
  • Change 3: the routing selection benchmark in our test suite, with Redis on, measuring p50 and p99 over many calls per cell, plus a rerun on the main branch after the merge.