feat(mcp): make the initialize limiter observable, and refill in minutes #23

Merged

jlxq0 merged 1 commit from feat-limiter-observable into main

2026-08-26 14:25:04 +00:00

jlxq0 commented

2026-08-26 14:21:26 +00:00

Owner

Closes #20.

What is true, and what is not

Not claimed: that we observed this limiter refusing Julian. We did not. The 429 seen from the client reads {"message":"Anthropic Proxy: Invalid content from server"}, which names the connector's proxy as the speaker and is equally consistent with its own rate limiting. Lucy was not a control either: her carddav mount answers whoami as julian@kampong.social, the same identity, and this limiter keys on the Logto sub.

What is claimed, all of it a fact about this code:

Eight failed attempts, and slots refill one at a time, so recovering from empty took four hours rather than the thirty minutes a reader would infer.
The refusal carried nothing actionable: no log line, no Retry-After, no error class.
The retry-after was computed by governor and discarded at the boundary — NotUntil::wait_time_from existed all along and check mapped it to a unit struct.
tool_calls_total reads zero during exactly this failure, because the limiter is Axum middleware outside the MCP router and a refusal never reaches the code that increments it. In the Infrastructure lead's words, from the investigation: "it measured a limiter that was not firing while the one that was firing recorded nothing."

None of that needs a 429 sighting.

The refill is the change that matters

burst    8  -> 24
refill   30 min -> 1 min          recovery from empty: 4 hours -> 24 minutes

At one slot per thirty minutes, someone who waits out what they believe is the window gets exactly one session, reconnects twice and is refused again. A wrong model of a limiter does not merely delay a user, it guarantees a second failure after the wait. That is why five attempts produced five identical failures.

Why 24 and not 32, and why the pool did not move

This is the fallback, taken deliberately. Burst 32 with MAX_SESSIONS at 256 takes the identities needed to exhaust the pool from 32 down to 8 — a per-user refusal converted into a global exhaustion, invisible in the diff, breaking somebody other than the person who triggered it.

Moving the pool with it requires the per-session memory measurement, and measuring means opening 32 authenticated sessions against a running pod, which needs a bearer this repo cannot currently obtain: Stalwart's directory is OIDC, so no password authenticates for anybody, which is the same blocker as #19.

So: 24 covers the observed workload exactly and moves the ratio to 10 identities knowingly. 1024 with burst 32 is the measured version, once a bearer exists.

Observability

tracing::warn! on the refusal branch with scope, user_hash and retry_after_seconds. user_hash is the domain-separated identity hash, which reverses against a candidate list and would have answered "whose refusal is this" in one grep. Verified by reproducing it: identity:julian@kampong.social gives f5a076c6a6c82848.
caldav_mcp_initialize_refusals_total{scope}.
A JSON body with a stable class, so a client distinguishes too many sessions from your write was rejected, and a message saying slots refill one at a time. The previous body's entire actionable content was "later".
Retry-After from the bucket, rounded up.

scope separates three diagnoses that were one value: bearer (one token reconnecting), subject (one identity across rotated tokens — the shape #21 describes), and bucket_capacity, which is not a quota refusal at all.

Verified by mutation, including one that found a useless test

mutation	died
subject refusal reported as bearer	`a_refusal_names_the_subject_bucket`
bearer refusal reported as subject	`a_refusal_names_the_bearer_bucket`
scope logs the wrong string	`refusal_scopes_have_distinct_log_values`
`retry_after_secs` rounds down	`retry_after_seconds_rounds_up`
wait re-derived from `quota.replenish_interval()`	`a_quota_refusal_carries_a_wait_within_the_refill_period`

The last row survived on the first attempt. The test asserted wait <= 60s, and the quota's replenish interval is exactly 60s, so it accepted the substitution it existed to forbid. Tightened to wait < 60s: time passes between spending the cell and asking, and only the bucket knows it. A test that permits the mutation it was written against is the same object as a green that carries no information, one level down.

Gates

cargo fmt --check, cargo clippy -D warnings, cargo test --all-features (116 + 9), cargo audit, cargo deny check bans licenses sources — green on rustc 1.98.0, through the shared build slot.

After merge

chore(release): prepare v0.2.0, tag, image, Renovate, pod digest. Production has not moved in forty hours and runs sha256:3d37a2eb; the digest goes to Clark and Alan when it lands.

Closes #20. ## What is true, and what is not **Not claimed:** that we observed this limiter refusing Julian. We did not. The 429 seen from the client reads `{"message":"Anthropic Proxy: Invalid content from server"}`, which names the connector's proxy as the speaker and is equally consistent with its own rate limiting. Lucy was not a control either: her `carddav` mount answers `whoami` as `julian@kampong.social`, the same identity, and this limiter keys on the Logto `sub`. **What is claimed, all of it a fact about this code:** - Eight failed attempts, and slots refill **one at a time**, so recovering from empty took four hours rather than the thirty minutes a reader would infer. - The refusal carried nothing actionable: no log line, no `Retry-After`, no error class. - The retry-after was **computed by `governor` and discarded at the boundary** — `NotUntil::wait_time_from` existed all along and `check` mapped it to a unit struct. - `tool_calls_total` reads zero during exactly this failure, because the limiter is Axum middleware **outside** the MCP router and a refusal never reaches the code that increments it. In the Infrastructure lead's words, from the investigation: *"it measured a limiter that was not firing while the one that was firing recorded nothing."* None of that needs a 429 sighting. ## The refill is the change that matters burst 8 -> 24 refill 30 min -> 1 min recovery from empty: 4 hours -> 24 minutes At one slot per thirty minutes, someone who waits out what they believe is the window gets **exactly one session**, reconnects twice and is refused again. A wrong model of a limiter does not merely delay a user, it guarantees a second failure after the wait. That is why five attempts produced five identical failures. ## Why 24 and not 32, and why the pool did not move **This is the fallback, taken deliberately.** Burst 32 with `MAX_SESSIONS` at 256 takes the identities needed to exhaust the pool from 32 down to **8** — a per-user refusal converted into a global exhaustion, invisible in the diff, breaking somebody other than the person who triggered it. Moving the pool with it requires the per-session memory measurement, and measuring means opening 32 authenticated sessions against a running pod, which needs a bearer this repo cannot currently obtain: Stalwart's directory is OIDC, so no password authenticates for anybody, which is the same blocker as #19. So: **24 covers the observed workload exactly and moves the ratio to 10 identities knowingly.** 1024 with burst 32 is the measured version, once a bearer exists. ## Observability - `tracing::warn!` on the refusal branch with `scope`, `user_hash` and `retry_after_seconds`. `user_hash` is the domain-separated identity hash, which **reverses against a candidate list** and would have answered "whose refusal is this" in one grep. Verified by reproducing it: `identity:julian@kampong.social` gives `f5a076c6a6c82848`. - `caldav_mcp_initialize_refusals_total{scope}`. - A JSON body with a stable class, so a client distinguishes *too many sessions* from *your write was rejected*, and a message saying slots refill one at a time. The previous body's entire actionable content was "later". - `Retry-After` from the bucket, rounded **up**. `scope` separates three diagnoses that were one value: `bearer` (one token reconnecting), `subject` (one identity across rotated tokens — the shape #21 describes), and `bucket_capacity`, which is not a quota refusal at all. ## Verified by mutation, including one that found a useless test | mutation | died | |---|---| | subject refusal reported as bearer | `a_refusal_names_the_subject_bucket` | | bearer refusal reported as subject | `a_refusal_names_the_bearer_bucket` | | scope logs the wrong string | `refusal_scopes_have_distinct_log_values` | | `retry_after_secs` rounds down | `retry_after_seconds_rounds_up` | | wait re-derived from `quota.replenish_interval()` | `a_quota_refusal_carries_a_wait_within_the_refill_period` | **The last row survived on the first attempt.** The test asserted `wait <= 60s`, and the quota's replenish interval is exactly 60s, so it accepted the substitution it existed to forbid. Tightened to `wait < 60s`: time passes between spending the cell and asking, and only the bucket knows it. A test that permits the mutation it was written against is the same object as a green that carries no information, one level down. ## Gates `cargo fmt --check`, `cargo clippy -D warnings`, `cargo test --all-features` (116 + 9), `cargo audit`, `cargo deny check bans licenses sources` — green on `rustc 1.98.0`, through the shared build slot. ## After merge `chore(release): prepare v0.2.0`, tag, image, Renovate, pod digest. Production has not moved in forty hours and runs `sha256:3d37a2eb`; the digest goes to Clark and Alan when it lands.

jlxq0 added 1 commit

2026-08-26 14:21:26 +00:00

feat(mcp): make the initialize limiter observable, and refill in minutes

CI / cargo (pull_request) Successful in 51s

Details

CI / docker (pull_request) Successful in 59s

Details

9757bed9e7

Eight failed attempts produced no server-side record of any kind: no log line,
no Retry-After, no error class, and tool_calls_total reading zero because the
limiter is Axum middleware outside the MCP router, so a refusal never reaches
the code that increments it. A correct refusal that carries no information is
the same object as a green that carries none.

The refill is the change that matters. At one slot per thirty minutes, someone
who waits out what they believe is the window gets exactly one session,
reconnects twice and is refused again — a wrong model of the rule does not
merely delay them, it guarantees a second failure after the wait. One per
minute takes recovery from empty from four hours to twenty-four minutes.

Burst goes 8 -> 24 and MAX_SESSIONS is unchanged, which is the fallback rather
than the preferred shape. Burst 32 with the pool at 256 would take the
identities needed to exhaust it from 32 down to 8, converting a per-user
refusal into a global exhaustion; moving the pool with it needs a per-session
memory measurement, and measuring means opening 32 authenticated sessions,
which needs a bearer this repo cannot currently obtain. 24 covers the observed
workload exactly and moves the ratio to 10 knowingly.

Retry-After comes from the bucket's own NotUntil, never from the configured
period. The two agree only until someone changes the quota, and this commit
changes it. Seconds round up: rounding down hands the client a moment at which
the slot still does not exist.

The 429 now carries a JSON body with a stable error class, so a client can tell
"too many sessions" from "your write was rejected", and the message says slots
refill one at a time — the previous body's entire actionable content was
"later", which is what produced the wrong model.

Also: a refusals counter labelled by scope, and the burst and refill as env
vars, because a number nobody can tune needed a rebuild to answer a user.

Refs #20

jlxq0 merged commit 74d6101284 into main

2026-08-26 14:25:04 +00:00

jlxq0 deleted branch feat-limiter-observable

2026-08-26 14:25:05 +00:00

jlxq0 referenced this pull request from a commit

2026-08-26 14:25:05 +00:00

Merge pull request 'feat(mcp): make the initialize limiter observable, and refill in minutes' (#23) from feat-limiter-observable into main