jlxq0/matrix-mcp

Fork 0

Recovery borrows the bearer of whichever claude.ai account spoke last, and the fleet flips accounts by design #143

New issue

Open

opened 2026-08-28 02:51:34 +00:00 by jlxq0 · 8 comments

jlxq0 commented

2026-08-28 02:51:34 +00:00

Owner

/setup/recover borrows the bearer of whichever claude.ai grant spoke to this server most recently. The fleet flips between two accounts by design, so that bearer belongs to the other account roughly half the time, and the recovery import fails with a 401 the page then attributed to the user's recovery key.

Nothing is blocked on Julian. He has done everything that could be asked; this is ours.

The mechanism

MatrixClientCache is keyed on the Matrix user alone: guard.get(&identity.mxid), one slot per mxid.

His private claude.ai account and the Hanso team account are two OAuth grants with two device-bound bearers, authenticating to this server as the same Matrix user. They share that one slot, and refresh_token_if_needed(&cached, identity, access_token, incoming_hash) swaps whichever arrived most recently onto the shared client. Last writer wins, and the loser's bearer is what /setup/recover may find when it calls get_if_cached.

This is why his report and the diagnosis were both true at once: each connector works flawlessly while it is the one speaking. He said so, and it is not a contradiction of the 401.

bin/claude-account.sh exists precisely so the fleet can move between subscriptions without a browser (save, use, who, list). A recovery path scoped to whichever account is currently active cannot be stable here, and it fails silently and blames the user.

Three things established from the code

A pod restart does not carry a stale bearer. The cache is Arc<RwLock<HashMap<String, CachedClientCell>>>, in memory. The SQLite stores persist crypto and room state and never a session token. for_user rebuilds from the bearer on the next request. So a connector re-add is not needed after a restart, and asking for one would have been the wrong instruction.

It slipped past rather than being chosen. ed88106, "fix(setup): stop panicking on /setup/recover when client is cached (#69)", says borrowing the bearer was the way out of a matrix-sdk 0.17 panic: restore_session on an already-initialised client raised AlreadyInitializedError and took the pod down. Grepping the source and docs/ for account switching finds nothing. Nobody weighed this against the account flipping because nobody knew to.

The cache refreshes on every MCP request, which is why the state looks healthy from every angle: tool calls work, the connector decrypts, and only the borrowing path sees the other grant's token.

What is not established, and it changes what he saw

Whether flipping revokes the other grant's bearer at MAS, or merely leaves it unused. Dead token, or live token scoped to a different device. It does not change the fix — one slot for two grants is wrong either way — but it decides which failure he actually hit.

This is a MAS records question rather than a server one, and needs cluster access rather than code here.

The measurement to take before writing anything

Does a client built with /setup's own session yield something device-scoped enough for the import?

The whole fix turns on it. The panic that forced the borrow was restore_session onto an initialised client, and eviction is the route the cache already uses for stale entries, so evict-then-build has the right shape. But /setup's grant is unbound, and recovery().recover() needs a device-scoped session, which is exactly why the borrow was reached for. If an evicted-and-rebuilt client is not device-scoped enough, this is not the route and something else is.

Establish that first. Writing the eviction and discovering it afterwards costs a release and another failed attempt by the person this has already cost three.

What the merged fixes did and did not do

#141 stopped the error attributing every failure to the last thing typed. #142 made the gate ask whether the bearer is alive rather than whether one is cached, and refuse before the key form.

They make the failure legible. They do not make it stop. Neither touches the dependency, and a legible failure on a path that still cannot work is not a fix.

Withdrawn, on the record

"Remove and re-add the claude.ai Matrix connector" was issued and then withdrawn, and should not be reissued. It is not needed after a restart, and it does not address a slot shared between two grants: the next flip puts the other account's bearer back. The instruction is still correct for a genuinely revoked connector, which is a different state and the one #142's page names.

#141, #142: the two legibility fixes.
The store layout comment in matrix_client.rs records that one stable matrix-sdk store is deliberately preserved for the Matrix device across token refreshes, which is the property the mxid keying was built for. Sharing across grants is the part nobody intended.

`/setup/recover` borrows the bearer of whichever claude.ai grant spoke to this server most recently. The fleet flips between two accounts by design, so that bearer belongs to the other account roughly half the time, and the recovery import fails with a 401 the page then attributed to the user's recovery key. **Nothing is blocked on Julian.** He has done everything that could be asked; this is ours. ## The mechanism `MatrixClientCache` is keyed on the Matrix user alone: `guard.get(&identity.mxid)`, one slot per mxid. His private claude.ai account and the Hanso team account are **two OAuth grants with two device-bound bearers, authenticating to this server as the same Matrix user**. They share that one slot, and `refresh_token_if_needed(&cached, identity, access_token, incoming_hash)` swaps whichever arrived most recently onto the shared client. **Last writer wins**, and the loser's bearer is what `/setup/recover` may find when it calls `get_if_cached`. This is why his report and the diagnosis were both true at once: **each connector works flawlessly while it is the one speaking.** He said so, and it is not a contradiction of the 401. `bin/claude-account.sh` exists precisely so the fleet can move between subscriptions without a browser (`save`, `use`, `who`, `list`). **A recovery path scoped to whichever account is currently active cannot be stable here**, and it fails silently and blames the user. ## Three things established from the code **A pod restart does not carry a stale bearer.** The cache is `Arc<RwLock<HashMap<String, CachedClientCell>>>`, in memory. The SQLite stores persist crypto and room state and **never a session token**. `for_user` rebuilds from the bearer on the next request. So a connector re-add is not needed after a restart, and asking for one would have been the wrong instruction. **It slipped past rather than being chosen.** `ed88106`, *"fix(setup): stop panicking on /setup/recover when client is cached (#69)"*, says borrowing the bearer was the way out of a matrix-sdk 0.17 panic: `restore_session` on an already-initialised client raised `AlreadyInitializedError` and took the pod down. Grepping the source and `docs/` for account switching finds nothing. **Nobody weighed this against the account flipping because nobody knew to.** **The cache refreshes on every MCP request**, which is why the state looks healthy from every angle: tool calls work, the connector decrypts, and only the borrowing path sees the other grant's token. ## What is not established, and it changes what he saw **Whether flipping revokes the other grant's bearer at MAS, or merely leaves it unused.** Dead token, or live token scoped to a different device. **It does not change the fix** — one slot for two grants is wrong either way — but it decides which failure he actually hit. **This is a MAS records question rather than a server one**, and needs cluster access rather than code here. ## The measurement to take before writing anything **Does a client built with `/setup`'s own session yield something device-scoped enough for the import?** The whole fix turns on it. The panic that forced the borrow was `restore_session` onto an initialised client, and eviction is the route the cache **already** uses for stale entries, so evict-then-build has the right shape. But `/setup`'s grant is unbound, and `recovery().recover()` needs a device-scoped session, which is exactly why the borrow was reached for. **If an evicted-and-rebuilt client is not device-scoped enough, this is not the route and something else is.** Establish that first. Writing the eviction and discovering it afterwards costs a release and another failed attempt by the person this has already cost three. ## What the merged fixes did and did not do **#141** stopped the error attributing every failure to the last thing typed. **#142** made the gate ask whether the bearer is alive rather than whether one is cached, and refuse **before** the key form. **They make the failure legible. They do not make it stop.** Neither touches the dependency, and a legible failure on a path that still cannot work is not a fix. ## Withdrawn, on the record **"Remove and re-add the claude.ai Matrix connector" was issued and then withdrawn, and should not be reissued.** It is not needed after a restart, and it does not address a slot shared between two grants: the next flip puts the other account's bearer back. The instruction is still correct for a genuinely revoked connector, which is a different state and the one `#142`'s page names. ## Related - #141, #142: the two legibility fixes. - The store layout comment in `matrix_client.rs` records that one stable matrix-sdk store is deliberately preserved for the Matrix device across token refreshes, which is the property the mxid keying was built for. **Sharing across grants is the part nobody intended.**

jlxq0 commented

2026-08-29 00:44:26 +00:00

Author

Owner

Measured on the running pod: two grants alternating, not one refreshing

The two outstanding checks are MAS and Synapse records: does MAS hold two live sessions on this device, and has Synapse's ed25519 changed under it. Neither shows two grants alternating over time on one Matrix user, which is the mechanism rather than the state, and the pod can see that.

@julian:kampong.social            8 distinct token hashes in 24h
each of the eight bot identities  1 distinct token hash

9b1a46ca   first  2026-08-28T23:36:02Z
02b7161c   then
9b1a46ca   again  2026-08-29T00:29:21Z

A bearer that returns after a different one held the slot is not refresh. Refresh is monotonic: a rotated token never comes back. A hash reappearing after another was in use is two grants taking turns, which is the account flipping observed rather than inferred from bin/claude-account.sh existing.

The per-bot control is what makes the eight readable. One identity, one grant, no recurrence, across all eight. So the asymmetry is about this user rather than about token lifetime, and a reader cannot dismiss the eight as ordinary rotation.

Both grants bind to MATRIXMCP-DLX8SKHK, which is the only device id this deployment advertises, read from the live endpoint:

scopes_supported: … urn:matrix:org.matrix.msc2967.client:device:MATRIXMCP-DLX8SKHK

So this is the state setup.rs's comment rejects for /setup — two OAuth sessions bound to one Matrix device, whose /keys/upload calls fight over Synapse's record — reached by a route nobody considered, because the comment reasoned about /setup being the second session and not about a second connector.

The first version of this check was broken and returned the reassuring answer

It printed "no hash recurs: monotonic, consistent with refresh rather than flipping" while the recurrence was visible in the sequence it had just printed. The awk tested whether the loop index was in the seen-set rather than the hash, so it could never fire.

Caught by reading the output rather than the verdict. A check that cannot return the positive, giving the answer that ends the investigation, is the same shape as the guard whose mutation stayed green and the log line that reports a 2xx as a verified device. The corrected run has a control beside it for that reason.

What this settles and what it does not

Settled: two grants are live and alternating on one Matrix user, and both bind to one advertised device id.

Not settled: whether Synapse's ed25519 for that device has actually changed under them, and whether MAS holds both sessions concurrently rather than sequentially. Those remain the two record checks, and this narrows what they are looking for rather than replacing them.

No code has been written, and #130 and #139 are not started, in case they share this cause.

## Measured on the running pod: two grants alternating, not one refreshing The two outstanding checks are MAS and Synapse records: does MAS hold two live sessions on this device, and has Synapse's ed25519 changed under it. **Neither shows two grants alternating over time on one Matrix user**, which is the mechanism rather than the state, and the pod can see that. @julian:kampong.social 8 distinct token hashes in 24h each of the eight bot identities 1 distinct token hash 9b1a46ca first 2026-08-28T23:36:02Z 02b7161c then 9b1a46ca again 2026-08-29T00:29:21Z **A bearer that returns after a different one held the slot is not refresh.** Refresh is monotonic: a rotated token never comes back. A hash reappearing after another was in use is two grants taking turns, which is the account flipping observed rather than inferred from `bin/claude-account.sh` existing. **The per-bot control is what makes the eight readable.** One identity, one grant, no recurrence, across all eight. So the asymmetry is about this user rather than about token lifetime, and a reader cannot dismiss the eight as ordinary rotation. Both grants bind to `MATRIXMCP-DLX8SKHK`, which is the only device id this deployment advertises, read from the live endpoint: scopes_supported: … urn:matrix:org.matrix.msc2967.client:device:MATRIXMCP-DLX8SKHK So this is the state `setup.rs`'s comment rejects for `/setup` — two OAuth sessions bound to one Matrix device, whose `/keys/upload` calls fight over Synapse's record — reached by a route nobody considered, because the comment reasoned about `/setup` being the second session and not about a second connector. ## The first version of this check was broken and returned the reassuring answer It printed *"no hash recurs: monotonic, consistent with refresh rather than flipping"* **while the recurrence was visible in the sequence it had just printed**. The awk tested whether the loop **index** was in the seen-set rather than the **hash**, so it could never fire. Caught by reading the output rather than the verdict. **A check that cannot return the positive, giving the answer that ends the investigation**, is the same shape as the guard whose mutation stayed green and the log line that reports a 2xx as a verified device. The corrected run has a control beside it for that reason. ## What this settles and what it does not **Settled**: two grants are live and alternating on one Matrix user, and both bind to one advertised device id. **Not settled**: whether Synapse's ed25519 for that device has actually changed under them, and whether MAS holds both sessions concurrently rather than sequentially. **Those remain the two record checks**, and this narrows what they are looking for rather than replacing them. No code has been written, and #130 and #139 are not started, in case they share this cause.

jlxq0 commented

2026-08-29 06:02:34 +00:00

Author

Owner

Mapping a bearer to a MAS session: not possible today, and the field that would make it possible

The question raised by Clark's records is which of the seven live MAS sessions on MATRIXMCP-DLX8SKHK issued the bearers this pod actually sees, because that decides which are dead weight and safe to revoke.

It cannot be answered from this server, and not because of a query nobody thought of.

IntrospectionResponse in src/mas.rs:44 deserialises active, sub, username, scope, aud, exp, device_id. Nothing identifies a session, and sub is the user, identical across all seven, as its own comment says.

Nor does MAS emit one. Reading crates/handlers/src/oauth2/introspection.rs on main, the response carries active, scope, client_id, username, token_type, exp, expires_in, iat, nbf, sub, aud, iss, jti, device_id, and no session id field of any kind. So no client can make this mapping from introspection.

jti is the join and we drop it. MAS sets jti: Some(access_token.jti()) for OAuth2 access tokens (and for refresh tokens), and None for compatibility and personal access tokens. A jti is a stable per-token identifier, so logging it beside the existing token_hash lets a MAS-side query join token to session and answer the question permanently, rather than once by hand.

That is one field on the struct and one field on the audit line here, plus one query on the MAS side. Cheaper than the hand-correlation it replaces, and it survives the next occurrence.

Bounded honestly: this is main of matrix-authentication-service, not necessarily the deployed version. The fields to confirm against the running MAS are jti present and session id absent.

One thing the same file settles

device_id in the introspection response is populated only for compatibility tokens — device_id: session.device.map(Device::into) — and is None for OAuth2 tokens. So for the claude.ai connector's bearers that field is always empty, and the device binding is read from the scope instead, which is what mas.rs:464 already does. The struct field is not wrong; it is simply never the source for the tokens that matter here.

What the pod can say about the seven

Across this pod's whole lifetime, @julian presented four distinct bearers in five runs, one recurring:

03:59:42Z  4ffbbd61
04:36:34Z  8a151a62
05:13:46Z  5544e38b
05:29:52Z  8a151a62   again, after 5544e38b held the slot
05:36:40Z  de3be047

At least two sessions are issuing traffic and at most four. So three of the seven have sent nothing here in the pod's lifetime, which narrows the revocation candidates without identifying them.

exp is fetched and never logged. Logging it alongside jti would separate lineages by lifetime as a second, independent signal.

## Mapping a bearer to a MAS session: not possible today, and the field that would make it possible The question raised by Clark's records is which of the seven live MAS sessions on `MATRIXMCP-DLX8SKHK` issued the bearers this pod actually sees, because that decides which are dead weight and safe to revoke. **It cannot be answered from this server, and not because of a query nobody thought of.** `IntrospectionResponse` in `src/mas.rs:44` deserialises `active`, `sub`, `username`, `scope`, `aud`, `exp`, `device_id`. **Nothing identifies a session**, and `sub` is the *user*, identical across all seven, as its own comment says. **Nor does MAS emit one.** Reading `crates/handlers/src/oauth2/introspection.rs` on `main`, the response carries `active`, `scope`, `client_id`, `username`, `token_type`, `exp`, `expires_in`, `iat`, `nbf`, `sub`, `aud`, `iss`, `jti`, `device_id`, and **no session id field of any kind**. So no client can make this mapping from introspection. **`jti` is the join and we drop it.** MAS sets `jti: Some(access_token.jti())` for OAuth2 access tokens (and for refresh tokens), and `None` for compatibility and personal access tokens. A `jti` is a stable per-token identifier, so logging it beside the existing `token_hash` lets a MAS-side query join token to session and answer the question permanently, rather than once by hand. That is one field on the struct and one field on the audit line here, plus one query on the MAS side. **Cheaper than the hand-correlation it replaces, and it survives the next occurrence.** **Bounded honestly**: this is `main` of `matrix-authentication-service`, not necessarily the deployed version. The fields to confirm against the running MAS are `jti` present and session id absent. ## One thing the same file settles **`device_id` in the introspection response is populated only for *compatibility* tokens** — `device_id: session.device.map(Device::into)` — and is `None` for OAuth2 tokens. So for the claude.ai connector's bearers that field is always empty, and the device binding is read from the **scope** instead, which is what `mas.rs:464` already does. The struct field is not wrong; it is simply never the source for the tokens that matter here. ## What the pod can say about the seven Across this pod's whole lifetime, `@julian` presented **four distinct bearers in five runs, one recurring**: 03:59:42Z 4ffbbd61 04:36:34Z 8a151a62 05:13:46Z 5544e38b 05:29:52Z 8a151a62 again, after 5544e38b held the slot 05:36:40Z de3be047 **At least two sessions are issuing traffic and at most four.** So three of the seven have sent nothing here in the pod's lifetime, which narrows the revocation candidates without identifying them. `exp` is fetched and never logged. Logging it alongside `jti` would separate lineages by lifetime as a second, independent signal.

jlxq0 commented

2026-08-29 06:02:58 +00:00

Author

Owner

Constraint on any per-grant device id, before anybody designs one

The root fix under discussion is that the Matrix device id must not be shared across grants. Whatever replaces the current scheme has to keep the property the current scheme exists for, and that property is not obvious from the symptom.

src/device_identity.rs ties the device id to the PVC deliberately, and says why:

Earlier iterations hard-coded the device id as a compile-time constant (MATRIXMCPCONNECTOR, then MATRIXMCP2). That has a sharp edge: if the PVC ever gets wiped, the SDK regenerates its ed25519 device keys but Synapse still remembers the old keys under the same device id. Every subsequent /keys/upload is rejected by Synapse with M_FORBIDDEN / SigningKeyChanged, and the only recovery is to ship a code change rotating the constant to a fresh id.

So the id and the crypto store must live and die together. A per-grant id that does not preserve that trades a contention bug for the SigningKeyChanged deadlock the current design was written to escape, and the deadlock is worse: it is unrecoverable without a release.

That is the shape of every failed repair here this week: the fix carrying the fault it repairs. It is on the record before the design rather than after it.

One cause, not three

#143, #130 and #139 are consistent with a single mechanism and should be treated as one until the code contradicts it.

seven live MAS sessions on MATRIXMCP-DLX8SKHK, finished_at IS NULL, back to 29 July
Synapse for that device:                      has_keys = 0   sigs = 0
control MATRIXMCP-KZ9M8ZY8, one session:      1 key          1 signature

Keyed and signed where one session held the device; keyless and unsigned where seven contend. Signing a device seven sessions overwrite is signing sand, so every route that ends in an upload is downstream of this.

#130 is the same fact from inside: Device::verify discards the failures map returned inside a 200, and the follow-up /keys/query that exists to check the signature attached is discarded too, so the SDK logs "Successfully signed our own device, the device is now verified" about an upload into nothing. @mantis_ai_bot produced that line and is unsigned in Clark's table.

#139 is the consequence: four of eight bot devices unsigned, and no session able to ask about itself.

Nothing in this repository signs or uploads anything that would behave differently under seven sessions than under one, so the code does not disagree.

## Constraint on any per-grant device id, before anybody designs one The root fix under discussion is that the Matrix device id must not be shared across grants. **Whatever replaces the current scheme has to keep the property the current scheme exists for**, and that property is not obvious from the symptom. `src/device_identity.rs` ties the device id to the PVC **deliberately**, and says why: > Earlier iterations hard-coded the device id as a compile-time constant (`MATRIXMCPCONNECTOR`, then `MATRIXMCP2`). That has a sharp edge: **if the PVC ever gets wiped, the SDK regenerates its ed25519 device keys but Synapse still remembers the old keys under the same device id**. Every subsequent `/keys/upload` is rejected by Synapse with `M_FORBIDDEN / SigningKeyChanged`, and the only recovery is to ship a code change rotating the constant to a fresh id. **So the id and the crypto store must live and die together.** A per-grant id that does not preserve that trades a contention bug for the `SigningKeyChanged` deadlock the current design was written to escape, and the deadlock is worse: it is unrecoverable without a release. **That is the shape of every failed repair here this week: the fix carrying the fault it repairs.** It is on the record before the design rather than after it. ## One cause, not three `#143`, `#130` and `#139` are consistent with a single mechanism and should be treated as one until the code contradicts it. seven live MAS sessions on MATRIXMCP-DLX8SKHK, finished_at IS NULL, back to 29 July Synapse for that device: has_keys = 0 sigs = 0 control MATRIXMCP-KZ9M8ZY8, one session: 1 key 1 signature **Keyed and signed where one session held the device; keyless and unsigned where seven contend.** Signing a device seven sessions overwrite is signing sand, so every route that ends in an upload is downstream of this. **`#130` is the same fact from inside**: `Device::verify` discards the `failures` map returned inside a 200, and the follow-up `/keys/query` that exists to check the signature attached is discarded too, so the SDK logs *"Successfully signed our own device, the device is now verified"* **about an upload into nothing**. `@mantis_ai_bot` produced that line and is unsigned in Clark's table. **`#139` is the consequence**: four of eight bot devices unsigned, and no session able to ask about itself. Nothing in this repository signs or uploads anything that would behave differently under seven sessions than under one, so the code does not disagree.

jlxq0 referenced this issue

2026-08-29 06:03:21 +00:00

The SDK reports a device verified after discarding both results that could say otherwise #130

jlxq0 referenced this issue

2026-08-29 06:03:22 +00:00

No session can ask whether its own bot is cross-signed, and four of eight are not #139

jlxq0 commented

2026-08-29 06:06:10 +00:00

Author

Owner

The one-cause reading is under a question: the control may have dissolved

Do not build on the single-mechanism conclusion in the comment above until this resolves.

That conclusion rests on a contrast, not on a single number:

DLX8SKHK   seven live sessions    has_keys = 0   sigs = 0
KZ9M8ZY8   one settled session    1 key          1 signature

The control device has since been reported as 8 live. If it holds eight live sessions and a key and a signature, then contention does not prevent keys from sticking, and the argument loses the only thing supporting it.

It may be two predicates rather than two answers, since settled and live need not count the same thing, and which query produced each is being established rather than guessed at.

What stands regardless: has_keys = 0, sigs = 0 on DLX8SKHK is a direct measurement of the device this issue is about. What is in doubt is the inference from it — that seven contending sessions are the reason — which is the part the one-cause statement, and its echo on the sibling issues, were built on.

Recording the doubt rather than retracting, because a retraction should carry the resolving measurement rather than the uncertainty. This line exists so that nobody reads the paragraph above it as settled in the meantime.

One thing that is settled and does not depend on any of it: the bearer-to-session mapping exists in MAS's database as oauth2_access_tokens.oauth2_session_id, so telling the seven apart needs no change to this server and no jti field. The runtime jti would be a convenience and nothing currently turns on it.

## The one-cause reading is under a question: the control may have dissolved **Do not build on the single-mechanism conclusion in the comment above until this resolves.** That conclusion rests on a contrast, not on a single number: DLX8SKHK seven live sessions has_keys = 0 sigs = 0 KZ9M8ZY8 one settled session 1 key 1 signature **The control device has since been reported as `8 live`.** If it holds eight live sessions *and* a key *and* a signature, then contention does not prevent keys from sticking, and the argument loses the only thing supporting it. It may be two predicates rather than two answers, since *settled* and *live* need not count the same thing, and which query produced each is being established rather than guessed at. **What stands regardless**: `has_keys = 0, sigs = 0` on `DLX8SKHK` is a direct measurement of the device this issue is about. **What is in doubt is the inference from it** — that seven contending sessions are the reason — which is the part the one-cause statement, and its echo on the sibling issues, were built on. **Recording the doubt rather than retracting**, because a retraction should carry the resolving measurement rather than the uncertainty. This line exists so that nobody reads the paragraph above it as settled in the meantime. **One thing that is settled and does not depend on any of it**: the bearer-to-session mapping exists in MAS's database as `oauth2_access_tokens.oauth2_session_id`, so telling the seven apart needs no change to this server and no `jti` field. The runtime `jti` would be a convenience and nothing currently turns on it.

jlxq0 commented

2026-08-29 06:07:06 +00:00

Author

Owner

Provenance of the one-cause statement

For whoever reads this thread later and needs to know where the confidence entered rather than only that it is in doubt.

Clark measured the two rows. Alan read them and instructed the conclusion, in writing, twice: "it is one cause rather than three… treat them as one and say so if the code disagrees", and "say the one-cause thing on the issues yourself — three readers should not each infer it." I wrote it onto three issues within the hour, adding the code reading that nothing here behaves differently under seven sessions than one, which is true and is not the same claim.

Neither of us asked which query produced the control row. The standard was available and had been applied that same hour to a different instrument, where a set of 401/403 counts were correctly marked as keyword matches to be replaced rather than confirmed. The rule was applied to our own numbers and not to a table handed to us.

Recorded because a claim under a question is easier to weigh when a reader can see whether it came from a measurement, an inference, or an instruction.

### Provenance of the one-cause statement For whoever reads this thread later and needs to know where the confidence entered rather than only that it is in doubt. Clark measured the two rows. **Alan read them and instructed the conclusion**, in writing, twice: *"it is one cause rather than three… treat them as one and say so if the code disagrees"*, and *"say the one-cause thing on the issues yourself — three readers should not each infer it."* **I wrote it onto three issues within the hour**, adding the code reading that nothing here behaves differently under seven sessions than one, which is true and is not the same claim. **Neither of us asked which query produced the control row.** The standard was available and had been applied that same hour to a different instrument, where a set of `401`/`403` counts were correctly marked as keyword matches to be replaced rather than confirmed. **The rule was applied to our own numbers and not to a table handed to us.** Recorded because a claim under a question is easier to weigh when a reader can see whether it came from a measurement, an inference, or an instruction.

jlxq0 commented

2026-08-29 06:09:03 +00:00

Author

Owner

Retracted: contention by session count is not the mechanism

The resolving measurement, both rows from the same query with the same predicate:

MATRIXMCP-KZ9M8ZY8   8 live sessions   has_keys 1   sigs 1
MATRIXMCP-DLX8SKHK   7 live sessions   has_keys 0   sigs 0

The device with more sessions is the signed one. The earlier "one settled session" was a mislabel: Synapse's sigs = 1 was read as a session count and written as one. The counts never moved; the noun was wrong.

So every inference from that contrast is withdrawn, including the one-cause statement placed on #143, #130 and #139, and the claim that seven contending sessions are why the record is empty. This replaces it rather than sitting beside it.

What stands, and always did: has_keys = 0, sigs = 0 on MATRIXMCP-DLX8SKHK is a direct measurement of that device. The why has gone, not the what.

Nothing here should be folded into a replacement hypothesis yet. One is on the table, that KZ9M8ZY8 has been quiescent for eight weeks while DLX8SKHK is still accumulating sessions, but that is a single variable chosen after seeing the outcome on two devices, which is the shape that fits perfectly and explains the wrong thing. The cheaper discriminator is whether DLX8SKHK was ever quiescent and still keyless, which would kill it without spending anybody's time.

How the retracted claim got here is on #143 under Provenance of the one-cause statement, so a later reader can see it arrived as an instruction on an unmeasured contrast rather than as a measurement.

## Retracted: contention by session count is not the mechanism The resolving measurement, both rows from the same query with the same predicate: MATRIXMCP-KZ9M8ZY8 8 live sessions has_keys 1 sigs 1 MATRIXMCP-DLX8SKHK 7 live sessions has_keys 0 sigs 0 **The device with *more* sessions is the signed one.** The earlier "one settled session" was a mislabel: Synapse's `sigs = 1` was read as a session count and written as *one*. The counts never moved; the noun was wrong. **So every inference from that contrast is withdrawn**, including the one-cause statement placed on #143, #130 and #139, and the claim that seven contending sessions are why the record is empty. **This replaces it rather than sitting beside it.** **What stands, and always did**: `has_keys = 0, sigs = 0` on `MATRIXMCP-DLX8SKHK` is a direct measurement of that device. **The *why* has gone, not the *what*.** **Nothing here should be folded into a replacement hypothesis yet.** One is on the table, that `KZ9M8ZY8` has been quiescent for eight weeks while `DLX8SKHK` is still accumulating sessions, but that is a single variable chosen after seeing the outcome on two devices, which is the shape that fits perfectly and explains the wrong thing. The cheaper discriminator is whether `DLX8SKHK` was ever quiescent and still keyless, which would kill it without spending anybody's time. **How the retracted claim got here is on #143** under *Provenance of the one-cause statement*, so a later reader can see it arrived as an instruction on an unmeasured contrast rather than as a measurement.

jlxq0 referenced this issue

2026-08-29 06:09:03 +00:00

No session can ask whether its own bot is cross-signed, and four of eight are not #139

jlxq0 referenced this issue

2026-08-29 06:09:04 +00:00

The SDK reports a device verified after discarding both results that could say otherwise #130

jlxq0 commented

2026-08-29 06:11:44 +00:00

Author

Owner

Churn-recency is dead too, and the state to leave here is a measurement with no mechanism

2026-08-01 → 08-06   DLX8SKHK had ZERO live sessions for five days
2026-08-06 → 08-15   exactly ONE live session for nine days
now                  has_keys = 0, sigs = 0

A lone session for nine days did not leave a key. So neither session count nor recency explains it: eight-session KZ9M8ZY8 is signed, and DLX8SKHK is keyless through two quiescent windows. Withdrawn, like contention-by-count before it.

The current-state reading is load-bearing rather than a gap: device keys live in e2e_device_keys_json keyed on (user, device), persist across sessions, and go when the device is deleted. The same device id spans all eight sessions including one that finished on 08-01 and is still in devices, so a key uploaded during the lone-session window would still be there. One device_lists_stream row and zero device_keys_json rows corroborate: one announcement, no key lifecycle.

has_keys = 0, sigs = 0 has now survived every explanation offered for it. That is the state to leave on this issue: a measurement with no mechanism, rather than a mechanism.

So do not build a contention fix, and the PVC constraint recorded above is a constraint on a design nobody should now be starting.

Where the upload is actually gated, and a candidate that is already filed

matrix-sdk-crypto-0.17.0/src/olm/account.rs:666:

let device_keys = self.shared().not().then(|| self.device_keys());

Device keys are offered for upload only while shared is false. Once that flag latches true, keys_for_upload returns None for device keys forever, and every later /keys/upload carries one-time keys only.

That is the exact mechanism #88 describes, open since 2026-05-21 and mergeable: the per-account shared flag latches on the first apparently-successful upload and is never re-verified against the homeserver, so a client operates against a device that does not exist server-side. Its repro was a device with 50 one-time keys and zero rows in e2e_device_keys_json.

The discriminator is one query and it is cheap: does @julian's account hold one-time keys for DLX8SKHK while holding no device keys? If yes, the state matches #88's signature exactly and #88 becomes a candidate fix rather than an adjacent issue. If no, it is something else again.

Offered as a candidate with a test, not as the answer. Two hypotheses have already died here and this one has the same shape as both: it fits, and fitting is what the last two did.

And one bound on my own contribution. #88's canonical SDK line, "Our own device might have been deleted", appears zero times in this pod's entire log. That is not evidence. There is no matrix-sdk client for @julian on this pod at all: sixteen introspects, zero tool spans, no client-lifecycle line, measured earlier in this thread. A line that only a running client can emit, absent where no client runs, says nothing. The control is that the same grep returns 39 for a string that is there.

Earlier in this thread I said #88 was not the fix. That was about the cross-signing warning and it remains correct for it: publication and signing are different layers. This is a publication measurement, which is #88's own subject.

## Churn-recency is dead too, and the state to leave here is a measurement with no mechanism 2026-08-01 → 08-06 DLX8SKHK had ZERO live sessions for five days 2026-08-06 → 08-15 exactly ONE live session for nine days now has_keys = 0, sigs = 0 **A lone session for nine days did not leave a key.** So neither session count nor recency explains it: eight-session `KZ9M8ZY8` is signed, and `DLX8SKHK` is keyless through two quiescent windows. **Withdrawn, like contention-by-count before it.** The current-state reading is load-bearing rather than a gap: device keys live in `e2e_device_keys_json` keyed on `(user, device)`, persist across sessions, and go when the **device** is deleted. The same device id spans all eight sessions including one that finished on 08-01 and is still in `devices`, **so a key uploaded during the lone-session window would still be there.** One `device_lists_stream` row and zero `device_keys_json` rows corroborate: one announcement, no key lifecycle. **`has_keys = 0, sigs = 0` has now survived every explanation offered for it.** That is the state to leave on this issue: a measurement with no mechanism, rather than a mechanism. **So do not build a contention fix**, and the PVC constraint recorded above is a constraint on a design nobody should now be starting. ## Where the upload is actually gated, and a candidate that is already filed `matrix-sdk-crypto-0.17.0/src/olm/account.rs:666`: let device_keys = self.shared().not().then(|| self.device_keys()); **Device keys are offered for upload only while `shared` is false.** Once that flag latches true, `keys_for_upload` returns `None` for device keys **forever**, and every later `/keys/upload` carries one-time keys only. **That is the exact mechanism #88 describes**, open since 2026-05-21 and mergeable: the per-account `shared` flag latches on the first apparently-successful upload and is never re-verified against the homeserver, so a client operates against a device that does not exist server-side. Its repro was a device with **50 one-time keys and zero rows in `e2e_device_keys_json`**. **The discriminator is one query and it is cheap**: does `@julian`'s account hold one-time keys for `DLX8SKHK` while holding no device keys? **If yes, the state matches #88's signature exactly** and #88 becomes a candidate fix rather than an adjacent issue. If no, it is something else again. **Offered as a candidate with a test, not as the answer.** Two hypotheses have already died here and this one has the same shape as both: it fits, and fitting is what the last two did. **And one bound on my own contribution.** #88's canonical SDK line, *"Our own device might have been deleted"*, appears **zero** times in this pod's entire log. **That is not evidence.** There is no matrix-sdk client for `@julian` on this pod at all: sixteen introspects, zero tool spans, no client-lifecycle line, measured earlier in this thread. **A line that only a running client can emit, absent where no client runs, says nothing.** The control is that the same grep returns 39 for a string that is there. Earlier in this thread I said #88 was not the fix. **That was about the cross-signing warning and it remains correct for it**: publication and signing are different layers. This is a publication measurement, which is #88's own subject.

jlxq0 commented

2026-08-29 12:30:44 +00:00

Author

Owner

This cannot detect the state it exists for

Read against the current measurement: MATRIXMCP-DLX8SKHK has e2e_device_keys_json rows = 0 on Synapse today, after a successful /setup/recover, which is this PR's target state.

The heal is right and the trigger cannot fire.

verify_device_keys_published reduces to:

let devices = client.encryption().get_user_devices(&user_id).await?;
Ok(devices.get(device_id).is_some())

Encryption::get_user_devices calls OlmMachine::get_user_devices, which is self.store().get_user_devices(user_id) — the local crypto store, with no homeserver round trip. wait_if_user_pending waits for an in-flight query and does not start one.

And Store::get_user_devices carries this note in the SDK's own source:

Note

: This method will include our own device which is always present in the store.

The memory store's get_own_device states the same invariant as an assertion: .expect("Invalid state: Should always have a own device").

So for our own user and our own device, the predicate is invariantly true. It returns Ok(true) on a device Synapse has never held a key for, keys_verified latches, and the wipe-and-rebuild never runs. The check asks the local store whether we know about ourselves, which we always do.

That is why this could sit for three months looking correct: its negative branch is unreachable, so nothing it guards was ever exercised, and no test would catch it because the predicate needs a live homeserver disagreeing with a live store.

What a working predicate has to do

Ask the homeserver, then read. secret_store.rs:443-452 already shows the shape used elsewhere in the SDK:

let (request_id, request) = olm_machine.query_keys_for_users([olm_machine.user_id()]);
client.keys_query(&request_id, request.device_keys).await?;

Force the /keys/query, then check the store, and treat a device absent from the response as the negative. Reading the store without forcing the query answers a different question.

The rest of the PR stands and is the valuable half: evict, wipe the per-MXID store subdirectory, rebuild, recurse exactly once, error on a second negative. That resets shared to false so keys_for_upload offers device keys again, which is the actual repair for the latch at matrix-sdk-crypto-0.17.0/src/olm/account.rs:666.

The gap this leaves in the diagnosis

Fixing the trigger makes the heal reachable; it does not explain why the keys never uploaded in the first place. The latch gates device keys and not one-time keys, so a device can go on publishing OTKs forever while shared says the identity is done. What set shared on a device Synapse never keyed is still unmeasured, and a heal that fires on every rebuild without that answer will wipe a crypto store repeatedly rather than once.

Cost of the heal, which belongs beside the decision: wiping the per-MXID store discards olm and megolm state. The PR's own figure is ~12 s to restore 924 keys across 31 rooms from key backup, which is cheap when key backup is present and recoverable and is not free otherwise.

## This cannot detect the state it exists for Read against the current measurement: `MATRIXMCP-DLX8SKHK` has `e2e_device_keys_json rows = 0` on Synapse today, after a successful `/setup/recover`, which is this PR's target state. **The heal is right and the trigger cannot fire.** `verify_device_keys_published` reduces to: let devices = client.encryption().get_user_devices(&user_id).await?; Ok(devices.get(device_id).is_some()) `Encryption::get_user_devices` calls `OlmMachine::get_user_devices`, which is `self.store().get_user_devices(user_id)` — **the local crypto store, with no homeserver round trip.** `wait_if_user_pending` waits for an in-flight query and does not start one. And `Store::get_user_devices` carries this note in the SDK's own source: > *Note*: This method will include our own device which is always present in the store. The memory store's `get_own_device` states the same invariant as an assertion: `.expect("Invalid state: Should always have a own device")`. **So for our own user and our own device, the predicate is invariantly `true`.** It returns `Ok(true)` on a device Synapse has never held a key for, `keys_verified` latches, and the wipe-and-rebuild never runs. **The check asks the local store whether we know about ourselves, which we always do.** That is why this could sit for three months looking correct: **its negative branch is unreachable**, so nothing it guards was ever exercised, and no test would catch it because the predicate needs a live homeserver disagreeing with a live store. ## What a working predicate has to do **Ask the homeserver, then read.** `secret_store.rs:443-452` already shows the shape used elsewhere in the SDK: let (request_id, request) = olm_machine.query_keys_for_users([olm_machine.user_id()]); client.keys_query(&request_id, request.device_keys).await?; Force the `/keys/query`, then check the store, and treat a device absent from the **response** as the negative. Reading the store without forcing the query answers a different question. **The rest of the PR stands and is the valuable half**: evict, wipe the per-MXID store subdirectory, rebuild, recurse exactly once, error on a second negative. That resets `shared` to false so `keys_for_upload` offers device keys again, which is the actual repair for the latch at `matrix-sdk-crypto-0.17.0/src/olm/account.rs:666`. ## The gap this leaves in the diagnosis **Fixing the trigger makes the heal reachable; it does not explain why the keys never uploaded in the first place.** The latch gates device keys and not one-time keys, so a device can go on publishing OTKs forever while `shared` says the identity is done. **What set `shared` on a device Synapse never keyed is still unmeasured**, and a heal that fires on every rebuild without that answer will wipe a crypto store repeatedly rather than once. **Cost of the heal, which belongs beside the decision**: wiping the per-MXID store discards olm and megolm state. The PR's own figure is ~12 s to restore 924 keys across 31 rooms from key backup, which is cheap **when key backup is present and recoverable** and is not free otherwise.