A channel mount goes deaf on any upstream blip and nothing inside a session can tell #137

New issue

Open

opened 2026-08-27 15:10:14 +00:00 by jlxq0 · 1 comment

jlxq0 commented

2026-08-27 15:10:14 +00:00

Owner

A channel mount can stop receiving pushes on any upstream interruption and stay that way until the session restarts, and no observation from inside a session distinguishes a dead stream from a quiet room.

Measured

Two messages never delivered to one bot, both readable on the server:

15:28:09Z  "Oh excellent"                    never delivered, that session was mid-turn
15:42:52Z  "How are you watching all that?"  never delivered, that session was idle
15:45:29Z  read_recent_messages returns all 8, decrypted, in one call

One lost while busy and one lost fourteen minutes later while idle kills the busy explanation with a single sample. The fetch path works throughout; only the push is dead.

The cause is visible only from the edge journal, which no session can see. vultr-sg's Caddy stopped listening 15:12:16Z to 15:17:09Z while BGP kept announcing, and minutes before it:

14:58:14Z  GET matrix-mcp.kampong.social /channel
           Accept: text/event-stream   User-Agent: claude-code/2.1.246
           "aborting with incomplete response"  error: "reading: context canceled"

A channel mount is a long-lived SSE stream through that edge. When Caddy dropped its listeners every stream died, and the mount looks healthy from every angle afterwards: whoami answers, list_joined_rooms answers, read_recent_messages answers.

Corroborated again tonight by a rollout: for fourteen minutes after the pod restarted it had logged three lines, all startup, and served zero authenticated requests while six sessions held mounts. Nothing reconnects on its own.

Two questions, and the first may not be ours

Does the client reconnect an SSE stream, and if not, should it. A push channel that dies silently on any upstream interruption and stays dead until a restart means the fleet goes deaf whenever an edge hiccups. Tonight it took an outage; a Caddy reload would do it, and edge-config converges /etc on all three edges every sixty seconds. If the server cannot make the client reconnect, that is a real answer and it moves the fix to restart discipline rather than leaving it looking like a defect this repository owes.

The mount cannot tell anyone it has stopped receiving. whoami proves the mount and says nothing about the stream. This is the third instance of the mount being unable to describe its own state, beside #127 and #129.

What the server knows, which is less than it looks

Peer::send_notification to a peer whose client has gone away resolves successfully — there is no delivery acknowledgement anywhere in the channel contract, and channel.rs's module docs say so. The only liveness signal is Peer::is_transport_closed, and it is known to false-positive: live_peers counts without evicting precisely because one false positive once removed a peer permanently and killed a channel for a whole session with nothing logged.

So a design that reports stream health has to establish what it is reading first. Neither a resolved send nor is_transport_closed is a measurement, and a heartbeat built on either would inherit the fault.

What would help, in order of how much it claims

A last-event-sent timestamp per peer, readable through whoami. Says when this server last wrote to that stream. Cheap, and it does not claim the client received anything.
A heartbeat the stream carries, which turns silence into an observation on the client's side. This is the one that needs the paragraph above settled first.

Nothing is lost meanwhile: undelivered messages are unacknowledged, so a restart replays them.

#118, a deploy that changes the tool set never reaching a running session. Same root, different half: what a session negotiated at connect time is fixed for its lifetime.
#127, acknowledged messages replaying.
#129, a bot that cannot see its own allowlist.

A channel mount can stop receiving pushes on any upstream interruption and stay that way until the session restarts, and **no observation from inside a session distinguishes a dead stream from a quiet room.** ## Measured Two messages never delivered to one bot, both readable on the server: 15:28:09Z "Oh excellent" never delivered, that session was mid-turn 15:42:52Z "How are you watching all that?" never delivered, that session was idle 15:45:29Z read_recent_messages returns all 8, decrypted, in one call **One lost while busy and one lost fourteen minutes later while idle kills the busy explanation with a single sample.** The fetch path works throughout; only the push is dead. The cause is visible only from the edge journal, which no session can see. `vultr-sg`'s Caddy stopped listening 15:12:16Z to 15:17:09Z while BGP kept announcing, and minutes before it: 14:58:14Z GET matrix-mcp.kampong.social /channel Accept: text/event-stream User-Agent: claude-code/2.1.246 "aborting with incomplete response" error: "reading: context canceled" A channel mount is a long-lived SSE stream through that edge. When Caddy dropped its listeners every stream died, and **the mount looks healthy from every angle afterwards**: `whoami` answers, `list_joined_rooms` answers, `read_recent_messages` answers. Corroborated again tonight by a rollout: for fourteen minutes after the pod restarted it had logged three lines, all startup, and served **zero authenticated requests** while six sessions held mounts. Nothing reconnects on its own. ## Two questions, and the first may not be ours **Does the client reconnect an SSE stream, and if not, should it.** A push channel that dies silently on any upstream interruption and stays dead until a restart means the fleet goes deaf whenever an edge hiccups. **Tonight it took an outage; a Caddy reload would do it**, and `edge-config` converges `/etc` on all three edges every sixty seconds. **If the server cannot make the client reconnect, that is a real answer** and it moves the fix to restart discipline rather than leaving it looking like a defect this repository owes. **The mount cannot tell anyone it has stopped receiving.** `whoami` proves the mount and says nothing about the stream. This is the third instance of the mount being unable to describe its own state, beside #127 and #129. ## What the server knows, which is less than it looks `Peer::send_notification` to a peer whose client has gone away **resolves successfully** — there is no delivery acknowledgement anywhere in the channel contract, and `channel.rs`'s module docs say so. The only liveness signal is `Peer::is_transport_closed`, and it is known to false-positive: `live_peers` counts without evicting precisely because one false positive once removed a peer permanently and killed a channel for a whole session with nothing logged. **So a design that reports stream health has to establish what it is reading first.** Neither a resolved send nor `is_transport_closed` is a measurement, and a heartbeat built on either would inherit the fault. ## What would help, in order of how much it claims - **A last-event-sent timestamp per peer, readable through `whoami`.** Says when this server last wrote to that stream. Cheap, and it does not claim the client received anything. - **A heartbeat the stream carries**, which turns silence into an observation on the client's side. This is the one that needs the paragraph above settled first. Nothing is lost meanwhile: undelivered messages are unacknowledged, so a restart replays them. ## Related - #118, a deploy that changes the tool set never reaching a running session. Same root, different half: what a session negotiated at connect time is fixed for its lifetime. - #127, acknowledged messages replaying. - #129, a bot that cannot see its own allowlist.

jlxq0 commented

2026-08-31 03:30:23 +00:00

Author

Owner

Next up. Triaged 2026-08-31: this is one of three I would take before anything else in my repositories.

It cost real time on 2026-08-29 and 30, in three separate confusions that all looked like different faults.

The fleet coordinates over Matrix, so a mount that is deaf and looks quiet is a session that stops receiving instructions with every mechanical signal healthy. That is the same family as #140's replay, from the other direction: one delivers twice, one delivers not at all, and neither is visible from inside.

**Next up. Triaged 2026-08-31: this is one of three I would take before anything else in my repositories.** It cost real time on 2026-08-29 and 30, in three separate confusions that all looked like different faults. **The fleet coordinates over Matrix**, so a mount that is deaf and looks quiet is a session that stops receiving instructions with every mechanical signal healthy. **That is the same family as `#140`'s replay**, from the other direction: one delivers twice, one delivers not at all, and neither is visible from inside.

jlxq0 referenced this issue

2026-09-02 00:47:03 +00:00

A session cannot enumerate the sessions of the identity it is, and the surface may be MAS rather than Synapse #147