jlxq0/jmap-mcp
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 1 Packages 1 Wiki Activity Actions

fix(ci): one writer to the buildcache ref #11

Merged
jlxq0 merged 1 commit from fix-buildcache-race into main 2026-08-25 03:45:44 +00:00
Conversation 1 Commits 1 Files changed 1 +10 -1
jlxq0 commented 2026-08-25 02:52:09 +00:00
Owner
Copy link

Closes jlxq0/m365-mcp#4 for this repository.

The race

Merging a PR and pushing the release tag are two pushes, so two docker jobs
run concurrently. Both exported to the same unqualified :buildcache ref:

#21 exporting cache to registry
#21 ERROR: error writing layer blob: unknown
error: failed to solve: error writing layer blob: unknown

Observed on m365-mcp runs 49 and 50, failing within half a second of each
other — after the image had already been pushed successfully.

The fix, and why it is not either option in the issue

The tag build now imports the cache and does not export it.

That works because a tag build is the same commit as the main build that
preceded it. Reading the branch: the two paths differ only in TAGS and PUSH,
which are output settings rather than build inputs. So the two caches would be
byte-identical — exporting twice was pure duplication, and the duplication was
the race. One writer removes it by construction rather than by scheduling.

The issue offered two shapes and I took neither:

  • A concurrency: group serialises two builds that produce the same bytes,
    which is paying a queue to do work twice. Worse, Forgejo job-level
    concurrency is unverified here    : nothing in the fleet uses it, this instance
    is 15.0.2, and an unsupported key is ignored silently — a fix that looks
    applied and does nothing, which is the failure mode worth avoiding above all
    others tonight.
  • A per-ref cache gives the tag build a cold cache, which for a Rust image
    is the expensive path, and accumulates refs on a registry whose disk is
    currently the fleet's bottleneck.

carddav-mcp already did this

It guards its export on refs/heads/main and has exactly one writer. So this is
four copies converging on the one that was right, not a new invention — the same
shape as the divergence in oauth_redirect.rs, resolved in favour of the copy
that already worked.

Worth correcting one line of the issue: it says all five export to the same ref,
which is true, but only four of them export from both paths. carddav-mcp
needed no change.

Verification

Not verified by running, and it cannot be yet: the forge runner is out of
disk and every cargo job fails before reaching a gate. A green docker job
would not be evidence either — buildkit does not use that volume.

Correction: the assertion I first wrote was the wrong one

I originally said the check was "exactly one --export-cache site remains".
That does not distinguish fixed from broken, and typst-mcp is the proof:
before this change it had exactly one export site, guarded by
GITHUB_EVENT_NAME != pull_request — which fires on a main push and on a
tag push. One site, two writers. A grep for the count would have reported it
already fixed.

The property is exactly one writer, which is about the guard on the
export, not the number of sites. For three of the four repos those coincide;
for typst they do not.

repo export sites before guard before writers before
caldav-mcp 2 refs/tags/v* and refs/heads/main 2
jmap-mcp 2 refs/tags/v* and refs/heads/main 2
typst-mcp 1 != pull_request 2
m365-mcp 2 refs/tags/v* and the else 2
carddav-mcp 1 refs/heads/main 1 — exempt

After this change every one of the four exports only under
refs/heads/main, stated explicitly rather than implied by "not a tag", so a
trigger added on another branch later cannot quietly become a second writer.

What is checked

  • the workflow YAML parses;
  • the build script extracted from it passes bash -n;
  • the single --export-cache is inside a refs/heads/main guard — read,
    not counted.

Do not merge on a gate you cannot read. This wants one run after the runner
is back: a merge to main and a tag push landing together, both green.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SP7njJZ7ZtrMgULuXh8ddq

Closes jlxq0/m365-mcp#4 for this repository. ## The race Merging a PR and pushing the release tag are two pushes, so two `docker` jobs run concurrently. Both exported to the same unqualified `:buildcache` ref: ``` #21 exporting cache to registry #21 ERROR: error writing layer blob: unknown error: failed to solve: error writing layer blob: unknown ``` Observed on `m365-mcp` runs 49 and 50, failing within half a second of each other — **after** the image had already been pushed successfully. ## The fix, and why it is not either option in the issue **The tag build now imports the cache and does not export it.** That works because a tag build is *the same commit* as the `main` build that preceded it. Reading the branch: the two paths differ only in `TAGS` and `PUSH`, which are output settings rather than build inputs. So the two caches would be byte-identical — **exporting twice was pure duplication, and the duplication was the race.** One writer removes it by construction rather than by scheduling. The issue offered two shapes and I took neither: - **A `concurrency:` group** serialises two builds that produce the same bytes, which is paying a queue to do work twice. Worse, **Forgejo job-level concurrency is unverified here**: nothing in the fleet uses it, this instance is 15.0.2, and an unsupported key is *ignored silently* — a fix that looks applied and does nothing, which is the failure mode worth avoiding above all others tonight. - **A per-ref cache** gives the tag build a cold cache, which for a Rust image is the expensive path, and accumulates refs on a registry whose disk is currently the fleet's bottleneck. ## `carddav-mcp` already did this It guards its export on `refs/heads/main` and has exactly one writer. So this is four copies converging on the one that was right, not a new invention — the same shape as the divergence in `oauth_redirect.rs`, resolved in favour of the copy that already worked. Worth correcting one line of the issue: it says all five export to the same ref, which is true, but only **four** of them export from both paths. `carddav-mcp` needed no change. ## Verification **Not verified by running, and it cannot be yet:** the forge runner is out of disk and every `cargo` job fails before reaching a gate. A green `docker` job would not be evidence either — buildkit does not use that volume. ### Correction: the assertion I first wrote was the wrong one I originally said the check was *"exactly one `--export-cache` site remains"*. **That does not distinguish fixed from broken**, and `typst-mcp` is the proof: before this change it had exactly **one** export site, guarded by `GITHUB_EVENT_NAME != pull_request` — which fires on a `main` push *and* on a tag push. One site, two writers. A grep for the count would have reported it already fixed. The property is **exactly one writer**, which is about the *guard* on the export, not the number of sites. For three of the four repos those coincide; for typst they do not. | repo | export sites before | guard before | writers before | |---|---:|---|---:| | caldav-mcp | 2 | `refs/tags/v*` and `refs/heads/main` | 2 | | jmap-mcp | 2 | `refs/tags/v*` and `refs/heads/main` | 2 | | typst-mcp | **1** | `!= pull_request` | **2** | | m365-mcp | 2 | `refs/tags/v*` and the `else` | 2 | | *carddav-mcp* | *1* | *`refs/heads/main`* | *1 — exempt* | After this change every one of the four exports **only** under `refs/heads/main`, stated explicitly rather than implied by "not a tag", so a trigger added on another branch later cannot quietly become a second writer. ### What is checked - the workflow YAML parses; - the build script extracted from it passes `bash -n`; - **the single `--export-cache` is inside a `refs/heads/main` guard** — read, not counted. **Do not merge on a gate you cannot read.** This wants one run after the runner is back: a merge to `main` and a tag push landing together, both green. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01SP7njJZ7ZtrMgULuXh8ddq
jlxq0 force-pushed fix-buildcache-race from 01f77e76cc
Some checks failed
CI / cargo (pull_request) Failing after 2s
Details
CI / docker (pull_request) Has been skipped
Details
to 26098bace1
All checks were successful
CI / cargo (pull_request) Successful in 1m48s
Details
CI / docker (pull_request) Successful in 18s
Details
2026-08-25 03:39:33 +00:00 Compare
jlxq0 merged commit 7d886e4ba2 into main 2026-08-25 03:45:44 +00:00
jlxq0 deleted branch fix-buildcache-race 2026-08-25 03:45:44 +00:00
jlxq0 commented 2026-08-25 03:55:50 +00:00
Author
Owner
Copy link

Merged as 7d886e4b. Merge builds: cargo 16532, docker 16533, both green.

The Closes jlxq0/m365-mcp#4 in the body above did not fire. Cross-repository references do not close issues on Forgejo, and all four pull requests carried the same line, so the issue stayed open with nothing recording that the work was done. It is closed by hand now, and m365-mcp#4 holds the record for all four repositories: merge commits, what was verified, and two instruments that were proposed and withdrawn before anyone used them.

Two writer paths this change does not close are tracked in m365-mcp#7. Neither affects this repository: it has no schedule: trigger, and the second needs two merges inside one build.

Evidence for this repository specifically, and it is the sharpest of the four. 16533 was the only job across all four merges that retained a log, and it reads:

#21 exporting cache to registry
...
#21 writing cache image manifest sha256:dc4a0ff9f980954d027c1cfa19d6224044a5fce67021be08ec7e1a88ba0a6682

That digest is byte-identical to what :buildcache held before the build. So this repository is the direct demonstration that an unchanged manifest digest does not mean no export happened, which is why the digest test proposed during this work was withdrawn. The log line is the signal; the digest is silent.

Merged as `7d886e4b`. Merge builds: cargo `16532`, docker `16533`, both green. The `Closes jlxq0/m365-mcp#4` in the body above did not fire. Cross-repository references do not close issues on Forgejo, and all four pull requests carried the same line, so the issue stayed open with nothing recording that the work was done. It is closed by hand now, and **[m365-mcp#4](https://forge.oddie.app/jlxq0/m365-mcp/issues/4) holds the record for all four repositories**: merge commits, what was verified, and two instruments that were proposed and withdrawn before anyone used them. Two writer paths this change does not close are tracked in [m365-mcp#7](https://forge.oddie.app/jlxq0/m365-mcp/issues/7). Neither affects this repository: it has no `schedule:` trigger, and the second needs two merges inside one build. Evidence for this repository specifically, and it is the sharpest of the four. `16533` was the only job across all four merges that retained a log, and it reads: ``` #21 exporting cache to registry ... #21 writing cache image manifest sha256:dc4a0ff9f980954d027c1cfa19d6224044a5fce67021be08ec7e1a88ba0a6682 ``` That digest is byte-identical to what `:buildcache` held **before** the build. So this repository is the direct demonstration that an unchanged manifest digest does not mean no export happened, which is why the digest test proposed during this work was withdrawn. The log line is the signal; the digest is silent.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
waiting-on-julian
Blocked on Julian. The ask is in the issue.

No labels
waiting-on-julian
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jlxq0/jmap-mcp!11
No description provided.