fix(ci): one writer to the buildcache ref #3

Merged

jlxq0 merged 2 commits from fix/one-writer-to-buildcache into main 2026-08-26 06:35:40 +00:00

Conversation 3 Commits 2 Files changed 2 +59 -1

jlxq0 commented 2026-08-26 05:31:11 +00:00

Owner

Copy link

This repository's last main push has read red since 2026-08-17 and the cause
is not the code.

Merging a pull request and pushing the release tag behind it are two pushes
seconds apart, so two docker jobs run at once, and both exported mode=max to
the same unqualified :buildcache ref.

run	ref	commit	docker
8	refs/heads/main	80c2d25	failed, twice, two seconds apart
9	refs/tags/v0.3.0	80c2d25	success, three seconds later

Same commit, same image, one red. v0.2.0 did the same: 12293 (main) failed while
12292 (tag) succeeded, started 14 seconds apart. v0.1.0 and v0.2.1 pushed tag and
main 80 to 90 seconds apart and both passed.

The fix

The tag build imports the cache and no longer exports it. A tag build is the same
commit as the main build that preceded it — the branches differ only in TAGS
and PUSH, which are output settings rather than build inputs — so the two
caches would have been byte-identical. Exporting twice was pure duplication, and
the duplication was the race.

This follows jlxq0/jmap-mcp#11, which closed jlxq0/m365-mcp#4 the same way
after reading the failure off a log this instance will not give us:

#21 exporting cache to registry
#21 ERROR: error writing layer blob: unknown

on m365-mcp runs 49 and 50, failing half a second apart, after the image had
already been pushed. caldav-mcp, typst-mcp and m365-mcp have converged on the
same shape, so this is a fifth copy of a fix rather than a new invention.

Not chosen

A concurrency: group. Forgejo job-level concurrency: is unverified on this
instance and an unsupported workflow key is ignored in silence, so it would look
applied and do nothing — which is the failure mode this whole class is about.
jlxq0/m365-mcp#7 is where verifying it belongs.

A per-ref cache. It gives every tag build a cold cache, which for a Rust image is
the expensive path, and it accumulates refs on a registry that is already the
fleet's disk bottleneck.

What is and is not verified

Not verified by running, and it cannot be from one push. Reproducing the race
needs a tag and a main push landing together, which is the next release. The
red this fixes is also the inferred cause here rather than the observed one:
job logs 404 on this instance over both the API and the web path, so a red job is
diagnosed by shape and by reproduction, never by reading it. What transfers from
m365-mcp is the error and the mechanism; what is measured here is the timing.

Checkable now, and checked:

• the YAML parses
• the extracted build step passes bash -n
• grep -c export-cache .forgejo/workflows/ci.yml is 1
• branch logic exercised for all three refs:

  GITHUB_REF	push	import args	export args
  refs/tags/v9.9.9	true	1	0
  refs/heads/main	false	1	1
  refs/pull/3/head	false	0	0

AGENTS.md records the one-writer invariant, that grep -c export-cache should
stay 1, and that adding a schedule: trigger would reintroduce a second writer,
since a cron run's GITHUB_REF is refs/heads/main — which is
jlxq0/m365-mcp#7's first surviving path, and the reason this repository is
currently unaffected by it.

This repository's last `main` push has read red since 2026-08-17 and the cause is not the code. Merging a pull request and pushing the release tag behind it are two pushes seconds apart, so two `docker` jobs run at once, and both exported `mode=max` to the same unqualified `:buildcache` ref. | run | ref | commit | docker | |---|---|---|---| | 8 | `refs/heads/main` | `80c2d25` | **failed**, twice, two seconds apart | | 9 | `refs/tags/v0.3.0` | `80c2d25` | success, three seconds later | Same commit, same image, one red. v0.2.0 did the same: 12293 (main) failed while 12292 (tag) succeeded, started 14 seconds apart. v0.1.0 and v0.2.1 pushed tag and main 80 to 90 seconds apart and both passed. ## The fix The tag build imports the cache and no longer exports it. A tag build is the same commit as the `main` build that preceded it — the branches differ only in `TAGS` and `PUSH`, which are output settings rather than build inputs — so the two caches would have been byte-identical. Exporting twice was pure duplication, and the duplication was the race. This follows `jlxq0/jmap-mcp#11`, which closed `jlxq0/m365-mcp#4` the same way after reading the failure off a log this instance will not give us: ``` #21 exporting cache to registry #21 ERROR: error writing layer blob: unknown ``` on m365-mcp runs 49 and 50, failing half a second apart, after the image had already been pushed. caldav-mcp, typst-mcp and m365-mcp have converged on the same shape, so this is a fifth copy of a fix rather than a new invention. ## Not chosen A `concurrency:` group. Forgejo job-level `concurrency:` is unverified on this instance and an unsupported workflow key is ignored in silence, so it would look applied and do nothing — which is the failure mode this whole class is about. `jlxq0/m365-mcp#7` is where verifying it belongs. A per-ref cache. It gives every tag build a cold cache, which for a Rust image is the expensive path, and it accumulates refs on a registry that is already the fleet's disk bottleneck. ## What is and is not verified **Not verified by running, and it cannot be from one push.** Reproducing the race needs a tag and a `main` push landing together, which is the next release. The red this fixes is also the *inferred* cause here rather than the observed one: job logs 404 on this instance over both the API and the web path, so a red job is diagnosed by shape and by reproduction, never by reading it. What transfers from m365-mcp is the error and the mechanism; what is measured here is the timing. Checkable now, and checked: - the YAML parses - the extracted build step passes `bash -n` - `grep -c export-cache .forgejo/workflows/ci.yml` is 1 - branch logic exercised for all three refs: | `GITHUB_REF` | push | import args | export args | |---|---|---|---| | `refs/tags/v9.9.9` | true | 1 | **0** | | `refs/heads/main` | false | 1 | 1 | | `refs/pull/3/head` | false | 0 | 0 | `AGENTS.md` records the one-writer invariant, that `grep -c export-cache` should stay 1, and that adding a `schedule:` trigger would reintroduce a second writer, since a cron run's `GITHUB_REF` is `refs/heads/main` — which is `jlxq0/m365-mcp#7`'s first surviving path, and the reason this repository is currently unaffected by it.

jlxq0 referenced this pull request from a commit 2026-08-26 06:21:11 +00:00

docs: a skipped docker job reports success, and a two-second death is contention

jlxq0 force-pushed fix/one-writer-to-buildcache from d1c5fbb666 to 009586b9fd 2026-08-26 06:21:11 +00:00 Compare

jlxq0 commented 2026-08-26 06:21:25 +00:00

Author

Owner

Copy link

Rebased onto main after #4, so its cargo job now has the h2 fix under it and should go green. Two additions since the first push.

What the real verification is, and what I will look for. This cannot be proven from a pull request: reproducing the race needs a tag push and a main push landing together, which is the next release. At that release, both runs should appear as docker tasks and both should be success — specifically the main one, which is now the only exporter. If the main docker job fails again inside a minute while the tag one succeeds, this fix was wrong and the writer is not what serialises them. I will read it off GET actions/tasks rather than off the commit statuses, for the reason below.

Two ways of misreading this CI, both of which I hit today, now in AGENTS.md.

A docker job skipped because needs: cargo failed posts success to the commit status. PR #2 shows failure CI / cargo at 05:23:03Z and success CI / docker at 05:23:04Z, one second later, with no docker task in the run list at all. PR #3 shows the same pair, 05:34:52Z and 05:34:53Z. I read both of those green ticks as builds and reported them as such before checking, which is why it is written down rather than remembered. matrix-mcp and m365-mcp have the same fault.

A job that ends within seconds of starting never ran. Run 16913 lasted two seconds; the retry of the same branch built in 63. The runner is capacity 1 and shared by every repository in the fleet, so that is contention, and the answer is to push again rather than to read the diff. A genuine failure in this workflow costs at least as long as the step that failed, which is why the buildcache race took 42 to 58 seconds — the image had to build before the export could lose.

Durations from actions/tasks need the same care: several 2026-08-17 runs carry an updated_at two days after their run_started_at, which is a backfill rather than a two-day build.

Rebased onto `main` after #4, so its `cargo` job now has the h2 fix under it and should go green. Two additions since the first push. **What the real verification is, and what I will look for.** This cannot be proven from a pull request: reproducing the race needs a tag push and a `main` push landing together, which is the next release. At that release, both runs should appear as docker tasks and both should be `success` — specifically the `main` one, which is now the only exporter. If the `main` docker job fails again inside a minute while the tag one succeeds, this fix was wrong and the writer is not what serialises them. I will read it off `GET actions/tasks` rather than off the commit statuses, for the reason below. **Two ways of misreading this CI, both of which I hit today, now in `AGENTS.md`.** A `docker` job skipped because `needs: cargo` failed posts `success` to the commit status. PR #2 shows `failure CI / cargo` at 05:23:03Z and `success CI / docker` at 05:23:04Z, one second later, with no docker task in the run list at all. PR #3 shows the same pair, 05:34:52Z and 05:34:53Z. I read both of those green ticks as builds and reported them as such before checking, which is why it is written down rather than remembered. `matrix-mcp` and `m365-mcp` have the same fault. A job that ends within seconds of starting never ran. Run 16913 lasted two seconds; the retry of the same branch built in 63. The runner is capacity 1 and shared by every repository in the fleet, so that is contention, and the answer is to push again rather than to read the diff. A genuine failure in this workflow costs at least as long as the step that failed, which is why the buildcache race took 42 to 58 seconds — the image had to build before the export could lose. Durations from `actions/tasks` need the same care: several 2026-08-17 runs carry an `updated_at` two days after their `run_started_at`, which is a backfill rather than a two-day build.

jlxq0 merged commit eded8e2098 into main 2026-08-26 06:35:40 +00:00

jlxq0 deleted branch fix/one-writer-to-buildcache 2026-08-26 06:35:41 +00:00

jlxq0 referenced this pull request from a commit 2026-08-26 06:35:41 +00:00

Merge pull request 'fix(ci): one writer to the buildcache ref' (#3) from fix/one-writer-to-buildcache into main

jlxq0 referenced this pull request 2026-08-26 06:54:18 +00:00

docs: duration does not tell you whether a job ran #6

jlxq0 referenced this pull request 2026-08-27 01:53:12 +00:00

docs: say why CI / docker is not a required status check #8

jlxq0 commented 2026-08-27 02:07:03 +00:00

Author

Owner

Copy link

The first release since this merged is out, and the result is green but does not verify the fix. Saying so because the green is easy to read as verification and I set this pull request up to be tested by exactly this release.

All four jobs on v0.4.0 succeeded, and all four exist as tasks rather than ticks:

17586  cargo   main     02:00:23Z -> 02:01:22Z   success
17592  cargo   v0.4.0   02:02:01Z -> 02:03:03Z   success
17594  docker  main     02:03:05Z -> 02:04:15Z   success
17604  docker  v0.4.0   02:05:39Z -> 02:06:06Z   success

Compare with the two releases that failed:

v0.2.0   12292 docker v0.2.0  01:57:35Z   12293 docker main  01:57:49Z   14s apart, main FAILED
v0.3.0   12451 docker v0.3.0  04:47:27Z   12453 docker main  04:47:41Z   14s apart, main FAILED

The two docker jobs did not overlap this time. main finished at 02:04:15Z and the tag started at 02:05:39Z, 84 seconds later. The runner has capacity 1 and serialised them, so there were never two writers to :buildcache regardless of what this branch changed. The v0.3.0 pair overlapped for nearly a minute; this pair did not overlap at all.

So the result is consistent with the fix and uninformative about it. An unpatched workflow would have passed this run too.

What did work as intended, and is checkable rather than inferred: the tag build imported the cache and did not export, grep -c export-cache .forgejo/workflows/ci.yml is still 1, and the tag docker job took 27 seconds against main's 70, which is the import-only path hitting a cache the main build had just written.

What would actually verify it is a release where the two docker jobs overlap, and that is not something to arrange by hand: pushing a tag faster does not help, because the queue decides. The honest position is that this stays unverified until a release happens to land while the runner has capacity for both, and the thing to record then is whether their windows overlap, not whether they are green.

I have added nothing to AGENTS.md claiming otherwise. The bullet there already says the real verification is the next release, and it should now say the next release with an overlap.

The first release since this merged is out, and the result is green but **does not verify the fix**. Saying so because the green is easy to read as verification and I set this pull request up to be tested by exactly this release. All four jobs on `v0.4.0` succeeded, and all four exist as tasks rather than ticks: ``` 17586 cargo main 02:00:23Z -> 02:01:22Z success 17592 cargo v0.4.0 02:02:01Z -> 02:03:03Z success 17594 docker main 02:03:05Z -> 02:04:15Z success 17604 docker v0.4.0 02:05:39Z -> 02:06:06Z success ``` Compare with the two releases that failed: ``` v0.2.0 12292 docker v0.2.0 01:57:35Z 12293 docker main 01:57:49Z 14s apart, main FAILED v0.3.0 12451 docker v0.3.0 04:47:27Z 12453 docker main 04:47:41Z 14s apart, main FAILED ``` **The two `docker` jobs did not overlap this time.** `main` finished at 02:04:15Z and the tag started at 02:05:39Z, 84 seconds later. The runner has capacity 1 and serialised them, so there were never two writers to `:buildcache` regardless of what this branch changed. The v0.3.0 pair overlapped for nearly a minute; this pair did not overlap at all. So the result is *consistent with* the fix and *uninformative about* it. An unpatched workflow would have passed this run too. What did work as intended, and is checkable rather than inferred: the tag build imported the cache and did not export, `grep -c export-cache .forgejo/workflows/ci.yml` is still 1, and the tag `docker` job took 27 seconds against `main`'s 70, which is the import-only path hitting a cache the `main` build had just written. **What would actually verify it** is a release where the two `docker` jobs overlap, and that is not something to arrange by hand: pushing a tag faster does not help, because the queue decides. The honest position is that this stays unverified until a release happens to land while the runner has capacity for both, and the thing to record then is whether their windows overlap, not whether they are green. I have added nothing to `AGENTS.md` claiming otherwise. The bullet there already says the real verification is the next release, and it should now say the next release with an overlap.

jlxq0 referenced this pull request 2026-08-27 02:07:28 +00:00

docs: a green release does not verify the buildcache fix, an overlap does #9

jlxq0 commented 2026-08-28 03:51:00 +00:00

Author

Owner

Copy link

Verified. v0.4.1 produced the overlap, and both jobs passed.

tag  docker 18091  03:49:40Z -> 03:49:52Z   12s   success
main docker 18090  03:49:40Z -> 03:50:10Z   30s   success
overlap: 12s

Both started in the same second and ran concurrently for twelve seconds, which is the condition this branch was written for and the one v0.4.0 did not produce. Compare the two releases that failed, where the main job died with the image already built:

v0.2.0   tag 01:57:35Z, main 01:57:49Z   14s apart, main FAILED
v0.3.0   tag 04:47:27Z, main 04:47:41Z   14s apart, main FAILED
v0.4.0   tag 02:05:39Z, main 02:03:05Z   no overlap, both passed, uninformative
v0.4.1   tag 03:49:40Z, main 03:49:40Z   12s overlap, both passed

Under the workflow this replaced, both of those concurrent jobs would have exported mode=max to the same unqualified :buildcache ref, which is what cost v0.2.0 and v0.3.0 a red main. Only the main job exports now, and the pair came through clean.

The durations are consistent with that being the mechanism rather than luck: the tag job took 12 seconds importing only, the main job 30 seconds importing and exporting.

This is the check AGENTS.md names, arrived at by waiting for a release that happened to overlap rather than by arranging one. #9's note that a green release is not verification stands as written, and this release is the exception it describes.

**Verified. `v0.4.1` produced the overlap, and both jobs passed.** ``` tag docker 18091 03:49:40Z -> 03:49:52Z 12s success main docker 18090 03:49:40Z -> 03:50:10Z 30s success overlap: 12s ``` Both started in the same second and ran concurrently for twelve seconds, which is the condition this branch was written for and the one `v0.4.0` did not produce. Compare the two releases that failed, where the `main` job died with the image already built: ``` v0.2.0 tag 01:57:35Z, main 01:57:49Z 14s apart, main FAILED v0.3.0 tag 04:47:27Z, main 04:47:41Z 14s apart, main FAILED v0.4.0 tag 02:05:39Z, main 02:03:05Z no overlap, both passed, uninformative v0.4.1 tag 03:49:40Z, main 03:49:40Z 12s overlap, both passed ``` Under the workflow this replaced, both of those concurrent jobs would have exported `mode=max` to the same unqualified `:buildcache` ref, which is what cost `v0.2.0` and `v0.3.0` a red `main`. Only the `main` job exports now, and the pair came through clean. The durations are consistent with that being the mechanism rather than luck: the tag job took 12 seconds importing only, the `main` job 30 seconds importing and exporting. **This is the check `AGENTS.md` names**, arrived at by waiting for a release that happened to overlap rather than by arranging one. `#9`'s note that a green release is not verification stands as written, and this release is the exception it describes.

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

waiting-on-julian

Blocked on Julian. The ask is in the issue.

No labels

waiting-on-julian

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

jlxq0/hevy-mcp!3

No description provided.