fix(ci): run the docker job on the builders host socket #30

Merged

jlxq0 merged 3 commits from fix/ci-docker-host-socket into main

2026-10-24 07:14:59 +00:00

jlxq0 commented

2026-10-24 07:06:45 +00:00

Owner

The docker job (build, smoke and publish) has been failing on every ref since ubuntu-22.04 moved to the standalone build-*-general builders. The workflow mounted and read DinD TLS files under /certs, which don't exist there. Merged main run 16158 (job 30358) failed with Could not read certificate from /certs/server-cert.pem before building, so v0.3.0 can't be tagged yet. Ops checked the builders: docker_host: automount, the host daemon on /var/run/docker.sock, buildkitd on the host, no /certs material. The Fondue DinD runners are scaled to zero on purpose.

Changes

Runner path. Removed the /certs mounts and the TLS/SAN daemon selection. The job now uses the automounted Unix socket and fails by name if it's missing. buildkitd and the smoke server's published port are reached through the job container's bridge gateway. The smoke server publishes on that address only and retries if another job already holds the host port.
Shared daemon. The host daemon is shared by every job on the builder and outlives each run:
- Every name now carries a per-run key that starts with its creation time.
- Publish tags the image ID recorded after load. It refuses if the per-run name no longer resolves to that ID, and checks the pushed tag afterwards.
- Teardown untags this run's image.
- A new first step removes smoke stacks over six hours old that a cancelled run left behind.
- :latest moves only to the highest release tag.
Credential. Any job on the builder can reach this job container through the daemon. The token is now passed through env: instead of being pasted into the script. It exists only during the cache build, removed by a trap, and around the release push. Pull requests still get no login.
smoke.sh. The step total is counted from the script; it had printed 11/10.
Docs. New AGENTS.md pitfall; stale "ten-step" wording fixed.

Verification

Ran the job's own step scripts, taken from this YAML, inside ghcr.io/catthehacker/ubuntu:act-22.04 on a separate bridge network, with only the host socket mounted. Docker 29.8.2, using the v0.3.0 image built from 5f638a8. Results:
- Socket and gateway were detected; smoke passed 11/11.
- The credential was gone after the build step and after the publish step.
- Publish pushed by image ID to a throwaway local registry. :latest was held for a tag below the highest and moved for the highest.
- Stale planted resources were removed and fresh ones kept.
- Teardown left no image behind.
Negative control: without the socket mounted, the job stops with the named error.
cargo +stable (1.99) fmt, clippy -D warnings and test: all green. No Rust changes.
Independent review of the first commit. Its findings are fixed in the second commit; review of the final head is requested here.

Not fixed here (runner-level, ops)

docker_host: automount hands the host socket to every job on these builders, pull-request jobs from any repo included. A hostile co-tenant job can still reach a release job's container while the token is present. The full fix is to keep pull-request jobs off the builders that run trusted jobs.

🤖 Generated with Claude Code

https://claude.ai/code/session_014Tow3aw74AFPQwy1Smuqdo

The `docker` job (build, smoke and publish) has been failing on every ref since `ubuntu-22.04` moved to the standalone `build-*-general` builders. The workflow mounted and read DinD TLS files under `/certs`, which don't exist there. Merged `main` run 16158 (job 30358) failed with `Could not read certificate from /certs/server-cert.pem` before building, so v0.3.0 can't be tagged yet. Ops checked the builders: `docker_host: automount`, the host daemon on `/var/run/docker.sock`, buildkitd on the host, no `/certs` material. The Fondue DinD runners are scaled to zero on purpose. ## Changes - **Runner path.** Removed the `/certs` mounts and the TLS/SAN daemon selection. The job now uses the automounted Unix socket and fails by name if it's missing. buildkitd and the smoke server's published port are reached through the job container's bridge gateway. The smoke server publishes on that address only and retries if another job already holds the host port. - **Shared daemon.** The host daemon is shared by every job on the builder and outlives each run: - Every name now carries a per-run key that starts with its creation time. - Publish tags the image ID recorded after load. It refuses if the per-run name no longer resolves to that ID, and checks the pushed tag afterwards. - Teardown untags this run's image. - A new first step removes smoke stacks over six hours old that a cancelled run left behind. - `:latest` moves only to the highest release tag. - **Credential.** Any job on the builder can reach this job container through the daemon. The token is now passed through `env:` instead of being pasted into the script. It exists only during the cache build, removed by a trap, and around the release push. Pull requests still get no login. - **smoke.sh.** The step total is counted from the script; it had printed `11/10`. - **Docs.** New AGENTS.md pitfall; stale "ten-step" wording fixed. ## Verification - Ran the job's own step scripts, taken from this YAML, inside `ghcr.io/catthehacker/ubuntu:act-22.04` on a separate bridge network, with only the host socket mounted. Docker 29.8.2, using the v0.3.0 image built from `5f638a8`. Results: - Socket and gateway were detected; smoke passed 11/11. - The credential was gone after the build step and after the publish step. - Publish pushed by image ID to a throwaway local registry. `:latest` was held for a tag below the highest and moved for the highest. - Stale planted resources were removed and fresh ones kept. - Teardown left no image behind. - Negative control: without the socket mounted, the job stops with the named error. - `cargo +stable` (1.99) fmt, clippy `-D warnings` and test: all green. No Rust changes. - Independent review of the first commit. Its findings are fixed in the second commit; review of the final head is requested here. ## Not fixed here (runner-level, ops) `docker_host: automount` hands the host socket to every job on these builders, pull-request jobs from any repo included. A hostile co-tenant job can still reach a release job's container while the token is present. The full fix is to keep pull-request jobs off the builders that run trusted jobs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_014Tow3aw74AFPQwy1Smuqdo

jlxq0 added 2 commits

2026-10-24 07:06:45 +00:00

fix(ci): run the docker job on the builders' host socket 23b3b03dd9

`ubuntu-22.04` now lands on the standalone `build-*-general` builders,
which run `docker_host: automount`: the host daemon is a Unix socket and
there is no DinD sidecar or /certs TLS material. The job failed on run
16158 with `Could not read certificate from /certs/server-cert.pem`
before building anything, so v0.3.0 could not be tagged.

- Drop the /certs bind mounts and the TLS/SAN daemon selection; use the
  automounted /var/run/docker.sock and fail by name when it is absent.
- Reach buildkitd and the smoke server's published port through the job
  container's bridge gateway, publishing on that address only, and retry
  a host port another job already holds.
- Name the smoked image per run. The host daemon is shared and outlives
  the job, so a fixed `typst-mcp:ci` could be retagged by a concurrent run
  between smoke and publish; untag it at teardown.
- Count smoke steps from the script, which printed `11/10`.

The untrusted pull-request credential boundary is unchanged: registry
login still runs only off pull requests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Tow3aw74AFPQwy1Smuqdo

fix(ci): keep the push credential and smoked image safe on a shared daemon

CI / cargo (pull_request) Successful in 1m59s

Details

CI / docker (pull_request) Successful in 2m2s

Details

f2bd7dc706

Review of the host-socket change: every job on the builder can drive the
host daemon, and through it this job container.

- The registry credential reaches the job through `env:` instead of being
  pasted into the script, is removed when the cache build ends (success or
  not), and is written again only around the release push.
- Publish tags the image id recorded after load, refuses if the per-run
  name no longer resolves to it, and checks the pushed tag still does.
- `:latest` moves only to the highest release tag, so rerunning an older
  tag cannot move it backwards.
- Names start with their creation time and a random part, and a new step
  removes smoke stacks older than six hours that a cancelled run left.
- smoke.sh refuses to run from a pipe, where counting its own steps would
  consume the rest of the script; stale ten-step docs updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Tow3aw74AFPQwy1Smuqdo

jlxq0 added 1 commit

2026-10-24 07:12:05 +00:00

fix(ci): check the pushed digest and read the highest tag after the push

CI / cargo (pull_request) Successful in 1m34s

Details

CI / docker (pull_request) Successful in 19s

Details

d086ff8e20

Second review of the shared-daemon change:

- Compare the digest the registry reports for the push with the smoked
  image, instead of where the tag points afterwards, which a retag during
  the push and back again would pass.
- Read the highest release tag after the version push, which can take
  minutes, and match only digits in it; never prompt for credentials.
- Say what the short credential window is: it narrows exposure to other
  jobs on the builder, and does not stop a hostile one, which only runner
  policy can.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Tow3aw74AFPQwy1Smuqdo

jlxq0 merged commit 5d9324935c into main

2026-10-24 07:14:59 +00:00

jlxq0 referenced this pull request from a commit

2026-10-24 07:15:01 +00:00

Merge pull request 'fix(ci): run the docker job on the builders host socket' (#30) from fix/ci-docker-host-socket into main