jlxq0/typst-mcp

Fork 0

Two http tests assert on wall-clock deadlines and fail under machine load #19

New issue

Open

opened 2026-08-26 05:24:58 +00:00 by jlxq0 · 1 comment

jlxq0 commented

2026-08-26 05:24:58 +00:00

Owner

run	load average (16 cores)	result
1	964	FAILED, both
2	964	FAILED, both (re-run alone, `--test-threads=1`, so not contention between tests)
3	~450	ok. 2 passed in 10.28 s

The two:

a_runaway_document_times_out_and_the_server_keeps_serving (tests/http.rs:1150).
Runs with COMPILE_TIMEOUT=3s. The first assertion — the runaway document gets 504 —
passes throughout. The one that breaks is the second: a trivial = Still here document
is expected to return 200 and returns 504, because a trivial compile in a debug
build did not finish inside 3 s under load.
the_big_five_exercises_five_uploaded_jpegs_end_to_end (tests/http.rs:856). Five
uploaded JPEGs through the hanso template against the 30 s default. Returns 504 with
compile exceeded its 30s deadline.

Why this is worth fixing rather than tolerating

The property each test exists to pin is real and worth keeping. a_runaway_document…
asserts containment that does not wedge the server, which is the whole point of the
subprocess design — "containment that wedges the server has contained nothing" is in the
test's own comment. the_big_five… is the multi-asset end-to-end path.

But neither property is about elapsed time, and both are currently asserted through
elapsed time. That makes them nondeterministic about an unchanged input: they fail on a
loaded machine and pass on an idle one, which is the definition of a test that cannot be
trusted in either direction. A green run tells you the machine was quiet. A red run tells
you it was not.

This machine is also the macos-27 Forgejo runner, and it was running a concurrent Xcode
build and an iOS simulator when runs 1 and 2 were taken. That is a normal state for it,
not an unusual one.

What not to do

Do not simply raise the timeouts. The existing values are load-bearing in the other
direction and AGENTS.md records why more than once: a deadline fixture must be CPU-bound
rather than a memory bomb, or the worker's RLIMIT_AS kills it and the request returns
500 instead of 504; and a debug integration-test binary does not share the stripped 1 GiB
production worker's ceiling. Raising a number to make a red test green is how both of
those were originally papered over. Whatever replaces this has to keep the runaway case
actually tripping the deadline.

Not scheduled

No decision has been taken on the approach. Raised because it was measured, not because a
fix is queued.

Found 2026-08-26 while running the locked gate for #17.

Two integration tests assert on wall-clock compile deadlines, so their outcome is determined by how busy the machine is rather than by the code under test. Same commit, same compiled binary, no rebuild between runs: | run | load average (16 cores) | result | |---|---|---| | 1 | 964 | **FAILED**, both | | 2 | 964 | **FAILED**, both (re-run alone, `--test-threads=1`, so not contention between tests) | | 3 | ~450 | **ok. 2 passed** in 10.28 s | The two: - `a_runaway_document_times_out_and_the_server_keeps_serving` (`tests/http.rs:1150`). Runs with `COMPILE_TIMEOUT=3s`. The first assertion — the runaway document gets 504 — passes throughout. The one that breaks is the second: a trivial `= Still here` document is expected to return 200 and returns **504**, because a trivial compile in a debug build did not finish inside 3 s under load. - `the_big_five_exercises_five_uploaded_jpegs_end_to_end` (`tests/http.rs:856`). Five uploaded JPEGs through the `hanso` template against the 30 s default. Returns 504 with `compile exceeded its 30s deadline`. ## Why this is worth fixing rather than tolerating The property each test exists to pin is real and worth keeping. `a_runaway_document…` asserts containment that does not wedge the server, which is the whole point of the subprocess design — "containment that wedges the server has contained nothing" is in the test's own comment. `the_big_five…` is the multi-asset end-to-end path. But **neither property is about elapsed time**, and both are currently asserted through elapsed time. That makes them nondeterministic about an unchanged input: they fail on a loaded machine and pass on an idle one, which is the definition of a test that cannot be trusted in either direction. A green run tells you the machine was quiet. A red run tells you it was not. This machine is also the `macos-27` Forgejo runner, and it was running a concurrent Xcode build and an iOS simulator when runs 1 and 2 were taken. That is a normal state for it, not an unusual one. ## What not to do **Do not simply raise the timeouts.** The existing values are load-bearing in the other direction and `AGENTS.md` records why more than once: a deadline fixture must be CPU-bound rather than a memory bomb, or the worker's `RLIMIT_AS` kills it and the request returns 500 instead of 504; and a debug integration-test binary does not share the stripped 1 GiB production worker's ceiling. Raising a number to make a red test green is how both of those were originally papered over. Whatever replaces this has to keep the runaway case actually tripping the deadline. ## Not scheduled No decision has been taken on the approach. Raised because it was measured, not because a fix is queued. Found 2026-08-26 while running the locked gate for #17.

jlxq0 commented

2026-08-27 02:58:54 +00:00

Author

Owner

A pointer, not a plan. This stays unscheduled.

Alan noted these are the same class as the wall-clock arms in jlxq0/mantis#129, and that
repository has already replaced them. I read bin/test.sh rather than the issue text,
because #129's body is mostly about fixed shared names and the replacement pattern is only
visible in the code. ~/Forge/mantis/wt/main/mantis/bin/test.sh:288-343. Three parts,
and the third is the one that transfers directly here.

Assert a count of overlapping intervals, not elapsed seconds. two builds at cap 1 serialise used to compare durations (cap2 5.00s cap1 4.22s). It now marks each build's
interval and counts how many were open at once, asserting unwrapped 2, cap2 2, cap1 1.
Load moves every one of those durations and moves none of those counts.

Flank the treatment with controls that must come out the other way. Its own comment:
without the unwrapped arm, a machine serialising by itself passes with the queue removed;
without the cap-2 arm, a wrapper that is merely slow passes as one that queues. Same
principle as an accepted push beside a refused one.

Use a timeout as a bound rather than as a measurement. Written there as: an uncontended
flock is instant at any load, and under a version that stranded the slot it never returns
at all, so a 120 s bound costs nothing on the good path and turns a hung suite into a red
case on the bad one.

How that maps onto these two

the_big_five_exercises_five_uploaded_jpegs_end_to_end is the clean case. Its property
is a five-asset branded document renders, and it is currently asserted through renders
within 30 s. Nothing about five JPEGs through the hanso template is a statement about
elapsed time. Give that fixture a deliberately generous bound, so it goes red on a hang and
on nothing else, and leave the 30 s default to be tested where the deadline is the subject.

a_runaway_document_times_out_and_the_server_keeps_serving conflates two properties and
only one of them is broken. The first assertion, that a runaway trips the deadline and
returns 504, is a statement about elapsed time, is the point of the test, and passed in
every run including at load 964. The second, that a trivial = Still here document returns
200 afterwards, is a statement about containment not wedging the server and is currently
asserted through the same COMPILE_TIMEOUT=3s that the first arm needs to be short. That
is what breaks: a trivial compile in a debug build did not finish inside 3 s under load, so
the test reported the server as wedged when it was serving.

Splitting those two so the follow-up request is not bound by the deadline the runaway arm
requires would fix it without touching either property.

Still not to be fixed by raising the numbers

#129's own body says it and AGENTS.md records the two reasons here: a deadline fixture
must be CPU-bound or the worker's RLIMIT_AS kills it and the request returns 500 instead
of 504, and a debug integration-test binary does not share the stripped 1 GiB production
worker's ceiling. Both were originally papered over by moving a number.

A pointer, not a plan. This stays unscheduled. Alan noted these are the same class as the wall-clock arms in `jlxq0/mantis#129`, and that repository has already replaced them. I read `bin/test.sh` rather than the issue text, because #129's body is mostly about fixed shared names and the replacement pattern is only visible in the code. `~/Forge/mantis/wt/main/mantis/bin/test.sh:288-343`. Three parts, and the third is the one that transfers directly here. **Assert a count of overlapping intervals, not elapsed seconds.** `two builds at cap 1 serialise` used to compare durations (`cap2 5.00s cap1 4.22s`). It now marks each build's interval and counts how many were open at once, asserting `unwrapped 2, cap2 2, cap1 1`. Load moves every one of those durations and moves none of those counts. **Flank the treatment with controls that must come out the other way.** Its own comment: without the unwrapped arm, a machine serialising by itself passes with the queue removed; without the cap-2 arm, a wrapper that is merely slow passes as one that queues. Same principle as an accepted push beside a refused one. **Use a timeout as a bound rather than as a measurement.** Written there as: an uncontended `flock` is instant at any load, and under a version that stranded the slot it never returns at all, so a 120 s bound costs nothing on the good path and turns a hung suite into a red case on the bad one. ## How that maps onto these two **`the_big_five_exercises_five_uploaded_jpegs_end_to_end`** is the clean case. Its property is *a five-asset branded document renders*, and it is currently asserted through *renders within 30 s*. Nothing about five JPEGs through the `hanso` template is a statement about elapsed time. Give that fixture a deliberately generous bound, so it goes red on a hang and on nothing else, and leave the 30 s default to be tested where the deadline is the subject. **`a_runaway_document_times_out_and_the_server_keeps_serving`** conflates two properties and only one of them is broken. The first assertion, that a runaway trips the deadline and returns 504, is a statement about elapsed time, is the point of the test, and passed in every run including at load 964. The second, that a trivial `= Still here` document returns 200 afterwards, is a statement about *containment not wedging the server* and is currently asserted through the same `COMPILE_TIMEOUT=3s` that the first arm needs to be short. That is what breaks: a trivial compile in a debug build did not finish inside 3 s under load, so the test reported the server as wedged when it was serving. Splitting those two so the follow-up request is not bound by the deadline the runaway arm requires would fix it without touching either property. ## Still not to be fixed by raising the numbers #129's own body says it and `AGENTS.md` records the two reasons here: a deadline fixture must be CPU-bound or the worker's `RLIMIT_AS` kills it and the request returns 500 instead of 504, and a debug integration-test binary does not share the stripped 1 GiB production worker's ceiling. Both were originally papered over by moving a number.

No labels

deferred

waiting-on-julian

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

jlxq0/typst-mcp#19

No description provided.

Rows
Columns