assess: recurrence exception handling (EXDATE, overrides, series truncation) #16
Labels
No labels
blocked
waiting-on-julian
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
jlxq0/caldav-mcp#16
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Assessment of a forwarded specification for recurrence exception handling. No branch, no code. Relayed by
mantisfrom another of Julian's sessions, so it is a well-informed proposal rather than a requirement.Everything below was checked against Stalwart's source at the version we run and against this repository. Two of the proposal's load-bearing claims do not hold.
The version under test
1.
Schedule-Reply: Fdoes not suppress anything on the operation that needs itThe proposal maps
suppress_schedulingtoSchedule-Reply: Fand defaults it to on for occurrence deletions. Stalwart parses the header:and reads
no_schedule_replyin exactly one file:crates/dav/src/calendar/update.rs— thePUTpath — never mentions it. Every occurrence operation in the proposal is aPUT. AddingEXDATE, removing an override, settingUNTIL: all read-modify-write on one href. So the header would be accepted, ignored, and the scheduling messages would go out anyway.That is the exact hazard the proposal names — a silent accept — appearing in the mechanism proposed to prevent it, and with the one failure mode that cannot be undone.
Two further details:
value == "F": exact, case-sensitive, and unlike theIf-Schedule-Tag-Matcharm three lines above it, not trimmed. A lowercasefis a silent no-op even onDELETE, where the header does work.DELETEStalwart is broader than RFC 6638. The RFC definesSchedule-Replyas an Attendee-side control that suppresses theREPLYwhen an attendee deletes a scheduling object (§8.1), and defines no header that suppresses an Organizer'sCANCEL. Stalwart's gate suppresses all iTIP for that delete, organizerCANCELincluded. So Delete All can be suppressed with it. Occurrence-level operations cannot.2. The sixty cancellations are not sixty, and they are not cancellations
An
EXDATEadded to the master is a changed entry on the main instance, and Stalwart's organizer-side diff emits:REQUEST— a full update of the series — one per external attendee perPUT. Not a per-occurrenceCANCEL. Per-instanceCANCELis emitted on a different path: when an override component present in the old object is absent from the new one.So a month of exclusions done as thirty
PUTs sends thirty re-invitations, and the same month batched into onePUTsends one. The proposal's own batching rule is the mitigation, and it is worth more than it claims. The arithmetic in the brief overstates the volume and understates how odd the messages are.3. The switch that does work is
SCHEDULE-AGENT, and it is data rather than a headeris_server_schedulingis set false when theATTENDEEcarriesSCHEDULE-AGENT=CLIENTorSCHEDULE-AGENT=NONE. That is the only per-attendee suppression Stalwart honours on aPUT.It is persistent. Writing it to suppress one batch of exclusions also suppresses every later legitimate update to that attendee until someone removes it. A
suppress_scheduling: boolparameter that quietly rewritesATTENDEEparameters would be a lasting change to the event wearing the costume of a per-call option, and the damage would show up weeks later as an attendee who stopped receiving updates.!self.email.is_localis worth reading twice: if the attendee's address is hosted on this Stalwart, no iMIP is generated at all. Whether the named attendee is local decides whether any of this matters, and that is one lookup rather than an assumption. I have not done it, because it is Julian's data and the question can be answered without me guessing at an address.4. What is already possible with what we have
update_event(src/caldav_client.rs:450) already does exactly the read-modify-write the proposal describes:GETthe whole resource, patch,PUTback to the same href withIf-Matchfrom either the caller's etag or theGET's. The plumbing exists; only the iCalendar manipulation is missing.patch_icsedits the firstVEVENTand passes every later component through untouched, so today'supdate_eventon a series carrying overrides does not destroy them. It also cannot address them.parse_ical_events(src/caldav_client.rs:886) already returns oneEventperVEVENTwith a shared href, so on a non-expanded fetch the master and its overrides are already distinguishable byrecurrence_id.is_overrideis derivable andmaster_hrefis the href itself.5. The gap it identifies is real, and the cause is not ours
list_eventsissues acalendar-querywith<c:expand>(src/caldav_client.rs:353), andsearch_eventsdelegates to it. RFC 4791 §9.6.5 requires an expanded component to have its recurrence properties removed, sorecurrence_rule: nullon an instance is the server obeying the specification rather than our parser losing it. There is no non-expanded read path exposed as a tool, so today there is no way to obtain the master'sRRULE,EXDATEs orRDATEs.get_event_rawwould expose somethingupdate_eventalready fetches on every call.6. What I could not measure, and what it would take
Unmeasured, and both matter:
EXDATEclaim. Structurally it looks right — a mismatched value type orTZIDsimply fails to match a generated occurrence, and nothing rejects it — but "looks right" is not a measurement, and this is the claimmantismost wants measured.Both need writes to a calendar. I have no CalDAV credential of my own: this service forwards the caller's bearer, so reaching the DAV server means using Julian's token, and I will not experiment on his data to find out.
What would make it measurable safely is a throwaway Stalwart account with its own mailbox. Then an attendee can be a local test address, any iMIP lands in a mailbox we can read, and nothing can escape to a third party. That is an infrastructure request rather than something to create unilaterally, and until it exists the honest answer to "does
Schedule-Replywork here" is the source reading above and not an experiment.7. Work split
Small, no new server behaviour, no expansion needed:
get_event_raw(event_href)— expose theGETwe already do.master_href,is_override, and the master'srecurrence_rule/exdates/rdateson expanded instances. Needs a second non-expanded fetch per series, or acalendar-multigetwithout<c:expand>.truncate_series(event_href, until, etag?)—UNTILon theRRULE, strip laterEXDATEs and overrides. Pure read-modify-write.EXDATEinsertion, and removing the matching override in the samePUT.Not small:
delete_occurrences_in_rangeneeds to know whichRECURRENCE-IDs a series actually generates, which means anRRULE/RDATEexpander. We do not have one and it is the single biggest piece of this. Everything the proposal says about matching anEXDATEto the generated occurrence depends on it; without it,delete_occurrencecan only trust therecurrence_idit is handed.RANGE=THISANDFUTURE: supporting it needs the expander too. Rejecting it explicitly is small and should be the first version.Blocked:
suppress_schedulingas specified. It cannot be built onSchedule-ReplyforPUToperations, and building it onSCHEDULE-AGENTis a different feature with a persistent effect that needs to be a decision rather than a default.The brief arrived truncated
The section "Tests worth writing first" is cut off mid-sentence at "a series spanning a". Everything above is assessed without it.
mantiscan obtain the rest; the missing section is the one most likely to change the work split, since it is the part that says what "correct" is supposed to look like.Build order, per Julian's actual requirement
The forwarded document is one route to that, not the requirement. Checklist against the order
mantisset:RANGEinstead of dropping it — PR #17. The silent-ignore was on the read path and live:parse_ical_temporalread onlyTZIDandVALUE, so aRANGE=THISANDFUTUREoverride was reported identically to a single-instance one. There is no write path that can accept aRANGEyet, so explicit rejection lands with the tools that could receive one, in the step below.update_eventpatching the wrong component — PR #17. Not in the original plan; found while assessing, and a live defect rather than a hypothetical one.recurrence_id_value/_tzid/_range— PR #17. Without these, anEXDATEbuilt from our output is the silently-ignoredEXDATEthis issue is about.get_event_raw,master_href,is_override, masterrecurrence_rule/exdates/rdateson expanded instances.is_overrideis now derivable fromrecurrence_id; the master'sRRULEstill needs a non-expanded fetch.delete_occurrenceandupdate_occurrence, series with noATTENDEEonly. No attendees means Stalwart generates no iTIP at all, so the whole scheduling hazard is absent. Not gated on an expander: the tool trusts therecurrence_idit is handed, and the tool description says so. That is exactly what a caller looking at an occurrence in a client already has.Requestper attendee for a batch rather than one per exclusion.truncate_series.delete_occurrences_in_range— waits for anRRULE/RDATEexpander. It is the one that would do the wrong thing at scale.Decision: a tool that would emit scheduling messages refuses by default
Recorded here rather than asked upward, and the code is the reason.
Schedule-Reply: Fis inert onPUTandSCHEDULE-AGENT=CLIENTis persistent data that suppresses every future update to that attendee too. There is therefore no way to make one of these operations quiet. The only honest options are "send the messages" and "do not touch the event", so the tool offers exactly those: refuse when the master or any override carries anATTENDEE, unless the call passes an explicit per-call opt-in, and name the attendees in the refusal.Per call, not per session or per client: the cost is paid per operation, so the consent is too.
One correction to how the refusal should be worded, from
snapshot.rs:send_update_messages()requires!email.is_local, so an attendee hosted on this same Stalwart generates no iMIP. We cannot tell local from external without knowing which domains the server hosts, so the refusal names every attendee and says the server decides which are actually written to. Naming too many is a recoverable error; naming too few is the one that puts mail in a stranger's inbox.Blocked on measurement
The silent-
EXDATEclaim is measured against the throwaway account Clark is creating, before any code that depends on anEXDATElanding. Nothing above depends on it except the two occurrence tools, and those are behind it.Written before the test account exists, so that running it is execution rather than design. A measurement designed after the data arrives can be shaped by it, and this one decides whether three of the refusals in PR #19 are necessary or superstition.
The claim under test
Instrument
curldirectly against Stalwart, not throughcaldav-mcp. Going through our own client puts our parser in the loop, and our parser is not the subject. Every request below is raw HTTP against the DAV base URL with the test account's credentials.Two reads, answering two different questions, and both are needed:
PUTpersist theEXDATE?GETthe resource, look for theEXDATElineREPORTcalendar-querywith<c:expand>over a window containing the occurrence, and look for itsRECURRENCE-IDRead B is the decisive one because the server performs the expansion, so it reports the server's own opinion of the recurrence set rather than ours. Read A alone proves storage, which is not the claim.
The failure under test is exactly: A says present, B says still there.
Instances are counted from the raw multistatus XML by counting
BEGIN:VEVENTand extractingRECURRENCE-IDvalues withgrep, never by parsing with our own code.Safety
Dedicated account, a calendar created for this and deleted after. No fixture carries
ATTENDEEorORGANIZER. With no attendees Stalwart generates no iTIP at all on any path, so no outcome of this experiment can put mail anywhere. That is a property of the fixtures, not a flag I have to remember to pass.Consequently this experiment measures nothing about scheduling. Those claims need a second experiment with an attendee on a second local mailbox, designed separately; bundling them would mean a fixture with an
ATTENDEEin it, and the whole reason this one is safe is that no such fixture exists.Fixtures, one resource per case
Each case gets its own
.icsresource so no case can contaminate another.EXDATEs accumulate, and restoring between cases is a step that can be skipped by accident.S-TZ — timed, zoned.
DTSTART;TZID=Asia/Singapore:20260901T090000,DTEND;TZID=Asia/Singapore:20260901T093000,RRULE:FREQ=WEEKLY;COUNT=6. Occurrences: 1, 8, 15, 22, 29 September and 6 October, all 09:00 SGT.S-DATE — all-day.
DTSTART;VALUE=DATE:20260901,DTEND;VALUE=DATE:20260902,RRULE:FREQ=DAILY;COUNT=6.S-FLOAT — floating, no zone.
DTSTART:20260901T090000,RRULE:FREQ=WEEKLY;COUNT=6. Included because "same value type andTZIDasDTSTART" has three forms, not two, and the floating one is the form nobody tests.Step 0, before any
EXDATE: read what the server generatesREPORTwith<c:expand start="20260901T000000Z" end="20261007T000000Z"/>over each series and record the exactRECURRENCE-IDvalues the server emits — value type,TZIDparameter orZsuffix, and count.This is the baseline every later read is compared against, and it answers on its own the question the whole rule rests on: what form does this server generate? If Stalwart emits
RECURRENCE-ID;TZID=Asia/Singapore:20260908T090000, then a UTC-instantEXDATEis a different value and the rule holds by construction. If it emits UTC instants, the rule is different from what the specification assumes.Cases
Every case:
PUTthe fixture,PUTagain with theEXDATEadded, then read A and read B.EXDATEwrittenEXDATE;TZID=Asia/Singapore:20260908T090000EXDATE;VALUE=DATE:20260903EXDATE:20260908T090000EXDATE:20260908T010000Z— the same instant, different formPUT2xxEXDATE:20260903T000000PUT2xxEXDATE;TZID=UTC:20260908T090000PUT2xxEXDATE;TZID=Asia/Singapore:20260909T090000(a Wednesday; the series is Tuesdays)EXDATE;TZID=Asia/Singapore:20260901T090000C1–C3 are the reason the null results mean anything. Without seeing the instrument move, "6 instances" could be a wrong expansion window, a malformed
REPORT, or a series that never generated that occurrence. If a positive control fails, every W result is void and the experiment stops until it passes.F1 decides a refusal in PR #19. That refusal is currently justified by "some servers treat
DTSTARTas implicitly included", which is received wisdom, not a measurement of this server. If Stalwart honours it, the refusal is superstition and comes out.W4 exists to keep me honest. If W1–W3 show "nothing excluded", W4 shows the same thing for a reason nobody disputes. The three wrong-form cases are only interesting if they are indistinguishable from W4 — which is the point: the server cannot tell you which kind of nothing happened.
What a 2xx with the occurrence still rendering looks like
Recorded here before seeing it, so it is recognisable rather than rationalised afterwards.
PUTreturns204 No Contentwith a newETagdiffering from the one sent inIf-Match. No body, no warning header, noprecondition-failed.EXDATEline present in the stored object, byte-for-byte as written.RECURRENCE-IDfor the occurrence supposedly excluded.Nothing in the
PUTresponse distinguishes this from a successful exclusion. That is the finding, if it holds, and it is why read B is mandatory rather than a nicety: the write is a success by every signal the writing client can see.Also recorded, in case the prediction is wrong in the other direction: if Stalwart rejects a malformed
EXDATE, the interesting output is the status and any<D:error>condition element, since a server that validates makes three of PR #19's refusals unnecessary.What each outcome changes
TZID-from-DTSTARTdesign is necessary; merge itCleanup
Delete the calendar. Record the results as a comment on this issue with the raw status lines and instance counts, not a summary of them.
Addendum to the design above, all of it fixed before the account exists.
Pre-registered consequences, per case
The earlier table grouped cases. Grouping is where interpretation hides, so here is one row per case, each naming the code change it causes. One refusal in PR #19 is on trial and that is exactly where the pressure to read a result charitably applies, so the reading is decided now.
EXDATEshape fromDTSTARTis load-bearing; PR #19 merges as writtenVALUE=DATEbranch comes out; keep it only if W1 and W3 disagree with W2TZIDTZID-from-DTSTARTdecision is load-bearing and the caller must never supply itdelete_occurrenceneeds no expander at all — it can trust the server to refuseWhy W4 is the row that reorders the work
If a correctly-shaped
EXDATEfor a value theRRULEnever generates is accepted and silently does nothing — and it is indistinguishable from W1–W3 — then there is no server-side signal that an exclusion was pointless, and a client cannot discover its own mistake from any response.That makes the
RRULEexpander the only possible pre-flight check, which promotes it from "needed fordelete_occurrences_in_range" to "the only waydelete_occurrencecan tell a caller their exclusion did nothing". It does not block PR #19 — the tool already says it trustsrecurrence_idas given — but it moves the expander ahead oftruncate_seriesand of the attended-series work in the order on this issue.Pre-registering that now, because after seeing the data the temptation is to file it as a known limitation and carry on.
The positive control's form is chosen by Step 0, and the rule is fixed now
If Stalwart's expansion emits
RECURRENCE-ID;TZID=Asia/Singapore:20260908T090000, C1 writesEXDATE;TZID=Asia/Singapore:20260908T090000. If it emitsRECURRENCE-ID:20260908T010000Z, C1 writes that instead — and W1 and C1 swap roles, because "the form the server generates" is what the rule is about, not the form I guessed.Fixing the literal in advance would give a positive control that fails for the wrong reason and void the run. Fixing the rule in advance — the control uses whatever Step 0 observed — is what keeps it a pre-registration.
Free observation while in there: does an ineffective
PUTbumpSEQUENCE?Costs one extra
grepper case and it is the justification for the idempotency refusal in PR #19.Recorded per case:
SEQUENCEbefore the write,SEQUENCEafter, and theETagbefore and after.PUTstoring anEXDATEthat excludes nothing still bumpsSEQUENCE, then a no-op write is a write whose only effect is the side effect — and on an attended series that side effect is mail for a change that did not happen. The idempotency short-circuit indelete_occurrencestops being a nicety and becomes the thing that prevents sending mail about nothing.SEQUENCE, the short-circuit still saves a round trip and anETagchurn, and I will say that is all it saves rather than claiming more.Either way the
ETagwill change, because the bytes changed.ETagis therefore not evidence about anything here and is recorded only so nobody later reads a changedETagas an exclusion having landed.Second mailbox
Requested for the scheduling experiment so it is not blocked a second time. It is not used by this one, and no fixture here will carry an
ATTENDEE— the absence is what makes this experiment safe, rather than any flag I have to remember to pass.Correction: two comments above are wrong about
is_local, and the code was tooVerified from source at v0.16.14 rather than relayed, because it changes what an experiment expects to see.
I wrote, twice above and in
delete_occurrence's refusal message, that Stalwart sends no iMIP to an attendee hosted on itself and that we cannot tell which domains those are. Both halves are wrong.Exact string membership. And
local_addressescomes frombuild_account_infoincrates/common/src/cache/principals.rs, which fills it with the authenticated account's own addresses and its groups' addresses, each expanded across the domain's names — plus the calendar owner's, when acting on someone else's calendar. It is an identity set, not the set of hosted domains.is_local_domain()exists separately and is not what this uses.So
!email.is_localmeans "not one of my own addresses". It exists to stop the organiser mailing themself. A second mailbox on the same Stalwart is not local to the acting account and does receive an iMIP.What it changes here, and what it does not
It does not change this experiment's safety. No fixture in it carries an
ATTENDEEorORGANIZER, so no iTIP is generated on any path regardless of whatis_localmeans. Safety was a property of the fixtures rather than of the belief, which is why the wrong belief cost nothing here.It does not change this experiment's pre-registration either, and I am not amending it to expect an iMIP. There are no attendees in it, so no iMIP is expected and its absence is not a signal — writing "expect mail" into a document whose fixtures cannot generate mail would make the run un-interpretable. The amendment belongs to the scheduling experiment, which is a separate design waiting on the second mailbox, and it is recorded here so it is right the first time:
It does change the code, at
1e88356. The refusal message said Stalwart decides which attendees are actually mailed and that one hosted on it receives no message. It now says to assume every named attendee is written to, because the only address suppressed is the caller's own. Same conservative behaviour, reached by a correct mechanism rather than one that understated the risk by exactly the population most likely to be in a private calendar — colleagues on the same server.The general shape, since it nearly cost a run
The belief was wrong and its conclusion — name every attendee, refuse by default — was right anyway. That is the kind of correction that gets skipped as academic. It was not academic, because someone was about to observe something because of it: an experiment built on it expects no mail for a same-server attendee, so a correct run would have looked like a bug and the finding would have been "Stalwart's suppression does not work" rather than "my model of it was wrong".
When correcting a mechanism whose conclusion holds anyway, the question is who is about to make an observation that depends on it.
Correction: my reading of the 401 was wrong, and no re-run could have caught it
I have been recording the blocker as "Stalwart's directory is OIDC, so no password authenticates for anybody". That is not what the 401 says, and it is not true.
The same 401 is returned by "this directory cannot hold passwords" and by "this principal has none". Nothing in the response separates them, so re-measuring it — which I did, more than once, and reported as diligence — could never have distinguished the two. A negative I re-ran carefully is still a negative from an instrument that has only one answer.
The discriminating measurement was a different population, and Clark ran it:
Six principals carry a password, so password authentication works in that directory. My hypothesis cannot survive that number.
And the precedent was already there.
scratch-spf-probeandsms-ingestare non-human principals with passwords, andcaldav_testandcaldav_test_attendeeexist as principals withcredEntries=0. The fixtures were never missing accounts. They were missing one field, and the established shape for a non-human principal on that directory was one query away.How I got there, since the mechanism is the reusable part
Clark gave me the OIDC reading with the inference marked: "Read that last sentence as the inference it is: the error names the credential type and the dispatch file, so I am reasoning from the message rather than from the source." I adopted it as fact, repeated it in this issue, in
#19's body and in three messages, and the marking did not survive the first retelling.That is worse than making the inference myself. A claim that arrives flagged and leaves unflagged has been laundered by the retelling, and every later reader sees a measurement.
What is actually blocking, stated so it can be checked
A Stalwart-local credential on
caldav_test, which Clark has dispatched. Both fixtures were checked against the live outbound routing table rather than against their names: neither appears in it, both havealiases=0andgroups=0, so a credential on them cannot affect anyone's mail.And the next result is pre-registered here rather than after the fact: if the
PROPFINDstill 401s with a credential set, that is a real directory finding and it goes to Clark as one. It is not a cue to build an alternative acceptance, and I will not.The experiment design above is unchanged.
Ran. Both guards I put on trial come out.
Executed against
caldav_test@kampong.socialon 2026-09-02, in a collection created for the purpose and deleted afterwards. Containment held as committed:MKCALENDARat/dav/cal/caldav_test%40kampong.social/exdate-probe/, href read back and asserted before the first write, every write beneath it,DELETEreturning 204 and the calendar home back to one collection. No fixture carriedATTENDEEorORGANIZER, so no outcome could generate mail.Credential verified with a control before anything else:
PROPFIND /dav/cal/returns 207 with it and 401 with a deliberately wrong password.Step 0 answered more than expected, and inverted the design
Stalwart's expansion emits
RECURRENCE-IDas a UTC instant in every case:The pre-registration said the positive control uses whatever form Step 0 observes, and that C1 and W1 swap roles if it emits UTC. Fixing the rule rather than the literal is what made the run interpretable: a control fixed to the zoned literal would have been the case I had labelled deliberately wrong.
Results, per instance rather than per count
Baselines: zoned series 6 instances with 8 Sep at
20260908T010000Z; all-day series 6 with 3 Sep.EXDATEwrittentzNonetzAEXDATE:20260908T010000ZtzBEXDATE;TZID=Asia/Singapore:20260908T090000tzCEXDATE;TZID=UTC:20260908T090000tzDEXDATE;TZID=Asia/Singapore:20260909T090000dtNonedtAEXDATE;VALUE=DATE:20260903dtBEXDATE:20260903T000000dtCEXDATE:20260903T000000Zf1EXDATE;TZID=Asia/Singapore:20260901T090000on the first occurrenceEvery
PUTreturned 201 regardless of outcome.Stalwart matches an
EXDATEby instant, not by literal form. Zoned, UTC, floating andVALUE=DATEall exclude when they denote the same moment.tzCfails because 09:00 UTC is a different instant from 09:00 SGT, not because the label differs.The pre-registered consequences, fired
W1 excluded anyway → the value-type refusal comes out.
dtBanddtCexclude an all-day occurrence with a date-time, which is exactly the case that refusal exists to prevent. It was superstition on this server.F1 honoured → the first-occurrence refusal comes out.
f1removedDTSTART's own occurrence, leaving five starting 8 Sep. "Some servers treatDTSTARTas implicitly included" is not true of this one.Both removals were written down before the run, and both cite this comment.
What survives, and it is the part the tool actually depends on: taking the
EXDATE's shape fromDTSTARTis still correct, because a shape derived from the series is guaranteed to denote a generated instant. It is now a simplification rather than a guard.W4 fired too, and it moves the roadmap
tzCandtzDare indistinguishable: both accepted with 201, both exclude nothing. So there is no server-side signal that an exclusion was pointless, and a caller cannot discover its own mistake from any response.As pre-registered, that promotes the
RRULEexpander from "needed fordelete_occurrences_in_range" to the only possible pre-flight check, ahead oftruncate_seriesand the attended-series work.Unchanged
The hazard itself is real and this run confirms it: a wrong instant is accepted and silently does nothing. Only its dependence on form was wrong.