Skip to content

FEAT-024 Phase 2C: Staging Proof Runbook

Purpose

Phase 2's staging proof is to "dark-write an allowlisted corpus, run shadow parity and load, force Stale/Rebuilding/Incompatible and kill-switch paths, and prove legacy response equivalence" (technical plan, Phase 2). This runbook is the operational half of that: which gates must close first, what to change, what to run, what each response means, and what evidence to keep.

It covers the project screening family only, on one staging project, with no page, SignalR or export consumer. Phase 2C is explicitly "no ordinary page or SignalR cutover", so the only reader of a materialized value during this proof is the administrator-only parity audit.

What this runbook is not

  • It is not an approval. Running it requires the gates in Preconditions to be closed, including an explicit go from the FEAT-024 programme owner.
  • It is not a production procedure. Nothing here may be pointed at production, and the soak evidence it produces is an input to a later separate production-pilot decision, not that decision.
  • It does not authorize a consumer cutover, a second project, a second family, or any legacy retirement.

Preconditions and gates

# Gate Why
1 syrf #3196 merged The parity audit endpoint lives on this pull request. As of 38a272d4c main has only backfill and rebuild; without #3196 there is no way to compare the projection against the authoritative calculation, which is the entire point of the proof.
2 syrf #3232 merged Binds capacity writes to durable-mode transactions.
2a syrf #3371 merged The administrative mode-transition surface (issue #3369). Without it no production code opens the fleet control or a project's narrow gate, so a completed backfill writes rows that can never serve and the parity audit reports Disabled for every scope — steps 3, 5 and 10 are unperformable and the proof establishes nothing.
3 syrf issue #3185 closed The README's pilot gate: until its single-transaction settings write lands, a Project.AgreementThreshold write can commit between a rebuild's pinned snapshot and its publication without advancing the control's source revision, so a freshly rebuilt row can fail the reader's row/control equality. The pilot must not be activated for any project before this closes.
4 Pilot project chosen and its GUID recorded See Choosing the pilot project.
5 Explicit go from the FEAT-024 programme owner The README holds every flag off and the allowlist empty "until the separately authorized single-project staging activation". This runbook does not grant that authorization.
6 An administrator account on staging Both endpoints sit behind ApplicationAuthorization.BatchAdminProjectsPolicy, whose activity BatchAdminProjects is granted to the administrator application group only. An authenticated non-administrator receives 403; an anonymous caller receives 401.

Confirm the deployed staging API and project-management images actually contain #3196 and #3232 before starting. A promoted chartTag in cluster-gitops is the intent; the running pod's image digest is the fact.

Step 1 — enable the pilot in cluster-gitops

The change is prepared as a draft pull request against camaradesuk/cluster-gitops, held closed behind the gates above. It touches two files:

File Change
syrf/environments/staging/api/values.yaml statistics flags into the existing featureFlags: map; allowlist into the existing env: map
syrf/environments/staging/project-management/values.yaml the same block

Both hosts are required. The API is the serving side and hosts the administrative endpoints; project-management is the source-transaction writing side. Enabling one without the other produces a projection that is either written and never read or read and never maintained.

Flag values

Three on, nine off:

Flag Value Note
materializedProjectStatisticsWrites true Global write kill switch.
materializedProjectStatisticsServing true Global serving kill switch; the runtime catalog also requires the write gate.
materializedProjectStatisticsScreening true The project screening family.
materializedProjectStatisticsMembershipScreening false A separate Phase 4 family, not part of the screening family.
materializedProjectStatisticsAnnotation false
materializedProjectStatisticsMembershipAnnotation false
materializedProjectStatisticsQuestionAnswers false
materializedProjectStatisticsSearchPopulation false
materializedProjectStatisticsDerivedSummaries false
materializedProjectStatisticsPages false Consumer gate — stays off for the whole proof.
materializedProjectStatisticsSignalR false Consumer gate — stays off for the whole proof.
materializedProjectStatisticsExports false Consumer gate — stays off for the whole proof.

ProjectStatisticsFlagMap.FamilyFlagKey maps ProjectStatisticsMetricFamily.ProjectScreening to materializedProjectStatisticsScreening and to nothing else, so the screening proof needs no second family gate. A family with no gate is denied rather than defaulted on.

Allowlist

ProjectStatistics:ProjectAllowlist is deployment configuration, not a generated feature flag. ProjectStatisticsAllowlistConfiguration reads it through IConfiguration, so a non-production administrator cannot widen it from the runtime flag admin UI — which matters, because it is the thing that decides which real projects a dark projection may serve.

It has no env-mapping.yaml entry, so it is set through the chart's generic .Values.env map, the same mechanism ASPNETCORE_ENVIRONMENT already uses:

env:
  SYRF__ProjectStatistics__ProjectAllowlist: "<STAGING_PILOT_PROJECT_ID>"

_deployment-dotnet.tpl copies .Values.env verbatim into the container environment, and the hosts call AddEnvironmentVariables("SYRF__") last, which strips the prefix and turns __ into :, yielding exactly ProjectStatistics:ProjectAllowlist.

The value is a comma-separated list of project GUIDs. There is no wildcard and no permissive failure: absent, empty and malformed all admit nothing. The draft therefore ships the literal placeholder <STAGING_PILOT_PROJECT_ID>, which cannot enable any project even if merged by mistake.

Verifying it rendered — read-only

After ArgoCD syncs, confirm the running pods carry the values. All three commands are reads.

# The rendered environment on each host. Expect the three true flags, the nine false ones,
# and the allowlist GUID.
kubectl -n syrf-staging get deploy api -o json \
  | jq -r '.spec.template.spec.containers[0].env[]
           | select(.name | test("MaterializedProjectStatistics|ProjectAllowlist"))
           | "\(.name)=\(.value)"'

kubectl -n syrf-staging get deploy project-management -o json \
  | jq -r '.spec.template.spec.containers[0].env[]
           | select(.name | test("MaterializedProjectStatistics|ProjectAllowlist"))
           | "\(.name)=\(.value)"'

# Prove the pods were actually replaced, rather than the Deployment spec having moved ahead
# of a stuck rollout.
kubectl -n syrf-staging get pods -l app.kubernetes.io/name=api \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].imageID}{"\n"}{end}'

kubectl exec ... env is an acceptable alternative; it is still a read. Do not use kubectl set env, kubectl apply, kubectl edit or helm upgrade to correct anything found here — a discrepancy is a cluster-gitops change, per the repository's GitOps-only policy.

The clearest functional confirmation costs nothing and comes in step 4: a project outside the allowlist answers the backfill endpoint with 409 and failureReason: "NotAllowlisted". If the pilot project answers that way, the allowlist did not reach the API host — a 404 there means something else, namely that the project document does not exist.

Choosing the pilot project

Record the decision and its reasoning in the evidence file. Criteria:

  • Real screening activity. The screening profiles must have something to distribute over: studies with include and exclude decisions, ideally from more than one reviewer, and ideally some studies with none. A project of all-zero counters proves that zero equals zero.
  • Small enough to backfill in minutes. The backfill is synchronous (see step 4), so its duration is a request duration. Prefer hundreds to low thousands of studies for the first run.
  • Not production-critical and not somebody's live work. Staging data is shared. Do not choose a project that a colleague is mid-review on: step 7 requires making a real screening decision on it.
  • Ideally one whose AgreementThreshold is stable. Configuration changes during the proof invalidate the control's configuration digest and complicate the report (ConfigurationMatchesCurrentSettings).

Record: project id, name, study count, reviewer count, screening decision count, and who confirmed it is safe to use. Read those counts from the project's existing UI or an authoritative read; do not write to the database to arrange them.

Step 2 — capture the "before" state

Before enabling anything, capture the project's current authoritative screening statistics through the ordinary application surface, and keep them in the evidence file. This is the legacy-response baseline that "prove legacy response equivalence" is measured against. With the consumer flags off, every page read is served authoritatively for the whole proof, so this baseline should still hold at the end for anything that did not change.

Step 3 — declare the fleet versions and open the fleet gate

POST https://api.staging.syrf.org.uk/api/admin/project-statistics/fleet/mode
Authorization: Bearer <administrator token>
Content-Type: application/json

{ "mode": "Enabled" }

Nothing serves until this runs. The flags in step 1 say what this deployment requests; the durable fleet control says what it is allowed to do, and it defaults to Disabled. Before this endpoint existed no production code created that control at all, so a completed backfill wrote rows that could never be read and the parity audit reported Disabled for every scope. Enabling is deliberately a separate act from backfilling: the rollout order is dark-write, prove parity, then serve, and a backfill that switched serving on would collapse the middle step it exists to make possible.

This one call does three things in one bounded transaction: it creates the singleton pmProjectStatisticsGlobalControl when the fleet has never had one, declares the catalogue/storage/source versions this build's writers produce, records the deployment's effective reviewer mode, and moves the durable gate to Enabled. A first declaration preserves both the write epoch and mode epoch at zero; it does not retire backfilled rows.

Before this first declaration, verify that every API and project-management replica agrees on ActiveReviewerTrackingEnabled && SignalRActive, and record that effective value in the evidence. The endpoint captures that process value once and stores it with the singleton and invalidation slot in one transaction. It does not assume the default false: an existing true deployment must retain its capacity semantics, including outside the pilot allowlist. This endpoint cannot verify other replicas' configuration. Use the coordinated static rollout; the runtime host-consistency gap remains

3360. Once a singleton exists, its reviewer mode is authoritative: a disagreeing enable returns

409 InvalidState and never silently changes the formula or resets its epoch.

The versions are not yours to choose. They are derived constants (ProjectStatisticsFleetVersions.Deployed) and are echoed back in the response. The reader applies two independent version equalities — fleet-versus-project-control and row-versus-control — and every published row is stamped from the deployed family writer's own constants, so a fleet declaring anything else would guarantee that no row is ever readable. Today they are catalogue 1, storage 0, source 1.

Run it before the backfill

The project control row copies the fleet's storage version verbatim when a backfill creates it, and the backfill refuses (ControlVersionMismatch) a project whose control disagrees with the fleet. Declaring first means the two agree by construction. It is also what makes the step-6 parity audit capable of reporting anything but Disabled.

Outcomes

Status outcome Means
202 Applied The gate is now Enabled. mode, writeEpoch and clientInvalidationRevision in the body are read back from the durable row.
200 AlreadyApplied The fleet was already Enabled under exactly these versions. Nothing changed and no revision was allocated.
409 InvalidState The recorded reviewer mode disagrees with this host, the fleet is serving under different versions — a version change is a disable/declare/re-enable sequence, never a silent edit under live traffic — or the requested stage is not legal from the current mode.
409 RebuildRequired This fleet control has been through a durable disable, so it re-opens only under rebuild evidence. See step 10.
409 QuarantineNotElapsed Only for a Disabled claim; see step 10.
409 InvalidationSlotUnavailable The fixed global invalidation slot could not be admitted, so the durable mode was deliberately left unchanged: a gate change nobody can be told about is worse than no gate change. Retry.
409 ConcurrencyLost A competing transition or writer won. The transaction was aborted; reload and retry.
400 — mode was not Enabled, Disabling or Disabled.

Both fleet and project transitions retry only an uncertain commit acknowledgement, at most three commit attempts on the same transaction. They never replay the transition body or rebuild sweep. If all acknowledgements remain uncertain, the error remains an outage rather than a false 409 claim that nothing committed; inspect the durable mode before resubmitting. Definitive write conflicts and duplicate-key races instead return 409 ConcurrencyLost after abort. Applied outcomes carry the committed invalidation revision; refusal responses do not allocate one.

Record the whole response body in the evidence file: mode, writeEpoch, clientInvalidationRevision and the three versions.

Step 4 — backfill

POST https://api.staging.syrf.org.uk/api/admin/project-statistics/{projectId}/backfill
Authorization: Bearer <administrator token>

No request body, no query string. The route parameter is deliberately named statisticsProjectId rather than projectId in source, so that the shared authorization handler does not probe for the project and leak its existence to an authenticated non-administrator; on the wire it is just the project GUID.

It is synchronous despite the 202

The work runs inline in the request, on the caller's cancellation token, and the response is returned only after every scope has been attempted. There is no background job, no hosted service, no Quartz schedule, no Location header, no job id and no polling endpoint. Treat 202 exactly as you would 200.

Size the client timeout for the whole rebuild. A client that times out aborts the request's cancellation token mid-run, which leaves scopes Stale and a rebuild lease held under the owner string stats.backfill.screening:admin:<32-hex-user-guid>. Use curl --max-time generously rather than a default.

202 Accepted — the success body

{
  "projectId": "...",
  "metricFamily": "ProjectScreening",
  "forced": false,
  "scopesRebuilt": 1,
  "scopesAlreadyCurrent": 0,
  "scopesStale": 0,
  "scopesAbsent": 0,
  "checkpointStatus": "Succeeded",
  "scopes": [
    {
      "scopeKind": "Project",
      "scopeKey": "project",
      "disposition": "Rebuilt",
      "status": "Succeeded",
      "failureReason": "None",
      "publishedGeneration": 1,
      "detail": null
    }
  ]
}

Screening declares exactly one scope, so scopes has one entry with scopeKind: "Project" and scopeKey: "project". disposition is one of Rebuilt, AlreadyCurrent, Stale, Absent. forced is false on /backfill and true on /rebuild.

A 202 is returned only when no scope is Stale and the bootstrap checkpoint was not refused. An Absent scope does not block a 202 — it means the authoritative calculator reported the scope does not exist and the rebuild returned it to Stale rather than publishing a fabricated zero row.

404 Not Found — empty body

One cause: the project document does not exist, so it declares no screening scope to rebuild. The service's explanatory detail is discarded, because NotFoundResult carries no body.

Note that this is not how backfill answers a project outside the allowlist — that is a 409 with failureReason: "NotAllowlisted" (below). The parity endpoint behaves differently and answers 404 for both; see step 6.

409 Conflict — not admitted

Nothing has been written; the check runs before any storage is touched. Key on the failureReason extension, not on status: Type and Instance are never set, so there is no RFC problem-type URI to match, and the Newtonsoft ProblemDetailsConverter flattens extensions to top level so status appears both as the framework's 409 and as a lifecycle enum name.

failureReason Meaning What to do
NotAllowlisted The project is outside ProjectStatistics:ProjectAllowlist. There is no implicit wildcard. Fix the cluster-gitops value, or the allowlist did not reach this host. This is the expected answer for every project today, since the allowlist is empty everywhere.
WriteDisabled materializedProjectStatisticsWrites is off. A backfill that ignored the global kill switch would be the one write the switch exists to stop. Turn the flag on through cluster-gitops.
FamilyNotServable materializedProjectStatisticsScreening is off. Turn the family flag on through cluster-gitops.
FleetVersionMismatch Either the fleet is serving but never declared its catalogue/source versions, or it declares versions the screening writer cannot produce. Every row published would be stamped at the writer's versions and rejected by the reader. Do not retry. This is a deployment-consistency problem: complete the fleet version declaration or deploy the matching family writer.
ControlVersionMismatch The project's control row carries catalogue/storage/source versions that do not match the writer's. Backfill only — the forced rebuild is exempt. Run POST .../rebuild, which reconciles the control and its current rows together inside the publication's own control compare-and-swap.

409 Conflict — partial failure or refused checkpoint

The title distinguishes three cases:

Title Means
The statistics rebuild could not publish every scope. at least one scope is Stale
The statistics rows are current, but the bootstrap checkpoint is incomplete. rows published, checkpoint refused
The statistics rebuild could not publish every scope, and the bootstrap checkpoint is incomplete. both

The body carries failureReason, status and a summary extension holding the full success DTO, so you can see which scopes did publish. Save it; it is evidence.

Common per-scope reasons and their responses:

failureReason Means Response
StatisticsRebuildBusy Another owner holds the per-scope rebuild lease, or the family guard's sole candidate slot is held. Wait for the lease to expire and retry. If it was your own timed-out client, wait it out rather than forcing.
FamilyFenced An active operation fence covers the family — a bulk, import or definition-rewrite operation. No visible scope can become servable while a fence covers it. Wait for the fencing operation to finish, then retry.
StatisticsInclusionRecalculationInProgress An inclusion recalculation or definition-rewrite fence is live. Wait and retry.
PublicationRaceLost A concurrent publication won the compare-and-swap. Retry.
LeaseLost The lease expired or was taken over mid-run — usually a run that took longer than expected. Retry.
DigestMismatch A replayed operation identity carried different content. Do not retry blindly; capture and escalate.
ScopeAbsent The authoritative calculator says the scope does not exist. Reported as Absent, not Stale; does not fail the run.
CheckpointCapacityExceeded, StatisticsScopeCapacityExceeded, StatisticsPublicationOperationCapacityExceeded A bounded capacity limit was hit. Capture; this is a Phase 1 provisional-limit finding worth reporting.

A refused bootstrap checkpoint on a non-forced run publishes nothing — current rows must not outrun the single bootstrap point they are paired with. The detail says to retry the backfill, and, if another checkpoint occupies the identity, to run a forced rebuild (which publishes and therefore moves the projection revision) and then backfill again to record the bootstrap at the new identity. The forced rebuild is exempt from this short-circuit.

500

Non-development hosts return {"error":"An unexpected server error occurred."}. Capture the timestamp and correlate with the API logs; do not retry in a loop.

Step 5 — open the project narrow gate

POST https://api.staging.syrf.org.uk/api/admin/project-statistics/{projectId}/narrow-gate
Authorization: Bearer <administrator token>
Content-Type: application/json

{ "enabled": true }

The fleet gate from step 3 is necessary and not sufficient: each project carries its own narrow serving gate on its control row, fenced by its own ProjectWriteEpoch, and it also defaults to Disabled. Both must be Enabled before the bundle reader will serve a materialized value or the parity audit can report anything but Disabled.

Run it after the backfill, not before. The narrow gate lives on the project's control row, and only a backfill or a materialized source write creates that row. There is deliberately no create-on-enable here: a control row created by this endpoint would carry no configuration digest and no versions, and the reader would refuse every row published against it for ever. A project with no control row is answered 404.

The route parameter is named statisticsProjectId in source for the same reason as the backfill route — so the shared authorization handler does not probe for the project and leak its existence to an authenticated non-administrator. On the wire it is just the project GUID.

Outcomes

Status outcome Means
202 Applied The narrow gate is now Enabled. The body carries the resulting mode and writeEpoch (the project write epoch) and, on a re-enable, scopesRebuilt.
200 AlreadyApplied The gate was already Enabled. Nothing changed.
404 — The project has no statistics control row. Run step 4 first.
409 RebuildRequired The gate has previously been disabled, so it re-opens only under a complete rebuild proof, and the sweep this call ran did not complete. rebuildFailureReason carries the first scope's typed reason — FamilyFenced, StatisticsRebuildBusy, PublicationRaceLost and so on — which is the thing to clear.
409 InvalidState An unfinished Disabling quarantine may not be jumped out of. Complete the disable, then re-enable.
409 QuarantineNotElapsed Only for a { "enabled": false } call claiming Disabled; see step 10.
409 InvalidationSlotUnavailable / ConcurrencyLost As for the fleet gate. Retry.

Confirm it took

Re-run the parity audit (step 6). Before this step it reports Disabled for every scope — the audit reader deliberately keeps every durable gate, overriding only the two flag-level serving switches, so an all-Disabled report is exactly what a shut narrow gate looks like. After it, the report should carry real scopeStates and a real inParity verdict.

An all-Disabled parity report is not a passed gate. It is the report saying nothing was audited. Read fallbackReason before concluding anything from it.

Step 6 — parity audit

GET https://api.staging.syrf.org.uk/api/admin/project-statistics/{projectId}/parity
Authorization: Bearer <administrator token>

This endpoint arrives with #3196 and is not on main as of 38a272d4c. It is read-only on both sides: nothing here creates a control row, family summary, guard or lease, and nothing advances a revision. It answers while serving is off, deliberately — the rollout order is dark-write, then prove parity, then serve.

Outcomes:

  • 200 OK with ProjectStatisticsParityReportDto.
  • 404 — not allowlisted, or the project does not exist. Here the two are deliberately indistinguishable, so the endpoint cannot be used to enumerate the pilot. This differs from backfill, which answers 409 NotAllowlisted for the first case and reserves 404 for the second.
  • 200 OK with fallbackReason: "Disabled" on every scope — the durable gates are shut. The audit reader overrides only the two flag-level serving switches; it deliberately keeps every durable gate, including the fleet control's mode and the project's narrow gate. Run steps 3 and 5. This is a report of nothing having been audited, not a passed gate.
  • 409, type: "urn:syrf:project-statistics:writes-disabled", title "There is no materialized projection to audit." — materializedProjectStatisticsWrites is off, so nothing is maintaining the projection and any report would describe a frozen artefact. Enable writes and backfill first.

Reading the report

{
  "projectId": "...",
  "metricKey": "project-screening",
  "inParity": true,
  "isInconclusive": false,
  "sourceAdvancedDuringAudit": false,
  "materializedAvailable": true,
  "readSource": "...",
  "fallbackReason": "...",
  "capacityFailure": "...",
  "checkpoint": { "checkpointSourceRevision": 0, "checkpointProjectionRevision": 0,
                  "checkpointModeEpoch": 0, "canonical": "..." },
  "watermarks": { "globalClientInvalidationRevision": 0, "sourceInvalidationRevision": 0,
                  "committedProjectionRevision": 0, "clientInvalidationRevision": 0,
                  "modeEpoch": 0, "catalogueVersion": 0, "storageVersion": 0,
                  "sourceVersion": 0, "configurationDigest": "..." },
  "scopeStates": [ { "selection": "...", "scope": "...", "availability": "...", "state": "...",
                     "isTombstone": false, "isPublished": true,
                     "publicationGeneration": 1, "lastChangedRevision": 0 } ],
  "metrics": [ { "metricKey": "screening.sufficientlyScreened",
                 "expected": 0, "actual": 0, "delta": 0, "inParity": true } ],
  "observedAtUtc": "...",
  "controlConfigurationDigest": "...",
  "currentSettingsConfigurationDigest": "...",
  "configurationMatchesCurrentSettings": true
}

inParity is the verdict. It is true only when a materialized value exists and equals the expected one, for every metric. Crucially, a bundle that legitimately refuses to serve is not "in parity" — it is unaudited, inParity is false, materializedAvailable is false, and fallbackReason says why. Read fallbackReason before concluding anything from a false verdict.

The comparison runs over the union of both key sets with absence read as zero, in the catalogue's declaration order. That union matters both ways: a metric the projection never wrote is a legitimate zero, and a counter the projection holds that the authoritative side does not produce is a real defect the report surfaces rather than skips.

isInconclusive means a write landed underneath the comparison, so the deltas may be an artefact of the race rather than real drift. The two halves are separate reads; the project's revision is checked either side of the authoritative half, a move is retried once internally, and a second move reports the run as inconclusive rather than as a divergence an operator would chase. sourceAdvancedDuringAudit reports the same underlying observation.

Retry rule for inconclusive. Re-run the audit unchanged. If it is still inconclusive, the project is under sustained write load and cannot be audited coherently at all — that is the report telling you so, not a transient. Re-run during a quiet window, or choose a quieter pilot project. Do not re-run repeatedly until it happens to look quiet, and never record an inconclusive run as a parity pass or as a divergence. Record every attempt in the evidence file, including the inconclusive ones.

configurationMatchesCurrentSettings is separate from the verdict on purpose:

  • true — the control row is stamped under the project's current settings.
  • false — the control is still stamped under settings the project no longer has, so its counters answer a superseded question even when they agree with today's calculation. This is drift to chase separately, not a divergence in the values compared here. It is also false whenever isInconclusive is true.
  • null — the comparison could not be made. Null never means it passed. It happens when there is nothing to digest, for example a project with no agreement threshold or one that vanished mid-run.

A fresh forced rebuild adopts the digest of the project it read, so it reports a match; the field says nothing about how the digest came to be stamped.

Also capture watermarks and scopeStates verbatim — availability, state, isPublished and publicationGeneration are what distinguish Fresh from Stale, Rebuilding, Incompatible, fenced and tombstoned, and they are the record of which of those paths the proof actually exercised.

Step 7 — exercise a real screening decision

With the projection backfilled and audited, make an actual screening decision on the pilot project through the ordinary UI, as an ordinary reviewer would. This is the point of the proof: the transactional delta path must move the projection in step with the source, without a rebuild.

  1. Note the pre-decision watermarks.committedProjectionRevision and the relevant metric values.
  2. Screen one study — include or exclude — through the normal screening surface. Note the study, the reviewer, the decision and the time.
  3. Re-run the parity audit.
  4. Expect inParity: true with the counters moved by exactly the decision made, and committedProjectionRevision advanced. A false verdict here with materializedAvailable: true and a non-zero delta is the finding the proof exists to catch: capture it in full and stop.
  5. Repeat for a decision of the opposite kind, and for a study that takes a profile from one populated bucket to another, so more than one counter is proven to move.

Worth exercising deliberately while here, since they are named in the Phase 2 staging proof:

  • Kill switch. With the flags turned off in cluster-gitops, confirm reads fall back to the authoritative calculation and the parity endpoint answers 409 writes-disabled. Turn them back on and confirm the projection is still there and still in parity — the rollback preserves data.
  • Stale / Rebuilding. Observe them in scopeStates.state around a rebuild.
  • Forced rebuild. POST .../rebuild and confirm the report is still in parity afterwards, with a new publicationGeneration.

Step 8 — capture evidence

Write one JSON file per proof run to evidence/phase2c-staging/, named phase2c-staging_<YYYY-MM-DD>_<short-run-label>.json. Start from the committed template, phase2c-staging-evidence-template.json, which follows the same shape as the Phase 0 baselines.

Record, at minimum: the date, environment, project id and selection rationale, the exact deployed flag values and allowlist as read from the running pods, the deployed image digests, the backfill outcome, every parity report including inconclusive attempts, the screening decisions made, the fallback reasons observed, and notes. Store the raw response bodies rather than a summary of them — a verdict without its watermarks and scopeStates cannot be re-interpreted later.

Never put reviewer identities, study titles or other project content into the evidence file. Ids, counts and the response envelope are enough.

Step 9 — soak

The README's soak gate and Phase 6's acceptance criteria require, before any production pilot:

  • at least seven actual days of staging soak — calendar days, not seven days' worth of traffic compressed into an afternoon;
  • 10,000 reads;
  • 1,000 relevant mutations;
  • 100% exact counter parity, zero stale materialized serves, 100% injected fallback success, no unresolved rebuild failures, and bounded delta-ledger and storage growth.

Two honest caveats about counting those numbers on a single staging project.

There is no read or mutation counter built for this. Nothing in the Phase 2C code exposes a "reads served" or "mutations applied" metric that can simply be read off. The counts have to come from log aggregation or from a driver that generates the load and counts its own calls. A driver is the more defensible option for the controlled 10,000/1,000 figures, since the plan calls them "controlled"; ambient staging traffic on one project will not reach them in seven days.

With every consumer flag off, ordinary page reads do not touch the projection at all. They are served authoritatively. So "10,000 reads" against a Phase 2C deployment means 10,000 reads of the projection, which during this phase means the parity/admin surface or a driver exercising the bundle reader — not 10,000 page loads. Decide which you are counting, write it down in the evidence file, and do not let the distinction blur.

Observing fallback reasons

Every gated read that declines to serve a materialized value logs a fallback reason. Watch the distribution across the soak: a fallback that is supposed to be rare (Stale, Rebuilding, Incompatible, an epoch mismatch, an active fence) appearing steadily is the signal the soak is for. The two reason enums, the exact log message templates and the substrings to query them by are in Fallback reasons and log fields below — read that section before planning the soak, because there are no metrics and nothing is logged at all while the pilot is dark.

Record the observed set — not just the counts, the set — in the evidence file. A fallback reason nobody expected to see is a finding whether or not parity held.

Step 10 — rollback

Rollback closes the durable gates first, through the API, and then the flags, through cluster-gitops. Both halves matter and they are not interchangeable: the flags stop this deployment from asking to serve, while the durable gates and their write epochs stop any deployment from serving the rows — including a replica whose per-process flag cache has not caught up. The flag cache is explicitly not a correctness authority.

The flag half is a cluster-gitops change, reverted through git and synced by ArgoCD. Never kubectl set env, never helm upgrade.

1. Close the project narrow gate.

POST /api/admin/project-statistics/{projectId}/narrow-gate   { "enabled": false }

Answers 202 with mode: "Disabling". That is not a half-done call — it is the first of two stages. The project write epoch advances at this commit, so a source transaction that observed the gate as Enabled and commits afterwards writes rows carrying the superseded epoch, which are already non-servable. Reads fall back to the authoritative calculation immediately.

2. Claim Disabled once the quarantine has elapsed. Repeat the identical call. Before the deployment-verified maximum transaction lifetime has passed it answers 409 with QuarantineNotElapsed; that interval, not the mode field, is the durable barrier. Afterwards it answers 202 with mode: "Disabled".

3. Close the fleet gate, the same two stages.

POST /api/admin/project-statistics/fleet/mode   { "mode": "Disabled" }

The first call answers 202 with mode: "Disabling" and advances the fleet write epoch; the second, after the quarantine, answers 202 with mode: "Disabled".

4. Turn the flags off. Set materializedProjectStatisticsWrites, materializedProjectStatisticsServing and materializedProjectStatisticsScreening back to false on both hosts.

5. Return SYRF__ProjectStatistics__ProjectAllowlist to the placeholder, or remove the key.

Any one of steps 1, 3, 4 and 5 alone is sufficient to stop the pilot serving — the allowlist admits nothing without a valid GUID, and the flags and the durable gates each gate every read independently — but do all of them, so the deployed state matches the intended state in one reading.

Confirm with the parity audit: after the gates are closed it reports Disabled again, and after the flags are off it answers 409 writes-disabled.

Re-enabling after a durable disable

A gate that has been through a disable does not re-open the way it was first opened. Its quarantine retired a whole epoch of rows, so:

  • The narrow gate re-opens only under a complete rebuild proof. { "enabled": true } produces that proof itself — it rebuilds every scope the project's routed families declare and presents the outcomes to the transition — and refuses with RebuildRequired plus the sweep's own rebuildFailureReason if any scope could not be published.
  • A successful re-enable still serves nothing until you rebuild again. The re-enable allocates a new project write epoch, and the sweep that produced the proof necessarily ran before that epoch existed, so its rows carry the superseded tuple and the reader answers EpochMismatch. This is the plan's stated residual — "a re-enable that serves nothing until a rebuild runs, never one that serves stale data" — not a defect. Run POST .../rebuild after the gate is open and confirm the parity report before treating the project as serving again.
  • The fleet gate answers 409 RebuildRequired and is deliberately not re-openable from this surface: a fleet re-enable's evidence would have to span the whole fleet, and no bounded transaction can establish that. Re-enabling a fleet that has been durably disabled is out of scope for the Phase 2C pilot; a pilot that needs it should be torn down and rebuilt from a fresh fleet control rather than coaxed back open.

What rollback leaves behind

Data, deliberately. "Rollback to authoritative reads is immediate and does not require data deletion", and "a flag rollback returns every read to the authoritative screening facets and preserves history". Concretely, after rollback:

  • The pmProjectStatistics* documents written during the proof — current rows, control row, delta ledger, the backfill-observed checkpoint — remain. They are simply never read.
  • Every read returns to the authoritative calculation immediately. There is no drain and no window in which a stale materialized value is served.
  • Source mutations continue committing normally. With writes off they commit the source, mark affected scopes Stale and fall back, so the projection stops tracking the source from the moment the flag goes false. This is why re-enabling requires a fresh backfill or forced rebuild, not just flipping the flags back: the rows are intact but no longer current, and the reader will correctly refuse them as Stale.
  • History is preserved. The bootstrap checkpoint keeps the identity it was captured under and is never rewritten.

Nothing needs to be deleted, and nothing should be. If the pilot is genuinely being abandoned rather than paused, removing the projection documents is a separate, separately approved cleanup — not part of rollback.

Fallback reasons and log fields

Read this section before planning the soak: what is observable is narrower than it looks.

There are no metrics

SyRF.ProjectManagement.Core/Telemetry/ProjectStatisticsTelemetry.cs declares a meter name (SyRF.ProjectManagement.ProjectStatistics), an instrument prefix (syrf.project_statistics.) and counter names including syrf.project_statistics.requests, ...materialized_reads, ...authoritative_fallbacks and ...flag_decisions, plus a reason tag. None of them record anything. No Meter is constructed, the meter name is never passed to AddMeter, and the class's own doc comment says so: "This wave declares the names only." The same holds after #3196.

Nor is there an OTLP destination: AddOpenTelemetryConfig exports only when OTEL_EXPORTER_OTLP_ENDPOINT is set, and that variable appears nowhere in the charts or in cluster-gitops.

So the soak is observed from logs, not metrics. Do not plan a dashboard query.

Nothing is logged at all while the pilot is dark

ReviewController.GetFullStats calls IsMaterializedReadRequested(projectId) first, which evaluates the flag and allowlist gate and logs nothing. If any of materializedProjectStatisticsWrites, ...Serving, ...Screening is off, or the project is outside the allowlist, the adapter is never entered and no FEAT-024 line is emitted. You cannot observe "reads falling back for reason X" in a fully dark deployment — the absence of log lines is the expected state before step 1, not a fault.

Once the gate passes, the adapter logs on every read: exactly one of the materialized-serve or fallback lines per call.

The two fallback enums

They are different types and both appear in one log line, separated by a literal /.

ProjectScreeningStatisticsFallbackReason — the adapter's reason, arriving with #3196:

Member Meaning
None the projection served
WritesDisabled materializedProjectStatisticsWrites off
ServingDisabled materializedProjectStatisticsServing off
FamilyDisabled materializedProjectStatisticsScreening off
ProjectNotAllowlisted project outside the pilot allowlist
NoCaller no caller identity to authorize against
NotAvailable bounded not-available: unauthorized, unknown or foreign selection
BundleFallback the reader decided the whole bundle is authoritative
BundleUnavailable the reader refused — visibility token or durable-mode disagreement
CapacityExceeded a bounded capacity ceiling
ScopeTombstoned the selection landed on an explicit deletion tombstone
ScopeRowMissing the materialized bundle carried no row for the scope
CheckpointUnresolved an all-zero (source revision, projection revision, mode epoch) tuple
ReaderFailed the read path threw

ProjectStatisticsFallbackReason — the reader's reason, already on main: None, Missing, Stale, Rebuilding, Incompatible, Disabled, EpochMismatch, Fenced, SnapshotPredicateFailed, InclusionRecalculationInProgress, DefinitionRewriteInProgress, DurableModeDisagreement, ProjectUnavailable.

Both render as their exact PascalCase member names — no attributes change the string form. The fallbackReason field of the parity report carries the reader enum's string.

For the soak, the reasons that should be rare are the interesting ones: a steady stream of BundleFallback/Stale, BundleFallback/Rebuilding, BundleFallback/Incompatible, .../EpochMismatch or .../Fenced is precisely what the soak is for. WritesDisabled, ServingDisabled, FamilyDisabled and ProjectNotAllowlisted mean the pilot is not actually on.

The log statements

All from ProjectScreeningStatisticsQueryAdapter (category SyRF.ProjectManagement.Core.Services.ProjectStatistics.Families.Screening.ProjectScreeningStatisticsQueryAdapter) except the last, which is from ReviewController.

Level Template
Debug FEAT-024 answered the project-screening section authoritatively for project {ProjectId}: {FallbackReason}/{ReaderFallbackReason}.
Debug FEAT-024 served the project-screening section from the projection for project {ProjectId} at checkpoint {CheckpointId}.
Error FEAT-024 project-screening materialized read failed for project {ProjectId}; answering from the authoritative screening facets.
Warning FEAT-024 declined to substitute the materialized screening section for project {ProjectId} at checkpoint {CheckpointId}: the materialized totals and the authoritative screening values differ; retaining the coherent response. Answering authoritatively.

The fallback line is Debug on purpose: while the flags are off every request takes that path, and a dark rollout that fills the log with warnings gets its warnings ignored.

The Warning line is the parity-divergence alarm. It is the only Warning-level FEAT-024 read-path signal and the one worth alerting on during the soak. A single occurrence is a finding: capture the project id and checkpoint, run the parity audit immediately, and record both.

Naming the query

API and project-management log plain text to stdout, not JSON. Their appsettings.json uses "WriteTo": [{ "Name": "Console" }] with no formatter key, so Serilog's default MessageTemplateTextFormatter renders the {ProjectId} and {FallbackReason} properties into the message string. Only Identity emits JSON.

Two consequences:

  1. Query by message substring, not by field predicate. FallbackReason="Stale" will not match anything; you must match the message text and parse the trailing <Guid>: <Reason>/<ReaderReason>.
  2. logging.format: json in staging.values.yaml is inert — no env var is mapped from logging.format in env-mapping.yaml or _env-blocks.tpl, so it switches nothing. Do not rely on it.

Substrings to query:

Purpose Substring
every gated read (count for the soak) FEAT-024 and the project-screening section
fallbacks, with reason FEAT-024 answered the project-screening section authoritatively for project
materialized serves FEAT-024 served the project-screening section from the projection for project
parity divergence — alert FEAT-024 declined to substitute the materialized screening section
read-path exception FEAT-024 project-screening materialized read failed for project

The baseline access is pod stdout, e.g. kubectl -n syrf-staging logs deploy/api --since=24h | grep 'FEAT-024'.

Three preconditions before any of this yields output:

  • 3196 merged and deployed — these identifiers do not exist on main;

  • the pilot flags on and the project in the allowlist, or nothing is logged at all;
  • the effective Serilog minimum level at Debug, for the two Debug lines.

On that last point: SYRF__Serilog__MinimumLevel comes from .Values.logging.level, and staging sets level: debug in lowercase, whereas production carries the comment "Serilog requires capitalized full word". Serilog's enum parse is case-insensitive so it should bind, but verify it live before depending on the Debug lines — confirm a known-Debug line appears in kubectl logs before starting a soak whose read count comes from them.

What is centrally visible

  • Sentry is enabled in staging (monitoring.sentry.enabled: true, environment: staging), so the Error and Warning lines above surface there. The two Debug lines do not.
  • Elastic APM is disabled in staging (elasticApm: { enabled: false }).
  • No OTLP, and no Loki/Elasticsearch/Fluent/Promtail/Alloy configuration exists in cluster-gitops. docs/architecture/system-overview.md describes "Aggregation: Loki or CloudWatch / Format: Structured JSON"; that is aspirational and does not match the deployed configuration. Confirm the actual cluster log pipeline out of band before writing a query against a named aggregator.

So, concretely: alert on Sentry for the divergence Warning; count reads from pod stdout; and take fallback-reason and revision observations from the parity endpoint, whose fallbackReason, readSource, capacityFailure and watermarks fields give the same information without depending on log levels or a log pipeline.

Counting mutations

There is no mutation counter either. The durable monotonic revisions on the control row are the practical proxy, and the parity report exposes them: watermarks.committedProjectionRevision, watermarks.sourceInvalidationRevision and watermarks.clientInvalidationRevision. The difference between two observations of committedProjectionRevision is a mutation count. Poll the parity endpoint at a fixed cadence through the soak and keep every response; that series is the mutation evidence, and it doubles as the parity-sampling evidence Phase 6B asks for.