`mutate` — would my suite catch a regression?¶

A green run means nothing the gate fails on was found. Even where every declared expectation held, that does not mean any of them could have failed.

An expectation of [200, 403, 404] matches whatever the target does. A 404 you read as a denial may be a route that answers 404 to everybody. An excludes_marker passes trivially if the marker never appears anywhere. All three pass forever, and none of them is a test.

mutate rewrites your expectations one at a time and re-grades the same observations. A mutant your suite kills is a check with the power to fail. A mutant that survives is a check that cannot.

Run it¶

The repository ships a second fixture API for this, on port 8766, separate from the quickstart's demo server so the numbers below cannot move when the demo changes.

python3.12 fixtures/veridor_mutation_fixture.py --port 8766

veridor mutate --config fixtures/veridor.mutation-demo.json

Without the fixture running, mutate reports THE BASELINE DID NOT REACH THE TARGET, a score of n/a, and exits 2. That is the correct answer — nothing was measured.

Mutation score: 0.92  (12 killed / 13 generated)
  survived 1  aberrant 0  errored 0  not applicable 2  skipped 3
  0.86 of 14 generated + limited (1 could not be evaluated). A mutation score is only as wide as the mutants it could evaluate.
  ...
  requests: 5 baseline, 7 issued by mutation (M4, M5, M6)

SURVIVING MUTANTS (1) -- these checks may be vacuous
  ...

  [SURVIVED] M1 invert-denial -- check 'widget-read-permissive', identity outsider
      mutation: declare 'outsider' receives [200] instead of [200, 403, 404]
      expected: the mutated expectation must now fail because the target
      REFUSED this identity
      observed: the mutated expectation still passed
      means: 'outsider' was served HTTP 200 on this request and the
      declaration [200, 403, 404] accepted it. The expectation cannot tell
      allowed from denied, so its pass may be vacuous: narrow it to the denial
      statuses you actually require.

Exit code 1.

Nothing in a mutation report is a finding about your target

mutate measures your declaration. It emits no SARIF, never touches latest.json, and its exit code is its own — a killed mutant surfacing in a code-scanning UI would be exactly the false alarm this verb exists to prevent. A mutant that fails is expected to fail; that failure is the evidence your check works.

And a mutation score of 1.00 does not mean the boundary is enforced. It means every mutant the engine could generate and evaluate changed a verdict.

The six operators¶

Tier 1 mutates the configuration only. The target is observed and never altered; the config is deep-copied before every perturbation.

Operator	Mutation	Survives when
`M1` invert-denial	an identity declared DENIED is re-declared to receive the success status the owner actually got	the identity was itself served that success status and the declaration accepted it anyway — the expectation cannot tell allowed from denied
`M2` invert-allow	an identity declared to SUCCEED is re-declared to be denied	the identity is in fact being denied and the declaration tolerated it — nobody would notice the owner losing access
`M3` drop-marker	an `excludes_marker` is inverted to `contains_marker`	nothing observed shows this path can surface the owner's marker at all, so the oracle could never have fired
`M4` strip-credential	the identity's request is re-issued with its credentials removed	the target answers the identity exactly as it answers a caller with no credential, and that answer is an authentication refusal
`M5` positive-control	the check is re-declared as "every identity is refused", corroborated by a request for a resource id that cannot exist	no identity the denial exists to guard was served, or the route answers `2xx` for the fabricated id too — a soft-404 envelope
`M6` sham-credential	the identity's request is re-issued with a fabricated credential belonging to nobody	the target answers the identity exactly as it answers that nobody: nothing establishes that this identity is a principal the boundary even applies to

M3 inverts rather than deletes, because deleting an excludes_marker from a correctly behaving target changes nothing — the assertion passed before and its absence passes after — so a literal deletion is unkillable by construction. contains_marker gets no mutant at all: it fails the moment the marker is absent, so a passing one has already demonstrated the oracle fires.

M4 and M6 are both needed. M4 asks whether the credential is read; M6 asks whether the credential names a principal this boundary applies to. A target that answers 401 to a missing header and 403 to an unrecognised one kills M4 for an identity whose token is expired, typo'd, or scoped away by a coarser gate that fires before the boundary under test is ever reached.

Three of the six reach the network. M4 sends a real unauthenticated request, M6 a real request with a fabricated credential, and M5 one request for a fabricated resource id. All three are refused outright for any method outside GET, HEAD and OPTIONS, all three are counted in the request budget, and --min-interval throttles all three.

What a mutant's outcome means¶

Outcome	Means	In the score?
`killed`	the verdict changed in the direction the operator requires	yes
`survived`	the verdict did not change. The check may be vacuous	yes, as a miss
`aberrant`	the verdict changed the wrong way. A fabricated credential outscoring a real one is not a kill	yes, as a miss — and it forces exit `2`
`errored`	the engine does not know — including a baseline probe the target never answered	yes, as a miss — and it forces exit `2`
`not_applicable`	no mutant was generated. Either structural, or a limit on the evidence	no
`skipped`	no mutant was generated by policy, not by measurement	no

A surviving mutant means this check may be vacuous — that its green result may be worthless. It never means your target is defective. The report's wording is enforced by a test, and the local viewer renders survivors amber rather than red for the same reason.

The survivors are the deliverable. The score alone is a vanity number, so every survivor carries a plain-English diagnosis and the text report prints them before the number is mentioned again.

`skipped`: nothing to mutate¶

Reported with its reason, so a shrinking population is visible:

  skipped, by reason -- no mutant was generated by POLICY, not by measurement:
      1 x the identity was already served 2xx here, so there is no denial
      whose meaning is in question
      1 x the identity declares no credential header, so a fabricated one (M6)
      would leave it already indistinguishable from a caller the target does
      not know
      1 x the request carries no credential to remove, so an unauthenticated
      copy (M4) would be the same request

`not_applicable` and LIMITS ON THE EVIDENCE¶

not_applicable covers two things that look identical in a summary and mean different things.

"There was nothing here to mutate" is structural. It recurs identically every run, and keeping it out of the score is correct — a suite must not improve its score by being unmutatable.

"This target cannot show us" is a limit on the evidence, and it is printed under its own heading beside the score:

LIMITS ON THE EVIDENCE (2) -- not scored, and not survivors: nothing here was measured either way
  order-tenant-isolation
      outsider: this operation answers HTTP 404 both to this identity and to a
      credential it could not have recognised, which is what hiding existence
      uniformly looks like. That answer says nothing about who the caller is,
      in either direction, so whether this identity is a principal the
      boundary applies to cannot be measured from outside this target.
  widget-read-permissive
      this check's only evidence is the status code. A status matches whether
      or not the response body carried another identity's data. Add
      contains_marker on the owner and excludes_marker on the other identities
      to get data-flow evidence.

These are printed precisely because they leave a 1.00 with an empty survivor list, which reads as a perfect result rather than as an unanswered question.

An unanswered baseline is errored, not not_applicable

A probe lost to a rate limit is neither structural nor permanent — that mutant would have been evaluated on a quieter afternoon. It used to be dropped from the denominator, which took one measured run from 15 generated mutants to 11 while still reporting 1.00 and exit 0. A third of the population gone, and the number describing it unchanged. errored makes that run exit 2.

The two denominators¶

The report prints both, because they answer different questions.

Line	Arithmetic	Answers
`Mutation score: 0.92 (12 killed / 13 generated)`	`killed / generated`	of the mutants that were evaluated, how many did your suite kill?
`0.86 of 14 generated + limited (1 could not be evaluated)`	`killed / (generated + limited)`	how wide is that first number?

Both are in the JSON as score / generated and score_of_evaluable / evaluable, alongside limited. A per-operator breakdown carries the same keys, so 0.80 from four status kills and one M4 survivor is legible as a different suite from one with five even results. The score is deliberately unweighted: a weight is a claim about which vacuity matters most, that claim is target-specific, and no evidence in this tool could falsify it.

`--require-evaluable`¶

The limited count is a caveat, and a caveat that cannot move a number cannot gate a build. --require-evaluable exits 3 when any mutant could not be evaluated at all — which is what stops a suite whose only cross-tenant assertion cannot fail from reporting 1.00 and exiting 0.

It is checked after the score's own exit code, so a survivor still returns 1.

What it costs your target¶

mutate re-issues every request you declared, DELETE included, and then issues probes of its own on top. It prints the number, the rate and the write posture before it opens a socket.

Veridor mutation -- this will re-issue your 5 declared request(s) to http://127.0.0.1:8766,
  plus up to 500 additional probe(s) of its own.
  rate:      2 req/s (0.5s apart), sequential
  measures:  your declaration. Nothing here is a finding about the target.

Flag	Default	Bounds
`--read-only`	off	refuses, with exit `2` before a single request, any verb outside `GET`, `HEAD`, `OPTIONS` — at either tier
`--min-interval`	`0.5`	seconds between every request, the re-issued baseline included. Pass `0` against a fixture you own
`--request-budget`	`500`	caps the additional probes only
`--max-requests`	`5,000`	bounds the whole plan, baseline included. A plan over the ceiling is refused with exit `2`, and the message names the number it refused

run is the gate and stays unthrottled. mutate is the before-merge analysis verb, so it takes map's posture instead — including map's rule that an oversized plan is refused, not merely narrated.

One deliberate asymmetry. map refuses a non-loopback target until you pass --acknowledge-target; mutate does not. map fans out from a specification across every identity, which is close to scanning, so it asks first. mutate re-issues the requests you declared in a configuration you wrote naming a base_url — exactly what run does — so gating it and not run would be incoherent.

Tier 2, briefly¶

Tier 1 mutates your config. Tier 2 mutates your target, using two commands you write: one that produces a build with a named guard removed, one that puts it back. Veridor does not parse your source and does not generate the mutation.

apply_command and restore_command are executed on your machine, as you, with your environment. They run as an argv list with no shell, so nothing in them is expanded or interpreted — but they are still your commands and they still run.

Four independent refusals, all before anything executes:

No --tier 2 without --hooks.
No --tier 2 without --i-understand-tier-2-runs-my-commands.
A hook with no restore_command is refused. There is no default, not even a no-op, because "I forgot the restore command" and "this target needs no restore" must not be indistinguishable.
The whole hooks file is validated together — hook 3 failing after hooks 1 and 2 have altered your target is a mess, not a diagnostic.

restore_command runs on every path. If it does not exit 0 the hook is reported errored, the report says the target may still be mutated, and every later hook is skipped rather than applied on top of a target in an unknown state. A tier-2 suite run is full of findings by design — it runs against a build with a guard deliberately removed — so its run documents go to their own directory and are the measurement, not results about your real target.

A worked example is in fixtures/tier2_guard_toggle.py.

Exit codes¶

Code	Meaning
`0`	every generated mutant was killed
`1`	at least one survived, so at least one check may be vacuous
`2`	at least one mutant was `aberrant` or `errored`; or nothing was generated at all; or the plan was refused before the first request, an output directory it cannot write included
`3`	`--require-evaluable` was passed and a mutant could not be evaluated

2 outranks 1, deliberately. aberrant and errored both mean the engine does not know whether those checks discriminate, and a run that generated nothing exits 2 rather than 0 — 0 claims every generated mutant was killed, which is vacuously true of an empty set.

Next¶

run — the gate this verb is auditing.
map — the operations your suite does not cover at all.
claims and limits — coverage and the mutation score multiply rather than add.

mutate — would my suite catch a regression?¶