Skip to content

Argo Rollouts

An Argo CD PostSync gate reports a bad version after it is live: a failed gate fails the sync and leaves the new version serving. A canary analysis is the blocking path. The new ReplicaSet gets a share, veridor run probes it through the canary Service before it is promoted, and a failed run aborts the rollout and keeps the stable ReplicaSet. Of the Kubernetes packagings, this is the only one that keeps a failing version from taking all the traffic.

The Veridor repository ships:

  • deploy/argo-rollouts/veridor-analysistemplate.yaml: an AnalysisTemplate named veridor-gate with one metric using the job provider.
  • deploy/argo-rollouts/rollout-example.yaml: a canary Rollout with canaryService / stableService and the steps setWeight: 50, analysis, setWeight: 100.

Not published

The template's default image, ghcr.io/veridor-authz/veridor:0.1.0, does not exist. Build the image from the repository's Dockerfile and pass it as the image argument.

The AnalysisTemplate

One metric, count: 1, failureLimit: 0, job provider. The Job is the same restricted-profile pod the veridor-gate chart renders (runAsNonRoot, uid 10001, seccompProfile: RuntimeDefault, no privilege escalation, read-only root filesystem, all capabilities dropped, no service-account token), with backoffLimit: 0 and activeDeadlineSeconds: 600. It runs:

veridor run --config /config/veridor.json --output-dir /work/.veridor \
            --base-url {{args.base-url}} --fail-on {{args.fail-on}} --require-complete \
            --wait-for-target {{args.wait-for-target}}
Argument Default What it is
base-url none, required The canary Service. A gate pointed at the stable Service measures the version already live.
config-map veridor-config ConfigMap holding veridor.json.
secret veridor-credentials Secret imported with envFrom; its keys must be the variable names the configuration references (VERIDOR_OWNER_TOKEN).
image ghcr.io/veridor-authz/veridor:0.1.0 The gate image. A placeholder.
fail-on new+confirmed --fail-on.
wait-for-target "60" --wait-for-target: seconds to wait, before the first request, for the canary Service to accept a TCP connection. The analysis can start as the canary pod turns Ready, before the Service routes to it; probes sent then are refused, and exit 3 aborts the rollout. It never changes a verdict. "0" turns it off.

The Job's other arguments are fixed in the template; edit it to add Veridor flags. The secretRef is optional: true: a missing Secret leaves the variables unset and Veridor refuses at once (exit 2, naming the variable), which fails the measurement instead of the pod waiting for its deadline.

A canary example

kubectl -n shop create configmap veridor-config --from-file=veridor.json
kubectl -n shop create secret generic veridor-credentials \
  --from-literal=VERIDOR_OWNER_TOKEN=... --from-literal=VERIDOR_OUTSIDER_TOKEN=...
kubectl -n shop apply -f veridor-analysistemplate.yaml
strategy:
  canary:
    canaryService: invoices-api-canary
    stableService: invoices-api-stable
    steps:
      - setWeight: 50
      - analysis:
          templates:
            - templateName: veridor-gate
          args:
            - name: base-url
              value: http://invoices-api-canary:8080
      - setWeight: 100

With no trafficRouting, Argo Rollouts splits by replica count and adds the pod-template-hash to each Service's selector, so the gate reaches the canary through invoices-api-canary while clients of invoices-api-stable stay on the stable ReplicaSet. The e2e measured the pinned selector on invoices-api-stable, and one ready canary pod and one ready stable pod when the analysis started. That pinning is also why the gate here probes the new version and not the one being replaced, which a PostSync gate on a RollingUpdate app does not guarantee (Kubernetes and Argo CD). With a traffic router (Istio, NGINX, ...) the same steps shift traffic by weight; that setup was not measured.

The wait-for-target wait holds the first request until the canary Service accepts a TCP connection. An accepted connection means a port is open, not that the application behind it is ready.

Abort semantics: every non-zero exit aborts

The job provider has two outcomes, not three. A Job that completes is a Successful measurement; a Job that fails is a Failed one, and with failureLimit: 0 one failure aborts the rollout.

Veridor exit Job Measurement Rollout
0 Complete Successful continues to setWeight: 100, promoted
1 findings Failed Failed aborted (status.abort: true), status.phase: Degraded, stable ReplicaSet kept
2 refused Failed Failed aborted
3 something went unmeasured Failed Failed aborted
70 internal error Failed Failed aborted
143 SIGTERM (for example the Job deadline) Failed Failed aborted

There is no Inconclusive outcome in the job provider, so exit 3 aborts the rollout exactly as a finding does, and so does a configuration mistake. That is the fail-closed direction, and it means a canary that could not be measured is not promoted. The only way to let a partially measured canary through is Veridor's own flags: the template passes --require-complete, and removing it lets a run with some unanswered probes pass. A run that answered nothing still exits 3 without it.

Measured by scripts/e2e_kubernetes.sh (Argo Rollouts v1.10.0, kind), as recorded in the Veridor repository's deploy/argo-rollouts/README.md, with this template as-is and the example Rollout's pod replaced by the demo API; not re-run for this page. The same script ran in a kind cluster in the project's CI on GitHub (workflow run 42508338432, success; the repository is private):

Step AnalysisRun Gate exit Rollout
revision 1, fixed (first rollout; steps are skipped) none — Healthy
revision 2, vulnerable canary Failed 1 Degraded, status.abort: true, stable ReplicaSet unchanged
revision 3, fixed canary Successful 0 Healthy, promoted

Exits 2, 3, 70 and 143 were not produced inside a Rollout; their rows above follow from the job provider's two outcomes and the Job failing on any non-zero exit.

Returning to exactly the stable spec is treated by Argo Rollouts as going back to stable, skipping the steps (its documented behaviour; not measured), so it does not exercise the gate.

Related: Kubernetes and Argo CD, exit codes, install.