Skip to content

CI/CD & Workflows

All automation lives in .github/workflows/. This page is the source of truth for what each workflow does, when it runs, and how to invoke it manually. The in-repo .github/workflows/README.md is a short pointer back here.

Workflow File Triggers
CI ci.yml push/PR to main/develop, manual dispatch
Integration Tests integration.yml PR + push to main on runtime paths
Extended Integration Tests integration-tests.yml manual dispatch only (full)
Install Confidence install-confidence.yml manual dispatch only (the same matrix also runs inside release.yml)
Release release.yml push of a vX.Y.Z tag, manual dispatch
Retracting a release release-retract.yml manual dispatch only
Pages — Docs pages-docs.yml push to main, manual dispatch, and workflow_call from release.yml (no tag trigger of its own)
Pages — Helm Repo pages-helm.yml manual dispatch and workflow_call from release.yml (no tag trigger of its own)
CodeQL codeql.yml push/PR to main, weekly schedule

Shared Go setup/caching lives in the composite action .github/actions/setup-go.

CI

ci.yml — runs on every push and PR to main/develop. Fast feedback; the gate that blocks merge. Jobs:

  1. Generated Artifacts In Sync (check-drift) — runs make check-drift (sync-all + bundle, then git diff --exit-code). Fails if any committed CRD, RBAC, deepcopy, Helm CRD, or OLM bundle file is stale (untracked generated files fail it too). Fix locally with make sync-all and commit the result. The same job then runs the static checks that have no generator to catch them: make helm-lint, make check-csv-coverage, make check-apiref-drift, make check-cli-asset-names, make check-release-notes-links, make check-cli-docs, make check-docs-release-pins, make check-examples-catalog, make check-docs-tables, make check-crd-catalog, and a gitleaks secret scan. All are static and seconds-fast; the failure summary distinguishes drift failures from the rest.
  2. Go Lint (go-lint) — builds golangci-lint against the current toolchain and runs the whole .golangci.yml (ci.yml calls it a merge gate), then cross-compiles kubectl-neo4j for every released platform (darwin, linux, windows) so a unix-only import fails here rather than at release time.
  3. Unit Tests (unit-tests) — make test-unit (race-enabled, envtest-backed controller suite + plain unit tests). No external cluster required. The job runs the suite through gotestsum (make test-unit GO_TEST_CMD="./bin/gotestsum …"), which emits a JUnit XML + test2json report; a summary step (scripts/gotest-summary.sh) writes the failed and slowest tests to the GitHub step summary.
  4. Invariant Guards (advisory) — runs scripts/check-invariants.sh and scripts/check-knowledge-drift.sh with continue-on-error, so a violation is reported in the run summary but never fails the job or blocks merge. See Agent guardrails.

Integration coverage lives in its own workflows, not in ci.yml: the fast contributor lane is Integration Tests; the full matrix is Extended Integration Tests.

Caching

The workflows cache the expensive, slowly-changing inputs so reruns stay fast:

  • ./bin tools + envtest assets — cached by the .github/actions/setup-go composite action, keyed on hashFiles('Makefile') (the tool versions are pinned there), so kustomize/controller-gen/ginkgo/setup-envtest aren't re-downloaded every run.
  • Go build/module cache (GOCACHE) — restored via actions/setup-go's built-in caching, keyed on go.sum.
  • Operator image layers — the integration lanes build with Buildx and a GitHub Actions layer cache (cache-from/to: type=gha), so unchanged build stages are reused.
  • Neo4j image tarball — the pulled neo4j:<tag>-enterprise image is saved to /tmp/neo4j-image.tar and restored with actions/cache, avoiding a Docker Hub pull on every integration run.

Branch protection

main is a protected branch. The rules enforce the gates above:

  • No direct pushes — all changes land via pull request.
  • 1 approval, from a CODEOWNER; stale approvals are dismissed on new commits; review threads must be resolved.
  • Required status checks (strict / branch must be up to date), matching the CI job display names exactly:
  • Generated Artifacts In Sync
  • Unit Tests
  • Linear history (squash/rebase merges); force-pushes and deletion blocked.

Two things to know:

  • The integration checks are intentionally not required. The Integration Tests lane is path-filtered (it doesn't run on, e.g., a docs-only change) and the Extended Integration Tests workflow is manual-dispatch only — so neither runs on every PR; a required check that doesn't run on a given PR would leave it stuck at "Expected — waiting for status." Reviewers gate on them manually for code changes.
  • enforce_admins is currently off so the maintainer can merge during the solo→team transition (GitHub forbids approving your own PR, and the sole CODEOWNER would otherwise be unable to merge anything). Turn it on once a second reviewer is onboarded: 
    gh api -X PUT repos/priyolahiri/neo4j-kubernetes-operator/branches/main/protection/enforce_admins

If you rename a required CI job in ci.yml, update the protection contexts to match or every merge will block.

Test tiers: core vs extended

Every integration spec carries a Ginkgo label on its top-level Describe:

  • core — reconcile contracts a routine change is most likely to break: standalone → Ready, cluster formation, Neo4jDatabase/Neo4jUser/Neo4jRole/ Neo4jRoleBinding CRUD, config rendering, plugin install, basic TLS. Small, fast, deterministic on both tracks.
  • extended — multi-node scaling, split-brain, rolling upgrade, the full backup/restore matrix, property sharding, MinIO/cloud, MCP, ABAC/OIDC. Slow, resource-heavy, or version-gated; high value before a release, low marginal value per PR.

The two workflows below select by label. Run a tier locally with ginkgo run --label-filter='core' ./test/integration/... (or 'extended', or 'core || extended' for everything). When you add a spec file, label its Describe or it runs in neither lane.

Integration Tests

integration.yml — the fast contributor lane. Runs the core subset against both supported Neo4j tracks in parallel:

  • 5.26-enterprise — the last SemVer LTS; exercises the SemVer-only operator paths (V2_ONLY discovery, system_bootstrapping_strategy).
  • the pinned CalVer tag (currently 2026.08.1-enterprise) — the track new users deploy; catches strict-mode fatals (duplicate conf keys, Cypher-25 defaults) that 5.26 tolerates.

Because it's the core subset and the two cells run in parallel, wall-clock ≈ the slower (CalVer) cell, not the sum. Triggers on pull_request + push to main when runtime paths change (internal/**, api/**, cmd/**, test/integration/**, config/**, Dockerfile, Makefile, go.{mod,sum}, the workflow itself) — never on docs-only changes.

This is the lane that should give a contributor a fast, legible yes/no on the contracts they touched, on the versions users actually run.

Caveat — a new push cancels the in-flight run. Each integration lane uses a per-PR concurrency group with cancel-in-progress: true, keyed on the PR number (integration-core-${{ github.event.pull_request.number || github.run_id }} and the Extended equivalent). Pushing a new commit while a run is still going terminates that run and starts a fresh one on the new head — even a docs-only or trivial push (on pull_request the path filter is evaluated against the cumulative base→head diff, so any push re-fires the lane if the PR has ever touched a runtime path). The cancelled run shows up as red/"cancelled", not a failure. If you're waiting on a green integration result, let it finish before pushing again, or batch your changes into one push.

Extended Integration Tests

integration-tests.yml — the full suite (core + extended, ≈90–150 min) against a real Kind cluster, on the pinned CalVer track. This is the release-readiness and deep-coverage run.

Extended never runs automatically — by design, the default PR/push signal is the fast core lane, keeping the dev cycle short. The extended suite is manual-dispatch only; there is no nightly schedule and no PR-label trigger (both were intentionally removed). Run it on demand from the Actions tab ("Run workflow") or with gh workflow run integration-tests.yml --ref <branch>, full suite, with inputs:

  • neo4j-version — the exact image tag, used verbatim as neo4j:<tag> (default the pinned CalVer, which already satisfies the CalVer-gated specs, property sharding included; pass 5.26-enterprise to verify the LTS floor). Dispatch against your branch to run the full suite before merging.
  • timeout-minutes — default 150 (CalVer is ~2× slower per spec).

Dispatch it explicitly for backup/restore/sharding/coordination changes that the core subset doesn't exercise, and on the release commit before tagging (see the release process below).

It builds and deploys the operator, runs the selected scope, uploads logs/cluster-state artifacts, and tears the cluster down.

Not everything labelled extended runs in this lane. The tier label only selects the lane; a runtime Skip can still exclude a spec there. The resource-heavy property-sharding suites self-skip in CI (they need the production 4Gi/server floor) and are local-only — only the minimal property_sharding_ci_smoke runs here. See Testing → What runs where for the full map and how to run the local-only suites.

Bumping the CalVer pin: the version is pinned (not floating) for deterministic CI, in two places that must move together (GitHub allows neither an env var in a matrix nor an expression in a dispatch-input default, so there's no single shared variable):

  1. the neo4j-version matrix list in integration.yml (the core lane), and
  2. the neo4j-version input default in integration-tests.yml (the Extended lane).

Bump both in the same PR — the bump is itself a tested change.

Neo4j version support: validation gates and the next LTS

The user-facing policy is Supported Neo4j Versions. What enforces it:

  • Hard-reject (version validator): anything older than the current LTS (pre-5.26 today) or a line past its Neo4j EOL.
  • Allow, don't block "newer than the validated anchor" within a supported track — a brand-new CalVer must not be rejected the day it ships.
  • CI anchors: integration suites run 5.26-enterprise + the latest CalVer. Invariant: exactly two anchors steady-state; a transition window may run three.

When the next LTS lands, the change is small and contained — touch:

  • the version validator's allowed/minimum set (internal/validation/),
  • the CI matrix anchors (.github/workflows/integration.yml, integration-tests.yml),
  • the "Supported Neo4j versions" line in CLAUDE.md,
  • the matrix at the top of Supported Neo4j Versions.

At the old LTS's EOL, raising the floor also lets you delete its now-dead version gates (e.g. the SemVer-only discovery paths once 5.26 is dropped).

Install Confidence

scripts/install-confidence.sh — the install/upgrade/uninstall matrix on a throwaway Kind cluster. Run locally with make install-confidence (~10–15 min; needs kind, helm, kubectl, docker), or dispatch install-confidence.yml manually. It builds the operator image from the working tree, loads it into a fresh Kind cluster, and walks five legs:

  1. Helm install, cluster mode — default values, then a smoke Neo4jEnterpriseStandalone CR proving the operator actually reconciles (StatefulSet appears).
  2. Helm install, namespaces mode + rbac.perNamespaceRoles — asserts the manager permissions land as a namespaced Role in the watched namespace and NO manager ClusterRole is created.
  3. Helm upgrade from the previously published chart — installs the latest released chart, applies the new CRDs with kubectl apply --server-side -f config/crd/bases/ (the step real users must run — Helm never upgrades crds/), upgrades to the working-tree chart, and probes a new-in-this-release CRD field to prove the refresh took. PREV_CHART_VERSION=<x.y.z> pins the starting chart (empty = latest published).
  4. Documented-order uninstall with a live CR — CRs first (finalizers need the running operator), then helm uninstall, then CRDs; asserts nothing wedges.
  5. kubectl server-side apply path — install, re-apply (idempotence), and ordered teardown of the non-Helm manifests.

The same matrix runs as a blocking job in release.yml — a release tag cannot publish an image or chart if any leg fails. Use the local run (or the dispatch workflow) to catch problems before tagging instead of failing the tag.

Output contract (#247): every sub-step prints a [HH:MM:SS +elapsed] prefix and waits pre-announce their timeout (helm --wait 180s/300s, rollout 180s, smoke poll 60s) — so a watcher can always tell slow from stuck. The docker build logs to /tmp/ic-build.log (tailed on failure). On any failure the script dumps cluster diagnostics — pods, the operator Deployment + its logs, recent events, helm releases — before exiting, and in Actions a verdict table (every passed check + the failing one) lands in the step summary on both pass and fail.

Release

release.yml — tag-driven release pipeline. Push a vX.Y.Z tag (or dispatch with a tag input). Jobs:

  1. determine-tag / validate-release — resolve the tag and run build/test validation, including make check-drift (a tag on a commit that bypassed per-PR CI must not ship stale generated artifacts).
  2. install-confidence — the full install/upgrade/uninstall matrix on Kind, in parallel with validate-release. Blocking: build-and-push depends on it, so a broken install path fails the release before anything is published.
  3. build-and-push — multi-arch (linux/amd64,linux/arm64) image to ghcr.io/priyolahiri/neo4j-kubernetes-operator, signed with Sigstore Cosign keyless (id-token: write OIDC — no long-lived secrets).
  4. create-release — assembles the kubectl-applyable bundles (neo4j-kubernetes-operator-complete.yaml, …operator.yaml), stamps the OLM CSV via make bundle-release (with a guard that refuses to publish if createdAt: is still the dev placeholder), renders the release body from .github/release-notes-template.md (its docs links point at the release's own vX.Y docs), and publishes the GitHub release.

The release workflow's final jobs (publish-docs, publish-helm) then call Pages — Docs and Pages — Helm Repo (below) once every gate and artifact exists, so a single tag publishes images, the release, the docs version, and the chart.

Cutting a release (runbook)

Releases are tag-driven. Do not bump Chart.yaml — version/appVersion are the 0.0.1 placeholder in the repo and the workflows stamp the real value from the tag.

Before tagging (on an up-to-date, green main):

  1. make ship-prep — regenerates everything, builds the bundle, and runs helm-lint + check-csv-coverage. Review git status and commit anything regenerated.
  2. (Recommended) make install-confidence — the same install/upgrade/uninstall matrix the release workflow runs as a blocking gate. Running it locally first means a failure costs you minutes, not a dead tag. 2b. Dispatch the extended suite on the final main and wait for green: gh workflow run integration-tests.yml --ref main. The extended lane is manual-only — it never runs on its own, so a PR that only core lanes validated can carry a stale extended-lane expectation (or a real regression) that would otherwise surface AFTER the release. This dispatch is the release gate for the extended coverage; tag the exact commit the suite validated.
  3. If there are breaking changes or notable upgrade steps, add/extend the Upgrading between future releases section in migration_guide.md (the Upgrade Guide). 3b. Bump the documented install pins (Helm --version, the RELEASE_VERSION examples, the ArgoCD targetRevision, go install) to the new version on main before pushing the tag — make check-docs-release-pins (run in the check-drift job) lets docs run ahead of the newest tag but fails once the tag exists and the docs still name the previous release. make ship-prep does not run it; run it yourself.
  4. Draft the What's Changed notes — git log <last-tag>..HEAD --pretty=oneline is a good starting point. (The release workflow renders only the boilerplate from .github/release-notes-template.md; the changelog is hand-written.)

Tag and push from main:

git tag v1.12.0 && git push origin v1.12.0

This fires Release only. The versioned docs (/vX.Y/ + /latest/) and the classic Helm repo (/charts/) are published by the Release workflow's final jobs after every gate and artifact exists — a failed release publishes nothing (#245). Push-to-main docs (/main/) are unaffected. Any job failure auto-files an incident issue with the failed jobs and run link.

After the workflows finish:

  1. Paste the What's Changed section into the GitHub release body (above the generated boilerplate).
  2. Verify:
  3. gh run list — Release, Pages — Docs, Pages — Helm Repo all green.
  4. gh release view v1.12.0 — has …-complete.yaml and …operator.yaml assets.
  5. helm repo update && helm search repo neo4j-operator/neo4j-operator --versions — new version listed.
  6. Docs /latest/ and /v1.12/ load and the version dropdown shows the release.
  7. cosign verify ghcr.io/priyolahiri/neo4j-kubernetes-operator:v1.12.0 … succeeds.
  8. The release run's Install Confidence Gate job is green (it ran automatically — no manual fresh-Kind helm install needed).
  9. Comment on every issue the release fixes that was reported by someone else, naming the released version and asking the reporter to re-verify against it — externally-reported issues stay open until verified on a released version.

Retracting a release

release-retract.yml (manual dispatch, #246) safely unpublishes a dire release. Doctrine: retract and supersede — never reuse a version (registry layers, Helm caches, OLM catalogs and Rekor cache immutably). Always run with dry-run: true first: the dry-run doubles as the grace-window probe, printing either "window OPEN" (tag delete-and-re-cut still safe — nothing published) or "window CLOSED at " (retract + ship vX.Y.Z+1). Execution repoints the moving image tags at previous-good-tag, banners the GitHub release as RETRACTED, removes the chart from both channels, republishes docs from the previous good tag, and files a retraction-record issue with the manual follow-ups (notably: a MERGED OperatorHub PR can only be superseded, never yanked). Deleting the pinned image or the git tag are explicit opt-in flags, default off.

Publishing to GitHub Pages

Both Pages workflows write to the gh-pages branch and share a gh-pages concurrency group so their pushes serialize instead of racing. Both authenticate with the built-in GITHUB_TOKEN — no extra secrets.

Pages — Docs

pages-docs.yml publishes the MkDocs (Material) site as versioned docs using mike. It first runs mkdocs build --strict, so a broken internal link or a nav entry pointing at a missing page fails the publish — treat it as a docs gate on main and tags.

What gets published where:

  • Push to main → the /main/ alias, a rolling preview of unreleased docs. Does not touch latest.
  • Called from release.yml (the publish-docs job, after create-release; the workflow has no tag trigger of its own — publishing on tag push ran in parallel with the release gates and could advertise a version that never shipped, #245) → published as /vX.Y/ (the patch is dropped, so a later vX.Y.Z+1 overwrites the same /vX.Y/), and /latest/ is moved to it. mike set-default latest also points the site root at /latest/.
  • Manual dispatch → publish under an arbitrary version-alias, optionally updating latest.

For readers: the site root redirects to /latest/ (newest release), and the version dropdown at the top of every page (extra.version.provider: mike in mkdocs.yml) switches between latest, each released /vX.Y/, and the /main/ preview. A page added on main appears under /main/ immediately and lands in /vX.Y/ + /latest/ automatically on the next release tag — no manual step.

(The rolling-preview alias was historically called dev; it is now main.)

Pages — Helm Repo

pages-helm.yml packages the chart in charts/neo4j-operator and appends it to the Helm repository index under /charts/ on gh-pages, so:

helm repo add neo4j-operator https://priyolahiri.github.io/neo4j-kubernetes-operator/charts
helm repo update
  • Called from release.yml (the publish-helm job, after create-release; no tag trigger of its own) → packages and publishes that version.
  • Manual dispatch → package a specific existing tag.

OpenShift / OLM

OLM/OperatorHub is a supported manual install path but is not covered by CI — a smoke test needs an OpenShift cluster, which standard GitHub runners can't provide. Build and validate the bundle locally per OpenShift OLM.

Secrets & permissions

  • CI / Extended Integration Tests: none beyond the default GITHUB_TOKEN.
  • Release: id-token: write (Cosign keyless OIDC) and packages: write (push to GHCR). No stored signing keys.
  • Pages (Docs + Helm): contents: write to push to gh-pages; the built-in GITHUB_TOKEN.