Release Verification

The pre-release verification journey: a manual + LLM-driven walk of the real user scenarios on a clean cluster, run from main before cutting any release. It is the canonical source of truth for what we verify, on which deployment, at what size, and why — keep it current as the product grows.

Why this exists alongside the automated suites

Unit and integration tests (Testing) validate the code against itself. This journey validates the published instructions and end-to-end behaviour against a clean machine — the only check that catches a doc that lies, a dead image tag, a tutorial ordering bug, or a CR-name/logical-name mismatch. Real passes have caught all of these (most recently the sharded target.name doc bug, a #270 follow-up).

The verify-journey skill executes this document. LLM agents and developers should follow the matrix below verbatim; the skill holds only the mechanics (build, kind-load retry, teardown, reporting) and defers to this page for what to test.

Ground rules

• KIND only. Enterprise images only (neo4j:*-enterprise) — the project invariants.
• Follow the published docs literally for Phases 1–2. A wrong/out-of-order step is a finding — record the exact doc location, don't silently fix it.
• One Enterprise deployment in the cluster at a time (the anti-wedge rule). Running standalone + cluster + sharding JVMs concurrently on a laptop VM wedges Bolt (the HTTP probe reports Ready but ports 7687/6362 time out). Phases run sequentially with full teardown between them; the Kind cluster, operator, and cert-manager stay up across phases. One deliberate exception: Phase 5 Part C runs two small clusters concurrently, because network-mode replication cannot be tested sequentially — kept small (2 servers each) to bound the wedging risk this rule exists to avoid.
• Restore must be walked on BOTH standalone and cluster — the mechanism differs (see the routing table) and one never covers the other.

Capability routing — why each lands where it does

Each capability is verified on the minimum deployment that actually exercises its code path. Standalone is the cheap workhorse; a capability moves to the cluster only when it needs clustering.

Capability	Where	Why
Database lifecycle (create/show/drop)	Standalone	Same CREATE DATABASE Cypher everywhere
Database topology (n PRIMARIES m SECONDARIES)	Cluster	Meaningless on standalone (validator only warns)
Users / Roles / RoleBindings	Standalone	Auth model is deployment-independent
Privilege matching (stored form; learn vs probe mode)	Standalone and Cluster	The diff compares against how Neo4j stores each statement (#409), learned from the operator's own grants (learn, the default) or a throwaway role (probe) (#410). The rendering is deployment-independent, so standalone covers the rules — but learn mode's before/after reads and probe's reads are only correct if they are causally ordered with the grant, which only a multi-server cluster actually tests
Upgrading from the previous release	Start of Phase 1, and Phase 3	Two changes promise nothing happens on upgrade: learn mode must revoke nothing on existing roles, and serverDefaultCypherLanguage must record what a deployment already runs without rewriting neo4j.conf (a rewrite rolls the servers — observed once, on a sharding cluster). Only an object created by the previous operator exercises either
Restarts the operator avoids (live spec.config, scale, upgrade; v1.20+)	Standalone (5.26) and Cluster (CalVer), plus Phase 2b's 5.26 cluster	The restart decision is made separately per kind: the cluster's ConfigMap hash with restart-neutral startup blocks, the standalone's semantic hash. Scaling exists only on a cluster, and the 5.26 async-channel block only on a 5.26 cluster, so only a 5.26 cluster created under the previous release shows that upgrading restarts nothing there
TLS reload on renewal, and the issued Secret on deletion (v1.20+)	Phase 2b: a CalVer TLS cluster created under the previous release, and a fresh CalVer TLS standalone	Reload needs CalVer 2025.03+ and cert-manager, and the Phase 1 standalone is 5.26. An upgraded cluster is the only place the "enabled at the next restart" path (RestartPending, TLSCertificateNeedsRestart) shows; a fresh standalone shows reload enabled from the start
Neo4j telemetry into OpenTelemetry (spec.monitoring.logs, the Prometheus endpoint; v1.20+)	Standalone (5.26) and Cluster (CalVer), with the example Collector	The log configuration is rendered per line (debug.log is text on 5.26, JSON on CalVer) and reaches each kind through a different ConfigMap mount (/conf vs /operator-conf), so both are walked. The Collector is not a Neo4j JVM, so it does not count against the one-deployment rule
Server default Cypher language (serverDefaultCypherLanguage)	Standalone (5.26) and Cluster (CalVer), Sharding for the upgrade	The two lines behave oppositely — nothing is ever written on 5.26, where the setting does not exist; CalVer writes CYPHER_25 for a new deployment — so each phase covers its own line
Plugins (APOC, …)	Standalone (ConfigMap path)	Cluster uses the env-var path; ConfigMap path is the cheaper of the two to smoke
Backup → restore	Both	Both restore via the in-place Cypher path, but the standalone differs in its target plumbing (StatefulSet name, pod labels, which CR carries the seed env) and is the only target with an offline neo4j-admin path (a point in time it cannot restore online, source.type: storage, cloud by pod identity without spec.podServiceAccountAnnotations — rules 108, 111, 112). A point-in-time restore runs online on a CalVer cluster (into a new database) and is refused there otherwise.
Split backup archives (splitArchivePartSize)	Standalone and Cluster, on a 2026.09+ image (not the CI anchor)	The parts are written by the backup Job and read by every consumer, so the walk needs both restore paths (offline neo4j-admin on a standalone, online seedURI from cloud and through the PVC proxy) plus PVC retention. A backup larger than one part needs >1GiB of incompressible data, which is why this runs outside CI
Property sharding	Cluster (CalVer)	Cluster-only by nature; needs a CalVer image and is memory-heavy
Aura orchestration (Aura* CRDs)	Phase 4 — no Kind deployment; needs Aura API credentials	Talks to a cloud API, not a local DBMS, so no phase above exercises it. Read-only checks need only a client ID/secret; write checks mutate real cloud resources — see the phase for what is and is not safe
Aura Fleet Management	Standalone (+ Aura creds for provision)	The plugin/registration path is deployment-independent, so the cheap phase covers it; operator-driven provision additionally needs Aura API credentials
Cross-cluster replication (Neo4jReplicaDatabase, Neo4jReplicaPromotion)	Phase 5 — two deployments, run SEQUENTIALLY; needs a 2026.08+ image	Needs an upstream and a downstream, which looks like it breaks the one-deployment-at-a-time rule — but backup-based replication couples the two only through a bucket, so the upstream can be torn down before the downstream comes up. See the phase
The CCDR proxy + cross-cluster TLS trust, together	Phase 5 Part E — two Kind clusters, make ccdr-e2e-up	The proxy, the certificate SANs, the advertised-address override and peer-CA trust only interact when TLS is on across two real clusters. Nothing smaller reaches it: the IP-SAN bug, the peer-CA mount bug and the unrepairable-forming-cluster bug were all invisible until this ran
Database aliases (Neo4jDatabaseAlias)	Standalone	Alias DDL is deployment-independent; the CCDR failover behaviour is covered in Phase 5
Composite databases (Neo4jCompositeDatabase)	Standalone	A composite stores no data and has no topology, so clustering adds nothing to its code path — it is alias DDL plus an ordering guard. A remote constituent would need a second DBMS; pointing one back at the same server proves the keystore path without a second Enterprise JVM, so it stays here too
Operator→Neo4j TLS verification (CA path + tls.crt pinning)	Standalone	Standalone never reads ca.crt server-side (no cluster SSL policy), so removing that key from the TLS Secret isolates the operator's client verification with nothing else changing. On a cluster the same edit also breaks intra-cluster mTLS and the two failures are indistinguishable
kubectl-neo4j — offline half (validate, explain <term>, exit codes)	Phase 0, no deployment	It reads manifests and its own guidance map. Nothing it checks needs a running database, so it costs no memory and can run before the first pod exists
kubectl-neo4j — the failure paths (diagnose, preflight)	Phase 0, on resources built to fail	diagnose exists for the layer below the CR — an unschedulable pod, ImagePullBackOff, a PVC that never binds. Those have to be caused on purpose, and a deployment that never schedules starts no JVM, so this is the one place they can be produced without touching the one-deployment-at-a-time rule
kubectl-neo4j — the healthy lens (status, connect/cypher, support-bundle, validate --connect)	Rides Phases 1–3, no deployment of its own	The CLI is a lens on whatever is deployed. Giving it a phase of its own would mean standing up a fourth Enterprise JVM for it to look at — forbidden by the anti-wedge rule, and it would show nothing the deployments already in Phases 1–3 do not
kubectl-neo4j export replica-database	Phase 5	Its only inputs are an upstream Neo4jBackup.status.replicationPullURI and an upstream cluster's status.internalAddresses, which exist nowhere else in the journey

The phase plan

Run Phases 0-3 every time (Phase 3 included — sharding regularly surfaces issues the lighter phases miss). Phase 0 is minutes and no memory, and it goes first for a second reason: it proves the operator, the CRDs and the CLI are wired before a real deployment costs twenty. Phase 4 (Aura) needs cloud credentials rather than a Kind cluster: run its read-only sweep whenever anything under internal/aura/ or an Aura* CRD changed. Phase 5 (CCDR) is now a standing gate, not an occasional one. Run it whenever a release touches anything under the replication CRDs, internal/resources/ccdr_proxy.go, the CCDR proxy controller, or spec.tls — that last one because the proxy, the certificate SANs and cross-cluster CA trust are one system, and Part E is the only place they are exercised together. It has found a release blocker on every pass so far. The image floor is 2026.08+, which is the pinned CI anchor as of v1.15.0, so "no such image" is no longer a reason to skip it; if a pass really is skipped, say so in the log rather than recording the phase as passed.

Phase 0 — CLI (no Neo4j deployment)

kubectl-neo4j ships on the operator's own tag, under the operator's own support statement, so it is verified on the same pass. It is not a deployment but a lens on one — which is why most of its scenarios ride Phases 1–3 (see the routing table). What lands here is everything that needs no healthy database: the offline half, and the failure paths that have to be caused deliberately.

Build it from the same tree as the operator and reach it through kubectl, because plugin discovery by binary name is part of what is being verified:

make build-cli          # stamps -X main.version=$(git describe --tags --always --dirty)
export PATH="$PWD/bin:$PATH"
kubectl plugin list | grep kubectl-neo4j

Part A — offline (no cluster contact):

Scenario	Verify
Plugin discovery	kubectl neo4j prints usage when invoked through kubectl, not only as ./bin/kubectl-neo4j. The binary name is the contract
The good case	kubectl neo4j validate -f config/samples/ → exit 0
The bad case	a manifest with a known-bad spec (spec.config.dbms.default_database, an out-of-range serverRoles[].serverIndex) → one line per finding, errors before warnings, each sorted by field path, exit 1
Every error, not the first	one manifest broken in several places at once — a bad spec.image, a deprecated spec.config key, an out-of-range serverRoles index, a missing acceptLicenseAgreement — must report all of them from a single run. The fix-one-rerun loop that #354 removed must not be back
Ruleset banner	every non---quiet run ends validated against operator rules <version>. On a journey build that is a git describe string such as v1.14.0-41-g3aa9f21, not a release version — only a tagged build prints a clean one. If it reads dev the ldflags did not apply, and the skew check in Part B is silently disabled
Skip taxonomy — the distinction that matters	a kind whose validator resolves cross-references (Neo4jDatabase) skips with "resolves cross-references; re-run with --connect". A kind with no operator-side validator at all (Neo4jRestore, Neo4jReplicaPromotion, any Aura*) skips with "no operator-side validator … kubectl apply --dry-run=server". Sending a user to --connect for a check that does not exist is the exact bug this wording was written to prevent
Pending is not failure	a manifest whose dependency cannot be satisfied yet renders … and does not change the exit code, including under --strict
Exit-code contract	0 clean · 1 a validation error, or a warning under --strict · 2 a usage problem (bad flag, unreadable file, undecodable YAML). Documented as stable, so CI users pin them
stdin	rendering the Helm chart with helm template and passing it to validate -f - reports normally
explain a term	kubectl neo4j explain ServersHealthy → meaning plus guidance; explain --list enumerates everything it knows
explain a condition REASON	kubectl neo4j explain CompositeDatabaseNameBlocked → (reason), the ordering trap and how to clear it; explain --list has a Reasons: section. Reasons are the most specific thing a status carries and were unreachable before v1.16.0
explain admits a gap	an invented term (explain NotARealCondition) prints no explanation for …, points at --list, and exits 2 — it never guesses. The other half of this, an unrecognised phase on a live resource naming the CLI's own version, is only reachable against an operator newer than the CLI; note it as unreachable rather than recording it as passed
preflight on the cloud substrate	on Kind, a cluster with spec.crossClusterReplication.enabled reports the Kind case (… — no load-balancer controller), not a false exposure problem. A cluster with a hard zone constraint (topologySpread.enabled, default DoNotSchedule) on a single unlabelled-node Kind reports ✗ zones. A spec.securityContext override that drops runAsNonRoot in a namespace labelled pod-security.kubernetes.io/enforce=restricted reports ✗ spec.securityContext naming each missing field. All three are new in v1.16.0 and need no running database

Part B — the failure paths (operator up, no Neo4j running):

Every resource below is built not to start, so none of them runs a JVM and none of them counts against the one-deployment-at-a-time rule. Apply, read, delete.

Scenario	Verify
Nothing ever reconciled	kubectl scale -n <operator-ns> deployment/<operator-deploy> --replicas=0, apply a standalone, wait past the 2-minute grace → status shows it with no phase, and diagnose reports "has no status after …" and names all three causes (operator not running, no RBAC for the kind, namespace-scoped and not watching). Scale back up afterwards. This is the only signal #282 has
Unschedulable	a standalone requesting more memory than any Ready node has → diagnose reports the pod as unschedulable with the scheduler's own message, plus the node-capacity guidance and the 1.5Gi Enterprise floor; exit 1
Image will not pull	the same manifest with a typo'd tag → ImagePullBackOff named as the cause, rather than the CR's Pending restated
StorageClass missing	spec.storage.className naming a class that does not exist → the operator refuses to build the StatefulSet at all, so there is no PVC and no pod: the CR goes Failed, status shows the operator's own message, and diagnose adds the StorageClassNotFound event. Then run preflight -f on the same file: it must have caught this before apply. Confirm the two agree — that pairing is the whole argument for preflight existing
PVC never bound	this appears in the unschedulable case above, not the missing-class one: with a WaitForFirstConsumer class the PVC stays Pending until a pod is scheduled, so diagnose reports it alongside the scheduling failure
preflight never overclaims	every run, clean or not, ends with the shape-only line (no bucket, registry or endpoint was contacted)
Backup credential shape	a Neo4jBackup whose credentialsSecretRef is absent, then one that exists but is missing a key the Job mounts → reported distinctly. This replaces the kubectl run backup-auth-check --image=amazon/aws-cli … ritual, which today is only reached after a backup has failed
Crash-loop is recognised	a setting Neo4j's strict config check refuses at startup — e.g. db.tx_log.rotation.size: "not-a-size" in spec.config, which validate passes — makes the server exit 1 on every start. (The old recipe, a 1Gi limit under a 2G heap, no longer reaches Neo4j: since v1.15.0 the operator's own heap-vs-limit check rejects it on standalones too.) diagnose must name the crash-loop with its restart count and point at kubectl logs --previous — the reason is in the log of the run that died. True OOMKilled / exit 137 needs a JVM that starts and then grows past the limit; record it as not run rather than faking it
Version skew warning	the kustomize dev deploy sets OPERATOR_VERSION=latest, which suppresses the check by design, so force it: kubectl -n <operator-ns> set env deployment/<operator-deploy> OPERATOR_VERSION=v0.0.1 → validate --connect prints ⚠ version skew naming both versions. Set it back to latest after. CI never has two versions, so this is reachable only by hand

→ Delete every broken resource before Phase 1, and confirm the operator is back to 1 replica.

Phase 1 — Standalone (1 pod, ~2Gi)

Neo4jEnterpriseStandalone → Ready, then on it:

Scenario	Verify
Standalone reconciles	status.phase=Ready, pod 1/1
Database lifecycle	Neo4jDatabase Ready; SHOW DATABASES shows it online. One named with two characters (w3) is in phase ValidationFailed, naming Neo4j's 3–63 rule, and nothing reaches the server
Users / Roles / RoleBinding	Neo4jUser/Role/RoleBinding Ready; SHOW USERS/SHOW ROLES. Reference at least one role by its hyphenated CR metadata.name (not spec.name) and confirm the grant lands (no RolesPending) and a RolesResolved event fires.
Upgrade from the previous release (run FIRST, before building from main)	Install the previous release (ghcr.io/priyolahiri/neo4j-kubernetes-operator:<previous tag>), then create a standalone and a Neo4jRole with a few privileges under it. Then deploy main without applying its CRDs, and then apply them. What the old CRDs do depends on the release: from v1.16 every Neo4jRole went Failed with the installed Neo4jRole CRD predates this operator (learn mode's status fields did not exist yet) and, once the CRDs were applied, Ready with PrivilegesSynced=Unknown/UnattributedPrivileges listing every pre-existing row; from v1.17 nothing fails at all. Either way SHOW ROLE … PRIVILEGES is unchanged at every step (nothing granted, nothing revoked), the standalone records status.effectiveCypherLanguage=CYPHER_5, its neo4j.conf is byte-identical to the old operator's (diff the ConfigMap), the StatefulSet's generation is unchanged and the pod did not restart. Remove one row by hand → it is restored (and, if it was unattributed, attributed). Also create a scheduled Neo4jBackup under the old release: from v1.18 it reads Ready=True/BackupScheduled as soon as main runs, where v1.17 left it Ready=Unknown forever. Then tear down and continue from a clean main install
Privileges match Neo4j's stored form	A role written in forms Neo4j rewrites — NODES *, ON DATABASES a, b, READ {name, age}, no segment, EXECUTE FUNCTIONS, an upper-case database name, and one granted through an alias — reaches PrivilegesSynced=True; SHOW ROLE … PRIVILEGES AS COMMANDS is unchanged across three forced reconciles (annotate the CR to poke it) and exactly one PrivilegesApplied event exists — provided the databases were Ready before the role was applied; applied together, the grants land in stages as each database appears (additions only, 0 revoked), which is convergence, not oscillation. Drift both ways: revoke one row that came from a list → restored; grant a foreign row → revoked; the next reconcile changes nothing
Learn mode (default)	status.privilegeRenderings lists each statement with the rows Neo4j stored (as SHOW shows them, backticks included); SHOW ROLES has no operator_privilege_probe_* roles at any point; restart the operator pod → zero new PrivilegesApplied (the state survived in status)
Probe mode	Redeploy the operator with --privilege-normalisation=probe (Helm privilegeNormalisation: probe) and repeat the stored-form row: same result, and afterwards SHOW ROLES YIELD role WHERE role STARTS WITH 'operator_privilege_probe_' returns 0 — each probe role is dropped. Switch back to learn before continuing
Dropping a database a role grants on	Delete a Neo4jDatabase while a role grants on it → a PrivilegeTargetDropped Warning on the role and on the database (an unaffected role gets none); the role stays Ready with PrivilegesResolve=False/DatabaseNotFound, and the operator log shows no reconcile-error storm (count "does not exist" errors over a minute: ~0, not ~20). Recreate it → the grant is back and PrivilegesResolve=True within seconds
privilegeRules	A role mixing spec.privileges with rules — ACCESS, MATCH with nodes: ["*"], READ on two properties of a label, a deny, a relationship segment, and a dashed database name — goes Ready and stores the expected rows. A rule naming a database that does not exist → PrivilegesResolve=False whose message says Named by spec.privilegeRules[N].onDatabase; apply that Neo4jDatabase → resolved in seconds, not the 30s requeue (that is the watch). A READ rule without properties → Failed naming spec.privilegeRules[N].properties, existing privileges untouched
Live config change (standalone)	Change a dynamic setting in spec.config (e.g. db.transaction.timeout): a ConfigAppliedLive event names it, the pod keeps its UID and restart count, SHOW SETTINGS YIELD name, value WHERE name = 'db.transaction.timeout' reads the new value, and it survives a pod delete. Change a static one (db.tx_log.buffer.size): a ConfigNeedsRestart event names it and the pod restarts
Logs on standard output (5.26)	Set spec.monitoring.logs.stdout: [query, debug]: the pod restarts once and a ConfigNeedsRestart event names server.logs.config; SHOW SETTINGS reads server.logs.config=/conf/server-logs.xml; kubectl logs carries one JSON object per query and per debug event, while /logs/debug.log stays Neo4j's text format. Change the list to [query, security]: no restart (same pod UID, restart count unchanged) and, within about two minutes, security events appear on standard output and debug events stop. Remove logs: one restart, the ConfigMap loses server-logs.xml, server.logs.config is back to the default. With server.logs.config also in spec.config: Failed, naming spec.config[server.logs.config], nothing rolled
Server default Cypher language (5.26)	The Phase 1 standalone records status.effectiveCypherLanguage=CYPHER_5 and its ConfigMap contains no db.query.default_language. A standalone with serverDefaultCypherLanguage: CYPHER_25, and one with db.query.default_language in spec.config, are each Failed naming the field, with no StatefulSet created
Plugin (APOC)	Neo4jPlugin Ready; an APOC function inside the plugin's allowedProcedures answers — with examples/plugins/apoc-plugin-example.yaml that is e.g. RETURN apoc.convert.toJson({a:1}). apoc.version() is outside that example's allowlist, so it fails as Unknown function while APOC works; that is the allowlist doing its job, not a failed install (standalone ConfigMap path)
Backup → restore (online)	add a marker node → back up (one-shot, PVC) → change the marker → restore over neo4j with replaceExisting → the marker returns. While it runs, poll a second database: it stays online throughout, the pod's restart count and UID do not change, no <restore>-restore Job exists, and a stopCluster: true on the restore draws an event saying it is ignored. A postRestore Cypher hook runs (rule 108)
Backup → restore (PVC differential, merged)	a scheduled PVC backup (backupType: AUTO) until a run records artifactType: DIFF, with data written between runs → restore it into a new database and over neo4j: online, no Job, every run's data present (the proxy's merge-chain init container merged the chain). On a 2026.09 image, repeat with splitArchivePartSize and with an allDatabases backup whose quiet database's newest run is empty. Then delete the chain's full from the PVC and restore again: Failed within a minute, carrying neo4j-admin's reason (rule 109)
Backup → restore (offline)	a point-in-time restore (source.pointInTime) on this 5.26 standalone without stopCluster: Failed, naming the reason (Neo4j 5.26 has no seedRestoreUntil; neo4j.com/point-in-time-path reads offline: …) → with stopCluster: true: the neo4j-admin Job restores the run status.resolvedSource pinned — the earliest that started at or after the point in time — to that point
Restore under a new name (standalone)	database: <new>: from a one-database backupRef (PVC and MinIO), and from an allDatabases backup with source.sourceDatabase — each new database is created online with the source's data. MinIO first without neo4j.com/auto-inherit-seed-creds: Failed, naming the standalone and the missing Secret/endpoint; then with it: exactly one restart, then Completed. On the offline path (source.type: storage + stopCluster: true) the Job's --from-path is a single file: the exact .backup file when backupPath names one, or, for a directory, the source database's newest file in it ($(ls '<dir>'/'<db>'-*.backup \| tail -1))
Standalone workload identity	The upgrade row's standalone (created under the previous release) keeps its pod UID, gets no <name>-neo4j ServiceAccount, and its StatefulSet metadata.generation does not change. Set spec.podServiceAccountAnnotations (e.g. eks.amazonaws.com/role-arn) → <name>-neo4j exists with that annotation and an owner reference to the standalone, and the pod is recreated with serviceAccountName: <name>-neo4j. Kind has no identity webhook, so stand in for one with the MinIO credentials in spec.extraEnvFrom + endpoint in spec.env, back up to MinIO, then remove credentialsSecretRef from the Neo4jBackup and restore into a new database: online, no Job, no restart. Remove the field → the pod returns to default and the ServiceAccount is deleted; the same restore is then refused, naming spec.podServiceAccountAnnotations
All-databases restore (standalone)	two user DBs → Neo4jBackup allDatabases: true → mutate both → Neo4jRestore allDatabases: true (options.replaceExisting: true); confirm both round-trip online, one database per pass, with no Job and no restart, status.databaseResults all Completed (system excluded) and status.completionTime set (#288, rule 108)
Standalone recommended labels	kubectl get pods -l app.kubernetes.io/name=neo4j returns the standalone pod (it carries app.kubernetes.io/{name,instance,managed-by})
Operator build metadata	scrape the operator's /metrics (kubectl port-forward -n <operator-ns> deploy/<operator-deploy> 8443:8443, then curl http://localhost:8443/metrics: the dev overlay binds :8443 and serves it as plain HTTP in --mode=dev, despite the --metrics-secure=true in its args — nothing listens on 8082, which this row named until 2026-10-02; production mode serves HTTPS with a bearer token when metrics.secure is on) and find neo4j_operator_build_info: exactly one series, value 1, with version, vcs_ref, build_date and go_version labels. vcs_ref must equal git rev-parse --short HEAD of the tree you built (a make deploy-dev-local build stamps version with the Makefile placeholder 0.0.1; only a release image carries a release version). A vcs_ref of unknown or a stale SHA means you are running an image that is not the tree under test
system is not restorable	a Neo4jRestore with database: system → Failed with an actionable message
Database alias	Neo4jDatabaseAlias → Ready; connect via the alias name and reach the target; SHOW ALIASES FOR DATABASE lists it. Then re-point spec.targetDatabase at a second database and confirm the same connection string now reaches the new one
Alias drift is corrected	ALTER ALIAS it elsewhere by hand → next reconcile restores spec.targetDatabase
Composite database	Run on BOTH lines — a CalVer deployment defaults to Cypher 25, which reads constituent names differently (knowledge rule 97; #426 was invisible on 5.26). Neo4jCompositeDatabase with two local constituents → Ready; status.observedConstituents lists them fully qualified (cineasts.latest), read back from the server rather than echoed from spec
Composite: where a constituent is reachable from	from a session on the composite (cypher-shell -d cineasts), USE cineasts.latest MATCH (n) RETURN count(n) succeeds and RETURN graph.names() lists the constituents. From the default database the same USE is refused with 42N04 — and following that error's own advice to "connect to cineasts.latest directly" fails with 22N51. Both halves matter: the guide tells users to ignore the server's suggestion, so a release that changed this would make the docs wrong. cypher-shell prints the messages without the GQL codes, so match on the text: "…only allowed when connected to it" for the first, "Unable to get a routing table for database 'latest'…" for the second (5.26.0, 2026-10-02)
Composite: the ordering trap is guarded	create CREATE ALIAS cineasts.latest by hand with no composite present, then apply a Neo4jCompositeDatabase named cineasts: the CR reports Failed with reason CompositeDatabaseNameBlocked and names the alias to drop. Drop it → the composite is created on the next reconcile. Neo4j's own 42N87 never mentions ordering, so the operator's message is the only route out
Composite: constituents are enforced	with enforceConstituents: true (the default), create an extra alias in the composite's namespace by hand → next reconcile removes it. Set false → a hand-made constituent survives and appears in observedConstituents
Composite: a GRAPH privilege on it is caught	a Neo4jRole granting MATCH {*} ON GRAPH <composite> → PrivilegesResolve=False with reason GraphPrivilegeOnComposite. Neo4j accepts and persists such a grant while it does nothing, so the operator's warning is the only signal
Composite: deletion removes aliases, not data	deletionPolicy: Delete → DROP COMPOSITE DATABASE ... CASCADE ALIASES; afterwards SHOW ALIASES has none of them and both target databases are still online with their data
Composite: remote constituent keystore	set spec.remoteAliasKeystore on the standalone, then a constituent with remote.credentialsSecretRef whose url points back at this same DBMS — enough to prove the keystore mounts and Neo4j can encrypt the alias credentials, without a second Enterprise JVM. Omitting the keystore must be refused at apply time naming the field, not inside the server with a 50N09/50N00 that mentions neither the CR nor the constituent. Walk it on the 5.26 LTS, not only CalVer: v1.16.0 shipped this path broken on 5.26 (every remote alias carried a CYPHER 25 directive the LTS rejects), and the URL must be neo4j+s:// or neo4j+ssc://
TLS: CA path	spec.tls.mode: cert-manager + ca-cluster-issuer → Ready, diagnostics collected, and no PINNING line in the operator log
TLS: certificate pinning	scale the cert-manager controller Deployment to 0 (so it cannot restore the key) — leave cert-manager-webhook up, or every Certificate write fails admission. (Before #423 the standalone wrote its Certificate on every reconcile, so a down webhook froze diagnostics; it now writes only on a change), kubectl patch secret <name>-tls-secret --type=json -p '[{"op":"remove","path":"/data/ca.crt"}]', then nudge a reconcile. Expect the operator to log verifying the server by PINNING its certificate from 'tls.crt' with a pinnedFingerprintSHA256 equal to openssl x509 -in tls.crt -noout -fingerprint -sha256, and status.diagnostics.collectionError to stay empty while lastCollected still advances at its refresh (diagnostics are written when they change or every 5 minutes, so allow up to 5) — the pinned connection is a real authenticated session, not a fallback that merely logs
TLS: pin rejects a mismatch	with ca.crt still absent, replace the Secret's tls.crt with a different certificate carrying the same SANs and do not restart the Neo4j pod (it keeps serving the keystore it loaded at startup). Next reconcile must fail with x509: certificate signed by unknown authority in status.diagnostics.collectionError. Restore tls.crt → the next reconcile recovers. This is the only place the pin is proven to reject; CI cannot reach it because every CI Secret has ca.crt
CLI: status on a healthy namespace	one kubectl neo4j status -n <ns> lists the standalone and its Database/User/Role/Plugin CRs with their phases; --problems prints only the all N Neo4j resource(s) look healthy line while everything is Ready. This replaces kubectl get against 26 kinds
CLI: connect tells the truth about TLS	run it around the TLS scenarios above: with TLS enabled, connect <name> prints bolt+s:// and states that plain bolt:// is rejected; before it, bolt://. The scheme is derived from the deployment, so a wrong one here is a real bug in the command customers reach for first
CLI: cypher never moves the password	kubectl neo4j cypher <name> -c "SHOW DATABASES" returns the same rows as the by-hand kubectl exec. connect prints no exec line at all — it points at kubectl neo4j cypher and says the credentials are already in the pod; it is cypher that execs, referencing $DB_USERNAME / $DB_PASSWORD by name (connect.go), so the value reaches neither your shell history nor the API audit log. Check by grepping connect's whole output for the real admin password: zero hits
CLI: explain against a live resource	kubectl neo4j explain Neo4jEnterpriseStandalone/<name> prints the phase, then each condition with the operator's own message and the guidance for it
CLI: support-bundle withholds the secret	support-bundle -n <ns> -o bundle.tar.gz, extract, and grep the whole archive for the admin password — zero hits. REDACTIONS.txt is present and enumerates what was withheld; Secrets appear as names, types and key names only; valueFrom references are still there, deliberately. Grepping for the real password is the check that matters — a redactor is only as good as its worst path
CLI: validate --connect and preflight on what is deployed	a Neo4jDatabase naming a cluster that does not exist → error; the same manifest naming this standalone → validated, and the kinds skipped offline in Phase 0 are no longer skipped. preflight -n <ns> with no arguments checks the deployed resources and still ends with the shape-only line

→ Tear down the standalone fully before Phase 2 — but KEEP the backup PVC (and the Neo4jBackup CR that owns it, so retention does not prune the artifact). Phase 2's cross-topology scenario restores that standalone-produced artifact into the cluster, and the one-deployment-at-a-time rule means you cannot stand the standalone back up alongside the cluster to re-make it. Deleting the PVC here costs a full standalone rebuild-and-teardown cycle later, so treat "a retained standalone backup" as one of Phase 1's outputs.

Phase 2 — Cluster, 3 servers (~6Gi)

Neo4jEnterpriseCluster with servers: 3 → Ready; pods {cluster}-server-0..2; SHOW SERVERS lists 3 Enabled/Available. Then:

Scenario	Verify
Cluster forms	3 members Enabled/Available; server-based pod names
Server default Cypher language (CalVer)	The new cluster records status.effectiveCypherLanguage=CYPHER_25, its neo4j.conf has db.query.default_language=CYPHER_25, and SHOW DATABASE neo4j YIELD defaultLanguage is CYPHER 25. Set serverDefaultCypherLanguage: CYPHER_5 → one rolling restart; a database created afterwards is CYPHER 5 while neo4j keeps CYPHER 25 (Neo4j fixes a database's language at creation)
Privilege matching on a cluster	Repeat Phase 1's stored-form row on the 3-server cluster: PrivilegesSynced=True, stable across forced reconciles, drift repaired both ways. This is the row that proves learn mode's before/after reads (and, with the flag, probe's) are causally ordered with the grant when the write lands on a leader the read might not
Database with topology	e.g. 3 PRIMARIES; SHOW DATABASE <db> shows the primaries online
Backup → restore (cluster)	in-place Cypher path: back up one DB (instanceRef + database), restore into a new database, confirm the data round-trips. Also one database out of an allDatabases backup under a new name with source.sourceDatabase
Point-in-time restore (online, cluster)	A scheduled MinIO backup (backupType: AUTO, every minute): after the first run write a Phase 1 node, note RETURN datetime() as the point in time, write Phase 2, wait for the next run (it holds both). Restore pointInTime into a new database: Completed online, no Job; neo4j.com/point-in-time-path=online; status.resolvedSource.backupStartedAt is the first run that started after the point; the database has Phase 1 and not Phase 2 (Neo4j truncated that run — it is not just an older run). The same restore over neo4j → Failed, naming the existing database, and neo4j keeps both phases
All-databases restore (cluster)	Neo4jBackup allDatabases: true → Neo4jRestore allDatabases: true (cloud-backed); confirm every user DB round-trips and status.databaseResults are all Completed (#222)
Cross-topology restore	restore the retained standalone backup from Phase 1 into the cluster via instanceRef; confirm the data round-trips. The cluster path stands up a short-lived backup-seed-proxy-<restore> Deployment that serves the backup PVC over HTTP. The seed URI has no status field of its own — it appears in Neo4jRestore.status.message (seedURI=…, while Running and once Completed) and in the RestoreStarted event (kubectl get events -n <ns> --field-selector reason=RestoreStarted), which carries seedURI=http://backup-seed-proxy-<restore>.<ns>.svc.cluster.local:8080/<path>/<artifact>.backup — an http:// URL, not a file:// path. The proxy Deployment appearing and then being reaped is the other half of the evidence
CLI: a healthy cluster reads as healthy	kubectl neo4j status shows the cluster Ready next to the topology-bearing Neo4jDatabase; diagnose reports 3/3 pods ready and exits 0 rather than inventing a problem. A diagnostic that cries wolf on a healthy cluster is worse than none
CLI: diagnose during a restart is not a failure	delete one server pod (kubectl delete pod <cluster>-server-1) and run diagnose inside the restart window: the not-ready pod is marked …, the exit code stays 0, and the pod returns. Enterprise takes tens of seconds to open Bolt, so an exit code that failed during a normal restart would be useless in the loop it exists for
A lost server is an outage, not formation (#444)	Delete one server pod: the cluster stays Ready throughout (watch .status.phase every few seconds — it must never read Forming), Degraded=True (reason ServersUnavailable) names the pod while it is down, ServersHealthy turns False and neo4j_operator_server_health drops to 0 for that pod's address — still one series per server, none left behind afterwards — and a Neo4jUser applied meanwhile reaches Ready. When it rejoins, Degraded=False and no ClusterFormationStarted event was raised. Kill one server's container instead (kubectl exec <pod> -c neo4j -- kill 1) and sample .status.phase every second: it never reads Forming either (the v1.20.0 walk caught it doing so once, while the client Service still routed to the dying container). Then hold a server down past the grace: redeploy the operator with --server-unavailable-grace=2m, kubectl cordon the Kind node and delete one server pod (it stays Pending; give the operator Deployment a toleration for node.kubernetes.io/unschedulable first if you will restart it while cordoned). At 2 minutes: phase Degraded, Ready=False/ClusterDegraded, one ClusterDegraded Warning; a new Neo4jUser still reconciles; an image change is held (UpgradeDeferred, StatefulSet image unchanged — revert it afterwards). Delete a second pod: phase Forming, one ClusterQuorumLost Warning, a new Neo4jUser waits with ClusterNotReady. kubectl uncordon → Degraded (already past the grace), then Ready; the waiting user reconciles. Walked on 2026.08.1, 2026-10-03
Scaling up restarts nothing	Under a loop writing a node every second, scale servers 3 → 5: Ready within about a minute, servers 0–2 keep their pod UIDs and restart counts, a ScaledWithoutRestart event, no failed write. SHOW DATABASE system YIELD address, role lists the two new servers as secondary, and each has /data/.neo4j-operator/system-database-mode reading SECONDARY. Restart server-3: still secondary. Scale 5 → 3: nothing restarts. (A scale-up that adds a system primary, such as 2 → 3, still restarts: that is a unit test, TestScale_RestartDecision)
Live config change (cluster)	Change a dynamic setting in spec.config: ConfigAppliedLive, no pod restarts, and SHOW SETTINGS on each server (kubectl exec <pod> -- cypher-shell -a bolt://localhost:7687 -d system …; -d system because the default neo4j database is not on every server) reads the new value. Remove it: each server reads the default again. Change a static one: ConfigNeedsRestart names it and the servers roll
Connections reused between passes	On the phase's quiet Ready cluster, count the operator's logins over 4 minutes, after it has run for a minute: kubectl exec <pod> -c neo4j -- grep -c 'logged in' /logs/security.log on each server, before and after. A handful (the per-pod split-brain checks, every 5 minutes), not ~17 a minute. Then delete one server pod: status.diagnostics shows it Unavailable and then Available again, and the operator logs no errors for the cluster
Clusters reconciled in parallel	The operator log shows Starting workers for neo4jenterprisecluster with "worker count": 4. Alongside the phase's cluster, apply three more clusters on an image tag that does not exist (e.g. 2099.01.0-enterprise: their pods stay in ImagePullBackOff, no JVM starts, so the one-deployment rule holds): the real cluster reaches Ready in its usual time, the three stay Forming naming ImagePullBackOff in status.message, and the operator log shows passes of different clusters overlapping. Adding --cluster-max-concurrent-reconciles=2 to the Deployment changes the worker count to 2; =0 stops the operator at startup with a message naming the flag. Delete the three afterwards
Neo4j telemetry through an OpenTelemetry Collector	kubectl apply -f examples/observability/otel-collector.yaml (a Collector DaemonSet with the debug exporter; delete it afterwards). With spec.monitoring.enabled: true, kubectl -n otel logs ds/otel-collector shows metrics from the three server pods only (no other annotated pods), the database moved out of the name into a database attribute (neo4j_database_transaction_committed_total, database: neo4j). Set spec.monitoring.logs.stdout: [query]: one rolling restart with ConfigNeedsRestart; a query slower than spec.monitoring.slowQueryThreshold (default 5s while monitoring is enabled; set it to 0s in the same change to log every query) arrives as a record with type: query, database, elapsedTimeMs, queryLang, k8s.pod.name, Neo4j's own timestamp and severity INFO. Change to [query, security, debug]: no pod restarts, the StatefulSet's generation is unchanged, and security and debug records arrive within about two minutes. Remove logs: one rolling restart, server-logs.xml gone from the ConfigMap. A cluster created under the previous operator is untouched by the upgrade (generation, pods and ConfigMap unchanged)
Split-brain checks are paced, not skipped	With the operator's logs at debug, leave the Ready cluster alone for 11 minutes: Starting split-brain detection appears about every 5 minutes (--split-brain-check-interval), and the passes between log Skipping split-brain detection. Delete one server pod: the pass that sees it gone checks in full, and so does the first pass after it rejoins — not five minutes later. Kill a server's container (kubectl exec <pod> -c neo4j -- kill 1): the restart count changes the pod fingerprint, so the pass after it checks in full. Redeploy with --split-brain-check-interval=0s: every pass checks in full. With --ready-poll-interval=2m in --mode=production, a quiet Ready cluster reconciles about once every 2 minutes. Dev mode (make deploy-dev-local) resyncs every informer every 30 seconds, so there a quiet cluster reconciles several times a minute whatever the interval; switch the Deployment's --mode=dev to --mode=production for this check and back afterwards. Walked on 2026.08.1, 2026-10-08
Split backup archives (2026.09+)	On a 2026.09+ standalone, load >1GiB of incompressible data (e.g. 1.3M nodes of 28 concatenated randomUUID()s, in 50k-row statements — one 1.3M-row CALL … IN TRANSACTIONS hits dbms.memory.transaction.total.max). Back up with compress: false and splitArchivePartSize: 1Gi to a PVC and to MinIO: each artifact is a 24-byte .backup plus .backup.1 (exactly 1073741824 bytes) and .backup.2, and status.history[].artifactFilename names the .backup. A scheduled CR with retention.maxCount: 1, deleted after two runs, leaves one artifact with its parts. Restore over neo4j on the standalone (marker node gone, every node back), then on a 2026.09 cluster restore into new databases from MinIO and from the PVC — both online with every node. Optionally, a replication-source chain split the same way seeds and pulls into a Neo4jReplicaDatabase. splitArchivePartSize: 1G goes Invalid (10⁹ bytes < 1GiB), and kubectl neo4j validate says the same. Walked 2026-10-05

3 servers (not 2) keeps split-brain / 3-primary quorum behaviour in the routine walk. → Tear down the cluster fully before Phase 3.

Phase 2b — Upgrading clusters restarts nothing, and TLS renewal (v1.20+)

Three short deployments, one at a time, each torn down before the next. The first two are created under the previous release (as in Phase 1's upgrade row), so start each with the previous operator, then deploy main with its CRDs. Before upgrading, note the StatefulSet's metadata.generation and every server pod's UID and restart count.

Scenario	Verify
5.26 cluster upgraded from the previous release	3 servers on 5.26-enterprise (5.26.29 or later; check CALL dbms.components()). After main runs: generation, pod UIDs and restart counts unchanged. A setting the previous release did not write is either applied live — a ConfigAppliedLive event names it (from v1.20.x: dbms.cluster.network.connect_timeout, the operator's 5s default) — or deferred: the ConfigMap's neo4j.com/deferred-settings names it and RestartPending=True names all three servers (from v1.19.x: dbms.cluster.raft.async_channel_acquisition_enabled, with startup.sh gaining the blocks between # operator:restart-neutral begin and end). With nothing deferred — any upgrade from v1.20.0 or later — RestartPending=False with reason NoSettingsAwaitRestart. Objects the previous release created and main no longer does are gone (from v1.20.x: the <name>-discovery Role, RoleBinding and Service; the <name>-discovery ServiceAccount stays). If RestartPending names servers, delete server-1: when it is back, RestartPending no longer names it. The existing servers stay system primaries
5.26 scale without restart	Scale that cluster 3 → 4: ScaledWithoutRestart, servers 0–2 keep their UIDs, server-3 hosts system as secondary. Optional, with a loop writing every second: deleting server-1 (which now has the setting) leaves no gap in the writes longer than a few seconds, where one without it stalls up to 30s
CalVer TLS cluster upgraded from the previous release	3 servers on the CI anchor with spec.tls.mode: cert-manager. After main runs: nothing restarts, and settings the previous release did not write appear as in the 5.26 row (from v1.20.x: ConfigAppliedLive for dbms.cluster.network.connect_timeout, RestartPending=False). Renew the certificate by deleting <name>-tls-secret (cert-manager issues it again). From v1.20.0 or later reload is already on: TLSCertificateReloaded events name every server with no restart — each event names the servers reloaded on that pass, and the rest are retried about 30s apart — and echo \| openssl s_client -connect localhost:7687 2>/dev/null \| openssl x509 -noout -fingerprint -sha256 inside each server matches the new Secret's tls.crt once its event has named it. From v1.19.x, RestartPending names all three servers with dbms.security.tls_reload_enabled and the renewal raises one TLSCertificateNeedsRestart Warning naming all three; delete server-1, wait for it, and renew again: TLSCertificateReloaded names server-1 with no restart, and the Warning names the other two
TLS Secret deleted with its cluster	Delete that cluster: <name>-tls-secret is gone with it
Rolling upgrade waits for each server's databases (#488)	Needs two published patch images of one line (e.g. 5.26.30 → 5.26.31). A 3-server cluster with about 30 databases of 3 primaries each (CREATE DATABASE dNN TOPOLOGY 3 PRIMARIES WAIT) and a client writing to a few of them through neo4j:// routing. Change the image: at least one step shows status.upgradeStatus.message "waiting for its databases to catch up (…)" naming databases, and CALL dbms.cluster.statusCheck([]) run on the server just restarted (cypher-shell -a bolt://localhost:7687 in its pod — the default neo4j:// routes to the system leader) shows those databases not yet APPLYING at that moment (to post-process its rows with WITH, run it with -d a user database — system refuses WITH); the partition lowers only after they are. No write made through a retrying transaction function fails; brief retryable errors and stalls of a few seconds around each restart are expected. Diagnostics keep being collected during the upgrade, but status is written only when they change or every 5 minutes (#475), so status.diagnostics.lastCollected moves in steps, not every --ready-poll-interval; ServersHealthy turns False naming a restarting server at least once, then True again
CalVer TLS standalone, created fresh	A Neo4jEnterpriseStandalone on the CI anchor with cert-manager TLS: its neo4j.conf has dbms.security.tls_reload_enabled=true. A new TLS standalone restarts once about 30 seconds after its first Ready (the neo4j.com/tls-cert-sans stamp is added once its Secret exists; pre-existing), so note the pod UID after that. Renew (delete <name>-tls-secret): TLSCertificateReloaded, same pod UID, and the presented certificate is the new one within about two minutes. Delete the standalone: <name>-tls-secret is gone. A non-TLS standalone with a hand-made <name>-tls-secret, deleted: the Secret is kept, and the operator logs why

→ Tear down each deployment before the next, and the last before Phase 3.

Phase 3 — Property sharding (cluster, the CalVer CI anchor, 3 servers)

Sharding's documented floor is 4Gi + 1 core per server. To fit a laptop we deliberately relax it — this phase is operator-mechanics verification, not doc-following. The relax is operator-side and DEV/TEST only:

# Downgrades the 4Gi/1-core hard rejects to warnings.
kubectl -n <operator-ns> set env deployment/<operator-deploy> \
  NEO4J_SHARDING_RELAX_MEMORY_MIN=true
kubectl -n <operator-ns> rollout status deployment/<operator-deploy>

Use the pinned CI anchor (2026.08.1-enterprise as of v1.15.0 — read it from .github/workflows/integration.yml) so local sharding matches what the extended suite validates. A node with 12Gi and 6 cores free can run the example's real 4Gi/2-core floor, in which case skip the relax flag. On a servers: 3 cluster (~2Gi each):

Scenario	Verify
Sharding cluster	CALL dbms.components() → the CI anchor you deployed (2026.08.1 enterprise as of v1.17.0); status.propertyShardingReady=true
Sharded database	Neo4jShardedDatabase whose metadata.name differs from spec.name (e.g. CR products-sharded / spec.name: products) → Ready; SHOW DATABASES WHERE name STARTS WITH '<logical>' lists the graph + property shards online
Sharded backup by CR name	Neo4jBackup with spec.shardedDatabase = the Neo4jShardedDatabase CR metadata name → Succeeded; status.history[].shardArtifacts lists every shard, and the filenames carry the LOGICAL names (<spec.name>-g000, -p000, -p001). Using the logical name instead fails with Neo4jShardedDatabase "<name>" not found. (This row said kind=ShardedDatabase with target.name until 2026-09-03; spec.target does not exist on Neo4jBackup and the row was unfollowable as written.)
Sharded PVC backups are full	The scheduled sharded backup on a PVC with backupType: AUTO (or none): every run's status.history[].shardArtifacts[].type is FULL (the Job command carries --type=FULL), and a new Neo4jShardedDatabase with seedBackupRef to it seeds from the latest run (the seed proxy serves it, no merge-chain init container). A Neo4jBackup with spec.shardedDatabase and backupType: DIFF is refused by validation, naming spec.options.backupType (Neo4j does not back up property shards differentially). To reach the differential path, take an all-databases AUTO backup on a PVC twice: the second run raises a BackupShardedDifferential Warning naming the graph shard; a seedBackupRef to it → Failed within seconds, naming the differential shard, with no backup-seed-proxy Deployment; after a sharded backup's full run lands and seedBackupRef points at it, the database seeds on its own requeue
Sharded seed from MinIO	Back the sharded database up to MinIO. On a sharding cluster with no MinIO credentials or endpoint in its spec but neo4j.com/auto-inherit-seed-creds: "true", a seedBackupRef to it: one SeedCredsAutoInherited event, the cluster gains the credentials in extraEnvFrom and AWS_ENDPOINT_URL_S3 + JAVA_TOOL_OPTIONS=-Daws.s3.forcePathStyle=true in spec.env in one rolling restart, and the database seeds after it. Without the annotation → Failed, naming both the credentials and the endpoint
Shard language follows the parent	SHOW DATABASES YIELD name, defaultLanguage WHERE name STARTS WITH '<logical>' → the parent, -g000 and every -p00n report the same language as the parent (measured 2026-09-29 on 2026.06.0: a parent created with SET DEFAULT LANGUAGE CYPHER 5 on a CYPHER_25 server gave a CYPHER 5 family). Neo4j fixes a database's language at creation, so also confirm the default neo4j database's defaultLanguage is what the design expects for this operator version — see cypher-language-defaulting.md §5.8–5.9
Sharded family language	The operator's CREATE (operator log, Executing sharded database creation) carries SET DEFAULT LANGUAGE CYPHER 25 exactly once; the parent and every shard report CYPHER 25
Shard status (v1.18+)	status.creationTime, graphShard, propertyShards (one per shard, with its index) and virtualDatabase are populated, and agree with SHOW DATABASES: within ~15s of Ready every shard reads state: online, ready: true. On the v1.18.0 walk the graph shard read starting for minutes after the family was online, because the status was written once, right after CREATE, and not re-read before the five-minute periodic reconcile
Upgrading a sharding cluster from the previous release	Create the sharding cluster under the previous release, note the StatefulSet's neo4j.com/cluster-template-hash annotation (on the StatefulSet itself, not its pod template), its metadata.generation, and the pod UIDs, then upgrade to main (CRDs first). status.effectiveCypherLanguage=CYPHER_25 (recorded from the ConfigMap the old operator wrote), the template hash and generation are unchanged, neo4j.conf differs only by settings main adds, each applied live with a ConfigAppliedLive event (from v1.20.x: the dbms.cluster.network.connect_timeout block), and no pod restarted. Then clear the stamp (kubectl patch … --subresource=status --type=merge -p '{"status":{"effectiveCypherLanguage":null}}') and restart the operator — scale it to 0 first, or the running operator re-stamps the field before you can see it cleared: re-derived CYPHER_25, still no restart — the etcd-restore case
CLI: the sharded CR is legible	status -n <ns> lists the Neo4jShardedDatabase by its CR metadata.name — the same name vs. spec.name distinction the row above exists to catch — with its phase, alongside the cluster. explain Neo4jShardedDatabase/<cr> either decodes its conditions or says plainly that it carries none this CLI knows, which is the correct answer rather than a guess

→ Tear down, then delete the Kind cluster.

Phase 4 — Aura (no Kind deployment; needs Aura API credentials)

BETA / best-effort surface. The Aura* CRDs talk to the Aura cloud API, so nothing in Phases 1-3 touches them. This phase is optional but strongly recommended whenever an Aura* CRD or internal/aura/ changed, because the published Aura OpenAPI spec is known to disagree with the live API (see docs/knowledge/operations.md id 87).

Take that disagreement seriously. As of 2026-08-01 every Aura surface except CMK has been driven against a real account, and three of them — fleet provisioning, IP filters and invites — were broken in ways the unit suite could not see, because the fixtures asserted the client matched the spec rather than the API. A green make test-unit is not evidence that an Aura write path works.

Cheap technique worth reusing: to learn an enum or a required field without mutating anything, send a deliberately invalid value — the API answers with the full list of accepted values, or names the missing field, and changes nothing.

Surface	Live-verified
Instances (v1 lifecycle), snapshots, restore	2026-08-01	✅
Databases + per-database backup/restore (v2beta1)	2026-07-31	✅
IP filters (v2beta1)	2026-08-01	✅ contract corrected
Console RBAC: members, invites (v2beta1)	2026-08-01	✅ invite contract corrected
Fleet provisioning (v2beta1)	2026-07-31	✅ contract corrected
Customer-managed keys (v1)	—	❌ never exercised; needs a real cloud KMS key

Set up an AuraProviderConfig (or an inline credentialsSecretRef) with an API client ID + secret from the Aura console.

Read-only sweep — always safe, do this every time:

Scenario	Verify
Credentials + auth	A token from /oauth/token authenticates both v1 and v2beta1 calls
Org/project discovery	GET /v2beta1/organizations and …/projects return {id,name}
Read shapes still match	Org users return user_id + organization_roles[] with organization-* values; ip-filters return a bare array (no data envelope); the v1 CMK list returns only id/name/tenant_id; DatabaseSummary returns only id
Fleet reads	Fleet deployment single-GET is data-wrapped; token field is auto_rotated; server field is mode_constraint (singular); shard/txn/lag data appears only on …/servers/{id}/databases

A GET-only sweep needs no throwaway resources and cannot damage anything. It is what caught four real client bugs during the 2026-07-30 re-diff.

Write checks — ONLY in a disposable Aura project, never production:

Scenario	Verify
AuraInstance lifecycle	create → Ready → resize (memory) → pause → resume → delete; status.instanceId pinned via the neo4j.com/external-instance-id annotation. Set deletionPolicy: Delete on the throwaway instance — the default is Orphan, so otherwise deleting the CR leaves the instance running and billing (see the teardown note below), and the delete path itself goes untested
AuraDatabase + backup	Needs an instance created with multiDatabase: true — a Business Critical tier alone is not enough, and the flag cannot be added later (docs/knowledge/operations.md id 90). Against any other instance expect Ready=False, reason InstanceNotMultiDatabase, with no requeue. On a multi-database instance: AuraDatabaseBackup reaches Completed (not on first observe — an empty status must read as Pending, and the backup does not appear in the backups LIST until it completes, so it must be polled by ID); AuraDatabaseRestore reports Submitted, never Completed
Multi-database create	An AuraInstance with multiDatabase: true + organizationId is created via v2beta1 (v1 type names are translated), then remains manageable through v1; status.atProvider.multiDatabase reports true. On a v1-created instance the field reports the true value from the v2beta1 probe — false is correct there, and it stays absent (unknown) only when no organizationId is resolvable, since the v1 GET carries no such field
Multi-database create options	Same instance, created with storage set to a non-default size and graphAnalytics set: the v2beta1 GET reports that storage and that graph_analytics. Separately, a non-multi-database instance with graphAnalytics: serverless is created through v2beta1 (it needs organizationId) and its GET reports graph_analytics: serverless with multi_database: false; the instance stays Ready with no update attempted, even though v1 reports graph_analytics_plugin: false for it (v1 cannot express serverless). graphAnalytics: plugin on a plain instance still goes through v1. vectorOptimized: true with multiDatabase is refused at apply time with Aura's reason (multi-database-capability-not-supported), never sent. Also confirm the API still REJECTS unknown create fields and still accepts restore by id — both changed or drifted in the spec before (id 87 items 12 and 15)
Console RBAC	AuraProjectMember for an existing org member is added without an invite; role PATCH bodies are accepted (they are {organization_roles:[…]} / {project_roles:[…]}, not a scalar)
Fleet provision	Deployment registered, token minted into the Secret, DBMS registers; deleting the Secret with the default tokenPolicy refuses to rotate and says why

→ Tear down every resource created here; Aura bills by the hour. Deleting the CRs is not sufficient proof: deletionPolicy defaults to Orphan, so a CR delete can leave a running instance behind. Confirm with GET /v1/instances/<id> → 404 and a project instance list of 0 before you call the phase done.

Phase 5 — Cross-cluster replication (two deployments; sequential for backup mode, concurrent for network mode; needs 2026.08+)

Run this only when a 2026.08+ enterprise image is available. Hosting a replica requires it, and the check is enforced — on an older image the CR fails with ReplicaVersionTooOld and nothing else in this phase can proceed. That is itself scenario 1: confirm the gate fires rather than silently doing nothing.

Parts A/B below (backup mode, source.mode: backup) look like they break the one Enterprise deployment at a time ground rule. They do not, and the reason is the point of that mode: the two clusters are coupled only through the object store, never over the network. So the upstream produces its chain, gets torn down, and then the downstream comes up and pulls from the bucket. Two deployments, never concurrent.

Network mode (source.mode: network, Parts C/D below) needs a live network path, but "two clusters" doesn't mean leaving Kind or leaving this single deployment session: Part C runs the real replication mechanism between two Neo4jEnterpriseClusters in different namespaces on this same Kind cluster, reachable over ordinary in-cluster DNS — no proxy, no crossClusterReplication, no LoadBalancer needed. Part D smoke-tests the crossClusterReplication toggle on a single plain Kind cluster, where the proxy's Service never gets an address. Part E is the one that matters most: two Kind clusters, TLS on with mutual CA trust, the proxy holding a real MetalLB address — the configuration CCDR is actually sold with, and the only one that exercises the proxy, the certificate SANs, the advertised-address override and cross-cluster trust together. See each part for what it does and does not cover.

Stand up MinIO as the shared bucket (same pattern the backup specs use) for Parts A/B.

Part A — upstream (standalone is enough):

Scenario	Verify
Replication-source backup	Neo4jBackup mode: replication-source + database + schedule → status.replicationPullURI is populated and points at the chain directory. Copy this value
Chain-breaking configs rejected	the same CR with retention set, or allDatabases: true, or no schedule → phase Invalid, message names the rule
A second writer to the same path is refused	while ANOTHER Neo4jBackup targets the same bucket + storage.path, the chain refuses to run and says why — "a second writer can interleave artifacts and break the differential chain a replica depends on" — pointing at storage.path or spec.chainFromBackup. Easy to trigger by accident: the rejected CRs from the row above collide with the good one unless they use a different path, and the chain stays stuck until they are deleted
A FULL then a DIFF	let (or force) two runs so the bucket holds a full and a differential — a seed-only chain does not exercise the pull path. status.history[].artifactType records each run's type; the run's Job log shows the transactions (Finished full backup … Downloaded from tx -1 to tx N then Finished differential backup … from tx N+1). Run 1 logs "Start differential … Falling back to full" — correct, there is no parent yet. Write to the CHAIN'S database between runs, not the default one: kubectl neo4j cypher has no -d flag, so use kubectl exec … cypher-shell -d <db> or the diff has nothing to carry
CLI replaces the copy step	kubectl neo4j export replica-database <name> --from-backup <backup> --cluster-ref <downstream> --upstream-database <db> emits a Neo4jReplicaDatabase on stdout only — redirect it and confirm the file is a clean manifest with the notes on stderr — carrying the same pullURI you copied by hand above. Then run validate -f on the generated file: the command already validates before printing, so a manifest validate rejects must never appear

→ Tear the upstream down completely. Leave the MinIO bucket intact.

Part B — downstream (fresh deployment, 2026.08+):

Scenario	Verify
Replica seeds	Neo4jReplicaDatabase with source.pullURI/seedURI from Part A → phase Replicating; SHOW DATABASES shows type=replica, access=read-only on every copy including primaries. (writer reads TRUE on 2026.08.1 — it names the leader, not write access; the write test below is what proves read-only)
Replica is read-only	a write via cypher-shell fails; the same query with --access-mode=read succeeds. Two traps: the replica is a database named after the CR (foo-replica) when spec.name is unset, which is what export replica-database emits, so -d foo hits a database that does not exist; and run the write on the server hosting it (SHOW DATABASE … YIELD address), where it fails with 08N08 database is read only. Routed from another server, Neo4j reports the same refusal as a transient 08N17/50N16 remote-execution error
Alias pre-staged against a replica	Neo4jDatabaseAlias targeting the replica → Ready while the target is still a replica (this is what removes work from the failover window)
Promotion	Neo4jReplicaPromotion → Completed; SHOW DATABASES now shows a standard read-write type. status.observedLagTxIds is set only when there was lag to make permanent: at zero lag it is absent by design and the message says "0 transactions of lag were made permanent"
Alias survives promotion	the same alias now resolves to a read-write database — no CR change, no client reconfiguration
Replica CR goes inert	the Neo4jReplicaDatabase is phase: Promoted; delete it and confirm the database is NOT dropped
Out-of-band promotion is safe	on a second replica, promote by hand at a cypher-shell, then let the operator reconcile → it goes Promoted with a ReplicaPromotionDetected event ("promotion detected (out-of-band)") rather than "correcting" the drift by dropping the database. Promote with CYPHER 25 CALL dbms.promoteReplicaDatabase('<name>', {}) on system — the procedure takes an options map as its second argument, and calling it with the name alone fails 42I13
Replica metrics (v1.18+)	run the operator with --kubernetes-cluster-name=<name> and scrape /metrics throughout Part B: neo4j_operator_replica_lag_transactions{namespace, cluster_name, replica, database, k8s_cluster} is present while a replica is Replicating (value = transactions behind, 0 here) with k8s_cluster set, and is removed — not frozen at its last value — when the replica is promoted by CR, promoted out of band, or its CR is deleted. neo4j_operator_replica_promotions_total{…, result="success"} counts the CR-driven promotion only, once; the out-of-band one does not increment it. A vector with no observations exports nothing, so the counter is absent until the first promotion. Repeat the lag check in Part E for network mode

That last row is the one worth doing carefully: it is the failure mode the whole observe-before-act design exists to prevent, and it is invisible to unit tests.

Part C — network mode core mechanism, same Kind cluster, no proxy (two concurrent deployments):

Automated — test/integration/ccdr_same_cluster_network_mode_test.go, Label("extended"), gated by isCCDRReplicaCompatible() (it needs a 2026.08+ image: it runs on the default CI anchor, 2026.08.1, in the dispatched Extended suite, and skips if you dispatch 5.26-enterprise or an older CalVer). Uses source.upstreamClusterRef rather than a hand-typed source.addresses FQDN — exercises the same underlying mechanism via the newer, higher-level API. Re-run manually only if you want the hand-typed-address path specifically, or a version this repo's CI cannot yet dispatch.

This is the highest-value scenario in this phase: it is the first time CREATE REPLICA DATABASE ... OPTIONS {replicaConfig: {remote, addresses}} runs against a real server, rather than being inferred from documentation.

Deploy two small Neo4jEnterpriseClusters (2 servers each, CI-sized resources) in different namespaces on this same Kind cluster — e.g. prod-cluster in namespace prod, dr-cluster in namespace dr. Leave spec.crossClusterReplication unset on the upstream; do not create the proxy at all. Point the downstream Neo4jReplicaDatabase directly at the upstream's plain internal pod FQDN — reachable across namespaces on the same cluster by ordinary in-cluster DNS, no Service changes needed:

prod-cluster-server-0.prod-cluster-headless.prod.svc.cluster.local:6000

This deliberately breaks the one-deployment-at-a-time rule — network replication cannot be tested sequentially the way backup mode can, the two clusters must be up together. Keep both small (2 servers, CI-appropriate resource requirements) to limit the Bolt-wedging risk the anti-wedge rule exists for; if Bolt does wedge here, that is itself a real finding about concurrent-deployment resource limits, not a test artifact to work around.

Scenario	Verify
Network replica seeds from a plain internal FQDN	Neo4jReplicaDatabase with source.mode: network, source.addresses set to the upstream's server-0 pod FQDN above → phase Replicating; SHOW DATABASES shows type=replica, access=read-only, writer=false
One address is sufficient	list only server-0's address, not all N — replication still reaches every ordinal, confirming the design doc's Q1 finding that the upstream hands back the addresses the downstream actually uses
Malformed/missing address rejected before any Cypher runs	source.addresses: [] or a host with no port → validator error, no CREATE REPLICA DATABASE attempt
Promotion works the same as backup mode	Neo4jReplicaPromotion against this replica → Completed, SHOW DATABASES now shows a standard read-write type
CLI replaces the hand-typed FQDN	kubectl neo4j export replica-database <name> --from-cluster prod-cluster --cluster-ref dr-cluster --upstream-database <db> emits the same manifest with source.addresses read from the upstream's status.internalAddresses — the FQDN typed by hand above. Confirm they match. Note in the log that on one Kubernetes cluster source.upstreamClusterRef is the better answer and the docs say so; the exported literal is for a downstream on a different cluster, which is the only case no ref can reach

This scenario does not exercise spec.crossClusterReplication itself — no proxy, no advertised-address override, no cert SAN, no NetworkPolicy rule. That gate depends on the proxy's Service reaching a real status.loadBalancer.ingress state, which is identical whether one Kind cluster or two are involved (see Part D) — this scenario is about the replication mechanism, deliberately isolated from that gate.

→ Tear down both deployments, then delete the Kind cluster.

Part D — network mode proxy toggle, Kind-only smoke check (single deployment):

Automated — test/integration/ccdr_proxy_test.go, Label("core"). Needs no version gate (nothing here hosts a replica), so it runs on every PR against both CI anchors, not just on dispatch — the manual walk below is useful only for exploring interactively, not for verifying this before a release.

Part C covers the replication mechanism; this part covers the one piece it deliberately left out — spec.crossClusterReplication itself. That gate depends on the proxy's Service reaching a real status.loadBalancer.ingress state, which a type: LoadBalancer Service never gets on plain Kind. make ccdr-lb now supplies it (hack/metallb-setup.sh — MetalLB L2 with a pool carved from the top of the shared kind Docker network), so the proxy path is verifiable locally, and Part E below walks it end to end across two Kind clusters. This part stays useful as the single-cluster smoke check: the toggle must never break normal cluster formation, whether or not a real load balancer is present to service it. Run it without make ccdr-lb — the rows below describe the no-address state on purpose.

What is verifiable on a single plain Kind cluster is that the toggle itself is safe — it must never break normal cluster formation, whether or not a real load balancer is present to service it:

Scenario	Verify
Proxy resources render	Neo4jEnterpriseCluster with spec.crossClusterReplication.enabled: true → a <name>-ccdr ConfigMap/Deployment/Service appear, and the cluster still reaches Ready (this proves the toggle doesn't wedge formation even when the LoadBalancer never gets an address on plain Kind)
NetworkPolicy stays additive	with spec.networkPolicy.enabled: true too → the existing 3 ingress rules are unchanged and a 4th rule admits only the ccdr-proxy pod on port 6000
Advertised address stays internal until ready	status.crossClusterReplication stays empty (Kind has no LoadBalancer controller, so the Service never gets an ingress address) → the EFFECTIVE config in the pod still advertises the pod itself: server.cluster.advertised_address=<pod FQDN>:6000, not the proxy's hostname. Do not hardcode the path — ask the process, because it has moved three times: kubectl exec <pod> -c neo4j -- sh -c 'CONF=$(ps aux | grep -o "config-dir=[^ ]*" | head -1 | cut -d= -f2); echo $CONF; grep ^server.cluster.advertised_address $CONF/neo4j.conf'. The ConfigMap never holds advertised addresses at all (it says so: "advertised addresses will be set dynamically by startup script") — buildStartupScriptForEnterprise substitutes them at pod start, overriding 6000 only once status.crossClusterReplication.LoadBalancerHostname is set. This row named the ConfigMap until 2026-09-12, /var/lib/neo4j/conf until 2026-09-13, then /tmp/neo4j-config until 2026-09-23 — when the env-var config fix (99e60e1/0cb2ee8) stopped the startup script pointing NEO4J_CONF at its own directory, so --config-dir went back to /var/lib/neo4j/conf and /tmp/neo4j-config no longer exists. A grep of an absent path returns nothing, which reads exactly like the override failing
Toggling off tears down cleanly	flip enabled back to false → the ConfigMap/Deployment/Service are deleted and status.crossClusterReplication clears

→ Tear down, then delete the Kind cluster.

Part E — the whole proxy path, two Kind clusters, TLS ON (two concurrent deployments):

This is the configuration CCDR is actually sold with, and the only part that exercises the proxy, the certificate SANs, the advertised-address override and cross-cluster CA trust together. All of that was verified for the first time on 2026-09-14, and three defects fell out of it that nothing else had reached — run it before every release that touches CCDR or TLS.

make ccdr-e2e-up      # both Kind clusters, cert-manager + issuer, MetalLB on
                      # the upstream, the operator built from this tree on both

Then apply a Neo4jEnterpriseCluster to each — by hand, from the published docs, because that is what is under test. Both get spec.tls.mode: cert-manager with strictPeerValidation: true (the default) and 2 CI-sized servers; the upstream also gets spec.crossClusterReplication.enabled: true. Once both are Ready:

make ccdr-e2e-trust   # exchange the CAs, patch additionalClusterTrustCAs on both
make ccdr-e2e-status  # where everything is
make ccdr-e2e-down    # delete both clusters

The upstream defaults to the dev cluster (neo4j-operator-dev). To keep the dev cluster beside the rig, set CCDR_UPSTREAM_CLUSTER=<name> on every target, down included: without it they act on neo4j-operator-dev, and down deletes it.

Put --context on every kubectl you run yourself. kind create cluster switches the current context, and a batch of commands aimed at the wrong cluster is the easiest mistake to make here — it produced a false blocker on the first walk. The scripted steps already do.

Scenario	Verify
CA exchange really crossed	copy each cluster's ca.crt out of its <name>-tls-secret into a Secret on the other cluster and reference it from spec.tls.additionalClusterTrustCAs. Then check it landed: md5sum /ssl/trusted/*.crt in each server-0 — the upstream's peer-ca-0.crt must equal the downstream's own ca.crt, and vice versa. Both clusters must still reach Ready; a peer CA that stops a pod starting is a blocker, not a warning
A wedged cluster can still be repaired	while you are here: if a template change does NOT reach a cluster that is stuck mid-formation, that is the bug fixed in 40713d7 regressing. Editing the CR must change the StatefulSet even when no pod is ready
The proxy gets a real address	status.crossClusterReplication.ready: true with one address per server ordinal (<ip>:16000, :16001), and server.cluster.advertised_address=<ip>:16000 in the effective config — find it with the --config-dir one-liner in Part D rather than a hardcoded path. RAFT and routing must stay internal; only the tx-shipping port is proxied. The override applies only once the Service has an ingress address, so this is the first place it is observable at all
The certificate covers that address	the proxy address is an IP, so it belongs in the certificate's IPAddresses, never DNSNames — an IP in DNSNames yields a certificate the cluster cannot form on. kubectl get certificate <name> -o yaml
Replica streams between the two clusters	Neo4jReplicaDatabase, source.mode: network, source.addresses: ["<proxy ip>:16000"] → Replicating, "0 transactions behind"
The stream is live, not just the seed	rows written upstream after the replica exists appear downstream; a write against the replica is refused
The cluster says what guards the exposed port	status.conditions[CrossClusterProxySecure] is True/MutualTLSRequired with TLS on. Enabling the proxy without spec.tls is refused: the cluster goes Failed, naming spec.tls.mode=cert-manager, and no StatefulSet or Service is created — nothing would authenticate the tx-shipping port, and the proxy authenticates nothing itself

→ Tear down both: make ccdr-e2e-down.

Coverage at a glance

	CLI (Phase 0)	Standalone	Cluster (3)	Sharding (CI anchor)	Aura (Phase 4)	CCDR (Phase 5, 2026.08+)
Reconcile → Ready		✅	✅	✅	✅ (instance)	✅
Database lifecycle		✅			✅ (AuraDatabase)
Database topology			✅		n/a (Aura-managed)	✅ (replica)
Users / Roles / Bindings		✅			n/a (console-RBAC only)
Privilege matching (stored form, learn / probe)		✅	✅
privilegeRules (+ Neo4jDatabase watch)		✅
Server default Cypher language		✅ (5.26: never written)	✅ (CalVer: CYPHER_25)	✅ (sharded family)
Upgrade from the previous release (nothing revoked, nothing restarted)		✅	✅ (Phase 2b, 5.26 and CalVer TLS)	✅
Restarts avoided (live config, scale)		✅ (live config)	✅ (scale, live config; 5.26 in Phase 2b)
TLS reload on renewal; issued Secret removed on deletion		✅ (CalVer, Phase 2b)	✅ (upgraded servers, Phase 2b)
Split-brain pacing, Ready poll interval			✅
Neo4j telemetry into OpenTelemetry (logs on stdout, metrics, Collector)		✅ (5.26, logs)	✅ (CalVer, Collector end to end)
Plugins (APOC)		✅ (ConfigMap)			n/a
Backup → restore		✅ (online + neo4j-admin)	✅ (Cypher)	✅ (sharded)	✅ (per-DB, API)	✅ (chain as source)
Point-in-time restore		✅ (offline, 5.26)	✅ (online, seedRestoreUntil)
Workload identity on the server pods		✅ (<name>-neo4j, simulated)
Sharded seed from PVC (full only) and MinIO (endpoint projected)				✅
Property sharding				✅
Aura Fleet Management		✅ (plugin + token)			✅ (provision)
Aura console RBAC					✅
Aura read-contract sweep					✅ (GET-only)
Cross-cluster replication						✅ backup mode + network mode mechanism (same-cluster) + the full proxy path across two Kind clusters with TLS on (make ccdr-lb, Part E)
Replica metrics (lag gauge, promotion counter, k8s_cluster)						✅ (backup mode and network mode)
Database aliases		✅				✅ (survives promotion)
Composite databases		✅ (local + the keystore path)
Operator→Neo4j TLS (CA + pinned)		✅			n/a (HTTPS to Aura)
CLI: offline (validate, explain, exit codes)	✅
CLI: failure paths (diagnose, preflight)	✅ (caused on purpose)	✅ (preflight on what is deployed)	✅ (… during a restart)
CLI: cloud substrate (preflight zones / PSA / default class / private LB)	✅ (all four reachable on Kind)
Operator build metadata (neo4j_operator_build_info)		✅ (scrape /metrics)
CLI: healthy lens (status, connect/cypher, support-bundle)		✅	✅	✅ (status)
CLI: manifest authoring (export replica-database)						✅ (both modes)
CLI ↔ operator version skew	✅ (forced by hand)

Keeping this current

When you add or change a capability, update this page in the same PR:

1. Add a row to the routing table (which deployment, and why there).
2. Add the scenario + its in-DB check to the relevant phase.
3. Tick the coverage matrix.
4. If it changes the operator install or sizing, update the phase headers.
5. If it adds or changes a CLI command, put it in the phase whose deployment it reads — Phase 0 if it needs none, or a broken resource if what it reports is a failure. A command verified only against a healthy cluster is verified in the state nobody runs it in.

When a release fixes a specific bug, add a one-line scenario that would have caught it (the v1.12.2 pass added the standalone-label, system-reject, role-CR-name, and sharded-backup-by-CR-name checks). Record each run below.

Verification log

Release	Date	Result	Findings
v1.21.0 release verification	2026-10-14	✅ Phases 0-3, 2b and Phase 5 (Parts A, B, E) pass on 80d3d80 (the release commit c8c190e adds only docs and dependency bumps); Phase 4 not run (nothing under internal/aura/ or api/v1beta1/aura* changed); Parts C/D left to the extended suite; no wedged cluster arose in Part E. Walked on the existing dev Kind cluster plus a renamed two-cluster rig (CCDR_UPSTREAM_CLUSTER=neo4j-ccdr-up). No release blocker; two should-fix defects (one new), eight stale rows corrected after the walk	Upgrade from v1.20.0 restarts nothing: the standalone (byte-identical ConfigMap, generation 1, same pod UID, role rows and scheduled backup unchanged), a 5.26.31 cluster, a CalVer TLS cluster and the sharding cluster all kept template hash, generation and pods, and each got ConfigAppliedLive for dbms.cluster.network.connect_timeout (SHOW SETTINGS 30s → 5s on every server). The -discovery Role, RoleBinding and port-5000 Service were deleted, the ServiceAccount kept, and the operator ClusterRole holds only get/list/watch/delete on roles and rolebindings. New this release, live: a 5.26.30 → 5.26.31 rolling upgrade with 30 three-primary databases waited on server-1's databases (d11, d12, … and 5 more → d14, d19, d23, d26, d27 → d19, d23, d26, d27), matching statusCheck on that server (21/31, 26/31, 27/31, then 31/31 APPLYING); the partition lowered 10s later; 3m26s in all, 0 failed writes to 10 databases written every 200ms, one write over 500ms (2.2s). Operator logins on a quiet 3-server cluster: 3 in 4 minutes (about 17 a minute before #496); 1 on the standalone; an admin password rotation took effect with no auth errors. Clusters reconciled 4 at a time (passes of all 6 cluster pairs interleaved; =2 honoured; =0 refused at startup); a real cluster recovered from a pod delete in 59s beside three ImagePullBackOff clusters (57s without). maxUnavailableDuringUpgrade: 2 raised ValidationWarning on both kinds. A lost server (#444) never showed Forming across 150 one-second samples. Should-fix: that maxUnavailableDuringUpgrade warning repeats on every reconcile (35 in about 3 minutes on a standalone; new in #492, the same pattern as the older no-effect warnings); a backup-mode Neo4jReplicaDatabase refuses an AWS_REGION supplied through the cluster's spec.extraEnvFrom (older; missingObjectStoreEnv reads only spec.env and container env names), and since spec.source is immutable the fix its message suggests (credentialsSecretRef) means recreating the CR. Rows corrected after the walk: the #488 row (lastCollected is written on change or every 5 minutes, not every poll); the two upgraded-cluster rows in Phase 2b (RestartPending expectations held only from v1.19.x); Phase 3's byte-identical neo4j.conf (connect_timeout is added, live); the OTel row (query records need a query slower than slowQueryThreshold, 5s); the offline restore --from-path (a directory backupPath still resolves to its newest file); the w3 phase (ValidationFailed); the seed URI (now also in status.message). Open, not blocking: Raft takes up the live connect_timeout only at each server's next restart, and RestartPending does not say so (a 10s write stall when server-1 rejoined on an upgraded 5.26 cluster); a restore Failed for missing seed credentials does not resume when the annotation is added — documented as terminal, but the message implies the annotation alone is enough; a mistyped static setting (db.tx_log.buffer.size: "1MiB") passes validation and crash-loops Neo4j; make ccdr-e2e-down deletes the dev cluster unless CCDR_UPSTREAM_CLUSTER is set; still present from v1.20.0: PrivilegesApplied re-emitted after alias changes, sharded CREATE … IF NOT EXISTS re-runs, DatabaseReady/BackupScheduled events about 6 a minute, the ImagePull guidance saying Enterprise images are not public. SeedCredsAutoInherited now reaches the API (open in v1.20.0). Not run: split archives (2026.09); the TLS cluster renewal check was inconclusive for server-1 (deleted before its retry).
v1.20.0 release verification	2026-10-09	✅ Phases 0-3, the new Phase 2b and Phase 5 (Parts A, B, E) pass on 3a5adaa (Phases 0-2) and 255bd85 (2b onward; #483 was CLI-only); Phase 4 not run (nothing under internal/aura/ or api/v1beta1/aura* changed); Parts C/D left to the extended suite. Walked on the existing dev Kind cluster plus two Part E clusters; two defects fixed and six rows or docs corrected in the journey-fix PR	Upgrade from v1.19.0 restarts nothing: the standalone (CYPHER_5, byte-identical neo4j.conf, generation 1, same pod UID, role rows unchanged, scheduled backup Ready), a 5.26.31 cluster (restart-neutral blocks added, neo4j.com/deferred-settings, RestartPending naming three servers, then two after one restarted), a CalVer TLS cluster, and the sharding cluster (template hash, generation and conf unchanged; CYPHER_25 re-derived). New this release, live: ConfigAppliedLive/ConfigNeedsRestart on the standalone and on each cluster server (and reset on removal); scale 3→5 and 3→4 with ScaledWithoutRestart, the new servers secondary and recorded on disk, 0 failed writes; with async channels no write gap over 2s while a follower restarted; TLSCertificateReloaded without a restart on the restarted cluster server and on a fresh standalone, TLSCertificateNeedsRestart for the others; {name}-tls-secret gone with its cluster and standalone, a hand-made one kept; split-brain full checks 5 minutes apart, immediate on a pod delete, a rejoin and a container kill; composites walked on both lines. Fixed in the journey-fix PR: a container kill wrote phase: Forming once on a formed cluster (twice in two tries) — an unreachable formed cluster with a Ready majority of pods is now left as it is for the grace period (rule 105); a database name under 3 or over 63 characters passed validation and failed at CREATE DATABASE. Rows and docs corrected: --ready-poll-interval has no visible effect in dev mode (30s informer resync — the row now says production mode, and operator-modes.md says so); the pinning row's lastCollected (refreshed every 5 minutes since #475); --problems prints the healthy line; SHOW SETTINGS per server needs -d system; a fresh TLS standalone rolls once after its first Ready; ccdr-e2e-up with a renamed upstream bound 6443 beside the dev cluster (fixed in hack/ccdr-two-cluster.sh); the SSO rolebinding example said usernames cannot contain @; the sharding guide now says a database sharing the logical name's prefix blocks the backup, and the API reference that deleting a Neo4jShardedDatabase keeps its databases. Open (pre-existing, not blocking): any alias or constituent change re-emits PrivilegesApplied "N added" on every role with nothing changed; diagnose shows a crash-loop in its Terminated window as ⚠ and exits 0; SeedBackupPending repeated on every pass exhausts the event budget, so SeedCredsAutoInherited never reached the API; a fresh cert-manager standalone rolls once for its tls-cert-sans stamp; TLSCertificateNeedsRestart uses singular wording for several servers and repeats every 30s; PluginInstalled emitted 66 times in 35 minutes; the sharded controller re-runs CREATE … IF NOT EXISTS every pass; deleting a Neo4jShardedDatabase keeps its databases (now documented); an offline restore from a PVC directory without sourceDatabase fails with no reason; the ImagePullBackOff guidance says Enterprise images are not public. Environment: kind load of a multi-arch image needed docker save \| ctr import --platform; no Bolt wedge. Added after the walk, walked on its own the same day (spec.monitoring.logs, OpenTelemetry): a 2026.08.1 cluster created under the previous operator was untouched by the upgrade; turning logs.stdout on and off each rolled the servers once (ConfigNeedsRestart on server.logs.config); changing the list reloaded without a restart in 45s (cluster) and 92s (5.26.31 standalone); the example Collector (contrib 0.162.0) received query, security and debug records with pod attributes, Neo4j's timestamps and severities, and metrics from the Neo4j pods only with the database attribute; a conflicting spec.config server.logs.config was refused. Found on the way: a 45-second log window looked like missing 5.26 debug records (a quiet standalone logs little after startup); records had no severity until the example mapped level
v1.19.0 release verification	2026-10-06	✅ Phases 0-3 and Phase 5 (Parts A, B, E) pass on 27baf02, after the fixes noted; Phase 4 not run (nothing under internal/aura/ or api/v1beta1/aura* changed); Parts C/D left to the extended suite; no wedged cluster arose to repair in Part E. Walked on an isolated Kind cluster beside an existing dev cluster	Upgrade from v1.18.0: standalone role rows, ConfigMap, StatefulSet template, generation and pod UID unchanged, scheduled backup stays Ready/BackupScheduled; sharding cluster hash, generation, ConfigMap and pod UIDs unchanged. New this release, live: a lost server keeps the cluster Ready (Degraded names the pod), then Degraded after the grace, Forming + ClusterQuorumLost on majority loss, image change deferred (UpgradeDeferred); split archives on 2026.09 (exact 1 GiB parts on PVC and MinIO, retention keeps parts, 1.3M-node restores on standalone and cluster); restore under a new name / sourceDatabase; online standalone restore (no Job, no restart, stopCluster ignored, hook ran); merged PVC differentials on 5.26 and 2026.09 incl. an empty quiet run; standalone workload identity (no ServiceAccount or roll for a v1.18.0 standalone; online pod-identity restore; refused once removed); online cluster PITR into a new database and refusal over neo4j; 5.26 offline PITR to the pinned run; sharded AUTO → FULL on a PVC, differential seed refusal (via an all-databases AUTO backup), MinIO credentials + endpoint in one roll. Fixed in the journey-fix PR: on 5.26 an empty differential whose chain lost its full was served as a full (left an offline database) — the merge now checks the result; a Completed one-time backup turned Waiting on a target restart and ran again after its Job was GC'd (pre-existing) — rule 113; Neo4j refuses backupType: DIFF for a sharded database ("Differential backups of property shards are not allowed") — now refused by validation, docs corrected; the Degraded message no longer says "waiting for cluster formation"; a cloud sharded seed under a new name needs seedSourceDatabase — documented. Open (pre-existing, not blocking): a deleted cluster's metrics stay exported until the operator restarts; validate --connect without -n checks namespace-less manifests in ""; validate --connect passes a remote composite whose target lacks remoteAliasKeystore; a standalone's plugin settings arrive as env vars and override the same key in spec.config. Matrix rows corrected: proxy without TLS is refused (no NoClusterTLS), history records artifactType, the sharded DIFF row
v1.18.0 release verification	2026-10-02	✅ Phases 0-3 and Phase 5 (Parts A, B, E) pass on 7d04d6c; Phase 4 not run (the Aura key needs rotating first; nothing under internal/aura/ changed beyond a no-effect warning); Parts C and D are covered by the extended suite. Five defects fixed in #443, two doc fixes and six stale or incomplete rows here corrected in #442	Upgrade from v1.17.0 (operator 0eceac4): with main on the v1.17.0 CRDs and then on its own, the role kept its 5 stored rows with no new PrivilegesApplied, the standalone kept CYPHER_5, a byte-identical ConfigMap, StatefulSet generation 1 and the same pod UID with 0 restarts, and a hand-revoked row came back as 1 added / 0 revoked. A scheduled backup created under v1.17.0 (Ready=Unknown/Pending) read Ready=True/BackupScheduled the moment main ran. The sharding cluster created under v1.17.0 kept its template hash, generation 1, ConfigMap and all three pod UIDs, and re-derived CYPHER_25 after the stamp was cleared with the operator stopped. First live scrape of the replica metrics (new in #437/#438): neo4j_operator_replica_lag_transactions present while replicating with k8s_cluster set from --kubernetes-cluster-name (journey-dev, and neo4j-dr on the Part E downstream for network mode), and removed on a CR promotion, an out-of-band promotion and a CR delete; replica_promotions_total{result="success"} = 1 for the CR promotion only. v1.18 behaviour verified live: New ValidationWarnings fire (kerberos as a provider, with plugin-Neo4j-Kerberos accepted silently); a standalone storage.size of 1.5Gi accepted; status --problems lists an Invalid backup; support bundle carries errors.txt and zero hits for four real secret values; Neo4jPlugin status fills installedVersion/installationTime; composites were walked on both lines (5.26 and a Cypher 25 CalVer cluster). status.upgradeStatus.phaseStartTime and upgrade_duration_seconds were not exercised: no rolling version upgrade on this walk. Fixed in code (#443): (1) every new standalone emitted a ReconcileFailed warning on its first reconcile (a cache race reading back the StatefulSet it had just created), which diagnose then reported — since the initial commit; (2) status.graphShard.state read starting for minutes after the sharded family was online; (3) the database's PrivilegeTargetDropped event came twice with the role names in different orders; (4) diagnose counted a just-replaced pod in 2/3 pods ready but named it nowhere while it was PodInitializing; (5) make ccdr-e2e-up built the operator with a bare docker build, so both Part E operators reported vcs_ref="unknown". Docs fixed: the backup tutorial's step 5 (an online restore into a new database) cannot be followed on a standalone, which Getting Started recommends — it fails asking for stopCluster: true, and the guide only said so after step 6; and three places said an allDatabases backup excludes system, when the backup captures it and only the restore skips it. Rows corrected here: build-info (the dev overlay serves plain HTTP on :8443, not :8082), the upgrade row (release-neutral, plus the scheduled-backup check), composite reachability (cypher-shell prints no GQL codes), replica read-only (the database is named after the CR; a write routed from a non-hosting server fails transiently with 08N17), promotion (observedLagTxIds is absent at zero lag), the out-of-band promote call (it takes an options map); new rows for shard status and replica metrics. Observed on the walk, fixed after it (#445): ConnectivityDegraded fired during normal formation of a new cluster (ten failures in its first 91 seconds), so diagnose showed a minutes-old ⚠ on a healthy one — it now also needs five minutes; export replica-database emitted an empty status: {}; the published examples pinned 5.26.0-enterprise (the first LTS patch) and the sharding examples 2026.06 against the 2026.08.1 anchor — now 5.26-enterprise or the anchor, with a guard test so the drift cannot return. Filed, not fixed: a single server pod restart flips a formed cluster to Forming (#444, a status-model decision). Not ours: cypher-shell cannot verify a cert-manager server over neo4j+s:// from inside the pod (kubectl neo4j cypher works around it).
v1.17.0 release verification	2026-09-30	✅ Phases 0-5 pass (Phase 5: Parts A, B, E); the extended suite on the release commit then caught a blocker the walk could not — composites broken on a new CalVer deployment (#426); six defects fixed in code (#422, #423, #424), two doc fixes, and several stale rows in this doc corrected above	Operator built from e1a89e0 for Phases 0-3 and 5A/5B (the commits since touch only Aura, the CLI and tests), and from 7cb0653 for Part E. Upgrade from v1.16.0 (the new headline scenario): v1.16.0 was visibly toggling privileges on alternate reconciles (2 added / 5 revoked, then 2 added / 2 revoked) — the bug this release fixes. main on the old CRDs: role Failed with the CRD-predates message, zero privileges changed. After the CRDs: Ready, PrivilegesSynced=Unknown/UnattributedPrivileges, nothing revoked; the standalone recorded CYPHER_5 with a byte-identical ConfigMap and no restart; a row removed by hand was re-granted and attributed. The sharding cluster, created under v1.16.0, recorded CYPHER_25 with template hash, generation, ConfigMap and pod UIDs all unchanged, and re-derived it after the stamp was cleared. Fixed in code: (1) preflight -f checked the -n/kubeconfig namespace, not the manifest's own — a standalone bound for a restricted namespace got a warning where admission rejects its pods (#422). (2) explain marked ClusterNotReady=False — every healthy role — as ✗ with "not Ready yet" guidance; six negative-polarity conditions now read the right way round (#422). (3) explain UnattributedPrivileges had no entry, though every upgraded role reports it (#422). (4) export replica-database and the preflight reference said credentialsSecretRef does not project the credentials; it has since 4f8948e (#422). (5) The standalone wrote its cert-manager Certificate on every reconcile, so a webhook outage froze status and diagnostics (#423). (6) MinIO withdrew its public images — quay.io/minio/* now answers 401, after Docker Hub's went on 2026-09-12 — breaking the published example, the guide and every MinIO-backed extended spec; moved to Chainguard's build (#424). Docs fixed: allowedProcedures is an allowlist that hides everything unmatched (the old APOC row's apoc.version() check fails against the published example); the CCDR examples still said network mode was unsupported. Stale rows corrected: the crash-loop recipe (now caught by validation), the stored-form single-event expectation (holds only when databases exist first), the pinning row (scale the cert-manager controller only), Phase 3's anchor, annotation name and stamp-clearing step, the replica writer column, and the out-of-band reason. Verified pass, highlights: stored-form privileges stable across forced reconciles on both lines, including CalVer's EXECUTE USER DEFINED FUNCTION rendering, and exactly one PrivilegesApplied on the 3-server cluster; probe mode left 0 probe roles (28 CREATE/DROP lines in the security log); a dropped database produced one log line, not a storm, and re-resolved in 2s; privilegeRules with a dashed database name, the watch resolving in 1s; a remote composite constituent with stored credentials created on 5.26 (broken in v1.16.0); CalVer serverDefaultCypherLanguage: CYPHER_5 took exactly one roll, with neo4j keeping CYPHER 25; standalone, cluster, all-databases (MinIO) and cross-topology restores all round-tripped; Part E streamed live across two Kind clusters with TLS and mutual CA trust. Phase 4 (Aura), 2026-09-30, "priyo example org" only: read sweep — token works on v1 and v2beta1; org users are data-wrapped with user_id + organization-* roles (new fields email, mfa_*, last_activity_at, ignored by the decoder); ip-filters a bare array; CMK list refused with tenant-incapable-of-action on this tier (CMK stays never-exercised). CEL refusals all fire at apply: graphAnalytics + graphAnalyticsPlugin, serverless + secondariesCount, multiDatabase + vectorOptimized. Serverless without an org is refused before any API call. An AuraInstance with graphAnalytics: serverless created through the operator went Ready in 3m22s via v2beta1; Aura itself reports graph_analytics: serverless, multi_database: false; the instance ID was pinned, the connection Secret written; pause and resume both converged and the URI survived the resume; changing graphAnalytics afterwards was refused; deletionPolicy: Delete removed it — GET /v1/instances/<id> → 404, project instance count 0. Two wording defects fixed (#427): the serverless refusal carried reason MultiDatabaseUnsupported, and the creation event called it a "multi-database" instance. Extended suite blocker (#426): since #416 a new CalVer deployment defaults to Cypher 25, which refuses the operator's unpinned `comp`.`name` constituent DDL (42NAA) and reads `comp.name` as the constituent where Cypher 5 reads a plain alias — every composite on a fresh CalVer deployment went Failed, and the OIDC remote path had never worked. This walk ran composites on 5.26 only, which is why it missed it: walk the composite rows on CalVer too (added to the rows below). Observations, not fixed: cypher-shell output carries "Picked up JAVA_TOOL_OPTIONS" once seed credentials project the path-style flag; kubectl neo4j cypher still has no database flag (USE <db> works); a sharded CREATE … IF NOT EXISTS ran on two consecutive reconciles (idempotent).
v1.16.0 release verification	2026-09-23	✅ Phases 0-3 and all of Phase 5 pass; eleven findings, nine fixed, two needed no code	Full journey on the release candidate. Phase 0 — every offline check, the exit-code contract, both skip taxonomies, and the three new cloud-substrate preflight checks on Kind; the new default-StorageClass check found a true finding immediately (Kind's standard is WaitForFirstConsumer with allowVolumeExpansion unset). The PSA check's claim that the operator's default satisfies restricted was proved against real admission: default admitted, override rejected. Phase 1 — standalone on 5.26.30; composites end to end including all three halves of the reachability trap (42N04, and the server's own advice failing with 22N51), the 42N87 ordering guard, and GraphPrivilegeOnComposite; users/roles with a hyphenated CR name and a real login; alias re-point and drift correction; APOC holding Ready across 8 samples; backup→restore with the marker returning; system refused; support bundle with zero hits for both real passwords. Phase 2 — 3 servers Available/Enabled, a 3-primary database, cypher routing correctly on a cluster, diagnose exit 0 both healthy and mid-restart, restore into a new database, all-databases restore, and cross-topology restore carrying a marker from a 5.26.30 standalone into a 2026.08.1 cluster — across a topology and a version boundary. Phase 3 — 2026.06.0, propertyShardingReady, the shard family online with the CR name differing from the logical name, sharded backup by CR name capturing all three artifacts, and the logical name correctly refused. Phase 5 — A/B on MinIO: replicationPullURI, all three chain-breaking configs rejected, a real FULL (tx -1 to 5) then DIFF (tx 6 to 6) chain, export replica-database emitting a clean manifest that validates; then the downstream pulled from the bucket after the upstream was deleted, all 4 rows arriving, write refused, alias pre-staged against a replica, promotion to standard, the alias surviving with generation still 1 (a write through it succeeded), the Promoted CR deleted without dropping the database, and out-of-band promotion observed rather than corrected. C and D passed in the dispatched extended suite. Part E — two Kind clusters, TLS, mutual CA trust: proxy on 172.18.255.200:16000/:16001, the IP in the certificate's ipAddresses, CA md5s matching both ways, replica Replicating 0 behind, and a row written upstream appearing downstream in 10s. Findings. Fixed: the composite constituent lookalike loop (#394); preflight contradicting the backup validator (#395); validate ignoring misspelled fields, plus a published example using spec.authentication that could never have worked (#396); the typo check skipping kinds that skip offline (#397); ccdr-e2e-up never able to create its second cluster — both bound 127.0.0.1:6443, so Part E's automation had never run (#401); the typo check covering only 13 of 27 kinds (#402); an auth-rule spec that deadlocks against the operator it tests (#403). Docs: the connect row, the validate kind counts, the restore seed URI (in an event, not status), and the advertised-address row now naming the file it warns against — this release's env-var fix moved --config-dir back to /var/lib/neo4j/conf and deleted /tmp/neo4j-config, so the row greps an absent path and reads exactly like the override failing. No change needed: --access-mode (docs already correct, my error), and writer=TRUE on a replica — Neo4j's own column alongside access=read-only, with the write functionally refused. Phase 4 not run — nothing under internal/aura/ changed.
v1.12.2	2026-06-14	✅ all phases pass	Doc bug: backup_restore.md sharded target.name said logical name, must be CR name (fixed, #270 follow-up). v1.12.2 surfaces (#260/#268/#269/#270) verified live.
(pre-merge, PR #333)	2026-08-07	⚠️ partial — Phases 1 (alias scenarios) + 5A/5B validation only	Bug found + fixed: spec.auth is optional on Cluster/Standalone, but five call sites dereferenced Spec.Auth.AdminSecret unguarded — a Neo4jDatabase against a standalone without spec.auth panicked the reconciler and the CR sat with an empty status forever, with nothing on the CR explaining why. Widest site was ResolvedTarget.NewClient, shared by the user/role/binding/authrule and replication controllers. Fixed to use the existing nil-safe helpers. Also: make deploy-dev-local reports "deployment configured" without restarting the pod when the :dev tag is unchanged, so a rebuilt binary silently does not take effect — needs kubectl rollout restart. Verified pass: alias create/resolve, blue-green re-point, drift correction, alias delete does not drop the target; replica version gate (2026.08 required, refused on 2026.06), network-mode rejection naming advertised_address, replication-source R1/R2/R4 rejections, status.replicationPullURI published. Not run: replica seeding, promotion, and alias-survives-promotion — need a 2026.08+ image, which is not yet published (highest on Docker Hub is 2026.06).
(pre-merge, PR #343)	2026-08-21	⚠️ partial — Phase 1 TLS scenarios only	Bug found + fixed: the operator's "insecure TLS fallback" for Secrets without ca.crt had never worked. It set InsecureSkipVerify: true, but the Bolt driver derives that field from the URI scheme and resets it (connector.tlsConfig()), so Go verified against system roots and every connection failed x509: certificate signed by unknown authority — while the operator logged that verification was disabled. So spec.tls.strictPeerValidation: false could not connect at all, and the warning named a state that never existed. Replaced with pinning tls.crt as a one-certificate RootCAs. A/B on one cluster, same Secret: pre-pinning binary failed 6/6 reconciles; pinning binary connected and collected diagnostics. Also verified: CA path unchanged (no PINNING line, diagnostics fresh), pinned fingerprint matches openssl, a mismatched tls.crt is refused, and restoring it recovers. Lesson: handshake unit tests dialed the config verbatim and so could not see the driver's override — they now emulate it (asDriverWouldUse).
(pre-release, v1.15.0 candidate)	2026-09-03	⚠️ Phases 0-3 walked incl. the TLS trio; Phase 5 blocked — all findings below fixed and re-verified live the same day	3 blockers. (1) Flags after a positional argument are silently ignored across diagnose/explain/connect/cypher/preflight/export — stdlib flag stops parsing at the first non-flag arg. It fails in the wrong direction: the wrong namespace is queried and reported as "not found". Every documented export invocation fails outright; preflight Neo4jBackup/nightly -n neo4j and cypher prod -n neo4j from the published docs silently target default. (2) cypher -c can hang forever and leaves an orphaned cypher-shell inside the database pod: it execs with -i and no --non-interactive, so the shell runs the query then reads stdin for more. Twelve orphans accumulated across two hangs and the database stopped answering over Bolt and HTTP while the pod stayed Ready — the anti-wedge signature, reached with a single deployment. (3) cypher is unusable against a cluster: it dials bolt://localhost on one chosen pod instead of routing, and the default neo4j database has 1 primary, so 2 of 3 servers answer "Database neo4j not found"; there is no database flag to work around it. Also: cross-topology restore (this doc's own Phase 2 scenario) fails — a single-database restore from an allDatabases backup demands artifactFilename, which such a backup never sets, while databaseArtifacts holds the exact filename; the error tells the user to "re-run the backup with a recent operator", which does not help. Neo4jPlugin phase flaps Ready↔Installing forever while APOC works. support-bundle collects no operator logs though its help text and docs promise them. explain does not know the phase Ready. Two validator asymmetries: dbms.mode is rejected on standalone but accepted on cluster; heap-vs-limit is rejected on cluster but accepted on standalone (demonstrated: a standalone that validated clean crash-looped). Three published examples fail validation (plugins/standalone-plugin-example.yaml, property_sharding/advanced-property-sharding.yaml, end-to-end/complete-deployment.yaml) — and all 8 of those errors pass kubectl apply --dry-run=server, which answers design Q1 empirically: validate earns its place. Second pass (TLS trio + Phase 3) found one more CLI defect and one doc defect. kubectl neo4j cypher could not connect to ANY TLS deployment: cypher-shell takes trust from the JVM truststore and has no CA flag, so a cert-manager certificate produced only "Failed to establish secured connection with the server". Fixed by building a throwaway truststore in the pod from /ssl/ca.crt (falling back to /ssl/tls.crt, which is the anchor the operator itself pins when ca.crt is absent) rather than dropping to the unverified +ssc scheme — and the verification was proven REAL by pointing it at a mismatched certificate and watching it refuse. The Phase 3 sharded-backup row named a field (spec.target) that does not exist; corrected above. TLS trio all pass: CA path (no PINNING line, diagnostics fresh); pinning engages with pinnedFingerprintSHA256 equal to openssl's and diagnostics still advancing; a same-SAN certificate from another CA is refused with x509: certificate signed by unknown authority, and restoring tls.crt recovers. Phase 3 all pass on 2026.06.0 enterprise with propertyShardingReady=true: a Neo4jShardedDatabase whose CR name (test-sharded-db) differs from its logical name (testdata) went Ready with testdata, testdata-g000, testdata-p000 and testdata-p001 all online, and one backup captured all three shard artifacts. Fixes verified live on a fresh cluster afterwards: the documented export form parses; cypher -c returns on a cluster (5/5, zero orphaned shells, writes included); diagnose Kind/name -n ns works; explain Ready gives guidance; the support bundle carries operator/<ns>/<pod>/manager.log (found via the Deployment's selector — the identifying label is on the Deployment, not its pods) with redaction intact; the plugin phase held Ready across 12 samples while APOC answered apoc.version(), and plugin reconciles fell from ~1/second to 2/minute. Verified pass: standalone Ready in 96s from the published URL, database/user/role incl. hyphenated CR-name resolution and all three privileges, APOC, backup→restore round trip, system not restorable, alias create/re-point/drift-correction, support-bundle redaction (grepped for all three real passwords, zero hits), cluster formation 3/3 with 3-primary topology, diagnose on healthy vs. restarting pods.
(v1.15.0 — network mode across TWO Kind clusters)	2026-09-13	✅ the last unverified path in CCDR now works end to end	With make ccdr-lb there is no longer a reason to leave this unrun. Two separate Kind clusters, operator in both, MetalLB on the upstream only: the proxy took 172.18.255.200, the downstream cluster reached it (nc -z from a pod in the second cluster), and a Neo4jReplicaDatabase with source.mode: network and source.addresses pointing at the proxy reached Replicating "0 transactions behind". type=replica/access=read-only/writer=false on both copies, the seeded rows readable downstream, and a row written to the upstream appeared downstream within 40s — so the live stream works, not just the seed. Two false alarms worth recording, both mine. kind create cluster switches the current context, so a batch of kubectl commands silently targeted the wrong cluster. Then the downstream appeared to be missing the Neo4jReplicaDatabase, Neo4jReplicaPromotion and Neo4jDatabaseAlias controllers in production mode — 23 started, not 26 — which looked exactly like a release blocker. It was the harness: config/default renders ghcr.io/…:latest, not controller:latest, so the sed that was meant to swap in the local build did nothing and the cluster ran a PUBLISHED release predating those controllers. With the right image all 26 start. If you deploy a second cluster by hand, check the image before believing a missing controller.
(v1.15.0 — CCDR proxy, first time ever)	2026-09-13	✅ the proxy path verified end to end with MetalLB; one security defect found and fixed	The gap this doc recorded as "no LoadBalancer-on-Kind tooling today" was hiding a live defect, not just an unverified path. spec.crossClusterReplication.loadBalancerInternal defaults to true and calls itself the main mitigation against public exposure; it was computed and thrown away — CCDRProxyLoadBalancerInternalEffective had no caller outside its own unit test, and no internal-LB annotation existed anywhere in the codebase. On AWS, Azure or GCP that meant a PUBLIC load balancer in front of the transaction-shipping port. Fixed in 9c46aaa, verified live on the Service. The harness: hack/metallb-setup.sh + make ccdr-lb, an L2 pool carved from the top of the kind Docker network (172.18.255.200-250), which Kind clusters share — so it is reachable from a second Kind cluster with no host routing. Verified with it: the proxy Service gets 172.18.255.200, status.crossClusterReplication goes ready: true publishing one address per server ordinal (:16000, :16001), and the servers advertise the proxy endpoint — server.cluster.advertised_address=172.18.255.200:16000. None of that had ever been observed. Also corrected: the Part D advertised-address row named the wrong file for the second day running; the server runs with --config-dir=/tmp/neo4j-config, and /var/lib/neo4j/conf/neo4j.conf is the image default the operator never writes.
(v1.15.0 — network mode with TLS ON, two Kind clusters)	2026-09-14	✅ the configuration CCDR is actually sold with now works; two operator bugs found, both release blockers	Every earlier network-mode run had TLS off. Turning it on — mode: cert-manager, strictPeerValidation: true, each cluster carrying the other's CA in spec.tls.additionalClusterTrustCAs — broke it twice, in ways no unit test could see. (1) The peer-CA feature could not work at all. Each CA was mounted as its own volume with a subPath at /ssl/trusted/peer-ca-<i>.crt, but /ssl/trusted/ is content of the certs volume itself (its own trusted/ca.crt item), and a volume cannot be mounted inside another volume's content: the kubelet refused the pod with "not a directory" and the server crash-looped with exit 128. The certs volume is now a projected volume carrying the cluster's own material plus one trusted/peer-ca-<i>.crt item per peer (d36c775) — which is what the API reference already claimed it did. (2) The fix changed nothing, which was the second bug. A forming StatefulSet accepted only hasCriticalTemplateChanges (image, resources, securityContext, serviceAccount) — not volumes, env or init containers. So a template defect that stops the pods going ready makes every reconcile take the conservative branch, which drops the repair, which keeps the pods from going ready: a cluster wedged by its own template could not be fixed by editing the CR, only by deleting the StatefulSet by hand. A forming StatefulSet now also accepts a template whose hash differs from the one the controller last stamped, since that is intent rather than churn (40713d7). Verified after both fixes: the corrected volume reached a StatefulSet at 1-of-2 ready, both servers rolled, both clusters went Ready; each side holds the other's CA in /ssl/trusted/ (md5 of dev's peer-ca-0.crt equals DR's own ca.crt and vice versa); a Neo4jReplicaDatabase in network mode against the proxy at 172.18.255.200:16000 reached Replicating "0 transactions behind"; the 3 seeded rows read downstream, a 4th written upstream appeared downstream, and a write to the replica was refused. Both bugs needed a real two-cluster TLS walk — the first is a kubelet behaviour, the second only appears when a cluster is already failing. Observation, not fixed: the replica's ClusterNotReady condition is negative-polarity (status: False + reason: ClusterReady means the cluster IS ready). It is consistent across the role, rolebinding and replica controllers, so it is a naming convention rather than a defect; renaming it is a status-API break and not release-day work.
v1.15.0 release verification	2026-09-13	✅ Phases 0-3 + TLS trio + Phase 5 pass on current main; one incomplete fix caught and finished	Full journey on the release candidate, after the day's CCDR and CLI work. Phase 0 — plugin discovery, exit-code contract, every example directory clean, --version, explain Ready; preflight caught the unschedulable request and the missing StorageClass before apply; diagnose named ImagePullBackOff, the unbound PVC (now marked … as the consequence it is), the missing class and the scheduling failure; version skew fires when forced. Phase 1 — standalone Ready from the published raw URL; users/roles incl. hyphenated CR-name resolution; APOC working with the plugin phase holding Ready across 8 samples (the flap fix); backup→restore round trip with the marker returning; cypher -c writing without hanging; support bundle carrying operator/<ns>/<pod>/ logs with zero hits for either real password. TLS trio all pass — CA path clean, pinning with a fingerprint equal to openssl's, a same-SAN rogue certificate refused with x509: certificate signed by unknown authority and recovery on restore; the CLI reached a TLS deployment and fell back to tls.crt while ca.crt was absent. Phase 2 — 3 servers Available, a 3-primary database online, cypher routed correctly on a cluster. Phase 3 — propertyShardingReady, the sharded family online, and a backup by CR name capturing all three shard artifacts. Phase 5 — preflight and the operator both catch a downstream with no AWS_REGION, the CR failing immediately with a message naming the variable and the cluster instead of stalling in Seeding; adding the variable unblocks it. The find: the morning's failure-surface fix was incomplete. It stopped duplicate status writes but not the alternation — Seeding was announced before every attempt and Failed written after it, so each pass was a genuine change and drove the next reconcile: 1153 failures logged in 45 seconds against a bucket that did not exist. Announcing Seeding only when the previous attempt had not already failed brought that to 0 in a settled minute, with the phase staying Failed and the server's own error in status.message. Only running it found that.
(follow-up, v1.15.0 — backup-mode credentials)	2026-09-13	✅ backup mode works; the gap is documentation, not machinery	Re-tested the open item from 2026-09-12. Backup-mode replication seeds and replicates as shipped, provided the downstream cluster carries the bucket credentials in spec.env — AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, plus AWS_ENDPOINT_URL_S3 for S3-compatible stores. Verified end to end on 2026.08.1: all four vars reached the server container, the replica went Replicating "0 transactions behind", and both seeded nodes were readable from the MinIO chain with type=replica/access=read-only on every copy. The seed and pull are performed by the SERVER via the AWS SDK's default chain, which is why the credentials belong there — so this is a documentation and failure-surface gap, not a missing feature, and the earlier "cannot work against object storage" reading was too strong. spec.source.credentialsSecretRef is still consumed nowhere and is now documented as such on the field itself, in the guide, in the troubleshooting table and in the example. Still worth fixing: the CR stays Seeding with an optimistic message while the server refuses, and retries a permanent ArgumentError about once a second — the error never reaches status.message, which is what made this expensive to diagnose.
(pre-release, v1.15.0 — Phase 5 B and C)	2026-09-12	⚠️ Parts B and C run on the first real 2026.08.1; two operator bugs fixed, one open	Three CCDR statements were unusable on the version that introduces the feature. CREATE REPLICA DATABASE and CALL dbms.promoteReplicaDatabase are Cypher 25 syntax, and system still defaults to Cypher 5 on 2026.08, so the server does not warn — it reports the feature as ABSENT ("Invalid input 'DATABASE': expected a graph pattern"; "There is no procedure with the name … registered"). Both fixed by prefixing with CYPHER 25 (aa1fea6, e242a9c) and verified end to end. The operations manual omits the prefix because its examples assume a Cypher 25 default — no unit test can see a parser's opinion, so only running it against a real server could find this. STILL OPEN — backup-mode replication cannot work against object storage: the seed and pull are performed by the SERVER, which therefore needs cloud credentials and a region in its own environment. AWS_REGION is projected only into backup/restore Jobs, never onto the servers, and Neo4jReplicaDatabase.spec.source.credentialsSecretRef is declared in the CRD and read nowhere in the codebase. Live result: CREATE REPLICA DATABASE reaches the server and fails "Unable to load region from any of the providers", while the CR sits at Seeding forever with a message that never mentions the error, retried ~1×/second on a permanent ArgumentError. Part C (network mode) passes fully and needs no object storage: replica Replicating "0 transactions behind", type=replica/access=read-only/writer=false on all three copies, data across the wire. Part B passes on a network-sourced replica, which exercises the same promotion machinery: write refused ("database is read only") while --access-mode=read succeeds; an alias pre-staged against the replica goes Ready and, unchanged, accepts writes after promotion; CR-driven promotion converts every copy; deleting a Promoted CR leaves the database and its data intact; and out-of-band promotion is observed, not "corrected" — promoted by hand at a cypher-shell, the operator reported Completed/Promoted without dropping the database, which is the failure mode the whole observe-before-act design exists to prevent. observedLagTxIds is absent at zero lag by design; the message carries "0 transactions of lag were made permanent" and lastCommittedTxn.
(pre-release, v1.15.0 — Phase 5 partial)	2026-09-12	⚠️ Phase 5: version gate + Parts A and D pass; Parts B and C still blocked	The 2026.08 tags on Docker Hub are EMPTY. 2026.08-enterprise and 2026.08.1-enterprise appeared on 2026-09-11 and are listed by the Hub tags API, but carry no image: images: [], docker manifest inspect says "no such manifest", and an explicit pull fails for both amd64 and arm64. A tag listing is not an image — check images[] is non-empty before concluding a version is available. The real pullable ceiling is still 2026.07.1. Ran anyway, because only hosting a replica needs 2026.08: scenario 1 — the version gate — fires exactly, Ready=False(ReplicaVersionTooOld) with "requires Neo4j 2026.08 or later on the downstream cluster; ccdr-cluster runs 2026.06-enterprise", and it is checked BEFORE a missing credentials Secret, so nothing else proceeds as designed. Part A all pass on a 2026.06 upstream: replicationPullURI published, all three chain-breaking configs refused with the rule named, a FULL (tx -1→2) then a real differential (tx 3→5) landing as 5.1KiB + 2.0KiB artifacts at exactly the published URI, and export replica-database emitting a clean manifest on stdout with notes on stderr that validate then accepts. Part D all pass: proxy ConfigMap/Deployment/Service render and the cluster still reaches Ready with the LoadBalancer stuck <pending>; NetworkPolicy goes 3 → 4 ingress rules with the new one scoped to the ccdr-proxy pod on 6000 only; the advertised address stays the pod's own; toggling off removes everything and returns the policy to 3 rules. Three doc rows corrected (the advertised-address row named the ConfigMap, which never holds the value; Part A gained the same-path guard; the FULL/DIFF row now says where the backup type actually lives). One CI defect found: minio/minio:latest and minio/mc:latest on Docker Hub are no longer anonymously pullable (pull access denied … insufficient_scope, on both the host and a Kind node), which would fail the extended MinIO specs; switched to quay.io, where the published example already pointed. Parts B and C need a real 2026.08+ image and remain unrun.
v1.14.0	2026-08-02	✅ all 4 phases pass (19/19 scenarios) — 3 bugs + 4 doc gaps found	Bugs: (1) deleting a Neo4jPlugin against a live deployment never finalizes — the removal Job is re-created every reconcile and fails already exists, so the CR stays Terminating and blocks namespace deletion; (2) examples/property_sharding/development-property-sharding.yaml does not boot — spec.propertySharding.config bypasses the memory-key exclusion that spec.config gets, so a user heap.max_size contradicts the operator-derived heap.initial_size and the JVM refuses to start; (3) make dev-up loads no aura* controller (the dev -controllers default omits all 12), so every Aura CR is accepted and then silently ignored — no status, no events, no logs. Doc gaps: Phase 1 teardown destroys the backup Phase 2's cross-topology scenario needs; Phase 4 teardown leaves a billing instance because deletionPolicy defaults to Orphan; the multiDatabase-stays-absent rule holds only with no org configured; sharding examples pin 2026.04 while the CI anchor is 2026.06. Also: a crash-looping member surfaces only as ConnectivityDegraded, which points at Bolt rather than the container.

See also

• Testing — the automated unit/integration suites and the core/extended label split.
• CI/CD & Workflows — what runs per-PR vs. the manual extended suite.
• kubectl-neo4j CLI — the command docs Phase 0 and the per-phase CLI rows follow, including the exit-code contracts.
• Backup & Restore, Property Sharding — the user docs this journey follows.
• Project Invariants — the hard constraints (KIND-only, Enterprise-only, etc.) every phase respects.