Backup & Restore Troubleshooting

Common backup and restore failures and their fixes. For the feature overview and configuration, see the Backup & Restore guide.

Tip

kubectl neo4j validate -f <manifest> catches a malformed Neo4jBackup before you apply it, kubectl neo4j preflight -f <manifest> checks the cluster-side preconditions the manifest depends on — the credentials Secret's keys, or the ServiceAccount's cloud-identity annotation — and kubectl neo4j explain Neo4jBackup/<name> decodes the phase of one that has already failed. See the CLI guide.

The credential checks below are written as kubectl run probes. preflight covers the shape of the same problems without a probe pod; reach for a probe when you need to prove the credentials actually work against the bucket, which preflight deliberately does not claim to do.

Common Backup Issues

Backup Job Failures

Symptom: Backup job fails to start

kubectl get jobs -l app.kubernetes.io/component=backup
# STATUS: Failed or no jobs created

Diagnosis:

# Check backup resource status — status.phase tells you which stage it's in:
#   Invalid   → spec failed validation (message has the details)
#   Waiting   → target cluster missing or not Ready yet (transient)
#   Pending   → transient precondition (chain parent missing, chain busy,
#               sharded preflight couldn't connect) — retried automatically
#   Failed    → terminal for one-shot backups
kubectl get neo4jbackup
kubectl describe neo4jbackup production-backup

# Check operator logs for backup controller errors
kubectl logs -n neo4j-operator-system -l control-plane=controller-manager --tail=-1 | grep -i backup

Common Causes & Solutions:

1. Missing ServiceAccount: ```bash

The Neo4jBackup controller creates the backup ServiceAccount when it reconciles a backup

(the cluster controller does not) — check it exists.

(No Role/RoleBinding is created: the backup Job needs no Kubernetes API access.)

kubectl get serviceaccount neo4j-backup-sa

If missing, the Neo4jBackup has not been reconciled yet (or already reached

Completed/Failed, which are terminal). For a pending or scheduled backup, trigger

reconciliation with a no-op annotation change on the Neo4jBackup itself:

kubectl annotate neo4jbackup production-backup troubleshooting.neo4j.com/reconcile="$(date +%s)" --overwrite ```

1. Storage Configuration Issues:

   # Verify storage configuration in backup spec
   spec:
     storage:
       type: s3
       bucket: "valid-bucket-name"    # Must exist
       path: "backups/"
       cloud:
         provider: aws
         identity:
           provider: aws
       # Credentials must be valid
   

2. Cluster Reference Problems:

   # Verify cluster exists and is ready
   kubectl get neo4jenterprisecluster production-cluster
   kubectl get pods -l neo4j.com/cluster=production-cluster
   

Symptom: Backup job starts but fails during execution

Diagnosis:

# Check backup Job's Pod log (one-shot: <neo4jbackup-name>-backup;
# CronJob child: named by Kubernetes after the CronJob, <neo4jbackup-name>-backup-cron-<suffix>
# — find it with `kubectl get jobs`).
kubectl logs -n <ns> job/<job-name>

# Check Neo4j server logs for backup-related errors (which server
# was the leader at backup time).
kubectl logs <cluster>-server-0 -c neo4j | grep -i backup

Common Solutions:

1. Insufficient Disk Space (PVC storage):

   # `kubectl exec` into any server pod to inspect the bound PVC.
   kubectl exec <cluster>-server-0 -c neo4j -- df -h /data
   

   Increase Neo4jBackup.spec.storage.pvc.size (only effective when the operator provisions the PVC — see bring-your-own PVC otherwise). Note that retention.maxCount / retention.maxAge do not prune continuously — PVC retention runs only when the Neo4jBackup CR is deleted, so a long-lived scheduled CR accumulates artifacts until then. Prune manually (or via a recurring neo4j-admin backup aggregate) if the volume fills.

2. Database Lock Issues:

   # Check for long-running transactions
   kubectl exec production-cluster-server-0 -- cypher-shell -u neo4j -p password \
     "SHOW TRANSACTIONS YIELD transactionId, currentQuery, elapsedTime WHERE elapsedTime > duration({seconds: 30})"
   
   # Solution: Wait for transactions to complete or schedule backups off-peak
   

3. Memory Issues in Backup Process: Backup pod resources default to a Burstable profile (request 100m CPU / 512Mi, limit 1 CPU / 2Gi) but are tunable via spec.options.resources (standard requests/limits). Raise them for large databases; spec.options.pageCache controls the page-cache hint passed to neo4j-admin.

Cloud Storage Issues

S3 Backup Failures

Authentication Issues:

# Check AWS credentials using the backup service account (default: neo4j-backup-sa)
kubectl run backup-auth-check --rm -it --image=amazon/aws-cli --serviceaccount=<backup-serviceaccount> -- aws sts get-caller-identity

# Test S3 access
kubectl run backup-auth-check --rm -it --image=amazon/aws-cli --serviceaccount=<backup-serviceaccount> -- aws s3 ls s3://your-backup-bucket/

Solutions:

1. IAM Role Issues:

   # Use IAM roles for service accounts (IRSA) on the Neo4jBackup CR.
   # The annotations land on the operator-managed neo4j-backup-sa.
   apiVersion: neo4j.neo4j.com/v1beta1
   kind: Neo4jBackup
   spec:
     storage:
       type: s3
       bucket: your-backup-bucket
       cloud:
         provider: aws
         identity:
           provider: aws
           autoCreate:
             annotations:
               eks.amazonaws.com/role-arn: "arn:aws:iam::123456789:role/Neo4jBackupRole"
   

2. Bucket Policy Problems:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::123456789:role/Neo4jBackupRole"
         },
         "Action": [
           "s3:GetObject",
           "s3:PutObject",
           "s3:DeleteObject",
           "s3:ListBucket"
         ],
         "Resource": [
           "arn:aws:s3:::your-backup-bucket",
           "arn:aws:s3:::your-backup-bucket/*"
         ]
       }
     ]
   }
   

3. Identity conflicts between CRs — backups that worked start failing auth after another backup/restore CR was added: neo4j-backup-sa / neo4j-restore-sa are shared per namespace, and two CRs declaring different identity.autoCreate.annotations overwrite each other (last writer wins). Look for a ServiceAccountAnnotationConflict Warning event (kubectl get events --field-selector reason=ServiceAccountAnnotationConflict). Fix: one identity per namespace with access to all backup locations, or split the CRs across namespaces.

Google Cloud Storage Issues

Service Account Problems:

# Check GCP credentials using the backup service account (default: neo4j-backup-sa)
kubectl run backup-auth-check --rm -it --image=google/cloud-sdk:slim --serviceaccount=<backup-serviceaccount> -- gcloud auth list

# Test GCS access
kubectl run backup-auth-check --rm -it --image=google/cloud-sdk:slim --serviceaccount=<backup-serviceaccount> -- gsutil ls gs://your-backup-bucket/

Solutions:

# Use Workload Identity on the Neo4jBackup CR
# (annotations land on the operator-managed neo4j-backup-sa)
apiVersion: neo4j.neo4j.com/v1beta1
kind: Neo4jBackup
spec:
  storage:
    type: gcs
    bucket: your-backup-bucket
    cloud:
      provider: gcp
      identity:
        provider: gcp
        autoCreate:
          annotations:
            iam.gke.io/gcp-service-account: "neo4j-backup@project.iam.gserviceaccount.com"

Azure Blob Storage Issues

Authentication Problems:

# Check Azure credentials using the backup service account (default: neo4j-backup-sa)
kubectl run backup-auth-check --rm -it --image=mcr.microsoft.com/azure-cli --serviceaccount=<backup-serviceaccount> -- az account show

# Test storage access
kubectl run backup-auth-check --rm -it --image=mcr.microsoft.com/azure-cli --serviceaccount=<backup-serviceaccount> -- az storage blob list --account-name storageaccount --container-name backups

Scheduled Backup Issues

Symptom: Scheduled backups not running

Diagnosis:

# Check CronJob status
kubectl get cronjob
kubectl describe cronjob production-backup-backup-cron   # the operator names it <neo4jbackup-name>-backup-cron

# Check backup schedule configuration
kubectl get neo4jbackup production-backup -o yaml | grep -A 10 schedule

Common Solutions:

1. Invalid Cron Expression:

   # Correct cron syntax
   spec:
     schedule: "0 2 * * *"    # Daily at 2 AM
     # NOT: "0 2 * * * *"     # Invalid - too many fields
   

2. Timezone Issues: Schedules run in UTC — there is no spec.timezone field, and TZ=/CRON_TZ= prefixes in the schedule string are rejected by the validator (Kubernetes refuses timezone-embedded CronJob schedules). Convert your local time to UTC:

   spec:
     schedule: "0 2 * * *"    # 02:00 UTC — adjust the hour for your timezone
   

3. Backup Window Conflicts:

   # Check for overlapping backup jobs
   kubectl get jobs -l app.kubernetes.io/component=backup --sort-by=.metadata.creationTimestamp
   

Common Restore Issues

Restore Job Failures

Symptom: Restore job fails to start

Diagnosis:

# Check restore resource status
kubectl get neo4jrestore
kubectl describe neo4jrestore production-restore

# Check operator logs
kubectl logs -n neo4j-operator-system -l control-plane=controller-manager --tail=-1 | grep -i restore

Common Solutions:

1. Invalid Backup Reference:

   # Verify backup exists
   kubectl get neo4jbackup production-backup
   
   # Check backup completion status. One-shot backups reach phase "Completed";
   # scheduled backups stay in "Scheduled" — check status.history for a
   # Succeeded run instead.
   kubectl get neo4jbackup production-backup -o jsonpath='{.status.phase}'
   kubectl get neo4jbackup production-backup -o jsonpath='{.status.history[*].status}'
   

   A restore whose referenced backup has no Succeeded run yet sits in Pending ("Waiting for backup to complete") and retries automatically — that is not an error.

2. Target Cluster Issues:

   # Ensure target cluster is ready
   kubectl get neo4jenterprisecluster target-cluster
   kubectl get pods -l neo4j.com/cluster=target-cluster
   

3. Storage Access Problems:

   # Run a transient pod to test S3 access from inside the cluster.
   kubectl run -it --rm s3-test --image=amazon/aws-cli \
     --restart=Never --env-from=secretRef/aws-backup-creds -- \
     s3 ls s3://backup-bucket/path/to/backup/
   

Symptom: Restore job fails during execution

Diagnosis:

# The restore's status and events say which path it took and what failed
kubectl describe neo4jrestore production-restore

# Online restores (the default): the database's own status names the cause
kubectl exec target-cluster-server-0 -c neo4j -- cypher-shell -u neo4j -p <password> \
  "SHOW DATABASE <name> YIELD name, currentStatus, statusMessage"

# Offline standalone restores only: the restore Job's log
kubectl logs job/production-restore-restore

Common Solutions:

1. Insufficient Storage Space:

   # Check available space on target cluster
   kubectl exec target-cluster-server-0 -- df -h /data
   
   # Solution: Increase PVC size before restore
   

2. Database Already Exists:

   # Overwrite the existing same-named database with the replaceExisting flag.
   spec:
     options:
       replaceExisting: true  # online: destructive recreate; offline standalone: --overwrite-destination=true
   

3. Version Incompatibility:

   # Check Neo4j versions
   kubectl exec source-cluster-server-0 -- neo4j version
   kubectl exec target-cluster-server-0 -- neo4j version
   

Symptom: Restore reports Failed with "server pods can't reach the seed source"

The Neo4jRestore message reads cluster "<name>"'s server pods can't reach the seed source: missing <credentials and/or endpoint settings>. The server JVM fetches the seed itself. Provide these on the cluster CR, or set annotation neo4j.com/auto-inherit-seed-creds="true" … (with standalone "<name>" on a standalone), and a SeedEndpointNotProjected Warning event carries the same text.

Cause: an online restore from cloud storage makes the Neo4j server pods fetch the seed, so they need the cloud credentials Secret projected via spec.extraEnvFrom (and, for MinIO / S3-compatible stores, the endpoint in spec.env).

Fix:

# On the Neo4jEnterpriseCluster or Neo4jEnterpriseStandalone CR:
spec:
  extraEnvFrom:
    - secretRef:
        name: <your-backup-creds-secret>

Or, set the annotation neo4j.com/auto-inherit-seed-creds=true on the cluster or standalone CR — the operator patches extraEnvFrom (and the endpoint) automatically, which triggers one rolling restart so Neo4j picks up the env vars.

Symptom: Restore reports Failed with "the seed proxy could not merge the backup chain"

Cause: the restore seeds from a PVC backup whose artifact is (or may be) a differential, so the seed proxy merges its chain into one full backup first — and that failed. The message carries neo4j-admin's reason. Common causes:

• A link of the chain is gone ("Unable to find valid backup chain"): the full backup the differential builds on was deleted — by hand, or by retention pruning that removed a full whose differentials remained.
• The scratch space ran out ("cannot copy … to the scratch volume"): merging copies the chain first. Set spec.options.tempStorage to at least the chain's size.

Fix: address the cause and re-trigger the restore by changing its spec, or restore from a run whose chain is complete.

Symptom: Standalone restore reports Failed with "cannot run against a running standalone (it restores offline: …)"

Cause: this standalone restore cannot run online — the message names why (a point-in-time restore that cannot run online, a source.type: storage path, cloud storage by pod identity on a standalone without spec.podServiceAccountAnnotations, a backup run with no recorded artifact) — so it runs neo4j-admin through a Job, which needs the instance stopped.

Fix: set spec.stopCluster: true. The operator stops the standalone for the restore and starts it again afterwards; every database on it is offline meanwhile.

Symptom: Restore stuck in Running, no Job created

Expected. Online restores — every cluster restore and most standalone ones — use Cypher (dbms.recreateDatabase or CREATE DATABASE OPTIONS{seedURI}); no Job is spawned. The restore's events show the seed URI it issued, and the database's status shows how seeding is going:

kubectl describe neo4jrestore <name>     # "Online restore into …", "Online restore seed failed"
kubectl exec <server-pod> -c neo4j -- cypher-shell -u neo4j -p <password> \
  "SHOW DATABASE <name> YIELD name, currentStatus, statusMessage"

If the status message is No seed providers found to satisfy the provided uri 's3://...', the server pods don't have the cloud creds projected — see the first symptom above. The restore fails once spec.timeout (default 5m) passes without the database coming online; raise it for large stores.

Symptom: Sharded DB restore rejected with "use Neo4jShardedDatabase.spec.replaceExisting"

Cause: Neo4jRestore doesn't support sharded databases — the Cypher shape (SET GRAPH SHARD / SET PROPERTY SHARDS) only fits CREATE DATABASE, not dbms.recreateDatabase.

Fix: restore via the Neo4jShardedDatabase CR's destructive-restore flow:

apiVersion: neo4j.neo4j.com/v1beta1
kind: Neo4jShardedDatabase
metadata:
  name: products
spec:
  # … existing sharding fields …
  seedBackupRef: products-backup
  replaceExisting: true
  force: true

See Property Sharding for details.

Symptom: Sharded database Failed with "holds differential backups of …"

Cause: the Neo4jShardedDatabase's seedBackupRef names a PVC backup whose latest run holds a differential of one or more shards. From a PVC, Neo4j seeds a sharded database only from a full backup of every shard, and a differential cannot be merged into one. On a PVC, backupType: AUTO takes full backups of a sharded database, so this comes from an all-databases backup, or a sharded backup's run taken by an earlier operator version; status.history[].shardArtifacts[].type shows which shards are DIFF.

Fix: seed from a sharded backup's full run: let the next run of a spec.shardedDatabase backup land, or create a one-shot Neo4jBackup with spec.shardedDatabase and point seedBackupRef at it. The sharded database retries and seeds once a full run lands. To keep differentials, back sharded databases up to cloud storage, where they seed directly.

Point-in-Time Recovery (PITR) Issues

A Neo4jRestore with source.pointInTime runs online on CalVer, from cloud storage, into a database that does not exist yet. Otherwise a standalone runs the neo4j-admin database restore --restore-until=… Job, and a cluster refuses. source.type: pitr always takes the Job, so it is standalone-only. See Point-in-Time Recovery.

Symptom: "point-in-time restore into cluster … cannot run: …"

Cause: the cluster cannot restore this point in time online, and a cluster has no offline path. The message names why:

• the database exists — restore under a new name, then compare and copy back what was lost;
• Neo4j 5.26 — upgrade to CalVer, or use a Neo4jDatabase with spec.seedConfig.restoreUntil on CalVer;
• a PVC backup — back up to cloud storage;
• no run started at or after the point in time — take a backup, or choose an earlier point.

Fix: address the cause and re-trigger the restore by changing its spec.

Symptom: PITR restore rejected for a cluster target

source.type=pitr is not supported for cluster targets …

Fix: on a cluster use source.type: backup with source.pointInTime (above), or a Neo4jDatabase with spec.seedConfig.restoreUntil; source.type: pitr is for standalones.

Symptom: PITR restore fails with timestamp errors

Solutions:

1. Missing PITR source config: source.type: pitr requires source.pitr.baseBackup or source.pointInTime (or both):

   spec:
     source:
       type: pitr
       pointInTime: "2025-01-15T14:30:00Z"   # ISO 8601 (metav1.Time)
       pitr:
         baseBackup:
           type: backup
           backupRef: production-backup
   

   The operator renders pointInTime into neo4j-admin database restore --restore-until="2025-01-15 14:30:00".

2. Timestamp Outside Backup Range:

   # Check backup time range
   kubectl logs job/production-backup-20250115 | grep -E "(start|end).*time"
   

3. Neo4j Version Compatibility:

   # PITR via Neo4jRestore (standalone target) is supported on Neo4j 5.26.x and 2025.x+
   spec:
     image:
       repo: "neo4j"
       tag: "2025.01.0-enterprise"
   

Performance Issues

Slow Backup Performance

Diagnosis:

# Monitor backup progress
kubectl logs job/production-backup-latest -f

# Check resource utilization during backup
kubectl top pod production-cluster-server-0

Optimization Strategies:

1. Reduce primary load: Use database-specific backups and schedule during low-traffic windows.

2. Avoid overlapping backups: Stagger Neo4jBackup schedules so only one job runs per cluster at a time.

3. Storage Performance Tuning: Back the Neo4jBackup Job's destination PVC with a high-performance storage class (e.g. fast-ssd) for backup staging.

4. Network Optimization:

   spec:
     config:
       # Increase buffer sizes for backup operations
       server.memory.off_heap.max_size: "2g"
       server.memory.pagecache.size: "4g"
   

Slow Restore Performance

Optimization:

1. Target Cluster Resources:

   spec:
     resources:
       requests:
         memory: "8Gi"
         cpu: "4"
       limits:
         memory: "16Gi"
         cpu: "8"
   

2. Storage Configuration:

   spec:
     storage:
       className: "fast-ssd"
       size: "1Ti"
   

Monitoring and Alerting

Backup Health Monitoring

Prometheus Metrics (exported by the operator):

# Backup attempts, labelled by result ("success"/"failure")
neo4j_operator_backup_total{cluster_name, namespace, result}
# Backup duration histogram
neo4j_operator_backup_duration_seconds{cluster_name, namespace}

# Alert rules
groups:
- name: neo4j-backup
  rules:
  - alert: BackupFailure
    expr: increase(neo4j_operator_backup_total{result="failure"}[24h]) > 0
    labels:
      severity: critical
    annotations:
      summary: "Neo4j backup failed"
      description: "Backup for cluster {{ $labels.cluster_name }} failed"

Log Monitoring:

# Monitor backup logs
kubectl logs -f job/production-backup-latest | grep -E "(ERROR|WARN|SUCCESS)"

# Set up log alerts
kubectl logs -f -n neo4j-operator-system -l control-plane=controller-manager | \
  grep -i "backup.*failed" --line-buffered | \
  while read line; do
    echo "BACKUP ALERT: $line"
    # Send to alerting system
  done

Backup Validation

Automated Validation Script:

#!/bin/bash
# Validate backup completeness

BACKUP_NAME="production-backup"
NAMESPACE="default"

validate_backup() {
  # One-shot backups reach phase "Completed". Scheduled backups stay in
  # "Scheduled" forever — check status.history for a Succeeded run instead.
  local backup_status=$(kubectl get neo4jbackup $BACKUP_NAME -n $NAMESPACE -o jsonpath='{.status.phase}')

  if [ "$backup_status" = "Scheduled" ]; then
    local last_run=$(kubectl get neo4jbackup $BACKUP_NAME -n $NAMESPACE \
      -o jsonpath='{.status.history[-1].status}')
    if [ "$last_run" != "Succeeded" ]; then
      echo "❌ Last scheduled run not successful: ${last_run:-no runs yet}"
      return 1
    fi
  elif [ "$backup_status" != "Completed" ]; then
    echo "❌ Backup failed or incomplete: $backup_status"
    return 1
  fi

  # Backup duration of the latest successful run (the operator does not
  # populate status.stats.size; check the .backup files in your storage backend
  # for the artifact size)
  local backup_duration=$(kubectl get neo4jbackup $BACKUP_NAME -n $NAMESPACE -o jsonpath='{.status.stats.duration}')
  echo "ℹ️  Latest backup duration: ${backup_duration:-unknown}"

  echo "✅ Backup validation passed"
  return 0
}

# Run validation
validate_backup

Emergency Recovery

For full disaster recovery (corrupted primary, restore to a new cluster from latest backup), follow the standard restore flow in the Backup & Restore guide § Restore Operations. The normal Neo4jRestore CR with source.type: backup + instanceRef pointing at a fresh cluster IS the emergency procedure — there's no separate path. Use spec.options.replaceExisting: true to overwrite existing data. To roll back to a specific timestamp before the corruption, use a Neo4jRestore with source.type: backup, source.pointInTime and a new spec.database name: on CalVer, from cloud storage, it runs online on a cluster or a standalone; otherwise a standalone restores offline and a cluster fails, naming the reason. See Point-in-Time Recovery.

See Also

• Backup & Restore Guide
• Performance Tuning
• Security Guide
• Split-Brain Recovery

Cluster restore fails with "the seed FAILED: …"

Symptom: a cluster-target Neo4jRestore goes Failed with a message like:

Database "mydb" was created but the seed FAILED: Object not found at the
path: s3://… — fix the cause, DROP DATABASE mydb IF EXISTS, and re-trigger
the restore

The quoted part is Neo4j's own statusMessage — it names the real cause.

Most common cause: Object not found at the path: s3://… on an S3-compatible endpoint (MinIO, on-prem) — the server JVMs are missing the endpoint configuration. The cluster Cypher restore makes the server pods fetch the seed, and the endpointURL field on the backup only reaches backup/restore Job pods. The operator emits a SeedEndpointNotProjected Warning event on the restore when it can verify the endpoint is missing. Add to the cluster CR:

spec:
  env:
  - name: AWS_ENDPOINT_URL_S3
    value: "http://minio.minio.svc:9000"
  - name: JAVA_TOOL_OPTIONS
    value: "-Daws.s3.forcePathStyle=true"

Recovery: fix the cause, DROP DATABASE <name> IF EXISTS (it holds no data), then re-trigger the restore (see the next section).

Retrying a finished restore

Completed/Failed are terminal for the current spec generation. Any spec change that bumps the generation issues a fresh attempt:

kubectl patch neo4jrestore <restore-name> --type=merge -p '{"spec":{"timeout":"10m"}}'

Restore stuck Pending on "Waiting for backup-seed-proxy"

Symptom: a PVC-backed cluster restore stays Pending with Waiting for backup-seed-proxy Deployment to become Ready (…), then goes Failed after the wait budget (3 minutes by default; spec.timeout overrides) with the proxy pod's condition in the message.

Cause: the operator serves the backup PVC to the server pods through a small HTTP proxy Deployment, and that proxy can't start. The parenthesised diagnosis names the reason; the most common is the backup PVC being ReadWriteOnce and still attached to another pod/node (e.g. a backup Job pod that hasn't terminated, or a node that still holds the volume).

Fix: release the PVC (wait for / delete the pod holding it), or move the backups to RWX storage or object storage. Then re-trigger the restore with a spec bump.