Troubleshooting Guide¶

This guide provides comprehensive troubleshooting information for the Neo4j Kubernetes Operator, covering both Neo4jEnterpriseCluster and Neo4jEnterpriseStandalone deployments.

Faster than working through this by hand

The kubectl-neo4j CLI automates most of what follows:

kubectl neo4j diagnose — start here. Names the Kubernetes-level cause: the pod that will not schedule, the container that was OOMKilled, the PVC that never bound
kubectl neo4j status — what exists and what is unhealthy, in one view
kubectl neo4j explain <Kind>/<name> — what a condition means and what to do about it
kubectl neo4j preflight — the cluster-side preconditions, before you apply
kubectl neo4j support-bundle — collect everything below into one redacted archive
kubectl neo4j cypher <name> — a shell, without the Secret round-trip

Use this guide when you want to look at things yourself, or when the CLI is not installed.

Quick Reference¶

Diagnostic Commands¶

With the CLI, the whole block below collapses to three commands:

kubectl neo4j status                 # every Neo4j resource, and which need attention
kubectl neo4j diagnose               # why the unhealthy ones are unhealthy
kubectl neo4j explain <Kind>/<name>  # what a condition means and what to do

diagnose reads the pod, PVC, StatefulSet, Job and event layers that the rest of this section reaches by hand, and it knows the operator's naming scheme and namespaces.

By hand¶

# Check deployment status
kubectl get neo4jenterprisecluster
kubectl get neo4jenterprisestandalone
kubectl get neo4jdatabase

# View detailed information
kubectl describe neo4jenterprisecluster <cluster-name>
kubectl describe neo4jenterprisestandalone <standalone-name>
kubectl describe neo4jdatabase <database-name>

# Check pod status
# Clusters
kubectl get pods -l neo4j.com/cluster=<cluster-name>
kubectl logs -l neo4j.com/cluster=<cluster-name>
# Standalone
kubectl get pods -l app=<standalone-name>
kubectl logs -l app=<standalone-name>

# Check events
kubectl get events --sort-by=.metadata.creationTimestamp

# Check operator logs
kubectl logs -n neo4j-operator-system -l control-plane=controller-manager --tail=-1

Common Port Forwarding Commands¶

# For clusters
kubectl port-forward svc/<cluster-name>-client 7474:7474 7687:7687

# For standalone deployments
kubectl port-forward svc/<standalone-name>-client 7474:7474 7687:7687

Common Issues and Solutions¶

1. Split-Brain Scenarios¶

Problem: Cluster nodes form multiple independent clusters¶

This is most common with TLS-enabled clusters where nodes fail to join during initial formation.

Quick Check:

# Check each node's view of the cluster
for i in 0 1 2; do
  kubectl exec <cluster>-server-$i -- cypher-shell -u neo4j -p <password> "SHOW SERVERS" | wc -l
done

Solution: See the Split-Brain Recovery Guide — the top of that page has a one-screen quick reference for detection and manual override commands.

Quick Fix:

# Restart minority cluster nodes (orphaned pods)
kubectl delete pod <cluster>-server-1 <cluster>-server-2

2. Deployment Validation Errors¶

Problem: Single-Node Cluster Not Allowed¶

servers must be at least 2 for clustering. For single-node deployments, use Neo4jEnterpriseStandalone instead

(The CRD schema also enforces spec.topology.servers >= 2, so kubectl apply itself may reject a smaller value with the apiserver's own "should be greater than or equal to 2" message.)

Solution: Use the correct CRD for your deployment type:

For development/testing (single-node):

apiVersion: neo4j.neo4j.com/v1beta1
kind: Neo4jEnterpriseStandalone
metadata:
  name: dev-neo4j
spec:
  acceptLicenseAgreement: "eval"
  image:
    repo: neo4j
    tag: "5.26-enterprise"
  storage:
    className: standard
    size: "10Gi"

For production (minimum cluster):

apiVersion: neo4j.neo4j.com/v1beta1
kind: Neo4jEnterpriseCluster
metadata:
  name: prod-cluster
spec:
  acceptLicenseAgreement: "eval"
  topology:
    servers: 2  # Minimum required for clustering
  image:
    repo: neo4j
    tag: "5.26-enterprise"
  storage:
    className: standard
    size: "10Gi"

Problem: Invalid Neo4j Version¶

Error: Neo4j version 5.25.0 is not supported. Minimum required version is 5.26.0

Solution: Update to a supported version:

spec:
  image:
    tag: "5.26-enterprise"  # or later

Supported versions:

Semver: any 5.26.x (5.26.x is the last semver LTS — no 5.27+ exists)
Calver: 2025.01.0 and later; each operator release validates one anchor CalVer, and newer ones are best-effort

See Supported Neo4j Versions for what each track guarantees.

3. Pod Startup Issues¶

Problem: Pods Stuck in Pending State¶

kubectl neo4j diagnose reports the scheduler's own reason and what to do about it, and kubectl neo4j preflight -f <manifest> catches the two usual causes — no node large enough, or a StorageClass that does not exist — before you apply. By hand:

# Check pod events
kubectl describe pod <pod-name>

# Common causes:
# - Insufficient resources
# - Storage issues
# - Image pull issues

Solutions:

Check Resource Availability:
```
kubectl describe nodes
kubectl get pv
```

Verify Storage Class:

kubectl get storageclass
kubectl describe storageclass <storage-class-name>

Check Image Pull:

kubectl describe pod <pod-name> | grep -A 5 "Events:"

Problem: Pods Crashing (CrashLoopBackOff)¶

kubectl neo4j diagnose distinguishes a crash loop from an OOMKill (exit 137) and names the pod and container to read. By hand:

# Check pod logs
kubectl logs <pod-name> --previous

Common causes and solutions:

Memory Issues:

spec:
  resources:
    requests:
      memory: "2Gi"
    limits:
      memory: "4Gi"

Configuration Issues:

# Check ConfigMap
kubectl get configmap <cluster-name>-config -o yaml

License Issues:

# Check license secret
kubectl get secret <license-secret> -o yaml

4. Connectivity Issues¶

Problem: Cannot Connect to Neo4j¶

# Test connectivity
kubectl port-forward svc/<service-name> 7474:7474 7687:7687
curl http://localhost:7474

# Check service
kubectl get svc -l app.kubernetes.io/name=neo4j
kubectl describe svc <service-name>

Solutions:

Check Service Configuration:

# For clusters
service: <cluster-name>-client

# For standalone (`<standalone-name>-service` also resolves
# but is deprecated — use `-client`)
service: <standalone-name>-client

Verify Network Policies:

kubectl get networkpolicies
kubectl describe networkpolicy <policy-name>

Check TLS Configuration:

# For TLS-enabled deployments
kubectl get certificates
kubectl describe certificate <cert-name>

5. Cluster-Specific Issues¶

Problem: Cluster Formation Fails¶

# Check cluster status
kubectl get neo4jenterprisecluster <cluster-name> -o yaml

# Check individual pod logs
kubectl logs <cluster-name>-server-0
kubectl logs <cluster-name>-server-1

Solutions:

🔧 Verify LIST Discovery Configuration

The operator uses LIST discovery with static pod FQDNs (port 6000). Check the startup script in the cluster ConfigMap:

kubectl get configmap <cluster-name>-config -o yaml | grep -A 3 "resolver_type"

# Neo4j 5.26.x should show:
# dbms.cluster.discovery.resolver_type=LIST
# dbms.cluster.discovery.version=V2_ONLY
# dbms.cluster.discovery.v2.endpoints=<cluster>-server-0.<cluster>-headless.<ns>.svc.cluster.local:6000,...

# Neo4j 2025.x+ should show:
# dbms.cluster.discovery.resolver_type=LIST
# dbms.cluster.endpoints=<cluster>-server-0.<cluster>-headless.<ns>.svc.cluster.local:6000,...

Verify Cluster Topology:

# Ensure minimum topology requirements
kubectl get neo4jenterprisecluster <cluster-name> -o jsonpath='{.spec.topology}'

Check Inter-Pod Communication:

# Test DNS resolution to headless service
kubectl exec -it <pod-name> -- nslookup <cluster-name>-headless

# Test V2 cluster port connectivity (port 6000 carries both V2 discovery
# and tcp-tx; port 5000 is V1-only and is never used by this operator).
kubectl exec -it <pod-name> -- nc -zv localhost 6000
kubectl exec -it <pod-name> -- nc -zv localhost 7000  # RAFT

Verify Discovery Labels:

# Check that only the discovery (headless) service carries the clustering label
kubectl get svc -l neo4j.com/cluster=<cluster-name> -o yaml | grep -A 3 -B 3 "neo4j.com/clustering"

Problem: A server is down (`Degraded`)¶

Symptoms: - The cluster is still Ready but its Degraded condition is True, naming the missing servers. - After the grace period (5 minutes; operator flag --server-unavailable-grace, Helm serverUnavailableGrace) the phase turns Degraded: status.ready is false and a ClusterDegraded Warning event is raised. - With a majority of servers lost, the phase is Forming with a ClusterQuorumLost Warning.

The cluster keeps serving while it has a majority: users, roles, databases and backups keep reconciling against it. A rolling image upgrade or a plugin install waits until every server is back.

Diagnosis:

kubectl get neo4jenterprisecluster <cluster-name> \
  -o jsonpath='{.status.phase}{"\n"}{range .status.conditions[?(@.type=="Degraded")]}{.message}{"\n"}{end}'
kubectl get pods -l neo4j.com/cluster=<cluster-name>
kubectl describe pod <cluster-name>-server-<n>      # why the missing server is not running
kubectl neo4j diagnose Neo4jEnterpriseCluster/<cluster-name>

Solution: bring the named server back (fix what keeps its pod from running). See Server availability.

Problem: Scaling Issues¶

# Check scaling validation
kubectl get events | grep -i scale

Solutions:

Verify Minimum Topology:

# Scaling cannot violate minimum requirements
spec:
  topology:
    servers: 2  # Minimum 2 servers; cannot scale below this

Check Resource Limits:

spec:
  resources:
    requests:
      cpu: "500m"
      memory: "2Gi"

6. Standalone-Specific Issues¶

Problem: Standalone Pod Won't Start¶

# Check standalone status
kubectl get neo4jenterprisestandalone <standalone-name> -o yaml

# Check pod events
kubectl describe pod <standalone-name>-0

Solutions:

Check Standalone Configuration:

# Uses unified clustering infrastructure (Neo4j 5.26+)
# No manual configuration needed for single-node operation

Verify Storage Configuration:

spec:
  storage:
    className: standard
    size: "10Gi"

Problem: Migration from Cluster to Standalone¶

# Create backup first
kubectl apply -f backup.yaml

# Deploy standalone
kubectl apply -f standalone.yaml

# Restore data
kubectl apply -f restore.yaml

7. Performance Issues¶

Problem: Slow Query Performance¶

# Check resource usage
kubectl top pods
kubectl top nodes

# Check Neo4j Prometheus metrics. Requires spec.monitoring.enabled: true,
# which creates a dedicated <cluster-name>-metrics service exposing port 2004.
# Metrics live on port 2004 at /metrics, not on the 7474 HTTP port.
kubectl port-forward svc/<cluster-name>-metrics 2004:2004
# Access http://localhost:2004/metrics

Solutions:

Adjust Memory Settings:

spec:
  config:
    server.memory.heap.initial_size: "2G"
    server.memory.heap.max_size: "4G"
    server.memory.pagecache.size: "2G"

Enable Query Logging:

spec:
  config:
    db.logs.query.enabled: "INFO"   # enum: OFF | INFO | VERBOSE (not a boolean)
    db.logs.query.threshold: "1s"

Check Storage Performance:

# Test storage I/O
kubectl exec -it <pod-name> -- dd if=/dev/zero of=/data/test bs=1M count=1000

8. Storage Issues¶

Problem: PVC Issues¶

kubectl neo4j diagnose reports any PVC that is not Bound and names its StorageClass; kubectl neo4j preflight checks that the class exists — and that it allows expansion — before you apply. By hand:

# Check PVC status
kubectl get pvc
kubectl describe pvc <pvc-name>

# Check storage class
kubectl get storageclass

Solutions:

Verify Storage Class:

spec:
  storage:
    className: fast-ssd  # Ensure this exists
    size: "50Gi"

Check Node Storage:

kubectl describe nodes
df -h  # On nodes

Problem: Data Corruption¶

# Check Neo4j consistency
kubectl exec -it <pod-name> -- neo4j-admin database check neo4j

Solutions:

Run Consistency Check:

kubectl exec -it <pod-name> -- neo4j-admin database check neo4j

Restore from Backup:

kubectl apply -f restore-from-backup.yaml

9. Backup and Restore Issues¶

Problem: Backup Job fails to start (ServiceAccount / permission errors)¶

Backups run as Kubernetes Jobs that execute neo4j-admin directly against a database — they do NOT exec into the Neo4j pods. Each Job runs under the neo4j-backup-sa ServiceAccount, which the operator creates automatically in the backup's namespace (and stamps with any workload-identity annotations from spec.storage.cloud.identity.autoCreate.annotations).

Solution: kubectl neo4j preflight Neo4jBackup/<name> -n <ns> checks this before the first run — that the ServiceAccount carries a cloud-identity annotation, or that the credentials Secret exists and holds every key the Job mounts. kubectl neo4j diagnose reports a Job that has already failed. By hand:

# The operator auto-creates this ServiceAccount; no Role/RoleBinding is needed
# because the Job runs neo4j-admin directly, not via pods/exec.
kubectl get serviceaccount neo4j-backup-sa -n <ns>

# Inspect the backup Job and its pod
kubectl describe neo4jbackup <backup-name> -n <ns>
kubectl get jobs -n <ns> -l app.kubernetes.io/part-of=<backup-name>

Problem: Backup path not found¶

Neo4j 5.26+ requires backup destination path to exist. The operator's backup Job creates the directory automatically (mkdir -p is prepended to the command for PVC targets; cloud targets get a trailing slash for directory semantics).

Solution: inspect the backup Job's Pod log:

# The Job is named "<neo4jbackup-name>-backup" (one-shot) or
# "<neo4jbackup-name>-backup-cron-<timestamp>" (CronJob child; the
# runID in status.history is this Job name).
kubectl logs -n <ns> job/<job-name>

10. Security Issues¶

Problem: Authentication Failures¶

# Check auth secret
kubectl get secret <auth-secret> -o yaml

# Check Neo4j auth logs
kubectl logs <pod-name> | grep -i auth

Solutions:

Verify Admin Secret:

apiVersion: v1
kind: Secret
metadata:
  name: neo4j-admin-secret
data:
  username: bmVvNGo=  # base64 encoded
  password: cGFzc3dvcmQ=  # base64 encoded

Check Password Policy: Set the Neo4j password-policy keys in spec.config:

spec:
  config:
    dbms.security.auth_minimum_password_length: "8"

Problem: TLS Certificate Issues¶

# Check certificate status
kubectl get certificates
kubectl describe certificate <cert-name>

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

Solutions:

Verify Issuer:

spec:
  tls:
    mode: cert-manager
    issuerRef:
      name: ca-cluster-issuer
      kind: ClusterIssuer

Check Certificate Details:

kubectl get secret <tls-secret> -o yaml

TLS Cluster Formation Issues:

TLS-enabled clusters are prone to split-brain during initial formation. If you see partial cluster formation:

# Check for split clusters
kubectl exec <cluster>-server-0 -- cypher-shell -u neo4j -p <password> "SHOW SERVERS"
kubectl exec <cluster>-server-1 -- cypher-shell -u neo4j -p <password> "SHOW SERVERS"

Prevention: The operator already configures discovery and RAFT membership timeouts with values tuned for in-cluster formation (including the longer windows TLS clusters need) — there are no spec.config knobs you need to set for this, and overriding the operator-managed cluster settings is not supported.

See Split-Brain Recovery Guide for detailed recovery procedures.

11. Database Creation Issues¶

Problem: Neo4jDatabase Creation Fails¶

# Check database status
kubectl get neo4jdatabase <database-name> -o yaml
kubectl describe neo4jdatabase <database-name>

# Check events specific to the database
kubectl get events --field-selector involvedObject.name=<database-name>

Common causes and solutions:

Cluster Not Ready:

# Error: Referenced cluster my-cluster not found
# Solution: Ensure cluster exists and is ready
spec:
  clusterRef: existing-cluster-name  # Must match actual cluster

Topology Exceeds Cluster Capacity:

# Error: database topology requires 5 servers but cluster only has 3 servers available
# Solution: Adjust topology to fit cluster capacity
spec:
  topology:
    primaries: 2     # Reduce from 3
    secondaries: 1   # Reduce from 2

Invalid Configuration Conflicts:

# Error: seedURI and initialData cannot be specified together
# Solution: Choose one data source method
spec:
  seedURI: "s3://my-backups/db.backup"
  # initialData: null  # Remove this section

Problem: Seed URI Database Creation Fails¶

# Check validation errors
kubectl describe neo4jdatabase <database-name>

# Check operator logs for seed URI specific errors
kubectl logs -n neo4j-operator-system -l control-plane=controller-manager --tail=-1 | grep -i seed

Common seed URI issues:

Authentication Failures:

# Check credentials secret exists
kubectl get secret <credentials-secret> -o yaml

# Verify required keys for your URI scheme
# S3: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
# GCS: GOOGLE_APPLICATION_CREDENTIALS
# Azure: AZURE_STORAGE_ACCOUNT + (AZURE_STORAGE_KEY or AZURE_STORAGE_SAS_TOKEN)

URI Access Issues:

# Test URI access from a pod
kubectl run test-pod --rm -it --image=amazon/aws-cli \
  -- aws s3 ls s3://my-bucket/backup.backup

# For GCS
kubectl run test-pod --rm -it --image=google/cloud-sdk:slim \
  -- gsutil ls gs://my-bucket/backup.backup

Invalid URI Format:

# Error: URI must specify a host
# Bad: s3:///path/backup.backup
# Good: s3://bucket-name/path/backup.backup

# Error: URI must specify a path to the backup file
# Bad: s3://bucket-name/
# Good: s3://bucket-name/backup.backup

Point-in-Time Recovery Issues:

# Warning: Point-in-time recovery (restoreUntil) is only available with Neo4j 2025.x
# Solution: Only use restoreUntil with Neo4j 2025.x clusters
seedConfig:
  restoreUntil: "2025-01-15T10:30:00Z"  # Neo4j 2025.x only

Performance Issues with Seed URI:

# Warning: Using dump file format. For better performance with large databases, consider using Neo4j backup format (.backup) instead.
# Solution: Use .backup format for large datasets
seedURI: "s3://my-backups/database.backup"  # Instead of .dump

# `seedConfig.config` is not a tuning surface for CloudSeedProvider (it only
# carries S3SeedProvider options such as `region`); for large seeds, give the
# server pods enough memory/CPU and a reliable path to the bucket.

Problem: Database Stuck in Creating State¶

# Check database status conditions
kubectl get neo4jdatabase <database-name> -o jsonpath='{.status.conditions[*].message}'

# Monitor database creation progress
kubectl get events -w --field-selector involvedObject.name=<database-name>

Solutions:

Check Cluster Connectivity:

# Ensure operator can connect to Neo4j cluster
kubectl logs -n neo4j-operator-system -l control-plane=controller-manager --tail=-1 | grep -i "connection failed"

Large Backup Restoration:

# Monitor restoration progress (seed URI databases)
kubectl logs <cluster-pod> | grep -i "restore\|seed"

# For large backups, restoration may take significant time
# Ensure adequate pod resources

Network Connectivity Issues:

# For seed URI, test network access from Neo4j pods
kubectl exec -it <cluster-pod> -- curl -I <your-backup-url>

Problem: Database Ready But No Data¶

# Connect to database and check
kubectl exec -it <cluster-pod> -- cypher-shell -u neo4j -p <password> -d <database-name> "MATCH (n) RETURN count(n)"

Solutions:

Initial Data Not Applied:

# Check if initial data import completed
kubectl get neo4jdatabase <database-name> -o jsonpath='{.status.dataImported}'

# Check for import errors in operator logs
kubectl logs -n neo4j-operator-system -l control-plane=controller-manager --tail=-1 | grep -i "initial data\|import"

Seed URI Data Not Restored:

# Check if seed restoration completed
kubectl get events --field-selector involvedObject.name=<database-name> | grep -i "DataSeeded"

# Verify seed URI is accessible and contains data

Advanced Troubleshooting¶

Debug Mode¶

Enable debug logging in the operator by appending a flag to the manager's existing arguments (a plain strategic-merge patch of args would replace them all). The Deployment is named neo4j-operator for a Helm release called neo4j-operator, and neo4j-operator-controller-manager for kustomize installs:

kubectl get deployment -n neo4j-operator-system   # find the operator Deployment name
kubectl patch deployment <operator-deployment> \
  -n neo4j-operator-system --type=json \
  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--zap-log-level=debug"}]'

For Helm installs, helm upgrade neo4j-operator ... --set logLevel=debug is the durable way (the chart renders --zap-log-level from logLevel).

Resource Monitoring¶

Monitor resource usage:

# Watch resource usage
watch kubectl top pods
watch kubectl top nodes

# Check resource limits
kubectl describe limitrange
kubectl describe resourcequota

Network Debugging¶

Test network connectivity:

# DNS resolution
kubectl exec -it <pod-name> -- nslookup <service-name>

# Port connectivity
kubectl exec -it <pod-name> -- telnet <service-name> 7687

# Network policies
kubectl get networkpolicies --all-namespaces

Collecting Diagnostic Information¶

When filing an issue, include the output of:

kubectl get neo4jenterprisecluster,neo4jenterprisestandalone -A
kubectl get pods,svc,pvc -l app.kubernetes.io/name=neo4j
kubectl get events --sort-by=.metadata.creationTimestamp | tail -30
kubectl logs -n neo4j-operator-system -l control-plane=controller-manager --tail=200
kubectl describe nodes | grep -A 5 "Allocated resources:"

Getting Help¶

Support Resources¶

Documentation: User Guide
API Reference: Neo4jEnterpriseCluster, Neo4jEnterpriseStandalone
Upgrade Guide: Upgrade Guide
Community: Neo4j Community Forum
Issues: GitHub Issues

When to Contact Support¶

Contact support when:

Data corruption is suspected
Cluster formation consistently fails
Performance is significantly degraded
Security incidents occur
Migration issues cannot be resolved

Always provide the diagnostic report and specific error messages when contacting support.

Troubleshooting Guide¶

Quick Reference¶

Diagnostic Commands¶

By hand¶

Common Port Forwarding Commands¶

Common Issues and Solutions¶

1. Split-Brain Scenarios¶

Problem: Cluster nodes form multiple independent clusters¶

2. Deployment Validation Errors¶

Problem: Single-Node Cluster Not Allowed¶

Problem: Invalid Neo4j Version¶

3. Pod Startup Issues¶

Problem: Pods Stuck in Pending State¶

Problem: Pods Crashing (CrashLoopBackOff)¶

4. Connectivity Issues¶

Problem: Cannot Connect to Neo4j¶

5. Cluster-Specific Issues¶

Problem: Cluster Formation Fails¶

Problem: A server is down (Degraded)¶

Problem: Scaling Issues¶

6. Standalone-Specific Issues¶

Problem: Standalone Pod Won't Start¶

Problem: Migration from Cluster to Standalone¶

7. Performance Issues¶

Problem: Slow Query Performance¶

8. Storage Issues¶

Problem: PVC Issues¶

Problem: Data Corruption¶

9. Backup and Restore Issues¶

Problem: Backup Job fails to start (ServiceAccount / permission errors)¶

Problem: Backup path not found¶

10. Security Issues¶

Problem: Authentication Failures¶

Problem: TLS Certificate Issues¶

11. Database Creation Issues¶

Problem: Neo4jDatabase Creation Fails¶

Problem: Seed URI Database Creation Fails¶

Problem: Database Stuck in Creating State¶

Problem: Database Ready But No Data¶

Advanced Troubleshooting¶

Debug Mode¶

Resource Monitoring¶

Network Debugging¶

Collecting Diagnostic Information¶

Getting Help¶

Support Resources¶

When to Contact Support¶

Problem: A server is down (`Degraded`)¶