Split-Brain Recovery Guide¶

This guide provides comprehensive troubleshooting and recovery procedures for Neo4j cluster split-brain scenarios when using the Neo4j Kubernetes Operator with server-based architecture.

Collect the evidence first

kubectl neo4j support-bundle -n <namespace> captures every server's logs, status and the namespace events in one redacted archive — worth doing before you start restarting pods, since recovery destroys the evidence of what went wrong. See the CLI guide.

Then kubectl neo4j diagnose Neo4jEnterpriseCluster/<name> shows the operator's own SplitBrainDetected warning alongside each server pod's state, so you can tell a genuine split-brain from servers that simply never started.

Quick reference¶

Detect — compare each server's view of the cluster. Different lists across pods mean split-brain:

for i in 0 1 2; do
  echo "=== server-$i ==="
  kubectl exec <cluster>-server-$i -- cypher-shell -u neo4j -p <pw> \
    "SHOW SERVERS YIELD name, state ORDER BY name"
done

Watch the operator do its job — auto-recovery is the default; manual intervention is rare:

kubectl get events --field-selector reason=SplitBrainDetected -A
kubectl get events --field-selector reason=SplitBrainRepaired -A
kubectl logs -n neo4j-operator-system -l control-plane=controller-manager -f \
  | grep -i splitbrain

Manual override if auto-recovery fails — restart the orphaned pod and verify:

kubectl delete pod <cluster>-server-N -n <namespace>
kubectl exec <cluster>-server-0 -- cypher-shell -u neo4j -p <pw> "SHOW SERVERS"

If the cluster doesn't reconverge after that, follow the full recovery procedure below.

Overview¶

Split-brain occurs when Neo4j cluster servers lose communication and form separate, independent clusters instead of one unified cluster. This can lead to data inconsistencies and cluster instability if not properly detected and resolved.

The Neo4j Kubernetes Operator includes automatic split-brain detection and repair to prevent and resolve these issues proactively.

Understanding Split-Brain Scenarios¶

What is Split-Brain?¶

Split-brain happens when:

Network partitions separate cluster servers
Servers cannot communicate with each other
Multiple independent "clusters" form within the same deployment
Each partition believes it is the authoritative cluster

Common Causes¶

Network partitions between Kubernetes nodes
Resource constraints causing pod communication failures
DNS resolution issues preventing server discovery
Storage problems affecting cluster state persistence
Configuration errors in discovery or networking

Automatic Split-Brain Detection¶

The operator includes comprehensive split-brain detection that runs automatically during cluster health checks.

Detection Process¶

Multi-Pod Analysis: Connects to each server pod individually over Bolt and runs SHOW SERVERS
Cluster View Comparison: Compares each server's view of cluster membership
Inconsistency Detection: Identifies pods that see servers outside the majority (largest) view
Automatic Repair: Restarts those orphaned/minority pods to rejoin the main cluster

Detection is skipped for single-server clusters (a single node cannot split-brain) and while the cluster has not yet reached status.phase=Ready (divergent views during initial formation are expected, not split-brain).

On a Ready cluster the full check runs at most every --split-brain-check-interval (default 5m; Helm splitBrainCheckInterval) while nothing changes — but at once when a server pod is deleted or recreated, restarts a container or changes readiness, or when SHOW SERVERS is short of a server. So a server that comes back and forms its own cluster is caught on the pass its pod changes, not five minutes later. 0s checks on every reconcile. See Many large clusters.

Detection Logs¶

Monitor operator logs for split-brain detection:

# Check for split-brain detection logs
kubectl logs -n neo4j-operator-system -l control-plane=controller-manager --tail=-1 | grep -i "split.*brain"

# Expected detection logs:
# Starting split-brain detection   cluster=production-cluster expectedServers=3
# Split-brain analysis complete    isSplitBrain=true repairAction=restart_pods orphanedPods=1
# Restarting orphaned pods to repair split-brain   pods=[production-cluster-server-2]

Kubernetes Events¶

The operator generates events for split-brain scenarios:

# Check for split-brain events
kubectl get events --field-selector reason=SplitBrainDetected
kubectl get events --field-selector reason=SplitBrainRepaired
kubectl get events --field-selector reason=SplitBrainRepairFailed   # manual intervention needed

# Example events:
# Warning   SplitBrainDetected   Neo4jEnterpriseCluster/production-cluster   Split-brain detected: 2 cluster groups found, 1 orphaned pods
# Normal    SplitBrainRepaired   Neo4jEnterpriseCluster/production-cluster   Split-brain automatically repaired by restarting orphaned pods

Manual Split-Brain Detection¶

Verify Cluster Health¶

Check Server Status:

# Connect to each server and check cluster membership
for i in 0 1 2; do
  echo "=== Server $i ==="
  kubectl exec production-cluster-server-$i -- cypher-shell -u neo4j -p password \
    "SHOW SERVERS YIELD name, state, health ORDER BY name"
  echo
done

Compare Cluster Views: Look for inconsistencies in server lists between different pods. In a healthy cluster, all servers should see the same cluster membership.

Check Database Allocation:

# Verify database distribution consistency
kubectl exec production-cluster-server-0 -- cypher-shell -u neo4j -p password \
  "SHOW DATABASES YIELD name, currentStatus, role, address"

Identify Split-Brain Symptoms¶

Indicators of Split-Brain:

Different server counts reported by different pods
Inconsistent database allocations across servers
Some servers showing as "offline" from others' perspectives
Database creation failures with "insufficient servers" errors
Application connection failures to some databases

Repair Strategies¶

Automatic Repair (Recommended)¶

The operator automatically repairs split-brain scenarios by:

Detection: Identifying orphaned servers with inconsistent cluster views
Analysis: Determining the main cluster and orphaned servers
Restart: Gracefully restarting orphaned pods to rejoin the main cluster
Verification: Confirming successful cluster reformation

No manual intervention required - the operator handles this automatically.

Manual Repair Procedures¶

If automatic repair fails or you need to intervene manually:

1. Identify the Main Cluster¶

# Check which partition has the majority of servers
kubectl exec production-cluster-server-0 -- cypher-shell -u neo4j -p password \
  "SHOW SERVERS YIELD name, state ORDER BY name"

# Count active servers in each partition
kubectl exec production-cluster-server-1 -- cypher-shell -u neo4j -p password \
  "SHOW SERVERS YIELD name, state ORDER BY name"

2. Restart Orphaned Servers¶

# Restart the server(s) that show inconsistent cluster views
kubectl delete pod production-cluster-server-2

# Wait for pod to restart and rejoin
kubectl wait --for=condition=Ready pod/production-cluster-server-2 --timeout=300s

3. Verify Cluster Recovery¶

# Confirm all servers show consistent cluster membership
kubectl exec production-cluster-server-0 -- cypher-shell -u neo4j -p password \
  "SHOW SERVERS YIELD name, state, health ORDER BY name"

# Check database status
kubectl exec production-cluster-server-0 -- cypher-shell -u neo4j -p password \
  "SHOW DATABASES"

4. Force Cluster Reformation (Last Resort)¶

If standard restart doesn't work, use cluster-wide restart:

# Delete all server pods simultaneously (data preserved in PVCs)
kubectl delete pods -l app.kubernetes.io/name=neo4j,neo4j.com/cluster=production-cluster

# Monitor cluster reformation
kubectl get pods -l app.kubernetes.io/name=neo4j -w

⚠️ Warning: Cluster-wide restart should only be used as a last resort and may cause temporary service interruption.

Prevention Strategies¶

Network Resilience¶

Node Affinity Configuration:

spec:
  topology:
    servers: 3
    placement:
      antiAffinity:
        enabled: true
        type: preferred    # Allow scheduling on same node if necessary
        topologyKey: kubernetes.io/hostname

Multi-Zone Deployment:

spec:
  topology:
    servers: 3
    placement:
      topologySpread:
        enabled: true
        topologyKey: topology.kubernetes.io/zone
        maxSkew: 1

Resource Allocation¶

spec:
  resources:
    requests:
      memory: "4Gi"    # Adequate memory to prevent OOM
      cpu: "2"
    limits:
      memory: "8Gi"
      cpu: "4"

Network Configuration¶

spec:
  config:
    # Cluster communication resilience (Neo4j 5.26+)
    dbms.cluster.raft.leader_failure_detection_window: "30s"

Monitoring and Alerting¶

Prometheus Metrics¶

Monitor these operator metrics for early split-brain detection:

# Incremented each time the operator's split-brain detection fires
neo4j_operator_split_brain_detected_total{cluster_name, namespace}
# 1 = healthy, 0 = unhealthy
neo4j_operator_cluster_healthy{cluster_name, namespace}
# Server counts: role="desired" (spec.topology.servers) and role="ready" (StatefulSet readyReplicas)
neo4j_operator_cluster_replicas_total{cluster_name, namespace, role}
# Per server: 1 = Enabled + Available, 0 = degraded (collected once the cluster has formed, also while a server is down)
neo4j_operator_server_health{cluster_name, namespace, server_name, server_address, k8s_cluster}

# Alert rules
groups:
- name: neo4j.split-brain
  rules:
  - alert: Neo4jSplitBrainDetected
    expr: increase(neo4j_operator_split_brain_detected_total[15m]) > 0
    labels:
      severity: critical
    annotations:
      summary: "Neo4j split-brain detected"
      description: "The operator detected a split-brain in cluster {{ $labels.cluster_name }} ({{ $labels.namespace }}); check SplitBrainDetected / SplitBrainRepaired events"
  - alert: Neo4jServersNotReady
    expr: |
      neo4j_operator_cluster_replicas_total{role="ready"}
        < neo4j_operator_cluster_replicas_total{role="desired"}
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Neo4j servers not ready"
      description: "Cluster {{ $labels.cluster_name }} has fewer ready servers than desired"

Log Monitoring¶

Set up log monitoring for split-brain events:

# Alert on split-brain detection logs
kubectl logs -f -n neo4j-operator-system -l control-plane=controller-manager | \
  grep -E "(split.*brain|Split.*Brain)" --line-buffered | \
  while read line; do
    echo "ALERT: $line"
    # Send to monitoring system
  done

Health Check Automation¶

#!/bin/bash
# Automated cluster health check script

CLUSTER_NAME="production-cluster"
NAMESPACE="default"

check_cluster_health() {
  local expected_servers=3
  local consistent_views=0

  for i in $(seq 0 $((expected_servers-1))); do
    local server_count=$(kubectl exec ${CLUSTER_NAME}-server-$i -n $NAMESPACE -- \
      cypher-shell -u neo4j -p password \
      "SHOW SERVERS YIELD name" 2>/dev/null | tail -n +2 | wc -l)   # tail skips the header row

    if [ "$server_count" -eq "$expected_servers" ]; then
      ((consistent_views++))
    fi
  done

  if [ "$consistent_views" -eq "$expected_servers" ]; then
    echo "✅ Cluster health: OK"
    return 0
  else
    echo "❌ Split-brain detected: $consistent_views/$expected_servers servers have consistent views"
    return 1
  fi
}

# Run health check
if ! check_cluster_health; then
  echo "🔄 Triggering operator reconciliation..."
  kubectl annotate neo4jenterprisecluster $CLUSTER_NAME -n $NAMESPACE \
    "troubleshooting.neo4j.com/reconcile=$(date +%s)" --overwrite
fi

Troubleshooting Common Issues¶

Split-Brain Detection Not Working¶

Check Operator Logs:

kubectl logs -n neo4j-operator-system -l control-plane=controller-manager --tail=100

Verify RBAC Permissions:

# The operator's ServiceAccount is neo4j-operator for a Helm release named neo4j-operator,
# neo4j-operator-controller-manager for a kubectl-apply install
kubectl auth can-i get pods --as=system:serviceaccount:neo4j-operator-system:neo4j-operator
kubectl auth can-i list pods --as=system:serviceaccount:neo4j-operator-system:neo4j-operator

The detector reaches each server over Bolt, so the operator does not need (and does not have) pods/exec.

Check Neo4j Connectivity:

# Test if operator can connect to Neo4j
kubectl exec production-cluster-server-0 -- cypher-shell -u neo4j -p password "RETURN 'test'"

False Split-Brain Detection¶

If the operator incorrectly identifies split-brain:

Check Resource Constraints:

kubectl describe pods -l app.kubernetes.io/name=neo4j
kubectl top pods -l app.kubernetes.io/name=neo4j

Verify Network Connectivity:

# Test inter-pod communication on the V2 discovery/transaction port (6000).
# Port 5000 is the V1 discovery port — never used by this operator.
kubectl exec production-cluster-server-0 -- nc -zv production-cluster-server-1 6000

Review Configuration:

kubectl get neo4jenterprisecluster production-cluster -o yaml | grep -A 20 "spec:"

Recovery Failures¶

If automatic recovery fails:

Check Pod Status:

kubectl get pods -l app.kubernetes.io/name=neo4j
kubectl describe pod production-cluster-server-0

Review Events:

kubectl get events --sort-by=.metadata.creationTimestamp | tail -20

Inspect Storage:

kubectl get pvc -l neo4j.com/cluster=production-cluster
kubectl describe pvc data-production-cluster-server-0

Emergency Recovery Procedures¶

Complete Cluster Reset¶

⚠️ Use only as a last resort - may cause data loss

# spec.topology.servers has a minimum of 2, so a cluster cannot be scaled to 0 —
# the API server rejects it. To rebuild from scratch, delete the cluster resource
# and recreate it, restoring data from a backup (see "Data Recovery from Backups").

# 1. Delete the cluster (PVCs are retained when the CR is deleted)
kubectl delete neo4jenterprisecluster production-cluster

# 2. Wait for pods to terminate
kubectl wait --for=delete pod -l neo4j.com/cluster=production-cluster --timeout=300s

# 3. Clean up cluster state (if necessary)
# Note: This may cause data loss - only do if cluster is completely corrupted
# kubectl delete pvc -l neo4j.com/cluster=production-cluster,neo4j.com/role=server

# 4. Re-apply the cluster manifest
kubectl apply -f production-cluster.yaml

# 5. Monitor recovery
kubectl get pods -l neo4j.com/cluster=production-cluster -w

Data Recovery from Backups¶

If split-brain causes data corruption:

# 1. Create restoration cluster
kubectl apply -f - <<EOF
apiVersion: neo4j.neo4j.com/v1beta1
kind: Neo4jEnterpriseCluster
metadata:
  name: recovery-cluster
spec:
  acceptLicenseAgreement: "eval"
  topology:
    servers: 3
  # ... same configuration as original cluster
EOF

# 2. Restore from backup
kubectl apply -f - <<EOF
apiVersion: neo4j.neo4j.com/v1beta1
kind: Neo4jRestore
metadata:
  name: split-brain-recovery
spec:
  instanceRef: recovery-cluster
  database: neo4j
  source:
    type: backup
    backupRef: latest-backup
  options:
    replaceExisting: true
EOF

# 3. Verify data integrity
kubectl exec recovery-cluster-server-0 -- cypher-shell -u neo4j -p password \
  "MATCH (n) RETURN count(n) as node_count"

Best Practices Summary¶

Prevention:
Use adequate resource allocation
Deploy across multiple zones
Configure proper network policies
Monitor cluster health continuously
Detection:
Rely on automatic split-brain detection
Set up monitoring and alerting
Regular health checks
Recovery:
Trust automatic repair mechanisms
Manual intervention only when necessary
Always verify cluster health after recovery
Monitoring:
Monitor operator logs for split-brain events
Set up Kubernetes event alerting
Track cluster consistency metrics

For additional troubleshooting help, see: