PHP小码农

Appearance

网络分区处理 ​

概述 ​

网络分区（Network Partition）是 RabbitMQ 集群中由于网络故障导致节点间通信中断，集群分裂为多个独立部分的现象。网络分区处理不当可能导致数据不一致、队列消息丢失等严重问题。

问题表现与症状 ​

常见症状 ​

┌─────────────────────────────────────────────────────────────┐
│                 网络分区典型症状                             │
├─────────────────────────────────────────────────────────────┤
│  1. 集群状态显示多个分区                                    │
│  2. 镜像队列同步中断                                        │
│  3. 客户端连接不稳定                                        │
│  4. 队列消息不一致                                          │
│  5. 节点间无法通信                                          │
│  6. 出现"split brain"现象                                   │
└─────────────────────────────────────────────────────────────┘

网络分区示意 ​

                    正常状态                    网络分区状态
    
         ┌─────────┐                        ┌─────────┐
         │ Node 1  │                        │ Node 1  │
         └────┬────┘                        └────┬────┘
              │                                  │
         ┌────┴────┐                        ┌────┴────┐
         │ Node 2  │                        │ Node 2  │
         └────┬────┘                        └─────────┘
              │                                  │
         ┌────┴────┐                        ┌─────────┐
         │ Node 3  │                        │ Node 3  │
         └─────────┘                        └────┬────┘
                                                  │
                                             ┌────┴────┐
                                             │ Node 4  │
                                             └─────────┘

问题原因分析 ​

1. 网络故障 ​

原因说明影响
网络设备故障交换机/路由器故障可能导致多个节点分区
网络链路中断光纤/网线断开节点间通信中断
网络抖动短暂网络不稳定可能自动恢复
防火墙规则端口被意外阻断特定端口通信失败

2. 资源问题 ​

原因说明影响
CPU过载导致响应超时可能误判为网络问题
内存不足GC时间过长节点响应缓慢
磁盘IO高磁盘响应慢节点无响应

3. 配置问题 ​

原因说明影响
心跳配置不当检测间隔过长或过短检测不及时或误判
超时配置不当等待时间不合理处理不及时

诊断步骤 ​

步骤1：确认网络分区 ​

bash
# 查看集群状态，检查分区
rabbitmqctl cluster_status

# 查看详细分区信息
rabbitmqctl eval 'rabbit_nodes:partitions().'

# 检查节点间连接
rabbitmqctl eval 'rabbit_nodes:running_names().'

步骤2：分析分区详情 ​

bash
# 查看每个分区的节点
rabbitmqctl eval '
  Partitions = rabbit_nodes:partitions(),
  [io:format("Partition ~p: ~p~n", [P, Nodes]) || {P, Nodes} <- Partitions].'

# 查看节点间通信状态
rabbitmq-diagnostics check_inter_node_communication

# 检查网络连通性
for node in node1 node2 node3; do
    echo "Testing $node:"
    ping -c 3 rabbit@$node
done

步骤3：评估影响范围 ​

bash
# 查看受影响的队列
rabbitmqctl list_queues name node policy synchronised_slave_nodes

# 查看镜像队列同步状态
rabbitmqctl eval 'rabbit_mirror_queue_misc:get_synced().'

# 查看消息状态
rabbitmqctl list_queues name messages messages_unacked

解决方案 ​

1. 自动恢复模式配置 ​

bash
# rabbitmq.conf

# 网络分区处理模式
# 可选值: ignore, pause_minority, pause_all, autoheal

# 暂停少数派节点（推荐）
cluster_partition_handling = pause_minority

# 或自动恢复
# cluster_partition_handling = autoheal

2. 手动处理网络分区 ​

php
<?php

class NetworkPartitionHandler
{
    private $clusterNodes = [];

    public function __construct(array $nodes)
    {
        $this->clusterNodes = $nodes;
    }

    public function diagnosePartition(): array
    {
        $result = [
            'timestamp' => date('Y-m-d H:i:s'),
            'partitions' => [],
            'recommendation' => '',
        ];

        exec('rabbitmqctl cluster_status', $output);
        
        $partitions = $this->parseClusterStatus($output);
        $result['partitions'] = $partitions;
        
        if (count($partitions) > 1) {
            $result['recommendation'] = $this->determineRecoveryStrategy($partitions);
        } else {
            $result['recommendation'] = '未检测到网络分区';
        }
        
        return $result;
    }

    private function parseClusterStatus(array $output): array
    {
        $partitions = [];
        $currentPartition = [];
        
        foreach ($output as $line) {
            if (preg_match('/Partitions/', $line)) {
                continue;
            }
            
            if (preg_match('/rabbit@[\w]+/', $line)) {
                preg_match_all('/rabbit@[\w]+/', $line, $matches);
                $currentPartition = $matches[0];
            }
            
            if (!empty($currentPartition)) {
                $partitions[] = $currentPartition;
                $currentPartition = [];
            }
        }
        
        return array_filter($partitions);
    }

    private function determineRecoveryStrategy(array $partitions): string
    {
        $partitionCount = count($partitions);
        
        if ($partitionCount === 2) {
            return $this->handleTwoPartitions($partitions);
        } elseif ($partitionCount > 2) {
            return $this->handleMultiplePartitions($partitions);
        }
        
        return '无法确定恢复策略';
    }

    private function handleTwoPartitions(array $partitions): string
    {
        $size1 = count($partitions[0]);
        $size2 = count($partitions[1]);
        
        if ($size1 > $size2) {
            $majority = $partitions[0][0];
            $minority = $partitions[1][0];
        } elseif ($size2 > $size1) {
            $majority = $partitions[1][0];
            $minority = $partitions[0][0];
        } else {
            return $this->forceAutoheal();
        }
        
        return "建议在 {$minority} 节点上执行: rabbitmqctl forget_cluster_node {$majority}";
    }

    private function handleMultiplePartitions(array $partitions): string
    {
        return $this->forceAutoheal();
    }

    private function forceAutoheal(): string
    {
        return "建议执行自动恢复: rabbitmqctl autoheal";
    }

    public function forceRecovery(): bool
    {
        echo "开始强制恢复...\n";
        
        echo "[1] 停止所有节点\n";
        foreach ($this->clusterNodes as $node) {
            exec("ssh {$node} 'systemctl stop rabbitmq-server'");
        }
        
        sleep(10);
        
        echo "[2] 清理Mnesia数据\n";
        foreach ($this->clusterNodes as $node) {
            exec("ssh {$node} 'rabbitmqctl stop_app && rabbitmqctl reset && rabbitmqctl start_app'");
        }
        
        sleep(30);
        
        echo "[3] 验证集群状态\n";
        exec('rabbitmqctl cluster_status', $output);
        
        return implode("\n", $output);
    }
}

3. 手动分区恢复脚本 ​

bash
#!/bin/bash
# partition_recovery.sh

echo "=== 网络分区恢复流程 ==="

echo "[1] 当前集群状态"
rabbitmqctl cluster_status

echo "[2] 检测分区"
PARTITIONS=$(rabbitmqctl eval 'rabbit_nodes:partitions().' 2>&1)
echo "$PARTITIONS"

if [ -z "$PARTITIONS" ] || [ "$PARTITIONS" == "[]" ]; then
    echo "未检测到网络分区"
    exit 0
fi

echo "[3] 选择恢复策略"
echo "1) pause_minority - 暂停少数派节点"
echo "2) autoheal - 自动恢复"
echo "3) 手动忽略"

read -p "请选择 (1/2/3): " choice

case $choice in
    1)
        echo "执行 pause_minority..."
        rabbitmqctl eval 'rabbit_nodes:handle_partition_pause_minority().'
        ;;
    2)
        echo "执行 autoheal..."
        rabbitmqctl autoheal
        ;;
    3)
        echo "手动处理..."
        ;;
    *)
        echo "无效选择"
        exit 1
        ;;
esac

echo "[4] 等待恢复"
sleep 30

echo "[5] 验证集群状态"
rabbitmqctl cluster_status

echo "[6] 检查队列状态"
rabbitmqctl list_queues name messages consumers

echo "恢复流程完成"

4. 强制节点加入 ​

bash
#!/bin/bash
# force_join.sh

TARGET_NODE=$1
JOIN_NODE=$2

if [ -z "$TARGET_NODE" ] || [ -z "$JOIN_NODE" ]; then
    echo "用法: $0 <target_node> <join_node>"
    exit 1
fi

echo "=== 强制节点加入 ==="

echo "[1] 在目标节点停止应用"
rabbitmqctl -n $TARGET_NODE stop_app

echo "[2] 重置节点"
rabbitmqctl -n $TARGET_NODE reset

echo "[3] 加入集群"
rabbitmqctl -n $TARGET_NODE join_cluster $JOIN_NODE

echo "[4] 启动应用"
rabbitmqctl -n $TARGET_NODE start_app

echo "[5] 验证状态"
rabbitmqctl cluster_status

echo "节点加入完成"

预防措施 ​

1. 网络配置优化 ​

bash
# rabbitmq.conf

# 心跳配置
heartbeat = 60

# 节点间通信超时
net_ticktime = 60

# 分区处理策略
cluster_partition_handling = pause_minority

2. 网络监控告警 ​

yaml
# Prometheus 告警规则
groups:
  - name: rabbitmq_partition
    rules:
      - alert: NetworkPartitionDetected
        expr: |
          rabbitmq_partitions > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "检测到网络分区"
          description: "RabbitMQ 集群检测到网络分区，需要立即处理"

      - alert: PartitionNotResolved
        expr: |
          rabbitmq_partitions > 0
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "网络分区未恢复"
          description: "网络分区已持续10分钟未恢复"

3. 网络健康检查脚本 ​

bash
#!/bin/bash
# network_health_check.sh

NODES=("rabbit@node1" "rabbit@node2" "rabbit@node3")
LOG_FILE="/var/log/rabbitmq/network_health.log"

log_message() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> $LOG_FILE
}

check_node_connectivity() {
    for node in "${NODES[@]}"; do
        result=$(rabbitmqctl -n $node ping 2>&1)
        if [ $? -eq 0 ]; then
            log_message "节点 $node: OK"
        else
            log_message "节点 $node: FAILED - $result"
        fi
    done
}

check_cluster_status() {
    status=$(rabbitmqctl cluster_status 2>&1)
    
    if echo "$status" | grep -q "Partitions"; then
        log_message "WARNING: 检测到网络分区"
        echo "$status" >> $LOG_FILE
        return 1
    fi
    
    log_message "集群状态正常"
    return 0
}

while true; do
    check_node_connectivity
    check_cluster_status
    sleep 30
done

注意事项 ​

  1. 分区处理策略要谨慎：根据业务需求选择合适模式
  2. 数据可能丢失：分区期间消息可能丢失
  3. 及时处理：分区时间越长，影响越大
  4. 提前测试：在测试环境验证恢复流程
  5. 监控要完善：及时发现分区问题

相关链接 ​