Redis 集群模式下的数据一致性问题实战分析
Practical Analysis of Data Consistency Issues in Redis Cluster Mode
| Kevin | 2026-08-23T10:54:01
最近线上遇到一个 Redis 集群脑裂导致数据丢失的问题，排查了两天才搞清楚原因。这篇文章记录下整个排查过程和最终的解决方案。
A practical walkthrough of debugging a Redis cluster split-brain issue that caused data loss in production, including the full investigation process and final solution.
前两天线上报了一个诡异的 bug：用户明明已经下过单了，但是再刷新页面就看不到订单了。一开始以为是前端缓存的问题，结果排查下来发现是 Redis 集群脑裂导致的。 问题现象 我们的订单状态是先写 Redis 再异步写 MySQL 的，正常情况下没啥问题。但是那天运维同学在做网络调整，导致 Redis 集群出现了短暂的网络分区。 具体表现就是：3 主 3 从的集群，其中一个主节点跟其他节点断开连接了大概 30 秒。这 30 秒内，这个主节点还在正常接受写入，但是等网络恢复之后，它发现自己已经被降级成从节点了，之前那 30 秒写入的数据就全丢了。 根因分析 Redis Cluster 用的是 Raft 类似的选举机制。当一个主节点失联超过 cluster-node-timeout（默认 15 秒），从节点就会发起选举。问题在于： 失联的主节点并不知道自己已经被替换了，它还在欢快地接受写请求 等网络恢复后，它收到了新主节点的 PING，才发现自己已经不是主了 这时候它会把自己降级为从节点，然后从新主同步数据，之前的写入全部丢弃 解决方案 查了一圈资料，最终用了这几个配置来缓解： # 设置最少从节点数，没有足够从节点时主节点拒绝写入 min-replicas-to-write 1 min-replicas-max-lag 10 # 缩短节点超时时间 cluster-node-timeout 5000 同时在业务层面也做了改进：关键数据（比如订单状态变更）不再只写 Redis，而是先写 MySQL 成功后再更新 Redis。虽然多了一次 DB 写入，但是数据安全性有了保障。 教训 说实话这个问题之前在测试环境完全没暴露过，因为测试环境就一个单节点 Redis。以后做架构设计的时候，分布式环境下的异常场景一定要提前考虑到。Redis 的 AP 模型注定了它在网络分区时可能丢数据，关键业务不能只依赖它。
We hit a weird bug in production recently: users placed orders successfully but couldn't see them after refreshing the page. Initially suspected a frontend caching issue, but it turned out to be caused by a Redis cluster split-brain. The Symptoms Our order status was written to Redis first, then asynchronously persisted to MySQL. This worked fine normally. However, during a network adjustment by our ops team, the Redis cluster experienced a brief network partition. Specifically: in our 3-master-3-replica cluster, one master node lost connection with the others for about 30 seconds. During those 30 seconds, this master kept accepting writes normally. But after the network recovered, it discovered it had been demoted to a replica, and all writes during those 30 seconds were lost. Root Cause Redis Cluster uses a Raft-like election mechanism. When a master is unreachable beyond cluster-node-timeout (default 15 seconds), replicas initiate an election. The problem is: The disconnected master doesn't know it's been replaced - it happily keeps accepting writes After network recovery, it receives a PING from the new master and realizes it's no longer the primary It then demotes itself to replica, syncs from the new master, and discards all previous writes Solution After researching, we applied these configurations: min-replicas-to-write 1 min-replicas-max-lag 10 cluster-node-timeout 5000 We also improved at the application level: critical data (like order status changes) now writes to MySQL first before updating Redis. It adds one more DB write, but ensures data safety. Lessons Learned This issue never surfaced in our test environment because we used a single Redis node there. Distributed failure scenarios must be considered upfront in architecture design. Redis's AP model means it can lose data during network partitions - critical business data shouldn't rely solely on it.