Kafka消息队列深度实践：从入门到性能调优
Apache Kafka Deep Dive: From Basics to Performance Tuning
| iDev Tech | 2026-08-02T10:00:00
全面讲解Kafka的核心概念、生产者/消费者最佳实践和生产环境性能调优策略。
Comprehensive guide to Kafka core concepts, producer/consumer best practices, and production performance tuning.
核心概念Topic（主题）是消息的逻辑分类，Partition（分区）是Topic的物理分片。消息按Key哈希分配到Partition，同一Partition内消息有序。Consumer Group内的消费者各自消费不同Partition，实现并行消费。Offset记录消费进度，支持精确一次（Exactly-Once）语义。生产者最佳实践batch.size和linger.ms控制批量发送策略（推荐batch.size=64KB, linger.ms=5）。acks=all确保消息被所有ISR副本确认。enable.idempotence=true防止重复发送。Key的选择直接影响Partition分布——建议使用业务ID（如订单ID）而非随机值。性能调优消费端：增加Partition数量提升并行度（建议Partition数=消费者实例数的2-3倍）。调大fetch.min.bytes和fetch.max.wait.ms减少拉取次数。开启压缩（compression.type=lz4）降低网络开销。监控Consumer Lag——持续增长说明消费跟不上生产。
Core ConceptsTopics are logical message categories, Partitions are physical shards of Topics. Messages are hash-distributed to Partitions by key; messages within a Partition are ordered. Consumers in a Consumer Group each consume different Partitions for parallel processing. Offsets track consumption progress, supporting Exactly-Once semantics.Producer Best Practicesbatch.size and linger.ms control batching strategy (recommended batch.size=64KB, linger.ms=5). acks=all ensures all ISR replicas acknowledge. enable.idempotence=true prevents duplicate sends. Key selection directly impacts Partition distribution — use business IDs (order ID) rather than random values.Performance TuningConsumer side: increase Partition count for parallelism (recommended Partitions = 2-3x consumer instances). Increase fetch.min.bytes and fetch.max.wait.ms to reduce fetch frequency. Enable compression (compression.type=lz4) to reduce network overhead. Monitor Consumer Lag — continuous growth indicates consumers can't keep up with producers.