Prometheus + Grafana 监控体系搭建：从指标采集到告警

# Prometheus + Grafana 监控体系搭建 ## 架构概览 ``` 应用 (metrics 端点) -> Prometheus (抓取+存储) -> Grafana (可视化) -> AlertManager (告警) ``` ## Spring Boot 指标暴露 ```xml io.micrometer micrometer-registry-prometheus ``` ```yaml management: endpoints: web: exposure: include: prometheus,health metrics: tags: application: my-app ``` ## 自定义业务指标 ```java @Component public class OrderMetrics { private final Counter orderCounter; private final Timer orderProcessingTimer; private final Gauge pendingOrderGauge; public OrderMetrics(MeterRegistry registry) { this.orderCounter = Counter.builder("orders.created.total") .description("创建的订单总数") .tag("type", "all") .register(registry); this.orderProcessingTimer = Timer.builder("orders.processing.duration") .description("订单处理耗时") .publishPercentiles(0.5, 0.95, 0.99) .register(registry); this.pendingOrderGauge = Gauge.builder("orders.pending.count", this, OrderMetrics::getPendingCount) .register(registry); } public void recordOrder() { orderCounter.increment(); } public Timer.Sample startTimer() { return Timer.start(); } public void stopTimer(Timer.Sample sample) { sample.stop(orderProcessingTimer); } } ``` ## PromQL 常用查询 ```promql # 请求 QPS（每秒请求数） rate(http_server_requests_seconds_count[5m]) # P99 延迟 histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m])) # 错误率 sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m])) / sum(rate(http_server_requests_seconds_count[5m])) # CPU 使用率 process_cpu_usage * 100 ``` ## AlertManager 规则 ```yaml groups: - name: app-alerts rules: - alert: HighErrorRate expr: | sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m])) / sum(rate(http_server_requests_seconds_count[5m])) > 0.05 for: 5m labels: severity: critical annotations: summary: "错误率超过 5%" description: "当前错误率: {{ $value | humanizePercentage }}" - alert: HighLatency expr: | histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m]) ) > 2 for: 5m labels: severity: warning annotations: summary: "P99 延迟超过 2 秒" ``` ## 总结 监控三板斧：指标采集（Prometheus）+ 可视化（Grafana）+ 告警（AlertManager），是微服务运维的基础设施。