17.10.5 Alertmanager与Webhook安全

Alertmanager 与 Webhook 安全需要同时关注告警数据完整性、传输机密性与接收端访问控制，避免伪造告警与通知风暴。

原理草图（告警流与安全控制点）#

安装与基础准备（示例环境）#

• Alertmanager 走 Nginx 反向代理，Nginx 终止 TLS 并做 Basic Auth。
• Webhook 接收端使用 Python/Flask，支持 HMAC 签名校验。

1) 安装 Nginx 与生成证书#

# Debian/Ubuntu
sudo apt update && sudo apt install -y nginx apache2-utils openssl

# 生成自签证书（实验环境）
sudo mkdir -p /etc/nginx/ssl
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout /etc/nginx/ssl/alert.key \
  -out /etc/nginx/ssl/alert.crt \
  -subj "/C=CN/ST=Beijing/L=Beijing/O=Ops/OU=Sec/CN=alert.example.com"

2) 生成 Basic Auth 账号#

sudo htpasswd -c /etc/nginx/.htpasswd alertuser
# 输入密码，用于访问 Alertmanager UI/API

3) 配置 Nginx 反向代理#

/etc/nginx/conf.d/alertmanager.conf

server {
    listen 443 ssl;
    server_name alert.example.com;

    ssl_certificate     /etc/nginx/ssl/alert.crt;
    ssl_certificate_key /etc/nginx/ssl/alert.key;

    # 限流：每 IP 每秒 5 个请求，突发 10
    limit_req_zone $binary_remote_addr zone=alert_zone:10m rate=5r/s;

    location / {
        limit_req zone=alert_zone burst=10 nodelay;
        auth_basic "Alertmanager";
        auth_basic_user_file /etc/nginx/.htpasswd;

        proxy_pass http://127.0.0.1:9093;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

sudo nginx -t && sudo systemctl reload nginx

Alertmanager 侧安全配置示例#

/etc/alertmanager/alertmanager.yml

global:
  resolve_timeout: 5m

route:
  receiver: "webhook-secure"

receivers:
  - name: "webhook-secure"
    webhook_configs:
      - url: "https://webhook.example.com/alert"
        send_resolved: true
        http_config:
          tls_config:
            insecure_skip_verify: false
          authorization:
            type: "Bearer"
            credentials: "TOKEN_123456"

命令解释
- send_resolved: 告警恢复也会通知，便于接收端做清理。
- authorization: 在 Header 中注入 Authorization: Bearer TOKEN_123456。

Webhook 接收端示例（HMAC 签名校验）#

/opt/webhook/app.py

from flask import Flask, request, abort
import hmac, hashlib, json, time

app = Flask(__name__)
SECRET = b"WEBHOOK_SHARED_SECRET"

def verify_signature(raw_body, signature):
    mac = hmac.new(SECRET, raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(mac, signature)

@app.route("/alert", methods=["POST"])
def alert():
    raw = request.get_data()
    sig = request.headers.get("X-Signature", "")
    token = request.headers.get("Authorization", "")
    if token != "Bearer TOKEN_123456":
        abort(401, "invalid token")
    if not verify_signature(raw, sig):
        abort(403, "invalid signature")

    data = request.json
    # 仅打印必要字段，避免敏感信息泄露
    print(json.dumps({
        "status": data.get("status"),
        "receiver": data.get("receiver"),
        "alerts": len(data.get("alerts", []))
    }))
    return "ok", 200

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=9001)

启动与依赖安装#

python3 -m venv /opt/webhook/venv
source /opt/webhook/venv/bin/activate
pip install flask
python /opt/webhook/app.py

发送测试告警（生成签名）#

BODY='{"status":"firing","receiver":"webhook-secure","alerts":[{"labels":{"alertname":"Test"}}]}'
SIG=$(python3 - <<'PY'
import hmac,hashlib
body=b'''{"status":"firing","receiver":"webhook-secure","alerts":[{"labels":{"alertname":"Test"}}]}'''
secret=b"WEBHOOK_SHARED_SECRET"
print(hmac.new(secret, body, hashlib.sha256).hexdigest())
PY
)
curl -k -X POST https://webhook.example.com/alert \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer TOKEN_123456" \
  -H "X-Signature: $SIG" \
  -d "$BODY"

预期效果：接收端返回 200 ok，日志输出告警摘要。

传输与证书校验（mTLS 可选示例）#

/etc/alertmanager/alertmanager.yml

receivers:
  - name: "webhook-mtls"
    webhook_configs:
      - url: "https://webhook.example.com/alert"
        http_config:
          tls_config:
            ca_file: /etc/alertmanager/ca.crt
            cert_file: /etc/alertmanager/client.crt
            key_file: /etc/alertmanager/client.key

命令解释
- ca_file: 服务器 CA，防止中间人。
- cert_file/key_file: 客户端证书，Webhook 端可验证来源。

接收端安全策略示例（Nginx 前置）#

/etc/nginx/conf.d/webhook.conf

server {
    listen 443 ssl;
    server_name webhook.example.com;

    ssl_certificate     /etc/nginx/ssl/webhook.crt;
    ssl_certificate_key /etc/nginx/ssl/webhook.key;

    client_max_body_size 1m;

    location /alert {
        limit_req_zone $binary_remote_addr zone=wh_zone:10m rate=3r/s;
        limit_req zone=wh_zone burst=5 nodelay;

        proxy_pass http://127.0.0.1:9001;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

排错与验证#

常见问题#

1) 401/403
- 令牌或签名不匹配。
2) TLS 握手失败
- 证书 CN/SAN 与域名不一致，或 CA 未信任。
3) 重试风暴
- Webhook 未返回 2xx 或超时。

快速排查命令#

# 检查 Alertmanager 到 Webhook 的 TLS
curl -v https://webhook.example.com/alert

# 查看 Alertmanager 发送记录
journalctl -u alertmanager -n 50

# 验证 Nginx 限流状态
grep "limit_req" /var/log/nginx/error.log

告警最小化与脱敏示例#

/etc/prometheus/alert.rules.yml

groups:
  - name: secure-alerts
    rules:
      - alert: HostDown
        expr: up == 0
        labels:
          severity: critical
        annotations:
          summary: "实例不可达"
          description: "实例 {{ $labels.instance | reReplaceAll \"([0-9]+\\.[0-9]+)\\.[0-9]+\\.[0-9]+\" \"$1.*.*\" }} 不可达"

练习#

1) 将 Alertmanager UI 仅对内网开放，外网访问返回 403。
2) 给 Webhook 增加 X-Signature 校验，并验证错误签名被拒绝。
3) 调整 limit_req，模拟 50 并发请求并观察限流日志。
4) 启用 mTLS，验证无客户端证书时请求失败。