Alerting

A monitor watches one signal (service metrics, traces, logs, metrics or RUM) and changes state when its condition holds. Each state change can notify a channel: Slack, PagerDuty, Opsgenie or a generic webhook. Browser push is set up separately; see Push notifications.

Channels

Create channels on the Alerts page, under Channels. Each monitor routes its warn and critical severities to a channel each (or to none). A warn also goes to the critical channel, and a recover goes to both.

• Slack: an incoming webhook URL (https://hooks.slack.com/...).
• PagerDuty: an Events API v2 routing key. Fire triggers an incident and recover resolves it.
• Opsgenie: an API key, and the region (US or EU). Fire creates an alert and recover closes it.
• Webhook: any HTTPS endpoint you run. Foredeck POSTs JSON to it.
• Email: you can create one, but email is not delivered yet. Alerts routed to an email channel are recorded and not sent. Use another channel for anything you need to hear about.

Secrets (routing keys, API keys and URLs) are never shown again after you save them.

Fire and recover

• Fire: the condition has held for the monitor's required number of consecutive evaluations. One notification is sent. Moving between warn and critical while still breaching counts as a new fire.
• Recover: the condition stops holding. A resolve is sent, so PagerDuty and Opsgenie close the incident on their own.
• Muted monitors aren't evaluated, so they send nothing until the mute ends.

Every notification for one monitor (and one group, if the monitor is grouped) carries the same dedup key, such as fd:3f9a…, until it recovers. It is the PagerDuty dedup_key and the Opsgenie alias, so a fire and its recover land on one incident instead of opening two. Use it the same way in your own webhook handler.

Webhook payload

action is trigger on fire and resolve on recover. state is alert (critical), warn or recovered. Any 2xx response counts as delivered.

{
  "action": "trigger",
  "dedup_key": "fd:3f9a1c0e5b7d2a4c6e8f0a1b",
  "state": "alert",
  "severity": "critical",
  "summary": "checkout p95: latency_p95 = 912 · above 500 — variation:on of new-checkout (80% concentrated), after deploy v42",
  "service": "checkout",
  "value": 912,
  "threshold": 500,
  "flag_key": "new-checkout",
  "variation": "on",
  "concentration": 0.8,
  "deploy_version": "v42"
}

When Foredeck can attribute the breach, flag_key, variation and concentration name the Foredeck Flag variation it is concentrated in, and deploy_version names the deploy it followed. They are null otherwise. A recover carries only state, severity, value, service, dedup_key and summary.

Destination rules

Slack and webhook URLs must meet all of these:

• https:// only. Plain http:// is rejected.
• Port 443. A URL with any other port (:8443, :8080) can't be reached from Foredeck's alerting network.
• A public host name. IP literals and names that resolve to private, loopback, link-local or carrier-grade NAT addresses are rejected.

If your endpoint is behind a firewall, open 443 to the address below, or put a public HTTPS relay in front of it.

Webhook source IP

Alerts are sent from one static address. Allowlist it on your webhook endpoint, a GitHub Enterprise or Atlassian IP allow list, or anything else that filters by source:

136.67.24.136

This address is a stable commitment. If it ever has to change, it will be announced here ahead of time. It sends outbound traffic only and accepts no inbound connections.

Test sends come from a different address

The Send test button on a channel sends from Google Cloud's shared address ranges, not from the address above. Real alerts come from the address above. So if you allowlist only that address, real alerts arrive and test sends are refused. To test an allowlisted endpoint end to end, breach a monitor on purpose (a low threshold on a busy service works) instead of using the button.

Troubleshooting

• The test send works but real alerts don't arrive. Check that your endpoint allows the webhook source IP above, and that the URL is HTTPS on port 443.
• Real alerts work but the test send is refused. Expected if you allowlist by IP; see above.
• Nothing arrives at all. Check on the Alerts page that the monitor actually changed state, that it isn't muted, and that the severity it reached (warn or critical) is routed to this channel.
• Duplicate incidents in PagerDuty or Opsgenie. Each monitor group gets its own dedup key, so a monitor grouped by service opens one incident per service. That's by design.