Monitor your services

Foredeck runs on Foredeck. Every service behind this app — the events API, the Kafka consumer, the experiment API and this web app — reports through the same public ingest contracts your services would use. This page is that setup, written down, including the outside-in piece we recommend and the parts that are easy to get wrong.

The three pieces

1. In-process instrumentation. One middleware per HTTP service gives you a server span, a request measurement, and RED counters per request. Logs ship from the logging framework you already use.
2. An external collector. Nothing inside a service can report that the service is down, that the broker is unreachable, or that the container is in a restart loop. A small agent samples that from outside on a fixed interval.
3. Monitors and SLOs as configuration. Provisioned once, versioned with the code, reconciled on every run.

1. Instrument the service

In Node, the SDK covers all four signals. Initialise once per process and the rest is middleware:

import express from 'express';
import { createClient, foredeckRequestMetrics, logs, tracing } from '@foredeck/sdk-js';

const foredeck = createClient({
  sdkKey: process.env.FOREDECK_SDK_KEY,
  baseUrl: process.env.FOREDECK_API_URL,
  eventsUrl: process.env.FOREDECK_EVENTS_URL,
  service: 'checkout-api',
  env: 'prod',
});

tracing.init({
  endpoint: process.env.FOREDECK_EVENTS_URL,
  sdkKey: process.env.FOREDECK_SDK_KEY,
  service: 'checkout-api',
  env: 'prod',
  serviceVersion: process.env.RELEASE_SHA,  // deploy markers, recorded passively
});

logs.init({
  endpoint: process.env.FOREDECK_EVENTS_URL,
  sdkKey: process.env.FOREDECK_SDK_KEY,
  service: 'checkout-api',
  env: 'prod',
});

const app = express();
app.use(foredeckRequestMetrics(foredeck));   // one measurement per handled request

Two rules are worth copying, because both were learned the hard way while instrumenting this platform:

• A 4xx is not your error. Count only 5xx (and unhandled exceptions) as status: 'error'. Counting client mistakes burns your error budget for someone else's bad request.
• Keep the resource dimension bounded. Record the route template (GET /orders/{id}), never the raw URL. The SDK middleware skips requests that matched no route rather than record one.

2. Connect the trace across process boundaries

A trace is only useful if it survives the hop. instrumentFetch() injects the W3C traceparent on outbound calls, and tracing.extract() continues an inbound one:

// caller
const fetch = tracing.instrumentFetch();
await fetch('https://payments.internal/charge', { method: 'POST', body });

// callee
const parent = tracing.extract(req.headers);
const span = tracing.startSpan('POST /charge', { kind: 'server', ...parent });

Queues need the same treatment. Foredeck's own pipeline carries the trace context as a Kafka message header, never in the payload — the message body is the ingest contract, and tracing must not change a stored field. An ingest request and the consumer work it triggers then land in one trace instead of two unrelated ones.

3. Watch the infrastructure from outside

In-process instrumentation has a blind spot it cannot close: a process cannot report its own death. If a service hangs, the code inside it hangs too; if the broker is unreachable, nothing inside your service can say so; a container in a restart loop is never alive long enough to complain. Closing that needs a small always-on process outside the ones it watches.

Report each dependency as its own service rather than as tags on the collector, so broker health lands on the broker's service page and its own SLO:

// every 60s, from a small always-on process
await fetch(`${eventsUrl}/counters`, {
  method: 'POST',
  headers: { Authorization: `Bearer ${sdkKey}`, 'Content-Type': 'application/json' },
  body: JSON.stringify({
    counter: 'kafka.consumer_lag',
    count: lag,
    service: 'kafka',            // reported AS kafka, not as the collector
    env: 'prod',
    tags: { topic, group, reported_by: 'selfmon-agent' },
  }),
});

Black-box probes belong on the target service too: record the probe result as a service metric on the service being probed, and an outage shows up where an on-call engineer will actually look.

4. Alert on it

A metric-source monitor sums a counter over its evaluation window, which shapes how you alert on point-in-time gauges:

• Level gauges (percentages, queue depth, lag): set window_seconds equal to your collection interval, so one window normally holds one sample, and set for_evaluations: 2 so a window that happens to catch two samples cannot fire on its own.
• Up-checks: alert on { op: 'below', critical: 1 } over three intervals. That sums to zero both when the dependency is down and when your collector has stopped reporting — one monitor covers “it is broken” and “we are blind”, and neither should ever be silent.

import { monitors } from '@foredeck/sdk-js';

const m = monitors({ endpoint, sdkKey });

await m.createMonitor({
  name: 'kafka down',
  source: 'metric',
  metric: 'kafka.up',
  query: { service: 'kafka', env: 'prod' },
  condition: { type: 'threshold', op: 'below', critical: 1 },
  window_seconds: 180,        // 3 x the 60s collection interval
  cadence_seconds: 60,
  routing: { critical: [pagerdutyChannelId] },
});

await m.reportDeploy({ service: 'checkout-api', version: process.env.RELEASE_SHA });

Rules for instrumentation that cannot page you at 3am

Telemetry runs inside your production request path, so it has to be un-noticeable when it breaks. Every Foredeck emitter follows the same four rules, and they are worth adopting for anything you write yourself:

1. Never raise. A failure in instrumentation is counted, not propagated.
2. Never block. Records go on a bounded queue drained by a background worker. A full queue drops the oldest record, so under overload you keep the telemetry describing the incident in progress.
3. Never loop. If your telemetry travels through your own pipeline, instrumenting that pipeline feeds it back into itself. Mark telemetry-derived traffic and skip instrumenting it — and prefer counters flushed on a fixed cadence over one counter per processed message, so volume does not scale with traffic.
4. Never fabricate. Missing configuration means emit nothing, not emit to the wrong place.

Python and other runtimes

The ingest contracts are plain authenticated JSON over HTTP, so any runtime can emit them directly: POST /counters, POST /v1/logs/intake, POST /v1/traces/intake, POST /v1/service-metrics, and POST /v1/deployments. Foredeck's own Python services use a dependency-free emitter implementing exactly those five, with an ASGI middleware and a stdlib logging handler — it ships in the platform repo at packages/foredeck-telemetry-py as a reference implementation of the four rules above.

import foredeck_telemetry as ft
from foredeck_telemetry.bootstrap import setup

setup("checkout-api", app=app)          # ASGI middleware + log handler

with ft.span("charge", kind="client") as span:
    span.set_attribute("psp", "stripe")
    ft.count("checkout.completed")