Operations Updated 27 Oct 2026

Four golden signals, RED and USE

Three widely used checklists for what to monitor. The four golden signals (latency, traffic, errors, saturation) and RED (rate, errors, duration) describe request-driven services from the user’s side; USE describes resources such as CPUs, disks and queues. Most systems need a service view and a resource view.

On this page

The four golden signals
The RED method
The USE method
Which to use

Framework	Signals	Applies to	Origin
Four golden signals	Latency, traffic, errors, saturation	User-facing services	Google SRE book
RED	Rate, errors, duration	Request-driven services, especially microservices	Tom Wilkie
USE	Utilization, saturation, errors	Every hardware or software resource	Brendan Gregg

The four golden signals

Latency: Time to serve a request. Track successful and failed requests separately: a fast error should not make latency look good, and a slow error is worse than a fast one. Use distributions or the share of requests under a threshold, not averages.
Traffic: Demand on the system: requests per second for an API, sessions or transactions for others, bytes for streaming.
Errors: Rate of failed requests, whether explicit (HTTP 5xx), implicit (a 200 with the wrong content) or by policy (slower than your latency commitment).
Saturation: How full the service is: the most constrained resource, such as CPU, memory, I/O, connection pool or queue depth. Many systems degrade well before 100%, and saturation predicts problems before they show as errors.

The RED method

RED trims the golden signals to three per-service metrics that are easy to standardise across every microservice, so every dashboard looks the same:

Rate: requests per second.
Errors: failed requests per second (or as a share of rate).
Duration: distribution of request latency.

Saturation is left to the resource view. Example PromQL for a service instrumented with a request counter and a latency histogram:

# Rate
sum by (service) (rate(http_requests_total[5m]))

# Errors (ratio)
sum by (service) (rate(http_requests_total{code=~"5.."}[5m]))
  / sum by (service) (rate(http_requests_total[5m]))

# Duration: 99th percentile
histogram_quantile(0.99,
  sum by (service, le) (rate(http_request_duration_seconds_bucket[5m])))

# Duration: share of requests taking 300 ms or less (SLI-friendly)
sum by (service) (rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
  / sum by (service) (rate(http_request_duration_seconds_count[5m]))

These use classic Prometheus histograms. The last query needs a bucket boundary at exactly 0.3 seconds, and le means “less than or equal”, so it counts requests of 300 ms or less; choose bucket boundaries to match your latency SLO thresholds. histogram_quantile interpolates within buckets, so it estimates the percentile. Native histograms use different expressions.

The USE method

For every resource, check:

Utilization: the share of time the resource was busy, or the share of capacity in use.
Saturation: extra work it cannot service yet, usually queued.
Errors: count of error events.

USE is a methodology for performance investigation: walk every resource systematically instead of guessing. Typical Linux resources:

Resource	Utilization	Saturation	Errors
CPU	Busy time per CPU and overall	Run-queue length above CPU count; CPU pressure (PSI); throttling in containers	Machine check exceptions (rare)
Memory	Used vs available memory	Swapping, page scanning, memory pressure (PSI), OOM kills	Allocation failures; ECC errors
Disk	Device busy time	Wait queue length; I/O latency; I/O pressure (PSI)	Device and filesystem errors
Network	Throughput vs link speed	Drops, overruns, retransmits; socket backlog	Interface errors
Connection pool, thread pool, queue	In use ÷ size	Waiters; queue age	Timeouts; rejected work

Which to use

SLOs and paging: base them on the service view (golden signals or RED), measured as close to the user as practical. See SLIs and SLOs.
Dashboards: a RED row per service with a USE section for its key resources makes triage fast.
Investigation: when a service signal is bad, USE tells you which resource is the bottleneck.
Capacity planning: utilization and saturation trends, combined with traffic growth.

Sources: Google, Site Reliability Engineering, ch. 6 “Monitoring Distributed Systems”; Julie Dam, “The RED Method: How to instrument your services”, reporting Tom Wilkie’s talk; Brendan Gregg, “The USE Method”.