Prometheus

Prometheus stores everything as time series: streams of timestamped values belonging to the same metric name and the same set of labelled dimensions. One sentence in the data model documentation decides most of how you will operate it: “Every time series is uniquely identified by its metric name and optional key-value pairs called labels.” Change any label value, add a label, remove a label, and you have created a different time series with its own index entry, its own chunks and its own memory.

Everything below follows from that, and from the second design choice: Prometheus goes and gets its data instead of waiting to be given it.

The pull model, and what it costs you

A Prometheus server is configured with a set of jobs. Each job resolves to a list of targets, either statically or through service discovery, and Prometheus scrapes an HTTP endpoint on each target at a fixed interval.

Two operational consequences matter more than the mechanism.

Target liveness comes free. Prometheus writes a synthetic up metric on every scrape, so up == 0 is a working “this thing is gone” alert with no cooperation from the target. There is nothing to configure and no heartbeat to forget.

The server needs to reach every target. Service discovery (Kubernetes, Consul, EC2, DNS, file based) exists so that the target list tracks reality without configuration changes, and relabel_configs exists to filter and rewrite that list before scraping. When the network is in the way, the documented preference is to move the Prometheus server, not to reverse the data flow: run it on the same network as the monitored instances, or use PushProx to traverse a firewall or NAT.

Push exists as an exception, and the exception is narrower than most people assume. The Pushgateway documentation lists three pitfalls: it becomes a single point of failure and a bottleneck; you lose the automatic up health signal; and it never forgets a series, so anything pushed is exposed to Prometheus forever unless deleted through its API. The documented legitimate case is one thing: the outcome of a service-level batch job, meaning a job not tied to a particular machine or instance, whose metrics therefore carry no instance label. For machine-related batch jobs (configuration management runs, security update cron jobs) the documented answer is the Node Exporter’s textfile collector instead.

Cardinality is the failure mode

Because a series is its full label set, the number of series a metric produces is the product of the distinct values of its labels. That multiplication is the single most common way to destroy a Prometheus server, and it is usually introduced by someone adding one label that looked harmless.

The project’s own worked example, from the instrumentation guidance: node_exporter exposes node_filesystem_avail per mounted filesystem, so each node produces on the order of tens of series. At 10,000 nodes that is roughly 100,000 series for that one metric, which the documentation says Prometheus handles fine. Now add a per-user quota dimension: with 10,000 users on 10,000 nodes you reach a double-digit number of millions of series, which the documentation states is too much for the current implementation.

The general guidance is stricter than most instrumentation actually is: keep the cardinality of a metric below 10, limit metrics that exceed that to a handful across the whole system, and note that “the vast majority of your metrics should have no labels”. Above a cardinality of 100, or with the potential to grow there, the advice is to reduce dimensions or move the analysis to a general-purpose processing system, not a monitoring one.

The labels that cause this are recognisable. The naming guidance names them: do not use labels for high-cardinality dimensions such as user IDs, email addresses, or other unbounded sets. In practice the list also includes request IDs, trace IDs, session tokens, full URL paths with identifiers embedded, customer names, container IDs and error messages. A label whose value set grows with traffic rather than with infrastructure is the one that will hurt you.

Series are also cheap to add and expensive to remove. A series that stops receiving samples still occupies index space and disk in every block covering the period it existed, so a deployment that emitted a request-ID label for an hour keeps costing you for the whole retention window. And a classic histogram multiplies the problem: one histogram with ten buckets and three labels of five values each is 5³ × 12 series, not 12.

Defences, in order of effectiveness: do not instrument it that way; drop it at scrape time with metric_relabel_configs; and set sample_limit per job so a bad deployment fails its own scrape instead of the server. Watch prometheus_tsdb_head_series and scrape_samples_scraped, and use the TSDB status page to see which metric names dominate.

The four metric types, and choosing correctly

The counter/gauge mistake is the common one and it is silent: a counter used for a value that can decrease breaks rate(), because rate() treats every decrease as a counter reset and adjusts for it. Current running processes is a gauge, not a counter. Note that the server flattens all types except native histograms into untyped float series. The type information lives in the instrumentation library and the exposition format, so nothing stops you from getting this wrong.

Histogram versus summary

A classic histogram exposes cumulative bucket counters as <name>_bucket{le="<upper bound>"}, plus <name>_sum and <name>_count; quantiles are computed on the server with histogram_quantile(). A summary computes streaming quantiles inside the instrumented process and exposes them as <name>{quantile="<φ>"}.

The difference that decides the choice:

Quantiles from summaries cannot be aggregated across instances, and this is arithmetic, not a limitation of the implementation. A quantile is a rank statistic. Knowing the 95th percentile of each of twenty backends tells you nothing about the 95th percentile of the twenty together, because you do not know how many observations each contributed or how they were distributed. The documentation marks the attempt explicitly:

avg(http_request_duration_seconds{quantile="0.95"})  // BAD

Histograms aggregate because buckets are counters, and counters add:

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))  // GOOD

The by (le) clause is required: histogram_quantile() needs the bucket boundary label to survive aggregation. Dropping it is a common and quiet error.

The trade-off runs the other way for accuracy near a specific threshold. Histogram error is bounded by bucket width, so if your objective is 300 ms and your buckets jump from 100 ms to 1 s, the estimate is close to useless: the interpolation has nothing to work with. Choose bucket boundaries around the values you will be asked about. A summary controls error in φ instead, which is precise but not aggregatable. The documented resolution of this tension is native histograms: exposed as composite samples, not one series per bucket, they need no bucket configuration, transfer atomically, and are always aggregatable with each other even across resolution changes. The Go and Java instrumentation libraries support them, and where they do the documentation says to prefer them.

If you cannot change instrumentation, the scrape-config option convert_classic_histograms_to_nhcb (default false) ingests classic histograms as native histograms with custom bucket boundaries, which buys the storage efficiency and atomic transfer but not the resolution or the aggregatability across bucket changes.

PromQL, and the mistakes

Two expression types carry most of the confusion.

• An instant vector is one sample per series at one instant. http_requests_total is an instant vector. A series only appears if its most recent sample is within the lookback period, which defaults to 5 minutes (--query.lookback-delta).
• A range vector is a series of samples over a window. http_requests_total[5m] is a range vector, and the range is left-open, right-closed.

You cannot graph a range vector, and you cannot do arithmetic on one. Range vectors exist to be fed to functions.

rate(v range-vector) gives the per-second average rate of increase over the window, adjusting for counter resets and extrapolating to the window edges to allow for missed scrapes. Three rules:

1. It requires a counter. On a gauge the result is meaningless, and worse, plausible: every decrease is read as a reset and added back in.
2. It requires a range. rate(http_requests_total) is a syntax error. This is the error people fix by writing rate(http_requests_total[5m]) without asking what 5m should be.
3. The range must contain enough samples. With scrape_interval: 1m, a [1m] range holds one or two samples and produces gaps whenever a scrape is late. The working rule is at least four scrape intervals, so [5m] at a 1-minute interval and [2m] at 15 seconds.

Related mistakes worth naming:

• rate() after aggregation. rate(sum(http_requests_total)[5m:]) hides counter resets, because the sum of restarting counters is not itself monotonic. Always sum(rate(...)), never rate(sum(...)).
• irate() for graphs and alerts. It uses only the last two samples, so it is spiky and it is easy to miss a spike entirely between resolution steps. Use it for fast-moving debugging, rate() for anything you alert on.
• increase() read as an exact count. It extrapolates over the whole range, so it can return a non-integer even for a counter that only steps by whole numbers.
• Ratios computed as averages of ratios. Aggregate numerator and denominator separately, then divide. The recording rule guidance is explicit that averaging a ratio, or averaging an average, is not statistically valid.
• Aggregating without saying what you drop. Prefer sum without (instance)(...) over sum by (job)(...), so labels you did not think about (job, cluster, le) are preserved and not silently discarded.

Recording rules and alerting rules

Recording rules precompute expressions on evaluation_interval and store the result as a new series. They exist for two reasons: to make expensive dashboard queries cheap, and to make a definition single-sourced so a dashboard and an alert cannot drift apart.

The naming convention is level:metric:operations: aggregation level and output labels, then the metric name with _total stripped when rate() is applied, then the operations applied with the newest first. It is a convention that catches mistakes: if the labels in a without clause are not reflected in the level of the output name, a rule is probably wrong. Colons are reserved for recording rules and should not appear in metric names from exporters or direct instrumentation.

groups:
  - name: api-slo
    interval: 30s
    rules:
      - record: instance_path:requests:rate5m
        expr: rate(http_requests_total{job="api"}[5m])

      - record: path:requests:rate5m
        expr: sum without (instance) (instance_path:requests:rate5m{job="api"})

      - record: instance_path:request_failures:rate5m
        expr: rate(http_requests_total{job="api",code=~"5.."}[5m])

      - record: job:request_failures_per_requests:ratio_rate5m
        expr: |2
            sum without (instance, path) (instance_path:request_failures:rate5m{job="api"})
          /
            sum without (instance, path) (instance_path:requests:rate5m{job="api"})

Alerting rules use the same file format. An alert becomes active whenever the expression returns any vector element, and each element’s label set is a separate alert instance.

groups:
  - name: api-alerts
    rules:
      - alert: ApiTargetDown
        expr: up{job="api"} == 0
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "Instance {{ $labels.instance }} of job {{ $labels.job }} is down"

      - alert: ApiErrorRatioHigh
        expr: job:request_failures_per_requests:ratio_rate5m{job="api"} > 0.02
        for: 10m
        keep_firing_for: 5m
        labels:
          severity: page
        annotations:
          summary: "API error ratio above threshold"
          description: "Error ratio is {{ $value | humanizePercentage }} over the last 5 minutes."

for is the difference between a usable alerting setup and pager fatigue. Without it, an alert fires on the first evaluation where the expression matched. At a 1-minute evaluation interval on a 5-minute rate window, that means a single slow minute pages someone. With for: 10m, Prometheus requires the element to be active at every evaluation for ten minutes; in between it is pending, visible in the UI and in the synthetic ALERTS{alertstate="pending"} series. keep_firing_for is the mirror image, holding an alert firing after the condition stops matching, which suppresses flapping.

The cost of for is detection latency, and it compounds with the rate window: a for: 10m alert on a rate(...[5m]) expression can take fifteen minutes to fire. Set it against how long the condition must persist before a human should be woken, not by habit.

Prometheus itself does not notify. The alerting rules documentation is explicit that they “are not a fully-fledged notification solution” and that summarisation, rate limiting, silencing and alert dependencies live in Alertmanager.

Alertmanager: grouping, inhibition, silencing

Alertmanager exists because a single failure produces many alerts. Its three mechanisms answer three different questions.

Grouping batches alerts sharing the labels named in group_by into one notification, so twenty instances of the same alert become one message with twenty entries.

group_interval also bounds the notification request itself: the documentation notes that if a receiver takes longer than group_interval to respond, the notification is cancelled. That is a real failure mode with small intervals and slow webhooks. repeat_interval is checked after each group_interval and is rounded up to a multiple of it.

Inhibition encodes dependency. An inhibition rule mutes a target alert while a source alert exists, provided both carry the same values for every label in equal. This is how “the data centre link is down” suppresses the two hundred “service unreachable” alerts that follow it.

Silencing is the manual, time-bounded mute, matched on labels and created by an operator through the UI or API. It is what you use during planned maintenance; it is deliberately not part of the configuration file, because it is an operational act, not a policy.

route:
  receiver: platform-default
  group_by: [alertname, cluster, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = page
      receiver: platform-oncall
      group_wait: 10s
      repeat_interval: 1h
    - matchers:
        - severity = ticket
      receiver: platform-tickets
      repeat_interval: 24h

inhibit_rules:
  - source_matchers:
      - alertname = ClusterNetworkDown
    target_matchers:
      - severity =~ "page|ticket"
    equal: [cluster]

receivers:
  - name: platform-default
    webhook_configs:
      - url: http://chatops.internal:9095/alerts
  - name: platform-oncall
    webhook_configs:
      - url: http://pager-bridge.internal:9096/page
  - name: platform-tickets
    webhook_configs:
      - url: http://ticket-bridge.internal:9097/create

Two structural notes. The routing tree is traversed top-down and stops at the first matching child unless continue: true, and child routes inherit unset timing parameters from the parent, so the grouping keys and intervals you set at the root are the ones most routes actually use. And a route’s own labels do not affect grouping; only group_by does.

Storage and retention

Samples are grouped into two-hour blocks on disk. The current block lives in memory, protected by a write-ahead log in 128 MB segments, replayed on restart; at least three WAL files are retained, more on high-traffic servers to cover two hours of raw data. Chunk segment files default to 512 MB. Background compaction merges two-hour blocks into larger ones spanning up to 10% of the retention period or 31 days, whichever is smaller. Source and destination blocks coexist briefly, so on-disk size can temporarily exceed a size-based retention limit.

Capacity planning is straightforward because the storage is dense: Prometheus stores an average of 1-2 bytes per sample, so needed_disk_space = retention_time_seconds × ingested_samples_per_second × bytes_per_sample. Note which term you can move. Halving the scrape interval halves samples per second; halving the number of series does more, because samples within a single series compress well and a new series does not.

The constraint that shapes architecture: local storage is neither clustered nor replicated. A Prometheus server is a single-node database and should be operated like one. Two Prometheus servers scraping the same targets gives you availability, not a cluster: they hold independent, slightly different data. For backups, the documentation recommends the snapshot API; a file-level copy that includes wal/, chunks_head/ and wbl/ risks an incoherent backup, and excluding them costs you the period they cover.

remote_write streams samples to an external system for long-term retention, cross-server queries, or downsampling. It is a stream, not a replication protocol: the local TSDB still applies its own retention, and write_relabel_configs, applied after external labels, is how you send a subset instead of everything.

remote_write:
  - url: https://metrics-store.internal/api/v1/write
    protobuf_message: io.prometheus.write.v2.Request
    remote_timeout: 30s
    name: long-term
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_.*|process_.*"
        action: drop
    queue_config:
      max_samples_per_send: 2000
      capacity: 10000
      max_shards: 50

protobuf_message defaults to prometheus.WriteRequest (Remote Write 1.0). Remote Write 2.0’s io.prometheus.write.v2.Request is more efficient and sends metadata, start timestamps and native histograms by default, but the documentation is explicit that you should confirm your remote storage supports it before switching. Watch prometheus_remote_storage_samples_pending, prometheus_remote_storage_samples_dropped_total and the queue’s shard count: a remote endpoint that falls behind causes memory growth locally, then drops.

Diagnostics

Prometheus exposes its own health and internal metrics at http://localhost:9090/metrics and status pages at http://localhost:9090/targets and http://localhost:9090/tsdb-status.

Targets showing as DOWN. Check the Targets page for the error message. Common causes:

# Test reachability from the Prometheus host:
curl -s http://<target>:9100/metrics | head -5

If the target is reachable manually but Prometheus reports it down, check scrape_timeout (default 10s). A slow exporter that takes 12 seconds to respond is marked unhealthy.

High memory usage / OOM kills. Prometheus stores all active time series in memory. Check current series count:

prometheus_tsdb_head_series

If this number is in the millions, you have a cardinality problem. Identify the culprit:

topk(10, count by (__name__)({__name__=~".+"}))

The fix is to drop high-cardinality labels at ingestion using metric_relabel_configs with action: drop or action: labeldrop, or to fix the exporter that is generating unbounded label values (user IDs, request paths, UUIDs in labels).

Slow queries / dashboard timeouts. Check the query log or the /api/v1/query endpoint response time. Common causes: queries that touch millions of series (use {__name__=~".+"} as a smoke test), high rate() ranges over long windows, or missing recording rules for expensive aggregations. Add recording rules for any query that takes more than 2 seconds:

groups:
  - name: recording
    rules:
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)

WAL corruption after crash. If Prometheus fails to start after an unclean shutdown with "msg"="WAL segment is corrupted" in logs, use the WAL repair tool:

promtool tsdb clean /var/lib/prometheus/data

This removes corrupted WAL segments. Some recent data points may be lost, but the TSDB recovers.

Alerts not firing. Check the Alerts page (/alerts). If the alert is in inactive state, the expression is not matching. Test the expression directly in the query UI. If it is in pending, the for duration has not elapsed yet. If it is firing but no notification arrives, the problem is in Alertmanager routing, not Prometheus.

Where Prometheus is the wrong tool

The overview says the important part plainly: “If you need 100% accuracy, such as for per-request billing, Prometheus is not a good choice, as the collected data will likely not be detailed and complete enough.”

That is a design decision, not a defect. Prometheus values availability of the monitoring system over completeness of the data. It samples at an interval, so anything between scrapes is not recorded. A failed scrape is a gap, not a retry. rate() extrapolates. Staleness markers end series that stop reporting. All of that is correct behaviour for answering “is this broken now?” and disqualifying for anything that must reconcile to an exact figure.

• Billing, invoicing, or anything audited. Use an event pipeline that guarantees delivery, and Prometheus for the operational view of that pipeline.
• Logs. Prometheus stores numbers, not text. A log line is not a sample, and a metric labelled with an error message is a cardinality incident. Use a log system; Loki if you want the same label model.
• Individual events and per-request traces. “Which request was slow, and where did the time go?” is a tracing question. Exemplars link a metric to a trace ID, which is the intended bridge.
• Long retention on one server. 15 days is the default for a reason. Use remote_write to a system built for it; do not grow one node’s disk.
• High-cardinality analytics. Per-user, per-tenant or per-request-ID breakdowns are a data warehouse question. The instrumentation guidance says so directly: above a cardinality of about 100, move the analysis to a general-purpose processing system.
• A durable, replicated metrics store out of the box. Local storage is single-node. If you need more, that is a federation, Thanos or Cortex/Mimir decision, and it should be made deliberately.

Where it fits, the reason is the same as the reason for its limits: a Prometheus server has no dependencies to fail, so it keeps answering while the system it monitors is falling apart. That property is worth the sampling, and it is why so many projects expose a /metrics endpoint by default - etcd documents Prometheus as the way to ingest its metrics and ships default alerting rules, and Kubernetes components expose Prometheus metrics throughout the control plane. Instrument accordingly: bounded label sets, counters and histograms rather than gauges and summaries, recording rules for anything a dashboard and an alert both need, and for durations set by how long a human should sleep through the condition.