If you're searching for a practical Prometheus Query Cheat Sheet, you're probably looking for one thing: reliable PromQL expressions you can copy, modify, and use immediately.

This guide is designed as both a learning resource and a production-ready reference. Instead of only listing functions, it explains when to use them, why they work, and common mistakes to avoid. Whether you're debugging a Kubernetes cluster, building Grafana dashboards, creating Alertmanager rules, or investigating application latency, you'll find examples that reflect real-world operational workflows.

Unlike many cheat sheets that only scratch the surface, this guide covers modern PromQL concepts including vector matching, histogram queries, label filtering, aggregation, subqueries, recording rules, and performance optimization aligned with current Prometheus best practices.

Quick Summary

In this guide you'll learn how to:

• Write valid PromQL expressions from scratch
• Filter metrics using labels and regular expressions
• Understand instant vectors, range vectors, scalars, and strings
• Use aggregation, comparison, arithmetic, and logical operators
• Query Node Exporter and Kubernetes metrics
• Calculate CPU, memory, disk, and network utilization correctly
• Build production-ready Grafana dashboards
• Create efficient alert expressions
• Avoid common PromQL mistakes that lead to inaccurate dashboards

Who should read this?

• DevOps Engineers
• Site Reliability Engineers (SREs)
• Platform Engineers
• Kubernetes Administrators
• Cloud Engineers
• Backend Developers
• Anyone working with Prometheus or Grafana

What Is Prometheus?

Prometheus is an open-source monitoring and alerting platform designed for collecting and querying time-series metrics.

Originally created at SoundCloud and now a graduated CNCF project, Prometheus has become the de facto monitoring solution for Kubernetes and cloud-native infrastructure.

Prometheus works by:

1. Scraping metrics from configured targets at regular intervals.
2. Storing metrics as time-series data.
3. Allowing users to query metrics using PromQL.
4. Evaluating alerting rules.
5. Sending alerts to Alertmanager.

Every collected metric consists of:

• Metric name
• Timestamp
• Value
• Labels (key-value metadata)

For example:

http_requests_total{
    method="GET",
    status="200",
    instance="api-01"
}

Rather than storing metrics in relational tables, Prometheus stores every unique label combination as its own time series.

What Is PromQL?

PromQL (Prometheus Query Language) is Prometheus' powerful query language for retrieving, transforming, aggregating, and analyzing time-series metrics.

Think of PromQL as SQL—but designed specifically for monitoring data.

With PromQL you can:

• Monitor infrastructure health
• Analyze application performance
• Detect anomalies
• Build Grafana dashboards
• Create alerting rules
• Calculate Service Level Indicators (SLIs)
• Investigate production incidents

For example, the simplest possible query is:

up

Result:

• 1 = Target is healthy
• 0 = Target is unreachable

A slightly more useful query calculates the request rate:

rate(http_requests_total[5m])

This returns the average number of HTTP requests per second over the previous five minutes.

PromQL Syntax Cheat Sheet

Most PromQL expressions are built from a small number of reusable building blocks.

Understanding these patterns makes it much easier to read complex queries.

Querying a Metric

Return every series for a metric.

node_memory_MemAvailable_bytes

Filter by Labels

Only return matching series.

http_requests_total{job="api"}

Multiple Labels

http_requests_total{
    job="api",
    method="GET"
}

Range Selector

Retrieve samples over a time window.

http_requests_total[5m]

Common ranges include:

• [1m]
• [5m]
• [15m]
• [30m]
• [1h]
• [6h]
• [24h]

Offset Modifier

Compare current metrics with historical values.

node_memory_MemAvailable_bytes offset 1h

Useful for:

• Historical comparisons
• Capacity analysis
• Trend investigation

Subqueries

Evaluate one query over another time range.

avg_over_time(
    rate(http_requests_total[5m])[1h:]
)

Subqueries are particularly useful when smoothing noisy metrics.

Aggregation

Group multiple time series.

sum(rate(http_requests_total[5m]))

Group by labels:

sum by(job)(
    rate(http_requests_total[5m])
)

Functions

Functions operate on vectors or ranges.

rate(http_requests_total[5m])

increase(http_requests_total[1h])

histogram_quantile(
    0.95,
    sum(rate(http_request_duration_seconds_bucket[5m]))
    by (le)
)

Understanding PromQL Query Types

Prometheus evaluates expressions differently depending on how they're executed.

Understanding query types helps prevent confusing results in dashboards and alerts.

Instant Queries

Instant queries evaluate a metric at a single point in time.

Example:

node_memory_MemAvailable_bytes

Common use cases:

• Alert rules
• Current server status
• Dashboard stat panels

Range Queries

Range queries return values across a time window.

Example:

rate(http_requests_total[5m])

Ideal for:

• Time-series graphs
• Historical analysis
• Trend visualization

Scalar Queries

Return a single numeric value.

Example:

scalar(up)

Useful when combining metrics with arithmetic operations.

String Queries

PromQL includes a string data type internally, although it's rarely used directly in operational queries.

Most production workloads primarily use vectors and scalars.

Prometheus Metric Types

Choosing the correct function depends entirely on the metric type.

Many incorrect dashboards happen because engineers use the wrong function for a metric.

Counter

A counter only increases.

Examples:

• HTTP requests
• API calls
• Login attempts
• Errors
• Packets transmitted

Examples:

http_requests_total

process_cpu_seconds_total

node_network_receive_bytes_total

Always use functions such as:

rate()

increase()

irate()

resets()

Avoid graphing raw counters directly.

Gauge

A gauge can increase or decrease.

Examples:

• Memory usage
• Temperature
• Queue depth
• CPU utilization
• Active connections

Examples:

node_memory_MemAvailable_bytes

node_load1

container_memory_usage_bytes

Gauges usually don't require rate().

Histogram

Histograms record distributions.

Typical examples:

• Request latency
• Response sizes
• Processing duration

Histograms expose three metric families:

http_request_duration_seconds_bucket

http_request_duration_seconds_sum

http_request_duration_seconds_count

Common functions:

histogram_quantile()

sum()

rate()

Summary

Summaries also measure distributions.

Unlike histograms, quantiles are calculated during metric collection rather than query execution.

Typical metrics include:

http_request_duration_seconds

Summaries cannot be meaningfully aggregated across multiple instances, making histograms the preferred choice for distributed systems.

Label Selectors Cheat Sheet

Labels make Prometheus incredibly powerful.

Every time series is uniquely identified by:

• Metric name
• Label set

Example:

http_requests_total{
    method="GET",
    status="200",
    instance="api-1"
}

Exact Match

http_requests_total{
    method="GET"
}

Not Equal

http_requests_total{
    method!="POST"
}

Regex Match

http_requests_total{
    job=~"api.*"
}

Matches:

• api
• api-v2
• api-prod

Negative Regex

http_requests_total{
    instance!~".*:9100"
}

Multiple Label Filters

http_requests_total{
    job="frontend",
    method="GET",
    status="200"
}

Match Empty Labels

my_metric{
    environment=""
}

Common Production Labels

You'll frequently encounter labels such as:

• job
• instance
• namespace
• pod
• container
• service
• endpoint
• method
• status
• node
• cluster
• region
• availability_zone

Understanding your label model is one of the most important skills in Prometheus. High-cardinality labels such as request IDs or user IDs should generally be avoided because they dramatically increase storage requirements and query latency.

PromQL Operators

PromQL supports several categories of operators that can be combined to create powerful expressions.

Arithmetic Operators

Use arithmetic operators to calculate percentages, ratios, and derived metrics.

+

-

*

/

%

^

Example:

(
node_memory_MemAvailable_bytes
/
node_memory_MemTotal_bytes
)
*100

Comparison Operators

Compare metric values.

Supported operators include:

==

!=

>

<

>=

<=

Example:

up == 0

Returns every target that is currently unreachable.

Logical Operators

Useful when combining vectors.

Supported operators:

and

or

unless

Example:

up == 0
and
node_load1 > 5

Aggregation Operators

Frequently used aggregations include:

• sum()
• avg()
• min()
• max()
• count()
• count_values()
• topk()
• bottomk()
• stddev()
• stdvar()
• quantile()
• group()

Grouping modifiers:

sum by(job)

sum without(instance)

Using by() preserves the specified labels, while without() removes them before aggregation. Choosing the correct modifier is essential for producing meaningful dashboards and avoiding duplicate or fragmented results.

Core Prometheus Query Examples

The following expressions are among the most useful day-to-day PromQL queries.

Check Target Health

up

Active Scrape Targets

up == 1

Down Targets

up == 0

HTTP Request Rate

rate(http_requests_total[5m])

Total Requests per Service

sum by(job)(
    rate(http_requests_total[5m])
)

Memory Utilization

100 *
(
1 -
(
node_memory_MemAvailable_bytes
/
node_memory_MemTotal_bytes
)
)

CPU Utilization

100 *
(
1 -
avg by(instance)(
rate(node_cpu_seconds_total{mode="idle"}[5m])
)
)

Unlike the raw node_cpu_seconds_total counter, this query calculates actual CPU utilization by subtracting idle CPU time from total CPU time over a rolling five-minute window.

Disk Usage Percentage

100 *
(
1 -
(
node_filesystem_avail_bytes{
fstype!~"tmpfs|overlay"
}
/
node_filesystem_size_bytes{
fstype!~"tmpfs|overlay"
}
)
)

Filtering pseudo-filesystems such as tmpfs and overlay helps produce more accurate capacity dashboards.

Network Receive Rate

rate(
node_network_receive_bytes_total[5m]
)

Error Rate Percentage

100 *
(
rate(http_requests_errors_total[5m])
/
rate(http_requests_total[5m])
)

This expression calculates the percentage of failed requests relative to all requests over the previous five minutes, making it suitable for dashboards and alert thresholds.

Working with Rate, Increase, Delta, Deriv, and Changes

Many Prometheus metrics are counters, meaning they only increase until the application restarts. Querying raw counter values rarely provides meaningful insight because the value continuously grows over time. Instead, PromQL provides functions that calculate rates, differences, and trends.

rate()

Calculates the average per-second increase of a counter across a time range.

rate(http_requests_total[5m])

Best for:

• Dashboard graphs
• Service throughput
• Request rates
• Long-term trends

Example:

sum by(job)(
    rate(http_requests_total[5m])
)

irate()

Uses only the two most recent samples to calculate an instantaneous rate.

irate(http_requests_total[1m])

Use irate() when monitoring rapidly changing metrics in dashboards.

Best Practice: Avoid using irate() in alerting rules because its sensitivity to short-term spikes can create noisy alerts.

increase()

Returns the total increase over a specified time period.

increase(http_requests_total[1h])

Example:

sum(
    increase(http_requests_total[24h])
)

Ideal for:

• Daily reports
• Billing metrics
• Request totals
• Business KPIs

delta()

Calculates the difference between the first and last sample in a range.

delta(node_temperature_celsius[30m])

Best used with gauges, not counters.

Examples include:

• Temperature
• Memory usage
• Queue depth
• Active sessions

deriv()

Calculates the derivative using linear regression.

deriv(
    node_memory_MemAvailable_bytes[30m]
)

Useful for:

• Trend detection
• Forecasting
• Capacity planning

changes()

Returns how many times a value changed.

changes(up[1h])

Useful for detecting:

• Service restarts
• State transitions
• Flapping targets

resets()

Counts counter resets.

resets(process_cpu_seconds_total[24h])

Useful for identifying:

• Pod restarts
• Process crashes
• Unexpected application resets

Choosing the Right Function

Use these guidelines when selecting a function:

• rate() – Smooth long-term trends for counters.
• irate() – Live dashboards with rapidly changing counters.
• increase() – Total increase across a time window.
• delta() – Absolute change for gauges.
• deriv() – Estimate trends using regression.
• changes() – Count value transitions.
• resets() – Detect counter resets after restarts.

Aggregation Functions

Aggregation combines multiple time series into meaningful summaries. Without aggregation, dashboards often become cluttered with one line per instance or container.

sum()

Calculate totals across multiple series.

sum(
    rate(http_requests_total[5m])
)

Group by service:

sum by(job)(
    rate(http_requests_total[5m])
)

avg()

Calculate averages.

avg by(instance)(
    node_load1
)

min() and max()

Find the smallest or largest values.

max(
    node_memory_MemAvailable_bytes
)

min(
    node_memory_MemAvailable_bytes
)

count()

Count the number of matching series.

count(up)

Running targets only:

count(up == 1)

count_values()

Count occurrences of each unique value.

count_values(
    "status",
    kube_pod_status_phase
)

topk()

Return the highest K values.

topk(
    5,
    rate(http_requests_total[5m])
)

Useful for identifying:

• Busiest APIs
• Highest CPU consumers
• Largest memory users

bottomk()

Return the lowest K values.

bottomk(
    5,
    node_memory_MemAvailable_bytes
)

quantile()

Calculate statistical percentiles.

quantile(
    0.95,
    node_load1
)

stddev() and stdvar()

Measure variation across series.

stddev(
    node_cpu_seconds_total
)

Helpful for spotting inconsistent performance across clusters.

Aggregation Modifiers

Keep specific labels:

sum by(namespace)(
    rate(container_cpu_usage_seconds_total[5m])
)

Drop labels before aggregation:

sum without(instance)(
    rate(http_requests_total[5m])
)

Use by() when you want to preserve dimensions such as job, namespace, or cluster. Use without() when instance-level differences are irrelevant.

Mathematical Functions

PromQL supports arithmetic between metrics, scalars, and vectors, making it possible to derive utilization percentages, ratios, and business metrics.

Memory Utilization

100 *
(
1 -
(
node_memory_MemAvailable_bytes
/
node_memory_MemTotal_bytes
)
)

Error Percentage

100 *
(
rate(http_requests_errors_total[5m])
/
rate(http_requests_total[5m])
)

Requests Per Minute

rate(http_requests_total[1m]) * 60

Disk Utilization

100 *
(
1 -
(
node_filesystem_avail_bytes
/
node_filesystem_size_bytes
)
)

Common Math Functions

Round values:

round(node_load1)

Round to two decimals:

round(node_load1, 0.01)

Square root:

sqrt(node_load1)

Natural logarithm:

ln(node_network_receive_bytes_total)

Base-10 logarithm:

log10(node_network_receive_bytes_total)

Clamp values to a maximum:

clamp_max(node_load1, 10)

Clamp values to a minimum:

clamp_min(node_load1, 0)

Clamp within a range:

clamp(node_load1, 0, 10)

These functions are especially useful when normalizing values before visualizing them in Grafana.

Time Functions

PromQL includes functions that work with timestamps and calendar values.

Current Evaluation Time

time()

Returns the current Unix timestamp.

Sample Timestamp

timestamp(
    node_load1
)

Useful for identifying stale metrics.

Hour of Day

hour()

Day of Week

day_of_week()

Day of Month

day_of_month()

Month

month()

Year

year()

Time functions are commonly used for:

• Business-hour alerts
• Maintenance windows
• Reporting
• Scheduled dashboards

Label Manipulation Functions

Label manipulation allows you to reshape metrics without changing the exporter.

These functions are especially useful when integrating multiple systems with different labeling conventions.

label_replace()

Create or modify labels using regular expressions.

label_replace(
    up,
    "host",
    "$1",
    "instance",
    "(.*):.*"
)

This extracts the hostname from the instance label.

label_join()

Combine multiple labels into one.

label_join(
    kube_pod_info,
    "pod_ref",
    "/",
    "namespace",
    "pod"
)

Result:

production/frontend-5f6cb

This is useful for dashboards that require concise resource identifiers.

Histogram & Native Histogram Queries

Histograms are the recommended metric type for measuring request duration and other latency distributions in distributed systems.

A histogram exposes three metric families:

*_bucket
*_count
*_sum

Example:

http_request_duration_seconds_bucket

http_request_duration_seconds_sum

http_request_duration_seconds_count

95th Percentile Latency

histogram_quantile(
    0.95,
    sum by(le)(
        rate(
            http_request_duration_seconds_bucket[5m]
        )
    )
)

99th Percentile

histogram_quantile(
    0.99,
    sum by(le)(
        rate(
            http_request_duration_seconds_bucket[5m]
        )
    )
)

Average Request Duration

rate(
http_request_duration_seconds_sum[5m]
)
/
rate(
http_request_duration_seconds_count[5m]
)

Requests Per Bucket

sum by(le)(
    rate(
        http_request_duration_seconds_bucket[5m]
    )
)

Native Histograms

Modern Prometheus releases support Native Histograms, reducing storage overhead while providing higher-fidelity latency distributions.

Benefits include:

• Lower cardinality
• More accurate percentile calculations
• Better storage efficiency
• Simpler instrumentation
• Improved query performance

When native histograms are enabled in your environment, prefer them over classic bucket-based histograms where supported by your exporters and visualization tools.

Vector Matching

Binary operations between vectors require matching label sets. When labels differ, Prometheus needs explicit instructions on how to align series.

Default Matching

Prometheus matches identical labels automatically.

metric_a / metric_b

on()

Match only specific labels.

rate(http_requests_total[5m])
/
on(job)
rate(http_requests_errors_total[5m])

ignoring()

Ignore selected labels during matching.

rate(requests_total[5m])
/
ignoring(instance)
rate(errors_total[5m])

This is useful when metrics differ only by labels such as instance or pod.

group_left()

Allow one-to-many matching.

container_cpu_usage_seconds_total
*
on(namespace,pod)
group_left(node)
kube_pod_info

Common use cases:

• Add node metadata
• Enrich container metrics
• Join Kubernetes labels

group_right()

Allow many-to-one matching.

metric_a
/
on(job)
group_right()
metric_b

Use with caution, as incorrect joins can produce unexpected results.

Subqueries & Offset

Subqueries enable multi-stage analysis by evaluating one query across another time range.

Basic Subquery

rate(
    http_requests_total[5m]
)[1h:]

Average Request Rate Over One Hour

avg_over_time(
    rate(http_requests_total[5m])[1h:]
)

This smooths short-term fluctuations while preserving overall trends.

Compare Current Metrics with One Hour Ago

node_memory_MemAvailable_bytes
-
node_memory_MemAvailable_bytes offset 1h

Compare CPU Usage Yesterday

100 *
(
1 -
avg by(instance)(
rate(
node_cpu_seconds_total{mode="idle"}[5m]
offset 24h
)
)
)

Forecast Future Growth

predict_linear(
    node_filesystem_free_bytes[6h],
    86400
)

This estimates available disk space 24 hours into the future, making it useful for proactive capacity planning.

Engineering Tip: Subqueries and offset are powerful analytical tools, but they increase query complexity. For dashboards that execute frequently, consider using recording rules to precompute expensive expressions and improve query performance.

Node Exporter PromQL Cheat Sheet

Node Exporter exposes Linux and Windows host metrics that are commonly used for infrastructure dashboards and alerts. The following queries assume the standard Node Exporter metric names.

CPU Utilization

100 *
(
1 -
avg by(instance)(
rate(node_cpu_seconds_total{mode="idle"}[5m])
)
)

CPU Load Average

node_load1

Five-minute load:

node_load5

Fifteen-minute load:

node_load15

Memory Usage Percentage

100 *
(
1 -
(
node_memory_MemAvailable_bytes
/
node_memory_MemTotal_bytes
)
)

Memory Available

node_memory_MemAvailable_bytes

Disk Usage Percentage

100 *
(
1 -
(
node_filesystem_avail_bytes{
fstype!~"tmpfs|overlay|squashfs"
}
/
node_filesystem_size_bytes{
fstype!~"tmpfs|overlay|squashfs"
}
)
)

Disk Read Rate

rate(
node_disk_read_bytes_total[5m]
)

Disk Write Rate

rate(
node_disk_written_bytes_total[5m]
)

Network Receive Throughput

rate(
node_network_receive_bytes_total[5m]
)

Network Transmit Throughput

rate(
node_network_transmit_bytes_total[5m]
)

Filesystem Free Space

node_filesystem_avail_bytes

Node Uptime

time() - node_boot_time_seconds

Detect Down Nodes

up{job="node"} == 0

These queries form the foundation of most infrastructure dashboards and capacity planning workflows.

Kubernetes PromQL Cheat Sheet

Prometheus is deeply integrated with Kubernetes through exporters such as kube-state-metrics, cAdvisor, and the kubelet metrics endpoint.

Pod CPU Usage

sum by(namespace,pod)(
rate(
container_cpu_usage_seconds_total{
image!=""
}[5m]
)
)

Pod Memory Usage

sum by(namespace,pod)(
container_memory_working_set_bytes
)

Container Restart Count

sum by(namespace,pod)(
increase(
kube_pod_container_status_restarts_total[1h]
)
)

Running Pods

count(
kube_pod_info
)

Pending Pods

count(
kube_pod_status_phase{
phase="Pending"
}
)

Failed Pods

count(
kube_pod_status_phase{
phase="Failed"
}
)

Node Ready Status

kube_node_status_condition{
condition="Ready",
status="true"
}

Node Memory Utilization

100 *
(
1 -
(
node_memory_MemAvailable_bytes
/
node_memory_MemTotal_bytes
)
)

PVC Usage

100 *
(
kubelet_volume_stats_used_bytes
/
kubelet_volume_stats_capacity_bytes
)

Deployment Replicas

kube_deployment_status_replicas_available

Detect CrashLoopBackOff Pods

kube_pod_container_status_waiting_reason{
reason="CrashLoopBackOff"
}

These queries are suitable for cluster health dashboards, SRE runbooks, and production alerting.

Grafana PromQL Examples

Grafana uses PromQL directly as its query language when Prometheus is configured as a data source.

A few production-ready dashboard queries include:

Requests Per Second

sum by(job)(
rate(http_requests_total[$__rate_interval])
)

Error Rate

100 *
(
sum(rate(http_requests_errors_total[$__rate_interval]))
/
sum(rate(http_requests_total[$__rate_interval]))
)

Top 10 Busy Services

topk(
10,
sum by(job)(
rate(http_requests_total[5m])
)
)

Slowest APIs

histogram_quantile(
0.95,
sum by(le,handler)(
rate(http_request_duration_seconds_bucket[$__rate_interval])
)
)

Dashboard Variables

Grafana variables frequently appear in PromQL:

$cluster

$namespace

$pod

$instance

$job

Example:

sum by(pod)(
rate(
container_cpu_usage_seconds_total{
namespace="$namespace"
}[5m]
)
)

Using dashboard variables makes panels reusable across multiple environments without rewriting queries.

Alerting Rules with PromQL

PromQL expressions power Prometheus alerting rules, which are evaluated continuously and forwarded to Alertmanager when conditions are met.

High CPU Usage

groups:
- name: infrastructure
  rules:
  - alert: HighCPUUsage
    expr: |
      100 *
      (
        1 -
        avg by(instance)(
          rate(node_cpu_seconds_total{mode="idle"}[5m])
        )
      ) > 90
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: High CPU usage
      description: CPU utilization has exceeded 90%.

Node Down

- alert: NodeDown
  expr: up == 0
  for: 2m

High Error Rate

- alert: HighErrorRate
  expr: |
    (
      rate(http_requests_errors_total[5m])
      /
      rate(http_requests_total[5m])
    ) > 0.05
  for: 5m

Low Disk Space

- alert: DiskSpaceLow
  expr: |
    (
      node_filesystem_avail_bytes
      /
      node_filesystem_size_bytes
    ) < 0.10

When writing alerts, prefer stable functions such as rate() instead of irate() to minimize false positives caused by short-lived spikes.

Recording Rules

Recording rules precompute expensive PromQL expressions and store the results as new metrics.

Benefits include:

• Faster dashboards
• Lower query latency
• Reduced Prometheus CPU usage
• Simpler Grafana panels
• Reusable metrics for alerts

Example:

groups:
- name: recording
  rules:
  - record: job:http_requests:rate5m
    expr: |
      sum by(job)(
        rate(http_requests_total[5m])
      )

Once evaluated, the new metric can be queried directly:

job:http_requests:rate5m

For frequently used expressions, recording rules are generally preferable to recalculating the same query across every dashboard panel.

PromQL Performance Optimization

Well-written PromQL scales effectively even in large environments, while inefficient queries can significantly increase latency and resource consumption.

Best Practices

• Use recording rules for expensive calculations.
• Keep range selectors as short as practical.
• Avoid unnecessary regular expressions.
• Minimize high-cardinality labels.
• Filter metrics before aggregation.
• Aggregate as early as possible.
• Prefer sum by() over large client-side aggregations.
• Limit expensive joins with group_left() and group_right().
• Remove unused labels at scrape time whenever possible.
• Review query execution time in the Prometheus expression browser.

A small optimization applied to frequently executed dashboard queries can substantially reduce Prometheus resource usage.

Common PromQL Mistakes

Even experienced engineers occasionally make these mistakes.

Using Raw Counters

❌

http_requests_total

✅

rate(http_requests_total[5m])

Using irate() for Alerts

irate() is designed for responsive dashboards, not alerting. Use rate() for stable alert conditions.

Ignoring Label Cardinality

Avoid labels containing:

• Request IDs
• Session IDs
• UUIDs
• User IDs
• Timestamps

These labels create excessive numbers of time series and increase storage costs.

Dividing Mismatched Vectors

Always verify label alignment.

If necessary, use:

• on()
• ignoring()
• group_left()
• group_right()

Graphing Gauges with rate()

Only counters should typically use rate().

Applying it to gauges usually produces misleading results.

Troubleshooting PromQL Queries

When a query produces unexpected results, work through the following checklist.

No Data Returned

Check:

• Target is being scraped.
• Metric name is correct.
• Labels match.
• Time range includes samples.

Empty Graphs

Verify:

up

If up returns 0, Prometheus cannot reach the exporter.

Unexpected Duplicate Series

Inspect labels:

label_replace()

label_join()

Or aggregate using:

sum by(...)

Slow Queries

Review:

• Regex selectors
• Large range vectors
• High-cardinality labels
• Complex joins
• Missing recording rules

Counter Appears to Decrease

Counters reset after process or container restarts.

Use:

rate()

increase()

resets()

rather than plotting the raw counter.

PromQL Function Reference

The following functions cover the majority of production PromQL use cases.

Rate & Counter Functions

• rate()
• irate()
• increase()
• delta()
• idelta()
• deriv()
• changes()
• resets()
• predict_linear()

Aggregation Functions

• sum()
• avg()
• min()
• max()
• count()
• count_values()
• group()
• topk()
• bottomk()
• quantile()
• stddev()
• stdvar()

Over-Time Functions

• avg_over_time()
• sum_over_time()
• min_over_time()
• max_over_time()
• count_over_time()
• last_over_time()
• present_over_time()
• quantile_over_time()

Mathematical Functions

• abs()
• absent()
• absent_over_time()
• ceil()
• floor()
• round()
• sqrt()
• exp()
• ln()
• log2()
• log10()
• clamp()
• clamp_min()
• clamp_max()
• scalar()
• vector()

Histogram Functions

• histogram_quantile()
• histogram_avg()
• histogram_sum()
• histogram_count()
• histogram_fraction()

Label Functions

• label_replace()
• label_join()

Time Functions

• time()
• timestamp()
• hour()
• day_of_week()
• day_of_month()
• day_of_year()
• month()
• year()

Together, these functions cover the overwhelming majority of day-to-day PromQL querying scenarios encountered in production environments.

Frequently Asked Questions

What is PromQL used for?

PromQL is the query language for Prometheus. It retrieves, transforms, and analyzes time-series metrics used in dashboards, alerting, troubleshooting, and capacity planning.

What is the difference between rate() and increase()?

• rate() returns the average per-second increase over a range and is ideal for graphs and alerts.
• increase() returns the total increase during the selected period and is useful for reports and cumulative totals.

Should I use irate() or rate()?

Use:

• rate() for dashboards, alerts, and long-term trends.
• irate() for highly responsive visualizations where recent changes are more important than smoothing.

Why isn't my PromQL query returning data?

Common causes include:

• Incorrect metric names.
• Missing or mismatched labels.
• Exporters not being scraped.
• Time ranges that exclude recent samples.
• Incorrect vector matching between metrics.

Can I use PromQL in Grafana?

Yes. Grafana supports Prometheus as a native data source, allowing you to use PromQL directly in panels, dashboard variables, alerts, and Explore mode.

How can I improve PromQL query performance?

Focus on reducing query complexity by using recording rules, minimizing high-cardinality labels, filtering early, limiting expensive joins, and selecting the shortest practical range vectors.

Conclusion

PromQL is much more than a query language—it is the analytical engine behind modern observability platforms. Mastering its syntax, functions, and data model enables you to move beyond basic monitoring and build meaningful dashboards, reliable alerts, and actionable operational insights.

Whether you're monitoring virtual machines, Kubernetes clusters, cloud-native applications, or large-scale microservices, understanding concepts such as counters, gauges, histograms, vector matching, and recording rules will help you write more accurate and efficient queries.

As your Prometheus deployment grows, prioritize reusable recording rules, keep label cardinality under control, and regularly review query performance. These practices improve scalability while making dashboards and alerts easier to maintain.

Keep this cheat sheet handy as a daily reference, experiment with queries in the Prometheus expression browser or Grafana Explore, and gradually build a library of production-tested PromQL expressions tailored to your own infrastructure.

🛠️ Free Engineering Tools

Working with Prometheus alert rules, Kubernetes manifests, or monitoring automation?

• Validate alert rule files, recording rules, and Kubernetes YAML using our YAML Formatter.
• Build and verify Alertmanager or recording schedules with our Cron Builder, which converts cron expressions into plain English before deployment.
• Explore additional browser-based engineering utilities for DevOps, SRE, Kubernetes, and cloud infrastructure workflows—all designed to simplify day-to-day operations without requiring local installations.