Cluster Carbon — Methodology

Overview

This dashboard estimates the carbon footprint of large language model (LLM) inference running on the lab's Kubernetes cluster (boettiger-lab/k8s) — one card per served model, whether it is running now or not. A model is identified by its vLLM served name and the node whose API server exports its metrics (qwen@nimbus, deepseek-v4-flash@nimbus2); a node that switches models gets a card for each, and a model scaled to zero keeps its card, with its record, for as long as Prometheus retains its data. All measurements are derived passively from existing telemetry — no instrumentation of the LLM service is required.

Step 1 — Measuring GPU Power

GPU power draw is read from NVIDIA DCGM Exporter (Data Center GPU Manager), which runs on every GPU node in the cluster. DCGM reads the hardware power sensor via NVML (nvmlDeviceGetPowerUsage) and exports it to Prometheus as:

DCGM_FI_DEV_POWER_USAGE{Hostname, namespace, pod, ...}  # watts, one series per GPU

Power is summed per node, and then attributed to whichever model is serving on that node at that moment. A model is "serving" while its vLLM server is exporting metrics (its vllm:num_requests_running gauge exists):

-- node GPU power, with a tensor-parallel worker's watts relabelled to its head
sum by (node) (
  label_replace(
    label_replace(DCGM_FI_DEV_POWER_USAGE{Hostname=~"cirrus|nimbus|nimbus2|nimbus4|nimbus3"},
                  "node", "$1", "Hostname", "(.*)"),
    "node", "nimbus2", "Hostname", "nimbus4")
)
-- ... joined onto the model serving there
  * on (node) group_right ()
(max by (node, namespace, model_name) (vllm:num_requests_running{...}) * 0 + 1)

The live figure wraps DCGM in avg_over_time(...[5m]) to smooth short spikes. Measurements are scraped every 30 seconds.

Step 2 — Token Throughput (Prompt + Generation)

Token throughput is read from vLLM's built-in Prometheus metrics. Two counters are tracked separately and summed into a total token rate:

-- Output (decode) tokens generated per second (2-minute window --
-- short enough to track our bursty single-user traffic without
-- diluting a burst across mostly-idle time on either side of it):
sum by (node, namespace, model_name) (
  rate(vllm:generation_tokens_total{namespace=~"vllm",node=~"cirrus|nimbus|nimbus2|nimbus3"}[2m])
)

-- Input (prefill) tokens processed per second:
sum by (node, namespace, model_name) (
  rate(vllm:prompt_tokens_total{namespace=~"vllm",node=~"cirrus|nimbus|nimbus2|nimbus3"}[2m])
)

Both phases consume GPU energy: the prefill phase runs attention over every input token before the first output token is produced; the decode phase generates one output token at a time. Counting only output tokens would therefore misrepresent the true cost of inference — especially for agentic workloads where a single user-visible response may involve tens of thousands of input tokens across multiple hidden tool calls.

The CO₂ per token metric is only reported when total throughput (input + output) is at least 5 tok/s. Below this threshold the ratio is dominated by near-idle power draw rather than the model's efficiency under real load.

Live and aggregate views

A single toggle switches every card between two records:

• Live — the model's current state: 5-minute power, 2-minute token rates, 15-minute request and latency figures, instant engine gauges. A model that is not running shows its aggregate for the selected window instead, labelled as such with when it was last seen.
• Aggregate — the model's record over the last 24 h, 7 d or 15 d, recomputed from Prometheus every five minutes. 15 days is the cluster's Prometheus retention, so it is the longest window offered; a model with no data in the chosen window falls back to the longest window that has some.

How aggregates are computed

Every aggregate is a ratio of totals over the window — never an average of ratios — so a few minutes at 5 tok/s cannot outweigh hours at 3,000 tok/s. Energy integrates the attributed power above at one-minute resolution:

energy_J       = sum_over_time((attributed_power)[W:1m]) × 60
serving_s      = count_over_time((serving)[W:1m]) × 60
tokens         = increase(vllm:prompt_tokens_total[W]) + increase(vllm:generation_tokens_total[W])
mean power     = energy_J / serving_s
CO₂/token      = energy_J / 3.6e6 × intensity / tokens          (all-in)
CO₂/token work = energy_J_while_>5_tok/s / 3.6e6 × intensity / tokens   (marginal)
decode tok/s   = increase(generation_tokens) / increase(inter_token_latency_seconds_sum)
uptime         = serving_s / (window, clipped to retention)

All-in CO₂/token counts every joule the model burned while loaded, idle hosting included — what the model actually cost. Working CO₂/token counts only minutes above 5 tok/s: the marginal cost of a token when the model is busy. Both use the same token denominator. Nothing is held in memory, so a restart loses nothing and needs no backfill.

Step 3 — Carbon Intensity

Every node is in one place: Berkeley, CA. Grid carbon intensity is therefore a fixed constant rather than a lookup table — the EPA eGRID 2022 average for the CAMX subregion (California):

Step 4 — Carbon Calculations

Three derived metrics are computed:

CO₂ grams per hour

CO₂_g/hr = P_watts × intensity_kg/kWh
         = P_watts × intensity   # because W × kg/kWh = W × kg/(1000W·h) × 1000 = g/h

CO₂ milligrams per token (total tokens)

total_tokens_per_sec = prompt_tokens_per_sec + generation_tokens_per_sec

CO₂_mg/token = P_watts × intensity_kg/kWh × (1e6 mg/kg) / (3.6e6 J/kWh) / total_tokens_per_sec
             = P_watts × intensity × 0.2778 / total_tokens_per_sec

The denominator is total tokens processed (prompt + generation), not output tokens alone. This is the correct normalization: the GPU expends energy on every token it touches — both the input context during prefill and each generated token during decode.

Commercial frontier comparison — per-model physics estimate

Rather than comparing against a fixed hypothetical workload, the dashboard computes a per-model, apples-to-apples power comparison: for each of our models' actual observed token throughput (prompt and generation rates), what total power would a commercial frontier model draw to serve the same tokens at the same rate?

Commercial frontier model specification

We model a commercial frontier LLM as a ~1.5 T parameter Mixture-of-Experts (MoE) with ~300 B active parameters per token, served in FP8 on H100-80GB GPUs. This is an informed estimate, not a known architecture — commercial providers (OpenAI, Anthropic, Google) do not publicly disclose hardware configurations or per-query power measurements. The closest available ground truth is Google's measured production figure of 0.24 Wh per median Gemini prompt (arXiv:2508.15734, August 2025), covering accelerators, CPU, DRAM, and datacenter overhead. An independent bottom-up analysis estimates a median of 0.34 Wh per query (IQR: 0.18–0.67 Wh) for models exceeding 200 B parameters on H100 hardware (arXiv:2509.20241). Our physics-based idle floor of ~5,400 W for 24× H100 GPUs is consistent with these per-query figures at realistic throughput: at 50 tok/s total and 0.34 Wh/query (~700 tokens), the implied power draw is ~87 W above the idle floor, broadly consistent with our marginal energy estimates below.

Marginal token energy (above idle)

Per-model comparison formula

frontier_watts = 24 × 225                                   # idle floor: 5,400 W
               + prompt_tokens_per_sec × 0.5                 # marginal prefill
               + generation_tokens_per_sec × 6.0             # marginal decode

ratio = frontier_watts / nrp_measured_watts

This formula is applied to each of our models using its actual observed prompt and generation token rates while working (tokens over working time in the selected window), against its measured working power — see where the comparison appears.

Example: nimbus at typical light traffic

nimbus measured: 11 W  (1× GB10, real DCGM reading)
Prompt rate:      40 tok/s
Generation rate:   1 tok/s

Commercial frontier equivalent:
  Idle floor:    24 × 225 = 5,400 W
  Prefill:       40 × 0.5 =    20 W
  Decode:         1 × 6.0 =     6 W
  Total:                     5,426 W

Ratio: 5,426 / 11 = 493× — nimbus uses 493× less energy for the same tokens.

Each card's aggregate view also shows tokens in and out and the mean tokens per request, for transparency about the workload driving the comparison.

Where the comparison appears

In the aggregate view, a card whose model did any work in the window shows one line — "~N× less energy than a frontier-scale deployment (estimate)" — computed as frontier_watts / measured_working_watts, with the frontier side fed the model's own working-time prompt and generation rates. It is informational, not a verdict, and is not colored. No per-model number on the page is colored against the others either: the models are different sizes on different hardware, and ranking the GB10s against a pair of RTX 8000s would present that as a judgment.

Energy per token (J/token)

J/token (joules per token) is a hardware- and grid-neutral measure of energy spent per token over the selected window, all-in like CO₂/token:

J_per_token = energy_J / (prompt_tokens + generation_tokens)

Unlike CO₂/token, this metric is independent of grid carbon intensity, making it directly comparable across models regardless of where they are hosted. J/token reflects batch size and workload as much as it reflects GPU health — it is not a "utilization" verdict. A personal, single-user box naturally runs higher J/token than a heavily-batched datacenter server, because personal single-stream inference is latency-bound, not throughput-bound: there's no queue of other requests to batch alongside, and there shouldn't be. Key drivers of J/token include:

• Batch size: more concurrent requests amortize the same idle power over more tokens — a difference in workload, not a problem to fix on a personal box
• Traffic level: idle power is a larger share of the total when fewer tokens are being processed in a given window
• Model architecture: Dense models activate all parameters on every token, while Mixture-of-Experts (MoE) models route each token through a subset (~20%) of total weights. A dense 31B model may therefore have higher J/token than a 397B MoE, because the MoE only activates ~60B parameters per token

Live activity metrics

Beside the carbon figures, each card reports the model's engine state and service quality, read directly from vLLM and DCGM. Engine gauges are absolute readings — a zero is as meaningful as any other value — so they are not gated on throughput. Latency percentiles are gated on request count (below):

Cumulative CO₂ emissions (time-series view)

The backend's timeseries endpoint queries the Prometheus range API at adaptive resolution (5-minute steps for 24 h, hourly for 7 d, 2-hourly for 15 d) and returns co2_grams_per_hour at each step (computed server-side as attributed_power_watts × 0.198, our fixed grid intensity — so only GPU time spent serving an LLM is counted). The dashboard then integrates that rate client-side using the rectangle rule, running-summed over the selected window:

total_g = Σ (co2_grams_per_hour_i × step_hours)

This total only grows — it never resets within a window — so the chart climbs steadily even while the cluster is mostly idle, rather than looking flat or empty the way a tiny, nearly-constant rate does.

Limitations and Future Work

• CPU, DRAM, and network power are excluded (typically adds 15–30% to total)
• Datacenter PUE (Power Usage Effectiveness) is not applied (adds 10–50%)
• Carbon intensity values are annual averages; real-time grid mix varies hour-to-hour
• DCGM reports a single power reading per GPU; we cannot decompose energy into prefill vs decode phases on our side
• The commercial frontier model spec is an informed estimate, not a known architecture. No empirical per-token energy measurement exists for GPT-4, Claude 3/3.5/3.7, or Gemini 1.5/2. The closest empirical anchor is Google's production measurement of 0.24 Wh/prompt for Gemini Apps (arXiv:2508.15734). Our physics-based estimate is broadly consistent with this and with independent bottom-up estimates (0.34 Wh median, arXiv:2509.20241)
• Decode marginal energy (6 J/token) assumes low batch matching our typical traffic; commercial providers batching hundreds of concurrent requests achieve lower per-token decode costs, but also require the same or greater hosting floor
• Agentic energy amplification is highly framework-dependent: measured research-grade frameworks show 62–137× energy vs single-turn (arXiv:2506.04301); production systems are likely lower but remain poorly characterized in the literature
• The eGRID intensity table in internal/carbon/intensity.go will be updated as new annual data is published
• History is bounded by Prometheus retention (15 days). A model unseen for longer drops off the dashboard
• A model is identified by its served name and head node. If the same name is reused for a different checkpoint on the same node, their records merge
• Energy and uptime integrate at one-minute resolution, so a model that serves for less than a minute is not seen

For a more complete lifecycle analysis including training, hardware manufacturing, and cooling overhead, see the references below.

References

Everyday Reference Frames

Absolute CO₂ figures for a single, efficient GB10 node are small (grams, not kilograms) and don't mean much in isolation. The dashboard converts the cumulative total for the selected window (see "Cumulative CO₂ emissions" above) into a few widely-cited everyday equivalents, in the style of CodeCarbon:

Source Code

This dashboard is open source: github.com/boettiger-lab/nimbus-carbon-api, forked from nrp-carbon-api. The fixed carbon intensity constant is in internal/carbon/intensity.go.