nimbus Carbon — Methodology

Overview

This dashboard estimates the carbon footprint of large language model (LLM) inference running on nimbus, a single GB10 DGX Spark node (boettiger-lab/k8s/vllm/nimbus). All measurements are derived passively from existing telemetry — no instrumentation of the LLM service is required.

Step 1 — Measuring GPU Power

GPU power draw is read from NVIDIA DCGM Exporter (Data Center GPU Manager), which runs on nimbus's single GPU (a GB10). DCGM reads the hardware power sensor via NVML (nvmlDeviceGetPowerUsage) and exports it to Prometheus as:

DCGM_FI_DEV_POWER_USAGE{namespace, container, pod, ...}  # watts, one GPU

nimbus has exactly one physical GPU, so no summing across GPUs is needed — just a namespace filter:

sum by (namespace) (
  avg_over_time(DCGM_FI_DEV_POWER_USAGE{namespace="default"}[5m])
)

The avg_over_time(...[5m]) wrapper smooths short-lived spikes into a 5-minute rolling average. This gives total GPU power in watts for each deployed model. Measurements are scraped every 30 seconds.

Step 2 — Token Throughput (Prompt + Generation)

Token throughput is read from vLLM's built-in Prometheus metrics. Two counters are tracked separately and summed into a total token rate:

-- Output (decode) tokens generated per second (2-minute window --
-- short enough to track nimbus's bursty single-user traffic without
-- diluting a burst across mostly-idle time on either side of it):
sum by (namespace, model_name) (
  rate(vllm:generation_tokens_total{namespace="default"}[2m])
)

-- Input (prefill) tokens processed per second:
sum by (namespace, model_name) (
  rate(vllm:prompt_tokens_total{namespace="default"}[2m])
)

Both phases consume GPU energy: the prefill phase runs attention over every input token before the first output token is produced; the decode phase generates one output token at a time. Counting only output tokens would therefore misrepresent the true cost of inference — especially for agentic workloads where a single user-visible response may involve tens of thousands of input tokens across multiple hidden tool calls.

The CO₂ per token metric is only reported when total throughput (input + output) is at least 5 tok/s. Below this threshold the ratio is dominated by near-idle power draw rather than the model's efficiency under real load. The dashboard cards break down throughput into Output tok/s, Input tok/s, and Total tok/s for transparency.

Token-weighted averaging

The 24-hour and 7-day CO₂/token averages shown on each model card use a token-weighted mean rather than an unweighted arithmetic mean. Because CO₂/token is a ratio (power ÷ throughput), samples taken during low-throughput periods produce extreme values — a model drawing 2,000 W at 5 tok/s yields 110 mg/token, versus 0.03 mg at 3,000 tok/s. An unweighted mean gives equal influence to both, inflating the average by orders of magnitude.

The weighted formula:

avg_CO₂/token = Σ(CO₂_per_token_i × tokens_per_sec_i) / Σ(tokens_per_sec_i)

This is equivalent to computing total CO₂ emitted divided by total tokens produced, which is the physically meaningful quantity. A 30-second sample at 5 tok/s (representing ~150 tokens) contributes proportionally less than a sample at 3,000 tok/s (representing ~90,000 tokens). Models under sustained heavy load naturally converge to their true operating efficiency.

Internally, samples are aggregated into hourly buckets (168 buckets for 7 days), each storing the weighted sum and token sum for that hour. On startup, the scraper backfills these buckets from 7 days of historical Prometheus data, so the 24-hour and 7-day averages are immediately correct after a service restart rather than starting from zero.

Step 3 — Carbon Intensity

nimbus is one machine in one place: Berkeley, CA. Grid carbon intensity is therefore a fixed constant rather than a lookup table — the EPA eGRID 2022 average for the CAMX subregion (California):

Step 4 — Carbon Calculations

Three derived metrics are computed:

CO₂ grams per hour

CO₂_g/hr = P_watts × intensity_kg/kWh
         = P_watts × intensity   # because W × kg/kWh = W × kg/(1000W·h) × 1000 = g/h

CO₂ milligrams per token (total tokens)

total_tokens_per_sec = prompt_tokens_per_sec + generation_tokens_per_sec

CO₂_mg/token = P_watts × intensity_kg/kWh × (1e6 mg/kg) / (3.6e6 J/kWh) / total_tokens_per_sec
             = P_watts × intensity × 0.2778 / total_tokens_per_sec

The denominator is total tokens processed (prompt + generation), not output tokens alone. This is the correct normalization: the GPU expends energy on every token it touches — both the input context during prefill and each generated token during decode.

Commercial frontier comparison — per-model physics estimate

Rather than comparing against a fixed hypothetical workload, the dashboard computes a per-model, apples-to-apples power comparison: for each nimbus model's actual observed token throughput (prompt and generation rates), what total power would a commercial frontier model draw to serve the same tokens at the same rate?

Commercial frontier model specification

We model a commercial frontier LLM as a ~1.5 T parameter Mixture-of-Experts (MoE) with ~300 B active parameters per token, served in FP8 on H100-80GB GPUs. This is an informed estimate, not a known architecture — commercial providers (OpenAI, Anthropic, Google) do not publicly disclose hardware configurations or per-query power measurements. The closest available ground truth is Google's measured production figure of 0.24 Wh per median Gemini prompt (arXiv:2508.15734, August 2025), covering accelerators, CPU, DRAM, and datacenter overhead. An independent bottom-up analysis estimates a median of 0.34 Wh per query (IQR: 0.18–0.67 Wh) for models exceeding 200 B parameters on H100 hardware (arXiv:2509.20241). Our physics-based idle floor of ~5,400 W for 24× H100 GPUs is consistent with these per-query figures at realistic throughput: at 50 tok/s total and 0.34 Wh/query (~700 tokens), the implied power draw is ~87 W above the idle floor, broadly consistent with our marginal energy estimates below.

Marginal token energy (above idle)

Per-model comparison formula

frontier_watts = 24 × 225                                   # idle floor: 5,400 W
               + prompt_tokens_per_sec × 0.5                 # marginal prefill
               + generation_tokens_per_sec × 6.0             # marginal decode

ratio = frontier_watts / nrp_measured_watts

This formula is applied to each nimbus model using its actual observed prompt and generation token rates. The comparison is shown as a plain-text watts bar on each model card, visible only while the model is actively generating (24h-avg combined tok/s above a small threshold) — not as a colored verdict, and separate from the CO₂/token and J/token card tiles, which are always-rendered absolute measurements.

Example: nimbus at typical light traffic

nimbus measured: 11 W  (1× GB10, real DCGM reading)
Prompt rate:      40 tok/s
Generation rate:   1 tok/s

Commercial frontier equivalent:
  Idle floor:    24 × 225 = 5,400 W
  Prefill:       40 × 0.5 =    20 W
  Decode:         1 × 6.0 =     6 W
  Total:                     5,426 W

Ratio: 5,426 / 11 = 493× — nimbus uses 493× less energy for the same tokens.

Each card also displays the observed prompt token percentage — what fraction of total tokens are input (prefill) vs output (decode), for transparency about the workload driving the comparison.

Dashboard visualizations

The dashboard presents the frontier comparison in two ways, both shown only while the model is actively generating (24h-avg combined tok/s > 1) — while idle, neither is shown, since there is nothing to compare and idling low is expected behavior for this hardware, not something to judge either way:

• Model card "N× less energy" metric: Shown only while actively generating. Each model card shows how many times more power the commercial frontier would require to serve the same tokens (e.g. "7.0× less energy"). This is frontier_watts / nimbus_measured_watts. There is no color judgment on this number — it's informational, not a verdict.
• Power comparison bar (per card): Shown only while actively generating. A horizontal bar comparing nimbus measured watts against the frontier equivalent (purple band), with a breakdown of idle floor vs. marginal power.

CO₂/token and J/token are both plain per-model card stats, not chart bars — a number (with a 24h average alongside where available), or — when there isn't enough throughput to make the ratio meaningful. Neither is colored by a comparative scale: nimbus almost always runs exactly one active model at a time (it's a single-GPU personal box, not a multi-tenant cluster), so any chart comparing bars against each other would only ever have one bar — and coloring a lone bar by its own maximum always renders it at the most "extreme" color, regardless of the actual value. A comparison visualization's entire reason to exist is comparing multiple things; with one thing, it can only produce a misleading result.

Energy per token (J/token)

J/token (joules per token) is a hardware- and grid-neutral measure of energy spent per token, computed from 24-hour averages (the same window CO₂/token uses):

J_per_token = power_watts_avg_24h / (prompt_tokens_per_sec_avg_24h + generation_tokens_per_sec_avg_24h)

Unlike CO₂/token, this metric is independent of grid carbon intensity, making it directly comparable across models regardless of where they are hosted. J/token reflects batch size and workload as much as it reflects GPU health — it is not a "utilization" verdict. A personal, single-user box naturally runs higher J/token than a heavily-batched datacenter server, because personal single-stream inference is latency-bound, not throughput-bound: there's no queue of other requests to batch alongside, and there shouldn't be. Key drivers of J/token include:

• Batch size: more concurrent requests amortize the same idle power over more tokens — a difference in workload, not a problem to fix on a personal box
• Traffic level: idle power is a larger share of the total when fewer tokens are being processed in a given window
• Model architecture: Dense models activate all parameters on every token, while Mixture-of-Experts (MoE) models route each token through a subset (~20%) of total weights. A dense 31B model may therefore have higher J/token than a 397B MoE, because the MoE only activates ~60B parameters per token

Live activity metrics

Each model also gets a companion "Live Activity" card, shown alongside its main carbon card, with real-time engine state read directly from vLLM and DCGM rather than derived from carbon math. Unlike CO₂/token or J/token, these are absolute engine-state readings — a value of zero is exactly as meaningful as any other value, so none of them are gated behind a minimum-throughput threshold:

Cumulative CO₂ emissions (time-series view)

The backend's timeseries endpoint queries the Prometheus range API at adaptive resolution (5-minute steps for 24 h, hourly for 7 d, 6-hourly for 30 d) and returns co2_grams_per_hour at each step (already computed server-side as power_watts × 0.198, nimbus's fixed grid intensity). The dashboard then integrates that rate client-side using the rectangle rule, running-summed over the selected window:

total_g = Σ (co2_grams_per_hour_i × step_hours)

This total only grows — it never resets within a window — so the chart climbs steadily even while nimbus is mostly idle, rather than looking flat or empty the way a tiny, nearly-constant rate does.

Limitations and Future Work

• CPU, DRAM, and network power are excluded (typically adds 15–30% to total)
• Datacenter PUE (Power Usage Effectiveness) is not applied (adds 10–50%)
• Carbon intensity values are annual averages; real-time grid mix varies hour-to-hour
• DCGM reports a single power reading per GPU; we cannot decompose energy into prefill vs decode phases on the nimbus side
• The commercial frontier model spec is an informed estimate, not a known architecture. No empirical per-token energy measurement exists for GPT-4, Claude 3/3.5/3.7, or Gemini 1.5/2. The closest empirical anchor is Google's production measurement of 0.24 Wh/prompt for Gemini Apps (arXiv:2508.15734). Our physics-based estimate is broadly consistent with this and with independent bottom-up estimates (0.34 Wh median, arXiv:2509.20241)
• Decode marginal energy (6 J/token) assumes low batch matching typical nimbus traffic; commercial providers batching hundreds of concurrent requests achieve lower per-token decode costs, but also require the same or greater hosting floor
• Agentic energy amplification is highly framework-dependent: measured research-grade frameworks show 62–137× energy vs single-turn (arXiv:2506.04301); production systems are likely lower but remain poorly characterized in the literature
• The eGRID intensity table in internal/carbon/intensity.go will be updated as new annual data is published
• History (24h/7d averages, and the cumulative emissions chart) is in-memory only and resets on every pod restart, backfilled only from Prometheus's own ~2-day retention. Persistent storage across restarts is a possible future improvement, not yet implemented
• Request latency (p50/p99 via histogram_quantile) is deliberately not shown. nimbus's traffic is bursty single-user generation — over a short window there are often only one or two completed requests, and a quantile computed from that few samples is not a meaningful percentile, just one or two data points dressed up as statistics (the same structural problem that led to dropping the old CO₂/token comparison chart). A longer-window mean (sum/count ratio, gated on a minimum request count) could be added later if useful

For a more complete lifecycle analysis including training, hardware manufacturing, and cooling overhead, see the references below.

References

Everyday Reference Frames

Absolute CO₂ figures for a single, efficient GB10 node are small (grams, not kilograms) and don't mean much in isolation. The dashboard converts the cumulative total for the selected window (see "Cumulative CO₂ emissions" above) into a few widely-cited everyday equivalents, in the style of CodeCarbon:

Source Code

This dashboard is open source: github.com/boettiger-lab/nimbus-carbon-api, forked from nrp-carbon-api. The fixed carbon intensity constant is in internal/carbon/intensity.go.