A benchmark is an experimental record
GPU timing numbers become useful only when hardware, synchronization, sampling, and raw observations remain attached to the result.
What this article establishes
- Separate device execution time from asynchronous host dispatch.
- Report distributions and uncertainty rather than one attractive number.
- Package enough environment data for a later comparison.
A benchmark result is not a number. It is a number attached to an environment, a protocol, raw observations, and a question. Remove any one of those and the result becomes difficult to interpret. Remove several and it becomes decoration.
Begin with the decision
“Which implementation is faster?” is incomplete. A useful benchmark names the workload and the decision it supports. Latency for batch size one, throughput under saturation, memory pressure during training, and energy per inference are different experiments.
The metric follows the decision. It should not be chosen because it produces the largest ratio.
Synchronization changes the measurement
CUDA kernels are dispatched asynchronously. Timing only the Python call often measures queue submission rather than device execution.
import time
import torch
def timed_run(operation, samples=30):
observations = []
for _ in range(samples):
torch.cuda.synchronize()
started = time.perf_counter_ns()
operation()
torch.cuda.synchronize()
observations.append(time.perf_counter_ns() - started)
return observations
Synchronization is not free, so the measured region must be documented. Device events are preferable when the question is strictly kernel execution. End-to-end wall time is preferable when host overhead is part of the user experience.
Keep the distribution
Thirty observations contain information that their mean discards. Report the median, an interval, and the raw values. A violin or empirical cumulative distribution can reveal multimodality caused by thermal behavior, compilation, or background work.
| Field | Example value | Why it matters |
|---|---|---|
| Warmups | 10 | Excludes compilation and initial allocation |
| Samples | 30 | Exposes run-to-run variation |
| Statistic | Median | Resists a small number of long stalls |
| Interval | Bootstrap 95% | Communicates estimation uncertainty |
Preserve the environment
Record the device model, driver, runtime, dependency lock hash, operating system, power policy, precision, shapes, dataset version, and full command. Reference source by immutable commit. Raw result files should use a run identifier and checksum.
A future result should be compared automatically only when the methodology declares the runs compatible. Otherwise, show both records and explain the environmental differences.
The honest conclusion
Benchmarks are valuable when they narrow a decision. “Implementation A was faster under this protocol” is stronger than an unsupported universal claim. The caveat does not weaken the work. It tells the reader where the evidence applies.