Latency Percentile Reporting Standards for Robot Inference Benchmarks

A mean latency figure for a robot inference benchmark hides exactly the requests that matter most. If a vision-language-action model responds in 40 milliseconds nine times out of ten but takes far longer on the tenth try, an average will report something reassuring, like 76 milliseconds, and tell the reader nothing about the fact that one in ten control cycles just blew through the robot's step budget. LLM inference distributions have heavy right tails: the slowest requests can run 5x to 20x slower than the median, and that tail behavior gets worse under load, not better, because of straightforward queuing effects. The distribution usually forms what practitioners call a hockey stick: P50 through P90 cluster close together, then the curve bends upward hard past P95. A benchmark standard that reports only a mean is, in a functional sense, not measuring the thing that breaks robots.

The percentile vocabulary robot benchmark authors need to know

Percentile terminology gets thrown around loosely, so each number's precise claim needs to be stated exactly.

P50, the median, describes typical-case latency. It's useful for characterizing what "normal" looks like, but it says nothing about how bad things get when they go wrong. P90 captures the experience of the large majority of requests without being thrown off by rare outliers, and P95 pushes that a bit further, covering the vast majority of traffic while still avoiding the noise sensitivity that P99.9 carries. P99 covers the worst 1 percent of requests and has become the standard gate in LLM serving benchmarks: Anyscale's LLMPerf methodology, for instance, states explicit interest in "the P50, P90, P95 and P99." P99.9 goes one layer deeper still, exposing the deepest-tail outlier behavior, and it is the percentile associated with safety-critical automotive inference benchmarking, where missing even rare latency spikes carries real consequences.

None of this works, though, unless the latency itself is broken into the right pieces. Time to First Token (TTFT) measures the delay before any output appears at all, and for a robot controller, that's the slice of latency it actually feels before it can start acting on a response. Time per output token, sometimes called inter-token latency (TPOT/ITL), covers the generation phase after that first token, and it matters most when a robot policy streams action tokens rather than waiting around for a full plan. End-to-end request latency is the number that counts when the complete response has to be ready before any action executes. And goodput, throughput restricted to only the requests that meet a latency service-level objective, is arguably the most honest number in the bunch: it measures how well a system serves the robot under a real constraint, not just how fast the system runs when nothing is watching the clock.

Each of those needs its own percentile column, not a blended average, because TTFT and TPOT fail independently, for different reasons, at different times. TTFT tends to degrade under prefill congestion, when a lot of long prompts hit the scheduler at once. TPOT tends to degrade under decode contention, when too many sequences are generating tokens in parallel and competing for the same compute. Averaging them together hides which failure mode is actually occurring, which makes the number close to useless for anyone trying to fix the system.

As of September 2026, the inference-serving stack (vLLM, NVIDIA TensorRT-LLM, SGLang) has largely converged on shared definitions for TTFT, inter-token latency, and percentile-based tail latency reporting. That's a meaningful development: the vocabulary for describing LLM serving performance has stabilized across the major serving engines, even though nothing equivalent yet exists for robot-specific inference. The words are ready. The standard that applies them to robotics isn't.

What robot and VLA control loops require from inference timing

Robot control loops don't care how elegant a percentile curve looks. They care whether a decision arrives before the next physical step needs to happen. A vision-language-action system generally needs real-time inference somewhere in the 10 to 100 millisecond range, just to keep pace with the rate at which visual signal actually arrives.

A 2026 VLA deployment study puts numbers on how tight that margin really is. At a robot control frequency of 15 Hz, the per-step budget works out to roughly 67 milliseconds. The VLA model's inference alone, in that study, ran 76 to 80 milliseconds, which is already past budget before anything else gets added. Adding LAN offloading tacks on another 34 to 40 milliseconds of network delay. Adding data processing adds another 10 to 20 milliseconds on top. Total end-to-end delay landed at 122 to 140 milliseconds, roughly a two-to-three-step lag behind where the robot's perception actually is.

That's the median case blowing through the control-loop budget. That's the median case blowing through the control-loop budget. Which means P95 and P99 in this context aren't academic tail curiosities, they represent real, physical missed steps: an arm that reaches a fraction of a second late, a gripper that closes on where an object used to be.

Edge hardware adds another layer of difficulty. Deploying VLA models on low-power onboard devices like the NVIDIA Jetson Orin introduces its own inference latency and caps how fast the control loop can run. Asynchronous inference can mask some of that latency by letting the robot keep moving while the next decision computes in the background, but it trades that speed for perception-execution misalignment: the action being executed no longer matches the world the model most recently perceived, and the reaction time to new events stretches out.

A harder line that no percentile can soften runs beneath all of this: safe robot operation requires safety routines to execute within a strict upper bound on end-to-end latency. Safe robot operation requires safety routines to execute within a strict upper bound on end-to-end latency. That's not a service-level objective a system can miss 1 percent of the time and call acceptable. It's a ceiling, and a benchmark standard that doesn't report against it isn't really measuring safety.

How MLPerf Inference v5.1 sets percentile gates

MLPerf Inference v5.1 drew a record 27 submitters and added three new benchmarks. The new benchmarks are a reasoning benchmark built on DeepSeek-R1, a speech-to-text benchmark built on Whisper Large v3, and a small LLM benchmark built on Llama 3.1 8B. How the percentile gates were structured task by task is what makes the round worth studying for robotics, not the submitter count.

Every gate in v5.1 is set at the 99th percentile for both TTFT and TPOT, but the actual numbers shift depending on the task and the scenario. LLM Q&A, run on Llama-2-70B against the OpenOrca dataset, gets a Conversational gate of 2000 ms TTFT and 200 ms TPOT, and a tighter Interactive gate of 450 ms TTFT and 40 ms TPOT. LLM Summarization, run on Llama-3.1-8B against CNN/DailyMail, sets 100 ms TPOT for Conversational, 500 ms TTFT and 30 ms TPOT for Interactive. The Reasoning benchmark on DeepSeek-R1 gets 80 ms TPOT. Long-context, on Llama-3.1-405B, allows 6000 ms TTFT and 175 ms TPOT.

The thresholds are task-conditioned, not universal, so the same underlying hardware gets judged against a different latency bar depending on whether the workload is conversational, interactive, reasoning, or long-context. The same underlying hardware gets judged against a different latency bar depending on whether the workload is conversational, interactive, reasoning, or long-context. That's a direct rebuke to the idea of a single latency number applying across an entire benchmark suite, and it's a lesson robotics has to absorb, because a pick-and-place task and a bimanual folding task don't share a latency budget any more than a chatbot and a code assistant do.

One more detail from v5.1: the single-stream scenario uses 90th-percentile per-sequence latency rather than TTFT/TPOT breakdowns, because per-token timing doesn't capture how the system performed for a given full sequence. The single-stream scenario uses 90th-percentile per-sequence latency rather than TTFT/TPOT breakdowns, because per-token timing doesn't capture how the system performed for a given full sequence. That distinction, measuring the whole sequence rather than its component tokens, maps almost exactly onto how a robot consumes an action chunk, since the controller ignores the timing of token seven inside the chunk and instead depends on the whole chunk arriving in time to execute.

MLPerf Automotive as the closest formal analog to what a robot standard needs

Automotive got there first, and the reasoning trail is instructive. An automotive benchmark built on a particular perception-labeling technique showed up inside MLPerf Inference v5.0, and by August 2025, MLCommons had spun out a standalone MLPerf Automotive suite (v0.5), a standardized suite built specifically for safety-critical, real-time inference.

MLCommons documentation states that a 99.9 percent latency percentile was chosen because automotive workloads are safety-critical. For the PointPainting/DeepLabV3+ benchmark, the primary metric is latency at P99.9, reported alongside throughput and accuracy rather than in place of them. MLCommons frames the rationale in blunt terms: "The main KPI is latency because automotive is a real-time, and often, functional safe system."

That sentence could be lifted directly into a robotics standard with only the word "automotive" swapped out. Robots share the same real-time, often functionally safe operating profile that justified P99.9 for cars, and nothing about a warehouse picking arm or a legged inspection robot makes its tail latency less consequential than a highway perception stack's. MLPerf Automotive isn't a perfect template, since its workloads don't map one-to-one onto VLA action generation. But its governing logic, that safety-critical, real-time systems earn a stricter percentile gate than general-purpose serving benchmarks, is exactly the logic robotics benchmarking has been missing.

What the emerging VLA-specific tooling measures, and what it omits

A handful of tools have started measuring VLA inference performance directly, and each one gets part of the picture right while leaving a gap somewhere else.

VLA-Perf takes an analytical, roofline-based approach: rather than running a model and clocking it, it predicts optimal inference latency and throughput for arbitrary model-system combinations. It supports varying model sizes and architectures, stateless and long-context inference, different action chunk sizes, asynchronous inference, and dual-system model pipelines. It's open-sourced in a research lab's public code repository. That breadth of configuration coverage is genuinely useful for early design decisions, though a roofline model, by nature, predicts rather than measures, so it's a design tool more than a reporting standard.

A unified inference runtime for VLA models, this tool actually reports a named percentile out of the box: request P90 and vision-stage latency in milliseconds, alongside CUDA memory use and device allocation, across a range of hardware configurations and supported VLA models. Reporting P90 natively puts it ahead of most tooling in this space, though P90 alone still misses the deep tail that P99 and P99.9 exist to catch.

vla-inference-bench, a Python benchmarking harness, goes a step further on percentile coverage, reporting per-action-chunk latency at both P50 and P95, plus throughput and peak memory, across CPU, Apple Silicon (MPS), and CUDA. That's a meaningfully wider percentile spread than vla.cpp offers, but it stops at P95. Neither P99 nor P99.9 shows up in its current output, so the exact tail behavior that safety-critical robot operation depends on, the same tail MLPerf Automotive built its entire KPI around, is invisible there.

vla-eval takes yet another angle. It's a unified evaluation harness spanning 18 simulation benchmarks and 13 model servers, and it aggregates a VLA leaderboard with results across its supported benchmarks. But its focus is task success rate, not latency percentiles. For a field trying to establish rigorous timing standards, that's a real gap: the most broadly aggregated VLA leaderboard available doesn't report latency tails.

Putting the four side by side reveals a pattern: coverage of percentiles is improving, tool by tool, but no single piece of tooling yet reports the full P50 through P99.9 spread, conditions it on workload and concurrency the way MLPerf does, or ties the two together. Coverage of percentiles is improving, tool by tool, but no single piece of tooling yet reports the full P50 through P99.9 spread, conditions it on workload and concurrency the way MLPerf does, or ties it to a hard safety ceiling the way automotive benchmarking does.

Specifications a rigorous percentile reporting standard for robot inference benchmarks must include

Put the precedents together, MLPerf Inference's task-conditioned gates, MLPerf Automotive's strict tail-latency floor, the VLA tooling's partial percentile coverage, and the shape of an actual standard starts to become clear.

The percentile floor should be P50, P90, P95, and P99 at minimum, for every workload class. Any application class carrying a safety-critical real-time constraint should be required to report P99.9 as well, following the MLPerf Automotive precedent directly. That's not a suggestion; a safety-critical system that can't show its P99.9 tail hasn't demonstrated it's safe, it's just demonstrated that it's usually fine.

Latency has to be broken into its component dimensions, each reported separately at every one of those required percentiles. TTFT, because the controller can't begin planning until the first token lands. TPOT/ITL, because streaming action tokens carry their own per-token timing budget separate from the first-token delay. End-to-end request latency, because plenty of VLA systems need a full action chunk in hand before any movement starts. And per-sequence, action-chunk latency, following the MLPerf single-stream precedent directly, since that's the number that reflects how the system performed for one complete unit of robot action, not just its average token rate.

None of that is meaningful without workload conditioning, either. A latency number reported without its context is barely a number. Control frequency in Hz, and the per-step budget derived from it, has to be stated up front, since that's the denominator every other figure gets measured against. Action chunk size matters too: larger chunks amortize the cost of TTFT across more output, but they stretch out end-to-end latency in exchange. Input modality and token length distribution need disclosure as a distribution, not a mean, since vision-language inputs carry different prefill costs than text-only inputs and a single average washes that difference out. Concurrency level has to be stated as well, because tail latency gets worse under load through queuing effects, and a result reported at one concurrency level says nothing about how the system behaves at three times that load. And inference location, onboard versus LAN-offloaded versus cloud, has to be disclosed explicitly: the 2026 VLA study cited earlier shows LAN offloading alone adding 34 to 40 milliseconds, more than half of the entire control-loop budget at 15 Hz.

Sample size is the last piece, and it's the one most benchmarks skip past. MLPerf's statistical rigor framework ties the required number of queries to the tail percentile being reported, the margin desired, and the confidence interval targeted, since P99.9 needs substantially more samples to be statistically reliable than P95 does. MLPerf requires 1,024 queries for its single-stream scenario; a benchmark claiming a P99.9 gate needs more than that floor to back the number up. A robot inference standard should state its sample-size floor explicitly, the same way, rather than leaving vendors to report a P99.9 figure drawn from a sample too small to say anything reliable about the tail.

None of this is complicated to write down. It's a matter of discipline, borrowing what MLPerf Inference and MLPerf Automotive have already proven out, and applying it to a domain that has, so far, mostly reported P50 and called it a day. The tooling is arriving. Robotics will adopt a shared percentile standard either before or after the first tail-latency failure that a mean latency number utterly failed to predict.