Inference Request Prioritization for Safety-Critical vs Background Robot Tasks

Robot control loops run at 50 Hz or faster because that's what stability demands, yet the foundation models robotics now leans on for perception and planning answer in hundreds of milliseconds, sometimes multiple seconds. That gap between what a control loop can wait for and what a transformer needs to finish a forward pass is a structural mismatch, not a rounding error to be optimized away later. It's the central engineering problem of putting large models into physical systems, and solving it means treating inference requests the way real-time operating systems have long treated tasks: not equally, but by consequence. The hardest case in the field is anomaly inference for an industrial emergency stop, which needs to complete inside 10 milliseconds, a window that a large model on embedded hardware doesn't fit into no matter how much you tune it. That's why prioritization has to happen before the model ever runs, not after. Missing the window doesn't just lower a quality score: the stop command fires on data that's already stale, or it fires too late to matter.

The three-tier task taxonomy that makes prioritization possible

Mixed-criticality systems theory, long applied in avionics and automotive control, hands robotics the vocabulary it needs. Under that framework, each task gets more than one estimate of its worst-case execution time, and only a slice of the system actually counts as safety-critical, subject to certification requirements that differ from those applied to the rest of the stack.

Applied to a robot, that framework splits into three tiers, and getting the split wrong is the mistake most implementations still make. Emergency stop, collision avoidance, and anomaly detection are Tier 1: hard deadlines, and a missed one has physical consequences, not a lower score. Tier 2 covers perception and operational work, lane detection, segmentation, cruise control, object detection, where deadlines are soft to firm and a degraded answer is recoverable rather than catastrophic. Tier 3 is background work: telemetry logging, map refinement, scene captioning over long horizons, all of it deadline-flexible and safe to defer or push off-device.

Skipping this taxonomy leaves every scheduler downstream guessing. A defensible call about which request runs next depends on knowing which tier it belongs to, and most production systems fall down exactly here: they schedule by arrival order because nobody assigned the tiers to begin with.

What first-come-first-served scheduling costs in a mixed-criticality robot system

Most LLM serving stacks in production today still batch requests first-come, first-served, and that's the wrong default for anything that touches physical hardware. FCFS works fine for a chatbot. For a robot it's close to indefensible, because it treats a background map-refinement query exactly the same as a collision-avoidance request that landed in the queue a millisecond later.

Under that scheme, a long planning token stream can occupy the inference pipeline while a genuinely urgent safety request sits behind it, and the value of that safety response decays toward zero the whole time it waits. TimelyLLM tested this directly, running real Tello drone traces and Neuromeka robot-arm traces against a vLLM baseline. The FCFS-style baseline produced substantially lower time utility, and TimelyLLM's scheduling cut overall waiting time by about 84% compared to that baseline.

The failure modes aren't symmetric, and that asymmetry is the whole argument against treating every request the same. A missed Tier 3 deadline burns some compute cycles and nothing else. A missed Tier 1 deadline can mean an actual safety incident. Scheduling a robot's inference requests as if all latency costs the same thing ignores the one fact that actually decides outcomes.

Deadline-driven scheduling with time-utility functions

Time-utility functions fix the accounting by attaching a deadline and a decay curve to every request. Safety-reactive inferences get a steep curve, one where the value of the response collapses almost immediately once the deadline passes. Background tasks get a flat or delayed curve, since a map update finished ten seconds late is still a useful map update.

A scheduler built around TUFs optimizes for total time utility, not throughput, and that distinction changes behavior in a concrete way: one high-utility safety request gets to preempt an entire batch of low-utility background requests, something a throughput-maximizing scheduler would never do on its own.

TimelyLLM's version introduces interrupt windows within the LLM's generation process, allowing a high-priority request to be served before the lower-priority planning stream resumes. That works because there is exploitable structure between generating a plan and executing it, and the scheduler takes advantage of that rather than treating generation as a single indivisible unit. Separately, mode-switching fixed-priority approaches appear in the literature for this class of problem: when the system detects a high-criticality event, it switches mode, drops or degrades the low-criticality tasks, and hands their compute to the tasks that actually need it.

DAG decomposition for multi-task inference pipelines

A robot's perception stack is rarely a flat queue of independent requests. It's a directed acyclic graph, where one model's output feeds straight into another model's input, and some of those outputs matter far sooner than others. Scheduling that graph as if it were a list, which is what most systems still do, throws away the one piece of structure that would let a safety-relevant result skip the line.

RED, a real-time DAG scheduling framework built for multi-task DNN inference on resource-constrained robots, targets exactly that structure, adapting at runtime to environmental change a fixed schedule can't anticipate. It runs on three mechanisms. An intermediate-deadline policy assigns sub-deadlines to individual DAG nodes and recomputes them online, letting an EDF-based dispatcher keep functioning even when requests arrive asynchronously or the graph itself changes shape. A MIMONet-aware refinement step splits inference into encoder and decoder sub-tasks and schedules them together so a shared encoder's output gets reused instead of recomputed. A safety-relevant result, object proximity, say, can be pulled off an early node and sent straight to the actuators without waiting on a lower-priority output like a scene description to finish computing. On-demand synchronization skips the periodic barriers that would otherwise add latency even when nothing downstream has actually changed.

An evaluation of RED found a 40.5% lower average deadline miss rate against an EDF baseline. A related ROS2 scheduler, ReDAGRT, showed a 29.7% reduction in deadline miss rate, a 42.9% reduction in 99th-percentile response time, and a further 40.8% reduction in interference through asymmetric per-DAG concurrency bounds. Those aren't small margins, and they point to the same lesson: treating a perception pipeline as a graph, not a list, is what makes early extraction of safety-relevant outputs possible.

Hardware partitioning to enforce isolation that software scheduling cannot

Real-time patches like PREEMPT_RT and Xenomai tighten scheduling latency inside a single operating system, but neither one gives spatial isolation. A background inference job can still evict the cache, saturate memory bandwidth, or push the chip into thermal throttling, and a safety-critical task sharing that hardware pays the cost no matter how well it's scheduled in software. Scheduling alone cannot solve a problem that lives below the scheduler.

Static partitioning hypervisors handle the isolation half by giving safety-critical partitions dedicated CPU cores and guaranteed memory bandwidth, a floor no software bug on the other partition can breach. Lifting that architecture wholesale into robotics runs into a real gap, though: the people modifying a robot's behavior in the field usually don't have the systems background the original platform engineers had.

Jiao's architecture closes that gap with three pieces. A dedicated hardware component provides override capability intended to operate independently of higher software layers. A Parameter Synchronization Service hides the cross-domain complexity so an integrator without deep systems training can modify workloads after deployment without breaking the partition contract. A Safety Communication Layer runs IEC 61508-aligned integrity checks with its own independent hardware override authority. Tested on an ARM Cortex-A55, this combination cut cycle-period jitter by 84.5% and brought tail timing error (p99 absolute jitter) down from 69.0 microseconds to 7.8 microseconds, eliminating every excursion above 50 microseconds. A system that's usually fine is not the same as one that's provably bounded, and for a Tier 1 task, usually fine is a failing grade.

Offloading background inference to edge-cloud to free local resources for safety tasks

If a background task can't be made safe to run alongside a safety-critical one on the same chip, take it off the device. Fog nodes and cloud compute exist for exactly this, whenever connectivity allows it, and for Tier 2 and Tier 3 work it's usually the right call.

For Tier 1 work, cloud inference hits a hard limit no amount of cloud processing power fixes: wide-area network latency and jitter make end-to-end response time unpredictable, and unpredictable is disqualifying for a 10-millisecond deadline. That's why hard-deadline safety tasks stay local, no matter how much heavier the on-device model has to be as a result.

RAPID's numbers show what that constraint costs in practice. Running vision-language-action inference entirely on-device measured around 1,274.4 milliseconds on a Jetson Orin and 667.2 milliseconds on a Thor, both far too slow for anything that has to react to the world in real time. RAPID's answer is an edge-cloud partitioning scheme that splits VLA inference across the two, accounting for visual noise and the step-to-step redundancy that shows up in embodied tasks, and it gets up to roughly three-quarters faster performance with only 5% to 7% overhead. That's a workable trade for Tier 2 and Tier 3 work. Nobody should accept it for an emergency stop.

Model compression as a way to create a fast local inference path for safety tasks

The pattern across all of this is straightforward, and it cuts against the instinct to run one model everywhere: put a small, quantized model on-device for the reactive Tier 1 decisions, and save the larger, unquantized models for the deliberative Tier 2 and Tier 3 work that can afford to wait.

Quantization is the main lever, and picking a bit depth is a real design decision. It shrinks a model enough to make on-device LLM deployment realistic on constrained hardware. Higher-bit formats keep more precision; lower-bit formats trade accuracy for speed. Picking a format means picking how much accuracy a safety task is allowed to give up in exchange for meeting its latency budget, and that trade should be made on purpose, not left to whatever the framework ships with by default.

Pruning and lightweight architectures fill in the rest, but only if the compression technique targets the actual bottleneck. Survey work on embodied foundation models at the edge notes that different model architectures face different bottlenecks, and that distinction tells an engineer which lever actually moves the needle. Compress a memory-bound model with a technique aimed at compute, and not much happens. The same survey is blunt about what reliable deployment actually requires: co-design across multiple system layers together, with a clean split between fast control and slower semantic reasoning rather than one model trying to do both.

How Scheduling, Partitioning, and Offloading Compose into a Tiered Architecture

None of these three mechanisms does the whole job alone, and treating any one of them as sufficient is how deployments end up with a safety task that's fast on the bench and unreliable in the field. They stack because each one covers what the others miss.

Hardware sits at the bottom. A static partitioning hypervisor guarantees Tier 1 tasks their own CPU cores and memory bandwidth, a floor that no scheduling decision made in software above it can violate, regardless of how the rest of the system gets configured.

Scheduling operates inside that floor. Deadline-driven dispatch using time-utility functions, plus sub-deadline assignment at each stage of a multi-stage inference pipeline, decides execution order and who gets to preempt whom, and mode switching drops Tier 2 and Tier 3 tasks the moment a high-criticality event occurs.

Model choice is what makes the other two tractable. A compressed fast-path model handles Tier 1 requests on-device, while full-scale models handle Tier 2 and Tier 3 work either locally inside a lower-priority partition or offloaded to edge-cloud.

If any one layer is pulled out, the other two can't cover for it. Scheduling alone can't stop a background job from starving a safety task's cache or memory bus. Partitioning alone doesn't tell the scheduler which request inside a partition to run first. Offloading alone does nothing for the request that has to finish within a sub-10-millisecond budget and therefore faces severe constraints on any offloading path. Building safety-critical robots on foundation models means building all three layers together, because the physical consequences of a missed deadline don't leave room to skip any of them.