The model is not the slow part
A language model's environment waits for it. A physical system does not. The object keeps falling, the foot keeps sliding, the contact transient came and went while the image was still being debayered. Everything difficult about embodiment follows from that asymmetry, and almost none of it is repaired by a larger network.
Three constraints do the damage, and all three are properties of the instrument rather than the policy:
- Latency is destabilising, not merely annoying. Delay in a feedback loop consumes phase margin at a rate set by the loop's bandwidth. Past a threshold the system does not degrade — it oscillates. §02 does the arithmetic.
- The interesting signals are fast and the sensors are slow. Slip, impact and stick–slip transitions occupy hundreds of hertz to kilohertz. Cameras deliver 30–60 Hz; the highest-resolution tactile sensors deliver 30 Hz. §05 and §06.
- Unsynchronised sensors are worse than fewer sensors. Fusing streams whose relative timing is unknown injects a spatial error you cannot calibrate away, because it scales with velocity. §04.
None of this argues that models do not matter. It argues that a great many published failures attributed to policy quality are, on inspection, a control loop that could not have been stable at any policy quality — and that the fix is a board, a clock and a bus, not a checkpoint.
A note on what follows. Numbers are taken from vendor specifications and published papers and are cited where used; the derivations from them are shown so they can be checked or rejected. Where a figure is illustrative rather than measured, its caption says so.
What a millisecond is worth
Take a loop with open-loop transfer and a pure transport delay . The delay contributes no magnitude change and pure phase lag, linear in frequency:
So at the gain crossover , the delay costs radians of phase margin outright. Requiring a margin to remain gives the design inequality that governs every embodied system:
Bandwidth and delay trade off hyperbolically. This is not a rule of thumb; it falls out of the Nyquist criterion, and it is the reason a policy's rate matters more than its accuracy for anything involving contact.
Where the delay comes from
A discrete-rate stage running at contributes, on average, half a period of zero-order-hold delay before its output is refreshed:
Add the pipeline. Using published figures — a global-shutter camera at 20 ms exposure plus 10–20 ms readout and transport, an OpenVLA-OFT forward pass at 112.2 ms, a policy issuing actions at 5 Hz, and a modest actuator response:
| Stage | Contribution | Source of the number |
|---|---|---|
| Exposure + readout | 30 ms | 20 ms global exposure, 10 ms readout/transport |
| Policy forward pass | 112 ms | OpenVLA-OFT measured latency |
| ZOH at 5 Hz | 100 ms | |
| Bus + driver + actuator | ~5 ms | Optimistic; §09 argues it is often far worse |
| Total transport delay τ | ≈ 247 ms |
Feed that into the inequality with a conventional rad:
Half a hertz. That is the entire closed-loop bandwidth available to a system built this way — the rate at which it can correct its own errors. It is adequate for quasi-static pick-and-place, where the world is patient. It cannot catch a falling object, cannot arrest a slip, and cannot stabilise a legged gait, and no amount of additional training will change that, because the limit is Nyquist's rather than the optimiser's.
Which is why architectures look the way they do
Action chunking — emitting a whole trajectory segment per forward pass rather than one action — is usually explained as a fix for compounding error. It is at least as much a fix for this: the chunk is replayed by a fast inner loop, so the ZOH term is set by the inner loop's rate, not the policy's. The slow policy stops being inside the feedback path and becomes a reference generator for something that is.
That is the standard resolution, and it works. Its cost is that the slow policy can no longer react within a chunk: whatever happens in those 200–1000 ms is handled by whatever fast controller you built, or not at all. The reflex layer is hardware you must design, and it is the layer most open platforms leave out.
The interactive in §03 computes this budget for parameters you choose.
Six decades of frequency, and the holes in them
Every subsystem in an embodied stack lives at a characteristic rate. Laid on one logarithmic axis, the mismatches that matter become obvious — and so does which of them are physics and which are merely the hardware nobody has built.
Reading the ladder
01 — The fast end is solved
Event cameras already resolve microseconds; current-mode motor drivers already close at tens of kilohertz; a modest microcontroller services a 1 kHz loop with room to spare. None of this is research. Most research platforms simply do not expose it.
02 — The gaps are integration failures
A 30 Hz tactile sensor and a 1 kHz torque loop cannot cooperate on a slip event — not because either is badly made, but because nobody built the sensor to report at the rate the phenomenon occupies.
03 — Where to put the boundary
Every stack draws a line between the learned slow layer and the hand-written fast one. Push it down and you lose reactivity; push it up and you need learning at rates nothing achieves. Most of §12 is about moving that line.
Interactive
Close your own loop
Enter a pipeline and the delay inequality from §02 is evaluated against it. The three presets are the three nested loops of one working system.
Pipeline
Stepping through the presets is the argument of this report in miniature. Put the policy inside the loop and you get 0.51 Hz. The chunk-replay layer, which carries no inference term because the trajectory is already in memory, gets 23 Hz. The torque loop underneath reaches 119 Hz. Every task harder than quasi-static needs the layers the outer loop cannot supply — and note how little the inference term matters once the rate is low, because the zero-order hold dominates. A faster network does not buy what a faster inner loop buys.
An unsynchronised sensor is a systematically wrong sensor
Two sensors whose timestamps disagree by describe the world at two different instants. If the scene has relative velocity , fusing them commits a registration error
which is not noise. It is a bias, proportional to speed, in the direction of motion — exactly the structure that a learned model will absorb as a systematic property of the world and then fail on when the speed distribution shifts.
| Regime | Speed | Δt = 10 ms | Δt = 1 ms | Δt = 10 µs | Sub-mm needs |
|---|---|---|---|---|---|
| Slow manipulation | 0.1 m/s | 1 mm | 0.1 mm | 1 µm | Δt < 10 ms |
| Brisk manipulation | 1 m/s | 10 mm | 1 mm | 10 µm | Δt < 1 ms |
| Legged locomotion | 3 m/s | 30 mm | 3 mm | 30 µm | Δt < 330 µs |
| Dynamic catching / UAV | 10 m/s | 100 mm | 10 mm | 100 µm | Δt < 100 µs |
Read the right-hand column as a hardware requirement. Anything past slow manipulation needs sub-millisecond alignment, which rules out timestamping in userspace on arrival — the arrival time of a USB frame tells you when the host got round to it, not when the photons landed.
What actually fixes it
Hardware trigger
One line, driven by one clock, into every sensor's external trigger input. Exposures start simultaneously by construction and skew is a wiring-length question. Cheapest correct answer, and the reason machine-vision cameras expose a trigger pin that webcams do not.
PTP / gPTP
IEEE 1588 and 802.1AS distribute a clock over the network. With hardware timestamping in the MAC, sub-microsecond alignment is routine; with software timestamping you are back to milliseconds and jitter. The distinction is a PHY feature, so it is decided when the board is designed.
Distributed clocks
EtherCAT's distributed-clock mechanism aligns slaves to well under a microsecond and is the standard answer for multi-axis motion. It puts the clock in the same wire as the control traffic, which removes an entire class of failure.
A gap worth checking before you build. Hardware triggering and PTP are standard in machine vision and industrial motion, but I am not aware of a widely-adopted open reference design aimed at robot labs — a board that takes one oscillator, emits hardware triggers to several cameras and IMUs, timestamps returning events, and presents one coherent clock to the host. If one exists it is not easy to find, and the practical consequence is that synchronisation quality is rarely reported alongside results.
Frames are the wrong abstraction for motion
A frame camera integrates light over an exposure window and reports a synchronous grid. Both halves of that are hostile to fast control: the integration blurs anything moving, and the synchrony means the information rate is constant regardless of whether anything happened.
What the specifications say
| Frame | Event (DVS) | |
|---|---|---|
| Temporal resolution | ~20 ms exposure | ~1 µs |
| Latency to first signal | 30–40 ms | < 1 ms |
| Dynamic range | ~60 dB | 120–143 dB |
| Output | Dense, constant rate | Sparse, scene-dependent |
| Motion blur | Yes | None by construction |
| Learning stack | Everything | Thin |
A ~140 dB dynamic range is the difference between working and not working when a robot moves from a shaded interior to direct sun — a failure mode that no amount of augmentation fixes, because the information was clipped at the sensor.
Why it has not taken over
The output is an asynchronous stream of tuples, which does not fit the dense-tensor assumption every pretrained vision backbone is built on. The usual remedy — accumulating events into frames over a window — reimposes exactly the latency and the temporal quantisation the sensor existed to remove.
So the honest position is that event cameras solve the physics and the community has not yet solved the representation. That is a research problem with a hardware component: sparse, asynchronous inference is poorly served by accelerators designed for dense matrix multiply, and the interesting near-term win is probably not a neuromorphic processor but a tightly co-located preprocessor that turns events into low-dimensional state — optical flow, corner tracks, time-to-contact — at source, at microsecond latency, so the slow policy never sees the raw stream at all.
That pattern generalises, and it is the strongest architectural recommendation in this report: put a small, fast, dumb computation physically next to the fast sensor, and let the large, slow, smart one consume its output. The alternative — streaming everything to one accelerator — spends the latency budget on transport and the bandwidth budget on data that was mostly redundant.
Exquisite spatial resolution at useless bandwidth
Vision-based tactile sensors — GelSight, DIGIT and their descendants — image an elastomer's deformation with a camera behind it. They resolve surface geometry at 20–30 µm, finer than human skin, and read out at the camera's rate of about 30 Hz. Contact events need the opposite balance.
Slip onset, stick–slip transitions and impact transients occupy hundreds of hertz to kilohertz. Sampling them at 30 Hz does not degrade the estimate; it aliases it. By the sampling theorem, detecting a transient at requires , so a 500 Hz slip signature needs at minimum a kilohertz sensor — a factor of 33 beyond what the highest-resolution devices provide.
The bandwidth arithmetic
The reason imaging sensors cannot simply be read faster is a data-rate problem. A vision-based fingertip streaming VGA colour at 30 Hz produces
per finger — over 1.1 Gbit/s for a five-fingered hand, in five independent camera pipelines, before any processing. Raising that to 1 kHz would demand 7.4 Gbit/s per finger, which is not a sensor problem but a wiring problem.
A taxel array inverts every term. One hundred taxels at 12 bits and 1 kHz is
— 184× less data at 33× the bandwidth, over a single SPI bus. The information you lose is spatial detail; the information you gain is the entire temporal band where contact events live.
What should be built
- Hybrid fingertips. A low-rate, high-resolution imaging channel for geometry and a high-rate, low-resolution channel for transients, sharing one substrate and one clock. BioTac does the multimodal half of this and has done for years; what is absent is an open, cheap version with fine spatial pitch.
- On-sensor feature extraction. Do not stream 221 Mbit/s to ask whether contact moved. Compute the shear field or a slip statistic at the fingertip and emit tens of bytes per millisecond.
- Durability as a specification. Tactile skins are consumables; an elastomer degrades over thousands of contacts and drifts before it fails, silently corrupting a dataset. Field-replaceable surfaces and an in-situ calibration routine are research infrastructure, not product polish.
- Published transfer functions. Of the sensors in Figure B, readout rate is documented but a measured frequency response generally is not. Without one you cannot know what your data aliased.
Gear ratio is a perception decision
A learned manipulation policy that regulates force needs an actuator that can be force-controlled and can sense force. Both properties are destroyed by the thing conventional robots use to get torque cheaply: a high-ratio gearbox.
Reflect the motor's rotor inertia through a reduction and it appears at the output scaled by the square:
Friction reflects similarly and, worse, asymmetrically. Comparing a classic harmonic drive against a quasi-direct-drive :
A 156-fold difference in reflected inertia is the difference between a joint you can backdrive by hand — and therefore one whose output torque you can estimate from motor current alone — and one where the gearbox dominates everything the sensor could tell you. Quasi-direct drive is not a mechanical preference; it is what makes proprioceptive force sensing possible without a torque sensor at every joint.
The sample rate sets the stiffness
There is a second, less obvious coupling between electronics and mechanics. A discrete controller rendering a virtual stiffness into a system with physical damping at sample period is passive — that is, guaranteed not to inject energy — only if
This is the Colgate–Brown condition, and it says the maximum stiffness you can render stably is inversely proportional to your loop period. Doubling the control rate doubles the achievable stiffness at the same physical damping. A stiff, crisp contact behaviour is bought in the timer configuration, not the gains.
What that implies for a board
- Current control in the kilohertz, closed locally on the motor controller — never over the host bus.
- Torque estimation from current requires a known, low, and characterised friction model. Publish it.
- Encoder resolution bounds velocity estimation: differentiating a coarse encoder at 1 kHz produces quantisation noise that the loop gain then amplifies. Position resolution is a velocity-loop specification in disguise.
- Thermal state changes the plant. Winding resistance rises with temperature, so the current-to-torque constant a policy learned when cold is wrong when hot. Log motor temperature as a first-class signal; almost no dataset does.
Watts, joules and the thing that has to carry them
Edge compute for robotics has stopped being capability-limited and started being power- and thermally-limited, which are problems for whoever designs the chassis rather than the network.
FP4, Blackwell
on Thor
configurable
at ~60 W
The energy per decision
A policy running at rate on a module drawing watts costs
so 130 W at 10 Hz is 13 J per action. Over an hour of operation that is 468 kJ — 130 Wh — which for a battery-powered platform is a substantial fraction of the pack before a single motor turns.
The comparison worth keeping in mind: the entire human central nervous system runs at roughly 20 W, closes contact loops at kilohertz, and does not need a heatsink. Nothing about that is achievable soon; it does indicate how far from any efficient frontier current stacks sit.
Thermal is the real constraint
130 W in a sealed, moving, dust-exposed enclosure is a hard mechanical problem. Fans are the least reliable part on most robots and the loudest; conduction to the chassis couples compute heat into structure whose dimensions then drift, which shows up as calibration drift over a session.
The consequence is that the module's sustained power — not its peak rating — sets what you can run, and sustained power is decided by your enclosure. A dev kit on a bench and the same module in a robot are different computers, and published inference rates rarely say which was measured.
The memory-bandwidth argument from the KV-cache report applies unchanged here and is sharper: an on-robot VLA decoding autoregressively is memory-bound, so its action rate is set by bandwidth rather than by TFLOPs. Thor's 128 GB of unified memory matters less for holding a bigger model than for holding a model, its KV cache, several camera buffers and a simulator without swapping.
Determinism is a property you buy at design time
Between the sensor and the controller sits a transport, and its worst case — not its average — is what enters the delay budget of §02. Jitter is more corrosive than latency: a constant delay can be modelled and compensated, while a varying one cannot.
| Transport | Cycle / period | Jitter | Where it belongs |
|---|---|---|---|
| EtherCAT | 12.5–100 µs | < 1 µs | Multi-axis motion; distributed clocks come free |
| CAN-FD | ~100 µs–1 ms | µs–ms | Robust, cheap, long runs; limited payload |
| SPI, local | < 10 µs | sub-µs | Sensor to co-located MCU. The right place for fast loops |
| TSN Ethernet | ~100 µs | < 1 µs | Mixed traffic with bounded latency; needs capable PHYs |
| Plain Ethernet | variable | ms, unbounded | Telemetry, logging. Not a control path |
| USB | 125 µs–1 ms frames | ms, unbounded | Cameras and convenience. Never a control path |
Nearly every hobby and research platform closes its loops over USB or plain Ethernet because both are convenient, then reports a control rate as though it were guaranteed. A rate with no worst case attached is not a specification — and the tail, not the mean, is what destabilises a contact loop.
The host is part of the loop
Stock Linux offers no scheduling latency bound. With PREEMPT_RT, a tuned system holds worst-case wake-up latency in the tens of microseconds; without it, millisecond outliers are routine and are caused by whatever else the machine was doing. Since §02 shows delay entering the stability condition directly, a garbage-collection pause or a network interrupt storm is a stability event.
The defensible architecture is therefore not a faster host but a smaller trusted one: put the hard-real-time loop on a microcontroller that has no scheduler to be preempted by, and let the host be soft-real-time and best-effort. This is also the honest reading of §02 — the fast reflex layer wants to be a separate piece of silicon, close to the actuator, running code you can bound by inspection.
A practical, publishable habit: report loop timing as a histogram with a maximum, measured with cyclictest or an oscilloscope on a GPIO toggle, and state what else was running. It is not a common disclosure, and it would settle a good number of arguments about irreproducible results.
The gap is mostly unmodelled electronics
Domain randomisation trains a policy to be robust across a distribution of simulator parameters, optimising
It works, within a strict limitation that is easy to state and easy to forget: it can only cover what the simulator parameterises. If the real plant differs in a dimension the model does not represent, then , and widening the randomisation cannot reach it — it only makes the policy more conservative everywhere else.
Now consider what is typically parameterised — masses, friction coefficients, restitution, link geometry — and what typically is not:
| Unmodelled effect | Why it is left out | What it does to a policy |
|---|---|---|
| Sensor and actuator latency | Most simulators step sensing and actuation instantaneously | Trains against a plant with no phase lag — the §02 failure, exactly |
| Timing jitter | Fixed-step integration has none | Policy assumes a regular interval it will not get |
| Stribeck and pre-sliding friction | Contact solvers use Coulomb | Fails precisely at the low velocities manipulation lives at |
| Backlash and drivetrain compliance | Joints are rigid and ideal | Learned gains oscillate on the real gearbox |
| Thermal drift | No thermal state at all | Torque constant shifts over a session; performance decays |
| Sensor noise colour | Modelled as white when it is not | Filters tuned in sim are mistuned in reality |
Every row is an electronics or drivetrain property, and the first two are free to fix: adding a measured delay and a measured jitter distribution to the simulator costs almost nothing and removes the single largest unmodelled term. That it is not standard practice is one of the cheapest available wins in the field.
Which makes measurement the deliverable
The corollary is that hardware for physical AI is not only the robot — it is the instrumentation to characterise the robot. A platform that ships with its measured actuator frequency response, its latency histogram, its friction curve and its sensor noise spectrum lets a simulator be parameterised from data rather than from optimism. A platform that ships without them requires every user to rediscover them, badly, and is the reason sim-to-real results transfer between labs so poorly.
Concretely, and in rough order of value: a chirp or swept-sine test per joint yielding a Bode plot; a GPIO round-trip latency histogram at the 99.9th percentile; a breakaway-torque and Stribeck curve per joint; a sensor noise PSD; and a repeated-pose accuracy figure with the thermal state recorded. None needs equipment beyond a scope, and the whole set is a weekend.
Four orders of magnitude, and no internet to scrape
Language models are trained on corpora of order tokens that already existed. Robot learning has no equivalent, because interaction data must be produced, on hardware, in real time, at roughly one second per second.
The arithmetic is unforgiving. At 30 s per demonstration with a human operator, one million demonstrations is 8,300 operator-hours — around four person-years of continuous teleoperation, per embodiment. Scaling by hiring does not converge; scaling by hardware might.
Three hardware answers, and what each buys
- Cheap parallel teleoperation. Low-cost bimanual rigs with leader–follower arms drove the cost per demonstration down by an order of magnitude and are the reason imitation learning became broadly reproducible. The constraint they hit is the operator, not the robot.
- Handheld capture. A gripper instrumented with the same camera and held in a human hand records manipulation without a robot present, decoupling data collection from robot count entirely — at the cost of an embodiment gap the policy must later cross.
- Cross-embodiment pooling. Aggregate across labs and platforms and treat embodiment as a conditioning variable. This is only sound if the data is comparable, which returns directly to §04 and §10: unsynchronised, uncalibrated, uncharacterised data does not pool, it averages.
What is missing from all three
Demonstrations are near-exclusively vision and proprioception. Force, torque, tactile, vibration, motor temperature — the channels that contact-rich behaviour actually depends on — are usually absent, because the rigs that collect data cheaply are the rigs least likely to carry that instrumentation.
So the largest datasets systematically under-represent the modality most needed for the tasks the field says it cares about. A teleoperation rig that records calibrated, time-aligned force and tactile alongside vision is a more valuable contribution than another policy architecture, and it is a hardware project.
Note the compounding failure: because tactile is rare in datasets, few models consume it; because few models consume it, few rigs bother to record it. Breaking that loop requires someone to build the rig first.
Specific, buildable, unbuilt
Collecting the gaps above into things that could be designed, fabricated and published. Each is scoped so that a small team could do it, and each removes a named obstacle rather than improving a metric.
Instruments
- An open synchronisation hub. One oscillator, hardware trigger outputs for N cameras and IMUs, hardware timestamp capture on return lines, PTP grandmaster to the host, and a published skew budget. Removes §04 for everyone downstream.
- A latency characterisation jig. GPIO-to-GPIO round trip through the full stack — sensor, transport, host, controller, actuator — reporting a histogram with a maximum. Turns §09's worst case from folklore into a number on a datasheet.
- A hybrid tactile fingertip. BioTac already pairs a coarse electrode array with a 2.2 kHz hydrophone, so the concept is proven. What is missing is an open, low-cost version that keeps the kilohertz channel and adds fine spatial pitch, with a published frequency response (§06).
Subsystems
- An event-camera front end. Co-located compute emitting flow, corner tracks or time-to-contact at microsecond latency, so the policy consumes state rather than an event stream (§05).
- A reflex board. A microcontroller between policy and actuator that executes action chunks, enforces an energy and force envelope, and falls back deterministically when the chunk expires or the host stalls. This is the layer §02 shows is mandatory and §09 shows must not live on the host.
- A characterised joint module. Quasi-direct drive, kilohertz current loop, thermal telemetry, and a shipped Bode plot, friction curve and noise spectrum (§07, §10).
Open problems that are not merely engineering
- Learning at kilohertz. Everything fast in current stacks is hand-written. Whether a learned controller can occupy the 1 kHz band at all — what architecture, what latency, on what silicon — is open, and the answer determines whether the boundary in §03 can move.
- Representations for asynchronous sensing. Events and tactile transients are sparse and irregular. Dense-tensor accelerators and frame-based backbones are the wrong shape for both, and the accumulate-into-frames workaround discards the property that motivated the sensor.
- Verifiable safety envelopes around learned policies. A learned policy cannot be certified. A hardware-enforced limit on energy, velocity and force can be, and it must degrade to a safe state without the host's participation. What that layer should expose to the policy — and how much authority it may take back — is unsettled.
- Calibration that survives. Thermal drift, wear and compliance move the kinematics within a session. Continuous self-calibration from proprioception, with a published drift budget, would make long-horizon autonomy considerably less brittle.
- Benchmarks with hardware error bars. Two labs running the same policy on nominally identical arms get different numbers, and no venue requires the timing and calibration disclosures that would explain why. Comparability is an instrumentation standard nobody has written.
The short version
Physical AI's binding constraints are measurable and mostly electrical. Delay sets a hard ceiling on closed-loop bandwidth, and current pipelines sit near half a hertz. Tactile sensing has its resolution and its bandwidth the wrong way round for contact. Unsynchronised sensors inject a velocity-proportional bias that training absorbs and then fails on. Simulators omit the latency and jitter that dominate the sim-to-real gap. Datasets omit the modality the hard tasks need. Every one of those is a board, a clock, a bus or a characterisation report — which is unusually good news, because it means the field's progress is gated on artefacts that a small team can design, build and publish.