LeRobot ACT and action chunking explained

Understand the predicted sequence

Action Chunking with Transformers was introduced in the ALOHA research project to learn manipulation from demonstrations. Instead of predicting only the next command, it predicts a sequence. Jointly predicting neighbouring actions gives the model a way to represent a coordinated part of a demonstration, such as approach, closure and initial lift, within one prediction.

This is a learned sequence, not a verified motion plan. It can reflect the patterns and errors in its demonstrations, and it does not inherently know whether an object was grasped successfully. The original research results concern specific tasks and setups. They motivate examining ACT, but they cannot establish how many demonstrations your own task requires or how reliably it will recover from a new disturbance.

References: Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware: project · ACT policy documentation at LeRobot v0.6.1

Relate ACT inputs to the task

The documented ACT architecture combines visual features and robot information to predict actions. Its training formulation includes a latent representation; the documented inference procedure uses a zero latent value. Readers need not tune that mechanism to begin a sensible evaluation. The first practical question is whether the observations distinguish the states that require different actions.

Consider a hypothetical gripper that has either securely picked up a block or closed just behind it. If both states look identical in the supplied images and joint readings, the next action may be ambiguous. Changing chunk length cannot add the missing evidence. Review camera coverage and demonstration labels before explaining every failure as an architectural limitation. The camera-placement guide addresses that observation problem separately.

References: ACT policy documentation at LeRobot v0.6.1 · Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware: project

Separate prediction, execution and ensembling

The v0.6.1 ACT configuration uses chunk_size for predicted sequence length and n_action_steps for the actions consumed from a prediction. Its defaults set both to 100 and leave temporal ensembling disabled. These are implementation defaults, not recommendations for a physical control period. Read the saved policy configuration because a downloaded checkpoint may use different values.

Temporal ensembling combines overlapping predictions for the current action. The tagged implementation requires n_action_steps to be one when ensembling is enabled, because it needs a new prediction at every step. This changes compute demand as well as output behaviour. A report that says only chunk size one hundred leaves the execution policy ambiguous; include all relevant temporal settings.

References: ACT configuration at LeRobot v0.6.1 · ACT execution implementation at LeRobot v0.6.1

Translate action steps into time

Step counts are meaningful only with a declared action timing. In an invented example at twenty actions per second, a forty-step prediction represents two seconds of nominal action samples. Consuming ten of those samples before replanning represents half a second of queued execution. Neither figure is an ACT speed specification, and neither guarantees that a real loop maintains the intended period.

Measure actual inference and observation timing as well as the configured frequency. A queued action may originate from an image captured before the most recent scene change. If the environment changes during the queue, the policy's response depends on when new observations are used for a fresh prediction. Keep this distinction visible when someone describes a smooth trajectory as responsive. Smoothness alone says little about timely correction.

References: ACT execution implementation at LeRobot v0.6.1

Understand queued feedback

In the tagged non-ensembled ACT execution path, the model is queried when the action queue is empty; otherwise the next queued action is returned. A new camera frame arriving during that interval does not automatically replace the queue. This behaviour explains why a policy can keep executing a plausible lift even after a missed grasp has become visible.

A useful simulation investigation changes the number of consumed actions while preserving the checkpoint and observation conditions where that comparison is supported. Record the retained prediction length, inference load, action discontinuities and task result. Shorter execution can create more opportunities to respond, but it also requires more frequent predictions and can expose inconsistent replanning. The experiment must decide whether those opportunities actually improve the task.

References: ACT execution implementation at LeRobot v0.6.1

Evaluate temporal ensembling as a separate mode

Overlapping predictions can disagree about the action for the same moment. Ensembling uses their weighted combination, which may reduce abrupt changes in the output. However, older predictions were based on older observations. The useful question is whether the selected weighting preserves the correction needed by the task, particularly after an unexpected object movement.

Compare the mode under the same permitted starts and measure full-loop timing on the target compute device. Record whether inference meets the intended step schedule and whether completion, interventions or visible oscillation change. Do not present the ensemble coefficient as a safety filter or universal stability setting. Any physical trial needs competent review of the entire control arrangement independently of how attractive the plotted actions appear.

References: ACT configuration at LeRobot v0.6.1 · ACT execution implementation at LeRobot v0.6.1 · Maintenance of work equipment

Work a correction experiment

Imagine a simulation where the intended task is to place a block inside a marked region. Compare two supported execution configurations over the same declared starts: one consumes a longer queue and one requests predictions more often. Introduce an approved simulated position change at a defined task phase, then record when the new condition first becomes visible and when the applied action first reflects it.

Keep ordinary attempts separate from disturbance attempts. A configuration could complete undisturbed starts quickly yet fail after the perturbation, while another corrects more often but takes longer. Report both results with counts and failure descriptions. Do not select only a disturbance timing that flatters one configuration. Retain event traces so a reviewer can separate sensing delay, queue behaviour and a prediction that never learned a suitable recovery.

References: Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware: project · ACT execution implementation at LeRobot v0.6.1

Control resets and compare complete behaviour

The v0.6.1 ACT implementation exposes reset behaviour for its action queue or temporal ensemble. An evaluation harness must reset policy state when the environment resets. Otherwise, the start of one attempt can inherit actions or ensemble history from the previous attempt, making results depend on evaluation order rather than the declared initial condition.

Finish the comparison with the checkpoint identity, dataset revision, input features, action representation and complete temporal configuration. State the task conditions under which the result was observed, including what changed and what stayed fixed. ACT's architecture explains possible behaviours; measured attempts determine whether those behaviours serve the project. Use the separate training guide for commands and checkpoint management so this evaluation remains focused on prediction and feedback.

References: ACT execution implementation at LeRobot v0.6.1 · Policy configuration at LeRobot v0.6.1

Checklist

• Record checkpoint identity and actual ACT temporal settings.
• Distinguish predicted length from executed action count.
• Translate action steps using declared and measured timing.
• Check whether the policy can observe a missed grasp or changed target.
• Evaluate queued execution and ensembling as distinct modes.
• Measure correction events as well as smoothness and completion.
• Reset policy state at every environment reset.
• Report ordinary and disturbed attempts with failures retained.

Common questions