Evaluating a diffusion policy in LeRobot

Define why diffusion is a candidate

Diffusion Policy models action sequences through an iterative denoising process conditioned on observations. The original project combines visual conditioning with receding-horizon execution and investigates tasks with more than one plausible action pattern. That makes it a candidate when demonstration behaviour is varied, but the research results do not establish an advantage for every new manipulation problem.

Write a specific hypothesis for your comparison. For example, a task may admit two viable approach directions and the existing baseline may produce inconsistent intermediate behaviour. Ask whether the diffusion implementation improves accepted completion under those conditions. An algorithm's popularity is not a testable hypothesis, and a lower training loss across different objectives is not evidence of better physical behaviour.

References: Diffusion Policy: author project and research references

Pin the implementation and inputs

Use an identified release or commit and keep the saved model configuration with its processors. Here the reference is LeRobot v0.6.1, checked on 6 September 2026. Its diffusion implementation and configuration are the authority for the behaviour described below. The original research repository provides algorithm context, not a promise that every LeRobot parameter has the same meaning or default.

Record dataset revision, feature names, action units, observation history and image processing. The tagged diffusion configuration requires robot state plus an image or environment-state input, and defines action output. Confirm that a candidate dataset supplies the intended meanings, not merely matching shapes. A joint-target dataset cannot be compared fairly with a Cartesian-action baseline without an explicit account of the different control interfaces.

References: Diffusion configuration at LeRobot v0.6.1 · Policy configuration at LeRobot v0.6.1

Separate the temporal horizons

The tagged diffusion policy distinguishes observation history, prediction horizon and executed actions. Its documented queue logic keeps a history of observations, generates a sequence and retains the action segment starting at the current step. The prediction horizon is measured from the earliest observation in that window, so it should not be described simply as that many future control steps.

Use the saved values when calculating timing and confirm the constraint linking executed actions to available predictions. The table identifies the quantities that matter without prescribing operating values. Also record the denoising scheduler and number of inference iterations. Reducing sampling iterations changes the candidate being evaluated; it is not a free acceleration that can be assumed to preserve behaviour.

References: Diffusion configuration at LeRobot v0.6.1 · Diffusion execution implementation at LeRobot v0.6.1

Measure sampling and loop timing

Measure model sampling separately from the full path that supplies observations and applies actions. Include decoding, resizing, transfer to the compute device and any command conversion in the end-to-end record. Distinguish startup from steady-state operation, and retain unusually slow observations rather than reporting only an average. On accelerated hardware, use a timing method that measures completed work rather than only asynchronous submission.

Worked hypothetical example: at twenty applied actions per second, executing eight queued actions represents four-tenths of a second of nominal action samples. That arithmetic does not prove the policy has a four-tenths-second response time. It may act on older images, incur sampling delay or wait in another buffer. Use observed event timing to determine whether a changed object position can influence the action soon enough for the task.

References: Diffusion execution implementation at LeRobot v0.6.1 · Camera documentation at LeRobot v0.6.1

Test stochastic variation deliberately

The tagged implementation accepts noise in its prediction path, and diffusion sampling can yield different sequences. Distinguish an offline repeated-prediction experiment from a full rollout experiment. For offline checks, hold the observation window and checkpoint fixed, vary the controlled sampling input or seed, and compare the generated action sequences. This isolates output variation without confusing it with a changing environment.

Then evaluate complete attempts with recorded initial conditions and randomness settings. Different trajectories may both be acceptable if they satisfy the same task outcome. Conversely, a visually consistent trajectory can fail consistently. Look for whether variation changes contact strategy, leaves the represented task region or produces incompatible intermediate actions. Do not select the most attractive sampled sequence after inspecting the outcome and report it as ordinary autonomous performance.

References: Diffusion execution implementation at LeRobot v0.6.1 · Diffusion Policy: author project and research references

Make the baseline comparison fair

ACT is a useful reference when it already runs through the same task interface, but fairness requires more than using the same episode count. Match the dataset revision, task starts, available observations, success definition and intervention rules. Declare any difference in preprocessing, compute or action representation. Compare both methods under a resource budget that answers the project's actual question.

An equal number of optimiser steps does not imply equal training effort or equal opportunity to tune. Keep a tuning record and reserve final test conditions that were not repeatedly used to choose configurations. When compute allows, use multiple training seeds and report variation between trained models separately from variation between diffusion samples. One favourable checkpoint is weak evidence for choosing an algorithm family.

References: ACT policy documentation at LeRobot v0.6.1 · Diffusion Policy: author project and research references · Training configuration at LeRobot v0.6.1

Work a task-outcome comparison

Consider an invented simulation comparison with thirty matched start conditions. A reference policy completes twenty-one attempts; a diffusion candidate completes twenty-four. Inspection shows that the candidate's three additional completions all occur in one rotated-start condition, while both methods fail when the target is occluded. That supports investigating orientation handling, not claiming that diffusion solved the whole task.

Report completion counts, attempts stopped for intervention, duration among accepted attempts, total experiment time and failure categories. Keep the ordinary test and any stress conditions separate. Repeat the informative rotated-start comparison with fresh starts before expanding the claim. If the candidate also misses action deadlines more often, include that cost in the selection decision rather than hiding it behind the completion percentage.

References: Diffusion Policy: author project and research references · Diffusion execution implementation at LeRobot v0.6.1

Verify reset and checkpoint behaviour

The tagged diffusion reset method clears observation and action queues. Ensure the evaluation harness resets that state whenever the environment resets; otherwise a new attempt can inherit history or queued actions from an earlier one. Verify the first few observations and actions after a reset explicitly, since plausible steady-state behaviour can conceal an incorrect initial history.

Deliver the selected checkpoint with preprocessing, temporal settings, scheduler configuration, dataset identity and the evaluation record. State whether the evidence comes from offline prediction, simulation or approved physical trials. Moving from a simulation to hardware requires competent review of action conversion, timing, calibration and stop arrangements. The diffusion sampler is not a safety controller, and a favourable benchmark cannot replace validation of the installed application.

References: Diffusion execution implementation at LeRobot v0.6.1 · Policy configuration at LeRobot v0.6.1 · Maintenance of work equipment

Checklist

• State the task-specific hypothesis for trying diffusion.
• Pin LeRobot v0.6.1, checkpoint and preprocessing configuration.
• Record observation history, prediction span and executed action count.
• Measure completed sampling and end-to-end timing separately.
• Distinguish sampling variation from training-seed variation.
• Match baseline data, task conditions and scoring rules.
• Clear policy queues on every environment reset.
• Retain failures, interventions and timing costs in the final comparison.

Common questions