Audit Robot Demonstration Data Before Training a Policy

Define what an accepted episode means

Write a dataset acceptance contract before choosing episodes. Specify the task, permitted initial conditions, recorded observation names, action meanings, termination rule, and outcome labels. Distinguish a technically valid recording from a successful demonstration: a failed attempt may contain intact measurements but still be unsuitable for the intended imitation target. Keep that distinction visible so that the dataset owner can explain inclusion decisions later.

The technical reference here is LeRobotDataset v3.0 in main documentation, checked September 5, 2026. It describes a schema and metadata alongside recorded signals and videos. Use those facilities as the foundation for your own quality contract, not as an automatic certificate of learning value. Give reviewers a small set of annotated examples representing acceptance, rejection, and uncertainty. Resolve disagreements in the labeling rule before asking people to review a long recording session.

参考资料： LeRobotDataset v3.0, main documentation

Check complete episodes and preserve originals

In the documented v3.0 layout, multiple episodes can share Parquet and video files, with metadata locating episode boundaries. Consequently, a file count is not an episode count, and deleting an inconvenient video file is not a sound episode-filtering method. Work through the matching dataset tools or loader, preserve the original recording, and record every transformation as a new dataset revision with a reason.

For each audited episode, verify that required features are present, values have the expected interpretation, and the beginning and ending belong to the intended attempt. Inspect missing values, abrupt discontinuities, truncated endings, and duplicate-looking stretches. Do not automatically smooth an unexplained jump: first determine whether it reflects a measurement error, an actual action, or an episode boundary mistake. Keep a quarantine list containing episode identifiers, observed defects, reviewer decisions, and any remaining uncertainty about the cause.

参考资料： LeRobotDataset v3.0, main documentation

Audit visibility and camera identity

Watch the contact phase as carefully as the approach. A wide view may establish where an object starts while concealing whether the tool has actually engaged it. Ask what evidence the policy receives at the moment it must choose an action. Review occlusion, motion blur, changing exposure, and backgrounds that could accidentally reveal the task label. These are questions about the proposed learning problem, not reasons to declare any particular camera inadequate.

The LeRobot camera guide warns that device identifiers can change after reconnection or reboot, depending on the operating system. Add a session-opening check that associates each recorded view name with a recognizable scene. Keep reference images in the internal session record and note deliberate camera changes. If a view moves, decide whether the session belongs in a different experimental condition. A valid image stream under the wrong name can be harder to notice than a missing stream.

参考资料： LeRobot Cameras, main documentation

Separate nominal timestamps from capture evidence

Do not infer physical synchronization merely because timestamp columns are orderly. The checked LeRobot recording tutorial explains that frame timestamps can be derived from frame index, so an underperforming recording loop can disagree with real elapsed time. Retain the capture diagnostics available in your installed revision and compare them with observed session duration. Describe a timing problem explicitly instead of presenting the configured recording rate as a measured result.

When ROS 2 is part of the capture path, inspect publisher and recorder compatibility. The official Jazzy QoS documentation explains that incompatible policies can prevent delivery. A reasonable audit therefore asks whether each required stream was actually received, whether gaps occurred around important events, and which clock produced each time field. Do not correct offsets by eye and call the result ground truth. Preserve the original timestamps and document any alignment estimate, its method, and its uncertainty.

参考资料： Imitation Learning on Real-World Robots, main documentation · Quality of Service settings, official ROS 2 Jazzy documentation source

Review behavior and outcome labels independently

Review whether the action sequence expresses the behavior you intend to learn. Long idle stretches, unrelated repositioning, hidden human help, and an inconsistent completion rule can make a technically intact episode ambiguous. Decide whether recovery behavior belongs in this training task. An expert correcting a mistake may provide valuable material for one objective and a conflicting target for another. Preserve the context instead of treating every successful ending as equivalent.

Use outcome labels that support investigation: completed without intervention, completed after intervention, incomplete, and uncertain are possible editorial starting points. Adapt them to the task and label observable events rather than inferred intent. In a hypothetical tray task, a block partly supported by the gripper is not clearly placed just because it appears above the tray. Reviewers should apply the same release-and-rest criterion, while the responsible integration team defines any physical testing and reset procedure.

参考资料： Imitation Learning on Real-World Robots, main documentation

Protect evaluation from near-duplicate material

Create splits around the independence claim you want to test. Randomly dividing neighboring frames can place almost the same scene in training and evaluation. Even whole episodes may remain closely related when recorded consecutively with unchanged objects and presentation. Group recordings by session, object instance, operator, or scene configuration where those factors matter to the intended claim. Document which differences are held out and which remain shared.

A hypothetical team evaluating a new collection day should hold out that day's episodes together, then explain that the result concerns that day's conditions. It should not describe the result as generalization to every environment. Preserve split membership in an explicit manifest rather than regenerating it silently whenever files change. Derive preprocessing choices from the training portion where applicable, and record exceptions. LeRobot supplies episode access; the independence of your evaluation remains an experimental design responsibility.

参考资料： LeRobotDataset v3.0, main documentation

Worked hypothetical example: spend the next session deliberately

Imagine an audit finds three distinct problems: one session uses swapped camera names, another contains operator interventions without labels, and a third lacks demonstrations near the edge of the declared starting area. These are invented findings for illustration. The first needs a verified mapping or quarantine; the second needs annotation review; the third may justify new collection. Treating all three as a request for more data would leave the original defects unresolved.

Write an action record for each finding with owner, evidence, proposed correction, and verification step. If a transformation repairs names, open corrected episodes through the same loader used for training and compare against the source material. If fresh collection is justified, specify the missing condition and review it immediately after the session. This creates a direct connection between an audit finding and the next expense, rather than a vague target for increasing dataset size.

参考资料： LeRobot Cameras, main documentation · LeRobotDataset v3.0, main documentation

Attach a quality record to every dataset revision

An internal dataset release should identify its source sessions, inclusion rules, known defects, split definition, transformations, and reviewer. State whether the audit examined every episode or a sample, and describe how that sample was chosen. List unresolved limitations in terms useful to a future experimenter: a view was absent in a session, a particular condition was never demonstrated, or capture timing was not independently measured. Avoid a single unexplained quality score.

ROS 2's recording tutorial provides a way to retain and inspect message data; a bag can complement the audit when ROS is involved, but it does not replace the learning dataset's semantics. Keep playback disconnected from physical command consumers unless a qualified team has approved that operation. For private recordings, check access and upload settings before sharing the release. The deliverable is evidence of what the dataset contains and can support, including what remains unknown.

参考资料： Recording and playing back data, official ROS 2 Jazzy documentation source · Imitation Learning on Real-World Robots, main documentation

检查清单

• Define acceptance, intervention, completion, and uncertainty labels.
• Preserve original recordings and identify every derived dataset revision.
• Inspect complete episodes through the intended training loader.
• Match each camera name to its actual view at session start.
• Review real capture diagnostics as well as nominal timestamps.
• Record evaluation split membership and the independence claim it supports.
• Convert each audit finding into a repair, quarantine, or collection decision.

常见问题