xrcollect

Collect robot manipulation demonstrations in VR. Put on a Meta Quest 3, drive simulated Franka arms in MuJoCo with the controllers or your bare hands, and get a lossless, replayable, labelled dataset out, ready for imitation learning (raw HDF5 or LeRobot), with camera images rendered after the fact.

stack_two_blocks · bimanual	duplo_tower · press-fit snapping	open_drawer · single arm, articulated fixture

_{The three bundled tasks, solved here by the built-in scripted expert (xrc validate-task) on the same control stack a human operator uses.}

Why

Teleoperating a real robot for demonstrations is slow, expensive and hard to reproduce. xrcollect moves the whole loop into simulation and into the headset:

• Natural input. 6-DoF controllers or hand tracking, a clutch so you can reposition your hand, motion scaling, a seated mode. Haptics and sound tell you when you grasp, place, snap or succeed.
• Lossless recording. Every step stores the full simulator state, so a take replays bit-exactly. Nothing is thrown away: raw head and hand poses (25 joints per hand) are kept too, so data can be re-retargeted later.
• Images are rendered, not recorded. Pick cameras, resolution and visual randomization at export time and re-render the whole dataset from the stored states. Add a camera next month; old takes get it too.
• Review without taking the headset off. A review card after every take shows whether the dataset would keep it and why; label success, quality and tags, then save or discard from your hands. Replay the take at 1:1 scale in VR and take over from any frame to record a correction.
• A task is data. Robot, scene, objects and success stages are YAML; an optional Python plugin covers the rest. Adding a task never touches platform code, and every episode carries its own task definition.
• One set of curation rules. Idle trimming, contact rejection, failed takes and stage cut-offs are decided in one place and used by the review card, the episode list and every exporter alike.

How it works

flowchart LR
    Q["Quest 3<br/>Quest Browser · WebXR"] -- "binary input frames<br/>(head, hands, buttons)" --> R
    R -- "body poses, every tick" --> Q
    D["Desktop dashboard"] <-- "status · commands" --> R
    subgraph R ["Runtime · one 50 Hz sim thread"]
        direction TB
        T["teleop<br/>filter · clutch · workspace"] --> I["IK<br/>(or a policy)"] --> F["safety filter<br/>no collisions"] --> P["MuJoCo<br/>physics + task stages"]
    end
    P --> H[("episode_*.hdf5<br/>sim/state per step")]
    H --> C["review + curation<br/>curation.json"]
    C --> E["export<br/>HDF5 · LeRobot<br/>cameras re-rendered"]

Loading

The server (Python, FastAPI) owns the simulation. The headset page and the dashboard (TypeScript, three.js, React) are thin clients: they draw the scene from streamed body poses and send every action as a command.

Screenshots

In the headset. HUD with recorder state, stages, session counters and link quality; the goal panel; labels next to each controller button. The right arm is engaged (glow, status chip).	Wrist menu (Y). Input mode, record and save, recenter, motion scale, panels; a Layout page for camera previews and panel placement; a Tools page for replay, scene editing, hand mapping and comfort.
Review card after every take: what a dataset would do with it and why, success, quality stars, tags, save or discard.	Landing page in Quest Browser: pick controllers or hands, then enter VR or mixed reality (passthrough).
Dashboard · Record. Live 3D view, camera previews, the goal and its stages, arm status.	Episodes. Every take with its verdict; replay with a timeline, label, exclude, take over.
Scene. The task editor: drag objects with a gizmo, edit properties, regions, cameras and stages, capture the goal. Edits are written back into the YAML and hot-reload in both clients.	Data. Dataset health and start-position coverage next to the export form, with a live preview of what would be exported.

Contents

• Quick start
• Running a session
• Controls
• The dashboard
• Writing a task
• Data format
• Exporting
• Driving with a policy
• Command reference
• Development

Developers: see docs/architecture.md for threads, the control tick, the wire protocol and the HTTP endpoints.

Quick start

You need Linux with git and curl (or wget), and a GPU driver or Mesa that provides EGL for offscreen rendering. For a USB connection to the headset you also need adb. The headset is a Meta Quest 3 (other WebXR headsets with controllers or hand tracking should work but are untested).

git clone https://github.com/HarunSMetin/MetaQuestDataCollector.git
cd MetaQuestDataCollector
./scripts/install.sh                # everything below, in one go; then:
uv run xrc serve                    # open the headset page (below) and http://localhost:8000/dashboard/

install.sh needs no sudo and is safe to re-run. It:

1. installs uv for your user if it is missing (uv then brings Python 3.12);
2. uses your Node.js if it is 20.19+ or 22.12+, otherwise downloads Node 22 LTS into .tools/node inside the repo (checksum-verified; Node is only needed to build the web client);
3. fetches the Franka Panda model from MuJoCo Menagerie into assets/third_party/;
4. creates .venv with uv sync and builds the web client into web/dist;
5. runs uv run xrc doctor and, if offscreen rendering fails, prints the package to install (sudo apt install libegl1 libgl1 libgles2, or libosmesa6 with MUJOCO_GL=osmesa on machines without a GPU).

Options: --lerobot (LeRobot exporter, brings torch), --mesh (xrc import-mesh), --e2e (Playwright and Chromium for the headset-free tests), --all, --yes (no questions), --no-doctor.

Manual install (what the script does)

./scripts/fetch_assets.sh           # Franka model from mujoco_menagerie -> assets/third_party/
uv sync                             # add --extra lerobot for the LeRobot exporter (brings torch)
                                    # add --extra mesh for `xrc import-mesh`
(cd web && npm ci && npm run build) # needs Node 20.19+ / 22.12+; the server serves web/dist
uv run xrc doctor                   # checks assets, EGL, the web build, protocol files, adb

No headset at hand? uv run xrc view tasks/stack_two_blocks opens a task in a desktop viewer, and uv run xrc validate-task tasks/stack_two_blocks watches the scripted expert solve it.

Running a session

uv run xrc serve                              # the task from config.yaml
uv run xrc serve --task tasks/duplo_tower     # another task
uv run xrc serve --mode hands                 # start in hand-tracking mode
uv run xrc serve --out data/session_02        # write takes somewhere else

There are two ways to connect the headset:

• USB (simplest). Enable developer mode on the Quest and plug it in. Run uv run xrc setup-quest, which sets up an adb reverse port forward; run it again after replugging. Then open http://localhost:8000/xr/ in Quest Browser.
• Wi-Fi. WebXR needs HTTPS off localhost. Run uv run xrc certs once, start xrc serve, and open the printed https://<lan-ip>:8443/xr/. Accept the self-signed certificate.

Open the dashboard on the desktop at http://localhost:8000/dashboard/. In the headset press Enter VR, or Enter Mixed Reality to see the simulated table in your room (passthrough).

A take goes like this:

1. Press A (record). After a 3 s countdown the recording starts.
2. Do the task. When success holds for a second, the take stops by itself; you can also press A to stop.
3. The review card shows what will be kept and why. Set success, tags and a quality score, then save (X) or discard (B).

Controls

Controller buttons are served by the server (GET /api/commands). The headset shows them as labels on the controller models; turn them off with Button hints in the wrist menu.

Input	Action
Grip (either hand)	Toggle: that hand moves its arm (clutch). Release, reposition, grip again to keep going.
Trigger	Close the gripper (analog)
A (right)	Record / stop, or cancel during the countdown
B (right)	Discard the take
X (left)	Save the take
Y (left)	Wrist menu
Left stick press	Recenter the robots in front of you
Ray + Trigger	Press panel buttons; grab a panel by its title bar, then push or pull it with the stick

Hands mode: curl your middle, ring and little finger into your palm and open them again to toggle the clutch (one curl = one toggle; index and thumb stay free). The thumb–index distance sets the gripper, and it holds still while the three fingers are curled, so the gesture never drops or grabs anything. Look at your left palm for the menu. If the gesture triggers too easily or not at all on your hands, tune teleop.curl_on / curl_off.

Control modes (wrist menu → Tools, or teleop.control in config.yaml):

Mode	The arm's target
clutched (default)	where it was when you engaged, plus your hand's motion since, times the motion scale. Like lifting a mouse.
absolute	your hand: the gripper goes where the hand is
rate	moves with a velocity proportional to the hand's offset from where you engaged; for large workspaces

In every mode the hand's rotation since engaging is applied to the gripper, so re-engaging never makes it jump.

Wrist menu:

• Main: input mode (controllers or hands), record, save as success or failure, discard, recenter, speed (motion scale), new scene, sound, button hints, camera view, camera previews.
• Layout: which camera previews show and how (column, row, grid); which panels show (status, goal, previews) and whether each stays in the world, follows your body or rides on your wrist; lock panels or reset the layout.
• Tools: replay the last take (opens the timeline: play, pause, seek, speed, step, take over here) or show its ghost; edit the scene (move objects, capture the goal); goal guide; swap hands (left-handed use, or both hands on one arm); control mode; table height (seated preset).

The headset gives feedback as haptics and sound cues on grasp, release, stage reached, snap clicks and success. The safety filter buzzes when it has to change a joint target (an arm about to hit the table or the other arm). It never helps with the task.

The dashboard

• Record: a live 3D view, camera previews, the goal and its stages, session counters, record controls, the operator name and auto-stop.

• Episodes: every take with its verdict (kept, cut, or left out, and why). Replay a take in the headset, label it (success, tags, quality, time annotations), or exclude it. Labels go to curation.json next to the episodes, and the episode files are never rewritten.

• Policy: load a policy runner, run it, share control, or stop it. Record its rollouts like any other take.

• Scene: the task editor. Switch tasks from the list at the top. In edit mode physics pauses:

  • Click an object in the 3D view and drag the gizmo (W to move, E to turn). Apply moves writes the new poses into the task file.
  • Edit an object's asset, mass, friction, colour and start region. Add or remove objects, regions and cameras.
  • Edit the stages and success predicate.
  • Capture goal records the current arrangement as the goal: it adds *_placed stages and a goal view.

  Each edit is written into the YAML (comments kept) and the scene reloads in both clients.

• Data: dataset health (takes, success rate, usable time, mixed task versions) and start-position coverage per object, plus take durations, success over time, contacts and operators. Next to it is the export form, with a live preview of what would be exported.

Writing a task

A task folder holds task.yaml, plus an optional task.py plugin and expert.py. It references four layers:

Layer	Where	What
embodiment	embodiments/*.yaml	robot adapter, arms and mounts, home pose, limits, safety margins
scene	scenes/*.yaml	table, props, lights, cameras
objects	objects/**.yaml	reusable assets: shape, size, mass, friction, geoms (or a builder / MJCF / mesh)
task	tasks/<name>/task.yaml	which objects, regions, start distribution, stages, success, events, goal view

Two embodiments ship: franka_bimanual (two Franka Panda arms) and franka_single. A complete task (tasks/stack_two_blocks/task.yaml):

task: stack_two_blocks
instruction: "Put the red block in the taped area, then the blue block on top of it."
embodiment: franka_bimanual
scene: table
objects:
  red_block:  {asset: blocks/cube_4cm, rgba: [0.85, 0.1, 0.1, 1], mass: 0.05, label: red block}
  blue_block: {asset: blocks/cube_4cm, rgba: [0.1, 0.3, 0.9, 1], mass: 0.05, label: blue block}
regions:                     # named areas, drawn as tape (visible to the cameras)
  build_area: {center: [0.50, 0.0], half: [0.06, 0.06]}
spawn:                       # start-state distribution, per object
  red_block:  {region: {center: [0.42, 0.22], half: [0.04, 0.04]}, yaw_deg: [-15, 15]}
  blue_block: {region: {center: [0.42, -0.22], half: [0.04, 0.04]}, yaw_deg: [-15, 15]}
stages:                      # ordered progress; each is a predicate over the simulator state
  - name: red_in_area
    label: red block in the area
    when: {all: [{inside: {object: red_block, region: build_area}}, {upright: red_block}]}
  - name: blue_on_red
    label: blue block on the red block
    when: {all: [{stage: red_in_area}, {on_top: {object: blue_block, base: red_block, tol_xy: 0.012, tol_z: 0.004}}]}
success:                     # omit: all stages done, nothing held, everything at rest
  all: [{stage: blue_on_red}, {not_held: any}, {at_rest: {objects: all, max_speed: 0.02}}]
events:                      # task feedback -> cues (config.yaml feedback.cues)
  - {on: stage_reached, cue: placed, toast: "{stage}"}
  - {on: success, cue: complete}
goal_view:                   # what the goal panel draws, relative to the anchor object
  anchor: red_block
  show:
    - {object: blue_block, pose: {pos: [0, 0, 0.04], yaw: 0}}

World frame: z up, +x away from the operator, +y to the operator's left, metres. An object's body frame is the bottom centre of its bounding box.

Predicates (xrcollect/task/predicates.py; the Scene tab lists them with templates):

Predicate	Holds when
inside	the object's whole footprint lies inside a region
on_top	the object rests on top of the base object, centred within tol_xy
pose_near / relative_pose	the object is at a world pose / at a pose in the base object's frame
contact	two bodies touch (objects, table, arm:<arm>, gripper:<arm>)
held / not_held	the object is (not) held, optionally by an arm; not_held: any = hands empty
at_rest	the objects move slower than max_speed
joint_at	a fixture joint (drawer, door) is at a value, or beyond min / max
upright	the object's z axis points up
snapped	the press fit between two objects is engaged
count	at least min (at most max) of the objects satisfy when ($ is the object)
all / any / not / stage	combinators; stage refers to an earlier stage

Predicates must depend on the simulator state only, because replay, review and export recompute them from recorded states.

Objects live in a shared library, referenced as asset:. A box is a few lines:

# objects/blocks/cube_4cm.yaml
label: cube
shape: box
size: [0.04, 0.04, 0.04]     # bounding box x, y, z
mass: 0.05
friction: 0.8
rgba: [0.7, 0.7, 0.7, 1]
geoms:
  - {type: box, size: [0.02, 0.02, 0.02], pos: [0, 0, 0.02]}

You can also build an object procedurally with builder: brick.py:brick and builder_args (see objects/duplo/), use an MJCF fixture with joints (objects/furniture/drawer.yaml), or import a mesh with uv run xrc import-mesh mug.obj kitchen/mug, which splits it into convex pieces. A task can override mass, friction, rgba, label and builder_args per object.

Press fits (DUPLO-style) are welds that switch on when two parts touch, line up and are pressed together, and switch off when pulled apart hard enough (pull_n). See tasks/duplo_tower.

Plugins: for logic YAML can't express, plugin: task.py:MyTask subclasses DeclarativeTask (xrcollect/task/declarative.py). Override build / bind / reset / stage_flags / after_step and so on. Any state the plugin keeps outside MuJoCo must go through extra_state / set_extra_state, or replays will diverge. tasks/duplo_tower/task.py is the worked example: a stud-grid snap test and a tower that may stand either way round.

Check a task before collecting with it:

uv run xrc view tasks/my_task              # look at it (and its start distribution)
uv run xrc validate-task tasks/my_task     # a scripted expert must reach success, from several seeds

The expert plans pick-and-place moves from the stages. For anything else, write expert.py with def script(expert) (see tasks/open_drawer/expert.py).

Every edit changes the task's task_version, a hash of all four layers. Episodes store their whole task bundle and compiled scene, so old takes still replay after you change the task. Exports refuse to mix versions unless you pass --allow-mixed.

Data format

One HDF5 file per take: <record.out_dir>/episode_XXXXXX.hdf5 (default data/episodes/). Time is axis 0; n = arms, J = joints per arm, k = objects. Images are not stored: sim/state replays the take exactly, and exporters render cameras from it.

Key	Shape	Content
time, wall_time	(T,)	sim time, host time [s]
obs/joint_pos, obs/joint_vel	(T, n·J)	arm order = attr arms
obs/gripper_width	(T, n)	finger opening [m]
obs/ee_pose	(T, n, 7)	TCP [x y z qw qx qy qz], world
obs/ee_wrench	(T, n, 6)	external force [N] and torque [Nm] on each hand
obs/objects	(T, k, 7)	object poses (privileged), order = attr objects
obs/fixture_joints	(T, f)	drawer / door joints
action/joint_pos	(T, n·(J+1))	per arm: J joint targets + gripper (0 closed … 1 open), after the safety filter
action/ee_pose	(T, n·7+n)	the TCP poses of those targets, plus grippers
sim/state	(T, S)	full simulator state before each step (MuJoCo + task extra state)
task/success, task/progress	(T,)	success flag, stages done
task/stages	(T, s)	each stage's predicate, order = attr stage_names
task/held	(T, n)	object index held by each arm (−1: none)
contacts	(T, 3)	link–table, arm–arm, fingertip–table contact, order = attr contact_kinds
control/source	(T,)	who drove: 0 operator, 1 policy, 2 script
safety/correction	(T,)	largest joint-target change the safety filter made [rad]
teleop/*	…	raw XR head and hand poses, buttons, 25 hand joints, pinch, engaged, tracked, RTT, input rate

File attributes include schema_version, task, task_version, instruction, operator, session, input_mode, source, success, fps, seed, model_xml, task_yaml (the resolved task bundle), config_yaml, action_names and stage_names. The full list, with branching and derived-episode fields, is in the docstring of xrcollect/recorder.py.

uv run xrc inspect ep.hdf5 prints a summary. uv run xrc resim ep.hdf5 replays the recorded actions from the first state and checks the result matches.

Manual labels (success override, tags, quality, notes, exclusions, time annotations) live in curation.json in the same folder. Labels never touch the episode files.

DART noise (record.action_noise) perturbs the executed targets, not the recorded ones, so the operator's corrections end up in the data. The executed targets can be recovered from sim/state.

Exporting

Which steps make it into a dataset is decided in one place, the curate section of config.yaml. The review card, the Episodes tab and every exporter use the same rules:

• Failed takes are left out unless include_failures is set.
• Idle time at the start and end is trimmed.
• Takes with too much table or arm contact are rejected, or just the contact moments are cut out.
• For shared-control takes, only the operator's steps are kept.
• A take can be cut at a given stage.

uv run xrc export my_set                                   # raw HDF5 with images -> data/exports/my_set
uv run xrc export my_set --exporter lerobot --cameras cam_high,cam_left_wrist
uv run xrc export my_set --cameras none --annotations      # state only; time annotations as segments

_{The scene cameras (cam_high, cam_low, both wrists) as an exporter renders them from sim/state: flat, matte light, no shadows or operator aids. Optional per-episode visual randomization varies lighting, colours and camera mounts.}

You can also export from the Data tab. Every export writes xrcollect_manifest.json: its feature layout, the episodes and spans used, the curate rules and the git commit. Channels that never change are flagged, since a policy input normalized by ~0 blows up at run time. The lerobot exporter needs uv sync --extra lerobot.

Driving with a policy

A policy runner is any class with reset(env) and act(env) -> (joint targets, grippers); see xrcollect/policy.py. xrcollect imports no ML library itself. Load a runner at start-up, or from the Policy tab:

uv run xrc serve --policy my_pkg.policies:Runner              # or path/to/file.py:Runner
uv run xrc serve --policy my_pkg.policies:Runner --control shared

The control modes:

• Run: the policy drives.
• Shared: the policy drives until you press Grip, then you do (DAgger-style corrections).
• Stop: you drive.

Policy actions go through the same safety filter. Each recorded step says who drove. A policy can check its expected feature layout against the export manifest it was trained on.

Command reference

Command
xrc serve	the server (--task, --mode, --config, --out, --no-https, --policy, --control)
xrc setup-quest / xrc certs	USB port forward / TLS certificate for Wi-Fi
xrc doctor	environment checklist
xrc tasks / xrc view [task]	list tasks / open a task in a viewer
xrc validate-task <task>	the scripted expert must reach success (--n, --embodiment, --save)
xrc replay / inspect / resim <ep.hdf5>	watch / summarize / bit-exact re-simulation check
xrc audit	dataset verdicts; --exclude ep.hdf5 --note why
xrc export <name>	build a dataset (--exporter, --cameras, --annotations, --allow-mixed)
xrc import-mesh <file> <asset>	mesh → object library entry
xrc codegen [--check]	regenerate the wire-protocol files

Platform settings (ports, teleop filtering and control mode, IK, recording, curation, safety, feedback cues) are in config.yaml. The world itself (robot, scene, objects, task) is never configured there.

Development

xrcollect/          Python backend: runtime (sim thread), server, teleop, IK, safety, recorder, audit, export, task/, scene/
web/src/xr/         headset page (three.js + WebXR): panels, wrist menu, review card, timeline, edit mode
web/src/dashboard/  desktop dashboard (React): Record, Episodes, Policy, Scene, Data
embodiments/ scenes/ objects/ tasks/   the four task layers
tests/              pytest suite
web/e2e/            headset-free end-to-end tests (emulated Quest 3) and the README screenshots
docs/               architecture.md, media/

• Tests: uv run --group dev pytest (about a minute), and cd web && npm run typecheck.
• Without a headset, web/e2e/ drives the real pages with Meta's IWER emulated Quest 3 in headless Chromium (npm test) and checks the dashboard's Scene and Data tabs (npm run test:dashboard). Install the tools with ./scripts/install.sh --e2e. Setup is at the top of each script; point them at a scratch copy of a task and a scratch record.out_dir, never at data/episodes.
• npm run dev in web/ runs Vite on :5173 with /api and /ws proxied to the server on :8000. The server itself serves web/dist, so run npm run build after web changes. If install.sh downloaded Node, it is in .tools/node/bin: PATH="$PWD/.tools/node/bin:$PATH" npm --prefix web run build.
• Screenshots in docs/media/ come from web/e2e/screenshots.mjs (instructions at its top). The GIFs are the scripted expert rendered offscreen with Expert.on_step.

Contributions are welcome. Read docs/architecture.md first; its last table lists which files have to change together.

Acknowledgements

The Franka Panda model comes from MuJoCo Menagerie (fetched by scripts/fetch_assets.sh, not redistributed here). Built on MuJoCo, three.js, IWER for headset-free testing, and LeRobot for the dataset format.

License

MIT