Object3d device fixes by Peter-Pater · Pull Request #487 · google/xrblocks

Peter-Pater · 2026-08-05T01:59:58Z

Object3d device fixes (on top of PR 417)

Description

Continues #417, which extracted the objects_3d pipeline into the objects3d addon. The extraction was right, but the pipeline didn't work on a headset: boxes were the wrong size, in the wrong place, all facing the same direction.

Root cause — the wrong camera. Rays were cast through a clone of the XR render camera (the ~90° union frustum between the eyes) while the pixels came from the physical passthrough camera (~48° FOV, near the right eye). Different intrinsics, different pose, so every ray left at the wrong angle.

Build the frozen camera from the SDK's device-camera model, falling back to the old clone only when unavailable — simulator behaviour is unchanged.
Wait for a fresh video frame and pair it with the pose at its captureTime; captures were pairing seconds-old pixels with the current head pose.
Enable matchDepthView (the demo overrode the SDK default off), so the depth mesh every ray lands on is no longer rotated relative to the view.
Drop the unconditional 90° yaw snap — no confidence gate, up to 45° error.
Snap instead to the room's wall direction, estimated from depth-mesh vertical normals. The old grid was the session origin, i.e. wherever the user happened to face at startup.
Estimate yaw from a minimum-area rectangle over the convex hull, gated on scatter confidence; PCA alone lands 45° off when two faces are visible.
Fix a sign bug the snapping hid: extents measured along (cos a, +sin a), drawn along (cos a, −sin a).
Replace the 2,452-line inline demo copy with the addon page (929 lines);

Orientation is selectable via {mode: 'roomFrame' | 'free' | 'cardinal'}, defaulting to roomFrame, which degrades to today's output if estimation fails.

Type of Change

Bug fix
New feature / enhancement
New demo or sample
Documentation update

Checklist

Tested in simulator & device: Verified functionality in desktop simulator and/or physical hardware (where applicable).
Large Assets ($\ge$ 1MB): Submitted separately to xrblocks/proprietary-assets via jsdelivr CDN.
SDK Dynamic Dependencies: All new SDK dependencies are dynamically loaded at runtime.
Security: Confirmed no hardcoded API keys or secrets are committed.

Disclaimer:

Real-world performers suffers from small objects, sensor accuracy, and object cluttering, so it is not as perfect as in the simulator.

dli7319 · 2026-08-05T03:39:11Z

Should we merge 417 first?

Peter-Pater · 2026-08-05T15:00:32Z

Should we merge 417 first?

I am not entirely sure what the standard practice should be here since this is a follow-up PR. But maybe it is good to merge 417 first? This PR inherits all the commits previously made by @salmanmkc, so it should still merge correctly after 417 is merged.

ruofeidu · 2026-08-05T20:00:45Z

Thank you Xincheng!

If this is merged, #417 can be closed I guess?

salmanmkc · 2026-08-05T20:09:39Z

i recommend making this a stacked pr, it came out 6 days ago:

it let's you see them both at the same time, #417 will be under and this on top and then they can both merge separately but in the same stack

or merge 417 first + rebase of main, it has another few commits that this pr doesn't have yet (probably conflicts)

dli7319 · 2026-08-05T21:43:10Z

@Peter-Pater Can you rebase onto main?

Peter-Pater · 2026-08-05T22:16:12Z

@Peter-Pater Can you rebase onto main?

Done :)

salmanmkc · 2026-08-05T22:53:17Z

Thanks for this @Peter-Pater! Was this on Galaxy XR that you tested this on?

dli7319

Tested it on Galaxy XR.
It's technically working but the whole app intermittently freezes on device during the detection.
And the WebGPU errors are still there.

ruofeidu · 2026-08-05T23:35:33Z

+/** A mesh of only horizontal (floor) triangles. */
+function floorMesh(): THREE.Mesh {
+  const positions: number[] = [];
+  for (let i = 0; i < 20; ++i) {


nit: comment on magic number

/*room_dim_m=*/20

or const ROOM_DIM_N = 20

ruofeidu · 2026-08-05T23:35:50Z

+  });
+
+  it('recovers a rotated room', () => {
+    for (const d of [17, 30, 41]) {


nit: magic number

const d of /room_dimension=/[17, 30, 41]
or const

ruofeidu · 2026-08-05T23:38:38Z

+const DEFAULTS = {
+  maxTriangles: 20000,
+  maxEdge: 0.2,
+  maxAbsNy: 0.25,


comment on this?

ruofeidu · 2026-08-05T23:38:45Z

+
+const DEFAULTS = {
+  maxTriangles: 20000,
+  maxEdge: 0.2,


what's the range, or a threshold?

dli7319 · 2026-09-14T21:39:09Z

@Peter-Pater Are you still planning on merging this?

Peter-Pater · 2026-09-15T00:28:50Z

@Peter-Pater Are you still planning on merging this?

Hi, yeah sorry was blocked by the paper crunch. Yeah, let me test with the new xrblocks version and then we can merge.

ruofeidu · 2026-09-15T03:24:26Z

Thanks, I think we can more easily resolve all upgrades and conflicts!

PR 417 was using the virtual camera's camera parameter rather than the physical ones. Also improved the real-world boundingbox performance by snapping it to the room frame rather than cardinal axises

…e raycast (cherry picked from commit 48d764862d400990729c2d164a1024854061b03d)

detect() awaits a fresh video frame, rebuilds the full-resolution depth mesh before cloning it, and asks the renderer for the XR camera when building the device frozen camera. The per-call-input test's xrblocks mock predates all of that, so it threw on depth.updateFullResolutionDepthMesh. Stub those (the device-camera model returns null, so the render-camera fallback is exercised) and wait for both backend requests instead of a single microtask.

Replace the inline detection pipeline with Object3DDetector and rebuild the XR control panel on the core UI (UICard root, UIPanel rows, UIButton / UIText / UIIcon leaves) now that the uiblocks addon is gone. The panel mirrors the DOM controls: detect / clear / copy diagnostics, live camera rotation-offset nudges, orientation mode, and the reload pickers for the depth toggles, detector and mask backends, plus a live diagnostics column fed from detector.diagnostics. Keeps the Gemini key overlay and the detection-prompt / MediaPipe threshold tuning from the inline version.

… unusable raycast" Superseded upstream: ca4910e caps the raycaster's line and point threshold at 1 cm in HitRegistry, which removes the 1 m Line-threshold trap that made the debug wireframes swallow every pointer ray. The per-object raycast opt-out is no longer needed. This reverts commit 5f05dda.

The page's key overlay only read {"key": ...} from ./keys.json, while xb.core.ai loads the same file as {"gemini": {"apiKey": ...}}. With an SDK-shaped file the overlay therefore blocked the page even though AI would have initialised fine. Accept both shapes.

The core UI style validator rejects unknown properties, and flexWrap is not one it knows. The throw inside DebugPanel.init() marked the script failed without any console output, so the floating panel silently never appeared in the simulator's 3D view or in XR while the DOM panel kept working. Verified via a headless probe: the card now mounts (364 UI meshes) and no script is in the failed set.

Review follow-ups: explain each RoomFrame default (triangle budget, edge gate, |normal.y| threshold with its angular meaning, support area, range) with the range it is tuned for, and replace the magic numbers in the RoomFrame tests (floor grid size / cell size, rotated-room yaws) with named constants.

The XR card is now just what you use while wearing the headset: title, status, Detect / Clear, and the detector, mask and orientation pickers. The developer knobs (rotation-offset nudges, depth toggles, the live diagnostics column, copy-diagnostics) stay on the DOM panel, where they are read at a desk. Also drops the per-frame diagnostics refresh, so the card does no work between detections.

Bump every font on the control card by two to three points (title 26, status and button labels 17, picker captions 12) and the icons to 26 px. The card keeps its size; it had room to spare.

Upstream now types core.renderer as WebGLRenderer | WebGPURenderer, and the WebGPU XR manager returns a plain ArrayCamera from getCamera(), which no longer satisfies the WebXRArrayCamera parameter of the device-camera helpers. Same cast upstream applied in PlanarVST.

Peter-Pater · 2026-09-16T23:22:00Z

@dli7319 I migrated this PR to xrblocks 0.21 and conducted a few tests. It should be ready to merge now.

ruofeidu requested a review from dli7319 August 5, 2026 19:59

Peter-Pater force-pushed the object3d_device_fixes branch from e63078b to 35f82d6 Compare August 5, 2026 22:14

dli7319 approved these changes Aug 5, 2026

View reviewed changes

ruofeidu approved these changes Aug 5, 2026

View reviewed changes

ruofeidu approved these changes Aug 6, 2026

View reviewed changes

salmanmkc mentioned this pull request Sep 12, 2026

Allow per-call object detection snapshots and backends #580

Merged

8 tasks

Peter-Pater added 4 commits September 15, 2026 14:46

Fixed PR 417 by using the correct cam parameters

ae9d905

PR 417 was using the virtual camera's camera parameter rather than the physical ones. Also improved the real-world boundingbox performance by snapping it to the room frame rather than cardinal axises

fix boundingbox raycast - causing performance degradation and unusabl…

5f05dda

…e raycast (cherry picked from commit 48d764862d400990729c2d164a1024854061b03d)

Peter-Pater force-pushed the object3d_device_fixes branch from 35f82d6 to 228450a Compare September 15, 2026 22:18

Peter-Pater added 8 commits September 15, 2026 18:26

Merge branch 'main' into object3d_device_fixes

1125e8d

demo(objects_3d): larger type on the XR card

2d38745

Bump every font on the control card by two to three points (title 26, status and button labels 17, picker captions 12) and the icons to 26 px. The card keeps its size; it had room to spare.

Merge branch 'main' into object3d_device_fixes

9ba235c

dli7319 merged commit 9d24ea1 into google:main Sep 16, 2026
11 checks passed

Conversation

Peter-Pater commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Object3d device fixes (on top of PR 417)

Description

Type of Change

Checklist

Disclaimer:

Uh oh!

dli7319 commented Aug 5, 2026

Uh oh!

Peter-Pater commented Aug 5, 2026

Uh oh!

ruofeidu commented Aug 5, 2026

Uh oh!

salmanmkc commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dli7319 commented Aug 5, 2026

Uh oh!

Peter-Pater commented Aug 5, 2026

Uh oh!

salmanmkc commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dli7319 left a comment

Choose a reason for hiding this comment

Uh oh!

ruofeidu Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

ruofeidu Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

ruofeidu Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

ruofeidu Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

dli7319 commented Sep 14, 2026

Uh oh!

Peter-Pater commented Sep 15, 2026

Uh oh!

ruofeidu commented Sep 15, 2026

Uh oh!

Peter-Pater commented Sep 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Peter-Pater commented Aug 5, 2026 •

edited

Loading

salmanmkc commented Aug 5, 2026 •

edited

Loading

salmanmkc commented Aug 5, 2026 •

edited

Loading