Follow-up to issue #1, for layer 2 (spatial understanding) once the core MVP loop is proven. Depends on #1.
Decision
This must work as a consumer product: open the app, point the camera, talk — no pre-scan step, no LiDAR session the user has to perform deliberately.
On devices with a LiDAR sensor (iPhone Pro models), ARKit builds a live scene mesh automatically in the background as the user naturally moves the phone around. No explicit "scanning mode."
On devices without LiDAR, fall back to ARKit visual-inertial tracking (rougher pose/point cloud, but still usable).
Spatial confidence builds up opportunistically over the first few seconds of normal use, not from a dedicated capture flow.
GPS/geolocation is coarse context only (where in the world the user roughly is) — it is not used for the fine-grained "where is this wall relative to the camera" reasoning, which comes from the live ARKit mesh/pose.
Scope
Integrate ARKit scene reconstruction (ARWorldTrackingConfiguration with sceneReconstruction where available) into the client.
Detect LiDAR availability and gracefully fall back when absent.
Feed the live mesh/point cloud to the backend alongside frame + pose (building on the #1 transport decision).
No UI "scan this room first" step — localization quality should just improve as the user moves naturally.
Out of scope
Any pre-scan/offline capture workflow.
Photogrammetry pipelines (explicitly not pursuing this — opportunistic ARKit mesh is the chosen approach).
Report back
Comment with: whether LiDAR vs. non-LiDAR devices were tested, how spatial confidence was measured/observed over time, and whether the "no explicit scan step" requirement held up in practice.
## Context
Follow-up to issue #1, for layer 2 (spatial understanding) once the core MVP loop is proven. Depends on #1.
## Decision
This must work as a consumer product: open the app, point the camera, talk — no pre-scan step, no LiDAR session the user has to perform deliberately.
- On devices with a LiDAR sensor (iPhone Pro models), ARKit builds a live scene mesh automatically in the background as the user naturally moves the phone around. No explicit "scanning mode."
- On devices without LiDAR, fall back to ARKit visual-inertial tracking (rougher pose/point cloud, but still usable).
- Spatial confidence builds up opportunistically over the first few seconds of normal use, not from a dedicated capture flow.
- GPS/geolocation is coarse context only (where in the world the user roughly is) — it is not used for the fine-grained "where is this wall relative to the camera" reasoning, which comes from the live ARKit mesh/pose.
## Scope
- Integrate ARKit scene reconstruction (`ARWorldTrackingConfiguration` with `sceneReconstruction` where available) into the client.
- Detect LiDAR availability and gracefully fall back when absent.
- Feed the live mesh/point cloud to the backend alongside frame + pose (building on the #1 transport decision).
- No UI "scan this room first" step — localization quality should just improve as the user moves naturally.
## Out of scope
- Any pre-scan/offline capture workflow.
- Photogrammetry pipelines (explicitly not pursuing this — opportunistic ARKit mesh is the chosen approach).
## Report back
Comment with: whether LiDAR vs. non-LiDAR devices were tested, how spatial confidence was measured/observed over time, and whether the "no explicit scan step" requirement held up in practice.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Context
Follow-up to issue #1, for layer 2 (spatial understanding) once the core MVP loop is proven. Depends on #1.
Decision
This must work as a consumer product: open the app, point the camera, talk — no pre-scan step, no LiDAR session the user has to perform deliberately.
Scope
ARWorldTrackingConfigurationwithsceneReconstructionwhere available) into the client.Out of scope
Report back
Comment with: whether LiDAR vs. non-LiDAR devices were tested, how spatial confidence was measured/observed over time, and whether the "no explicit scan step" requirement held up in practice.