Read docs/overview.md, AGENTS.md, and CLAUDE.md first — architecture, rationale, and working conventions are there.
One-line pitch: a FaceTime-style live call with an agent using the phone's rear camera, so the agent reasons about physical space in real time (AR co-pilot). Origin use case: the sauna/Harg cabin build.
Scope for this issue (MVP only)
Prove the core loop. No AR overlay rendering yet.
Scaffold a native iOS client (Swift, ARKit) that captures camera frames + ARKit pose and streams them (roughly once per second) along with voice, to a backend.
Scaffold a minimal server-side agent backend that receives frame + pose + voice and replies in text or speech.
Verify manually that the agent's spatial references (e.g. "the wall on your left", "two meters up") are coherent against what's actually in frame — this is the MVP success criterion, not a polished product.
Explicitly out of scope for now
AR annotation/overlay rendering on the camera view
Localization against a reference geometry/CAD model (layer 2 in the overview) — can follow once the core loop works
Web dashboard
Open questions to resolve during scaffolding
Realtime transport: WebRTC vs. periodic frame upload over WebSocket/HTTPS
Pose + frame payload format and sync (timestamps, camera intrinsics)
Which vision-capable model and latency budget per turn
Report back
Comment on this issue with: what was scaffolded, which transport/model choices were made and why, and whether the MVP success criterion was met with a concrete example (what was said, what the agent replied, whether it matched reality).
## Brief
Read `docs/overview.md`, `AGENTS.md`, and `CLAUDE.md` first — architecture, rationale, and working conventions are there.
One-line pitch: a FaceTime-style live call with an agent using the phone's rear camera, so the agent reasons about physical space in real time (AR co-pilot). Origin use case: the sauna/Harg cabin build.
## Scope for this issue (MVP only)
Prove the core loop. No AR overlay rendering yet.
1. Scaffold a native iOS client (Swift, ARKit) that captures camera frames + ARKit pose and streams them (roughly once per second) along with voice, to a backend.
2. Scaffold a minimal server-side agent backend that receives frame + pose + voice and replies in text or speech.
3. Verify manually that the agent's spatial references (e.g. "the wall on your left", "two meters up") are coherent against what's actually in frame — this is the MVP success criterion, not a polished product.
## Explicitly out of scope for now
- AR annotation/overlay rendering on the camera view
- Localization against a reference geometry/CAD model (layer 2 in the overview) — can follow once the core loop works
- Web dashboard
## Open questions to resolve during scaffolding
- Realtime transport: WebRTC vs. periodic frame upload over WebSocket/HTTPS
- Pose + frame payload format and sync (timestamps, camera intrinsics)
- Which vision-capable model and latency budget per turn
## Report back
Comment on this issue with: what was scaffolded, which transport/model choices were made and why, and whether the MVP success criterion was met with a concrete example (what was said, what the agent replied, whether it matched reality).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Brief
Read
docs/overview.md,AGENTS.md, andCLAUDE.mdfirst — architecture, rationale, and working conventions are there.One-line pitch: a FaceTime-style live call with an agent using the phone's rear camera, so the agent reasons about physical space in real time (AR co-pilot). Origin use case: the sauna/Harg cabin build.
Scope for this issue (MVP only)
Prove the core loop. No AR overlay rendering yet.
Explicitly out of scope for now
Open questions to resolve during scaffolding
Report back
Comment on this issue with: what was scaffolded, which transport/model choices were made and why, and whether the MVP success criterion was met with a concrete example (what was said, what the agent replied, whether it matched reality).