diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 0000000..5eb2e6f --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,51 @@ +# spatial-agent-app: Overview + +## Purpose + +A FaceTime-style live call with an AI agent, using the phone's rear camera. The agent sees what the user sees, hears the user talk, and answers spatially: where something goes, at what angle, in what order. Think persistent AR co-pilot, not "take a photo and ask about it." + +## Origin and use case + +Summer 2026 sauna cabin build (see harg-bastu): carpentry, roofing, floors, sauna installation. The loop was: photograph a detail, describe it in text, wait, go to CAD, extract, print. Spatial questions ("here, not there, at this angle") are hard in text and trivial when shown. + +Goal: collapse that loop into one live camera conversation. + +## Strategy + +Dogfood on the sauna/Harg build first. Only consider packaging as a product (potentially under Definitely Mabe) if it proves genuinely useful to the author. + +## Architecture (3 layers) + +1. **Capture/session layer.** Streams camera frames, continuous transcript, and camera pose/depth (ARKit/ARCore). Pose is the key: it lets anything said map to a point in 3D space, not just "the picture." +2. **Spatial understanding layer.** Localizes the live feed against a known reference geometry. For the sauna: the as-built scan or CAD model, not OpenStreetMap. The agent then knows "east wall, two meters up," not just "a wall." +3. **Reasoning/overlay layer.** Agent reasons in text as normal. Spatial answers become AR annotations (arrows, highlighted regions, ghost outlines of where a batten goes) rendered back onto the camera view. + +## MVP scope + +Skip live AR overlay in v1. Prove the core loop only: + +- Phone app streams camera frames + ARKit pose + voice to a backend (about once per second for frames). +- Agent receives frame + pose + voice, replies in text or speech. +- Manually verify its spatial references are coherent before building any on-screen rendering. + +Success criterion: the agent's spatial statements ("the vapor barrier overlaps here") match reality when checked on site. + +## Distribution + +Native iOS app, iOS-first. Mobile browsers do not expose depth or world-tracking pose the way ARKit/RealityKit do. + +- Thin client: stream frames, pose, audio up; render annotations returned. +- Agent loop, 3D reasoning, and reference geometry stay server-side. +- A web dashboard for reviewing sessions and managing projects can come later; it is not part of the AR loop. + +## Reference pattern + +[fltman/wasteland-builder](https://github.com/fltman/wasteland-builder): OpenStreetMap, to Blender 3D city, to three.js game, orchestrated by an AI coding agent. Relevant as a model for "real-world data in, structured 3D geometry out, agent-orchestrated." Author has a fork on GitHub. + +## Open questions for the first session + +- Realtime transport: WebRTC vs. periodic frame upload over WebSocket/HTTPS for the MVP. +- Pose + frame payload format and sync (timestamps, intrinsics). +- Reference geometry source for the sauna: existing CAD export vs. fresh LiDAR scan. +- Localization approach: how the agent is given the model and the camera pose in the same frame of reference. +- Which vision-capable model and latency budget per turn.