docs: add project overview and architecture
CD / Detect unsubstituted template (push) Successful in 1s
CD / Lint / Test / Vet (push) Successful in 5s
CD / Deploy via GitOps (push) Failing after 1s
CD / Build & Import (push) Successful in 4s

This commit is contained in:
2026-10-06 13:13:43 +00:00
parent 60ff7ea496
commit 1df7bf6d39
+51
View File
@@ -0,0 +1,51 @@
# spatial-agent-app: Overview
## Purpose
A FaceTime-style live call with an AI agent, using the phone's rear camera. The agent sees what the user sees, hears the user talk, and answers spatially: where something goes, at what angle, in what order. Think persistent AR co-pilot, not "take a photo and ask about it."
## Origin and use case
Summer 2026 sauna cabin build (see harg-bastu): carpentry, roofing, floors, sauna installation. The loop was: photograph a detail, describe it in text, wait, go to CAD, extract, print. Spatial questions ("here, not there, at this angle") are hard in text and trivial when shown.
Goal: collapse that loop into one live camera conversation.
## Strategy
Dogfood on the sauna/Harg build first. Only consider packaging as a product (potentially under Definitely Mabe) if it proves genuinely useful to the author.
## Architecture (3 layers)
1. **Capture/session layer.** Streams camera frames, continuous transcript, and camera pose/depth (ARKit/ARCore). Pose is the key: it lets anything said map to a point in 3D space, not just "the picture."
2. **Spatial understanding layer.** Localizes the live feed against a known reference geometry. For the sauna: the as-built scan or CAD model, not OpenStreetMap. The agent then knows "east wall, two meters up," not just "a wall."
3. **Reasoning/overlay layer.** Agent reasons in text as normal. Spatial answers become AR annotations (arrows, highlighted regions, ghost outlines of where a batten goes) rendered back onto the camera view.
## MVP scope
Skip live AR overlay in v1. Prove the core loop only:
- Phone app streams camera frames + ARKit pose + voice to a backend (about once per second for frames).
- Agent receives frame + pose + voice, replies in text or speech.
- Manually verify its spatial references are coherent before building any on-screen rendering.
Success criterion: the agent's spatial statements ("the vapor barrier overlaps here") match reality when checked on site.
## Distribution
Native iOS app, iOS-first. Mobile browsers do not expose depth or world-tracking pose the way ARKit/RealityKit do.
- Thin client: stream frames, pose, audio up; render annotations returned.
- Agent loop, 3D reasoning, and reference geometry stay server-side.
- A web dashboard for reviewing sessions and managing projects can come later; it is not part of the AR loop.
## Reference pattern
[fltman/wasteland-builder](https://github.com/fltman/wasteland-builder): OpenStreetMap, to Blender 3D city, to three.js game, orchestrated by an AI coding agent. Relevant as a model for "real-world data in, structured 3D geometry out, agent-orchestrated." Author has a fork on GitHub.
## Open questions for the first session
- Realtime transport: WebRTC vs. periodic frame upload over WebSocket/HTTPS for the MVP.
- Pose + frame payload format and sync (timestamps, intrinsics).
- Reference geometry source for the sauna: existing CAD export vs. fresh LiDAR scan.
- Localization approach: how the agent is given the model and the camera pose in the same frame of reference.
- Which vision-capable model and latency budget per turn.