generated from mathias/template-go-web
docs: add project overview and architecture
This commit is contained in:
@@ -0,0 +1,51 @@
|
||||
# spatial-agent-app: Overview
|
||||
|
||||
## Purpose
|
||||
|
||||
A FaceTime-style live call with an AI agent, using the phone's rear camera. The agent sees what the user sees, hears the user talk, and answers spatially: where something goes, at what angle, in what order. Think persistent AR co-pilot, not "take a photo and ask about it."
|
||||
|
||||
## Origin and use case
|
||||
|
||||
Summer 2026 sauna cabin build (see harg-bastu): carpentry, roofing, floors, sauna installation. The loop was: photograph a detail, describe it in text, wait, go to CAD, extract, print. Spatial questions ("here, not there, at this angle") are hard in text and trivial when shown.
|
||||
|
||||
Goal: collapse that loop into one live camera conversation.
|
||||
|
||||
## Strategy
|
||||
|
||||
Dogfood on the sauna/Harg build first. Only consider packaging as a product (potentially under Definitely Mabe) if it proves genuinely useful to the author.
|
||||
|
||||
## Architecture (3 layers)
|
||||
|
||||
1. **Capture/session layer.** Streams camera frames, continuous transcript, and camera pose/depth (ARKit/ARCore). Pose is the key: it lets anything said map to a point in 3D space, not just "the picture."
|
||||
2. **Spatial understanding layer.** Localizes the live feed against a known reference geometry. For the sauna: the as-built scan or CAD model, not OpenStreetMap. The agent then knows "east wall, two meters up," not just "a wall."
|
||||
3. **Reasoning/overlay layer.** Agent reasons in text as normal. Spatial answers become AR annotations (arrows, highlighted regions, ghost outlines of where a batten goes) rendered back onto the camera view.
|
||||
|
||||
## MVP scope
|
||||
|
||||
Skip live AR overlay in v1. Prove the core loop only:
|
||||
|
||||
- Phone app streams camera frames + ARKit pose + voice to a backend (about once per second for frames).
|
||||
- Agent receives frame + pose + voice, replies in text or speech.
|
||||
- Manually verify its spatial references are coherent before building any on-screen rendering.
|
||||
|
||||
Success criterion: the agent's spatial statements ("the vapor barrier overlaps here") match reality when checked on site.
|
||||
|
||||
## Distribution
|
||||
|
||||
Native iOS app, iOS-first. Mobile browsers do not expose depth or world-tracking pose the way ARKit/RealityKit do.
|
||||
|
||||
- Thin client: stream frames, pose, audio up; render annotations returned.
|
||||
- Agent loop, 3D reasoning, and reference geometry stay server-side.
|
||||
- A web dashboard for reviewing sessions and managing projects can come later; it is not part of the AR loop.
|
||||
|
||||
## Reference pattern
|
||||
|
||||
[fltman/wasteland-builder](https://github.com/fltman/wasteland-builder): OpenStreetMap, to Blender 3D city, to three.js game, orchestrated by an AI coding agent. Relevant as a model for "real-world data in, structured 3D geometry out, agent-orchestrated." Author has a fork on GitHub.
|
||||
|
||||
## Open questions for the first session
|
||||
|
||||
- Realtime transport: WebRTC vs. periodic frame upload over WebSocket/HTTPS for the MVP.
|
||||
- Pose + frame payload format and sync (timestamps, intrinsics).
|
||||
- Reference geometry source for the sauna: existing CAD export vs. fresh LiDAR scan.
|
||||
- Localization approach: how the agent is given the model and the camera pose in the same frame of reference.
|
||||
- Which vision-capable model and latency budget per turn.
|
||||
Reference in New Issue
Block a user