# spatial-agent-app: Overview

## Purpose

A FaceTime-style live call with an AI agent, using the phone's rear camera. The agent sees what the user sees, hears the user talk, and answers spatially: where something goes, at what angle, in what order. Think persistent AR co-pilot, not "take a photo and ask about it."

## Origin and use case

Summer 2026 sauna cabin build (see harg-bastu): carpentry, roofing, floors, sauna installation. The loop was: photograph a detail, describe it in text, wait, go to CAD, extract, print. Spatial questions ("here, not there, at this angle") are hard in text and trivial when shown.

Goal: collapse that loop into one live camera conversation.

## Strategy

Dogfood on the sauna/Harg build first. Only consider packaging as a product (potentially under Definitely Mabe) if it proves genuinely useful to the author.

## Architecture (3 layers)

1. **Capture/session layer.** Streams camera frames, continuous transcript, and camera pose/depth (ARKit/ARCore). Pose is the key: it lets anything said map to a point in 3D space, not just "the picture."
2. **Spatial understanding layer.** Localizes the live feed against a known reference geometry. For the sauna: the as-built scan or CAD model, not OpenStreetMap. The agent then knows "east wall, two meters up," not just "a wall."
3. **Reasoning/overlay layer.** Agent reasons in text as normal. Spatial answers become AR annotations (arrows, highlighted regions, ghost outlines of where a batten goes) rendered back onto the camera view.

## MVP scope

Skip live AR overlay in v1. Prove the core loop only:

- Phone app streams camera frames + ARKit pose + voice to a backend (about once per second for frames).
- Agent receives frame + pose + voice, replies in text or speech.
- Manually verify its spatial references are coherent before building any on-screen rendering.

Success criterion: the agent's spatial statements ("the vapor barrier overlaps here") match reality when checked on site.

## Distribution

Native iOS app, iOS-first. Mobile browsers do not expose depth or world-tracking pose the way ARKit/RealityKit do.

- Thin client: stream frames, pose, audio up; render annotations returned.
- Agent loop, 3D reasoning, and reference geometry stay server-side.
- A web dashboard for reviewing sessions and managing projects can come later; it is not part of the AR loop.

## Reference pattern

[fltman/wasteland-builder](https://github.com/fltman/wasteland-builder): OpenStreetMap, to Blender 3D city, to three.js game, orchestrated by an AI coding agent. Relevant as a model for "real-world data in, structured 3D geometry out, agent-orchestrated." Author has a fork on GitHub.

## Open questions for the first session

- Realtime transport: WebRTC vs. periodic frame upload over WebSocket/HTTPS for the MVP.
- Pose + frame payload format and sync (timestamps, intrinsics).
- Reference geometry source for the sauna: existing CAD export vs. fresh LiDAR scan.
- Localization approach: how the agent is given the model and the camera pose in the same frame of reference.
- Which vision-capable model and latency budget per turn.