A FaceTime-style live call with an AI agent, using the phone's rear camera. The agent sees what the user sees, hears the user talk, and answers spatially: where something goes, at what angle, in what order. Think persistent AR co-pilot, not "take a photo and ask about it."

Origin and use case

Summer 2026 sauna cabin build (see harg-bastu): carpentry, roofing, floors, sauna installation. The loop was: photograph a detail, describe it in text, wait, go to CAD, extract, print. Spatial questions ("here, not there, at this angle") are hard in text and trivial when shown.

Goal: collapse that loop into one live camera conversation.

Strategy

Dogfood on the sauna/Harg build first. Only consider packaging as a product (potentially under Definitely Mabe) if it proves genuinely useful to the author.

Architecture (3 layers)

Capture/session layer. Streams camera frames, continuous transcript, and camera pose/depth (ARKit/ARCore). Pose is the key: it lets anything said map to a point in 3D space, not just "the picture."
Spatial understanding layer. Localizes the live feed against a known reference geometry. For the sauna: the as-built scan or CAD model, not OpenStreetMap. The agent then knows "east wall, two meters up," not just "a wall."
Reasoning/overlay layer. Agent reasons in text as normal. Spatial answers become AR annotations (arrows, highlighted regions, ghost outlines of where a batten goes) rendered back onto the camera view.

MVP scope

Skip live AR overlay in v1. Prove the core loop only:

Phone app streams camera frames + ARKit pose + voice to a backend (about once per second for frames).
Agent receives frame + pose + voice, replies in text or speech.
Manually verify its spatial references are coherent before building any on-screen rendering.

Success criterion: the agent's spatial statements ("the vapor barrier overlaps here") match reality when checked on site.

Distribution

Native iOS app, iOS-first. Mobile browsers do not expose depth or world-tracking pose the way ARKit/RealityKit do.

Thin client: stream frames, pose, audio up; render annotations returned.
Agent loop, 3D reasoning, and reference geometry stay server-side.
A web dashboard for reviewing sessions and managing projects can come later; it is not part of the AR loop.

Reference pattern

fltman/wasteland-builder: OpenStreetMap, to Blender 3D city, to three.js game, orchestrated by an AI coding agent. Relevant as a model for "real-world data in, structured 3D geometry out, agent-orchestrated." Author has a fork on GitHub.

Open questions for the first session

Realtime transport: WebRTC vs. periodic frame upload over WebSocket/HTTPS for the MVP.
Pose + frame payload format and sync (timestamps, intrinsics).
Reference geometry source for the sauna: existing CAD export vs. fresh LiDAR scan.
Localization approach: how the agent is given the model and the camera pose in the same frame of reference.
Which vision-capable model and latency budget per turn.

3.1 KiB Raw Blame History

spatial-agent-app: Overview

Purpose