Compare commits
2
Commits
gitea-pages
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4f9b32205d | ||
|
|
fa817affb8 |
@@ -1,3 +1,7 @@
|
||||
# homelab
|
||||
|
||||
Personal AI platform — architecture and capabilities overview
|
||||
Personal AI platform — architecture and capabilities overview
|
||||
|
||||
## Docs
|
||||
|
||||
- [What a Client PoC Taught Us About Our Own AI Delivery Pipeline](docs/articles/what-a-client-poc-taught-us-about-cad.md) — cross-repo review of Continuous Agentic Development (CAD) against an external Requirements-Engineering PoC, plus what it surfaced across this stack.
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
# What a Client PoC Taught Us About Our Own AI Delivery Pipeline
|
||||
|
||||
*A cross-repo review of Continuous Agentic Development (CAD), and the fifteen-repo stack that runs it — prompted by comparing notes with a from-scratch AI Requirements-Engineering PoC.*
|
||||
|
||||
---
|
||||
|
||||
## The setup
|
||||
|
||||
A few weeks ago I was pulled into a proof-of-concept for a client project: an AI-driven Requirements-Engineering platform, built from scratch by two humans and a swarm of coding-agent sessions. Call it **the RE-PoC**. It ingests source documents, runs an elicitation bot against a human author, gates every output through a structural quality check and an LLM-judged governance review, and commits the result as a pull request — never merging autonomously.
|
||||
|
||||
I wasn't looking for a case study. But halfway through reading its process docs, I recognized the shape of a problem I'd already spent months solving in my own homelab: **how do you let coding agents build software continuously, without turning "the agent said it's done" into the only signal anyone checks?**
|
||||
|
||||
That's also the plain-English definition of **Continuous Agentic Development (CAD)** — the pipeline my own stack runs, visualized end to end in a tool I call the Atlas. So I did the obvious thing: a full structural comparison, then pointed the same lens at my *own* fifteen-repo stack to see if it held up as well as I assumed.
|
||||
|
||||
It didn't, not entirely. Here's what came out of it.
|
||||
|
||||
---
|
||||
|
||||
## What CAD actually is
|
||||
|
||||
Nine stages, one feedback loop:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[00 Signals] --> B[01 TELOS]
|
||||
B --> C[02 Strategic session]
|
||||
C --> D[03 Spec to Gitea issue]
|
||||
D --> E{04 Human dispatch gate}
|
||||
E --> F[05 Execute — agent swarm]
|
||||
F --> G[06 PR to CI]
|
||||
G --> H[07 CD to pod]
|
||||
H --> I[08 Loop back — scored vs TELOS]
|
||||
I -. feedback .-> B
|
||||
|
||||
style E fill:#f9d67a,stroke:#333
|
||||
```
|
||||
|
||||
Three gates sit across that pipeline, each guarding a different failure mode:
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Gates["Three orthogonal gates"]
|
||||
direction LR
|
||||
G1["Spec integrity<br/><i>is the issue untampered</i>"]
|
||||
G2["Repo eligibility<br/><i>may agents run here at all</i>"]
|
||||
G3["Output correctness<br/><i>does the PR actually satisfy the spec</i>"]
|
||||
end
|
||||
```
|
||||
|
||||
The whole thesis of the stack's audit layer is that **the trace *is* the audit package** — if you build the pipeline right, you don't write a compliance report after the fact, you render one from the event stream.
|
||||
|
||||
That's the pitch. The RE-PoC was a chance to test it against an independently-built system solving the same problem from a different angle.
|
||||
|
||||
---
|
||||
|
||||
## The single most useful thing the comparison surfaced
|
||||
|
||||
Both systems, built independently, hit the exact same failure mode — and fixed it in two different ways worth knowing about.
|
||||
|
||||
**The failure:** an agent (or a human standing in for one) reviews another agent's work, says "approved," and it turns out the work wasn't actually done. A rubber stamp with a signature on it.
|
||||
|
||||
**Fix #1 — the process answer (the RE-PoC).** After a real incident where a reviewer wrote "ACCEPTED" over a known-unmet checklist item, they hardened their review protocol: every review comment must open with exactly one of three words (`APPROVED` / `REWORK` / `ESCALATE`), and the reviewer must transcribe the acceptance checklist verbatim, item by item, *before* writing that word. "Approved" written above an unchecked box is now defined as self-contradictory — automatically treated as rework, no judgment call needed.
|
||||
|
||||
That's a good fix. It's also a fix that depends on the reviewer following the convention correctly, every single time.
|
||||
|
||||
**Fix #2 — the mechanical answer (my own stack, independently).** Across four separate repos in my pipeline, I found the same problem solved a structurally different way: **the thing being judged never gets to supply its own verdict.**
|
||||
|
||||
- The executor's review gate hard-overrides an LLM "approve" to "revise" if the actual test suite didn't pass — approval is documented as "a signal, not a guarantee."
|
||||
- A separate spec-compiler's PR gate injects the pass/fail logic into a sandboxed process that runs *against* the candidate, rather than trusting a verdict the candidate hands back. The internal shorthand for this, after a debate about it: *"a self-claim has no authority."*
|
||||
- A third component never trusts an agent's self-reported "done" — it polls the actual CI status on the actual commit, independently, every time.
|
||||
|
||||
The difference matters: the RE-PoC's fix makes a false "approved" *visually self-contradictory*. Mine makes it *impossible to act on*, because nothing downstream ever reads the agent's own opinion of its work as the verdict. One is a better habit. The other doesn't care whether you're having a good day.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph P["Process fix"]
|
||||
R1[Reviewer writes verdict] --> C1{Checklist transcribed first?}
|
||||
C1 -->|yes, consistent| OK1[Verdict stands]
|
||||
C1 -->|contradicts checklist| RW1[Auto rework]
|
||||
end
|
||||
subgraph M["Mechanical fix"]
|
||||
R2[Agent proposes verdict] -.ignored.-> X((discarded))
|
||||
T[Independent check: tests / CI / gate] --> V2[Real verdict]
|
||||
end
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Four more things worth stealing — in both directions
|
||||
|
||||
**1. Nobody in my stack has a claim protocol, and the RE-PoC does — because they got burned first.**
|
||||
Two independent agent sessions on their side once read the same "unblocked" signal and both built the same task concurrently — landed on the identical branch name, wasted real compute. Their fix: a lightweight claim-comment protocol (post a claim before touching anything, cap work-in-progress, re-check freshness immediately before pushing, not just before claiming). My own stack has *nothing* like this anywhere — one component gets close (a distributed lock for a single watcher process) but nothing coordinates multiple agents against a shared backlog. It's a gap I only found because someone else hit the failure first.
|
||||
|
||||
**2. My stack independently reinvented a "sovereignty gate" three times — theirs has none.**
|
||||
Three unrelated components in my pipeline all converged on the same idea without coordinating: before any content reaches a cloud LLM call, check whether it's allowed to leave the local network at all, and fail closed if unsure. The RE-PoC has no equivalent — not a flaw exactly, more a sign it's never had to operate under a "don't leak client data" constraint. Worth them at least deciding that on purpose rather than by omission, especially given their own stated premise is demonstrating *governed* agent-driven delivery.
|
||||
|
||||
**3. Whether an LLM-judged gate gets to *block* anything is a decision I haven't actually made yet.**
|
||||
The RE-PoC's governance review can outright block a merge. A knowledge-quality service in my own stack does the opposite on purpose: its LLM judge is confirmation-only, never allowed to execute or block anything, routed to a human above a severity threshold. Two live precedents, opposite philosophies, and I have two more components in my own stack about to need this exact decision with no answer recorded anywhere. That's not a maturity gap, it's an undecided fork — worth closing before either one gets built on autopilot.
|
||||
|
||||
**4. Glossaries rot the moment nobody enforces them — theirs is CI-gated, three of mine aren't.**
|
||||
Cheap, mechanical, easy to add: flag any new domain term that shows up in code or docs but isn't in the glossary. Advisory only, doesn't block anything, just stops the thing from going stale. They do it. I have three separate glossaries and zero enforcement on any of them.
|
||||
|
||||
---
|
||||
|
||||
## The consolidation side-quest
|
||||
|
||||
Doing this review meant reading fifteen-plus of my own repos back to back for the first time in a while, which is its own kind of audit. Two clusters of genuine redundancy fell out:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph "Already done (precedent)"
|
||||
ISV["ingestion-svc<br/>(archived 2026-05-14)"] -.absorbed into.-> HGI[hyperguild/ingestion]
|
||||
end
|
||||
subgraph "Same move, not yet done"
|
||||
BIN[brain-ingest-ntfy<br/>CI red 2 months] -.propose absorb.-> HGI
|
||||
BWI[brain-weekly-ingest<br/>never actually built] -.propose absorb.-> HGI
|
||||
end
|
||||
subgraph "Separate lifecycle question"
|
||||
J[journal<br/>self-flagged provisional] -.propose fold into.-> B[brain]
|
||||
end
|
||||
```
|
||||
|
||||
One repo (`journal`) had literally flagged itself for a consolidation review in its own README. Another pair of repos were doing the same job as a component that had *already* absorbed a third repo doing that exact job a year earlier — the precedent was sitting right there and nobody had followed it since. Filed as tracked issues, cross-linked both directions, not silently merged — consolidation is a decision, not a cleanup script.
|
||||
|
||||
---
|
||||
|
||||
## Nine takeaways, if you're building something similar
|
||||
|
||||
1. **A rubber-stamp failure will happen to you too.** It's not a sign of a bad review process — it's a sign you have a review process with a human or an LLM in the loop. Plan for it.
|
||||
2. **Prefer a mechanical verdict over a well-written convention, when you can.** Conventions degrade under drift. A gate that computes its own answer independently of the thing it's judging doesn't.
|
||||
3. **If more than one agent might touch the same backlog, you need a claim protocol before you need it.** Waiting for the collision to design the fix is expensive and avoidable.
|
||||
4. **Decide your LLM-gate's authority on purpose.** Blocking vs. advisory-only is a philosophy choice with real consequences, not a default you back into.
|
||||
5. **If you operate under a data-sovereignty constraint, build the gate before you need it,** not after the first accidental cloud call.
|
||||
6. **A glossary without enforcement is a glossary that's already stale.** Advisory-only CI checks are cheap insurance.
|
||||
7. **The best comparison you'll get is an independently-built system solving your exact problem differently.** Convergent design (three repos reinventing the same gate without coordinating) is a stronger signal than any one of them alone.
|
||||
8. **When you find a repo that's already flagged itself for cleanup, believe it and act — don't wait for the calendar date it names.**
|
||||
9. **Write the comparison down somewhere durable, cross-link it to the actual work items it implies, and send fixes back to whoever you learned from.** A comparison that lives only in a chat transcript is a comparison that didn't happen.
|
||||
|
||||
---
|
||||
|
||||
## What we sent back
|
||||
|
||||
Two of these findings went back to the RE-PoC's own tracker: a suggestion to pair their process-based rubber-stamp fix with a mechanical fallback, and a question about whether their governance-review LLM call needed a sovereignty check of its own. Owed, since half the value here came from reading their incident log in the first place.
|
||||
|
||||
*Everything referenced from my own stack above is tracked and cross-linked in the relevant repos' issue trackers as of this writing.*
|
||||
Reference in New Issue
Block a user