The scrubber had a rule for `postgres://user:pass@` and none for the
general `scheme://user:secret@host` form — which is exactly what a git
credential store holds:
https://<user>:<token>@git.d-ma.be
Verified against all 13 DefaultRules with an invented value: it matched
none of them. Not authorization-header, not bearer-token, and not
homelab-env-token, which needs a `_TOKEN=`-style prefix a bare URL does
not have. So such a turn was ingested rather than dropped, against the
stated intent in this file's own doc comment:
Fail-closed by design: we'd rather lose signal than ingest
credentials into a public-readable brain.
It failed closed for the shapes it named and open for this one.
How it surfaced: a test suite in mathias/infra executed a script that
configures git while inheriting the operator's real $HOME, read koala's
live ~/.git-credentials, and printed two live values. ~/.claude/projects
is this watcher's own input, so the scrubber was the last line of defence
for that mistake. A sweep of that tree found 12 occurrences of 2 distinct
credential-shaped URLs across 4 transcripts in 3 sessions — so the shape
had been passing for a while, not just that once. Both credentials are
being rotated (infra#466).
The generalisable point is not "do not read the credential file". That
mistake will recur. It is that this shape must fail closed, because the
scrubber is what decides whether such a mistake is survivable or
permanent.
Ordering: the new rule sits after postgres-uri-with-password so a DSN
still reports the more specific name, the same convention as
authorization-header before bearer-token.
Both directions are tested, because a rule that dropped every URL with a
colon would discard most of the signal this watcher exists to keep. The
{8,} floor on the secret half keeps `https://host:443/path` and
`https://user:@host` out; the `[^:\s/@]+` user half stops a host:port
pair being read as userinfo. Must-not-scrub cases include a host with a
port, a password-less DSN, a colon in a query string, and the literal
printf format string `https://oauth2:%s@%s`.
Closes#142
Refs mathias/infra#466
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CLAUDE_INGEST_SKIP_PROJECTS takes filepath.Match globs tested against
the project directory NAME, and a matching directory is skipped during
the directory WALK — so its files are never opened and their content
never enters the process. That is a stronger property than filtering
after reading, and the difference is observable: without SkipDir,
WalkDir lists the directory's entries, and an unreadable one aborts the
entire walk, silently stopping ingestion for every other project.
Globs rather than exact names because a project directory is derived
from the session's working directory, so a git worktree yields an
ephemeral name no exact-match list could keep up with.
An unparseable pattern refuses to run. filepath.Match returns an error
ALONGSIDE a false match, so a bad pattern checked per-file would ingest
the directory while the log complained — the failure nobody notices.
Validated once, up front, in both entry points.
This complements the client-name scrubber rather than replacing it. The
scrubber drops individual turns whose content matches a client
identity; this excludes whole workspaces whose content is confidential
in ways no pattern reliably recognises. Learnings from excluded
sessions are captured deliberately through the session-close ritual
instead.
Recorded caveat: the project directory reflects the session's working
directory, NOT the repositories it touches. A session run from ~/dev
that reads an excluded repo is not excluded by this. The scrubber
remains the control for content; this is a control for workspaces.
A mutation test earned the strongest assertion here: the original tests
could not distinguish SkipDir from filtering after the walk, because
neither opens files. The unreadable-directory test can.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A ~/.zshrc read surfaced OP_SERVICE_ACCOUNT_TOKEN into a transcript. The
_TOKEN= form was already caught by homelab-env-token, but a bare ops_<b64>
value was not. Add an op-service-account rule (ordered early). Tests cover
both env-assigned and bare forms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A shell mangle that glued a key to a preceding word ('yes'+'sk-...') had
no word boundary, so the leading \b in the openai-sk rule failed to match
and a LiteLLM master key leaked past the scrubber into ingest (2026-06-11).
- openai-sk: drop leading \b, match sk- shape anywhere ({32,} floor keeps
short task-/disk- words clean).
- add jwt rule for bare header.payload.sig tokens (no Bearer prefix).
- regression tests: the exact yessk- evasion, standalone sk-, bare JWT,
plus clean-content guards for task-/disk-.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pre-rollout guard. Source code stays clean — client identities come
from CLAUDE_INGEST_CLIENT_BLOCK env (sourced from a SOPS-encrypted k8s
secret in infra repo). Env value is a regex alternation; main wraps
it with `(?i)\b(...)\b` so word-boundary matching avoids false hits
inside longer identifiers (e.g. "Sebastian" doesn't trigger on "SEB").
DefaultRules (credential shapes) still take precedence so any leak
that's BOTH a client mention AND a credential shape logs as the
credential — strictly more dangerous, points triage at the right
thing. Tests cover precedence + case variations + word-boundary
respect + invalid-pattern rejection.
Refs: infra#73 Track E.1 pre-rollout grill (option B).
Bump-Type: minor
New package internal/claudewatcher. The volume gate (24 turns/week of
agentsquad logs vs 500/week gate) exposed that the real signal lives
in daily Claude Code usage at ~/.claude/projects/*/<uuid>.jsonl, not
in agentsquad output. This package captures that signal. See infra#73
Track E + hyperguild#27 for the full reframe.
Components:
- parser: tolerant JSONL parser over the observed Claude Code session
schema (user / assistant / attachment / system + bookkeeping types).
Skip-flag fast-paths queue-operation, last-prompt, permission-mode,
ai-title, bridge-session, file-history-snapshot.
- scrubber: 11-rule fail-closed regex set for credential shapes
(bearer, postgres URIs, PEM, ssh-key, ghp_/sk-/sk-ant-/AKIA, homelab
env tokens, SOPS markers). Drop turn + log on match.
- cursor: postgres-backed claude_session_cursors table, keyed by
(host, file_path) with byte_offset. Resumable across pod restarts.
- watcher: poll loop. Walks SessionsDir, processes each .jsonl from
its cursor offset, runs scrubber, emits a Batch per file to a
Sink interface, advances cursor on successful Ingest.
No classifier integration in this commit — every kept turn is emitted
in a per-session batch. The cmd/server wiring (next commit) routes
batches to brain/wiki/claude-sessions/facts/. Classifier-driven hall
routing (decisions / failures / hypotheses) is a follow-up.
19 unit tests across parser + scrubber + watcher. task check green.
Refs: infra#73, hyperguild#27
mathiasandClaude Opus 5