mathiasandClaude Opus 4.8
cd461b95f8
Wire the metrics package into the live paths and serve it: - summarizer: per-endpoint latency by model/outcome(success|error|parse_error)/fallback + slog. - youtube.FetchTranscript: latency by outcome (captions|none|rate_limited) + slog. - chat: answer latency by model + slog. - llm usage hook → token counts (prompt|completion) per model, wired in buildSummarizer/buildChat. - oidc callback: login counter. - cmdServe: wrap Router in metrics.HTTPMiddleware (request count + latency by bounded route pattern) and serve /metrics on TAPIR_METRICS_ADDR (default :9090), a SEPARATE port — never on the public app mux. BDD: observability.feature scenarios un-pended + mapped. TDD: summarizer wiring tested black-box via the /metrics scrape; metrics-not-on-public-mux asserted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
47 lines
2.4 KiB
Gherkin
47 lines
2.4 KiB
Gherkin
Feature: Observability — timing and metrics for performance and UX (ADR-030, #15)
|
|
As the maintainer running Tapir for pilot users
|
|
I want timing and Prometheus metrics for the activities that drive performance and UX
|
|
So that I can see latency, model behaviour, and usage — and feed the Stage-0 eval gate
|
|
|
|
# AI metrics are the priority (ADR-030 R3). Each scenario maps to a Go test in
|
|
# test/acceptance/scenario_coverage_test.go (the BDD name-coverage gate).
|
|
|
|
Scenario: Summarization latency is recorded per endpoint
|
|
Given the summarizer runs a transcript through its endpoint chain
|
|
When an endpoint returns a parseable summary
|
|
Then the summarize latency is recorded with the model, outcome "success", and whether it was a fallback
|
|
|
|
Scenario: A failing summarizer endpoint records its failure outcome
|
|
Given the summarizer runs a transcript through its endpoint chain
|
|
When an endpoint errors or returns unparseable output
|
|
Then the summarize latency is recorded with outcome "error" or "parse_error" before the chain advances
|
|
|
|
Scenario: Caption fetch latency is recorded by outcome
|
|
Given a caption fetch is attempted for a video
|
|
When it resolves to captions, no captions, or a rate limit
|
|
Then the caption-fetch latency is recorded labelled by that outcome
|
|
|
|
Scenario: LLM token usage is recorded from the completion
|
|
Given an LLM completion returns a usage block with prompt and completion tokens
|
|
When the client finishes the call
|
|
Then the prompt and completion tokens are recorded for that model
|
|
|
|
Scenario: Q&A answer latency is recorded
|
|
Given a user asks a question about a video
|
|
When the answer is produced from the stored transcript
|
|
Then the chat answer latency is recorded for the answering model
|
|
|
|
Scenario: HTTP requests are counted by route, method, and status
|
|
Given the metrics HTTP middleware wraps the app
|
|
When a request is served against a registered route
|
|
Then it is counted and timed under the bounded route pattern, not the raw path
|
|
|
|
Scenario: A successful login is counted
|
|
Given a user completes the OIDC callback and a session is established
|
|
Then the login counter is incremented
|
|
|
|
Scenario: The metrics endpoint is not on the public app port
|
|
Given the service is running
|
|
When the public app mux is inspected
|
|
Then it exposes no /metrics route — metrics are served on the dedicated metrics port only
|