diff --git a/docs/use-cases/observability.feature b/docs/use-cases/observability.feature new file mode 100644 index 0000000..3a1a933 --- /dev/null +++ b/docs/use-cases/observability.feature @@ -0,0 +1,54 @@ +Feature: Observability — timing and metrics for performance and UX (ADR-030, #15) + As the maintainer running Tapir for pilot users + I want timing and Prometheus metrics for the activities that drive performance and UX + So that I can see latency, model behaviour, and usage — and feed the Stage-0 eval gate + + # AI metrics are the priority (ADR-030 R3). Each scenario maps to a Go test in + # scenarioCoverage once implemented; tagged @pending until the TDD step lands it. + + @pending # TestObserveSummarizeRecordsModelOutcomeFallback + Scenario: Summarization latency is recorded per endpoint + Given the summarizer runs a transcript through its endpoint chain + When an endpoint returns a parseable summary + Then the summarize latency is recorded with the model, outcome "success", and whether it was a fallback + + @pending # TestObserveSummarizeRecordsFailureOutcomes + Scenario: A failing summarizer endpoint records its failure outcome + Given the summarizer runs a transcript through its endpoint chain + When an endpoint errors or returns unparseable output + Then the summarize latency is recorded with outcome "error" or "parse_error" before the chain advances + + @pending # TestObserveCaptionFetchByOutcome + Scenario: Caption fetch latency is recorded by outcome + Given a caption fetch is attempted for a video + When it resolves to captions, no captions, or a rate limit + Then the caption-fetch latency is recorded labelled by that outcome + + @pending # TestLLMUsageHookRecordsTokens + Scenario: LLM token usage is recorded from the completion + Given an LLM completion returns a usage block with prompt and completion tokens + When the client finishes the call + Then the prompt and completion tokens are recorded for that model + + @pending # TestChatAnswerLatencyRecorded + Scenario: Q&A answer latency is recorded + Given a user asks a question about a video + When the answer is produced from the stored transcript + Then the chat answer latency is recorded for the answering model + + @pending # TestHTTPMiddlewareRecordsByRoutePattern + Scenario: HTTP requests are counted by route, method, and status + Given the metrics HTTP middleware wraps the app + When a request is served against a registered route + Then it is counted and timed under the bounded route pattern, not the raw path + + @pending # TestLoginCounted + Scenario: A successful login is counted + Given a user completes the OIDC callback and a session is established + Then the login counter is incremented + + @pending # TestMetricsNotOnPublicMux + Scenario: The metrics endpoint is not on the public app port + Given the service is running + When the public app mux is inspected + Then it exposes no /metrics route — metrics are served on the dedicated metrics port only