55 lines
2.8 KiB
Gherkin
55 lines
2.8 KiB
Gherkin
Feature: Observability — timing and metrics for performance and UX (ADR-030, #15)
|
|
As the maintainer running Tapir for pilot users
|
|
I want timing and Prometheus metrics for the activities that drive performance and UX
|
|
So that I can see latency, model behaviour, and usage — and feed the Stage-0 eval gate
|
|
|
|
# AI metrics are the priority (ADR-030 R3). Each scenario maps to a Go test in
|
|
# scenarioCoverage once implemented; tagged @pending until the TDD step lands it.
|
|
|
|
@pending # TestObserveSummarizeRecordsModelOutcomeFallback
|
|
Scenario: Summarization latency is recorded per endpoint
|
|
Given the summarizer runs a transcript through its endpoint chain
|
|
When an endpoint returns a parseable summary
|
|
Then the summarize latency is recorded with the model, outcome "success", and whether it was a fallback
|
|
|
|
@pending # TestObserveSummarizeRecordsFailureOutcomes
|
|
Scenario: A failing summarizer endpoint records its failure outcome
|
|
Given the summarizer runs a transcript through its endpoint chain
|
|
When an endpoint errors or returns unparseable output
|
|
Then the summarize latency is recorded with outcome "error" or "parse_error" before the chain advances
|
|
|
|
@pending # TestObserveCaptionFetchByOutcome
|
|
Scenario: Caption fetch latency is recorded by outcome
|
|
Given a caption fetch is attempted for a video
|
|
When it resolves to captions, no captions, or a rate limit
|
|
Then the caption-fetch latency is recorded labelled by that outcome
|
|
|
|
@pending # TestLLMUsageHookRecordsTokens
|
|
Scenario: LLM token usage is recorded from the completion
|
|
Given an LLM completion returns a usage block with prompt and completion tokens
|
|
When the client finishes the call
|
|
Then the prompt and completion tokens are recorded for that model
|
|
|
|
@pending # TestChatAnswerLatencyRecorded
|
|
Scenario: Q&A answer latency is recorded
|
|
Given a user asks a question about a video
|
|
When the answer is produced from the stored transcript
|
|
Then the chat answer latency is recorded for the answering model
|
|
|
|
@pending # TestHTTPMiddlewareRecordsByRoutePattern
|
|
Scenario: HTTP requests are counted by route, method, and status
|
|
Given the metrics HTTP middleware wraps the app
|
|
When a request is served against a registered route
|
|
Then it is counted and timed under the bounded route pattern, not the raw path
|
|
|
|
@pending # TestLoginCounted
|
|
Scenario: A successful login is counted
|
|
Given a user completes the OIDC callback and a session is established
|
|
Then the login counter is incremented
|
|
|
|
@pending # TestMetricsNotOnPublicMux
|
|
Scenario: The metrics endpoint is not on the public app port
|
|
Given the service is running
|
|
When the public app mux is inspected
|
|
Then it exposes no /metrics route — metrics are served on the dedicated metrics port only
|