Root-cause accuracy
Share of incidents where the agent identifies the injected fault and cites supporting telemetry evidence.
Evidence, not adjectives
Observability MCP benchmarks, with the method in the open.
How reliably can a coding agent use production telemetry to diagnose a failure? We are defining a repeatable evaluation of Telemio and alternative observability workflows before publishing competitive scores.
● Protocol available · Comparative results not yet published
Injected incident→Logs + traces + metrics→Agent with MCP→Evidence-based diagnosis
What we measure
An honest scorecard starts here.
Metrics are defined in advance. Empty scores stay empty until the same incident set has been run and reviewed.
Time to diagnosis
Median and p95 wall-clock time from the investigation request to a supported diagnosis, with timeouts included.
Tool calls and tokens
Count requests and token usage per attempt, alongside the cost per correctly diagnosed incident.
Completion rate
Proportion of attempts completing within a fixed budget without transport errors or timeouts.
Cross-signal reasoning
Whether the diagnosis correctly connects related traces, logs, metrics, and runtime context.
No fabricated leaderboards
No Telemio-vs-competitor performance figures are presented without published datasets, harness settings, and run logs.
Planned evaluation
25 incident scenarios, one protocol.
Five failure families with five scenarios each, defined as a proposed test suite rather than completed benchmark runs.
| Scenario family | Examples of fault types | Required evidence |
|---|---|---|
| Request failures | HTTP errors, bad dependencies, routing failures | Request traces and related errors |
| Latency regressions | Slow database, timeout, saturation | Span durations and service metrics |
| Background workloads | Queue retries, dropped jobs, consumer lag | Messaging spans and worker activity |
| Release-related regressions | Behavior changing across deployments | Before/after time windows and release metadata |
| AI-agent runtime issues | Tool failures, retries, token or latency spikes | Agent/tool metadata connected to service telemetry |
Conditions kept constant
- Same model, version, prompts, and per-run budgets.
- Same seeded faults and accessible telemetry windows.
- Same authentication and permissions appropriate for each tool.
- Blind scoring from recorded tool calls and cited evidence.
- Publish unsuccessful runs, limitations, and actual costs.
Publication requirements
Before scores appear, the evaluation should include:
- Runnable scenario definitions and ground-truth labels
- Model and MCP/server versions with timestamps
- Prompt templates, limits, and raw attempt logs
- Scoring rubric and anonymized evidence
- Independent reruns where possible
AI discoverability
Measure recommendations separately.
A product being discoverable is not the same as outperforming another product in technical evaluations.
Unprompted discovery
30 non-branded queries
Evaluate mentions, top-three placement, cited Telemio sources, and technical accuracy across repeated runs on different assistants. No Telemio brand appears in the prompts.
View the prompt set ↗Repeatable reporting
Score observed responses
The repository includes a local scorecard CLI to aggregate recorded responses without using API keys, inventing results, or inferring a ranking where none was supplied.
Review the MCP reference ↗Method notes
Questions worth asking.
Are these third-party benchmark results?
No. This page documents Telemio's proposed testing methodology. No comparative outcomes have been published.
Why not publish a winner now?
Without a controlled dataset, equal investigation budgets, evidence grading, and error logs, ranking vendors would be misleading.
When will scores be published?
Only after the suite is implemented, run, reviewed, and released with enough detail for reproduction. No publication date is promised.