Agent observability without storing every prompt

It is easy to make an AI observability product into a second copy of the application database. Turn on full prompt logging, add tool payloads, keep retrieved documents, and suddenly an operational trace contains support tickets, source code, customer details, and secrets that were never meant to live there.

That is avoidable. Most day-to-day questions about an agent do not require the full content that passed through it.

Begin with the questions you actually debug

Teams usually want to know why a run failed, why it was slow, why it cost more than expected, which tool caused the problem, whether a release changed behavior, or whether the same context is being loaded repeatedly. Those questions can be answered with structured metadata.

Record the agent or workflow name, model and provider, operation type, tool name, status, duration, input and output token counts, retry count, estimated cost, release, and the identifiers that connect the run to the rest of the application.

Use identifiers when content is not the point

A stable context ID is often more useful than the context body. It lets you see that the same document set was loaded across several runs. Add a version when the context changes. Add token or byte size when you need to understand cost. Add a sensitivity label if the application already classifies data.

A fingerprint can help when you need to know whether two pieces of content are the same without exporting the content itself. Telemio exposes a SHA-256 fingerprint helper for that case.

await telemio.context.loaded({
  id: "ctx:architecture",
  kind: "documentation",
  version: process.env.GIT_COMMIT,
  estimatedTokens: 4200,
  sensitivity: "internal"
})

const fingerprint = await telemio.agent.fingerprint(contextText)

The fingerprint is a correlation aid. It is not a substitute for a proper access-control or privacy policy, and low-entropy values can still be guessable by someone who already knows the possible inputs.

Prompt capture should have a reason

There are cases where sampled prompt or completion content is useful. A team debugging a narrow quality regression may decide to capture a small, redacted sample. That decision should name the purpose, the fields that are removed, the retention period, and who can access the sample.

Full content capture is much harder to reason about when it is simply the default. It also makes telemetry volume larger and can bury the operational fields that are useful across every run.

Tool calls deserve their own telemetry

Tools are where many agent failures become concrete. A model may choose the right action and still fail because a tool timed out, a permission changed, a database was slow, or an MCP server returned an error. Give tool execution its own span and preserve the parent agent or workflow relationship.

Record the tool name and type, timing, status, retry behavior, and a safe identifier for the target when that is useful. Avoid copying authorization headers, raw SQL parameters, customer records, or entire tool results into attributes.

Keep agent traces connected to the application

An agent does not run in isolation. A browser action or API request may start the workflow. The run may read from a database, enqueue work, call an external API, and write a final result. When those operations share trace context, you can follow the incident without jumping between unrelated tools.

This is one reason Telemio uses OpenTelemetry as the base. The agent-specific metadata sits inside the same operational model as the rest of the software.

What to keep by default

• agent, workflow, model, provider, and operation names
• tool and MCP operation names
• duration, status, retries, and errors
• input/output token counts and cost metadata
• release, environment, service, request, job, and session identifiers
• context IDs, versions, sizes, reuse metadata, and optional fingerprints

That set is small enough to govern and rich enough to answer most operational questions. Add content only when there is a specific reason to do so.

Use it in Telemio

Install @telemio/sdk for explicit agent and context telemetry. The agent helpers are metadata-first by default. For repository setup, the coding-agent instructions at /agent/setup.md tell the agent to keep credentials, payment data, and sensitive content out of telemetry while it instruments the rest of the application.