# GPU and inference runtimes

Self-hosted inference is both an application runtime and an infrastructure workload. Instrument the serving request, model operation, and host/GPU capacity together.

Relevant systems include vLLM, Hugging Face TGI, NVIDIA Triton, Ollama, Ray Serve, KServe, and custom model servers.

Useful metadata and metrics include model identity/version, request latency, queue time, time to first token, tokens per second, batch size, input/output token counts, GPU utilization, VRAM utilization, cache utilization, model load/cold-start time, errors, and cancellations.

Do not record raw prompts, completions, embeddings, retrieved documents, or model weights by default.

Use standard OpenTelemetry and Prometheus exporters where the inference runtime already provides them. `telemio setup gpu` prints a starting checklist.