Virtual machines and ephemeral hosts

Identify every machine

Give each VM a stable host identity and each short-lived allocation a unique run identity.

Use host.id to correlate CPU and memory with application traces, logs, and agent work. Prefer an actual cloud instance ID or a machine ID unique to that allocation. Golden images must not contain a reused host identity. Set telemio.vm.lifecycle to persistent or ephemeral, and use telemio.vm.run.id to distinguish short-lived allocations on reused hosts.

The Collector monitors the host. A Node.js agent, Java service, or Python worker uses its own instrumented process and can share the same host.id for correlation.

Install the host Collector

Use the official OpenTelemetry Collector Contrib distribution on the VM, with a metrics-only pipeline by default.

Install otelcol-contrib using the official Linux package instructions. Use the Contrib build because hostmetrics is not part of the core-only distribution. Save this configuration as otelcol-contrib.yaml:

receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      load: {}
      disk: {}
      filesystem:
        metrics:
          system.filesystem.utilization:
            enabled: true
      network: {}
      processes: {}
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
  resource/vm:
    attributes:
      - key: service.name
        value: ${env:TELEMIO_HOST_SERVICE}
        action: upsert
      - key: deployment.environment.name
        value: ${env:TELEMIO_ENVIRONMENT}
        action: upsert
      - key: host.id
        value: ${env:TELEMIO_VM_ID}
        action: upsert
      - key: host.name
        value: ${env:TELEMIO_HOST_NAME}
        action: upsert
      - key: telemio.vm.lifecycle
        value: ${env:TELEMIO_VM_LIFECYCLE}
        action: upsert
      - key: telemio.vm.run.id
        value: ${env:TELEMIO_VM_RUN_ID}
        action: upsert
      - key: telemio.vm.group
        value: ${env:TELEMIO_VM_GROUP}
        action: upsert
  batch:
    timeout: 5s
exporters:
  otlphttp/telemio:
    endpoint: ${env:TELEMIO_OTLP_ENDPOINT}
    headers:
      x-api-key: ${env:TELEMIO_API_KEY}
    sending_queue:
      enabled: true
service:
  pipelines:
    metrics/host:
      receivers: [hostmetrics]
      processors: [memory_limiter, resource/vm, batch]
      exporters: [otlphttp/telemio]

Only host metrics are collected automatically. Application logs, agent prompts, and source files are not included. The Collector uses HTTPS and the standard Telemio x-api-key ingest header. Its endpoint is the OTLP base URL, not the read-only API URL or a path ending in /v1/metrics.

Configure a persistent VM

Configure the machine once and keep a consistent identity across reboots.

Create a root-owned environment file with your unique VM identity and scoped ingest credential. For a persistent machine, TELEMIO_VM_ID stays the same across Collector restarts. Replace all placeholder values in this template:

TELEMIO_API_KEY=replace-with-vm-host-scoped-ingest-key
TELEMIO_OTLP_ENDPOINT=https://otlp.telemio.sh
TELEMIO_HOST_SERVICE=vm-host
TELEMIO_ENVIRONMENT=production
TELEMIO_VM_ID=replace-with-unique-instance-id
TELEMIO_HOST_NAME=replace-with-hostname
TELEMIO_VM_LIFECYCLE=persistent
TELEMIO_VM_RUN_ID=replace-with-unique-boot-or-allocation-id
TELEMIO_VM_GROUP=default-vm-group

After downloading the example files from your repository, install the configuration and environment file. If your package uses a different Collector config path, adapt it to the installed systemd unit.

sudo install -d -m 0700 /etc/telemio
sudo install -m 0600 vm-collector.env /etc/telemio/vm-collector.env
sudo install -m 0644 otelcol-contrib.yaml /etc/otelcol-contrib/config.yaml
sudo install -d /etc/systemd/system/otelcol-contrib.service.d
printf '[Service]\nEnvironmentFile=/etc/telemio/vm-collector.env\nTimeoutStopSec=30\n' | sudo tee /etc/systemd/system/otelcol-contrib.service.d/10-telemio.conf
sudo systemctl daemon-reload
sudo systemctl enable --now otelcol-contrib
sudo journalctl -u otelcol-contrib -n 50 --no-pager

Set the systemd service to use the installed Collector configuration path. Watch its journal after startup for credential errors, exporter delivery failures, and permission issues.

Instrument ephemeral agents and VM workers

Create a fresh allocation identity during provisioning and let the Collector flush before shutdown.

Bake the Collector binary and safe YAML into the VM image, but never the ingest key, host.id, or run identifier. At boot, inject a new allocation ID, credentials from your secret manager, and a useful bounded group name (for example build-workers).

TELEMIO_VM_ID="${INSTANCE_ID}"
TELEMIO_VM_RUN_ID="${ALLOCATION_ID}"
TELEMIO_VM_LIFECYCLE="ephemeral"
TELEMIO_VM_GROUP="build-workers"
TELEMIO_HOST_NAME="$(hostname -s)"
TELEMIO_HOST_SERVICE="vm-host"
TELEMIO_ENVIRONMENT="production"
TELEMIO_OTLP_ENDPOINT="https://otlp.telemio.sh"

INSTANCE_ID and ALLOCATION_ID are values supplied by your VM provisioner, not built-in shell variables. Write the resolved values and the host-scoped ingest key to the protected environment file before starting the Collector. Gracefully stop the service when a job completes and allow exporter drain time. Forced termination can still lose buffered telemetry.

Correlate a running agent

Attach the same VM identity to an agent process, but use a separate service-scoped ingest credential.

The first-party TypeScript SDK supports resourceAttributes. Use them to attach host.id and telemio.vm.run.id to agent activity, so the Infrastructure view can show the agent beside its host metrics.

import { createTelemio } from "@telemio/sdk"

const telemetry = createTelemio({
  endpoint: "https://otlp.telemio.sh",
  apiKey: process.env.TELEMIO_AGENT_API_KEY,
  serviceName: "vm-agent",
  environment: process.env.TELEMIO_ENVIRONMENT || "production",
  resourceAttributes: {
    "host.id": process.env.TELEMIO_VM_ID || "",
    "host.name": process.env.TELEMIO_HOST_NAME || "",
    "telemio.vm.lifecycle": process.env.TELEMIO_VM_LIFECYCLE || "persistent",
    "telemio.vm.run.id": process.env.TELEMIO_VM_RUN_ID || "",
    "telemio.vm.group": process.env.TELEMIO_VM_GROUP || "default-vm-group"
  }
})

export async function runToolWork(): Promise<void> {
  await telemetry.tracer.withSpan("agent.tool.execute", async (span) => {
    span.setAttribute("agent.tool.name", "knowledge.lookup")
    await telemetry.logger.info("Agent tool started")
    await telemetry.metrics.count("agent.tool.invocations", 1)
  })
}

The agent uses TELEMIO_AGENT_API_KEY, a separate ingest key scoped to vm-agent. The Collector uses a vm-host key. Telemio enforces service-scoped ingest identity, so do not reuse one key to export records with another service.name. For Java and Python, use the same host resource attributes in their OpenTelemetry SDK setup. Continue with AI agents, Node.js SDK, or Java services.

Verify signals in Telemio

Check the Collector, inspect the new VM inventory, then trace activity back to its host.

1. Start the Collector and allow at least 30–60 seconds for its first hostmetrics sample.
2. In the Telemio console, open Observe → Infrastructure → Virtual machines and hosts.
3. Confirm the host, lifecycle, group, last-seen time, CPU busy percentage, used-memory percentage, and root filesystem utilization when available.
4. Run an instrumented agent task and confirm its service appears alongside vm-host for the same host.id.
5. If nothing arrives, inspect journalctl -u otelcol-contrib, make sure outbound HTTPS to otlp.telemio.sh is allowed, and check the service/environment binding of both keys.

The VM inventory uses the existing workspace-scoped OTLP data, via GET /api/virtual-machines for authorized console readers. Missing utilization percentages show as unavailable rather than invented values. The status indicator is based on the most recent signal, not a cloud API power-state lookup.

For collectors that also receive multiple services or a Kubernetes cluster, follow the Collector & Kubernetes guide. Multi-service export requires appropriate service-scoped credentials.