AI Agent Observability — Internal Platform Telemetry Layer

Built the AI agent observability layer of an internal DevOps platform. OpenTelemetry signals from Claude Code and an open-source agent runtime (Hermes) flow into VictoriaLogs/VictoriaMetrics and surface in Grafana, attributing agent usage, cost, and success rate per user and per agent across the engineering organization.

Architecture

Gallery

Agent operations board — throughput, success rate, and outcome breakdown

InfoGrab DevOps Platform

2026.04 - 2026.06

InfoGrab Platform Team

Tech Stack

Claude Code Hermes OpenTelemetry VictoriaLogs VictoriaMetrics VictoriaTraces Grafana Multi-agent Observability Cost Engineering

Highlights

Built an end-to-end OTel exporter → VictoriaLogs (events) + VictoriaMetrics (metrics) → Grafana pipeline on the internal platform
Per-user and per-agent cost attribution — separates parent session and subagent tokens/cost to expose real multi-agent spend
Monthly run-rate estimation against plan gives organization-level spend visibility
Agent operations metrics — session, turn, and tool-call throughput with success rate, decomposing failures into incomplete / interrupted / timed_out
Per-agent cache hit ratio and output ratio reveal how prompt structure translates into cost
Secrets in prompt bodies are masked at collection and detection counts are kept as a metric, so turning observability on does not create a new leak path
Distributed the OTel configuration through the internal marketplace so every teammate's agent emits signals in the same shape
Archived logs and metrics to object storage (backup moved from CronJob to sidecar, retrying after 60 seconds on failure)

Key Tasks

Telemetry Pipeline

Enabled OpenTelemetry exporters on the agent runtimes and standardized endpoints
Collected events (prompts, agent spawn, completion, errors) into VictoriaLogs
Collected metrics (tokens, cost, latency, tool calls) into VictoriaMetrics
Standardized agent identity labels through the parent_session_id chain
Added VictoriaTraces and a collector traces pipeline, wired into Grafana as a datasource
Corrected spanmetrics dimensions to key on model name and provider
Packaged the OTel configuration as an internal marketplace item so teammates could enable it on their own clients

Productivity Board — Usage and Cost

Active sessions / Cost / Monthly run-rate / Tokens KPI row
Cost and token split by model family to identify what actually drives the bill
Per-user and per-agent summary table (active, cost, cache_hit, output_ratio, last_seen)
Run-rate parameterized against plan ceiling for early overrun detection

Agent Operations Board — Throughput and Failure

Fleet metrics: active agents, sessions created, turns per minute, tool calls per hour, error rate
Per-agent session and turn throughput with success rate
Outcome breakdown — completed / incomplete / interrupted / timed_out
Operation class share (llm / agent / api / cron / tool) to characterize load

Collection-path Safety and Retention

Secrets in prompt bodies masked at collection, with detection counts kept as a separate metric
Logs archived to object storage via an S3-compatible exporter
Metric backup moved from a standalone CronJob to a sidecar, retrying 60 seconds after failure

Cost and Cache Analysis

Per-agent cost attribution exposes the share of multi-agent calls in real spend
Compared prompt-cache hit ratios across tools and agents to surface effectiveness gaps
Prompt length distribution (avg/median/p95) to detect outliers that drive cost spikes

삶을 기록하는 두 축

Architecture

Gallery