Observability

Photon ships with OpenTelemetry instrumentation baked in — traces, metrics, and logs are emitted automatically during every tool call. By default the instrumentation is a no-op. Set one environment variable and install the OTel SDK to route everything to any OTLP-compatible backend (Jaeger, Grafana Tempo, SigNoz, Honeycomb, DataDog, …).

MCP logging/setLevel and notifications/message are compatibility features as of 2026-07-28. Use stderr for human-readable stdio diagnostics and this OpenTelemetry path for production telemetry. The complete migration is in Migrating MCP roots, sampling, and logging.

Quick start

# 1. Install the OTel SDK as a peer dep of your deployment
bun add @opentelemetry/sdk-node @opentelemetry/api \
        @opentelemetry/exporter-trace-otlp-http \
        @opentelemetry/exporter-metrics-otlp-http \
        @opentelemetry/exporter-logs-otlp-http \
        @opentelemetry/resources \
        @opentelemetry/semantic-conventions

# 2. Point at a collector
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_SERVICE_NAME=my-photon-service

# 3. Run anything — CLI, MCP server, Beam — they all export now
photon cli kanban stats
photon beam
photon mcp my-photon

That is the entire setup. Photon's CLI entry point calls initOtelSdk() before any command runs, so the SDK is live before the first span is created.

What gets exported

Traces

Span	Attributes
gen_ai.tool.call {photon}.{tool}	gen_ai.tool.name, gen_ai.agent.name, gen_ai.operation.name, photon.instance, photon.caller, photon.trace_id, photon.stateful
gen_ai.agent.invoke {photon}	gen_ai.agent.name, gen_ai.operation.name, gen_ai.agent.description

• Error spans auto-set sampling.priority=1 so they survive head-based sampling.
• recordException captures the full stack trace via Error.cause.
• HTTP headers or _meta can carry bounded W3C traceparent, tracestate, and baggage; Photon rejects conflicting or malformed inbound values.
• Nested this.call(), daemon workers, durable tasks, and external MCP calls propagate that validated context, so multi-hop workflows remain connected.
• Baggage is context only. Photon never expands untrusted baggage entries into metric labels.

Metrics

Instrument	Type	Unit	Attributes
photon.tool.duration	histogram	ms	gen_ai.agent.name, gen_ai.tool.name, status, photon.stateful, photon.error_type
photon.tool.calls	counter	1	same
photon.tool.errors	counter	1	same
photon.circuit_breaker.transitions	counter	1	gen_ai.agent.name, gen_ai.tool.name, from, to, photon.instance
photon.rate_limit.rejections	counter	1	gen_ai.agent.name, gen_ai.tool.name, photon.instance
photon.bulkhead.rejections	counter	1	gen_ai.agent.name, gen_ai.tool.name, photon.instance

Structured error responses

When a tool call fails, the MCP response sets isError: true and attaches a machine-readable payload so agents can make typed retry decisions:

{
  "content": [{ "type": "text", "text": "Tool Error: add ..." }],
  "isError": true,
  "structuredContent": {
    "error": {
      "type": "circuit_open",
      "retryable": true,
      "message": "Circuit open: add has failed 5 consecutive times. Resets in 12s"
    }
  },
  "_meta": { "photon": { "type": "...", "retryable": true, "message": "..." } }
}

Error type values: validation_error, timeout_error, network_error, permission_error, not_found_error, circuit_open, rate_limited, bulkhead_full, implementation_error, runtime_error. retryable is true for transient failures (circuit_open, timeout, network) and false for deterministic ones (validation, permission, not_found).

Logs

Every record emitted by the photon Logger is forwarded to the OTel logs bridge with severity mapped to the standard scale and ambient context auto-attached: photon.name, photon.tool, photon.trace_id, photon.caller_id. This gives trace-log correlation in any OTLP backend without a pino dependency.

Running a local collector

For development, the easiest stack is Grafana's OTel-LGTM image — it includes Tempo (traces), Mimir (metrics), and Loki (logs) behind a single Grafana pane:

docker run -p 3000:3000 -p 4317:4317 -p 4318:4318 \
  --rm -ti grafana/otel-lgtm

Then OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 and open http://localhost:3000. Every tool call will show up in the "Explore" view under the photon service.

For Jaeger-only traces:

docker run -p 16686:16686 -p 4318:4318 \
  jaegertracing/all-in-one:latest

Health checks

Beam exposes two HTTP endpoints for liveness/readiness probes:

Endpoint	Purpose
GET /api/health	Top-level status with per-subsystem breakdown. Returns 200 when all subsystems are ok, 503 when any is degraded. Mount as Kubernetes livenessProbe + readinessProbe.
GET /api/health/circuits	Per-circuit state (closed/open/half-open) plus summary counts. Useful for ops dashboards.

The /api/health body shape:

{
  "status": "ok",
  "uptime_s": 1234,
  "timestamp": "2026-04-15T07:50:00.000Z",
  "subsystems": {
    "runtime":  { "status": "ok" },
    "photons":  { "status": "ok", "detail": "5 photon(s) loaded" },
    "circuits": { "status": "ok", "detail": "0 open of 3 tracked" }
  }
}

Environment variables

Variable	Purpose
OTEL_EXPORTER_OTLP_ENDPOINT	Master switch. Unset = SDK stays disabled.
OTEL_EXPORTER_OTLP_PROTOCOL	grpc, http/protobuf, or http/json. Defaults per SDK.
OTEL_SERVICE_NAME	Service name attached to every record.
OTEL_RESOURCE_ATTRIBUTES	Extra resource attributes (e.g. deployment.environment=prod).
OTEL_TRACES_SAMPLER	Override the default sampler. Error spans are always kept.
OTEL_LOG_LEVEL	SDK's own log level.

These are standard OTel variables; Photon doesn't invent any of them.

AG-UI integration

AG-UI clients running against Photon benefit from everything in this guide automatically:

• Structured errors — RUN_ERROR events include code (error type), retryable (boolean), runId, and threadId, so clients can classify failures and auto-retry transient ones without bubbling to the user.
• Trace correlation — every AG-UI event carries rawEvent.traceparent, letting clients deep-link a UI event to the backing span in Jaeger/Tempo.
• Capability handshake — MCP initialize advertises experimental['ag-ui'].features (structured-errors, trace-correlation, proxy-mode, local-mode) so clients can negotiate without probing.

Philosophy

Photon's instrumentation is intentionally zero-dependency: the runtime only imports @opentelemetry/* via dynamic import() inside try/catch, so unused installs stay tiny. The value-add of this guide is that once you do install the SDK and set one env var, every promise in PROMISES.md Intent 10 becomes observable with no photon-specific configuration.