Observability reference v7

AIDB emits OpenTelemetry (OTel) traces, logs, and metrics for its own function calls, background workers, and SQL query execution. For the GUCs that turn this feature on and configure it, see Configuring AIDB parameters — OpenTelemetry tracing.

Schemas

When aidb.otel_client = 'database', AIDB writes traces, logs, and metrics into three tables in the aidb_otel schema:

• aidb_otel.traces — one row per span.
• aidb_otel.logs — one row per log record.
• aidb_otel.metrics — one row per batch of metric readings, such as a periodic collection of instrument values.

A span describes an operation and its timing. AIDB stores log messages as separate OTel log records, not as events embedded inside spans. A log record can link to its span through the trace_id and span_id columns. Governance decisions are spans named purpose_decision in aidb_otel.traces.

This follows OpenTelemetry's recommendation to record new events as logs correlated with spans. The Span Events API is being deprecated; this doesn't remove span events from the OTel data model or invalidate existing data.

EDB Agent Governance reads spans and log records directly from the database. It doesn't require an OTLP collector and doesn't display all AIDB telemetry. Export to an external monitoring system is configured separately through the exporter worker.

By default, AIDB grants read access to these tables to aidb_governance, not to aidb_users. Administrators can grant a separate login role read access to the tables it needs. For the dashboard, use the dedicated viewer role: membership in aidb_governance also permits retention cleanup and purpose management.

All three tables share a common set of columns, plus a few columns specific to each.

Common columns

Every row in all three tables carries these columns:

At least one of payload_json/payload_binary is always populated — see Storage format.

Starting in 7.9.0, use source to distinguish AIDB's own decisions from decisions reported by external senders. Rows retained from an earlier version are labeled internal during upgrade and have no principal; their original source can't be verified from those columns.

aidb_otel.traces — additional columns

Each row is exactly one span. The identifier and attribute columns are stored separately from the payload, so you can query them without parsing JSON or protobuf.

recorded_at is the time the row was stored, which can be later than the operation itself. Use the span's start and end times to measure its duration.

trace_id/span_id/parent_span_id are lowercase hex, the same representation payload_json's own traceId/spanId/parentSpanId fields use — the two match as strings, so you can look up a row by either. (This isn't the same as payload_binary's raw OTLP protobuf bytes fields, which aren't hex or text at all.)

aidb_otel.logs — additional columns

Each row is one log record, with two columns linking it back to the span it was emitted under:

Configuring storage format

The aidb.otel_client_storage_format GUC controls which payload column the database client (one of the aidb.otel_client destinations — see Configuring AIDB parameters) writes to: payload_json, payload_binary, or both — the two columns described under Common columns. Values:

aidb.otel_client_storage_format only affects the database client (aidb.otel_client = 'database'). It has no effect on noop (nothing is written), stdout or grpc (both always emit the full OTLP payload directly, not through these columns), or log, which always writes payload_json regardless of this setting, since a server log line is a human-readable debugging aid, not storage a collector later forwards.

When exporting a row that has both payloads populated (storage_format = both), the exporter worker sends only payload_binary, since it's cheaper to have produced.

Starting in 7.9.0, telemetry ingested from external senders retains the format received: JSON or protobuf. aidb.otel_client_storage_format controls AIDB's own telemetry, not these incoming records.

Triggering retention cleanup

The cleaner background worker deletes old rows automatically (see Running background workers), but this function lets you trigger a cleanup pass immediately instead of waiting for its next tick.

aidb.run_otel_retention_cleanup()

Manually triggers a full retention cleanup pass, deleting rows older than aidb.otel_cleaner_retention_days from all three aidb_otel tables, in the same bounded batches the cleaner background worker uses. Runs immediately, regardless of aidb.otel_cleaner_enabled — useful for an ops task or a test that shouldn't wait for the worker's own wake-up interval.

Requires the aidb_governance role (see Managing audit access) — aidb_users has no EXECUTE on this function.

Takes no arguments and returns void. Safe to call while the cleaner background worker is running — the two don't compete over the same rows.

SELECT aidb.run_otel_retention_cleanup();

Internally, the database client writes through aidb_otel.store_metrics(), aidb_otel.store_logs(), and aidb_otel.store_traces(). These aren't meant to be called directly — aidb_users has EXECUTE on them (needed for any traced call to work) but no SELECT on the tables they write to.

Running background workers

Two background workers run per database, each spawned only when enabled for that database (see Configuring AIDB parameters for the governing GUCs).

Exporting traces, logs, and metrics

Export is disabled by default. Enable aidb.otel_exporter_enabled and configure aidb.otel_exporter_endpoint to forward stored telemetry. EDB Agent Governance reads the database directly and doesn't need this worker.

Forwards pending, due rows from aidb_otel.{traces,logs,metrics} to the OTLP/HTTP collector at aidb.otel_exporter_endpoint, on a 15-second tick. payload_json rows are sent as OTLP/HTTP+JSON. payload_binary rows are sent as OTLP/HTTP+protobuf (Content-Type: application/x-protobuf) — a real OTel Collector's HTTP receiver dispatches on that header, so both formats share the same endpoint. Every outbound request goes through AIDB's egress allowlist.

Failed deliveries back off per row (not per tick), so one row's retry schedule never blocks the rest of the batch. After aidb.otel_exporter_max_retries attempts, a row is marked export_status = 'failed' and stops being retried automatically — find these with:

SELECT * FROM aidb_otel.traces WHERE export_status = 'failed';

Cleaning up expired rows

Deletes rows older than aidb.otel_cleaner_retention_days from all three tables, in batches of aidb.otel_cleaner_batch_size, on an hourly tick. Set aidb.otel_cleaner_retention_days = 0 to disable cleanup entirely without stopping the worker itself.

Traced attributes

AIDB records attributes automatically when telemetry is enabled. It follows OpenTelemetry semantic conventions where applicable (gen_ai.*, db.*, error.type), with custom aidb.* attributes for other AIDB concepts. The reference below covers the attributes emitted by AIDB on spans, logs, and metrics. Each section identifies where they appear; an attribute can be absent when it doesn't apply to the operation.

Database telemetry from AIDB 7.8.0 uses schema version 1; AIDB 7.9.0 introduces schema version 2. The schema version identifies the telemetry structure and changes independently of the AIDB release number. Upgrading doesn't add new attributes to older records. These fields are available in the stored telemetry; the dashboard displays a subset.

Setting resource attributes

Set once per process, shared by every signal (traces, logs, and metrics):

Setting root-span attribute

Set once per trace, only on its root span:

db.postgresql.session_id uses the format <start_time_hex>.<pid_hex>, matching Postgres's own %c log_line_prefix escape. It's per-connection, so it's set only on each trace's root span — not on every span, and not as a resource attribute — since sharing it with the metrics pipeline would inflate metric cardinality. It's promoted to the indexed pg_session_id column on aidb_otel.traces.

Recording Agent Hub attributes

Set on spans for agent conversations, tasks, and messages:

Recording Tools Hub attributes

Set on spans for tool calls, including those imported from an MCP server:

Recording purpose-enforcement decisions

With telemetry enabled, purpose-scoped tool SQL produces purpose_decision spans with these attributes. Agents without a purpose don't produce these decision records.

Referenced relations describe the submitted statement, including relations in joins and subqueries. They aren't a complete record of indirect access through functions or views, and they don't identify the object that caused a denial. An absent attribute means relation information wasn't captured. An empty list means the captured statement named no relations.

The denial class, function name, message, and SQLSTATE apply to PostgreSQL privilege errors. They aren't recorded for decisions rejected before execution, such as a missing purpose role or a caller who isn't a member of that role. See Decision records for interpreting the outcome.

Tracing SQL queries

Set on any span that executes SQL:

Tracing pipelines, semantic KBs, and models

These spans cover pipeline operations, semantic search, model inference and maintenance, data processing, and volume access.

Recording memory activity

Set on spans for memory and memory-session operations:

Recording memory metrics

The aidb.memory.session.operations counter counts completed memory operations. The aidb.memory.session.operation.duration histogram records their duration in seconds. Their data points carry these attributes:

Recording registry and catalog activity

Set on spans for catalog discovery and purpose or plugin management:

Recording failures as logs

Failure log records can carry an error attribute containing the error text. This is separate from the span's error.type classification. When the log is emitted within a span, its trace and span IDs link it to that operation. Error text can contain sensitive data; restrict read access to the telemetry accordingly.