Monitor self-hosted Intelligence with OpenTelemetry

Overview#

A self-hosted CopilotKit Intelligence deployment can send its telemetry to an OpenTelemetry collector that you run. From there, you forward it to the tracing and logging tools that your team already uses. This page lists each stream, what it contains, how to turn it on, and which signals show that the installation is healthy.

Intelligence does not install a collector. The streams on this page go only to the endpoints that you configure, and each stream is off until you configure one.

When should I use this?#

• You self-host Intelligence with the Helm chart or the ECS bundle.
• You want request traces, errors, and logs from Intelligence in your own observability tools.
• You want to see whether a new installation or an upgrade is healthy.

Cloud-hosted Intelligence does not export these streams to customers.

Prerequisites#

• A running self-hosted Intelligence deployment.
• An OpenTelemetry Collector (the otel/opentelemetry-collector-contrib distribution) that the Intelligence services can reach over HTTP. Intelligence sends OTLP over HTTP only. It does not use gRPC.
• For pod logs on Kubernetes: a collector that runs as a DaemonSet and can read /var/log/pods on each node.

The streams#

Application logs are not sent over OTLP. Your collector or log agent reads them from the container output.

Send traces and metrics#

Helm#

Set a full OTLP/HTTP traces endpoint. The path must end in /v1/traces.

observability:
  tracing:
    enabled: true
    endpoint: http://otel-collector.observability.svc:4318/v1/traces
    environment: production

• environment is required when tracing is on. Use 1 to 64 letters, numbers, dots, underscores, slashes, or hyphens.
• The chart sends traces from the app API, the realtime gateway, and the Automatic Learning scheduler to this endpoint.
• The scheduler also sends its metrics to the same collector, at the same base URL with /v1/metrics.
• helm install and helm upgrade fail if the endpoint does not end in /v1/traces.

ECS#

Set otlpEndpoint to the base URL of your collector, without a path.

{
  "otlpEndpoint": "https://otel-collector.example.internal:4318"
}

• The release stack sends traces from the app API and the realtime gateway to <otlpEndpoint>/v1/traces.
• cpki-ecs deploy removes a trailing / or /v1/traces, so a value in that form also works.
• The ECS bundle sets deployment.environment.name to ecs. You cannot change it.
• The collector must be reachable from the private subnets of the deployment.

What every span carries#

Each span has these resource attributes:

The services trace inbound HTTP and WebSocket upgrades, outbound HTTP, PostgreSQL, and Redis commands that run inside a request. Useful span attributes include:

• http.request.method, http.route, and http.response.status_code
• db.operation.name on PostgreSQL spans
• cpki.organization.id and cpki.project.id on requests from a known project
• provider.effect spans on the realtime gateway for Slack and Microsoft Teams calls, with cpki.provider.adapter and cpki.provider.outcome
• replay.restore spans on the realtime gateway, with thread_id, organization_id, and project_id

The services remove these attributes before a span leaves the process:

• db.statement and db.query.text (SQL text)
• http.url, url.full, url.path, and url.query (full URLs, paths, and query strings)
• every http.request.header.* and http.response.header.* attribute

Health probe requests do not create spans.

Send product events#

Product events are OTLP log records that describe product actions, for example a thread that was created or a skill that was approved. Use them to build your own usage reports.

appApi:
  channelTelemetry:
    endpoint: http://otel-collector.observability.svc:4318/v1/logs
    tokenSecret:
      name: otel-collector-token
      key: token

• The endpoint must end in /v1/logs.
• Set tokenSecret or headers. The chart fails if you set neither.
• The app API sends the token as Authorization: Bearer <token>.
• To send other headers, set headers to a literal key=value,key=value string. If both are set, the token wins for Authorization.
• This stream has no ECS equivalent.

Every attribute name starts with copilotkit.. The key that you use to remove duplicates after a retry is different for each event family:

• Thread and Learning events: copilotkit.event_id, in the form <event>:<organization>:<project>:<resource>. On eip.learning.skill_pulled, the resource part is a new random ID for each pull. Two pulls of the same skill are two events.
• eip.channel.inbound: copilotkit.channel_event_id, the ID of the inbound message event.
• eip.channel.created and eip.channel.attached: no event ID. Do not remove duplicates of these events by an ID.

Product events never contain message content, raw provider payloads, credentials, Channel names, thread names, skill text, or insight text. They do contain identifiers: organization, project, thread, run, and API key IDs, and the ID of the signed-in user. eip.channel.created and eip.channel.attached can also carry copilotkit.principal_email, the email address of the user who made the change.

Collect application logs#

The app API, the realtime gateway, the thread compaction worker, the connector outbox, and the thread culler write one JSON object per line to stdout. Two services write plain text:

• App frontend: Nginx access and error logs in the default Nginx text format.
• Embeddings: the text-embeddings-inference server log.

For the JSON services:

• Level: set it with config.logLevel on Helm.
• Request lines: the app API and the realtime gateway end each HTTP request with an http.server.request.completed line that has duration_ms, http.method, http.route, http.status_code, and outcome. In the app API line, method, route, and status_code are nested under an http object. The frontend writes no such line. To see frontend errors, read the status code in its Nginx access log.
• Trace links: app API lines carry trace_id and span_id when tracing is on. Realtime gateway lines carry dd.trace_id and dd.span_id in the Datadog decimal format.
• Redaction: app API logs replace authorization headers, cookies, API keys, and fields named like password, token, or secret with [REDACTED].

On Kubernetes, read the logs with the collector's filelog receiver on /var/log/pods. On ECS, the logs are in the CloudWatch log groups /ecs/<deploymentName>/....

Configure the collector#

This configuration receives all OTLP streams, reads pod logs, and forwards everything to your backend. Replace the exporter with the one for your tool.

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
  filelog:
    # Replace intelligence with the namespace of your release.
    include: [/var/log/pods/intelligence_*/*/*.log]
    include_file_path: true
    operators:
      - type: container
      # Parse the JSON services. Nginx and embeddings lines stay as text.
      - type: json_parser
        if: 'body matches "^\\s*\\{"'
      # Link app API lines to their traces.
      - type: trace_parser
        if: 'attributes.trace_id != nil'
        trace_id:
          parse_from: attributes.trace_id
        span_id:
          parse_from: attributes.span_id

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  # A second layer. The services already remove these attributes.
  transform/redact:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - delete_key(attributes, "db.statement")
          - delete_key(attributes, "db.query.text")
          - delete_key(attributes, "url.full")
          - delete_key(attributes, "url.query")
  batch: {}

exporters:
  otlphttp:
    endpoint: https://otlp.your-backend.example.com
  debug:
    verbosity: basic

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, transform/redact, batch]
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]
    logs:
      receivers: [otlp, filelog]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]

• Point observability.tracing.endpoint at http://<collector-service>.<namespace>.svc:4318/v1/traces.
• Point appApi.channelTelemetry.endpoint at the same host with /v1/logs.
• The filelog operators parse the JSON lines into attributes, so you can query fields such as http.status_code. Lines that are not JSON keep their text in the body.
• The trace_parser maps the app API trace_id and span_id onto the log record. The realtime gateway dd.trace_id is a decimal number, so it does not map to an OpenTelemetry trace ID.
• To check the setup, add debug to a pipeline's exporters and read the collector output.

Check installation health#

Probes#

With the marketplace build, the app API uses /api/health/live and /api/health/ready.

Signals to watch#

• Error rate: error spans, http.server.request.completed lines with http.status_code 500 or higher, and frontend Nginx access lines with a 5xx status.
• Database: app API app-api health check failed log lines, and failed postgresql.* spans.
• Realtime gateway memory: gateway_memory lines every 5 seconds, and gateway_memory_pressure warnings when a container uses 80% or more of its memory limit.
• Product Analytics pipeline: realtime_gateway.analytics.projector.health lines, when Product Analytics is on. The line logs at warning level when events wait in the queue, fail, or go to quarantine, and at error level when the gateway cannot read the queue. Watch analytics_projector.queue_depth, analytics_projector.quarantine_depth, and analytics_projector.oldest_queue_age_ms.
• Configuration: a telemetry.config_rejected warning at startup. It means that an endpoint was set but rejected, and that stream is off.

What is not checked#

• No probe checks Redis, object storage, or the connection between the app API and the realtime gateway. A Redis failure shows up only as request errors and error spans.
• No service exports health metrics or connection counts. Use the signals above in your own tools.

Next steps#

• Self-host on Kubernetes has the full Helm configuration reference.
• Self-host on ECS has the full ECS configuration.
• Product Analytics shows usage inside Intelligence.