Monitoring & Observability

The platform exposes operational signals through a combination of log streams, Redis-backed queue metrics, and a health endpoint. These signals are intended for operators monitoring a deployed instance.

Queue Depth

The chat-proc worker consumes a Redis-backed message queue. Queue depth (number of pending messages) is observable via standard Redis list length commands against the configured queue key. Elevated queue depth indicates processor backpressure or under-provisioning.

Processor Health

Each processor instance reports liveness through its internal health check. In ECS deployments the ALB target group tracks instance health; in Compose deployments a container health check is defined. Unhealthy processor instances are replaced or restarted by the orchestrator.

Log Streams

All platform services emit structured log output to stdout, captured by the container runtime. In ECS deployments logs flow to CloudWatch Logs; in Compose deployments they are available via docker compose logs. Log lines include request IDs, tenant/project scope, bundle ID, and error classification where applicable.

Metrics Endpoint

The Operations REST API (served by chat-proc) exposes operational data accessible to authorized callers. Bundle-level metrics (turn counts, economics state, error rates) can be queried via the Operations API using a privileged token. Prometheus-format scrape endpoints are not currently built in but can be added as a bundle operation.

Connection Pooling

Redis and Postgres connection pools are created once per worker process during FastAPI lifespan startup and stored in app.state. Total connections scale with worker count.

Redis Pools

Pool size is controlled by GATEWAY_CONFIG_JSON.pools.<component>.redis_max_connections. If unset, the pool is unbounded and grows with load. Pub/sub and blocking calls consume connections from the pool, not on top of the cap.

Postgres Pool

Each worker creates one asyncpg pool. Size is controlled by pools.<component>.pg_pool_max_size and pg_pool_min_size. If not set, it defaults to service_capacity.<component>.concurrent_requests_per_process.

# Total connections per instance
pg_conns_per_instance = workers × pg_pool_max_size
redis_conns_per_instance = workers × pools_per_worker × redis_max_connections

Connection Lifecycle

Shared Redis clients set socket_connect_timeout=5, health_check_interval=30, socket_keepalive=True, and retry_on_timeout=True. A centralized Redis health monitor tracks PING health and emits up/down events. On reconnect, the service automatically rebuilds SSE relay subscriptions and reconnects gateway config pub/sub listeners.