Monitoring¶
Service UIs¶
Most tool UIs are reached through the shared Caddy dev-proxy at *.vectis.eto (the containers only expose: their ports internally). Meilisearch and the RustFS S3 API are host-mapped directly.
| Tool | URL (dev) | Purpose |
|---|---|---|
| Temporal UI | temporal.vectis.eto | Workflow execution history, task queue status, running/failed workflows (internal port 8080) |
| Redpanda Console | redpanda.vectis.eto | Topic inspection, consumer group lag, message browsing |
| Meilisearch Dashboard | localhost:7700 | Search index stats, document counts (host-mapped) |
| RustFS | localhost:9000 | Object storage (S3 API, host-mapped). The web console listens on internal port 9001 (RUSTFS_CONSOLE_ENABLE=true); bucket inspection also works with any S3 client, e.g. mc |
| Strawberry GraphQL IDE | api.vectis.eto/graphql | Interactive API testing |
| Prometheus | prometheus.vectis.eto | Metrics scrape + ad-hoc PromQL |
| Grafana | grafana.vectis.eto | Dashboards (default login admin / admin); see the seeded dashboards below |
Application Logging¶
Vectis uses Python's standard logging module. All services log to stdout for container-native collection.
Log Levels¶
| Level | When Used |
|---|---|
INFO |
Request handling, order creation, state transitions |
WARNING |
Non-critical issues — stale cache, missing optional config |
ERROR |
Failures — database errors, payment gateway failures, unhandled exceptions |
DEBUG |
Detailed tracing — SQL queries, event payloads (development only) |
Configure with the LOG_LEVEL environment variable (default: INFO).
Temporal Workflows¶
Monitor long-running business processes in the Temporal UI. Notable workflows:
- OrderLifecycleWorkflow — tracks order from creation through fulfillment
- RecurringOrderWorkflow — scheduled subscription order placement
- CustomerApprovalWorkflow + OrderApprovalWorkflow — B2B account registration and order approval
- BulkPriceImportWorkflow / ImportEntityWorkflow — bulk data imports
- RefundExecutionWorkflow (refund_approval module) — per-tender refund execution with idempotent retry
- VoidExpiringCardAuthsWorkflow — daily sweep of card auths that aged past their expiry
- ExpireStaleReservationsWorkflow — TTL expiry on HELD inventory reservations
- GcExpiredOverdraftDraftsWorkflow — store-credit overdraft draft GC
- GcExpiredRmaDraftsWorkflow, GcExpiredCartTendersWorkflow, CleanupExpiredCartsWorkflow — periodic GC
- RebuildComplianceCacheWorkflow — nightly compliance cache refresh
Check the Task Queues tab to verify workers are connected and processing tasks.
Warning
If workflows accumulate in "Running" state without progress, check that the Temporal worker is running and connected: make worker or the temporal-worker Docker service.
Redpanda Events¶
Topics follow the versioned convention vectis.<domain>.<event>.v<N>. Key topics to monitor:
| Topic | Normal Volume | Alert If |
|---|---|---|
vectis.orders.placed.v1 |
Proportional to order volume | Consumer lag > 1000 messages |
vectis.orders.modified.v1 |
Proportional to order edits | Consumer lag > 1000 messages |
vectis.cart.line_added.v1 |
Proportional to cart activity | Consumer lag growing steadily |
vectis.inventory.stock_changed.v1 |
Proportional to stock adjustments | Consumer lag growing steadily |
vectis.accounts.created.v1 |
Low (account creation) | Any consumer errors |
vectis.fraud.check_requested.v1 |
Proportional to risky orders | Any consumer errors |
vectis.leads.created.v1 |
Low (lead capture) | Any consumer errors |
Use the Redpanda Console to check consumer group lag and browse recent messages for debugging. The check-redpanda-topics gate (part of make check) fails the build if a producer emits a topic that isn't documented in vectis/docs/REDPANDA_TOPICS.md, preventing topic drift.
Key Metrics¶
For production monitoring, expose and track:
| Metric | Source | Threshold |
|---|---|---|
| API response time (p95) | Uvicorn access logs | < 500ms |
| Database connection pool utilization | SQLAlchemy pool stats | < 80% |
| Valkey memory usage | Valkey INFO command |
< available memory |
| Temporal workflow failure rate | Temporal metrics | < 1% |
| Redpanda consumer lag | Consumer group offset | < 1000 |
| Meilisearch index freshness | Last indexed timestamp | < 5 min lag |
Alerting Recommendations¶
- API 5xx rate > 1% — check application logs for stack traces
- Database connections exhausted — increase pool size or investigate slow queries
- Temporal task queue backlog — add worker replicas
- Redpanda consumer lag increasing — event consumer crashed or overwhelmed
- Workflow faults emitted — check
Analytics → Workflow Faultsin the admin; a sudden spike usually means a setting change broke a workflow assumption - Schema validation warnings on Redpanda emit — schemas in
backend/vectis/events/schemas/diverged from a producer; fix the producer or update the schema
Application Metrics¶
The API exposes two endpoints for orchestration and metrics scraping:
| Endpoint | Purpose |
|---|---|
GET /metrics |
Prometheus-format metrics from prometheus-fastapi-instrumentator: http_requests_total, http_request_duration_seconds, http_requests_inprogress, http_response_size_bytes |
GET /ready |
Readiness probe — parallel-checks DB, Valkey, and Redpanda; returns 200 OK only when all three are healthy. Use this for load-balancer health and Kubernetes readiness probes |
GET /health |
Cheap liveness check (no dependency probes). Use for container restart loops |
{ health } GraphQL query |
Legacy in-graph health field (returns "Vectis Commerce API is healthy") |
Six Grafana dashboards ship in infra/grafana/dashboards/ and are auto-provisioned:
| Dashboard | Focus |
|---|---|
vectis_api.json |
Request rate, error rate by status, latency (p50/p95/p99), in-progress requests, payload size (p95), plus a Totals header |
vectis_cache.json |
Valkey cache hit/miss and eviction metrics |
vectis_postgres.json |
PostgreSQL connections, transactions, and query stats (via postgres-exporter) |
vectis_redis.json |
Valkey memory, connections, and ops/sec (via redis-exporter) |
vectis_redpanda.json |
Redpanda broker throughput and consumer lag |
vectis_temporal.json |
Temporal workflow/activity execution and task-queue metrics |
Token-Bucket Rate Limiting¶
The rate limiter (backend/vectis/core/rate_limit.py) is a proper token-bucket: burst capacity is the ceiling, not a steady-state floor. Per-endpoint policies live in the rate_limit_policies table (model in backend/vectis/modules/rate_limit/models.py) and reload on change. The limiter attributes calls by X-Forwarded-For, so a BFF that forwards the real client IP gets accurate per-client buckets rather than collapsing everything into one BFF-IP bucket.
JSON Schema Event Registry¶
Producer-side payloads validate against the schema registry in backend/vectis/events/schemas/. Dev and test default to strict mode (invalid payloads raise); production defaults to warn-and-publish (logs a structured warning so a bad payload never blocks a transaction). Override with EVENT_SCHEMA_MODE=strict|warn. The set of registered topics lives in vectis/docs/REDPANDA_TOPICS.md.