Monitoring & Dashboards¶

PatternOps provides built-in observability with automatic dashboard generation for multiple platforms.

Observability Stack¶

Signal	Standard	Correlation
Traces	OpenTelemetry OTLP	W3C Trace Context `trace-id`
Metrics	MetricsEnvelope (canonical)	tenancy/namespace/pipeline/runId
Logs	Structured JSON	correlationId (same trace-id)
Lineage	OpenLineage	pipeline/runId/stage

Cross-Signal Correlation¶

A single correlationId (W3C Trace Context trace-id) links all signals for a pipeline execution:

Trace: trace-id=abc123 → spans for each stage
Metrics: correlationId=abc123 → records_processed, duration, errors
Logs: correlationId=abc123 → structured log entries
Lineage: runId=run-001 → data flow graph

Dashboard Generation¶

The Documentation Engine generates monitoring dashboards automatically:

DashboardBundle dashboards = documentationEngine.generateDashboards(
    ComputePlatformProfile.DATABRICKS
);

// dashboards.grafana()    → Grafana JSON model
// dashboards.kibana()     → Kibana NDJSON export
// dashboards.splunk()     → Splunk XML dashboard
// dashboards.datadog()    → Datadog JSON dashboard
// dashboards.cloudwatch() → CloudWatch JSON dashboard

Key Metrics¶

Metric	Unit	Description
`records.processed`	count	Records processed per stage
`stage.duration`	milliseconds	Stage execution time
`pipeline.duration`	milliseconds	Total pipeline execution time
`bytes.read`	bytes	Data volume read
`bytes.written`	bytes	Data volume written
`error.count`	count	Errors per stage
`quality.score`	ratio	Data quality score (0.0–1.0)
`consumer.lag`	count	Messaging consumer lag
`replication.throughput`	bytes_per_second	Replication speed

Alerting¶

Configure alerts through the Observability capability:

observabilityProvider.configureAlerts(List.of(
    new AlertRule("high-error-rate", "error.count > 100", "5m", "critical"),
    new AlertRule("slow-pipeline", "pipeline.duration > 3600000", "1m", "warning"),
    new AlertRule("quality-breach", "quality.score < 0.95", "immediate", "critical")
));

Structured Logging¶

All services emit structured JSON logs:

{
  "timestamp": "2024-01-15T10:30:00.123Z",
  "level": "INFO",
  "service": "pathway-service",
  "tenancy": "acme-corp",
  "namespace": "production",
  "pipeline": "daily-ingest",
  "runId": "run-001",
  "correlationId": "abc123def456",
  "message": "Stage 'transform-orders' completed in 45230ms"
}