Skip to content

Documentation Engine

The Documentation Engine automatically generates complete documentation for every deployed pipeline, including technical specs, operational runbooks, data product pages, and platform-specific monitoring dashboards.

Output Formats

Format Use Case Content
MKDOCS Developer/operator docs Markdown with tables, code blocks, mermaid diagrams
OPENAPI API-style documentation OpenAPI 3.0 YAML with endpoints for each pipeline stage
JSON_SCHEMA_DOCS Schema documentation JSON Schema 2020-12 definitions for pipeline structure
MCP_TOOL_CATALOG AI agent integration MCP tool definitions with inputSchema for each pipeline

Usage

Generate Full Documentation Bundle

DocumentationEngine engine = new DefaultDocumentationEngine();

DocumentationContext context = new DocumentationContext(
    OutputFormat.MKDOCS, true, true, Map.of()
);

DocumentationBundle bundle = engine.generate(pipeline, context);

// bundle.technical() → stage details, capability bindings, transformations
// bundle.operational() → monitoring metrics, alerting rules, runbook
// bundle.dataProduct() → ownership, quality, SLA, schema info
// bundle.crossReferences() → links between related docs

Incremental Regeneration

Only regenerate changed sections:

DocumentationBundle updated = engine.regenerate(pipeline, List.of("stage-config.yaml"));
// Only technical doc is regenerated; operational and data product remain "[unchanged]"

Change detection rules:

Changed Source Contains Regenerates
stage, capability, transform Technical
monitor, alert, retry, timeout Operational
schema, quality, sla, dataset Data Product
(unrecognized) All three

Generate Monitoring Dashboards

DashboardBundle dashboards = engine.generateDashboards(ComputePlatformProfile.DATABRICKS);

// dashboards.grafana() → Grafana JSON dashboard model
// dashboards.kibana() → Kibana NDJSON saved objects
// dashboards.splunk() → Splunk XML dashboard
// dashboards.datadog() → Datadog JSON dashboard
// dashboards.cloudwatch() → AWS CloudWatch JSON widgets

Generated Content Examples

MkDocs Technical Documentation

Generated content includes:

  • Pipeline overview table (name, tenancy, namespace, dataset, execution mode, schedule, platform)
  • Stage details with type, capability, provider, timeout, retry policy
  • Configuration blocks rendered as YAML
  • Transformation definitions (Spark mode + SQL mode)
  • Extension and policy references

MkDocs Operational Runbook

Generated content includes:

  • Key metrics table with alert thresholds
  • Per-stage monitoring (timeout, retry config)
  • Critical alerts (Pipeline Failure, Data Quality Breach, SLA Violation)
  • Warning alerts (Elevated Latency, Schema Drift, Resource Pressure)
  • Recovery procedures for hangs, quality failures, partial failures
  • Contact information (pipeline owner, platform team, data steward)

MkDocs Data Product Documentation

Generated content includes:

  • Ownership (domain, team, pipeline, dataset, platform)
  • Quality dimensions table (completeness, accuracy, timeliness, consistency, uniqueness)
  • Quality rules from QUALITY/VALIDATION stages
  • SLA commitment (availability, freshness, completeness, recovery time)
  • Pipeline data flow diagram
  • Per-stage schema information
  • Access and consumption rules

Dashboard Generation

Each platform profile generates dashboards with platform-specific datasources and metric namespaces:

Platform Grafana Datasource Metric Prefix
DATABRICKS databricks-prometheus databricks.pipeline.*
SNOWFLAKE snowflake-prometheus snowflake.pipeline.*
AWS_NATIVE cloudwatch-prometheus aws.pipeline.*
AZURE_NATIVE azure-monitor-prometheus azure.pipeline.*

Dashboard panels include: - Pipeline execution status (stat) - Duration percentiles p50/p95 (timeseries) - Records processed per stage (timeseries) - Error rate gauge with thresholds - Stage latency breakdown table

Cross-References

Generated bundles include cross-references linking:

  • Technical ↔ Operational ↔ Data Product docs
  • Each stage → its capability contract documentation
  • Each stage → its provider implementation documentation
  • Transformations → transformation reference documentation
  • Platform profile → platform documentation