Skip to content

Architecture Overview

Layered Architecture

PatternOps uses a four-layer architecture that separates intent from implementation:

graph TB
    subgraph "Layer 1: Pipeline Definitions"
        PD[Pipeline Definition Model]
        CP[Configuration Parser/Printer]
    end

    subgraph "Layer 2: Control Plane"
        IN[Intake Service]
        PW[Pathway Service]
        PE[Policy Engine]
        EC[Echo Service]
        PR[Provider Registry]
        MCP[MCP Layer]
        DE[Documentation Engine]
    end

    subgraph "Layer 3: Capability Contracts"
        SC[Source Acquisition]
        EX[Execution]
        TR[Transformation]
        SE[Security]
        DQ[Data Quality]
        MORE[+ 15 more...]
    end

    subgraph "Layer 4: Provider Layer"
        SP[Spark, Flink, Snowflake...]
        OP[Airflow, Prefect, Dagster...]
        MP[Kafka, Pulsar, SQS...]
    end

    PD --> IN & PW
    CP --> PD
    IN --> SC
    PW --> TR
    PE --> EC
    PR --> SC & EX & TR & SE & DQ & MORE
    SC --> SP
    EX --> SP
    TR --> SP

Design Principles

1. Capability-Driven

Every operation in PatternOps is expressed through a Capability Contract — a stable interface that defines WHAT must happen without specifying HOW.

// The contract defines the operations
interface TransformationContract extends BaseProviderContract {
    ValidationResult validateTransformation(TransformationDefinition definition);
    ExecutionResult execute(TransformationDefinition definition, String inputRef, String outputRef);
    Explanation explain(TransformationDefinition definition);
    FieldLevelLineage lineage(TransformationDefinition definition);
    TransformationDefinition optimise(TransformationDefinition definition);
}

// Providers implement the HOW
class SparkTransformationProvider implements TransformationContract { ... }
class SnowflakeTransformationProvider implements TransformationContract { ... }
class DbtTransformationProvider implements TransformationContract { ... }

2. Technology-Free Pipeline Definitions

Pipeline YAML/JSON never contains provider-specific constructs. The validator actively rejects:

  • Spark configs (sparkConf, numExecutors, driverMemory)
  • Airflow constructs (dag_id, task_id, operator)
  • Cloud-specific references (s3Bucket, lambdaArn, glueJobName)
  • Databricks specifics (clusterSpec, notebookPath, dbfsPath)

3. Multi-Platform Portability

Dual-mode transformations declare both Spark and SQL implementations. The platform profile determines which executes:

flowchart TD
    A[Pipeline Stage] --> B{Platform Profile?}
    B --> C[Databricks]
    B --> D[Snowflake]
    B --> E[AWS Native]

    C --> F{Spark Available?}
    F -->|Yes| G[Spark Execution]
    F -->|No| H[SQL Fallback]

    D --> I[SQL Execution]
    E --> J{EMR or Glue/Athena?}
    J -->|EMR| G
    J -->|Glue/Athena| I

4. Observable by Design

Every component emits:

  • Traces via OpenTelemetry SDK
  • Metrics in the canonical MetricsEnvelope format
  • Structured logs as JSON with W3C Trace Context correlation
  • Lineage events compatible with OpenLineage

5. Policy Engine as Authority

The Policy Engine is the single decision-maker for:

  • Retry strategies (fixed, exponential, linear backoff)
  • Recovery actions (restart from checkpoint, quarantine, halt)
  • Scaling decisions (up/down based on metrics)
  • Quality breach responses (quarantine failing records, halt pipeline)

Control Plane Services

Service Responsibility Key Operations
Intake Data acquisition orchestration acquire, checkpoint, resume
Echo Telemetry hub (state, metrics, lineage, events) transitionState, persistMetric, queryLineage
Pathway Transformation orchestration transform, resolveExecutionStrategy
Policy Engine Governance and automation evaluate, handleFailure, scheduleCatchUp
Provider Registry Provider lifecycle and resolution register, resolve, healthCheck
MCP Layer AI integration discover, invoke, getSafetyClass
Documentation Engine Auto-generated docs generate, generateDashboards

Request Flow

sequenceDiagram
    participant User
    participant PDM as Pipeline Definition Model
    participant PR as Provider Registry
    participant CP as Control Plane
    participant Echo as Echo Service
    participant Provider as Resolved Provider

    User->>PDM: Submit pipeline config
    PDM->>PDM: Parse & validate schema
    PDM->>PR: Resolve capability bindings
    PR-->>PDM: Provider instances
    PDM-->>User: Validation result

    User->>CP: Execute pipeline
    CP->>Provider: Submit pipeline
    Provider->>Echo: Emit state transition (RUNNING)

    loop Each Stage
        Provider->>Provider: Execute stage lifecycle
        Provider->>Echo: Emit metrics + lineage
    end

    Provider->>Echo: Emit state transition (COMPLETED)

Stage Lifecycle

Every stage follows the same lifecycle regardless of type:

configure → initialise → execute → observe → persist state → emit event

Terminal outcomes:

  • Complete — Stage succeeded
  • Retry — Stage failed but is retryable (up to configured max, default 3, max 10)
  • Fail — Retries exhausted or non-retryable error