Skip to content

PatternOps User Guide

Enterprise Data Engineering Operating System

PatternOps decouples pipeline intent from execution technology. Define WHAT your data pipelines must do; let interchangeable Providers determine HOW.

graph LR
    A[Pipeline Definition] --> B[Control Plane]
    B --> C[Capability Contracts]
    C --> D[Provider Layer]
    D --> E[Spark / Flink / Snowflake / ...]

Why PatternOps?

Problem PatternOps Solution
Vendor lock-in Technology-free pipeline definitions
Rewrite on migration Swap providers without touching pipelines
Inconsistent observability OpenTelemetry + structured logging everywhere
Manual documentation Auto-generated docs for every pipeline
AI integration complexity MCP tools expose the full platform to AI agents

Implementation Status

Module Component Tests
core Domain Models, Pipeline Definition Model, Provider Registry, Echo Service, Policy Engine, Stage Lifecycle 231
capabilities 21 Capability Contract Interfaces —
providers Apache Spark (file, JDBC, transform, quality), Apache Flink (stream, transform, storage) 44
mcp MCP Layer (14 tools, 5 categories), AccessController, Safety Classification 44
agent AI Pipeline Constructor, Signal Analysis, Pattern Detection, Confidence Scoring 33
build Distribution Compiler, Dependency Resolver, Platform Compatibility 21
migration Infrastructure Scanner, Pipeline Generator, Risk Assessor 29
documentation Doc Engine (MkDocs, OpenAPI, JSON Schema, MCP Catalog), Dashboard Templates 34
echo-api Spring Boot REST API (state, metrics, lineage, events, quality) 10
Total 9 modules, all fully implemented 446+ tests

Key Principles

  1. Capability-Driven — Define WHAT must happen; providers determine HOW
  2. Technology-Free Pipelines — No Spark configs, no Airflow DAGs, no cloud-specific constructs in your pipeline YAML
  3. Multi-Platform Portable — Dual-mode transformations (Spark + SQL) with automatic execution strategy selection
  4. Observable by Design — OpenTelemetry traces, structured JSON logs, W3C Trace Context correlation
  5. Multi-Tenant by Default — Tenancy and namespace isolation at every layer
  6. AI-Native — MCP tools with safety classification for AI agent integration

Quick Example

name: customer-orders-pipeline
tenancy: acme-corp
namespace: production
dataset: customer-orders
executionMode: batch
platformProfile: databricks

stages:
  - name: acquire-orders
    type: source
    capability: source-acquisition
    timeout: PT5M
    config:
      mode: incremental
      watermark: last_modified

  - name: validate-quality
    type: quality
    capability: data-quality
    timeout: PT3M

  - name: transform-orders
    type: transformation
    capability: transformation
    timeout: PT10M
    transformation:
      spark:
        style: dataframe
        reference: com.acme.transforms.OrderAggregation
      sql:
        style: sql
        reference: transforms/order_aggregation.sql

  - name: publish-output
    type: publish
    capability: storage
    timeout: PT5M

policies: []
extensions: []

No Spark configs. No Airflow operators. No S3 bucket names. Just intent.

Architecture at a Glance

Layer Components
Pipeline Definitions YAML/JSON configs parsed by the Pipeline Definition Model
Control Plane Intake, Echo, Pathway, Policy Engine, Provider Registry, MCP Layer, Documentation Engine
Capability Contracts 20+ stable interfaces (Source, Execution, Transformation, Security, etc.)
Provider Layer Pluggable implementations (Databricks, Snowflake, Kafka, Airflow, etc.)

Next Steps

Define Your First Pipeline

Start with a simple YAML pipeline definition. PatternOps supports YAML, JSON, and HOCON formats. See the Quick Start guide.

Choose Your Providers

PatternOps ships with Apache Spark and Apache Flink providers out of the box. See the Spark Provider Guide or Flink Provider Guide.

Explore Capabilities

20+ capability contracts cover source acquisition, transformation, storage, security, data quality, and more. See the Capabilities Overview.