Execution Modes

PatternOps supports four execution modes within a single schema definition.

Modes

Mode	Description	Use Case
batch	Finite dataset, bounded processing	Daily/hourly ETL, reports
streaming	Continuous dataset, unbounded processing	Real-time analytics, event processing
cdc	Change data capture processing	Database replication, incremental sync
hybrid	Multiple engines in one pipeline	Mixed batch + streaming stages

Batch Mode

Process a finite dataset from start to finish.

name: daily-orders
executionMode: batch
stages:
  - name: acquire
    type: source
    capability: source-acquisition
    timeout: PT10M
    config:
      mode: snapshot
  - name: transform
    type: transformation
    capability: transformation
    timeout: PT30M
  - name: publish
    type: publish
    capability: storage
    timeout: PT5M

State machine: PENDING → SUBMITTED → RUNNING → COMPLETED

Streaming Mode

Continuously process an unbounded data stream.

name: realtime-events
executionMode: streaming
stages:
  - name: consume-events
    type: source
    capability: source-acquisition
    timeout: PT1H
    config:
      mode: streaming
  - name: process
    type: transformation
    capability: transformation
    timeout: PT1H
  - name: emit
    type: publish
    capability: messaging
    timeout: PT1H

State machine: DEPLOYING → STARTING → RUNNING → STOPPED

Additional streaming states: DEGRADED, BACKPRESSURED, LAGGING, DRAINING, UPGRADING, RESTARTING

CDC Mode

Process change data capture feeds.

name: customer-sync
executionMode: cdc
stages:
  - name: capture-changes
    type: cdc
    capability: source-acquisition
    timeout: PT5M
    config:
      mode: change_feed
      trackDeletes: true
  - name: apply-changes
    type: transformation
    capability: transformation
    timeout: PT10M
  - name: publish-changes
    type: publish
    capability: storage
    timeout: PT5M

Hybrid Mode

Combine multiple execution engines in one pipeline. Different stages can use different providers.

name: mixed-pipeline
executionMode: hybrid
stages:
  - name: stream-ingest
    type: source
    capability: source-acquisition
    timeout: PT1H
    config:
      mode: streaming
  - name: batch-transform
    type: transformation
    capability: transformation
    timeout: PT30M
    transformation:
      spark:
        style: dataframe
        reference: com.acme.BatchTransform
  - name: stream-publish
    type: publish
    capability: messaging
    timeout: PT1H

In hybrid mode, the Control Plane resolves multiple Execution Providers and coordinates data handoff between engine boundaries.

State Machines

Batch Pipeline States

stateDiagram-v2
    [*] --> PENDING
    PENDING --> SUBMITTED
    SUBMITTED --> RUNNING
    SUBMITTED --> CANCELLED
    RUNNING --> COMPLETED
    RUNNING --> FAILED
    RUNNING --> CANCELLED
    FAILED --> RETRYING
    RETRYING --> RUNNING
    RETRYING --> CANCELLED

Streaming Pipeline States

stateDiagram-v2
    [*] --> DEPLOYING
    DEPLOYING --> STARTING
    DEPLOYING --> FAILED
    STARTING --> RUNNING
    STARTING --> FAILED
    RUNNING --> DEGRADED
    RUNNING --> BACKPRESSURED
    RUNNING --> LAGGING
    RUNNING --> DRAINING
    RUNNING --> FAILED
    RUNNING --> STOPPED
    DEGRADED --> RUNNING
    DEGRADED --> FAILED
    DEGRADED --> STOPPED
    BACKPRESSURED --> RUNNING
    BACKPRESSURED --> DEGRADED
    LAGGING --> RUNNING
    LAGGING --> DEGRADED
    FAILED --> RESTARTING
    FAILED --> STOPPED
    RESTARTING --> STARTING
    DRAINING --> STOPPED
    STOPPED --> DEPLOYING

Choosing a Mode

If you need...	Use
Process all data once per schedule	batch
Continuous real-time processing	streaming
Track and apply database changes	cdc
Mix batch and streaming in one pipeline	hybrid