Skip to content

Define data trust once. Use it from development to production.

Bad data usually appears after it has reached a dashboard or decision.
LakeLogic turns version-controlled YAML contracts into executable checks for your data pipelines.

Define schema, quality rules, PII handling, lineage, service levels, and materialization once. Run the contract with Polars, DuckDB, or Spark.

Git-native  ·  Runs in your environment  ·  Apache 2.0

Run the 5-Minute Quickstart View on GitHub

orders_contract.yaml
version: 1.0.0
dataset: orders

info:
  owner: data-team@company.com
  target_layer: silver

model:
  fields:
    - name: order_id
      type: integer
      required: true
    - name: customer_email
      type: string
      required: true
      pii: true
      masking: partial
    - name: amount
      type: float
      required: true

quality:
  row_rules:
    - name: valid_email
      sql: "customer_email LIKE '%@%.%'"
    - name: positive_amount
      sql: "amount > 0"
Python
from lakelogic import DataProcessor

processor = DataProcessor(
    "orders_contract.yaml",
    engine="polars",
)
result = processor.run_source("orders.csv")

print(f"Accepted: {result.good_count}")
print(f"Quarantined: {result.bad_count}")
Result
Accepted: 98
Quarantined: 2

Failed records retain diagnostic context:
- valid_email: customer_email is invalid
- positive_amount: amount must be greater than 0

Row-level failures can be inspected or written to a quarantine target. Dataset-level rules can stop downstream processing when a configured threshold is breached.

Pre-deployment validation
lakelogic validate \
  --contract orders_contract.yaml \
  --gates breaking_change,pii_classification,lineage_break

Add the command to your pull-request workflow to reject changes when a configured gate fails. Some gates require comparison or lineage context from your repository.

Start in Five Minutes

Install the base package:

pip install lakelogic

Then open the five-minute Colab quickstart. It creates sample data, runs a contract, and shows accepted and quarantined records without requiring Spark or cloud credentials.

Choose an Execution Engine

The contract model is shared across engines, but their storage and catalog capabilities are not identical.

Engine Start here when you need Notes
Polars Local development, notebooks, CI, and fast single-node processing Included in the base package. Delta operations use delta-rs.
DuckDB Embedded analytical SQL and local workflows Included in the base package. Some materialization combinations differ from Spark.
Spark Distributed lakehouse workloads and managed catalogs Install with pip install "lakelogic[spark]". Platform configuration still applies.

Review the engine and format capability matrix before selecting a catalog, table format, or materialization strategy.

Choose Your Path

Define schemas, row rules, dataset rules, quarantine behaviour, service levels, and materialization.

Run guided notebooks for quality checks, edge cases, dimensional models, and integrations.

Compare local, CI, Spark, Delta, Iceberg, and catalog-aware execution paths.

Structure domains, systems, ownership, shared defaults, and registries without centralising business meaning.

Declare PII-handling and retention patterns, then connect them to the access, audit, and legal controls your organisation requires.

Use configured providers to assist contract onboarding or extract structured data before normal contract validation.

What LakeLogic Controls

LakeLogic parses contract declarations, executes supported validation and transformation rules, separates accepted and failed records, and emits run evidence. It can also materialize outputs, evaluate service-level thresholds, and invoke configured notification integrations.

LakeLogic does not create domain ownership, cloud permissions, alert-delivery infrastructure, retention jobs, or regulatory compliance by itself. Those responsibilities remain with your platform and organisation. The contract provides a versioned control surface that can participate in them.

For catalog-specific behaviour, see capabilities, cloud integration, and automatic credentials.

Continue Learning

Ready to try it? Run the quickstart or install LakeLogic locally.