Support
Search Shortcut cmd + k | ctrl + k
Search cmd+k ctrl+k
1.0 (stable)
Delta Extension

The delta extension adds support for the Delta Lake open-source storage format. It is built using the Delta Kernel. The extension offers read support for delta tables, both local and remote.

For implementation details, see the announcement blog post.

Warning The delta extension is currently experimental and is only supported on given platforms.

Installing and Loading

The delta extension will be transparently autoloaded on first use from the official extension repository. If you would like to install and load it manually, run:

INSTALL delta;
LOAD delta;

Usage

To scan a local delta table, run:

SELECT *
FROM delta_scan('file:///some/path/on/local/machine');

To scan a delta table in an S3 bucket, run:

SELECT *
FROM delta_scan('s3://some/delta/table');

For authenticating to S3 buckets, DuckDB Secrets are supported:

CREATE SECRET (
    TYPE S3,
    PROVIDER CREDENTIAL_CHAIN
);
SELECT *
FROM delta_scan('s3://some/delta/table/with/auth');

Features

While the delta extension is still experimental, many (scanning) features and optimizations are already supported:

  • multithreaded scans and Parquet metadata reading
  • data skipping/filter pushdown
    • skipping row-groups in file (based on Parquet metadata)
    • skipping complete files (based on Delta partition information)
  • projection pushdown
  • scanning tables with deletion vectors
  • all primitive types
  • structs
  • S3 support with secrets

More optimizations are going to be released in the future.

Supported DuckDB Versions and Platforms

The delta extension requires DuckDB version 0.10.3 or newer.

The delta extension currently only supports the following platforms:

  • Linux AMD64 (x86_64): linux_amd64 and linux_amd64_gcc4
  • macOS Intel and Apple Silicon: osx_amd64 and osx_arm64

Support for the other DuckDB platforms is work-in-progress.

About this page

Last modified: 2024-07-01