Search Shortcut cmd + k | ctrl + k
lance_conversion
Downloads 696this week
GitHub stars 0
Extension descriptor (YAML)

Stream any DuckDB query into Lance datasets with Blob v2 storage and scalar, vector, text, and Bloom filter indexes

Maintainer(s): dentiny

Installing and Loading 

INSTALL lance_conversion FROM community;
LOAD lance_conversion;

Example 

INSTALL lance_conversion FROM community;
LOAD lance_conversion;

COPY (
  SELECT *
  FROM read_parquet('events/**/*.parquet')
) TO 'events.lance' (FORMAT LANCE);

About lance_conversion

lance_conversion registers Lance as a native DuckDB COPY destination. DuckDB reads, filters, joins, casts, and projects the source query, then streams its output into the Rust Lance writer without an intermediate Parquet export or full-input materialization.

Any source that DuckDB can query can be written to Lance, including DuckDB tables and views, Parquet, CSV, JSON, and S3-backed data:

COPY events TO 'events.lance' (FORMAT LANCE);

COPY (
  SELECT *
  FROM read_json_auto('events.json')
) TO 'events_from_json.lance' (FORMAT LANCE);

The extension also provides read_huggingface for Hugging Face Parquet datasets and read_warc for local or S3 WARC/WARC.GZ files:

COPY (
  SELECT * FROM read_huggingface('lhoestq/demo1')
) TO 'demo1.lance' (FORMAT LANCE);

COPY (
  SELECT id, type, date, content_type, body
  FROM read_warc('archive.warc.gz')
) TO 'archive.lance' (FORMAT LANCE);

read_huggingface fetches only the projected Parquet columns and decodes row groups in the background. max_read_parallelism bounds concurrent row-group reads, and read_ahead_bytes caps the decoded data each one may queue (0 disables read-ahead):

SELECT id, package_name
FROM read_huggingface(
  'lhoestq/demo1',
  max_read_parallelism = 8,
  read_ahead_bytes = 67108864
);

Storage timeouts and retries for S3 and Hugging Face I/O are configured through the lance_conversion_storage_* settings:

SET lance_conversion_storage_io_timeout_ms = 30000;
SET lance_conversion_storage_retry_max_times = 5;

Lance writes support Create (the default), Append, and Overwrite modes:

COPY new_events TO 'events.lance' (FORMAT LANCE, APPEND);
COPY replacement TO 'events.lance' (FORMAT LANCE, OVERWRITE);

When row order need not be preserved, every DuckDB thread runs its own Lance writer, and all writers commit as one dataset version. Use PRESERVE_ORDER false for the fastest writes:

COPY events TO 'events.lance' (FORMAT LANCE, PRESERVE_ORDER false);

SAMPLE_PERCENT and SAMPLE_ROWS write a random sample of the source rows instead of every row:

COPY events TO 'events_10pct.lance' (FORMAT LANCE, SAMPLE_PERCENT 10);
COPY events TO 'events_sample.lance' (FORMAT LANCE, SAMPLE_ROWS 1000);

Blob v2 layout, data file sizing, and indexes are configurable directly from the COPY statement:

COPY (
  SELECT id, event_id, category, description, embedding, asset_uri
  FROM read_parquet('catalog/**/*.parquet')
) TO 'catalog.lance' (
  FORMAT LANCE,
  TARGET_FILE_SIZE 536870912,
  BLOB_INLINE_SIZE_THRESHOLD 2097152,
  BLOB_DEDICATED_SIZE_THRESHOLD 16777216,
  BLOB_COLUMNS (asset_uri),
  SCALAR_INDEX_COLUMNS (id, category),
  VECTOR_INDEX_COLUMNS (embedding),
  TEXT_INDEX_COLUMNS (description),
  BLOOM_FILTER_INDEX_COLUMNS (event_id)
);

Local and S3-compatible Lance destinations are supported. S3 credentials, endpoint, region, TLS, and URL style are resolved from DuckDB secrets and passed to the OpenDAL-backed Lance object store.

Added Functions

function_name function_type description comment examples
read_atif table Reads ATIF agent trajectories from a local or S3 JSON or JSONL file. NULL [SELECT * FROM read_atif('trajectories.jsonl');]
read_huggingface table Reads a Hugging Face dataset's Parquet files. NULL [SELECT * FROM read_huggingface('lhoestq/demo1');]
read_warc table Reads records from a local or S3 WARC file. NULL [SELECT * FROM read_warc('archive.warc.gz');]

Overloaded Functions

This extension does not add any function overloads.

Added Types

This extension does not add any types.

Added Settings

name description input_type scope aliases
lance_conversion_storage_io_timeout_ms IO operation and body method timeout per attempt (ms) BIGINT GLOBAL []
lance_conversion_storage_retry_factor Storage exponential retry backoff multiplier for reads and writes DOUBLE GLOBAL []
lance_conversion_storage_retry_max_delay_ms Maximum exponential retry backoff (ms) BIGINT GLOBAL []
lance_conversion_storage_retry_max_times Maximum retries after the initial attempt (0 disables retries) BIGINT GLOBAL []
lance_conversion_storage_retry_min_delay_ms Initial exponential retry backoff (ms) BIGINT GLOBAL []
lance_conversion_storage_timeout_ms Control operation timeout per attempt (ms) BIGINT GLOBAL []