Downloads n/athis week

GitHub stars 1

Extension descriptor (YAML)

BM25 full-text search for DuckDB — tokenization and Snowball stemming via tantivy, through DuckDB's C Extension API (quack-rs), no Arrow IPC

Installing and Loading

INSTALL tantivy FROM community;
LOAD tantivy;

Example

INSTALL tantivy FROM community;
LOAD tantivy;

CREATE TABLE docs(doc_id BIGINT, body VARCHAR);
INSERT INTO docs VALUES (0, 'the cat sat'), (1, 'a cat here'), (2, 'the dog barked');

SELECT * FROM bm25_search('docs', 'cat');
-- doc_id | score
--        1 | 0.92...
--        0 | 0.92...

SELECT * FROM bm25_search('docs', 'cat', top_k := 1);

About tantivy

tantivy — BM25 Full-Text Search for DuckDB

A native Rust DuckDB loadable extension adding BM25 full-text search (via tantivy, the Rust search engine behind Quickwit) plus tokenization and Snowball stemming for 18 languages — written against DuckDB's C Extension API (ABI v1.2.0) through quack-rs. Table functions bind to real tables and stream real columns: no Arrow IPC, no JSON shims, no external search engine.

Functions

Unicode text is NFC-normalized on both the indexed documents and the query, so canonically-equivalent spellings (precomposed vs decomposed) match.

Hard limits (by design — clean SQL errors, never crashes or silent data loss)

Each call builds an in-RAM index; the extension is a high-precision full-text search layer for medium-sized tables, not a persistent search-engine replacement.

Known limitation: temporary tables and uncommitted transactions are not visible to bm25_search (the C-API callbacks receive no caller connection, so the extension queries through a connection it opens itself). Permanent and attached tables work fine.

Windows and musl builds are excluded pending CI coverage, matching the other quack-rs based community extensions.

→ GitHub

Added Functions

Overloaded Functions

This extension does not add any function overloads.

Added Types

This extension does not add any types.

Added Settings

This extension does not add any settings.