Search Shortcut cmd + k | ctrl + k

duckir

A DuckDB extension for Information Retrieval. Fast vector search fully integrated with DuckDB, with filtered search on any predicate, indexes larger than memory, and an index that updates alongside your table.

Maintainer(s): lkuffo, Noorts, peterboncz

Compatibility Matrix

This table covers all DuckDB releases that have not reached end-of-life, including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.

Platform	1.4.4	1.4.5	1.5.5	1.5.6	2.0.0
Linux AMD64
Linux ARM64
macOS AMD64
macOS ARM64
Windows AMD64
Windows ARM64
WebAssembly

Installing and Loading

INSTALL duckir FROM community;
LOAD duckir;

Example

-- Create a table with an empty vector index
CREATE TABLE movies (
  title VARCHAR,
  genres VARCHAR[],
  rating DOUBLE,
  embedding FLOAT[1536]
);
CREATE INDEX movies_idx ON movies USING DUCKIR_VSS (embedding) WITH (metric = 'cosine');

-- Insert the data directly from Hugging Face. The index updates alongside the table.
INSERT INTO movies
  SELECT title, genres, imdb.rating, plot_embedding::FLOAT[1536]
  FROM 'hf://datasets/MongoDB/embedded_movies@~parquet/default/train/0000.parquet';

-- Run a filtered vector search. The filters are evaluated first, pushing down the predicates whenever possible.
SET VARIABLE q = (SELECT embedding FROM movies WHERE title = 'The Matrix');

SELECT title, genres, rating
FROM movies
WHERE list_contains(genres, 'Comedy') AND rating > 7
ORDER BY array_cosine_distance(embedding, getvariable('q'))
LIMIT 5;

About duckir

DuckIR is a DuckDB extension for Information Retrieval. Vector Search is ready, and Full Text Search and Hybrid Search are on their way.

Why DuckIR:

Vector search fully integrated with DuckDB, from query execution and predicate pushdown to transactional correctness, checkpoints, and crash recovery.
Fast vector indexing and search. Index millions of vectors in seconds and search them in milliseconds.
Out-of-core: Create and search indexes larger than memory.
Filtered search on any predicate DuckDB can evaluate.
Lightweight maintenance of indexes alongside your table, with no full rebuilds.
Morsel-driven parallelism, one row group at a time, as DuckDB does.

Status: Alpha. Indexes might need to be recreated after an extension update.

See our documentation for maintenance, persistence, out-of-core search, and the known limitations.

Added Functions

function_name	function_type	description	comment	examples
duckir_vss_encode_query	scalar	Reads a list literal written as a string and returns its float32 values as a BLOB	Use it as the query vector of a search. It bypasses the cost of binding a list literal element by element.	[SELECT id FROM t ORDER BY array_distance(embedding, duckir_vss_encode_query('[0.25, 0.79, -0.50]')) LIMIT 10;]
duckir_vss_index_info	table	Returns metadata about all DuckIR vector indexes	One row per index, with its table, index options, and in-memory size.	[CALL duckir_vss_index_info();]
duckir_vss_sync_index	table	Synchronize the index to force it to process its maintenance backlog	Otherwise, the first search after a bulk load pays this indexing cost. Returns the index name and whether it had rows to index.	[CALL duckir_vss_sync_index('my_idx');]

Overloaded Functions

function_name	function_type	description	comment	examples
array_distance	scalar	Euclidean distance between a FLOAT[d] array and a query vector given as a BLOB of float32 values	DuckIR adds this BLOB overload, also as the <-> operator. The BLOB must hold exactly d float32 values.	[SELECT id FROM t ORDER BY array_distance(embedding, duckir_vss_encode_query('[0.25, 0.79, -0.50]')) LIMIT 10;]
array_cosine_distance	scalar	Cosine distance between a FLOAT[d] array and a query vector given as a BLOB of float32 values	DuckIR adds this BLOB overload, also as the <=> operator. The BLOB must hold exactly d float32 values.	[SELECT id FROM t ORDER BY array_cosine_distance(embedding, duckir_vss_encode_query('[0.25, 0.79, -0.50]')) LIMIT 10;]
<->	scalar	Euclidean distance between a FLOAT[d] array and a query vector given as a BLOB of float32 values	Operator form of the array_distance BLOB overload that DuckIR adds. The BLOB must hold exactly d float32 values.	[SELECT id FROM t ORDER BY embedding <-> duckir_vss_encode_query('[0.25, 0.79, -0.50]') LIMIT 10;]
<=>	scalar	Cosine distance between a FLOAT[d] array and a query vector given as a BLOB of float32 values	Operator form of the array_cosine_distance BLOB overload that DuckIR adds. The BLOB must hold exactly d float32 values.	[SELECT id FROM t ORDER BY embedding <=> duckir_vss_encode_query('[0.25, 0.79, -0.50]') LIMIT 10;]

Added Types

This extension does not add any types.

Added Settings

name	description	input_type	scope	aliases	typed_value
duckir_vss_bruteforce_max_rows	the most passing rows a filtered LATERAL index join searches by brute force (default: 100000, 0 disables it)	UBIGINT	GLOBAL	[]	100000
duckir_vss_bruteforce_rows_per_cluster	filtered LATERAL index joins search the passing rows by brute force, without the index, when they number at most this many per cluster of the row groups they are in, and at most duckir_vss_bruteforce_max_rows (default: 2, 0 disables it)	DOUBLE	GLOBAL	[]	2.0
duckir_vss_cache_tiers	in indexes loaded with it on, the clusters searches read most often are the last DuckDB evicts from memory (default: true)	BOOLEAN	GLOBAL	[]	true
duckir_vss_cluster_paging	indexes loaded from a database file read their clusters from it as searches need them, so they can be larger than memory_limit; false loads them fully when they load (default: true)	BOOLEAN	GLOBAL	[]	true
duckir_vss_cluster_sketch_check	when searches of indexes loaded with it check a cluster's sketch before probing it: 'never' or 'when_cluster_not_cached', when the cluster must be read from disk (default: when_cluster_not_cached)	VARCHAR	GLOBAL	[]	when_cluster_not_cached
duckir_vss_experimental_max_cluster_size	incremental maintenance splits clusters when they reach this size, between 32 and 2048 (default: 384)	UBIGINT	GLOBAL	[]	384
duckir_vss_experimental_on_the_fly_indexing	filtered LATERAL index joins whose passing rows are too many for a brute-force search build indexes over only the passing rows and search them instead of the row groups: 'always', 'never' or 'auto' (when there are at most duckir_vss_on_the_fly_indexing_threshold passing rows per query, queries as estimated; default: never)	VARCHAR	GLOBAL	[]	never
duckir_vss_n_probe	override the n_probe parameter when scanning DuckIR VSS indexes (default: 24, must be >= 0)	INTEGER	GLOBAL	[]	NULL
duckir_vss_on_the_fly_indexing_max_row_groups	the most consecutive row groups whose passing rows go into one index built on the fly (default: 0, no limit: the memory left decides)	UBIGINT	GLOBAL	[]	0
duckir_vss_on_the_fly_indexing_threshold	with duckir_vss_experimental_on_the_fly_indexing = 'auto', the indexes over the passing rows are built when there are at most this many passing rows per query, queries as estimated (default: 20)	UBIGINT	GLOBAL	[]	20
duckir_vss_paging_counters	indexes loaded with it on count their cluster cache's acquires, misses and bytes read, of the clusters and of their sketches, and the blocks read, in duckir_vss_index_info (default: false)	BOOLEAN	GLOBAL	[]	false
duckir_vss_rank_clusters_across_row_groups	LATERAL index joins probe clusters nearest to the query across all row groups (default: true); false probes the n_probe nearest of each row group	BOOLEAN	GLOBAL	[]	true
temp_directory	Set the directory to which temporary files are written.	VARCHAR	GLOBAL	[]	post.db.tmp

DuckDB Home Code of Conduct Trademark Use Blog

duckir

In this article