Provides access to faiss indices from DuckDB.
Maintainer(s):
JAicewizard,
arjenpdevries
Installing and Loading
INSTALL faiss FROM community;
LOAD faiss;
Example
-- Generate semi-random input data and queries
-- Note that the dimensionality of our data will be 5
CREATE TABLE input AS SELECT i AS id, apply(generate_series(1, 5), j-> CAST(hash(i*1000+j) AS FLOAT)/18446744073709551615) AS data FROM generate_series(1, 1000) s(i);
CREATE TABLE queries AS SELECT i AS id, apply(generate_series(1, 5), j-> CAST(hash(i*1000+j+8047329823) AS FLOAT)/18446744073709551615) AS data FROM generate_series(1, 10) s(i);
-- Create the index and insert data into it
CALL FAISS_CREATE('name', 5, 'IDMap,HNSW32');
CALL FAISS_ADD((SELECT id, data FROM input), 'name');
-- Get 10 results with uneven id
SELECT id, UNNEST(FAISS_SEARCH_FILTER('name', 10, data, 'id%2==1', 'rowid', 'input')) FROM queries;
-- Get 10 results with even id
SELECT id, UNNEST(FAISS_SEARCH_FILTER('name', 10, data, 'id%2==0', 'rowid', 'input')) FROM queries;
-- Get 10 results
SELECT id, UNNEST(FAISS_SEARCH('name', 10, data)) FROM queries;
About faiss
The FAISS extension allows DuckDB users to store vector data in faiss, and query this data, making reliable vector search more accessible. On all linux platforms, this platform also supports GPU indexes, you can move a supported index to the GPU using CALL FAISS_MOVE_GPU({index_name}, {gpu number}). Currently only CUDA is supported, note that GPU support may be split into a seperate extension in the future.
Some (most) indices are not supported for gpus, however this is very easily resolvable. Please open an issue over at our repository in order to get this resolved!
Added Functions
| function_name | function_type | description | comment | examples |
|---|---|---|---|---|
| __faiss_create_mask | table | NULL | NULL | [] |
| faiss_add | table | NULL | NULL | [] |
| faiss_create | table | NULL | NULL | [] |
| faiss_create_params | table | NULL | NULL | [] |
| faiss_destroy | table | NULL | NULL | [] |
| faiss_load | table | NULL | NULL | [] |
| faiss_manual_train | table | NULL | NULL | [] |
| faiss_save | table | NULL | NULL | [] |
| faiss_search | scalar | NULL | NULL | [] |
| faiss_search_filter | scalar | NULL | NULL | [] |
| faiss_search_filter_set | scalar | NULL | NULL | [] |
| faiss_to_gpu | table | NULL | NULL | [] |