Real-time DuckLake tables, exactly once. Keeps SQL views over Kafka as DuckLake tables, updated every few seconds.
Compatibility Matrix
This table covers all DuckDB releases that have not reached end-of-life, including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.
| Platform | 1.5.5 | 1.5.6 | 2.0.0 |
|---|---|---|---|
| Linux AMD64 | |||
| Linux ARM64 | |||
| macOS AMD64 | |||
| macOS ARM64 | |||
| Windows AMD64 | |||
| Windows ARM64 | |||
| WebAssembly |
Installing and Loading
INSTALL duckwake FROM community;
LOAD duckwake;
Example
-- A DuckLake catalog in PostgreSQL and a Kafka topic of JSON events.
CALL duckwake_connect('ducklake:postgres:dbname=lake host=localhost user=lake password=lake', data_path := 'lake_data/');
CALL duckwake_create_source('clicks', 'kafka', '{"brokers":"localhost:9092","topic":"clicks","event_time":"ts",
"columns":{"ts":"TIMESTAMP","game_id":"VARCHAR","amount":"DECIMAL(18,2)"}}');
CALL duckwake_create_view('analytics.clicks_per_min',
'SELECT window_start, game_id, count(*) AS n, sum(amount) AS revenue
FROM TUMBLE(clicks, ts, INTERVAL 1 MINUTE) GROUP BY ALL');
CALL duckwake_worker_start();
CALL duckwake_start('analytics.clicks_per_min');
FROM duckwake_jobs();
About duckwake
DuckWake keeps a SQL view's table in a DuckLake lake current as events arrive from Kafka. Each epoch (every
10 seconds by default) commits the view's new rows, its state and the Kafka offsets in one DuckLake
transaction, so a crash never loses or repeats an event, and duckwake_verify checks a view against its
query run as one batch. Views can filter, join lake tables, aggregate over tumbling and hopping windows or
per group, and read other views and lake tables through their change feed.
The extension runs a worker inside the DuckDB process (duckwake_worker_start); the same engine also runs
as the duckwaked server and the duckwake CLI. It needs a DuckLake catalog in PostgreSQL and Kafka
(JSON messages). Documentation: https://duckwake.github.io
Added Functions
| function_name | function_type | description | comment | examples |
|---|---|---|---|---|
| duckwake_add_source_column | table | NULL | NULL | |
| duckwake_alter_view | table | NULL | NULL | |
| duckwake_checkpoints | table | NULL | NULL | |
| duckwake_connect | table | NULL | NULL | |
| duckwake_create_source | table | NULL | NULL | |
| duckwake_create_view | table | NULL | NULL | |
| duckwake_drain | table | NULL | NULL | |
| duckwake_drop | table | NULL | NULL | |
| duckwake_explain | table | NULL | NULL | |
| duckwake_jobs | table | NULL | NULL | |
| duckwake_metrics | table | NULL | NULL | |
| duckwake_pause | table | NULL | NULL | |
| duckwake_reset | table | NULL | NULL | |
| duckwake_sources | table | NULL | NULL | |
| duckwake_start | table | NULL | NULL | |
| duckwake_stop | table | NULL | NULL | |
| duckwake_verify | table | NULL | NULL | |
| duckwake_wait | table | NULL | NULL | |
| duckwake_worker_start | table | NULL | NULL | |
| duckwake_worker_status | table | NULL | NULL | |
| duckwake_worker_stop | table | NULL | NULL | |
| duckwake_workers | table | NULL | NULL |
Overloaded Functions
This extension does not add any function overloads.
Added Types
This extension does not add any types.
Added Settings
| name | description | input_type | scope | aliases | typed_value |
|---|---|---|---|---|---|
| temp_directory | Set the directory to which temporary files are written. | VARCHAR | GLOBAL | [] | post.db.tmp |