tidbfs

5 downloads this week 1 star Extension descriptor

Read and write analytics files on TiDB Filesystem through its HTTP API, with experimental native DuckDB database storage.

Compatibility Matrix

This table covers all DuckDB releases that have not reached end-of-life, including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.

Installing and Loading

INSTALL tidbfs FROM community;
LOAD tidbfs;

Example

INSTALL tidbfs FROM community;
LOAD tidbfs;

-- Use your TiDB Filesystem data-plane endpoint and API key.
CREATE TEMPORARY SECRET fs (
    TYPE tidbfs,
    ENDPOINT 'https://YOUR_FILESYSTEM_ENDPOINT',
    TOKEN 'YOUR_FILESYSTEM_API_KEY',
    SCOPE 'tidbfs:///'
);

COPY (SELECT i AS id, i * 10 AS value FROM range(100) t(i))
TO 'tidbfs:///example.parquet' (FORMAT PARQUET);

SELECT count(*), sum(value) FROM read_parquet('tidbfs:///example.parquet');

About tidbfs

tidbfs is a native C++ filesystem extension for TiDB Filesystem. It calls the existing HTTP data plane directly, without FUSE or a Go runtime. Its initial community release targets DuckDB 2.0.0 on Linux amd64 (glibc) and macOS arm64. OpenSSL and libcurl are statically linked into the artifact.

The tidbfs:/// scheme supports Parquet, CSV and JSON reads, globbing, Hive partition pruning, COPY TO, partitioned output, and database EXPORT/IMPORT. Reads use conditional HTTP Range requests and parallel reads. Writes stage locally and publish a complete file using the server's revision checks and multipart upload APIs. They require local scratch space; multi-file operations are not atomic transactions.

Experimental tidbfs+db:/// storage can place native DuckDB database and WAL files on the same filesystem using the public extent API and C++ SDK. It preserves the existing sessions, advisory file locks and persistence contract. All writers must cooperate with these locks; this is not a replacement for DuckDB's concurrency rules or a claim of arbitrary multi-process writer support. Native mode requires an unscoped filesystem token and a server with the documented extent/session APIs.

See the compatibility matrix for the supported server capabilities, recovery tests and remaining limits. The publication guide documents the official build and artifact-level regression tests.

Added Functions

This extension does not add any functions.

Overloaded Functions

This extension does not add any function overloads.

Added Types

This extension does not add any types.

Added Settings