CREATE SECRET statement creates a new secret in the Secrets manager, which provides unified user interface for secrets across all backends that use them. Secrets can be scoped, so different storage prefixes can have different secrets, allowing for example to join data across organizations in a single query. Secrets can also be persisted, so that they do not need to be specified every time DuckDB is launched.
Secrets were introduced with DuckDB version 0.10.
Warning Persistent secrets are stored in unencrypted binary format on the disk.
Secrets are typed, their type identifies which service they are for. Currently, the following cloud services are available:
- AWS S3 (
S3), through the
- Google Cloud Storage (
GCS), through the
- Cloudflare R2 (
R2), through the
- Azure Blob Storage (
AZURE), through the
For each type, there are one or more “secret providers” that specify how the secret is created. Secrets can also have an optional scope, which is a file path prefix that the secret applies to. When fetching a secret for a path, the secret scopes are compared to the path, returning the matching secret for the path. In the case of multiple matching secrets, the longest prefix is chosen.
Secrets can be temporary or persistent. Temporary secrets are used by default – and are stored in-memory for the life span of the DuckDB instance similar to how settings worked previously. Persistent secrets are stored in unencrypted binary format in the
~/.duckdb/stored_secrets directory. On startup of DuckDB, persistent secrets are read from this directory and automatically loaded.
To create a secret, a Secret Provider needs to be used. A Secret Provider is a mechanism through which a secret is generated. To illustrate this, for the
AZURE secret types, DuckDB currently supports two providers:
CONFIG provider requires the user to pass all configuration information into the
CREATE SECRET, whereas the
CREDENTIAL_CHAIN provider will automatically try to fetch credentials. When no Secret Provider is specified, the
CONFIG provider is used. For more details on how to create secrets using different providers checkout the respective pages on httpfs and azure
To create a temporary unscoped secret to access S3, we can now use the following:
CREATE SECRET (
Note that we implicitly use the default
CONFIG secret provider here.
In order to persist secrets between DuckDB database instances, we can now use the
CREATE PERSISTENT SECRET command, e.g.:
CREATE PERSISTENT SECRET my_persistent_secret (
This will write the secret (unencrypted) to the
Secrets can be deleted using the
DROP SECRET statement, e.g.:
DROP PERSISTENT SECRET my_persistent_secret;
If two secrets exist for a service type, the scope can be used to decide which one should be used. For example:
CREATE SECRET secret1 (
CREATE SECRET secret2 (
Now, if the user queries something from
secret2 will be chosen automatically for that request. To see which secret is being used, the
which_secret scalar function can be used, which takes a path and a secret type as parameters:
SELECT which_secret('s3://my-other-bucket/file.parquet', 'S3');
Secrets can be listed using the built-in table-producing function, e.g., by using the
duckdb_secrets() table function:
Sensitive information will be redacted.