The aws extension adds functionality (e.g., authentication) on top of the httpfs extension's S3 capabilities, using the AWS SDK.

Installing and Loading

The aws extension will be transparently autoloaded on first use from the official extension repository. If you would like to install and load it manually, run:

INSTALL aws;
LOAD aws;

In most cases, the aws extension works in conjunction with the [httpfs extension](/docs/stable/extensions/httpfs/overview.html.

Configuration and Authentication

The preferred way to configure and authenticate to AWS S3 endpoints is to use secrets.

config Provider

The default provider, config (i.e., user-configured), allows access to the S3 bucket by manually providing a key. For example:

CREATE OR REPLACE SECRET secret (
    TYPE s3,
    PROVIDER config,
    KEY_ID 'AKIAIOSFODNN7EXAMPLE',
    SECRET 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',
    REGION 'us-east-1'
);

Tip If you get an IO Error (Connection error for HTTP HEAD), configure the endpoint explicitly via ENDPOINT 's3.your_region.amazonaws.com'.

Now, to query using the above secret, simply query any s3:// prefixed file:

SELECT *
FROM 's3://your_bucket/your_file.parquet';

credential_chain Provider

The credential_chain provider allows automatically fetching credentials using mechanisms provided by the AWS SDK. For example, to use the AWS SDK default provider:

CREATE OR REPLACE SECRET secret (
    TYPE s3,
    PROVIDER credential_chain
);

Again, to query a file using the above secret, simply query any s3:// prefixed file.

DuckDB also allows specifying a specific chain using the CHAIN keyword. This takes a semicolon-separated list (a;b;c) of providers that will be tried in order. For example:

CREATE OR REPLACE SECRET secret (
    TYPE s3,
    PROVIDER credential_chain,
    CHAIN 'env;config'
);

The possible values for CHAIN are the following:

• config
• sts
• sso
• env
• instance
• process

The credential_chain provider also allows overriding the automatically fetched config. For example, to automatically load credentials, and then override the region, run:

CREATE OR REPLACE SECRET secret (
    TYPE s3,
    PROVIDER credential_chain,
    CHAIN config,
    REGION 'eu-west-1'
);

Legacy Features

Deprecated The load_aws_credentials function is deprecated.

Prior to version 0.10.0, DuckDB did not have a Secrets manager, to load the credentials automatically, the AWS extension provided a special function to load the AWS credentials in the legacy authentication method.

Load AWS Credentials (Legacy)

To load the AWS credentials, run:

CALL load_aws_credentials();

The function takes a string parameter to specify a specific profile:

CALL load_aws_credentials('minio-testing-2');

There are several parameters to tweak the behavior of the call:

CALL load_aws_credentials('minio-testing-2', set_region = false, redact_secret = false);