Add per-operation timeout and retry for httpfs extension
Maintainer(s):
dentiny
Installing and Loading
INSTALL httpfs_timeout_retry FROM community;
LOAD httpfs_timeout_retry;
Example
SET httpfs_timeout_file_operation_ms = 30000; -- 30 seconds
SET httpfs_timeout_list_ms = 60000; -- 60 seconds
SET httpfs_timeout_delete_ms = 30000; -- 30 seconds
SET httpfs_timeout_stat_ms = 10000; -- 10 seconds
SET httpfs_timeout_create_dir_ms = 20000; -- 20 seconds
SET http_timeout = 30000; -- This will be used for operations without per-operation settings
SET httpfs_retries_file_operation = 3; -- applies to open, read, and write operations
SET httpfs_retries_list = 3;
SET httpfs_retries_delete = 2;
SET httpfs_retries_stat = 2;
SET httpfs_retries_create_dir = 3;
SET http_retries = 3; -- This will be used for operations without per-operation settings
About httpfs_timeout_retry
This extension allows per-operation timeout and retry config to DuckDB httpfs extension. It supports a few key features:
- It's 100% compatible to httpfs extension, so users don't need to change any of their code.
- It supports a few IO operations, like, open/read/write, list, stats, etc.
Added Functions
This extension does not add any functions.
Overloaded Functions
This extension does not add any function overloads.
Added Types
This extension does not add any types.
Added Settings
| name | description | input_type | scope | aliases |
|---|---|---|---|---|
| httpfs_retries_create_dir | Maximum number of retries for creating directories | UBIGINT | GLOBAL | [] |
| httpfs_retries_delete | Maximum number of retries for deleting files | UBIGINT | GLOBAL | [] |
| httpfs_retries_file_operation | Maximum number of retries for file operations (open/read/write) | UBIGINT | GLOBAL | [] |
| httpfs_retries_list | Maximum number of retries for listing directories | UBIGINT | GLOBAL | [] |
| httpfs_retries_stat | Maximum number of retries for stat/metadata operations | UBIGINT | GLOBAL | [] |
| httpfs_timeout_create_dir_ms | Timeout for creating directories (in milliseconds) | UBIGINT | GLOBAL | [] |
| httpfs_timeout_delete_ms | Timeout for deleting files (in milliseconds) | UBIGINT | GLOBAL | [] |
| httpfs_timeout_file_operation_ms | Timeout for file operations (open/read/write) (in milliseconds) | UBIGINT | GLOBAL | [] |
| httpfs_timeout_list_ms | Timeout for listing directories (in milliseconds) | UBIGINT | GLOBAL | [] |
| httpfs_timeout_stat_ms | Timeout for stat/metadata operations (in milliseconds) | UBIGINT | GLOBAL | [] |