Downloads 9this week

GitHub stars 0

Extension repository on GitHub

Extension descriptor (YAML)

Better HTTP statistics for DuckDB

Installing and Loading

INSTALL http_stats FROM community;
LOAD http_stats;

Example

-- Run a query against remote source(s) with profiling enabled.
EXPLAIN ANALYZE SELECT * FROM (
    SELECT count(*) FROM read_json('https://httpbin.org/json')
    UNION ALL
    SELECT count(*) FROM read_json('https://jsonplaceholder.typicode.com/todos/1')
);

-- ┌──────────────────────────────────────┐
-- │┌────────────────────────────────────┐│
-- ││             HTTP Stats             ││
-- ││                                    ││
-- ││    jsonplaceholder.typicode.com    ││
-- ││    ────────────────────────────    ││
-- ││          Total Requests: 2         ││
-- ││           HEAD: 1  GET: 1          ││
-- ││     in: 83 bytes  out: 0 bytes     ││
-- ││          Total Time: 0.19s         ││
-- ││                                    ││
-- ││             httpbin.org            ││
-- ││             ───────────            ││
-- ││          Total Requests: 2         ││
-- ││           HEAD: 1  GET: 1          ││
-- ││     in: 429 bytes  out: 0 bytes    ││
-- ││          Total Time: 1.12s         ││
-- │└────────────────────────────────────┘│
-- └──────────────────────────────────────┘

About http_stats

The HTTP stats extension transparently intercepts all HTTP traffic and collects request statistics (request counts, bytes sent and received and request time) on a per-host basis. The collected statistics are surfaced through DuckDB's EXPLAIN ANALYZE output.

The extension works alongside the HTTPFS extension, which provides DuckDB with HTTP(S) and S3 file system support. When both extensions are loaded, HTTP stats wraps HTTPFS's HTTP backend to measure traffic while HTTPFS handles the actual transport.

Key advantages over HTTPFS's built-in profiling:

• Works correctly when querying DuckDB databases stored on S3.
• Provides per-host breakdown for queries that fetch data from multiple remote servers.

For detailed documentation, visit the extension repository.

Added Functions

| function_name | function_type | description | comment | examples | |—————|—————|————-|———|———-|

Overloaded Functions

| function_name | function_type | description | comment | examples | |—————|—————|————-|———|———-|

Added Types

| type_name | type_size | logical_type | type_category | internal | |———–|———-:|————–|—————|———-|

Added Settings

| name | description | input_type | scope | aliases | |——|————-|————|——-|———|