gorz

883 downloads this week 1 star Extension descriptor

Read and write GORpipe .gorz files and .gord dictionary folders as native DuckDB tables

Compatibility Matrix

This table covers all DuckDB releases that have not reached end-of-life, including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.

Installing and Loading

INSTALL gorz FROM community;
LOAD gorz;

Example

-- Write a query result out as a GORZ file (rows must be in GOR order),
-- then read it back — with an optional GOR -p style range filter.
COPY (SELECT * FROM (VALUES ('chr1', 1000, 'A', 'G'), ('chr1', 2000, 'C', 'T'))
        t(chrom, pos, ref, alt) ORDER BY chrom, pos)
  TO 'variants.gorz' (FORMAT gorz);

SELECT * FROM read_gor('variants.gorz', range := 'chr1:1500-2500');
-- Equivalently, in plain SQL on the bare file name (the chrom / pos
-- predicates are pushed down into a block seek):
SELECT * FROM 'variants.gorz' WHERE chrom = 'chr1' AND pos BETWEEN 1500 AND 2500;

-- Write a .gord dictionary folder: one sorted .gorz per genomic range (here
-- 1 Mb per file) plus thedict.gord. The input needs no ORDER BY.
COPY (SELECT * FROM 'variants.gorz') TO 'variants.gord' (FORMAT gord, SPLIT 1000000);
SELECT * FROM 'variants.gord' WHERE chrom = 'chr1';

-- Read a partitioned .gord dictionary table (not shown here), keeping only
-- the partitions tagged sample1 or sample99 (GOR's -f). See the GORpipe paper
-- (Guðbjartsson et al., Bioinformatics 2016, doi:10.1093/bioinformatics/btw199)
-- and "Ultra-fast joint-genotyping with SparkGOR" (Guðbjartsson et al.,
-- bioRxiv 2022, doi:10.1101/2022.10.25.513331):
SELECT * FROM read_gord('example.gord', f := ['sample1', 'sample99']);

About gorz

The gorz extension reads and writes GORpipe genomic files — block-compressed .gorz and .gord dictionary folders — as native DuckDB tables. Files it writes are read and seeked natively by gorpipe, and vice-versa.

Reading

• read_gor(path) — auto-detects .gorz vs .gord by extension.
• read_gorz(path) / read_gord(path) — force the kind.
• A replacement scan, so a bare literal works: SELECT count(*) FROM 'variants.gord';
• pgor … | write x.gord folders — pass the folder itself (FROM 'x.gord'); its x.gord/thedict.gord is read.
• Projection & parallel scan — only selected columns are read; large files scan across threads.
• WHERE chrom/pos → block seek — range predicates seek into the right block instead of scanning the whole file.
• range := 'chrN:start-end' — GOR's -p semantics as a hard positional filter (also 'chrN', 'chrN:start-', 'chrN:pos').
• Partition filters — GOR's -f / -ff semantics prune a .gord's file list and expose a Source column: f := ['A','B'] is an inline tag list, ff := 'tags.txt' is a tag file (one tag per line / first tab-delimited column, # lines skipped). source := 'PN' renames the exposed column.
• Object stores — paths resolve through DuckDB's FileSystem, so s3://… works for reads and writes when httpfs is loaded.

Writing

COPY (SELECT chrom, pos, ref, alt FROM my_variants ORDER BY chrom, pos)
  TO 'out.gorz' (FORMAT gorz);   -- or (FORMAT gor) for plain sorted TSV

Single-threaded, standard block-zip; validates GOR order (chromosomes ascend lexicographically, positions non-decreasing) and errors out otherwise. META also writes the .gorz.meta sidecar (columns, types, line count, range, MD5), which read_gor uses to hand the column types back.

COPY (SELECT * FROM my_variants) TO 'out.gord' (FORMAT gord, SPLIT 100);

Writes a dictionary folder, as pgor … | write out.gord does: one sorted .gorz + .gorz.meta per genomic range and out.gord/thedict.gord. The query needs no ORDER BY; it runs once and each range is sorted on its own, so memory follows the size of a range rather than the size of the input.

• SPLIT 'chrom' (default) — one file per chromosome.
• SPLIT n, n ≤ 1000 — n genome partitions (a step of 3,000,000,000 / n bases within each chromosome, as pgor -split=n).
• SPLIT n, n > 1000 — n bases per file.
• OVERWRITE — replace an existing .gord folder.
• STAGING 'dir' — local directory for the temporary files (default: next to the output; DuckDB's temp_directory for an object-store target).

Object stores

LOAD httpfs;
CREATE SECRET (TYPE s3, KEY_ID '…', SECRET '…', REGION 'eu-west-1');
COPY (SELECT * FROM my_variants) TO 's3://bucket/out.gord' (FORMAT gord, SPLIT 100);
SELECT count(*) FROM 's3://bucket/out.gord' WHERE chrom = 'chr17';

.gorz files and .gord folders can be read from and written to S3-compatible stores (s3://…). For a folder write the temporary files stay on local disk and only the finished .gorz, .meta and thedict.gord objects are uploaded, the dictionary last.

References

• Guðbjartsson H. et al. GORpipe: a query tool for working with sequence data based on a Genomic Ordered Relational (GOR) architecture. Bioinformatics 32(20):3081–3088, 2016.
• Guðbjartsson H. et al. Ultra-fast joint-genotyping with SparkGOR. bioRxiv, 2022.

Added Functions

Overloaded Functions

This extension does not add any function overloads.

Added Types

This extension does not add any types.

Added Settings