Search Shortcut cmd + k | ctrl + k
Search cmd+k ctrl+k
0.10 (stable)
Data Chunks

Data chunks represent a horizontal slice of a table. They hold a number of vectors, that can each hold up to the VECTOR_SIZE rows. The vector size can be obtained through the duckdb_vector_size function and is configurable, but is usually set to 2048.

Data chunks and vectors are what DuckDB uses natively to store and represent data. For this reason, the data chunk interface is the most efficient way of interfacing with DuckDB. Be aware, however, that correctly interfacing with DuckDB using the data chunk API does require knowledge of DuckDB’s internal vector format.

The primary manner of interfacing with data chunks is by obtaining the internal vectors of the data chunk using the duckdb_data_chunk_get_vector method, and subsequently using the duckdb_vector_get_data and duckdb_vector_get_validity methods to read the internal data and the validity mask of the vector. For composite types (list and struct vectors), duckdb_list_vector_get_child and duckdb_struct_vector_get_child should be used to read child vectors.

API Reference

duckdb_data_chunk duckdb_create_data_chunk(duckdb_logical_type *types, idx_t column_count);
void duckdb_destroy_data_chunk(duckdb_data_chunk *chunk);
void duckdb_data_chunk_reset(duckdb_data_chunk chunk);
idx_t duckdb_data_chunk_get_column_count(duckdb_data_chunk chunk);
duckdb_vector duckdb_data_chunk_get_vector(duckdb_data_chunk chunk, idx_t col_idx);
idx_t duckdb_data_chunk_get_size(duckdb_data_chunk chunk);
void duckdb_data_chunk_set_size(duckdb_data_chunk chunk, idx_t size);

Vector Interface

duckdb_logical_type duckdb_vector_get_column_type(duckdb_vector vector);
void *duckdb_vector_get_data(duckdb_vector vector);
uint64_t *duckdb_vector_get_validity(duckdb_vector vector);
void duckdb_vector_ensure_validity_writable(duckdb_vector vector);
void duckdb_vector_assign_string_element(duckdb_vector vector, idx_t index, const char *str);
void duckdb_vector_assign_string_element_len(duckdb_vector vector, idx_t index, const char *str, idx_t str_len);
duckdb_vector duckdb_list_vector_get_child(duckdb_vector vector);
idx_t duckdb_list_vector_get_size(duckdb_vector vector);
duckdb_state duckdb_list_vector_set_size(duckdb_vector vector, idx_t size);
duckdb_state duckdb_list_vector_reserve(duckdb_vector vector, idx_t required_capacity);
duckdb_vector duckdb_struct_vector_get_child(duckdb_vector vector, idx_t index);
duckdb_vector duckdb_array_vector_get_child(duckdb_vector vector);

Validity Mask Functions

bool duckdb_validity_row_is_valid(uint64_t *validity, idx_t row);
void duckdb_validity_set_row_validity(uint64_t *validity, idx_t row, bool valid);
void duckdb_validity_set_row_invalid(uint64_t *validity, idx_t row);
void duckdb_validity_set_row_valid(uint64_t *validity, idx_t row);

duckdb_create_data_chunk


Creates an empty DataChunk with the specified set of types.

Note that the result must be destroyed with duckdb_destroy_data_chunk.

Syntax


duckdb_data_chunk duckdb_create_data_chunk(
  duckdb_logical_type *types,
  idx_t column_count
);

Parameters


  • types

An array of types of the data chunk.

  • column_count

The number of columns.

  • returns

The data chunk.


duckdb_destroy_data_chunk


Destroys the data chunk and de-allocates all memory allocated for that chunk.

Syntax


void duckdb_destroy_data_chunk(
  duckdb_data_chunk *chunk
);

Parameters


  • chunk

The data chunk to destroy.


duckdb_data_chunk_reset


Resets a data chunk, clearing the validity masks and setting the cardinality of the data chunk to 0.

Syntax


void duckdb_data_chunk_reset(
  duckdb_data_chunk chunk
);

Parameters


  • chunk

The data chunk to reset.


duckdb_data_chunk_get_column_count


Retrieves the number of columns in a data chunk.

Syntax


idx_t duckdb_data_chunk_get_column_count(
  duckdb_data_chunk chunk
);

Parameters


  • chunk

The data chunk to get the data from

  • returns

The number of columns in the data chunk


duckdb_data_chunk_get_vector


Retrieves the vector at the specified column index in the data chunk.

The pointer to the vector is valid for as long as the chunk is alive. It does NOT need to be destroyed.

Syntax


duckdb_vector duckdb_data_chunk_get_vector(
  duckdb_data_chunk chunk,
  idx_t col_idx
);

Parameters


  • chunk

The data chunk to get the data from

  • returns

The vector


duckdb_data_chunk_get_size


Retrieves the current number of tuples in a data chunk.

Syntax


idx_t duckdb_data_chunk_get_size(
  duckdb_data_chunk chunk
);

Parameters


  • chunk

The data chunk to get the data from

  • returns

The number of tuples in the data chunk


duckdb_data_chunk_set_size


Sets the current number of tuples in a data chunk.

Syntax


void duckdb_data_chunk_set_size(
  duckdb_data_chunk chunk,
  idx_t size
);

Parameters


  • chunk

The data chunk to set the size in

  • size

The number of tuples in the data chunk


duckdb_vector_get_column_type


Retrieves the column type of the specified vector.

The result must be destroyed with duckdb_destroy_logical_type.

Syntax


duckdb_logical_type duckdb_vector_get_column_type(
  duckdb_vector vector
);

Parameters


  • vector

The vector get the data from

  • returns

The type of the vector


duckdb_vector_get_data


Retrieves the data pointer of the vector.

The data pointer can be used to read or write values from the vector. How to read or write values depends on the type of the vector.

Syntax


void *duckdb_vector_get_data(
  duckdb_vector vector
);

Parameters


  • vector

The vector to get the data from

  • returns

The data pointer


duckdb_vector_get_validity


Retrieves the validity mask pointer of the specified vector.

If all values are valid, this function MIGHT return NULL!

The validity mask is a bitset that signifies null-ness within the data chunk. It is a series of uint64_t values, where each uint64_t value contains validity for 64 tuples. The bit is set to 1 if the value is valid (i.e., not NULL) or 0 if the value is invalid (i.e., NULL).

Validity of a specific value can be obtained like this:

idx_t entry_idx = row_idx / 64; idx_t idx_in_entry = row_idx % 64; bool is_valid = validity_mask[entry_idx] & (1 « idx_in_entry);

Alternatively, the (slower) duckdb_validity_row_is_valid function can be used.

Syntax


uint64_t *duckdb_vector_get_validity(
  duckdb_vector vector
);

Parameters


  • vector

The vector to get the data from

  • returns

The pointer to the validity mask, or NULL if no validity mask is present


duckdb_vector_ensure_validity_writable


Ensures the validity mask is writable by allocating it.

After this function is called, duckdb_vector_get_validity will ALWAYS return non-NULL. This allows null values to be written to the vector, regardless of whether a validity mask was present before.

Syntax


void duckdb_vector_ensure_validity_writable(
  duckdb_vector vector
);

Parameters


  • vector

The vector to alter


duckdb_vector_assign_string_element


Assigns a string element in the vector at the specified location.

Syntax


void duckdb_vector_assign_string_element(
  duckdb_vector vector,
  idx_t index,
  const char *str
);

Parameters


  • vector

The vector to alter

  • index

The row position in the vector to assign the string to

  • str

The null-terminated string


duckdb_vector_assign_string_element_len


Assigns a string element in the vector at the specified location. You may also use this function to assign BLOBs.

Syntax


void duckdb_vector_assign_string_element_len(
  duckdb_vector vector,
  idx_t index,
  const char *str,
  idx_t str_len
);

Parameters


  • vector

The vector to alter

  • index

The row position in the vector to assign the string to

  • str

The string

  • str_len

The length of the string (in bytes)


duckdb_list_vector_get_child


Retrieves the child vector of a list vector.

The resulting vector is valid as long as the parent vector is valid.

Syntax


duckdb_vector duckdb_list_vector_get_child(
  duckdb_vector vector
);

Parameters


  • vector

The vector

  • returns

The child vector


duckdb_list_vector_get_size


Returns the size of the child vector of the list.

Syntax


idx_t duckdb_list_vector_get_size(
  duckdb_vector vector
);

Parameters


  • vector

The vector

  • returns

The size of the child list


duckdb_list_vector_set_size


Sets the total size of the underlying child-vector of a list vector.

Syntax


duckdb_state duckdb_list_vector_set_size(
  duckdb_vector vector,
  idx_t size
);

Parameters


  • vector

The list vector.

  • size

The size of the child list.

  • returns

The duckdb state. Returns DuckDBError if the vector is nullptr.


duckdb_list_vector_reserve


Sets the total capacity of the underlying child-vector of a list.

Syntax


duckdb_state duckdb_list_vector_reserve(
  duckdb_vector vector,
  idx_t required_capacity
);

Parameters


  • vector

The list vector.

  • required_capacity

the total capacity to reserve.

  • return

The duckdb state. Returns DuckDBError if the vector is nullptr.


duckdb_struct_vector_get_child


Retrieves the child vector of a struct vector.

The resulting vector is valid as long as the parent vector is valid.

Syntax


duckdb_vector duckdb_struct_vector_get_child(
  duckdb_vector vector,
  idx_t index
);

Parameters


  • vector

The vector

  • index

The child index

  • returns

The child vector


duckdb_array_vector_get_child


Retrieves the child vector of a array vector.

The resulting vector is valid as long as the parent vector is valid. The resulting vector has the size of the parent vector multiplied by the array size.

Syntax


duckdb_vector duckdb_array_vector_get_child(
  duckdb_vector vector
);

Parameters


  • vector

The vector

  • returns

The child vector


duckdb_validity_row_is_valid


Returns whether or not a row is valid (i.e., not NULL) in the given validity mask.

Syntax


bool duckdb_validity_row_is_valid(
  uint64_t *validity,
  idx_t row
);

Parameters


  • validity

The validity mask, as obtained through duckdb_vector_get_validity

  • row

The row index

  • returns

true if the row is valid, false otherwise


duckdb_validity_set_row_validity


In a validity mask, sets a specific row to either valid or invalid.

Note that duckdb_vector_ensure_validity_writable should be called before calling duckdb_vector_get_validity, to ensure that there is a validity mask to write to.

Syntax


void duckdb_validity_set_row_validity(
  uint64_t *validity,
  idx_t row,
  bool valid
);

Parameters


  • validity

The validity mask, as obtained through duckdb_vector_get_validity.

  • row

The row index

  • valid

Whether or not to set the row to valid, or invalid


duckdb_validity_set_row_invalid


In a validity mask, sets a specific row to invalid.

Equivalent to duckdb_validity_set_row_validity with valid set to false.

Syntax


void duckdb_validity_set_row_invalid(
  uint64_t *validity,
  idx_t row
);

Parameters


  • validity

The validity mask

  • row

The row index


duckdb_validity_set_row_valid


In a validity mask, sets a specific row to valid.

Equivalent to duckdb_validity_set_row_validity with valid set to true.

Syntax


void duckdb_validity_set_row_valid(
  uint64_t *validity,
  idx_t row
);

Parameters


  • validity

The validity mask

  • row

The row index


About this page

Last modified: 2024-03-29