⌘K ctrl+k
Search Shortcut cmd + k | ctrl + k
Spark API

The DuckDB Spark API implements the PySpark API, allowing you to use the familiar Spark API to interact with DuckDB. All statements are translated to DuckDB's internal plans using our relational API and executed using DuckDB's query engine. The functions it covers are tested against PySpark 4.2.0 and 3.5.9.

Warning The DuckDB Spark API is experimental and receives bug fixes only: missing features will not be added, and a later release will deprecate it. We recommend SQLFrame, which runs PySpark DataFrame code on DuckDB, as its replacement. SQLFrame does not support DuckDB 2.0 yet: SQLFrame 4.5.0 requires duckdb<1.6 for its DuckDB engine. Please report bugs in the Spark API on GitHub.

Example 

from duckdb.experimental.spark.sql import SparkSession as session
from duckdb.experimental.spark.sql.functions import lit, col

spark = session.builder.getOrCreate()

df = spark.createDataFrame(
    [(34, "Joan"), (45, "Peter"), (23, "John"), (56, "Bob")],
    ["age", "name"],
)
df = df.withColumn("location", lit("Seattle"))
res = df.select(col("age"), col("location")).collect()

print(res)

[Row(age=34, location='Seattle'), Row(age=45, location='Seattle'), Row(age=23, location='Seattle'), Row(age=56, location='Seattle')]

createDataFrame also accepts a pandas DataFrame, which needs pandas to be installed.

Contribution Guidelines

The experimental Spark API accepts bug fixes only. When contributing a fix, please test it with our pytest testing framework instead of using temporary files.

© 2026 DuckDB Foundation, Amsterdam NL
DuckDB Home Code of Conduct Trademark Use Blog