DuckDB ran out of memory mid-query. Almost always because a large aggregation, join, or sort tried to hold more rows in RAM than the process was allowed, and the spill-to-disk buffer either was not set or ran out of temp space.
- ·The query is doing a big JOIN, GROUP BY, ORDER BY, or PIVOT on a large table
- ·memory_limit is set too low (default is 80% of RAM, but explicit settings can be much lower)
- ·temp_directory is unwritable, on a full disk, or points at a small /tmp
- ·You are reading Parquet/CSV without projection pushdown, materializing every column
- ·Running inside a container with a hard memory cgroup lower than what DuckDB is trying to use
What you're seeing
The query terminates with one of the following. The exact wording depends on your client (CLI, Python, Node, JDBC) but the substance is identical.
Error: Out of Memory Error: could not allocate block of size X (Y/Z used)
Error: Out of Memory Error: failed to allocate data of size X
# Python client variant:
duckdb.OutOfMemoryException: Out of Memory Error:
could not allocate block of size 262144 (17.1 GB/17.1 GB used)duckdb out of memory, duckdb allocation failure, duckdb memory issues, OutOfMemoryExceptionWhat's causing this
Ranked most-likely first.
- 1
The query genuinely needs more memory than it is allowed
DuckDB is aggressive about vectorized execution but joins, group-bys, distinct, order-by, and window functions can all buffer intermediate results in RAM. If memory_limit is 4 GB and the hash join build side is 8 GB of rows, you hit this immediately.
- 2
temp_directory is missing or unwritable
DuckDB spills to disk when memory pressure hits, but only if temp_directory is set to a writable location. In in-memory mode (:memory: databases) temp_directory defaults to empty, so spill is disabled entirely and every allocation must fit in RAM.
- 3
The temp partition ran out of space during spill
If temp_directory points at /tmp on a small partition, or at an ephemeral container mount, the spill files can fill it and DuckDB will still surface an out-of-memory-shaped error because the fallback path failed.
- 4
You are inside a container with a memory cgroup
Docker, Kubernetes, and other container runtimes enforce hard memory ceilings that DuckDB may not detect correctly. DuckDB's default memory_limit is 80% of what it thinks the host has, which can exceed the cgroup limit and get the process OOM-killed by the runtime before you see the DuckDB error.
- 5
The workload materializes far more data than it needs to
Reading a wide Parquet with SELECT * and only using two columns forces DuckDB to decode everything. Same for CSV without projection. The fix is a smarter query, not more RAM.
How to fix it
Step 1: check the current memory and temp settings
Confirm what DuckDB actually thinks the limits are. The defaults you expect are often not what a specific client, config, or session ends up with.
SELECT current_setting('memory_limit') AS memory_limit,
current_setting('temp_directory') AS temp_directory,
current_setting('threads') AS threads;Step 2: give DuckDB a writable temp_directory so it can spill
This alone resolves most out-of-memory errors on in-memory databases. Point temp_directory at a path with plenty of free space, ideally on a fast local SSD.
SET temp_directory = '/var/tmp/duckdb-spill';
-- Or for a local dev machine:
SET temp_directory = '/tmp/duckdb-spill';Step 3: raise memory_limit if the host actually has more RAM
The default is 80% of detected RAM, but explicit settings can be much lower. Raise it if there is headroom. Do not exceed the container cgroup limit if you are running in Docker or Kubernetes.
-- Set an absolute value:
SET memory_limit = '12GB';
-- Or match a container-safe fraction:
SET memory_limit = '80%';Step 4: reduce threads if the query is aggregating heavily
Each parallel thread holds its own partial results. Cutting thread count can cut peak memory in proportion, at the cost of longer wall-clock time.
SET threads = 4;
-- Or match the physical core count if hyperthreading is inflating it:
SET threads = 8;Step 5: rewrite the query to materialize less
The cheapest fix is often to project fewer columns, add a WHERE filter earlier, or replace a huge DISTINCT with an approximate one. DuckDB has APPROX_COUNT_DISTINCT and APPROX_QUANTILE for exactly this reason.
-- Instead of:
SELECT COUNT(DISTINCT user_id) FROM events;
-- Consider:
SELECT APPROX_COUNT_DISTINCT(user_id) FROM events;
-- Instead of SELECT * on a wide Parquet, project explicitly:
SELECT user_id, event_ts FROM 'events.parquet';Step 6: if in-memory database, switch to a file database for spill support
In-memory databases (opened with :memory: or no path) disable spilling by default. Opening a file-based database with a real temp_directory lets DuckDB stream through data larger than RAM.
# Python client. Instead of:
con = duckdb.connect(':memory:')
# Use a file:
con = duckdb.connect('scratch.duckdb')
con.execute("SET temp_directory = '/var/tmp/duckdb-spill'")Prevention
Set memory_limit and temp_directory in a startup script or connection init so every session inherits sane defaults. Especially for shared analytics environments where users will not think to check.
If you are running in Docker or Kubernetes, set DuckDB's memory_limit to something safely below the container's memory request so the runtime never has to OOM-kill the process. A 12 GB container should not be running DuckDB with a 12 GB memory_limit.
For workloads that regularly touch datasets larger than RAM, prefer file-based databases over :memory: connections. The overhead is small and spill support is worth it.
Debug DuckDB faster
1bench is a native GUI for DuckDB. Inspect queries, connections, and settings without leaving the app. See what's happening before you have to Google it.
Open DuckDB in 1bench