Chunking Async Iterables

Split arrays or async iterables into fixed-size batches. Provides a synchronous version for arrays (chunks) and an async version for iterables (chunksAsync) using Repeater.

When to Use

• Processing database rows in batches
• Sending bulk API requests with size limits
• Splitting large datasets into manageable pages
• Batching items from an async stream for bulk operations

The Pattern

Batch array processing

const items = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
const batches = chunks(3, items);
// [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]

Stream database rows in batches

const rowStream = streamRows(cursor);
const batchedRows = chunksAsync(100, rowStream);

for await (const batch of batchedRows) {
  await bulkInsert(batch);
}

Batch API requests

const ids = getIdsToFetch();
for await (const batch of chunksAsync(50, ids)) {
  const results = await api.fetchMany(batch);
  process(results);
}

Synchronous chunks for arrays

Slice an array into fixed-size sub-arrays:

export function chunks(size, array) {
  if (!Array.isArray(array)) {
    throw new TypeError("array arg must be an array");
  }
  if (!Number.isInteger(size) || size <= 0) {
    throw new TypeError("size must be a positive integer");
  }
  
  let cursor = 0;
  const result = [];
  while (true) {
    const chunk = array.slice(cursor, cursor += size);
    if (chunk.length > 0) {
      result.push(chunk);
    } else {
      break;
    }
  }
  return result;
}

Async chunks for iterables

Wrap an async iterable in a Repeater that accumulates items and pushes each full batch:

import { Repeater } from "@repeaterjs/repeater";

export function chunksAsync(count, iterator) {
  if (typeof count !== "number" || count <= 0) {
    throw new TypeError("count must be a positive number");
  }
  
  return new Repeater(async (push, stop) => {
    let chunk = [];
    
    for await (const item of iterator) {
      chunk.push(item);
      
      if (chunk.length === count) {
        await push(chunk);
        chunk = [];
      }
    }
    
    if (chunk.length > 0) {
      await push(chunk);
    }
    
    stop();
  });
}

Key details:

• await push(chunk) applies backpressure — the producer waits until the consumer processes each batch before accumulating the next one.
• Remainder handling — a final partial batch is pushed if the source ends mid-chunk.
• stop() — signals the consumer that no more batches will arrive.

Trade-offs

Approach	Pros	Cons
chunks() sync	Simple, no deps	Arrays only
chunksAsync()	Any iterable, backpressure	Needs Repeater
Manual accumulation	No abstraction	Boilerplate, easy to miss remainder

Batch size selection: Larger batches reduce overhead (fewer I/O round-trips) but increase memory usage and latency before the first batch is ready. Choose based on downstream constraints (API limits, memory, transaction sizes).