Code auditing

When Generators Actually Makes Sense

Posted on by dams

When Generators Actually Makes SenseWhen Generators Actually Makes Sense

Generators are one of PHP 8 best-loved features. Sprinkle a yield into a function and suddenly you’re processing millions of rows with constant memory. At least, that’s the promise. The reality is more nuanced: lazy evaluation only pays off when the underlying data source can produce records incrementally. If the format, the driver, or the algorithm forces the whole dataset into memory anyway, wrapping it in a generator changes nothing except the shape of your code.

This post draws the line between the two situations, with concrete examples of each.

The principle in one sentence

Lazy evaluation helps when data can be created or read at the moment it is yielded; it is useless when the whole dataset must be materialized in memory before the first item can be produced, or when peacemeal generation is actually slower than doing it once.

In the first case, memory stays flat, one record at a time, at the cost of unit reads: many small I/O operations instead of one big one. In the second case, the memory peak has already happened before your generator yields anything. The yield is cosmetic.

A quick corollary: laziness also requires that the consumer doesn’t need the whole set at once. Even a perfectly streamable source is defeated by a sort() at the end.

Part 1 — Where lazy evaluation genuinely helps

1. Computed sequences

The textbook case. Nothing exists until you ask for it:

<?php

function integers(int $from = 1, ?int $to = null): Generator
{
    for ($i = $from; $to === null || $i <= $to; $i++) {
        yield $i;
    }
}

foreach (integers(1, 10_000_000) as $n) {
    // one integer in memory at a time
}
?>

range(1, 10_000_000) allocates the full array up front. The generator version runs in constant memory and even supports infinite sequences, with $to = null, which an array can’t represent at all.

Even with this textbook example, there is a size under which the size of the loop will not improve. Here, we had to make it significantly large to make it visible: who handles 10 millions things at once in PHP?

2. Reading lines from a file

Files are byte streams; the OS happily hands them to you piece by piece. file() slurps everything; fgets() doesn’t:

<?php
function lines(string $path): Generator
{
    $h = fopen($path, 'rb');
    try {
        while (($line = fgets($h)) !== false) {
            yield rtrim($line, "\r\n");
        }
    } finally {
        fclose($h);
    }
}
?>

A 2 GB log file is processed in kilobytes of memory. The same pattern works for CSV with fgetcsv(): the format is line/record oriented, so each record is self-contained and can be parsed independently of the rest.

3. Reading packets from a socket

Network data is the ultimate streaming source: the rest of the data may not even exist yet when you process the first packet. It is better to process it, and, incidentally, wait for the next packet during the same time. One stone, two birds.

<?php
function packets($socket, int $size = 8192): Generator
{
    while (!feof($socket)) {
        $chunk = fread($socket, $size);
        if ($chunk === false || $chunk === '') {
            break;
        }
        yield $chunk;
    }
}
?>

Here laziness isn’t just an optimization; it’s the only correct model. Buffering the whole response from a long-lived connection is impossible by definition.

4. Line-delimited formats: JSONL / NDJSON

JSON Lines is JSON redesigned for streaming: one complete document per line. Each line decodes independently, so a generator composes beautifully:

<?php
function jsonl(string $path): Generator
{
    foreach (lines($path) as $line) {
        if ($line !== '') {
            yield json_decode($line, true, flags: JSON_THROW_ON_ERROR);
        }
    }
}
?>

Only one decoded object lives in memory at a time. This is why data-export APIs, and tools like bulk endpoints prefer NDJSON over a single giant JSON array.

5. Unbuffered database queries

Databases can stream: you just have to ask nicely. With pdo_mysql, buffering is on by default; turn it off and rows travel from the server as you fetch them:

<?php
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);

function rows(PDO $pdo, string $sql): Generator
{
    $stmt = $pdo->query($sql);
    while ($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
        yield $row;
    }
}
?>

With mysqli, the equivalent is MYSQLI_USE_RESULT. PostgreSQL offers cursors, with keywords like DECLARE ... CURSOR / FETCH, for the same effect. Now the generator is honest: memory stays flat even for millions of rows. The trade-offs are real, though: the connection is pinned until the result set is fully consumed, you can’t run another query on it meanwhile, and rowCount() is unavailable because nobody knows the total yet.

6. Paginated APIs

A generator hides pagination from the consumer while fetching pages only on demand:

<?php
function apiItems(HttpClient $http, string $url): Generator
{
    while ($url !== null) {
        $page = $http->getJson($url);
        yield from $page['items'];
        $url = $page['next'] ?? null;
    }
}
?>

If the consumer stops after 50 items, only one page was ever downloaded. Laziness saves not just memory here, but bandwidth and API quota.

7. Pipelines of transformations

Because generators compose, you can chain filter/map steps that each hold one item:

<?php
function parsed(iterable $lines): Generator
{
    foreach ($lines as $line) {
        yield str_getcsv($line);
    }
}

function onlyErrors(iterable $rows): Generator
{
    foreach ($rows as $row) {
        if ($row[2] === 'ERROR') {
            yield $row;
        }
    }
}

foreach (onlyErrors(parsed(lines('app.log'))) as $error) {
    // whole pipeline is lazy, end to end
}
?>

The pipeline is only as lazy as its laziest stage.

8. Streaming output

Laziness works in both directions. Generating a large CSV download by writing straight to php://output, row by row, from an unbuffered query, means the response starts immediately and the server never holds the full file. Compression via stream filters, such as compress.zlib://, streams too.

Part 2 — Where lazy evaluation doesn’t help much

1. json_decode() on a regular JSON document

A JSON array is one syntactic unit: it isn’t valid until the closing ], and json_decode() parses the entire document into PHP structures in one shot. This generator is a lie:

<?php
function items(string $path): Generator
{
    $data = json_decode(file_get_contents($path), true); // 💥 peak memory happens HERE
    foreach ($data as $item) {
        yield $item; // too late to save anything
    }
}
?>

By the time the first yield runs, the file and its full decoded representation, typically several times larger than the raw JSON, have both been in memory. The yield only changes how the caller iterates, not what it costs.

Here is a workarounds: switch the format to JSONL if you control it, or use an incremental parser such as halaxa/json-machine or cerbero90/json-parser, which tokenize the stream and yield items one at a time. That’s laziness pushed down into the parser, where it belongs.

2. Buffered database reads

This is the most common false economy. With pdo_mysql’s default buffered mode, the entire result set is transferred and stored in PHP’s memory as soon as the query executes, before the first fetch(). Fetching piecemeal with fetch()/fetchAssoc() in a loop just walks a buffer that’s already fully resident:

<?php
// Looks lazy. Isn't.
$stmt = $pdo->query('SELECT * FROM events'); // full result set buffered here
while ($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
    yield $row; // the memory bill was paid two lines up
}
?>

With the mysqlnd driver, that buffer even counts against memory_limit, which is why large exports die with “Allowed memory size exhausted” despite the innocent-looking row-by-row loop. The fix isn’t a generator; it’s unbuffered mode or chunked queries (LIMIT/keyset pagination). The same trap exists one level up: many ORMs hydrate results eagerly, so foreach ($repository->findAll() as $e) buffers everything regardless of how you iterate afterwards.

3. Whole-document formats: XML DOM, unserialize, spreadsheets

The JSON problem generalizes to any format that must be parsed as a unit:

  • DOMDocument and SimpleXML build the entire tree in memory. Lazily iterating $dom->getElementsByTagName('row')saves nothing. The streaming alternative is XMLReader, a pull parser that pairs naturally with generators.
  • unserialize() produces the whole object graph at once: there is no streaming variant.
  • XLSX: PhpSpreadsheet loads the workbook into memory; a generator over its rows is decorative. Streaming readers/writers, like OpenSpout exist precisely because the default can’t be lazy. XLSX is also a ZIP container, compressed, non-line-oriented, another hint that “read it line by line” doesn’t apply.

The pattern: when the parser’s API says “give me the whole input, receive the whole structure,” laziness has to be implemented inside a different parser, not wrapped around the eager one.

4. Algorithms that need the whole dataset

Even with a perfectly streamable source, some operations are inherently eager:

  • Sorting: you can’t emit the smallest element until you’ve seen them all. sort(), usort(), SplPriorityQueue drained at the end: all require full materialization (or external merge sort on disk, which is a different technique, not a generator).
  • Deduplication, grouping, joins across the full set, array_unique, building an index keyed by ID.
  • “How many are there?” before processing: counting a generator consumes it.
  • Anything ending in iterator_to_array(), which converts your careful laziness back into an array in one call.

Aggregations are the interesting middle ground: sum, min, max, averages, and top-N with a bounded heap only need constant state, so they stream fine. The question to ask: does the operation need to remember everything it has seen, or just a running summary?

5. Multiple passes and random access

A generator is forward-only and single-use. Iterate it twice and you get an exception like Cannot traverse an already closed generator; ask for $items[42] and there’s no such API. If your algorithm needs rewinding, random access, or two passes, e.g., first compute a total, then compute percentages, you either re-create the source, paying the I/O twice, or materialize once and accept the memory cost. Laziness is not free flexibility; it trades capabilities for memory.

6. Small datasets

Under a few thousand items, an array is simpler, rewindable, count()-able, debuggable in var_dump(), and usable with the whole array_* toolbox. Generators add call/resume overhead per item and remove capabilities. Reserve them for cases where the memory or latency win is real.

The final checklist

Before reaching for yield, ask yourself three questions:

  1. Can the source produce records incrementally? Lines, CSV rows, socket packets, NDJSON, unbuffered/cursor query results, paginated APIs: yes. A monolithic JSON document, a DOM tree, a serialized blob, a buffered result set: no.
  2. Is every stage of the pipeline lazy? One json_decode(file_get_contents(...)), one buffered query, one iterator_to_array() anywhere in the chain, and the flat-memory guarantee is gone. The pipeline is as eager as its most eager stage.
  3. Can the consumer work with one record at a time? Filtering, mapping, writing out, constant-state aggregation: yes. Sorting, global dedup, multiple passes, random access: no.

Three criteria and generators will genuinely keep your memory flat, whatever the dataset size, at the price of many small reads, single-pass semantics, and a long lasting connection. Any no, and the honest fix is elsewhere: change the format, change the parser, change the driver mode, or accept that this particular job simply needs the memory.

Also, consider the middle ground between doing everything by the unit, or all at once: it is doing it by batches. Group data by batches of 5, 10 or a hundred. That may save some time by reading or writing a significant amount of data, and processing them in a batch. It is also known as the problem of the elephpant and the bananas.

yield is a delivery mechanism, not a memory-saving spell. It saves memory only when there was something incremental to deliver in the first place.