---
title: "A Hard Memory Ceiling for PHP Scripts That Run for Hours"
url: https://www.exakat.io/a-hard-memory-ceiling-for-php-scripts-that-run-for-hours/
date: 2026-10-09
modified: 2026-10-09
lang: en
author: "dams"
description: "A Hard Memory Ceiling for PHP Scripts That Run for Hours You inherit a legacy table full of auto-incrementing integers and a new architecture that wants UUIDs. You write a..."
categories:
  - "Technology"
tags:
  - "php"
  - "sqlite"
  - "weakmap"
image: https://www.exakat.io/wp-content/uploads/2026/10/cap.640.jpg
word_count: 1496
---

# A Hard Memory Ceiling for PHP Scripts That Run for Hours

# A Hard Memory Ceiling for PHP Scripts That Run for Hours

You inherit a legacy table full of auto-incrementing integers and a new architecture that wants UUIDs. You write a CLI worker, build an in-memory lookup array mapping old IDs to new ones, and let it run. Forty thousand rows in, the terminal stops mid-scroll and hands you the line every PHP developer eventually meet one day: *Allowed memory size exhausted*.

The instinctive fix is to raise `memory_limit`. The correct fix is to notice that you built an unbounded array and called it a lookup table. And once you've fixed that, a second problem is waiting right behind it: this migration runs for hours, across a table nobody trusts, on infrastructure nobody promised wouldn't reboot. If the fix for memory doesn't also make the script resumable, you haven't solved anything: you've just delayed the crash until row two million, at which point restarting from zero stops being an inconvenience and starts being your weekend.

One structure fixes both problems, because they're the same problem wearing two hats: an LRU cache backed by SQLite, doing double duty as a checkpoint log. Let's see how one can do that.

## Why WeakMap doesn't apply here

PHP 8.0 gave us [`WeakMap`](https://www.php.net/manual/en/class.weakmap.php), and it's tempting to reach for it the moment "memory" and "cache" appear in the same sentence: the garbage collector drops an entry the moment nothing else references its key, no manual eviction required. It's a real improvement over `SplObjectStorage`, which has been quietly holding strong references to its keys since PHP 5.3 and leaking memory in exactly the scenario you'd use it to avoid.

But `WeakMap` keys must be objects. In our case, the lookup table is keyed on database IDs, that is strings or integers, and `WeakMap` won't accept them at all. It responds with `WeakMap key must be an object`, which is basically the exact contrary of the reaction of an array. Even granting yourself the trouble of wrapping every ID in a throwaway object, you'd still be trusting PHP's generational garbage collector, incrementally improved since 7.4, but never deterministic, to decide *when* to reclaim memory. You want to keep control. You want a hard cap of exactly `$maxItems` entries, enforced on every write, not a random moment, sometime after PHP notices the cycle. `WeakMap` solves a reference-tracking problem. You have a capacity problem. Different tool.

## Hot data in RAM, everything else on disk

The actual shape you want is a two-tier cache: a small PHP array holds the most recently touched entries, and SQLite holds all of them, unbounded, on disk. Reads check the array first; on a miss, they pull from SQLite and promote the entry into the array. Writes go to both, immediately.

PHP arrays are ordered by insertion, which makes LRU bookkeeping nearly free: `unset()` a key and reassign it, and it moves to the end. `array_key_first()` hands you the eviction candidate without a separate tracking structure. The only work SQLite does is being reliable, fast enough, and, crucially, still there after the process dies.

It is worth pointing at the `remember()` extraction, because its absence is exactly where a first draft of this class would broke. `set()` on an existing key looks harmless, `$this->memoryCache[$key] = $value`, but PHP does not reorder an array on reassignment to an existing key. It only moves to the end on `unset()` + insert. If you sSkip that step, you have a structure calling itself an LRU cache that never promotes a re-set key. Eventually, it evicts values based on first-write order instead of last-touch order. Cheap bug, and one you'd only notice under load, exactly when you can least afford to.

## The checkpoint you already own

`set()` writes to SQLite before it writes to the array. That means the database on disk is never more than one call behind the true migration state, which is a checkpoint mechanism, whether you asked for one or not. Use it deliberately: keep a `progress` table, read `last_id` on startup, resume from there.

That `INSERT OR IGNORE` seeding line is doing more work than its size suggests. An empty `progress` table with only an `UPDATE` statement pointed at it is a checkpoint that silently never checkpoints. `UPDATE` against zero matching rows succeeds, reports nothing wrong, and the resumable script will restart from ID 0. It's the kind of bug that passes every manual test, because you only ever run it interrupted-and-resumed by accident once, in production, at 2am. I would recommend grepping your own scripts for that exact pattern: any `UPDATE ... WHERE` used as a substitute for an upsert, on a table, should be reviewed. Yes, this is me giving you homework.

The `do...while` around the fetch matters too. A single `LIMIT 1000` query with no outer loop processes one batch and exits, which quietly turns a resumable migration into a migration you have to manually re-invoke a few thousand times. This is all fine if that's what your process supervisor is for; but it is worth being honest with yourself about which one you want to built.

| What broke | Why it passed a glance | The fix |
| ---------- | ---------------------- | ------- |
| `set()` doesn't reorder existing keys | Looks identical to a fresh insert | Route both paths through one `remember()` that always `unset()`s first |
| `progress` table starts empty | `UPDATE` on zero rows raises no error | Seed the row with `INSERT OR IGNORE`before ever updating it |
| Single `LIMIT 1000`fetch, no loop | Runs fine, just stops after 1000 rows | Wrap the fetch/checkpoint pair in a `do...while` on batch size |

## The trade off of this approach

Reading through SQLite is slower than reading through a PHP array, for sure. But, what it buys back isn't just a memory ceiling; it's the automated checkpoint, because durability and eviction turned out to be the same write. The cost worth naming explicitly: every `set()` here is its own implicit transaction, so PDO commits to disk on every single row. For a migration touching millions of rows, that's millions of fsyncs. Wrapping each 1000-row batch in `$db->beginTransaction()` / `$db->commit()` buys real throughput. This is the price to pay for the risk of losing up to one batch's progress on a hard crash mid-transaction, instead of losing at most one row. Decide which of those numbers you can live with before you're the one explaining the choice in an incident channel.

## The bigger picture

Once a structure is durable, resumable, and survives its own process dying mid-write, it is tempting to call it a cache. In fact, a cache is allowed to lose data. In this case, it is deliberately built so it doesn't and can't. Every property that makes it safe for a nine-hour migration is a property a database has, not a property a cache has. The eviction policy is the only thing left distinguishing `data_store` from a table you'd just call your database.

In the end, we have now set up a robust system can both limit the amount of memory it uses to process even more data, and can even restart. This is a feature that rest on the shoulders of the data organisation, rather than PHP. Yet, with generators, and tools that walk datasets rather than process then in one time, it is not difficult to prepare PHP to be reborn after dying. Again, and again, and again...