Two Large PHP Datasets, and How to Read Them With Itself

PHP has a large code base online. At Exakat, we run the engine on a corpus of 3000+ open source projects. If you want to train, fine-tune, or evaluate an AI model on PHP code, or just mine a few hundred thousand real-world snippets for research, there are two open datasets on Hugging Face worth knowing about. They are big, they are plain text, and, refreshingly, you just need PHP to access them.

This post covers what’s in each one, how to download them without melting your disk, and how to read them efficiently from PHP itself. From there, do whatever you want with them.

Searching for PHP source code datasets

When looking for dataset, I mostly check HuggingFace.

Other options, such as kaggle.com, github.com or Open Data on AWS did not yield any interesting results: I still mention them as this might change, or you’re better at finding them than me: in every case, please ping me about it!

So far, HuggingFace has landed some interesting finds. I’ll share two interesting datasets just below. There are also datasets, such as The Stack which have up to 400, 600 or 900 programming languages, as one big depot. For PHP, this means downloading them and filtering them before using them. I’ve set them aside for another time.

The two datasets

PHP-Code-Large

PHP-Code-Large is the simple one: a pile of PHP scraped from open source projects, two fields per row.

{"code": "<?php function baseurl() { ... }", "language": "PHP"}

That’s it. Individual code values run from a single character up to ~2 MB, and the mix is exactly what you’d expect from the open-source PHP world: WordPress-style hook functions, Laravel controllers, CodeIgniter-era mysql_query() calls, Smarty templates, half-PHP/half-HTML view files, and a fair amount of code with comments in languages other than English. The dataset card advertises 12M+ lines of PHP; the Hub’s viewer estimates about 8.07M rows. Rows are files (or file fragments), not lines, so treat both numbers as rough scale indicators rather than detailled counts.

cornstack-php-v1

cornstack-php-v1 is a different animal. It’s the PHP slice of CoRNStack (paper, ICLR 2025), built by Nomic to train code-retrieval models like nomic-embed-code and CodeRankEmbed. Each row pairs a natural-language query with the PHP that answers it, plus a set of plausible-but-wrong alternatives:

{
  "query": "return boolean as string 'true' / 'false'",
  "document": "function bool2str($bool) { ... }",
  "metadata": {"objective": {"self": [], "paired": [], "triplet": [["query","document","negatives"]]}},
  "negatives": ["function bool_s($boolean) {...}", "public static function strbool($bool){...}", "..."],
  "negative_scores": ["0.80348504", "0.80206877", "..."],
  "document_score": "0.8142804",
  "document_rank": "0"
}

Two things are worth internalising before you build anything on it.

First, the queries were derived from function docstrings, not from humans asking questions. They read like documentation comments, because they are documentation comments. Nomic filtered out docstrings that weren’t English, were too short, or contained URLs and HTML, then applied a dual-consistency pass that drops a pair unless the code shows up among the most similar candidates for its docstring.

Second, the negatives are hard negatives: mined to look nearly right. In the example above, every negative is also a boolean-to-string helper. That’s the point: they’re what makes the dataset useful for contrastive training, and it’s also why you can’t treat them as “wrong code.” They’re wrong matches, not broken programs. Rows carry anywhere from 3 to about 100 of them, with a parallel negative_scores array.

You’ll also notice rows where document_score is "0.0" and document_rank is "-1". Those are cases where the positive document wasn’t ranked by the retriever at all. If you want only the cleanly-mined triplets, filtering on document_rank >= 0 is a reasonable first cut.

Getting the data

Plan for disk first

99 GB plus 209 GB is 308 GB before you decompress anything, and the CoRNStack shards will expand to several times their compressed size. Don’t decompress them. PHP reads gzip transparently, as we’ll see, so leave them as-is and save yourself half a terabyte.

Also: you almost certainly don’t need all of it. Both datasets are shard-per-file, and any single shard is a representative sample. Start with one or two.

The hf CLI

The Hugging Face CLI was renamed from huggingface-cli to hf in mid-2025. Install it, then grab what you need:

>pip install -U "huggingface_hub[hf_xet]"

# One shard, to see what you're dealing with
hf download xormania/PHP-Code-Large \
    php_script_only_0000.jsonl \
    --repo-type dataset --local-dir ./php-code-large

# The first ten shards of CoRNStack
hf download nomic-ai/cornstack-php-v1 \
    --repo-type dataset \
    --include "shard-0000*.jsonl.gz" \
    --local-dir ./cornstack-php

# Everything, if you're sure
hf download xormania/PHP-Code-Large --repo-type dataset --local-dir ./php-code-large

The hf_xet extra matters here. Both repos are Xet-backed (you’ll see the badge on the Files tab), and the Xet client does chunk-level deduplication and parallel transfer — noticeably faster than plain HTTP on repos this size. Downloads resume if interrupted; just re-run the same command.

Or just curl

Every file on the Hub has a stable resolve URL, and these datasets are public, so no token is needed:

curl -L -C - -O \
  https://huggingface.co/datasets/xormania/PHP-Code-Large/resolve/main/php_script_only_0000.jsonl

curl -L -C - -O \
  https://huggingface.co/datasets/nomic-ai/cornstack-php-v1/resolve/main/shard-00000.jsonl.gz

-C - resumes a partial file, which you will need at least once at these sizes.

git clone also works if you have git-lfs installed, but it’s the worst option: you end up with the data twice (working tree plus .git), which turns 99 GB into ~200 GB.

Sampling without downloading anything

For a first look, the Dataset Viewer API serves rows straight over HTTP, up to 100 at a time, with no download and no auth for public datasets:

<?php

$url = 'https://datasets-server.huggingface.co/rows?' . http_build_query([
    'dataset' => 'nomic-ai/cornstack-php-v1',
    'config'  => 'default',
    'split'   => 'train',
    'offset'  => 0,
    'length'  => 20,
]);

$data = json_decode(file_get_contents($url), true, 512, JSON_THROW_ON_ERROR);

foreach ($data['rows'] as $r) {
    printf("%-60s => %d negatives\n",
        substr($r['row']['query'], 0, 60),
        count($r['row']['negatives'])
    );
}
?>

There’s a /search and a /filter endpoint too, plus /size for exact row counts. Handy for prototyping a schema before you commit to 200 GB.

Reading JSONL efficiently in PHP

Now the interesting part. The whole game is: never load a file into memory, never decode a row you’re going to throw away.

The streaming reader

This one generator handles both datasets, gzipped or not:

<?php

declare(strict_types=1);

/**
 * Stream a .jsonl or .jsonl.gz file, one decoded row at a time.
 *
 * @return \Generator<int, array<string, mixed>>
 */
function jsonl_rows(string $path): \Generator
{
    $uri = str_ends_with($path, '.gz') ? 'compress.zlib://' . $path : $path;

    $fh = fopen($uri, 'rb');
    if ($fh === false) {
        throw new \RuntimeException("Cannot open {$path}");
    }
    stream_set_chunk_size($fh, 1 << 20); // 1 MiB reads instead of 8 KiB

    $lineNo = 0;
    try {
        while (($line = fgets($fh)) !== false) {
            $lineNo++;
            if (trim($line) === '') {
                continue;
            }
            $row = json_decode($line, true, 512, JSON_INVALID_UTF8_SUBSTITUTE);
            if (!is_array($row)) {
                fwrite(STDERR, "skipping malformed line {$lineNo} in {$path}\n");
                continue;
            }
            yield $lineNo => $row;
        }
    } finally {
        fclose($fh);
    }
}
?>

Usage is exactly what you’d hope:

<?php

foreach (jsonl_rows('shard-00000.jsonl.gz') as $lineNo => $row) {
    // $row['query'], $row['document'], $row['negatives'] …
}

?>

Four details in there are doing real work:

compress.zlib:// is PHP’s built-in stream wrapper for gzip. fgets() works through it line by line, so a 1.1 GB compressed shard costs you a few megabytes of RAM. No temp files, no shelling out to gunzip.

stream_set_chunk_size() bumps the read buffer from the 8 KiB default to 1 MiB. On files this size that’s a free speedup.

JSON_INVALID_UTF8_SUBSTITUTE is not optional, and this is the gotcha that will bite you. Real-world PHP is full of Latin-1 comments, mangled encodings, and stray bytes from 2009. Plain json_decode() returns null on a malformed UTF-8 sequence with no explanation beyond “Malformed UTF-8 characters” — so you’d silently drop those rows. With the flag, bad bytes become U+FFFD and the row survives:

<?php

$s = "{\"code\":\"<!--?php // \xe9 legacy\"}"; var_dump(json_decode($s, true)); // NULL var_dump(json_decode($s, true, 512, JSON_INVALID_UTF8_SUBSTITUTE)); // array -->

?>

fgets() with no length limit reads a full line however long it is. PHP-Code-Large has code values around 2 MB and CoRNStack has documents up to ~1 MB; both stream fine: I measured ~6 MiB peak RSS reading a 2 MB row under a 64 MB memory_limit. Don’t pass a length argument to be safe, as you’ll silently split records.

Filter before you decode

json_decode() is the expensive step. If you’re looking for a needle, test the raw line first:

<?php

$fh = fopen('big.jsonl', 'rb');
stream_set_chunk_size($fh, 1 << 20); while (($line = fgets($fh)) !== false) { if (stripos($line, 'new PDO') === false) { continue; // cheap reject, no decode } $row = json_decode($line, true, 512, JSON_INVALID_UTF8_SUBSTITUTE); // … } 

?>

On a 98 MB test file with a 2% hit rate, that took 0.10 s versus 0.35 s for decode-everything. That is about 3.5x on the same data. Scale that across 99 GB and it’s the difference between a coffee and a lunch break.

One caveat: you’re matching against JSON-escaped text. Newlines are \n, quotes are \", and forward slashes may appear as \/. So stripos($line, 'echo "hello"') won’t match, because the file contains echo \"hello\". Stick to needles without quotes, backslashes, or newlines, such as function names, keywords, class names, and do exact matching after the decode.

Random access with an offset index

Sequential streaming is right for most jobs, but sometimes you want row 4,812,003 specifically. Build a fixed-width index once, 8 bytes per row, so ~64 MB for 8M rows, and seek forever after:

<?php

/** One 64-bit little-endian byte offset per row. */
function build_index(string $jsonl, string $idx): int
{
    $in  = fopen($jsonl, 'rb');
    $out = fopen($idx, 'wb');
    stream_set_chunk_size($in, 1 << 20);

    $rows = 0;
    $offset = 0;
    while (($line = fgets($in)) !== false) {
        fwrite($out, pack('P', $offset));
        $offset += strlen($line);
        $rows++;
    }
    fclose($in);
    fclose($out);
    return $rows;
}

/** Fetch row $n (0-based) without scanning. */
function get_row(string $jsonl, string $idx, int $n): ?array
{
    $ih = fopen($idx, 'rb');
    fseek($ih, $n * 8);
    $packed = fread($ih, 8);
    fclose($ih);
    if ($packed === false || strlen($packed) < 8) {
        return null;                    // past end of file
    }

    $fh = fopen($jsonl, 'rb');
    fseek($fh, unpack('P', $packed)[1]);
    $line = fgets($fh);
    fclose($fh);

    return $line === false ? null
        : json_decode($line, true, 512, JSON_INVALID_UTF8_SUBSTITUTE);
}
?>

This gives you shuffled batches, deterministic train/test splits by row number, and reproducible sampling. It only works on uncompressed files, as gzip streams aren’t seekable, so it’s a natural fit for PHP-Code-Large, and for CoRNStack you’d index after decompressing a shard you care about.

Making it queryable: SQLite + FTS5

Once you’ve filtered down to a workable subset, put it somewhere you can actually search. SQLite with an FTS5 index is hard to beat for this, and it ships with most PHP builds, thanks to pdo_sqlite:

<?php

$db = new PDO('sqlite:php_corpus.sqlite', null, null, [
    PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
]);
$db->exec('PRAGMA journal_mode = WAL');
$db->exec('PRAGMA synchronous = OFF');       // bulk load; turn back on afterwards

$db->exec('CREATE TABLE IF NOT EXISTS snippets (
    id    INTEGER PRIMARY KEY,
    sha1  BLOB UNIQUE,
    shard TEXT,
    line  INTEGER,
    code  TEXT
)');
$db->exec('CREATE VIRTUAL TABLE IF NOT EXISTS snippets_fts
           USING fts5(code, content=snippets, content_rowid=id, tokenize="unicode61")');

$insert = $db->prepare(
    'INSERT OR IGNORE INTO snippets (sha1, shard, line, code) VALUES (?, ?, ?, ?)'
);

$shard = 'php_script_only_0000.jsonl';
$db->beginTransaction();
$n = 0;
foreach (jsonl_rows($shard) as $line => $row) {
    $code = $row['code'] ?? $row['document'] ?? null;
    if ($code === null) {
        continue;
    }
    $insert->execute([sha1($code, true), $shard, $line, $code]);
    if (++$n % 5000 === 0) {            // commit in batches
        $db->commit();
        $db->beginTransaction();
    }
}
$db->commit();
$db->exec("INSERT INTO snippets_fts(snippets_fts) VALUES('rebuild')");

?>

Two things here worth noticing. Wrapping inserts in transactions, where we are committing every few thousand rows rather than per row: it is the single biggest performance factor in SQLite bulk loading, typically by orders of magnitude. And the sha1 unique column deduplicates as you go, which matters more than you’d think: scraped corpora are full of vendored libraries, so the same Mobile_Detect.php appears over and over.

Then you can search:

<?php

$st = $db->prepare(
    "SELECT s.shard, s.line, snippet(snippets_fts, 0, '[', ']', '…', 12) AS excerpt
     FROM snippets_fts
     JOIN snippets s ON s.id = snippets_fts.rowid
     WHERE snippets_fts MATCH ?
     ORDER BY bm25(snippets_fts)
     LIMIT 20"
);
$st->execute(['password_hash']);
foreach ($st as $r) {
    echo "{$r['shard']}:{$r['line']}  {$r['excerpt']}\n";
}

?>

Parallelism: one process per shard

PHP’s threading story is still unfolding, but you don’t need threads. The data is already sharded, and the work is easily parallel. Write a script that handles one file, then fan out with xargs:

ls php_script_only_*.jsonl | xargs -P 8 -n 1 php worker.php

Each worker writes its own output file (TSV, JSONL, or its own SQLite database that you merge later). No locking, no shared state, no pcntl extension required, and you can kill and resume individual shards. Set -P to roughly your core count. You’ll usually hit I/O limits before CPU limits, especially on the gzipped shards where decompression is the bottleneck.

Run everything under the CLI SAPI, and bump the memory limit for comfort: php -d memory_limit=512M worker.php.

Things to watch out for

Dataset licence ≠ code licence. PHP-Code-Large is published under MIT and CoRNStack under Apache-2.0, but those cover the dataset compilations. The underlying code was written by thousands of people under whatever licences they chose, GPL included. For model training this is contested legal territory; for anything where you’d paste a snippet into a product, do your own diligence.

There are real credentials in there. Scraped source code contains hardcoded database passwords, API keys, and salts. The PHP-Code-Large preview on the Hub has a $dbpw = 'root' config in the first page of rows. Scan for secrets before you publish anything derived from this, and don’t be surprised when a model trained on it emits a plausible-looking API key. This is were all of this is coming from.

PSA: Never execute what you read. No eval(), no include of extracted files, not even in a container you think is disposable. If you need to analyse structure, parse it: nikic/php-parsergives you an AST without running a line.

Expect near-duplicates. Exact-hash dedup catches vendored copies; it won’t catch the same file with one changed constant. If duplicate ratio matters for your work, look at MinHash or SimHash over token shingles.

Encoding is messy. Beyond the invalid-UTF-8 problem, you’ll find BOMs at the start of files and mixed line endings. Strip \xEF\xBB\xBF if you’re tokenising.

Where to start

If you want a PHP corpus for pretraining, tokenizer work, or static-analysis research, take PHP-Code-Large, pull two or three shards, dedup them, and see whether the quality mix suits you before committing to 99 GB.

If you’re building code search, an embedding model, a reranker, or an evaluation set for one, CoRNStack is the better fit, and its hard negatives are the reason. Start with one 1.1 GB shard, filter to document_rank >= 0, and you’ll have a few hundred thousand usable triplets on your laptop.

Either way, the tooling is a fifty-line generator and whatever you already know about PHP. You don’t need a GPU cluster to start looking at the data and looking at the data is almost always the step people skip.

In the end, it could be useful to have a organized collection of source code. These datasets are not too old, but, by simple mass, they are still promoting lots of old PHP syntax, which have been modernized and should not be available anymore. We can leave the compilation of these dataset to random authors, or we could take ownership of that and drive AI in the right direction. Anyone has a few hard drives with peta bytes of space, and a few paralell CPUs to improve these?