--- title: "Two Large PHP Datasets, and How to Read Them With Itself" url: https://www.exakat.io/two-large-php-datasets-and-how-to-read-them-with-itself/ date: 2026-08-12 modified: 2026-08-12 author: "dams" description: "Two Large PHP Datasets, and How to Read Them With Itself PHP has a large code base online. At Exakat, we run the engine on a corpus of 3000+ open..." categories: - "Code auditing" tags: - "AI" - "dataset" - "php" image: https://www.exakat.io/wp-content/uploads/2026/08/heap.320.jpg word_count: 2657 --- # Two Large PHP Datasets, and How to Read Them With Itself # Two Large PHP Datasets, and How to Read Them With Itself PHP has a large code base online. At [Exakat](https://www.exakat.io/), we run the engine on a corpus of 3000+ open source projects. If you want to train, fine-tune, or evaluate an AI model on PHP code, or just mine a few hundred thousand real-world snippets for research, there are two open datasets on Hugging Face worth knowing about. They are big, they are plain text, and, refreshingly, you just need PHP to access them. This post covers what's in each one, how to download them without melting your disk, and how to read them efficiently from PHP itself. From there, do whatever you want with them. ## Searching for PHP source code datasets When looking for dataset, I mostly check [HuggingFace](https://huggingface.co/datasets). Other options, such as [kaggle.com](https://www.kaggle.com/), [github.com](https://www.github.com/) or [Open Data on AWS](https://aws.amazon.com/fr/opendata/) did not yield any interesting results: I still mention them as this might change, or you're better at finding them than me: in every case, please ping me about it! So far, HuggingFace has landed some interesting finds. I'll share two interesting datasets just below. There are also datasets, such as [The Stack](https://huggingface.co/datasets/bigcode/the-stack) which have up to 400, 600 or 900 programming languages, as one big depot. For PHP, this means downloading them and filtering them before using them. I've set them aside for another time. ## The two datasets | | **xormania/PHP-Code-Large** | **nomic-ai/cornstack-php-v1** | | --- | --------------------------- | ----------------------------- | | Content | Raw PHP source files | ``retrieval triplets | | Rows | ~8.07M (viewer estimate) | ~19M (viewer estimate) | | Size on disk | 99.3 GB | 209 GB (gzipped) | | Files | `php_script_only_0000.jsonl`, … | `shard-00000.jsonl.gz`, … | | Compression | None | gzip | | Licence | MIT | Apache-2.0 | ## PHP-Code-Large **[PHP-Code-Large](https://huggingface.co/datasets/xormania/PHP-Code-Large) **is the simple one: a pile of PHP scraped from open source projects, two fields per row. That's it. Individual `code` values run from a single character up to ~2 MB, and the mix is exactly what you'd expect from the open-source PHP world: WordPress-style hook functions, Laravel controllers, CodeIgniter-era `mysql_query()` calls, Smarty templates, half-PHP/half-HTML view files, and a fair amount of code with comments in languages other than English. The dataset card advertises 12M+ lines of PHP; the Hub's viewer estimates about 8.07M rows. Rows are files (or file fragments), not lines, so treat both numbers as rough scale indicators rather than detailled counts. ## cornstack-php-v1 **[cornstack-php-v1](https://huggingface.co/datasets/nomic-ai/cornstack-php-v1)** is a different animal. It's the PHP slice of CoRNStack ([paper](https://arxiv.org/abs/2412.01007), ICLR 2025), built by Nomic to train code-retrieval models like `nomic-embed-code` and `CodeRankEmbed`. Each row pairs a natural-language query with the PHP that answers it, plus a set of plausible-but-wrong alternatives: Two things are worth internalising before you build anything on it. First, the queries were derived from function docstrings, not from humans asking questions. They read like documentation comments, because they are documentation comments. Nomic filtered out docstrings that weren't English, were too short, or contained URLs and HTML, then applied a dual-consistency pass that drops a pair unless the code shows up among the most similar candidates for its docstring. Second, the `negatives` are hard negatives: mined to look nearly right. In the example above, every negative is also a boolean-to-string helper. That's the point: they're what makes the dataset useful for contrastive training, and it's also why you can't treat them as "wrong code." They're wrong matches, not broken programs. Rows carry anywhere from 3 to about 100 of them, with a parallel `negative_scores` array. You'll also notice rows where `document_score` is `"0.0"` and `document_rank` is `"-1"`. Those are cases where the positive document wasn't ranked by the retriever at all. If you want only the cleanly-mined triplets, filtering on `document_rank >= 0` is a reasonable first cut. ## Getting the data ### Plan for disk first 99 GB plus 209 GB is 308 GB before you decompress anything, and the CoRNStack shards will expand to several times their compressed size. Don't decompress them. PHP reads gzip transparently, as we'll see, so leave them as-is and save yourself half a terabyte. Also: you almost certainly don't need all of it. Both datasets are shard-per-file, and any single shard is a representative sample. Start with one or two. ### The hf CLI The Hugging Face CLI was renamed from `huggingface-cli` to `hf` in mid-2025. Install it, then grab what you need: The `hf_xet` extra matters here. Both repos are Xet-backed (you'll see the badge on the Files tab), and the Xet client does chunk-level deduplication and parallel transfer — noticeably faster than plain HTTP on repos this size. Downloads resume if interrupted; just re-run the same command. ### Or just curl Every file on the Hub has a stable resolve URL, and these datasets are public, so no token is needed: `-C -` resumes a partial file, which you will need at least once at these sizes. `git clone` also works if you have git-lfs installed, but it's the worst option: you end up with the data twice (working tree plus `.git`), which turns 99 GB into ~200 GB. ### Sampling without downloading anything For a first look, the Dataset Viewer API serves rows straight over HTTP, up to 100 at a time, with no download and no auth for public datasets: There's a `/search` and a `/filter` endpoint too, plus `/size` for exact row counts. Handy for prototyping a schema before you commit to 200 GB. ## Reading JSONL efficiently in PHP Now the interesting part. The whole game is: never load a file into memory, never decode a row you're going to throw away. ### The streaming reader This one generator handles both datasets, gzipped or not: Usage is exactly what you'd hope: Four details in there are doing real work: `compress.zlib://` is PHP's built-in stream wrapper for gzip. `fgets()` works through it line by line, so a 1.1 GB compressed shard costs you a few megabytes of RAM. No temp files, no shelling out to `gunzip`. `stream_set_chunk_size()` bumps the read buffer from the 8 KiB default to 1 MiB. On files this size that's a free speedup. `JSON_INVALID_UTF8_SUBSTITUTE` is not optional, and this is the gotcha that will bite you. Real-world PHP is full of Latin-1 comments, mangled encodings, and stray bytes from 2009. Plain `json_decode()` returns `null` on a malformed UTF-8 sequence with no explanation beyond "Malformed UTF-8 characters" — so you'd silently drop those rows. With the flag, bad bytes become U+FFFD and the row survives: `fgets()` with no length limit reads a full line however long it is. PHP-Code-Large has `code` values around 2 MB and CoRNStack has documents up to ~1 MB; both stream fine: I measured ~6 MiB peak RSS reading a 2 MB row under a 64 MB `memory_limit`. Don't pass a length argument to be safe, as you'll silently split records. ### Filter before you decode `json_decode()` is the expensive step. If you're looking for a needle, test the raw line first: On a 98 MB test file with a 2% hit rate, that took 0.10 s versus 0.35 s for decode-everything. That is about 3.5x on the same data. Scale that across 99 GB and it's the difference between a coffee and a lunch break. One caveat: you're matching against JSON-escaped text. Newlines are `\n`, quotes are `\"`, and forward slashes may appear as `\/`. So `stripos($line, 'echo "hello"')` won't match, because the file contains `echo \"hello\"`. Stick to needles without quotes, backslashes, or newlines, such as function names, keywords, class names, and do exact matching after the decode. ### Random access with an offset index Sequential streaming is right for most jobs, but sometimes you want row 4,812,003 specifically. Build a fixed-width index once, 8 bytes per row, so ~64 MB for 8M rows, and seek forever after: This gives you shuffled batches, deterministic train/test splits by row number, and reproducible sampling. It only works on uncompressed files, as gzip streams aren't seekable, so it's a natural fit for PHP-Code-Large, and for CoRNStack you'd index after decompressing a shard you care about. ### Making it queryable: SQLite + FTS5 Once you've filtered down to a workable subset, put it somewhere you can actually search. SQLite with an FTS5 index is hard to beat for this, and it ships with most PHP builds, thanks to `pdo_sqlite`: Two things here worth noticing. Wrapping inserts in transactions, where we are committing every few thousand rows rather than per row: it is the single biggest performance factor in SQLite bulk loading, typically by orders of magnitude. And the `sha1` unique column deduplicates as you go, which matters more than you'd think: scraped corpora are full of vendored libraries, so the same `Mobile_Detect.php` appears over and over. Then you can search: ### Parallelism: one process per shard PHP's threading story is still unfolding, but you don't need threads. The data is already sharded, and the work is easily parallel. Write a script that handles one file, then fan out with `xargs`: Each worker writes its own output file (TSV, JSONL, or its own SQLite database that you merge later). No locking, no shared state, no `pcntl` extension required, and you can kill and resume individual shards. Set `-P` to roughly your core count. You'll usually hit I/O limits before CPU limits, especially on the gzipped shards where decompression is the bottleneck. Run everything under the CLI SAPI, and bump the memory limit for comfort: `php -d memory_limit=512M worker.php`. ## Things to watch out for Dataset licence ≠ code licence. PHP-Code-Large is published under MIT and CoRNStack under Apache-2.0, but those cover the dataset compilations. The underlying code was written by thousands of people under whatever licences they chose, GPL included. For model training this is contested legal territory; for anything where you'd paste a snippet into a product, do your own diligence. There are real credentials in there. Scraped source code contains hardcoded database passwords, API keys, and salts. The PHP-Code-Large preview on the Hub has a `$dbpw = 'root'` config in the first page of rows. Scan for secrets before you publish anything derived from this, and don't be surprised when a model trained on it emits a plausible-looking API key. This is were all of this is coming from. PSA: Never execute what you read. No `eval()`, no `include` of extracted files, not even in a container you think is disposable. If you need to analyse structure, parse it: `nikic/php-parser`gives you an AST without running a line. Expect near-duplicates. Exact-hash dedup catches vendored copies; it won't catch the same file with one changed constant. If duplicate ratio matters for your work, look at MinHash or SimHash over token shingles. Encoding is messy. Beyond the invalid-UTF-8 problem, you'll find BOMs at the start of files and mixed line endings. Strip `\xEF\xBB\xBF` if you're tokenising. ## Where to start If you want a PHP corpus for pretraining, tokenizer work, or static-analysis research, take PHP-Code-Large, pull two or three shards, dedup them, and see whether the quality mix suits you before committing to 99 GB. If you're building code search, an embedding model, a reranker, or an evaluation set for one, CoRNStack is the better fit, and its hard negatives are the reason. Start with one 1.1 GB shard, filter to `document_rank >= 0`, and you'll have a few hundred thousand usable triplets on your laptop. Either way, the tooling is a fifty-line generator and whatever you already know about PHP. You don't need a GPU cluster to start looking at the data and looking at the data is almost always the step people skip. In the end, it could be useful to have a organized collection of source code. These datasets are not too old, but, by simple mass, they are still promoting lots of old PHP syntax, which have been modernized and should not be available anymore. We can leave the compilation of these dataset to random authors, or we could take ownership of that and drive AI in the right direction. Anyone has a few hard drives with peta bytes of space, and a few paralell CPUs to improve these?