Datasets
Three corpora licensed for training.
Collected under one licence policy and delivered as JSONL or Parquet. See the catalogue and an evaluation sample before you decide.
Multilingual text
The long tail of human language.
A text corpus for the languages and subjects that general web crawls cover poorly: rare and low-resource languages, and institutional and specialist writing in science, law, government, medicine, finance and engineering.
- Volume
- Growing every day. Dated figures are in the catalogue.
- Languages
- More than 130, many of them low-resource
- Subjects
- More than 140
- Licence classes
- Public domain, CC0, attribution (CC BY)
- Formats
- JSONL, Apache Parquet
Source code
From COBOL to Rust.
Banks, insurers and public bodies still run on COBOL, Fortran, JCL and PL/I, and models that modernise those systems need to learn them. This corpus pairs that legacy code with modern languages, under permissive licences only.
- Volume
- Growing. Dated figures are in the catalogue.
- Legacy
- COBOL, Fortran, JCL, PL/I
- Modern
- Rust, C, C++, Java, Go, MATLAB
- Licences
- MIT, Apache-2.0, BSD, ISC, Zlib, 0BSD, CC0, Unlicense. No GPL or AGPL.
- Licence evidence
- Recorded for every file
- Credentials
- Files with credential-like strings are delivered only with synthetic values.
Biomedical and life science
For drug discovery and clinical language.
Biomedical text from pharmacology and neurology to tropical medicine and biochemistry, with structured records available alongside it. Nothing here is patient-level data.
- Volume
- Growing. Dated figures are in the catalogue.
- Subjects
- Pharmacology, neurology, biochemistry, immunology, epidemiology, tropical medicine and more
- Structured records
- Regulatory drug labelling, de-identified adverse-event reports, clinical-trial registry records, measured binding affinities, gene and pathway annotations
- Not offered
- Patient-level data. Adverse-event reports are de-identified regulatory filings, not a patient cohort.
Coverage
Languages and subjects in the text corpus.
Volumes differ a lot by language. The catalogue shows what is available for the ones you need.
Languages include
Abkhazian · Acehnese · Afrikaans · Albanian · Amharic · Ancient Greek · Armenian · Avar · Aymara · Azerbaijani · Bambara · Bashkir · Basque · Belarusian · Bengali · Bislama · Breton · Bulgarian · Burmese · Catalan · Cebuano · Chamorro · Chechen · Chichewa · Chinese · Cornish · Croatian · Czech · Danish · Dutch · Dzongkha · English · Esperanto · Estonian · Fijian · Finnish · French · Fula · Galician · Ge'ez · Georgian · German · Gothic · Greek · Guaraní · Gujarati · Hausa · Hawaiian · Hindi · Hungarian · Icelandic · Igbo · Indonesian · Inuktitut · Irish · Italian · Japanese · Javanese · Kazakh · Khmer · Kinyarwanda · Korean · Kyrgyz · Lao · Latin · Latvian · Lezgian · Lingala · Lithuanian · Lower Sorbian · Luganda · Macedonian · Maithili · Malay · Malayalam · Maltese · Manx · Māori · Marathi · Marshallese · Mon · Mongolian · Nahuatl · Nepali · Norwegian · Occitan · Odia · Oromo · Ossetian · Polish · Portuguese · Punjabi · Quechua · Romanian · Russian · Samoan · Sanskrit · Serbian · Sesotho · Shona · Sinhala · Slovak · Slovenian · Somali · Spanish · Sundanese · Swahili · Swedish · Tagalog · Tajik · Tamil · Tatar · Telugu · Thai · Tibetan · Tigrinya · Tok Pisin · Tongan · Tsonga · Tswana · Turkish · Turkmen · Ukrainian · Uzbek · Vietnamese · Welsh · Wolof · Xhosa · Yoruba · Zulu
Subjects include
Science and research · Law, courts and legislation · Government and parliamentary records · Medicine · Traditional medicine, Ayurveda and Islamic medicine (Tibb Nabawi) · Indigenous knowledge · Banking, finance and tax · Engineering and aeronautics · Agriculture · Architecture · Astronomy and chemistry · Linguistics · Theology and humanities · Education · Legacy IT systems
A language is listed as recorded on the document. Languages still under quality review appear in the catalogue with that status.
Every record
A selection you can audit and reproduce.
- Licence class, licence and the evidence behind it
- Attribution notice wherever a licence requires one
- Language as recorded, and subject
- Token count and a content hash
- Quality flags, reported and never silently removed
{ "text": "…", "language": "de", "subject": "medicine", "licence_class": "public-domain", "licence_evidence": "source statement", "tokens": 1842, "text_hash": "9f2c…e41a", "quality_flags": [] }
Licence policy
One rule for everything we deliver.
A document is delivered only if its licence allows commercial use, including training. Anything we cannot classify is held back, not delivered.
What we deliver
- Public domain and CC0
- CC BY, with the attribution delivered alongside the data
- Code under MIT, Apache-2.0, BSD, ISC, Zlib, 0BSD, CC0 and Unlicense
What we never deliver
- Non-commercial (NC) and no-derivatives (ND) licences
- Share-alike (SA) licences
- GPL, AGPL and other copyleft code licences
- Anything with a missing or unclear licence
Delivery
Delivered in the size you need.
Long documents are cut into self-contained units at sentence boundaries, so the volume per subject matches what you need.
pages per unit
pages per unit
pages per unit
How licensing works
From first look to delivery.
Catalogue
Volumes by language, subject and licence, rounded down and dated.
Evaluation sample
Real units in the delivery format, so you can judge schema and quality.
NDA and agreement
We sign an NDA, then agree scope and licence terms.
Delivery
JSONL or Parquet over an encrypted transfer, with documentation and attribution notices.
Questions buyers ask.
Can we see data before we sign anything?
Yes. The catalogue lists volumes by language, subject and licence. An evaluation sample in the delivery format follows, and larger evaluation packs are shared under NDA.
Do you disclose where the data comes from?
No. Sourcing is confidential. Each document carries its licence class and the evidence for it, and attribution is delivered wherever a licence requires it.
Is personal data removed?
Personal-data detection can be run on any delivery, and the detected rates are reported with it. Automated detection does not find everything, so we report what was found and never claim that nothing remains.
Can we license a single language or subject?
Yes. Every document is labelled with its language and subject, so a delivery can be cut by language, subject, licence class or size.
How is a delivery sized?
Long documents are cut into self-contained units at sentence boundaries, in three sizes: up to about 2 pages, about 2 to 20 pages, and about 20 to 100 pages.
Do you provide a quality report with the data?
Yes. A benchmark report can be delivered with the dataset. It measures duplicates, detected personal data, licence mix and overlap with evaluation sets.
Ask for the catalogue.
Tell us which languages, subjects or programming languages you need.