This is the Linux app named CC-Net whose latest release can be downloaded as cc_net1.0.0sourcecode.tar.gz. It can be run online in the free hosting provider OnWorks for workstations.
Download and run online this app named CC-Net with OnWorks for free.
Siga estas instrucciones para ejecutar esta aplicación:
- 1. Descargue esta aplicación en su PC.
- 2. Ingrese en nuestro administrador de archivos https://www.onworks.net/myfiles.php?username=XXXXX con el nombre de usuario que desee.
- 3. Cargue esta aplicación en dicho administrador de archivos.
- 4. Inicie el emulador en línea OnWorks Linux o Windows en línea o el emulador en línea MACOS desde este sitio web.
- 5. Desde el SO OnWorks Linux que acaba de iniciar, vaya a nuestro administrador de archivos https://www.onworks.net/myfiles.php?username=XXXXX con el nombre de usuario que desee.
- 6. Descarga la aplicación, instálala y ejecútala.
SCREENSHOTS
Ad
CC-Net
DESCRIPCIÓN
cc_net provides tools to download, segment, clean, and filter Common Crawl to build large-scale text corpora, including monolingual datasets and the multilingual CC-100 collection introduced in the associated paper. It includes pipelines to fetch snapshots, extract text, de-duplicate, identify language, and apply quality filtering based on heuristics and language models. The outputs are intended for pretraining language models and for creating standardized corpora that can be reproduced or updated with new crawls. The repository documents practical concerns like HTTP failures, snapshot differences, and stats JSONs, reflecting community use across many languages. While powerful, the repo has been archived and is read-only, so users should expect to run it as-is or fork for maintenance. Even in archived state, issues and releases pages remain useful references for implementation details and dataset lineage.
Caracteristicas
- End-to-end Common Crawl download and extraction
- Language identification and monolingual segmentation
- Quality filtering and de-duplication pipelines
- Support for building multilingual datasets like CC-100
- Reproducible statistics and corpus metadata outputs
- Scripts and configs for snapshot-by-snapshot processing
Lenguaje de programación
Python
Categorías
This is an application that can also be fetched from https://sourceforge.net/projects/cc-net.mirror/. It has been hosted in OnWorks in order to be run online in an easiest way from one of our free Operative Systems.
