This is the Linux app named CC-Net whose latest release can be downloaded as cc_net1.0.0sourcecode.tar.gz. It can be run online in the free hosting provider OnWorks for workstations.
Download and run online this app named CC-Net with OnWorks for free.
Follow these instructions in order to run this app:
- 1. Downloaded this application in your PC.
- 2. Enter in our file manager https://www.onworks.net/myfiles.php?username=XXXXX with the username that you want.
- 3. Upload this application in such filemanager.
- 4. Start the OnWorks Linux online or Windows online emulator or MACOS online emulator from this website.
- 5. From the OnWorks Linux OS you have just started, goto our file manager https://www.onworks.net/myfiles.php?username=XXXXX with the username that you want.
- 6. Download the application, install it and run it.
SCREENSHOTS:
CC-Net
DESCRIPTION:
cc_net provides tools to download, segment, clean, and filter Common Crawl to build large-scale text corpora, including monolingual datasets and the multilingual CC-100 collection introduced in the associated paper. It includes pipelines to fetch snapshots, extract text, de-duplicate, identify language, and apply quality filtering based on heuristics and language models. The outputs are intended for pretraining language models and for creating standardized corpora that can be reproduced or updated with new crawls. The repository documents practical concerns like HTTP failures, snapshot differences, and stats JSONs, reflecting community use across many languages. While powerful, the repo has been archived and is read-only, so users should expect to run it as-is or fork for maintenance. Even in archived state, issues and releases pages remain useful references for implementation details and dataset lineage.
Features
- End-to-end Common Crawl download and extraction
- Language identification and monolingual segmentation
- Quality filtering and de-duplication pipelines
- Support for building multilingual datasets like CC-100
- Reproducible statistics and corpus metadata outputs
- Scripts and configs for snapshot-by-snapshot processing
Programming Language
Python
Categories
This is an application that can also be fetched from https://sourceforge.net/projects/cc-net.mirror/. It has been hosted in OnWorks in order to be run online in an easiest way from one of our free Operative Systems.