This is the Linux app named CC-Net whose latest release can be downloaded as cc_net1.0.0sourcecode.tar.gz. It can be run online in the free hosting provider OnWorks for workstations.
Download and run online this app named CC-Net with OnWorks for free.
このアプリを実行するには、次の手順に従ってください。
-1。このアプリケーションをPCにダウンロードしました。
--2。ファイルマネージャーhttps://www.onworks.net/myfiles.php?username=XXXXXに必要なユーザー名を入力します。
-3。このアプリケーションをそのようなファイルマネージャにアップロードします。
-4。このWebサイトからOnWorksLinuxオンラインまたはWindowsオンラインエミュレーターまたはMACOSオンラインエミュレーターを起動します。
-5。起動したばかりのOnWorksLinux OSから、必要なユーザー名でファイルマネージャーhttps://www.onworks.net/myfiles.php?username=XXXXXにアクセスします。
-6。アプリケーションをダウンロードし、インストールして実行します。
スクリーンショット：
CCネット
説明：
cc_net provides tools to download, segment, clean, and filter Common Crawl to build large-scale text corpora, including monolingual datasets and the multilingual CC-100 collection introduced in the associated paper. It includes pipelines to fetch snapshots, extract text, de-duplicate, identify language, and apply quality filtering based on heuristics and language models. The outputs are intended for pretraining language models and for creating standardized corpora that can be reproduced or updated with new crawls. The repository documents practical concerns like HTTP failures, snapshot differences, and stats JSONs, reflecting community use across many languages. While powerful, the repo has been archived and is read-only, so users should expect to run it as-is or fork for maintenance. Even in archived state, issues and releases pages remain useful references for implementation details and dataset lineage.
オプション
- End-to-end Common Crawl download and extraction
- Language identification and monolingual segmentation
- Quality filtering and de-duplication pipelines
- Support for building multilingual datasets like CC-100
- Reproducible statistics and corpus metadata outputs
- Scripts and configs for snapshot-by-snapshot processing
プログラミング言語
Python
カテゴリー
This is an application that can also be fetched from https://sourceforge.net/projects/cc-net.mirror/. It has been hosted in OnWorks in order to be run online in an easiest way from one of our free Operative Systems.