EnglishFrenchSpanish

Ad


OnWorks favicon

Web Crawler Security Tool download for Linux

Free download Web Crawler Security Tool Linux app to run online in Ubuntu online, Fedora online or Debian online

This is the Linux app named Web Crawler Security Tool whose latest release can be downloaded as crawler_v1.0.1.tar.gz. It can be run online in the free hosting provider OnWorks for workstations.

Download and run online this app named Web Crawler Security Tool with OnWorks for free.

Follow these instructions in order to run this app:

- 1. Downloaded this application in your PC.

- 2. Enter in our file manager https://www.onworks.net/myfiles.php?username=XXXXX with the username that you want.

- 3. Upload this application in such filemanager.

- 4. Start the OnWorks Linux online or Windows online emulator or MACOS online emulator from this website.

- 5. From the OnWorks Linux OS you have just started, goto our file manager https://www.onworks.net/myfiles.php?username=XXXXX with the username that you want.

- 6. Download the application, install it and run it.

SCREENSHOTS

Ad


Web Crawler Security Tool


DESCRIPTION

Last update on tue mar 26 16:25 UTC 2012

The Web Crawler Security is a python based tool to automatically crawl a web site. It is a web crawler oriented to help in penetration testing tasks. The main task of this tool is to search and list all the links (pages and files) in a web site.

The crawler has been completely rewritten in v1.0 bringing a lot of improvements: improved the data visualization, interactive option to download files, increased speed in crawling, exports list of found files into a separated file (useful to crawl a site once, then download files and analyse them with FOCA), generate an output log in Common Log Format (CLF), manage basic authentication and more!

Many of the old features has been reimplemented and the most interesting one is the capability of the crawler to search for directory indexing.

Features

  • Crawl http and https web sites (even web sites not using common ports). Crawl http and https web sites (even web sites not using common ports). Crawl http and https web sites (even web sites not using common ports).
  • (new!) It allows to determine the depth of the crawling (-C <depth> option)
  • (new!) Generates a summary at the end of the crawling with statistics about the crawl results
  • (new!) Implemented HEAD method for analysing file types before crawling. This feature improves the speed of the crawler significantly.
  • Uses regular expressions to find 'href', 'src' and 'content' links.
  • Identifies relative links.
  • Identifies non-html files and shows them.
  • Not crawl non-html files.
  • Identifies directory indexing.
  • Crawl directories with indexing (not yet implemented in v1.0)
  • Uses CTRL-C to stop current crawler stages and continue working. Very useful stuff...
  • Identifies all kind of files by reading the content-type header field of the response.
  • Exports (-e option) in a separated file a list of all files URLs found during crawling.
  • Select type of files to download (-d option). Ex.: png,pdf,jpeg,gif or png,jpeg.
  • Select in an interactive way which type of files to download (-i option).
  • Save the downloaded files into a directory. It only creates the output directory if there is at least one file to download.
  • Generates a output log in CLF (Common Log Format) of all the request done during crawling.
  • (beta) Login with basic authentication. Feedback is welcome!
  • Tries to detect if the website uses a CMS (like wordpress, joomla, etc) (not yet implemented in v1.0)
  • It looks for '.bk' or '.bak' files of php, asp, aspx, jps pages. (not yet implemented in v1.0)
  • It identifies and calculates the number of unique web pages crawled. (not yet implemented in v1.0)
  • It identifies and calculates the number of unique web pages crawled that contains parameters in URL. (not yet implemented in v1.0)
  • It works in Windows, but didn't save results yet


Audience

Information Technology


User interface

Console/Terminal


Programming Language

Python



This is an application that can also be fetched from https://sourceforge.net/projects/webcrawler-py/. It has been hosted in OnWorks in order to be run online in an easiest way from one of our free Operative Systems.


Free Servers & Workstations

Download Windows & Linux apps

  • 1
    Asuswrt-Merlin
    Asuswrt-Merlin
    Asuswrt-Merlin is a third party
    firmware for select Asus wireless
    routers. Based on the Asuswrt firmware
    developed by Asus, it brings tweaks, new
    features and ...
    Download Asuswrt-Merlin
  • 2
    Atom
    Atom
    Atom is a text editor that's
    modern, approachable and full-featured.
    It's also easily customizable- you
    can customize it to do anything and be
    able to ...
    Download Atom
  • 3
    Osu!
    Osu!
    Osu! is a simple rhythm game with a well
    thought out learning curve for players
    of all skill levels. One of the great
    aspects of Osu! is that it is
    community-dr...
    Download Osu!
  • 4
    LIBPNG: PNG reference library
    LIBPNG: PNG reference library
    Reference library for supporting the
    Portable Network Graphics (PNG) format.
    Audience: Developers. Programming
    Language: C. This is an application that
    can also...
    Download LIBPNG: PNG reference library
  • 5
    Metal detector based on  RP2040
    Metal detector based on RP2040
    Based on Raspberry Pi Pico board, this
    metal detector is included in pulse
    induction metal detectors category, with
    well known advantages and disadvantages.
    RP...
    Download Metal detector based on RP2040
  • 6
    PAC Manager
    PAC Manager
    PAC is a Perl/GTK replacement for
    SecureCRT/Putty/etc (linux
    ssh/telnet/... gui)... It provides a GUI
    to configure connections: users,
    passwords, EXPECT regula...
    Download PAC Manager
  • More »

Linux commands

  • 1
    4ti2-output
    4ti2-output
    4ti2 - software package for algebraic,
    geometric and combinatorial problems on
    linear spaces ...
    Run 4ti2-output
  • 2
    4ti2-ppi
    4ti2-ppi
    4ti2 - software package for algebraic,
    geometric and combinatorial problems on
    linear spaces ...
    Run 4ti2-ppi
  • 3
    convertToAtac
    convertToAtac
    sim4dbutils - utilities to work with
    sim4db-generated alignment files
    convertPolishes - convert between sim4db
    and GFF3 formats filterPolishes - filter
    alignme...
    Run convertToAtac
  • 4
    convertToExtent
    convertToExtent
    sim4dbutils - utilities to work with
    sim4db-generated alignment files
    convertPolishes - convert between sim4db
    and GFF3 formats filterPolishes - filter
    alignme...
    Run convertToExtent
  • 5
    fweelin
    fweelin
    freewheeling � live looping musical
    instrument ...
    Run fweelin
  • 6
    fwexec
    fwexec
    fwexec - program to upload and rexecute
    image file to a connected NXT device ...
    Run fwexec
  • More »

Ad