This is the command webcheck that can be run in the OnWorks free hosting provider using one of our multiple free online workstations such as Ubuntu Online, Fedora Online, Windows online emulator or MAC OS online emulator
PROGRAM:
NAME
webcheck - website link checker
SYNOPSIS
webcheck [OPTION]... URL
DESCRIPTION
webcheck will check the document at the specified URL for links to other documents, follow
these links recursively and generate an HTML report.
-i, --internal=PATTERN
Mark URLs matching the PATTERN (perl-type regular expression) as an internal link.
Can be used multiple times. Note that the PATTERN is matched against the full URL.
URLs matching this PATTERN will be considered internal, even if they match one of
the --external PATTERNs.
-x, --external=PATTERN
Mark URLs matching the PATTERN (perl-type regular expression) as an external link.
Can be used multiple times. Note that the PATTERN is matched against the full URL.
-y, --yank=PATTERN
Do not check URLs matching the PATTERN (perl-type regular expression). Like the -x
flag, though this option will cause webcheck to not check the link matched by regex
whereas -x will check the link but not its children. Can be used multiple times.
Note that the PATTERN is matched against the full URL.
-b, --base-only
Consider any URL not starting with the base URL to be external. For example, if
you run
webcheck -b http://www.example.com/foo
then http://www.example.com/foo/bar will be considered internal whereas
http://www.example.com/ will be considered external. By default all the pages on
the site will be considered internal.
-a, --avoid-external
Avoid external links. Normally if webcheck is examining an HTML page and it finds
a link that points to an external document, it will check to see if that external
document exists. This flag disables that action.
--ignore-robots
Do not retrieve and parse robots.txt files. By default robots.txt files are
retrieved and honored. If you are sure you want to ignore and override the
webmaster's decision this option can be used.
For more information on robots.txt handling see the NOTES section below.
-q, --quiet, --silent
Do not print out progress as webcheck traverses a site.
-d, --debug
Print debugging information while crawling the site. This option is mainly useful
for developers.
-o, --output=DIRECTORY
Output directory. Use to specify the directory where webcheck will dump its
reports. The default is the current directory or as specified by config.py. If this
directory does not exist it will be created for you (if possible).
-c, --continue
Try to continue from a previous run. When using this option webcheck will look for
a webcheck.dat in the output directory. This file is read to restore the state
from the previous run. This allows webcheck to continue a previously interrupted
run. When this option is used, the --internal, --external and --yank options will
be ignored as well as any URL arguments. The --base-only and --avoid-external
options should be the same as the previous run.
Note that this option is experimental and it's semantics may change with coming
releases (especially in relation to other options). Also note that the stored
files are not guaranteed to be compatible between releases.
-f, --force
Overwrite files without asking. This option is required for running webcheck non-
interactively.
-r, --redirects=N
Redirect depth. the number of redirects webcheck should follow when following a
link. 0 implies to follow all redirects.
-u, --userpass=URL
Specify a URL with username and password information to use for basic
authentication when visiting the site.
e.g. http://test:secret@example.com/
This option may be specified multiple times.
-w, --wait=SECONDS
Wait SECONDS between document retrievals. Usually webcheck will process a url and
immediately move on to the next. However on some loaded systems it may be desirable
to have webcheck pause between requests. This option can be set to any non-
negative number.
-v, --version
Show version of program.
-h, --help
Show short summary of options.
URL CLASSES
URLs are divided into two classes:
Internal URLs are retrieved and the retrieved item is checked for syntax. Also, the
retrieved item is searched for links to other items (of any class) and these links are
followed.
External URLs are only retrieved to test whether they are valid and to gather some basic
information from them (title, size, content-type, etc). The retrieved items are not
inspected for links to other items.
Apart from their class, URLs can also be considered yanked (as specified with the --yank
or --avoid-external options). The URLs can be either internal or external and will not be
retrieved or checked at all. URLs of unsupported schemes are also considered yanked.
EXAMPLES
Check the site www.example.com but consider any path with "/webcheck" in it to be
external.
webcheck http://www.example.com/ -x /webcheck
NOTES
When checking internal URLs webcheck honors the robots.txt file, identifying itself as
user-agent webcheck. Disallowed links will not be checked at all as if the -y option was
specified for that URL. To allow webcheck to crawl parts of a site that other robots are
disallowed, use something like:
User-agent: *
Disallow: /foo
User-agent: webcheck
Allow: /foo
ENVIRONMENT
<scheme>_proxy
Proxy url for <scheme>.
REPORTING BUGS
Bug reports shoult be sent to the mailing list <webcheck-users@lists.arthurdejong.org>.
More information on reporting bugs can be found on the webcheck homepage:
http://arthurdejong.org/webcheck/
COPYRIGHT
Copyright © 1998, 1999 Albert Hopkins (marduk)
Copyright © 2002 Mike W. Meyer
Copyright © 2005, 2006, 2007, 2008, 2009, 2010 Arthur de Jong
webcheck is free software; see the source for copying conditions. There is NO warranty;
not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
The files produced as output from the software do not automatically fall under the
copyright of the software, unless explicitly stated otherwise.
Use webcheck online using onworks.net services