calamaris - Online in the Cloud

PROGRAM:

NAME

calamaris - generate text and graphical statistics out of log files from Proxy-Cache-
Servers

SYNOPSIS

cat log | calamaris [ --config-file .../calamaris.conf ] [ switches ]

DESCRIPTION

Calamaris is used to produce statistical output from Squid, NetCache, Inktomi Traffic
Server, Oops! proxy server, Compaq Tasksmart, Cisco Content Engines or related Proxy log
files. The resulting output can be ascii or html with or without graphic and with or
without frames. It is possible to cache calculated data in a file to use them in later
runs.

This manual page describes the options of Calamaris and gives a few examples.

OPTIONS

Configuration File
--config-file file

Not all reports and modification can be made through command-line-switches. To use
all options of Calamaris you´ll have to use the configuration file. You'll find
the configuration-directives below, always inside of braces. Examples are in the
calamaris.conf which should come with this package.

Reports
--all-useful-reports|-a
extracts all useful reports available, --all-useful-reports equals --size-
distribution-report 10 --domain-report 20 --performance-report 60 --requester-
report 20 --status-report --type-report 20 --response-time-report --errorcode-
distribution-report

--domain-report|-d n ($domain_report)
switches the top level and the second level report on. The data is derived from
the URL. The output is limited by n. (-1 is unlimited)

--domain-report-limit n ($domain_report_limit)
limit the domain-report to lines which have n or more requests.

--domain-report-n-level|-N n ($domain_report_n_level)
All URL-Host reports will be switched from 2nd-level to n-level-reports. (-1 shows
a full urlhost-report)

Note: This option is only useful with activated domain-report.

--errorcode-distribution-report ($errorcode_distribution_report)
shows the Response code distribution over all objects

($object_freshness_report)
shows the freshness of objects in your cache. Calamaris looks for freshness tags
like 'TCP_HIT', 'TCP_REFRESH_MISS', ... and make statistics on it. With this
information you can optimize the caching behaviour of your cache depending on the
objects content type. E.g. squid admins could use this information to configure
the refresh_pattern. This option needs more configuration in the configuration-
file.

--peak-report|-p type ($peak_report)
Measures the peaks of the Proxy usage in requests per second, per minute and per
hour. It reports peaks for TCP, UDP and ALL requests. If set to 'old' these values
were calculated with the old slow method, if set to 'new' the new faster (but still
slow) method is used.

--performance-report|-P n ($performance_report)
Shows the throughput of TCP requests for every n minutes.

--performance-report-adjust|-T n ($performance_report_adjust)
Time: Adjust the Performancereport in minutes for non GMT-Timezoners.

--requester-report|-r n ($requester_report)
Switches the UDP and TCP requester reports on. The output is limited by n. (-1 is
unlimited)

--requester-report-no-dns-lookup|-n ($requester_report_no_dns_lookup)
Switches the IP lookup for the requesters off.

--requester-report-use-user-info|-u ($requester_report_use_user_info)
Switches the usage of eventually available ident information for requester reports
on.

Warning: This breaks the privacy of your users! (see PRIVACY-Section below)

--requester-report-with-targets|-R n ($requester_report_with_targets)
adds to each line of the requester report the requested URLs. The output is limited
by n. (-1 is unlimited, and can result in very very long reports.)

Warning: Using this option breaks the privacy of your users! (see PRIVACY-Section
below)

--response-time-report ($response_time_report)
sums up the time distribution over all objects

(@response_time_report_interval)
This array defines the time steps, which should be reported in the response-time-
report.

--size-distribution-report|-D n ($size_distribution_report)
shows size-based distribution of requested objects, smaller numbers result in more
verbose reports. (choose 2, 10 or 100 for useful output.)

--status-report|-s ($status_report)
alters the default behaviour of Calamaris and makes the status reports more
verbose.

--type-report|-t n ($type_report)
switches the content type and the file extension report on. The output is limited
by n. (-1 is unlimited)

--type-report-ignore-case|-c ($type_report_ignore_case)
Switch to case-insensitive. This is useful for the 'Requested extensions' report.

Input
--input-format|-f type ($input_format)
sets the type of input logfiles. If set to

'auto' Calamaris tries to guess the input file format. This is the Default.

Note: If the first line of your input file is corrupted, Calamaris will stop with
an error.

'squid' Calamaris expects native logfile derived from Squid V1.1.beta26-V2.x or
OOPS.

'squid-extended' Calamaris expects native logfile derived from Squid
V1.1.alpha1-V2.x with log_mime_hdrs enabled or Squid with Smartfilter-Patch or
squid-style logfiles out of Cisco Content Engines. (This only enables parsing of
these kind of logfile, the additional data will be ignored.)

(Logging of MIME-Headers breaks the privacy of your users! (see PRIVACY-Section
below)

'squid-old' Calamaris expects native logfile derived from Squid
V1.1.alpha1-V1.1.beta25.

'nc' Calamaris expects Logfiles from NetCache up to V4.x. (Please see the README on
this.)

'its' Calamaris expects Logfiles from Inktomi Traffic Server.

'elff' Calamaris expects Logfiles in Extended Logfile Format (i.e. from Compaq
Tasksmart, Novell Internet Caching System or NetCache V5.x)

'nse' Calamaris expects Logfiles in Netscape Extended-1 or Netscape Extended-2
Logfile Format (from Netscape/iPlanet/SunOne Proxy-Server )

--ipfilter-exclude IP/range ($ipfilter_exclude)
all IPs are analyzed, except IP/range. Format: 1.1.1.1/32:1.1.2.0/24 or
1.1.1.1/255.255.255.255:1.1.2.0/255.255.255.0

IP list separated by ':' This switch needs the perl Module NetAddr::IP.

Warning: This breaks the privacy of your users! (see PRIVACY-Section below)

--ipfilter-include IP/range ($ipfilter_include)
no IPs are analyzed, except IP/range. Format: see --ipfilter-exclude

Warning: This breaks the privacy of your users! (see PRIVACY-Section below)

--no-input|-z ($no_input)
Switches reading from standard input off. You can use this to merge many cache
files to one (see --cache-input-file and --cache-output-file) or to generate a
report out of cache files.

--time-interval|-I t-t ($time_interval)
defines which time-interval should be parsed. t has to be the format
yyyymmddhhmmss (localtime)

Note: omitting the beginning or ending date is allowed.

Output
Standard output format is plain ascii with 80 chars width.

($column1_color)
($column2_color) defines the colors for the columns in graphics. (only useful with
--output-format graph)

($formats[n])
Through the config-file you are able to modify the width of the report and alter
the culomns that are displayed in the reports. n is the number of the report, as
displayed by --help in the --show-reports-option.

--hostname|-H name ($hostname)
The name for the title or subject of the output. If set to 'lookup' Calamaris
looks up the host name of the system its been run on.

--image-type ($image_type)
Sets the image type to gif, png, jpeg, gd or gd2. Only useful when --output-format
graph is set. The available images types are dependend on your GD::Graph
installation. Default is 'png'.

--logo|-l string ($logo)
add a custom string to a HTML-Report. It'll be added to a table on the top of the
output. -l '<A HREF="http://cord.de/"><IMG BORDER=0
SRC="http://cord.de/Images/cord/cordlog2n.gif" ALT="Cord"></A>' will add my logo
with a link to the Report.

Note: --logo works only in combination with --output-format html or html-frame

--meta|-M string ($meta)
Meta: adds a custom string or the content of a file into the <HEAD> of a HTML-
Report. Useful if you want to add Stylesheets or something to the Report.

Note: --meta works only in combination with --output-format html or html-frame

--output-format|-F type[,type[,type[,...]]] ($output_format)
Format: sets the format of the output-report. If set to

'mail' adds a subject header to the beginning of the report.

'html' all output is given in html with tables. Can be combined with 'mail' to
send html mails.

'html-frame' all output is given in html frames with tables.

'html-embed' all output is given in html with tables without HTML-Headers. Useful
for Server-Side-Includes.

'graph' enables graphics for html, html-embed or html-frame.

'unformatted' gives out the raw numbers separated by spaces. Useful for re-using
the output in other scripts. If you use this along with -U, the byte values are
calculated in the given Unit, and displayed without indication along with the
numbers. the indication moves up to the header of the report.

--output-path ($output_path)
output calamaris statistics to /path. In case of graph output, the graphics
destination is /path and the filename is index.html, else it is calamaris.txt. If
--output-path is not given, all graphics are written to the working directory.

--output-file ($output_file)
alter the filename of --output-path.

--output-file-prefix ($output_file_prefix)
adds a prefix to --output-file %t is replaced by the timerange of the report, %h by
the hostname (see --hostname )

--show-reports|-S n[,n[,n[,...]]] ($show_reports)
Show: Shows only the defined reports in the specified order. Default is to display
the reports as they are defined through the report-switches above. The following
numbers are defined:

0 Summary
1 Incoming request peak per protocol
2 Incoming transfer volume peak per protocol
3 Incoming requests by method
4 Incoming UDP-requests by status
5 Incoming TCP-requests by status
6 Outgoing requests by status
7 Outgoing requests by destination
8 Request-destinations by 2ndlevel-domain
9 Request-destinations by toplevel-domain
10 TCP-Request-protocol
11 Requested content-type
12 Requested extensions
13 Incoming UDP-requests by host
14 Incoming TCP-requests by host
15 Size Distribution Diagram
16 Performance in n minute steps
17 UDP-Request duration distribution in msec
18 TCP-Request duration distribution in msec
19 UDP Response code distribution
20 TCP Response code distribution

Note: Using this doesn't make Calamaris any faster, the internal calculations will be done
as the report-switches were set (see above).

--sort-order|-O ($sort_order)
Changes the sort order in the reports to request size, default is sorting by number
of requests.

($text_color)
defines the colors for text/axis/legend/labels in graphics. (only useful with
--output-format graph )

--unit|-U string ($unit)
You can define this to K(ilo), M(ega), G(iga) or T(era) for the Byte-units.

($width)
defines the width of the graphics. height is calculated from this with a 3:2-ratio.
(only useful with --output-format graph )

($x_scale)
defines how many datasets should be drawn on the graph. 30 is a good value, but you
can play with this. if $x_scale gets to big, you're on your own ;-)

--generate-index ($generate_index)
generates an index for all reports that match --output-file-prefix.

Caching
--cache-input-file|-i file ($cache_input_file)
You can reuse a cache file generated with --cache-output-file file to add old data
to a new report. Several files can be separated with a ':'.

Note: if you use more than one cache file, make sure they are chronologicaly
ordered (oldest first).

Note: if you reuse cache-files, which were not created with -d -1 -r -1 -t -1 -R -1
the number of 'others' would be wrong everywhere. In this case the number of
'others' are omitted.

--cache-output-file|-o file ($cache_output_file)
Calamaris stores a summary of the computed information in file and you can reuse it
at a later time with --cache-input-file

Note: The output file can be the same as the input file: it is simply overwritten
after reading the data. It is not recommended to change the options between
different runs if you include older data as this can result in strange
measurements.

Misc
--benchmark|-b n ($benchmark)
benchmark: A switch for the impatient as it prints a '#' for every n parsed lines.

--copyright|-C
Prints the copyright information of Calamaris

--help|-h
Prints a brief description of the command line options.

--version|-V
Prints out the Version-Number.

Debug
--dump-loop|-L
prints the internal loop to STDERR. (for Debugging)

($test)
activates some small tests for the programmer.

--verbose|-v ($verbose)
print more information about what is Calamaris is doing and believing.

EXAMPLES

This example mails the daily statistics to root:
cat /var/log/squid/access.log | nice -39 calamaris --all-useful-reports --hostname
"daily worf" --output-format mail | mail root

This one only caches a summary for later use:
cat /var/log/squid/access.log | calamaris --all-useful-reports --cache-output-file
daily.`date +"%w"` > /dev/null

You can then use the caches to have weekly statistics:

if [ $DAYOFWEEK = "0" ]; then
calamaris --all-useful-reports --cache-input-file
daily.1:daily.2:daily.3:daily.4:daily.5:daily.6:daily.0 --no-input --output-format
mail --hostname "weekly worf" | mail root ;

fi

Use calamaris online using onworks.net services