awffull - Online in the Cloud

PROGRAM:

NAME

AWFFull - A Webalizer Fork, Full o' features

SYNOPSIS

awffull [...] [log-file]

DESCRIPTION

AWFFull is a web server log analysis program based on The Webalizer. AWFFull produces
usage statistics in HTML format for viewing with a browser. The results are presented in
both columnar and graphical format, which facilitates interpretation. Yearly, monthly,
daily and hourly usage statistics are presented, along with the ability to display usage
by site, URL, referrer, user agent (browser), user name, search strings, entry/exit pages,
and country (some information may not be available if not present in the log file being
processed).

AWFFull supports the following log formats shown in the following variable list:

CLF (common log format) log files

Combined
log formats as defined by NCSA and others, and variations of these which it
attempts to handle intelligently

xferlog
wu-ftpd formatted log files allowing analysis of ftp servers, and squid proxy logs.
Note

Logs may also be compressed, via gzip. If a compressed log file is detected, it
will be automatically uncompressed while it is read. Compressed logs must have the
standard gzip extension of .gz.

This documentation applies to AWFFull Version 3.8.2

CHANGES FROM WEBALIZER

AWFFull is based on The Webalizer code and has a number of large and small changes. These
include:

· Beyond the raw statistics: Making use of published formulae to provide additional
insights into site usage

· GeoIP IP Address look-ups for more accurate country detection

· Resizable graphs

· Integration with GNU gettext allowing for ease of translations. Currently 32 languages
are supported.

· Display more than 12 months of the site history on the front page.

· Additional page count tracking and sort by same.

· Some minor visual tweaks, including Geolizer's use of Kb, Mb etc for Volumes

· Additional Pie Charts for URL counts, Entry and Exit Pages, and Sites

· Horizontal lines on graphs that are more sensible and easier to read

· User Agent and Referral tracking is now calculated via PAGES not HITS

· GNU style long command line options are now supported (eg --help)

· Can choose what is a page by excluding ‘what isn't’ vs the original ‘what is’ method

· Requests to the site being analysed are displayed with the matching referring URL

· A Table of 404 Errors, and the referring URL can be generated

· An external CSS file can be used with the generated html

· Manual performance optimisation of the config file is now easier with a post analysis
summary output

· Specified IP Addresses can be assigned to a given country

· Additional Dump options for detailed analysis with other tools

· Lotus Domino v6 logs are now detected and processed

Additional changes and improvements are planned and undergoing implementation. See the
TODO file for details.

NEW REPORT MEASUREMENTS

With version 3.8.1 of AWFFull, several new measured results have been added to the
detailed report monthly page.

Single Access
Single Access Pages - the only page seen within a given visit

Stickiness
How useful a given entry page is to draw Visitors deeper into your site

Popularity
The Ratio of Page Entries to Page Exits

These metrics can help towards improving insight in the usage of the processed web site.
And hence allow the site owner to make positive change to make the site more useful to
site visitors. All three metrics appear in the ‘Entry Pages’ Report. ‘Popularity’ is also
on the ‘Exit Pages’ Report.

Single Access

More completely: ‘Single Access Pages’. This is a report on the number of times that a
given page was the only page viewed within a Visit. Or in English, Someone came to your
website. They only viewed one page. The number is the cumulative count of people who did
this for that particular page. Why is this useful? Identifying those entry pages that
don't draw visitors deeper into your site. Or seeing entry pages that shouldn't be entry
pages. It's also a reality check against the next two values which are calculated from
this number. The number generated should be a subset of the ‘Entry Page Views’ and/or
‘Exit Page Views’ metric. If it isn't? Let me know, we have a bug. :-)

Stickiness

Is calculated as 1 - (Single Access / Entry Page Views) expressed as a percentage. In
essence Stickiness describes how useful a given entry page is to draw Visitors deeper into
your site. The stickier the page, the more folk are caught by it. :-) The closer to 100%
the better. Generally. Certain pages within YOUR website may not make sense to have a
high stickiness or even > 5%. This measurement is a clue to understanding how your site is
used, it is not a rule. How is this useful? How and where are people entering your web
site. Does that make sense? Should it be here or there? What can you change to fix this
and hence improve their use of your website.

Popularity

Popularity is the Ratio of Page Entries to Page Exits. o If it equals 1.0? Then the number
of visitors to your site who started with that page, equals the number who left at that
page. o If greater then 1.0, then more people entered here then left. o If less then 0?
More people left from here then entered. I personally find this metric one of the more
useful "At a Glance: How are Pages Performing" metrics. One of the difficulties with using
this particular metric is that certain numbers will NOT make sense for YOUR site. In that
a natural exit page would expect to have a very low Popularity. It's an exit page, not an
entry page. So if an exit page has a high popularity, then you have a real problem.
Likewise, a low Popularity for an entry page is unlikely to be a Good Thing(tm).

"Where & Why?" All three of these metrics are covered very nicely in Hack #58 from "Web
Site Measurement Hacks" [1]. Which is where, credit where credit due, the inspiration to
add these metrics came from.

RUNNING AWFFULL

AWFFull is designed to be run from a Unix command line prompt or as a crond(8) job. There
is no need to run with super-user privleges, and indeed, is preferable NOT to.

Once executed, the general flow of the program is:

A default configuration file is scanned for, /etc/awffull/awffull.conf and, if found, is
used.

Any command line arguments given to the program are parsed. This may include the
specification of one or more configuration files, which are processed at the time it is
encountered.

It can be useful to have multiple config files. A master used for multiple sites, and
individualised config files. Do be aware that last option set wins. So last config file,
or if after a config file, command line options. Useful if you desire to send the output
to an alternate directory.

If a log file was specified, it is opened and made ready for processing. If no log file
was given, STDIN is used for input. If the log filename '-' is specified, STDIN will be
forced.

If an output directory was specified, AWFFull changes to that directory in preparation for
generating output. If no output directory was given, the current directory is used.

If no hostname was given, the program attempts to get the hostname using a uname(2) system
call. If that fails, localhost is used.

A history file is searched for in the current directory (output directory) and read if
found. This file keeps totals for previous months, which is used in the main index.html
HTML document. Note: The file location can now be specified with the HistoryName
configuration option.

If incremental processing was specified, a data file is searched for and loaded if found,
containing the 'internal state' data of the program at the end of a previous run. Note:
The file location can now be specified with the IncrementalName configuration option.

Main processing begins on the log file. If the log spans multiple months, a separate HTML
document is created for each month. After main processing, the main index.html page is
created, which has totals by month and links to each months HTML document.

A new history file is saved to disk, which includes totals generated by AWFFull during the
current run.

If incremental processing was specified, a data file is written that contains the
'internal state' data at the end of this run.

INCREMENTAL PROCESSING

Version 1.2x of The Webalizer added incremental run capability. Simply put, this allows
processing large log files by breaking them up into smaller pieces, and processing these
pieces instead. What this means in real terms is that you can now rotate your log files as
often as you want, and still be able to produce monthly usage statistics without the loss
of any detail. Basically, AWFFull saves and restores all internal data in a file named
awffull.current. This allows the program to 'start where it left off' so to speak, and
allows the preservation of detail from one run to the next. The data file is placed in the
current output directory, and is a plain ASCII text file that can be viewed with any
standard text editor. It's location and name may be changed using the IncrementalName
configuration keyword.

Some special precautions need to be taken when using the incremental run capability of
AWFFull. Configuration options should not be changed between runs, as that could cause
corruption of the internal data stored. For example, changing the MangleAgents level will
cause different representations of user agents to be stored, producing invalid results in
the user agents section of the report. If you need to change configuration options, do it
at the end of the month after normal processing of the previous month and before
processing the current month. You may also want to delete the awffull.current file as
well.

AWFFull also attempts to prevent data duplication by keeping track of the timestamp of the
last record processed. This timestamp is then compared to current records being processed,
and any records that were logged previous to that timestamp are ignored. This, in theory,
should allow you to re-process logs that have already been processed, or process logs that
contain a mix of processed/not yet processed records, and not produce duplication of
statistics.

The only time this may break is if you have duplicate timestamps in two separate log
files. Any records in the second log file that do have the same timestamp as the last
record in the previous log file processed, will be discarded as if they had already been
processed. There are lots of ways to prevent this however, for example, stopping the web
server before rotating logs will prevent this situation, or using a tool such as cronolog
(⟨http://cronolog.org/⟩). This setup also necessitates that you always process logs in
chronological order, otherwise data loss will occur as a result of the timestamp compare.

REVERSE DNS LOOKUPS

AWFFull no longer supports DNS lookups. Please use an external program such as DNShistory
or DNSTran instead.

· ⟨http://www.summary.net/soft/dnstran.html⟩

· ⟨http://www.stedee.id.au/dnshistory⟩

With version 3.7.1 of AWFFull, GeoIP capability can be used for improved country
detection.

COMMAND LINE OPTIONS

AWFFull supports many different configuration options that will alter the way the program
behaves and generates output. Most of these can be specified on the command line, while
some can only be specified in a configuration file. The command line options are listed
below, with references to the corresponding configuration file keywords. See also
awffull.conf(5).

General Options

-h, --help
Display all available command line options and exit program

-V, --version
Display program version and exit program

-v --verbose
Verbosity Display debugging information for errors and warnings. Multiple v's will
increase the amount of information displayed.

--match_counts
Display optimisation useful information pertaining to the number of matches against
various Group, Hide and Ignore options.

-i --ignore_history
IgnoreHist Ignore history. USE WITH CAUTION. This will cause AWFFull to ignore any
previous monthly history file only. Incremental data (if present) is still
processed.

-p --preserve_state
Incremental Preserve internal data between runs.

-T --timing
TimeMe. Force display of timing information at end of processing.

-c --config=FILE
Use configuration file FILE

-n NAME
HostName. Use the hostname NAME.

-o --output=DIR
OutputDir. Use output directory DIR.

-t NAME
ReportTitle. Use NAME for report title.

F --logtype=TYPE
LogType. Specify log type to be processed. Value can be one of: auto, clf,
combined, domino, ftp or squid format. If not specified, will default to auto
format. FTP logs must be in standard wu-ftpd xferlog format. A value of ‘auto’
states that the log format automatically ascertained.

-f --fold
FoldSeqErr. Fold out of sequence log records back into analysis, by treating as if
they were the same date/time as the last good record. Normally, out of sequence log
records are simply ignored.

-Y CountryGraph. Suppress country graph.

-G HourlyGraph. Suppress hourly graph.

-x NAME
HTMLExtension. Defines the HTML file extension to use on the created report files.
If not specified, defaults to html. Do not include the leading period.

-H HourlyStats. Suppress hourly statistics.

-L GraphLegend. Suppress color coded graph legends.

-l NUM GraphLines. Use background lines. The number of lines and where to place are
automatically calculated. For backwards compatibility, any number > 0 enables. Use
zero ('0') to disable the lines.

-P NAME
PageType. Specify file extensions that are considered pages. Sometimes referred to
as pageviews.

-m NUM VisitTimeout. Specify the Visit timeout period. Specified in number of seconds.
Default is 1800 seconds (30 minutes). Sometimes referred to as sessions.

-I NAME
IndexAlias. Use the filename name as an additional alias for index.

-M NUM MangleAgents. Mangle user agent names according to the mangle level specified by
num.

Mangle levels are:

5 - Browser name and major version

4 - Browser name, major and minor version

3 - Browser name, major version, minor version to two decimal places

2 - Browser name, major and minor versions and sub-version

1 - Browser name, version and machine type if possible

0 - All information (left unchanged).

-g NUM GroupDomains. Automatically group sites by domain. The grouping level specified by
num can be thought of as 'the number of dots' to display in the grouping. The
default value of 0 disables any domain grouping.

Hide Options

-a NAME
HideAgent. Hide user agents matching name.

-r NAME
HideReferrer. Hide referrer matching name.

-s NAME
HideSite. Hide site matching name.

-X NAME
HideAllSites. Hide all individual sites (only display groups).

-u NAME
HideURL. Hide URL matching name.

Table size options

-A --top_agents=NUM
TopAgents. Display the top num user agents table.

-R --top_refers=NUM
TopReferrers. Display the top num referrers table.

-S --top_sites=NUM
TopSites. Display the top num sites table.

-U --top_urls=NUM
TopURLs. Display the top num URL's table.

-C --top_countries=NUM
TopCountries. Display the top num countries table.

-e --top_entry=NUM
TopEntry. Display the top num entry pages table.

-E --top_exit=NUM
TopExit. Display the top num exit pages table.

Other Options

--use_geoip
Enables the use of the Maxmind GeoIP capability for more accurate detection of
countries.

NOTE! Do not enable GeoIP if you analyse files that have had the IP Address
translated to a Fully Qualified Host Name. Use either raw IP Addresses and GeoIP,
or Names and disable GeoIP. ie. Don't use GeoIP AND DNShistory.

--match_counts
Display the various Group/Hide etc Match Counts. This option is ideal for
optimisation of the awffull.conf file. Just be careful with optimising Agents in
particular, as the order is typically important.

CONFIGURATION FILES

See the awffull.conf(5) man page for complete details of all configuration options.

Configuration files are standard ASCII(7) text files that may be created or edited using
any standard editor.

Blank lines and lines that begin with a pound sign ('#') are ignored.

Any other lines are considered to be configuration lines, and have the form ‘Keyword
Value’, where the ‘Keyword’ is one of the currently available configuration keywords (see
awffull.conf(5)), and ‘Value’ is the value to assign to that particular option.

Any text found after the keyword up to the end of the line is considered the keyword's
value, so you should not include anything after the actual value on the line that is not
actually part of the value being assigned. The file sample.conf provided with the
distribution contains lots of useful documentation and examples as well.

Certain "Keywords" will accept a 2nd value. In those situations, the first value may be
enclosed in double quotes (") to allow for whitespace.

Use awffull online using onworks.net services