This is the command mailcross that can be run in the OnWorks free hosting provider using one of our multiple free online workstations such as Ubuntu Online, Fedora Online, Windows online emulator or MAC OS online emulator
mailcross - a cross-validation simulator for use with dbacl.
mailcross command [ command_arguments ]
mailcross automates the task of cross-validating email filtering and classification
programs such as dbacl(1). Given a set of categorized documents, mailcross initiates
simulation runs to estimate the classification errors and thereby permits fine tuning of
the parameters of the classifier.
Cross-validation is a method which is widely used to compare the quality of classification
and learning algorithms, and as such permits rudimentary comparisons between those
classifiers which make use of dbacl(1) and bayesol(1), and other competing classifiers.
The mechanics of cross-validation are as follows: A set of pre-classified email messages
is first split into a number of roughly equal-sized subsets. For each subset, the filter
(by default, dbacl(1)) is used to classify each message within this subset, based upon
having learned the categories from the remaining subsets. The resulting classification
errors are then averaged over all subsets.
The results obtained by cross validation essentially do not depend upon the ordering of
the sample emails. Other methods (see mailtoe(1),mailfoot(1)) attempt to capture the
behaviour of classification errors over time.
mailcross uses the environment variables MAILCROSS_LEARNER and MAILCROSS_FILTER when
executing, which permits the cross-validation of arbitrary filters, provided these satisfy
the compatibility conditions stated in the ENVIRONMENT section below.
For convenience, mailcross implements a testsuite framework with predefined wrappers for
several open source classifiers. This permits the direct comparison of dbacl(1) with
competing classifiers on the same set of email samples. See the USAGE section below.
During preparation, mailcross builds a subdirectory named mailcross.d in the current
working directory. All needed calculations are performed inside this subdirectory.
mailcross returns 0 on success, 1 if a problem occurred.
Prepares a subdirectory named mailcross.d in the current working directory, and
populates it with empty subdirectories for exactly size subsets.
add category [FILE]...
Takes a set of emails from either FILE if specified, or STDIN, and associates them
with category. All emails are distributed randomly into the subdirectories of
mailcross.d for later use. For each category, this command can be repeated several
times, but should be executed at least once.
clean Deletes the directory mailcross.d and all its contents.
learn For every previously built subset of email messages, pre-learns all the categories
based on the contents of all the subsets except this one. The command_arguments
are passed to MAILCROSS_LEARNER.
run For every previously built subset of email messages, performs the classification
based upon the pre-learned categories associated with all but this subset. The
command_arguments are passed to MAILCROSS_FILTER.
Prints statistics for the latest cross-validation run.
review truecat predcat
Scans the last run statistics and extracts all the messages which belong to
category truecat but have been classified into category predcat. The extracted
messages are copied to the directory mailcross.d/review for perusal.
Shows a list of available filters/wrapper scripts which can be selected.
testsuite select [FILTER]...
Prepares the filter(s) named FILTER to be used for simulation. The filter name is
the name of a wrapper script located in the directory /usr/share/dbacl/testsuite.
Each filter has a rigid interface documented below, and the act of selecting it
copies it to the mailcross.d/filters directory. Only filters located there are used
in the simulations.
testsuite deselect [FILTER]...
Removes the named filter(s) from the directory mailcross.d/filters so that they are
not used in the simulation.
Invokes every selected filter on the datasets added previously, and calculates
Describes the scheduled simulations.
Shows the cross validation results for all filters. Only makes sense after the run
The normal usage pattern is the following: first, you should separate your email
collection into several categories (manually or otherwise). Each category should be
associated with one or more folders, but each folder should not contain more than one
category. Next, you should decide how many subsets to use, say 10. Note that too many
subsets will slow down the calculations rapidly. Now you can type
% mailcross prepare 10
Next, for every category, you must add every folder associated with this category. Suppose
you have three categories named spam, work, and play, which are associated with the mbox
files spam.mbox, work.mbox, and play.mbox respectively. You would type
% mailcross add spam spam.mbox
% mailcross add work work.mbox
% mailcross add play play.mbox
You can now perform as many simulations as desired. Every cross validation consists of a
learning, a running and a summarizing stage. These operations are performed on the
classifier specified in the MAILCROSS_FILTER and MAILCROSS_LEARNER variables. By setting
these variables appropriately, you can compare classification performance as you vary the
command line options of your classifier(s).
% mailcross learn
% mailcross run
% mailcross summarize
The testsuite commands are designed to simplify the above steps and allow comparison of a
wide range of email classifiers, including but not limited to dbacl. Classifiers are
supported through wrapper scripts, which are located in the /usr/share/dbacl/testsuite
The first stage when using the testsuite is deciding which classifiers to compare. You
can view a list of available wrappers by typing:
% mailcross testsuite list
Note that the wrapper scripts are NOT the actual email classifiers, which must be
installed separately by your system administrator or otherwise. Once this is done, you
can select one or more wrappers for the simulation by typing, for example:
% mailcross testsuite select dbaclA ifile
If some of the selected classifiers cannot be found on the system, they are not selected.
Note also that some wrappers can have hard-coded category names, e.g. if the classifier
only supports binary classification. Heed the warning messages.
It remains only to run the simulation. Beware, this can take a long time (several hours
depending on the classifier).
% mailcross testsuite run
% mailcross testsuite summarize
Once you are all done with simulations, you can delete the working files, log files etc.
% mailcross clean
The progress of the cross validation is written silently in various log files which are
located in the mailcross.d/log directory. Check these in case of problems.
mailcross testsuite takes care of learning and classifying your prepared email corpora for
each selected classifier. Since classifiers have widely varying interfaces, this is only
possible by wrapping those interfaces individually into a standard form which can be used
by mailcross testsuite.
Each wrapper script is a command line tool which accepts a single command followed by zero
or more optional arguments, in the standard form:
wrapper command [argument]...
Each wrapper script also makes use of STDIN and STDOUT in a well defined way. If no
behaviour is described, then no output or input should be used. The possible commands are
filter In this case, a single email is expected on STDIN, and a list of category filenames
is expected in $2, $3, etc. The script writes the category name corresponding to
the input email on STDOUT. No trailing newline is required or expected.
learn In this case, a standard mbox stream is expected on STDIN, while a suitable
category file name is expected in $2. No output is written to STDOUT.
clean In this case, a directory is expected in $2, which is examined for old database
information. If any old databases are found, they are purged or reset. No output is
written to STDOUT.
IN this case, a single line of text is written to STDOUT, describing the filter's
functionality. The line should be kept short to prevent line wrapping on a
In this case, a directory is expected in $2. The wrapper script first checks for
the existence of its associated classifier, and other prerequisites. If the check
is successful, then the wrapper is cloned into the supplied directory. A courtesy
notification should be given on STDOUT to express success or failure. It is also
permissible to give longer descriptions caveats.
toe Used by mailtoe(1).
foot Used by mailfoot(1).
Right after loading, mailcross reads the hidden file .mailcrossrc in the $HOME directory,
if it exists, so this would be a good place to define custom values for environment
This variable contains a shell command to be executed repeatedly during the running
stage. The command should accept an email message on STDIN and output a resulting
category name. It should also accept a list of category file names on the command
line. If undefined, mailcross uses the default value MAILCROSS_FILTER="dbacl -T
email -T xml -v" (and also magically adds the -c option before each category).
This variable contains a shell command to be executed repeatedly during the
learning stage. The command should accept a mbox type stream of emails on STDIN for
learning, and the file name of the category on the command line. If undefined,
mailcross uses the default value MAILCROSS_LEARNER="dbacl -H 19 -T email -T xml
This directory is exported for the benefit of wrapper scripts. Scripts which need
to create temporary files should place them a the location given in TEMPDIR.
The subdirectory mailcross.d can grow quite large. It contains a full copy of the training
corpora, as well as learning files for size times all the added categories, and various
Cross-validation is a widely used, but ad-hoc statistical procedure, completely unrelated
to Bayesian theory, and subject to controversy. Use this at your own risk.
The source code for the latest version of this program is available at the following
Use mailcross online using onworks.net services