wulfstat - Online in the Cloud

PROGRAM:

NAME

wulfstat - A simple project template

SYNOPSIS

wulfstat [-h] [-v] [-t display_type] [-d delay] [-c count]
[-f /path/to/wulfhosts] [-l]

WULFSTAT OPTIONS

-h shows help (command synopsis).
-v makes execution verbose for debugging or the bored.
-t display_type selects display type from list below
-d delay (in seconds) selects update loop delay
-c count causes it to output only count times and exit (for debugging)
-f /path/to/wulfhosts to use a particular wulfhosts file
-l show localhost only (use no wulfhosts file from any location)

DESCRIPTION

wulfstat is a simple yet powerful tty (ncurses) based cluster monitoring tool. It
requires xmlsysd (running on each system to be monitored) to efficiently provide it with
system and proc-derived information that is processed and provided to the user in one of
several user-selectable display formats. With it a user can monitor things across en
entire beowulf, cluster, or workstation LAN systems descriptors such as load average,
memory consumption, swap, page, and interrupt activity and network loads or can even
retrieve and display such mundane information is CPU make and base clock, system time,
uptime or other potentially useful but slowly varying system descriptors. The information
presented is updated regularly after a user-selectable delay. The tool allows displays to
be selected or changed while the application is running, and more hosts or nodes can be
displayed than will easily fit on a single e.g. xterm by paging through them with key-
based commands.

WULFHOST

To run wulfhost as anything but a monitor of the local host one REQUIRES a wulfhost file.
wulfstat run with no viable wulfhost file defaults to a localhost connection. A localhost
connection can also be forced (overriding the search for a wulfhost file) with the -l
command line argument.

The wulfhost file tells wulfstat where to to connect to xmlsysd's. It consists of any mix
of the following xml discriptors:

<?xml version="1.0"?>

<wulfstat>

<root/>
<user>rgb</user>
<task>On_spin3d</task>

<host>
<name>ganesh</name>
</host>

<host>
<ip>192.168.1.132</ip>
<port>7887</port>
<host>

<host>
<name>lucifer</name>
<ip>192.168.1.131</ip>
<port>7887</port>
</host>

<hostrange>
<hostfmt>g%02d</hostfmt>
<imin>1</imin>
<imax>15</imax>
<port>7887</port>
</hostrange>

<iprange>
<ipmin>152.3.182.193</ipmin>
<ipmax>152.3.182.200</ipmax>
<port>7887</port>
</iprange>

</wulfstat>

From this example, one sees that the <host></host> tag defines a host to connect to.
Within this tag, the host can be specified by the <name></name> tag (which can contain any
name resolvable by gethostbyname()) or the <ip></ip> tag, most commonly used for hosts in
a cluster that haven't been named. In addition, for each host one can specify a
<port></port> if one for any reason is running the xmlsysd on a different port than its
installation default.

This information can easily be overspecified. In most cases, for example, it is better to
just use the default port (7887) and let local hostname ip address lookup take care of
determining the interface IP number. Note that xml doesn't care how the tags are laid out
as long as they are nested correctly, and that there can be more than one <host>,
<hostrange>, or <iprange> tagset in a wulfhosts to specify the simultaneous monitoring of
any mix of hosts, clusters, lans.

Note also that xml DOES preserve whitespace, so

<host><name>b0 </name></host>

is NOT the same is

<host><name>b0</name></host>

and would likely not work correctly. If you do enter port, name, and ip explicitly and
incorrectly or inconsistently, be prepared for odd behavior.

The <hostrange> is hopefully self explanatory. It can be used to rapidly define an entire
cluster on the basis of a systematic ordering of hostname. The contents of the <hostfmt>
tag should be a SIMPLE printf-format string for a presumed integer that will be iterated
from <imin> to <imax> in steps of one. In this way a single xml tag can define an entire
cluster e.g. g01-g15.

The <iprange> is similar, except that it uses ip number directly in <ipmin> and <ipmax>.
Use caution -- in almost all cases the first three tuples in the ip number should be the
SAME in <ipmin> and <ipmax>. This option is provided in case the hosts don't have a well-
defined and published hostname and are accessible only by e.g. dhcp-assigned ip number in
any event.

All forms of defining a host or list of hosts permit an optional <port> to be assigned to
override xmlsysd's installation default of 7887.

wulfstat will connect to these hosts as fast as it can in a parallel thread, and then will
periodically attempt to REconnect to any hosts that might be down or that might go down
while wulfstat is running. wulfstat itself is thus moderately robust against cluster node
state changes, although this is an area still being debugged (1/9/03).

wulfstat looks for a useable wulfhosts file in many places (in order):

NO wulfhosts file (localhost only) via the -l flag;
where it is directed by the -f /path/to/wulfhosts command line option;
in ./wulfhosts (in the current working directory);
in $HOME/.wulfhosts (in the home directory, note the ".");
in the file pointed to by environment variable WULFHOSTS;
in /etc/wulfhosts;

and if it can find none of them it tries to come up on localhost.

Note that any hosts that do not resolve are displayed but marked unknown. Any hosts that
resolve but that cannot accept a connection (which could mean that no daemon is installed
or running, the daemon has more connections than the number permitted in e.g.
/etc/xinetd.d/xmlsysd, or that the host is down) are marked down.

DISPLAY TYPES

The following display types are supported by wulfstat:

0 - load and status only (default), a very useful display for cluster
users
1 - stat -- information and rates primary derived from /proc/stat
2 - memory only (similar to running "free" on each host)
3 - network rates
4 - time displays system clocks, uptime, cpu type and clock
5 - "user" interface for monitoring distributed tasks by username or
taskname (experimental).

Each of these interfaces can also be reached interactively within wulfstat by typing the
first letter as shown on the menu.

INTERACTIVE COMMANDS

wulfstat also permits things like delay, display type and quit to be selected with single
keystrokes when the mouse focus is in the window. Single-key commands it recognizes
include:

q(uit)

+/- increments or decrements the delay clock, zero is the minimum permitted value and
basically spins wulfstat as fast as it can spin, which is useful for creating a modest
"load" on cpu or network so you can see wulfstat work.

PgUp displays the next page of hosts
PgDn displays the last page of hosts
UpArrow scrolls the display up
DnArrow scrolls the display down

in a circular queue. Using these keys a user can rapidly cycle through quite large
clusters.

l(oad) selects the cpu load display (likely the most useful)
m(emory) selects the memory display
n(etwork) selects the network load display
t(ime) selects the time (and cpu info) display
u(ser) selects the user task interface (experimental).

Finally, if there is enough room (it is recommended that wulfstat be used with at least
35x80 xterms or larger) a debug window will rarely display interesting messages. Starting
wulfstat with a -v option will increase the noisiness in this window, but isn't
recommended unless you are debugging the wulfstat source for some reason.

DEBUGGING

To help debug wulfstat (or problems you might have with wulfhosts), note the table of
verbose/debugging values that is printed as part of its Usage (-h flag). This yields
anything from a simple trace of a particular subsystem such as connect_hosts() to
everything the program does. To limit the output, one can also use the -c count flag to
only display a single cycle. It is a good idea to pipe stderr into a logfile separately
so that the display output is unaltered. The logfile can be examined later or mailed back
to me for analysis.

An example of this might be:

wulfstat -l -c 1 -v 10 2>connect_hosts.log

to trace what wulfstat does connecting to localhost.

Use wulfstat online using onworks.net services