sluice - Online in the Cloud

PROGRAM:

NAME

sluice - a tool to control data flow at a specified rate

SYNOPSIS

sluice [options]

DESCRIPTION

sluice reads input and outputs a specified data rate. It has various data rate controlling
mechanisms that can be tuned for specific use cases where necessary.

OPTIONS

sluice options are as follow:

-a append output to a file (used in conjunction with the -t 'tee' or -O options).
Instead of creating a new file or truncating an existing file, this option appends
data to the file.

-c delay
enables a constant delay time (in seconds) between writes. This option adjusts the
output buffer size to try and keep the data rate constant. The output buffer size
in this mode is initially set to the data rate × the delay.

This option is mutually exclusive to the -i option and implicitly enables the -o
overrun and -u underrun buffer management options to dynamically re-size the
read/write buffer to keep the data rate constant. By default this adjusts the
buffer based on the total amount of data transferred and the time to write this
(equivalent to the -s 0 turning mode). However, if the -s shift value is greater
than 0, then the new size is adjusted by the previous size right shifted by the
shift value.

-d discard data, do not copy it to stdout. This makes sluice act as a data sink.

-D mode
select the delay mode. The are various approaches to when to perform the data rate
delays. The default is to perform the read, then write and finally the delay for
each iteration. However, the -D option allows one to select the delay mode as
follows:

Mode Delay strategy Delay Duration
0 Read, Write, Delay (default) 1 × delay time
1 Delay, Read, Write 1 × delay time
2 Read, Delay, Write 1 × delay time
3 Delay, Read, Delay, Write 2 × 1/2 delay time
4 Read, Delay, Write, Delay 2 × 1/2 delay time
5 Delay, Read, Delay, Write, Delay 3 × 1/3 delay time

Note that modes 3 and 4 perform two delays each comprising of 1/2 the delay time
and mode 5 performs 3 delays each comprising of 1/3 the delay time.

Modes 1, 3, 5 maybe considered as not entirely accurate in terms of the total run
duration. In these modes an extraneous delay occurs before the final end-of-file
empty read is performed.

-e ignore read errors. The failed read is replaced by zeros.

-f freq
specify the frequency of -v verbose statistics updates. The default is 1/4 of a
second. Note that sluice will try to emit updates close to the requested frequency,
however, if the read/write rate is less than the frequency then the updates occur
only at the read/write rate.

-h show help

-i size
specify the read/write size in bytes. The K, M, G, T and P suffixes allow one to
specify size in Kilobytes, Megabytes, Gigabytes, Terabytes and Petabytes
respectively. This option is mutually exclusive to the -c option.

In this mode, the delays between writes are used to control the data rate. By
default the delay is based on the total amount of data transferred and the time
taken to write this. This is equivalent to the -s 0 tuning mode. However, if the
-s shift value is greater than 0, then the new delay is adjusted by the previous
delay right shifted by the shift value.

A special hybrid rate control mode can be invoked by also using the -o overflow and
-u underflow options to also enable dynamic re-sizing of the read/write buffer. By
default this adjusts the buffer based on the total amount of data transferred and
the time to write this (equivalent to the -s 0 turning mode). However, if the -s
shift value is greater than 0, then the new size is adjusted by the previous size
right shifted by the shift value.

-I file
read input from file rather than from stdin.

-m size
specify amount of data to process, the default size is in bytes, but the K, M, G, T
and P suffixes can specify size in Kilobytes, Megabytes, Gigabytes, Terabytes and
Petabytes respectively. If this size is less than the write size, then the write
size is truncated to be the -m size.

-n no rate control. This is just a straight data copy much like cat and all data rate
controls cannot be used. Combined with the -v and -S options one can observe the
data rates of the copy.

-o detect overrun and re-size read/write buffer size to try and stop overrun. This
will shrink the buffer each time consecutive overruns are detected. See the -s
option for details of the size re-adjustment mechanism.

-O file
send output to file, equivalent to -dt file

-p enable verbose stats showing % progress and ETA information. This is only valid
using the -I or -m option and the if file size is non-zero. See the -v option for
more details.

-P pidfile
write the process ID of sluice in file pidfile. The file is removed when sluice
exits.

-r rate
specify the data rate in bytes per second. The K, M, G and T suffixes can specify
the rate in Kilobytes/sec, Megabytes/sec, Gigabytes/sec and Terabytes/sec
respectively. This option must always be provided except when the -n option is
used.

-R do not read from stdin, instead read random data from /dev/urandom.

-s shift
modify the rate adjustment shift. This is a data rate tuning scaling factor used by
the -r, -c, -o and - options.

For the -r option, the delay between each write is controlled by modifying the
previous delay by adding or subtracting the previous delay right shifted by this
shift value. The larger the shift value the longer it takes to adjust up/down to
the specified rate. The smaller the shift value the quicker it takes to reach the
optimal delay, however, this can result in a highly fluctuating rates at the the
beginning because the delay varies by a large amount causing large overruns and
underruns. A shift value of 3 works well for most fast rates.

For the -c, -o and -u options, the size of the buffer is modified by adding or
subtracting the previous size shifted by the shift value. Again, a shift value of 3
works well for most fast rates.

If the shift value is set to 0, then the shift rate adjustment tuning mechanism is
explicitly turned off and data rates are adjusted based on the total amount of data
transferred and the time to write this.

Small -s shift values of 1 and 2 can cause rapid oscillations before data rate
damping fully kicks into action. The value of -s 0 (the default) is recommended for
accurate low-speed data transfers.

-S print various performance and buffering statistics to stderr when end of file is
reached.

-t file
tee output to the specified file. Output is written to both stdout and to the named
file. By default, the file will be created if it does not exist or re-written if it
already exists. Use the -a option to append to an existing file.

-T t stop slice test after t seconds. One can also specify the units of time in seconds,
minutes, hours, days or years with the suffix s, m, h, d or y.

-u detect underrun and re-size read/write buffer size to try and stop underrun. This
will expand the buffer each time consecutive underruns are detected. The buffer
will not be expanded any more than 4MB in size. See the -s option for details of
the size re-adjustment mechanism.

-v write verbose statistics to stderr. By default, this will display the current data
rate, the last data rate adjusment ('-' = underrun, '+' = overrun), total bytes
transferred, duration and the current buffer size.

With the -p option, the progess statistics are displayed. This will display the
current data rate, total bytes transferred, duration, percentage complete so far
and the estimated time to completion. Note that the estimation is available using
the -I and -m options and if the file size is non-zero.

-V print version information to standard out and exit successfully.

-w warn if a long burst of continuous data rate underrun occurs, the warning is issued
just once. To overcome the underrun increase the -i read/write buffer size or use
the -u option to auto-expand the read/write buffer. Too many underruns implies
that too small a buffer or not enough CPU is available to keep up with the required
data rate.

-z do not read from stdin, instead generate a stream of zeros (equivalent to reading
from /dev/zero).

SIGUSR1 SIGINFO
Sending SIGUSR1 (or SIGINFO on BSD systems) will toggle the verbose data rate mode
on/off.

SIGUSR2
Toggle underrun/overrun (-u, -o) options on/off.

NOTES

If neither -i or -c options are used, then sluice defaults to using a write buffer size of
1/32 of the data rate and bounded between the limits of 1 byte and 64MB. Sluice will try
to keep the data rate steady by adjusting the delay between writes. To tune this, see the
-s option.

EXAMPLES

Read /dev/zero and write in 4K sizes at the rate of 1MB/sec to the file 'example.dat'
cat /dev/zero | sluice -i 4K -r 1M > example.dat

Read 32MB from /dev/zero and write at the rate of 64K/sec to stdout with feedback on
duration and ETA on stderr using 4K buffer writes and a tuning shift of 4.
cat /dev/zero | sluice -r 64K -vp -m 32M -i 4K -s 4

Generate a stream of zeros and write at a rate of 1MB/sec to a fifo named 'myfifo' with
underrun and overrun buffer management
sluice -z -u -o -r 1MB -O myfifo

Write random data at 5MB per second to the file 'myfile' doing a write every 0.1 seconds
sluice -R -r 5M -c 0.1 > myfile

Write zeros to the file 'example-file' in 64K chunks and measure write rate as a crude
throughput test
sluice -nzSv -f 0.1 -i 64K > example-file

Read data from somehost.com on port 1234 at a rate of 2MB per second and discard the data,
e.g. this is a constant rate data sink.
nc somehost.com 1234 | sluice -d -r 2MB -i 8K

EXIT STATUS

Sluice sets the exit status as follows:

Status Decription
0 Exited successfully.
1 Invalid or out of range option provided.
2 File open error.
3 Sleep error.
4 Failed to get time of day.
5 Signal handler setup error.
6 Read error (file or stdin).
7 Write error (file or stdout).
8 Buffer allocation failed.

Use sluice online using onworks.net services