This is the command fpsync that can be run in the OnWorks free hosting provider using one of our multiple free online workstations such as Ubuntu Online, Fedora Online, Windows online emulator or MAC OS online emulator
PROGRAM:
NAME
fpsync — Synchronize directories in parallel using fpart and rsync.
SYNOPSIS
fpsync [-h] [-v] [-n jobs] [-f files] [-s size] [-w wrks] [-d shdir] [-t tmpdir]
[-r jobname] [-o rsyncopts] [-O fpartopts] [-S] src_dir/ dst_dir/
DESCRIPTION
The fpsync tool synchronizes directories in parallel using fpart(1) and rsync(1). It
computes subsets of src_dir/ and spawns rsync(1) jobs to synchronize them to dst_dir/.
Synchronization jobs can be executed either locally or remotely (using SSH workers, see
option -w) and are executed on-the-fly while filesystem crawling goes on. This makes fpsync
a good tool for migrating large filesystems.
OPTIONS
-h Print help
-v Verbose mode. Can be be specified several times to increase verbosity level.
-n jobs
Start jobs concurrent sync jobs (either locally or remotely, see below). Default: 2
-f files
Transfer at most files files per sync job. Default: 2000
-s size
Transfer at most size bytes per sync job.
Default: 4294967296 (4 GB)
-w wrks
Use remote SSH wrks to synchronize files. Synchronization jobs are executed locally
when this option is not set. wrks is a space-separated list of login@machine
connection strings and can be specified several times. You must be allowed to
connect to those machines using a SSH key to avoid user interaction.
-d shdir
Set fpsync shared directory to shdir. This option is mandatory when using SSH
workers and set by default to tmpdir when running locally. The specified directory
must be an absolute path ; it will be used to handle communications with SSH hosts
(sharing partitions and log files) and, as a consequence, must be made available to
all participating hosts (e.g. through a r/w NFS mount), including the master one
running fpsync.
-t tmpdir
Set fpsync temporary directory to tmpdir. This directory remains local and does not
need to be shared amongst SSH workers when using the -w option. Default: /tmp/fpsync
-r jobname
Resume job jobname and restart synchronizing remaining partitions from a previous
run. jobname can be obtained using verbose mode (see option -v). Note that
filesystem crawling is skipped when resuming a previous run. As a consequence,
options -f, -s, -o, -O, -S, src_dir/, and dst_dir/ are ignored.
-o rsyncopts
Override default rsync(1) options with rsyncopts. Use this option with care as
certain options are incompatible with a parallel usage (e.g. --delete). Default:
-av --numeric-ids
-O fpartopts
Override default fpart(1) options with fpartopts.
Default: -x .zfs -x .snapshot* -x .ckpt
-S Sudo mode. Use sudo(8) for filesystem crawling and synchronizations.
src_dir/
Source directory. It must be absolute and available on all participating hosts
(including the master one, running fpsync).
dst_dir/
Destination directory. It must be absolute and available on all participating
workers.
RUNNING FPSYNC
Each fpsync run generates a unique jobname, which is displayed in verbose mode (see option
-v) and within log files. You can use that jobname to resume a previous run (see option
-r). fpsync will then restart synchronizing data from the parts that were being synchonized
at the time it stopped.
This unique feature gives the administrator the ability to stop fpsync and restart it later,
without having to restart the whole filesystem crawling and synchronization process. Note
that resuming is only possible when filesystem crawling step has finished.
During synchronization, you can press CTRL-C to interrupt the process. The first CTRL-C
prevents new synchronizations from being submitted and the process will wait for current
synchronizations to be finished before exiting. If you press CTRL-C again, current
synchronizations will be killed and fpsync will exit immediately.
On certain systems, CTRL-T can be pressed to get the status of current and remaining parts
to be synchronized. This can also be achieved by sending a SIGINFO to the fpsync process.
Whether you use verbose mode or not, everything is logged within shdir/log/.
EXAMPLES
Here are some examples:
fpsync -n 4 /usr/src/ /var/src/
Synchronizes /usr/src/ to /var/src/ using 4 local jobs.
fpsync -n 2 -w login@machine1 -w login@machine2 -d /mnt/fpsync /mnt/src/ /mnt/dst/
Synchronizes /mnt/src/ to /mnt/dst/ using 2 concurrent jobs executed remotely on 2
SSH workers (machine1 and machine2). The shared directory is set to /mnt/fpsync and
mounted on the machine running fpsync, as well as on machine1 and machine2. The
source directory (/mnt/src/) is also available on those 3 machines, while the
destination directory (/mnt/dst/) is mounted on SSH workers only (machine1 and
machine2).
LIMITS
Parallelizing rsync(1) makes several options not usable, such as --delete. If your source
directory is live while fpsync is running, you will have to delete extra files from
destination directory. This is usually done by using a final -offline- rsync(1) pass that
will use this option.
fpsync enqueues synchronization jobs on disk, within the tmpdir/queue directory. Be careful
to host this queue on a filesystem that can handle fine-grained mtime timestamps (i.e. with
a sub-second precision) if you want the queue to be processed in order when fpart(1)
generates several jobs per second. On FreeBSD, VFS(9) timestamps' precision can be tuned
using the 'vfs.timestamp_precision' sysctl. See vfs_timestamp(9).
Use fpsync online using onworks.net services
