dss

The dyadic snapshot scheduler


[README] [NEWS] [Download] [INSTALL] [License] [Contact] [Man page]

README

dss creates hardlink-based snapshots of a given directory on a remote or local host using rsync’s link-dest feature.

dss is admin friendly: It is easy to configure and needs little attention once configured to run in daemon mode. It keeps track of the available disk space and removes snapshots if disk space becomes sparse or snapshots become older than the specified time. Also, due to the hardlink-based approach, there is only one type of backup. Hence no full, incremental or differential backups need to be configured, and there is no database to maintain.

dss is also user-friendly because users can browse the snapshot directories without admin intervention and see the contents of the file system at the various times a snapshot was taken. Each snapshot looks like a full backup, so users can easily restore accidentally removed files by using their favorite file browser to simply copy files from the snapshot directory back to the live system.

dss gives your data an additional level of security besides the usual tape-based backups: If the file server goes down and all data is lost you can simply use the most recent snapshot as an immediate replacement – no need for a restore from tape that takes days to complete.

Snapshot pruning takes place in a dyadic fashion: Many recent snapshots are available, but the number of snapshots per time interval decreases exponentially. For example, one can configure dss so that it keeps 16 snapshots not older than one week, 8 snapshots between one and two weeks old, 4 snapshots between two and three weeks old, and so on.


NEWS


1.0.0 (2017-11-19)

After almost ten years of development, dss has finally reached version 1.0.0. Many new features have been implemented for this release. The list below contains the user-visible changes since the previous 0.1.7 release. For the complete changelog consult the git log.

The biggest user-visible change is that the former modes of operation have been replaced by subcommands. For example, the former -R option to start dss in “run” mode corresponds to the “run” subcommand of version 1.0.0. Each subcommand has its own set of options. Global options and options for subcommands are split with the usual “–” separator. For example, to specify the global –loglevel option and the run-specific –daemon option, use dss –loglevel 2 – run –daemon.

Although the new version has different syntax rules, the on-disk snapshot format remains the same. No conversion is required to continue existing snapshot directories created with older dss versions.

The 0.1.x series will still be maintained for a while as the “maint” branch of the git repository, but this branch will only receive bug fixes.


0.1.7 (2017-04-17)


0.1.6 (2015-08-05)


0.1.5 (2014-01-14)


0.1.4 (2010-11-08)

This version of dss contains some new features, many improvements of existing features and several bug fixes.

Support for resuming snapshot creation after restart has been added and snapshot removal is deferred until at least one current snapshot exists. The internal handling of the various hooks has been simplified and many error messages have been clarified. On SIGHUP, dss now writes its configuration and internal state to the log file.


0.1.3 (2009-06-06)

You can now specify pre-rm/post-rm-hooks that are executed whenever a snapshot is deleted. This release adds better compatibility with rsync 3.0, and avoids busy loops when automatically restarting the rsync process.


0.1.2 (2009-03-05)

This release includes the reason why a snapshot gets removed in the log message. It will never remove the snapshot that is currently being created. It will really pass the full path to the last complete snapshot in the post_create_hook.


0.1.1 (2008-11-13)

This release prevents busy loops on rsync exit code 13. It ignores any snapshot directory with creation time >= completion time. It opens /dev/null for reading and writing when executing rsync. It shows human readable snapshot creation duration when listing snapshots. It restarts the rsync process if it returned with exit code 13.


0.1.0 (2008-10-10)

Initial public release.


Download

Only the source code is available for download. Use git to clone the dss repository by executing

git clone git://git.tuebingen.mpg.de/dss

or grab the tarball of the current master branch. If you prefer to download the tarball of the latest release, select the corresponding snapshot link on the dss gitweb page


INSTALL

dss is known to compile on Linux, FreeBSD and NetBSD. However, it is run-tested only on Linux.

Note that lopsub is required to compile dss.

Type

    make

in the dss source directory to build the dss executable and the man page. If lopsub is installed in a non-standard path, you may need to run make as follows:

    make CPPFLAGS=-I$HOME/lopsub/include LDFLAGS=-L$HOME/lopsub/lib

Then type

    sudo make install

to install in /usr/local, or

    make install PREFIX=/somewhere/else

to install in /somewhere/else.

Also make sure that rsync is installed on your system. Version 2.6.1 or newer is required.

Examples:

Suppose you’d like to create snapshots of the existing directory

    /foo/bar

in the directory

    /baz/qux.

Create the config file

    ~/.dssrc

that contains the values for the source and the destination directories as follows:

    echo 'source-dir "/foo/bar"' > ~/.dssrc
    echo 'dest-dir "/baz/qux"' >> ~/.dssrc

Then execute the commands

    mkdir /baz/qux
    dss run

In order to print the list of all snapshots created so far, use

    dss ls

Yes, it’s really that easy.

The second example involves a slightly more sophisticated config file. It instructs dss to exclude everything which matches at least one pattern of the given exclude file, prevents rsync from crossing file system boundaries and increases the number of snapshots.

    source-dir "/foo/bar"
    dest-dir "/baz/qux"
    # exclude files matching patterns in /etc/dss.exclude
    rsync-option "--exclude-from=/etc/dss.exclude"
    # don't cross filesystem boundaries
    rsync-option "--one-file-system"
    # maintain 2^6 - 1 = 63 snapshots
    num-intervals "6"

The /etc/dss.exclude file could look like this (see rsync(1) for more examples)

     - /proc
     - /**/tmp/

Note that dss supports many more features and config options such as taking snapshots from remote hosts and several hooks that are executed on certain events, for example whenever a snapshot was created successfully. Try

    dss -h

for an overview of all supported command line options or

    dss --detailed-help

for the full help text.


License

dss is open source software, licensed under the GNU General Public License, Version 2.


Contact

Email: André Noll, maan@tuebingen.mpg.de, Homepage: http://people.tuebingen.mpg.de/maan/

Comments and bug reports are welcome. Please provide enough info such as the version of dss you are using and relevant parts of the logs. Including the string [dss] in the subject line is also a good idea.

Man page

dss

NAME

dss - the dyadic snapshot scheduler

SYNOPSIS

dss [global-options...] [--] [<subcommand> [subcommand-options...]]

DESCRIPTION

dss creates hardlink-based snapshots of a given directory on a remote or local host using rsync’s link-dest feature.

OPTIONS

General options
-h
, --help

print help and exit

--detailed-help

print help, including all details, and exit

-V, --version

print version and exit

-c, --config-file=<path>

use alternative config file (default: ~/.dssrc)

Options may be given at the command line or in the configuration file. As usual, if an option is given both at the command line and in the configuration file, the command line option takes precedence.

However, there is one exception to this rule: The run subcommand re-reads the configuration file when it receives the HUP signal. In this case the options in the config file override any options that were previously given at the command line. This allows changing the configuration of a running dss process by sending SIGHUP.

-l, --loglevel=<level>

set loglevel (0-6)

default: 4

Lower values mean more verbose logging.

-n, --dry-run

only print what would be done

This flag does not make sense for all subcommands. The run subcommand refuses to start if this option was given while the ls subcommand silently ignores the flag.

--source-dir=<dirname>

the remote directory to snapshot

The directory on the remote host from which snapshots are taken. Of course, the user specified as --remote-user must have read access to this directory.

This option is mandatory for the create and run subcommands: It must be given at the command line or in the config file.

--dest-dir=<dirname>

where snapshots are stored

The destination directory on the local host where snapshots will be written. This must be writable by the user who runs dss.

This option is mandatory for all subcommands except kill.

--mountpoint

abort if destination directory is not a mountpoint

This option checks whether a file system is mounted on the directory specified as the argument to --dest-dir. Operation proceeds only if this is the case. Otherwise dss exits unsuccessfully without performing any action. Use this option to prevent snapshot creation if the snapshot file system is not mounted.

This option is silently ignored for subcommands which do not depend on the destination directory.

Controlling how rsync is run
These options are only relevant to the run and the create subcommands.
-H
, --remote-host=<hostname>

host to take snapshots from

default: localhost

If this option is given and its value differs from the local host, then rsync uses ssh. Make sure there is no password needed for the ssh connection. To achieve that, use public key authentication for ssh and, if needed, set the remote user name by using the --remote-user option.

-U, --remote-user=<username>

Remote user name (default: current user)

Set this if the user that runs dss is different from the user on the remote host.

--checksum=<permille>

run rsync with --checksum occasionally

default: 0

If a file on the backup becomes corrupt in a way that file size and modification time still match the original file, rsync will not consider the file for transfer ("quick check"). Hence the corruption stays on the backup until the file is modified on the source. The --checksum option of rsync disables the quick check and compares the contents of each file, fixing such corruptions. Since computing the checksums adds a significant slowdown due to a lot of disk I/O, the option is not enabled by default.

The argument to the --checksum option of dss is a number between 0 and 1000, inclusively, which determines the probability of adding --checksum to the rsync options each time a snapshot is created. The default value zero means to never add the option. The value 100 will create every tenth snapshot (on average) using checksums, and the value 1000 will always pass --checksum to rsync.

-O, --rsync-option=<option>

further rsync options

This option may be given multiple times. The given argument is passed verbatim to the rsync command. Note that in order to use rsync options that require an argument, you have to specify the option and its argument as separate --rsync-options, like this:

--rsync-option --exclude --rsync-option /proc

Fine tuning the number of snapshots per time unit
Snapshot aging is implemented in terms of intervals. There are two command line options related to intervals: the duration u of a unit interval and the number of unit intervals, denoted n below.

dss removes snapshots older than n times u and tries to keep 2^(n - k - 1) snapshots in interval k, where the interval number k counts from zero to n - 1, with zero being the most recent unit interval.

Hence the oldest snapshot will at most be u * n days old (4 days * 5 intervals = 20 days, if default values are used). Moreover, there are at most 2^n - 1 snapshots in total (2^5 - 1 = 31 by default). Note that for this to work out your system must be fast enough to create at least 2^(n - 1) snapshots per unit interval (16 snapshots in 4 days = one snapshot in 6 hours), because this is the number of snapshots in interval zero.
-u
, --unit-interval=<days>

the duration of a unit interval

default: 4

Increasing this number instructs dss to create fewer snapshots per time unit while the number of snapshots to keep stays the same.

-n, --num-intervals=<num>

the number of unit intervals

default: 5

Increasing this number by one doubles the total number of snapshots.

Commands to be run on certain events
All hooks default to "true". That is, the true(1) utility (which always returns with exit code zero) is executed if the hook command is not specified.
-r
, --pre-create-hook=<command>

executed before a snapshot is created

default: true

This command is executed before dss runs rsync to create a new snapshot. If the command returns with a non-zero exit status, no snapshot will be created and the operation is retried later.

For example, the command could execute a script that checks whether all snapshot-related file systems are mounted.

Another possible application of the pre-create hook is to return non-zero during office hours in order to not slow down the file systems by taking snapshots.

--post-create-hook=<command>

executed after a snapshot has been created

default: true

This is only executed if a snapshot has successfully been created. The full path of the newly created snapshot is passed to the hook as the first argument. The exit code of this hook is ignored.

For instance this hook could count the number of files per user and/or compute disk usage patterns to be stored in a database for further analysis.

--pre-remove-hook=<command>

executed before a snapshot is removed

default: true

The full path to the snapshot which is about to be removed is passed to the command as the first argument. If the command returns with a non-zero exit status, the snapshot is not going to be removed and the operation is retried later.

For example, one could execute a script that checks whether the snapshot to be deleted is currently used by another process, e.g. by a tape-based backup system that runs concurrently to dss.

Another possible application of this is to record disk-usage patterns before and after snapshot removal.

--post-remove-hook=<command>

executed after snapshot removal

default: true

As for the pre-remove hook, the full path of the removed snapshot is passed to the hook as the first argument. The exit code of this hook is ignored.

--exit-hook=<command>

executed before the run command exits

default: true

This hook is only relevant to the run subcommand. It is executed just before dss terminates. The reason for termination is passed as the first argument.

One possible application for this hook is to send email to the system administrator to let her know that no more snapshots are going to be created.

Disk space monitoring
The options of this section control the aggressiveness of snapshot removal. That is, they define under which circumstances existing snapshots are removed. These options are only relevant to the run and the prune subcommands.
-m
, --min-free-mb=<megabytes>

minimal amount of free disk space

default: 100

If disk space on the file system containing the destination directory gets low, the run subcommand suspends the currently running rsync process and starts to remove snapshots in order to free disk space. This option specifies the minimal amount of free disk space. If less than the given number of megabytes is available, snapshots are being deleted. See also the --min_free_percent and the min-free-percent-inodes options below.

A value of zero deactivates this check.

-p, --min-free-percent=<percent>

minimal percentage of free disk space

default: 2

This is like --min-free-mb but the amount of free disk space is specified as a percentage. It is not recommended to set both --min-free-mb and --min-free-percent to zero as this will cause your file system to fill up quickly.

-i, --min-free-percent-inodes=<percent>

minimal percent of free inodes

default: 0

The minimum amount of free inodes on the file system containing the destination dir. If the percentage of free inodes drops below the given value, snapshot removal kicks in like in case of low disk space.

The number of free inodes is determined from the f_ffree field of the statvfs structure. However, some file systems set this field to zero, indicating that the number of inodes is basically unlimited. Moreover it is not possible to reliably detect whether this is the case. Therefore this feature is disabled by default. It’s safe to enable it for ext2/ext3/ext4 file systems on linux though.

A value of zero (the default) deactivates this check.

-k, --keep-redundant

prune by disk space only

By default, redundant and outdated snapshots are removed automatically to keep the number of snapshots in harmony with the configured policy. If this flag is given, dss removes such snapshots only if disk space or number of free inodes becomes low.

--min-complete[=<num>]

minimal number of complete snapshots to keep

default: 1

This option is only relevant if snapshots must be deleted because disk space gets low.

dss refuses to remove old snapshots if there are fewer complete snapshots left than the given number. The default value of one guarantees that at least one complete snapshot is available at all times.

If only <num> complete snapshots are left, and there is not enough disk space available for another snapshot, the program terminates with a "No space left on device" error.

SUBCOMMANDS

dss supports the subcommands described below. If no subcommand is given, the list of available subcommands is shown and the program terminates successfully without performing any further action.

run - start creating and pruning snapshots
Usage: run [--daemon] [--logfile=<path>] [--max-rsync-errors=<count>]

This is the main mode of operation. Snapshots are created in an endless loop as needed and pruned automatically. The loop only terminates on fatal errors or if a terminating signal was received. See also the --exit-hook option.
-d
, --daemon

run as background daemon

If this option is given, the dss command detaches from the console and continues to run in the background. It is not possible to let a daemonized process re-attach to the console by editing the config file and sending SIGHUP. However, log output may be redirected to a different file in this way.

See --logfile.

-l, --logfile=<path>

where to write log output

default: /dev/null

This option is only honored if --daemon is given, in which case log messages go to the given file. Otherwise the option is silently ignored and log output is written to stderr.

--max-rsync-errors=<count>

terminate after this many rsync failures

default: 10

If the rsync process exits with a fatal error, dss restarts the command in the hope that the problem is transient and subsequent rsync runs succeed. After the given number of consecutive rsync error exits, however, dss gives up, executes the exit hook and terminates. Set this to zero if dss should exit immediately on the first rsync error.

The only non-fatal error is when rsync exits with code 24. This indicates a partial transfer due to vanished source files and happens frequently when snapshotting a directory which is concurrently being modified.

create - execute rsync once to create a new snapshot
Usage: create

This command does not check the amount free disk space. The pre-create and post-create hooks are honored, however.

Specify --dry-run to see the rsync command which is executed to create snapshots.

prune - remove redundant and outdated snapshots
Usage: prune

A snapshot is considered outdated if its interval number is greater or equal than the specified number of unit intervals. See --unit-interval and --num-intervals above.

A snapshot is said to be redundant if the interval it belongs to contains more than the configured number of snapshots.

The prune command gets rid of both outdated and redundant snapshots. At most one snapshot is removed per invocation. If --dry-run is given, the subcommand only prints the snapshot that would be removed.

ls - print the list of all snapshots
Usage: ls

The list contains all existing snapshots, no matter of their state. Incomplete snapshots and snapshots being deleted will also be listed.

kill - send a signal to a running dss process
Usage: kill [--signal=<signal>] [--wait]

This sends a signal to the dss process that corresponds to the given config file. If --dry-run is given, the PID of the dss process is written to stdout, but no signal is sent.
-s
, --signal=<signal>

send the given signal rather than SIGTERM

default: SIGTERM

Like for kill(1), alternate signals may be specified in three ways: as a signal number (e.g., 9), the signal name (e.g., KILL), or the signal name prefixed with "SIG" (e.g., SIGKILL). In the latter two forms, the signal name and the prefix are case insensitive, so "sigkill" works as well.

Sending SIGHUP causes the running dss process to reload its config file.

-w, --wait

wait until the signalled process has terminated

This option is handy for system shutdown scripts which would like to terminate the dss daemon process.

Without --wait the dss process which executes the kill subcommand exits right after the kill(2) system call returns. At this point the signalled process might still be alive (even if SIGKILL was sent). If --wait is given, the process waits until the signalled process has terminated or the timeout expires.

If --wait is not given, the kill subcommand exits successfully if and only if the signal was sent (i.e., if there exists another dss process to receive the signal). With --wait it exits successfully if, additionally, the signalled process has terminated before the timeout expires.

It makes only sense to use the option for signals which terminate dss.

configtest - run a configuration file syntax test
Usage: configtest

This command checks the command line options and the configuration file for syntactic correctness. It either reports "Syntax Ok" and exits successfully or prints information about the first syntax error detected and terminates with exit code 1.

COPYRIGHT

Written by Andre Noll
Copyright (C) 2008 - present Andre Noll
License: GNU GPL version 2
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Report bugs to Andre Noll

SEE ALSO

ssh(1), rsync(1)