Category Archives: Rsync

DeltaCopy, Rsync, Cygwin and Windows to Linux backups

I wrote a while back regarding Rsync backups from Windows servers to Linux using DeltaCopy.

It’s all been working ok but I’ve had some files failing with a “file has vanished’ message.

At first I tried searching for a solution based on the failures being any file with a special accent character.

The log email that I get sent by DeltaCopy contained something like this:

[text]
Profile datafiles failed to execute.
Execution log
————-
sending incremental file list
file has vanished: "/cygdrive/D/datfiles/A standard Windows type file?.jpg"
file has vanished: "/cygdrive/D/datfiles/Another file with two questionmarks ??.docx
….
[/text]

Investigating the Windows server the files did exist and they were not “in use” during the backup. However, some (note: not all)  some, files had special non-standard English characters in the file names. So characters with umlauts etc, like È, É, Ê  were giving issues alongside some punctuation characters that appeared normal but I suspect have been prepared from a non-English keyboard or system with a different character set.

So assuming that the character set was the issue I went searching.

I had narrowed it to the Cygwin toolkit rather than Rsync itself and came across this old message thread that included a link to a modified cygwin1.dll as http://www.okisoft.co.jp/esc/utf8-cygwin/  except that the link was broken. So I chased a modified cygwin1.dll and found what appears to be the updated link at http://www.oki-osk.jp/esc/utf8-cygwin/.

Downloading the compiled library file cygwin1-dll-20-11-18.tar.bz2 and extracting it to my Windows server using 7-Zip I renamed the old dll as cygwin1.dll.orig and copied the modified cygwin1.dll into the DeltaCopy directory.

Then I re-ran a data file backup that had given me “file vanished” errors and it seemed to work. But the next time it ran, it gave errors again.

So I delved deeper and found that the version of Rsync that is distributed with DeltaCopy is quite old, replacing the Rsync toolkit that supports DeltaCopy was the next thing to try and it worked a treat. I will document that process shortly.

 

 

Rsync on Debian

I’ve acquired a few more Linux servers recently with Debian Squeeze installed (version 6.0.6 according to #cat /etc/debian_version).

Installing Rsync is easy:

[bash]
apt-get install rsync
[/bash]

The tricks are in getting the config files done, actually running the service, connecting, etc.

The default config file is in /etc/default/rsync  which sets up how rsync will run.

The default settings file is expected as /etc/rsync.conf  which I change to set to /etc/rsyncd/rsyncd.conf   just so I am clear which config is ‘mine’ to work on.  The actual change in the /etc/default/rsync file is shown here with the RSYNC_CONFIG_FILE line enabled (removed the # at the start) and with the path setting as described.

[text]
# which file should be used as the configuration file for rsync.
# This file is used instead of the default /etc/rsyncd.conf
# Warning: This option has no effect if the daemon is accessed
#          using a remote shell. When using a different file for
#          rsync you might want to symlink /etc/rsyncd.conf to
#          that file.
RSYNC_CONFIG_FILE=/etc/rsyncd/rsyncd.conf

[/text]

The other important bit is the Rsync is not enabled by default and is therefore not going to run or be running!  In the same /etc/default/rsync file edit the RSYNC_ENABLE option to change false to true

[text]
# start rsync in daemon mode from init.d script?
#  only allowed values are "true", "false", and "inetd"
#  Use "inetd" if you want to start the rsyncd from inetd,
#  all this does is prevent the init.d script from printing a message
#  about not starting rsyncd (you still need to modify inetd’s config yourself).
RSYNC_ENABLE=true
#
#
[/text]
[bash]
#service rsync restart
[/bash]

I used the restart command even though I knew that it was not running (yet) as I can reuse the command via Ctrl-R when I do more edits to the config settings

[bash]
/etc/default# service rsync restart
Restarting rsync daemon: rsyncrsync daemon not running, attempting to start. … (warning).
missing or empty config file /etc/rsyncd/rsyncd.conf … failed!
failed!
/etc/default#
[/bash]

And as expected there is a warning that it was not running to be able to stop it, and then the start action fails as there is no config file, as yet.

So that is all good.

Next create the appropriate config file. Easiest to start with the example conf file from the default install

[bash]
/etc/rsyncd/# cp /usr/share/doc/rsync/examples/rsyncd.conf .
[/bash]

Then edit it…

[text]
# sample rsyncd.conf configuration file
# GLOBAL OPTIONS
#motd file=/etc/motd
log file=/var/log/rsyncd
# for pid file, do not use /var/run/rsync.pid if
# you are going to run rsync out of the init.d script.
# pid file=/var/run/rsyncd.pid
#syslog facility=daemon
#socket options=
[/text]

The sections in the config file are based on the sample config file and in the top Global Options I only check that the log file path is correct and enabled

For each backup set that is going to be inbound to this server I add a new section to the config file:

[text]
# MODULE OPTIONS

[ftp]
comment = public archive <– a clever comment
path = /var/www/pub <– set the path
      use chroot = yes
#      max connections=10 <– do not set it to 1
lock file = /var/lock/rsyncd
# the default for read only is yes…
read only = yes <– change this to no so source files can be written
list = yes
       uid = nobody <– the username for the newly uploaded files
       gid = nogroup <– the group for the files
#       exclude =
#       exclude from =
#       include =
#       include from =
#       auth users = <– comma space delimited list of names that appear in the secrets file
#       secrets file =/etc/rsyncd.secrets <– a text file with a username:password
        strict modes = yes
#       hosts allow = <– ip address for the source system
#       hosts deny =
       ignore errors = no
       ignore nonreadable = yes
       transfer logging = no
#       log format = %t: host %h (%a) %o %f (%l bytes). Total %b bytes.
       timeout = 600
refuse options = checksum dry-run
       dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz
[/text]

and after each update to the config file restart the service to load the new config and confirm that Rsync will run with it.

[bash]
/etc/default# service rsync restart
[/bash]

I’ve not looked at SSH / SSL type connections for Rsync in this. The main thing was to get Rsync setup as a server and control the inbound traffic based on accounts, servers, and ip restrictions.

Another post will update SSH configuration.