Category Archives: Rsync

DeltaCopy, Rsync, Cygwin and Windows to Linux backups

I wrote a while back regarding Rsync backups from Windows servers to Linux using DeltaCopy.

It’s all been working ok but I’ve had some files failing with a “file has vanished’ message.

At first I tried searching for a solution based on the failures being any file with a special accent character.

The log email that I get sent by DeltaCopy contained something like this:

[text]
Profile datafiles failed to execute.
Execution log
————-
sending incremental file list
file has vanished: "/cygdrive/D/datfiles/A standard Windows type file?.jpg"
file has vanished: "/cygdrive/D/datfiles/Another file with two questionmarks ??.docx
….
[/text]

Investigating the Windows server the files did exist and they were not “in use” during the backup. However, some (note: not all)  some, files had special non-standard English characters in the file names. So characters with umlauts etc, like È, É, Ê  were giving issues alongside some punctuation characters that appeared normal but I suspect have been prepared from a non-English keyboard or system with a different character set.

So assuming that the character set was the issue I went searching.

I had narrowed it to the Cygwin toolkit rather than Rsync itself and came across this old message thread that included a link to a modified cygwin1.dll as http://www.okisoft.co.jp/esc/utf8-cygwin/  except that the link was broken. So I chased a modified cygwin1.dll and found what appears to be the updated link at http://www.oki-osk.jp/esc/utf8-cygwin/.

Downloading the compiled library file cygwin1-dll-20-11-18.tar.bz2 and extracting it to my Windows server using 7-Zip I renamed the old dll as cygwin1.dll.orig and copied the modified cygwin1.dll into the DeltaCopy directory.

Then I re-ran a data file backup that had given me “file vanished” errors and it seemed to work. But the next time it ran, it gave errors again.

So I delved deeper and found that the version of Rsync that is distributed with DeltaCopy is quite old, replacing the Rsync toolkit that supports DeltaCopy was the next thing to try and it worked a treat. I will document that process shortly.

 

 

Rsync on Debian

I’ve acquired a few more Linux servers recently with Debian Squeeze installed (version 6.0.6 according to #cat /etc/debian_version).

Installing Rsync is easy:

[bash]
apt-get install rsync
[/bash]

The tricks are in getting the config files done, actually running the service, connecting, etc.

The default config file is in /etc/default/rsync  which sets up how rsync will run.

The default settings file is expected as /etc/rsync.conf  which I change to set to /etc/rsyncd/rsyncd.conf   just so I am clear which config is ‘mine’ to work on.  The actual change in the /etc/default/rsync file is shown here with the RSYNC_CONFIG_FILE line enabled (removed the # at the start) and with the path setting as described.

[text]
# which file should be used as the configuration file for rsync.
# This file is used instead of the default /etc/rsyncd.conf
# Warning: This option has no effect if the daemon is accessed
#          using a remote shell. When using a different file for
#          rsync you might want to symlink /etc/rsyncd.conf to
#          that file.
RSYNC_CONFIG_FILE=/etc/rsyncd/rsyncd.conf

[/text]

The other important bit is the Rsync is not enabled by default and is therefore not going to run or be running!  In the same /etc/default/rsync file edit the RSYNC_ENABLE option to change false to true

[text]
# start rsync in daemon mode from init.d script?
#  only allowed values are "true", "false", and "inetd"
#  Use "inetd" if you want to start the rsyncd from inetd,
#  all this does is prevent the init.d script from printing a message
#  about not starting rsyncd (you still need to modify inetd’s config yourself).
RSYNC_ENABLE=true
#
#
[/text]
[bash]
#service rsync restart
[/bash]

I used the restart command even though I knew that it was not running (yet) as I can reuse the command via Ctrl-R when I do more edits to the config settings

[bash]
/etc/default# service rsync restart
Restarting rsync daemon: rsyncrsync daemon not running, attempting to start. … (warning).
missing or empty config file /etc/rsyncd/rsyncd.conf … failed!
failed!
/etc/default#
[/bash]

And as expected there is a warning that it was not running to be able to stop it, and then the start action fails as there is no config file, as yet.

So that is all good.

Next create the appropriate config file. Easiest to start with the example conf file from the default install

[bash]
/etc/rsyncd/# cp /usr/share/doc/rsync/examples/rsyncd.conf .
[/bash]

Then edit it…

[text]
# sample rsyncd.conf configuration file
# GLOBAL OPTIONS
#motd file=/etc/motd
log file=/var/log/rsyncd
# for pid file, do not use /var/run/rsync.pid if
# you are going to run rsync out of the init.d script.
# pid file=/var/run/rsyncd.pid
#syslog facility=daemon
#socket options=
[/text]

The sections in the config file are based on the sample config file and in the top Global Options I only check that the log file path is correct and enabled

For each backup set that is going to be inbound to this server I add a new section to the config file:

[text]
# MODULE OPTIONS

[ftp]
comment = public archive <– a clever comment
path = /var/www/pub <– set the path
      use chroot = yes
#      max connections=10 <– do not set it to 1
lock file = /var/lock/rsyncd
# the default for read only is yes…
read only = yes <– change this to no so source files can be written
list = yes
       uid = nobody <– the username for the newly uploaded files
       gid = nogroup <– the group for the files
#       exclude =
#       exclude from =
#       include =
#       include from =
#       auth users = <– comma space delimited list of names that appear in the secrets file
#       secrets file =/etc/rsyncd.secrets <– a text file with a username:password
        strict modes = yes
#       hosts allow = <– ip address for the source system
#       hosts deny =
       ignore errors = no
       ignore nonreadable = yes
       transfer logging = no
#       log format = %t: host %h (%a) %o %f (%l bytes). Total %b bytes.
       timeout = 600
refuse options = checksum dry-run
       dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz
[/text]

and after each update to the config file restart the service to load the new config and confirm that Rsync will run with it.

[bash]
/etc/default# service rsync restart
[/bash]

I’ve not looked at SSH / SSL type connections for Rsync in this. The main thing was to get Rsync setup as a server and control the inbound traffic based on accounts, servers, and ip restrictions.

Another post will update SSH configuration.

Rsync Backups Windows Servers

Backups are problematic. It is a love / hate thing for me. I love having them when I need them but I hate the prep work that goes into getting them done. Of course, that is offset by the hate for the work I have to do when there isn’t one…..  … in any case backups are necessary.

Ok. So setting up yet another server for Rsync backups. This time Windows to a Linux based server. Of course I’ve never documented the process in the past so this time I will spend the extra minutes and make a complete note of what I am doing so I have a consistent process for the next time. I also decided to look at what options are available to provide minimum fuss.

I looked at the various options of cwRsync (which I have used before), Grsync, Cygwin with Rsync as a self-build, a few tools that appear to only support their specific hosted solution like rsync.net.  After doing some reading and some basic testing I scrapped all of them and was about to give up when I tried QtdSync and finally found something that worked as intended, but…  I cannot get it to schedule the backups.

For the record:

cwRsync: skip this. The Windows GUI version is paid only without a trial to see what it is like. The free GPL version is command line and the instructions are vague. I know I have used this in the past but the developer seems to have changed focus.

Grsync: Appears to be a useful GUI but it is unclear as to how to set it to connect to a remote Rsync server and as a result I uninstalled within a few minutes.

QtdSync. SourceForge, GPL Free Software. Simple installation. On the test server it worked fine. I then noted all the steps as I installed to a production server. But I cannot get it to run as a service or schedule to work without being logged into the system. I think it looks like a great utility for desktop to server / NAS type devices for home or small business but it just does not cut it for this. I think it will be up for it at some stage, given a bit of new breath in the development.

DeltaCopy: Is held apparently in high regard and I spent several hours on this and gave up.  It connected and everything appeared to be ok, except I kept getting an error that it was wanting a directory to exist on the remote backup server before it would transfer files. If I manually added the directory it worked fine. I tried many things but could not figure it out.

That was a few days ago and I have come back to it today and it worked first time. The chair to keyboard interface playing up again!

I’ll document how I’ve used it as another post. Suffice to say I like DeltaCopy for Windows server Rsync backups.