How to synchronize files between two servers

Suppose you have a collection of files which are replicated on two different servers. The two replicas are then modified independently, and you want whatever changes made in one replica to be propagated to the other, so that both replicas remain in sync.

There are several file mirroring tools on Linux, such as rsync or duplicati. However, these tools are meant for "uni-directional" file sync (i.e., pushing or pulling incremental updates in one direction), and so two-way sync would require running such tools twice, one for each direction.

Unison is an open-source file synchronization tool that natively supports bi-directional file synchronization. Unison is available on multiple platforms including Linux, FreeBSD, Windows and MacOS X. In Linux, Unison is available as a command-line tool as well as a GUI program with GTK+ interface.

In this tutorial, I will describe how to synchronize files between two servers with Unison command-line utility.

Install Unison on Linux

To install Unison on Debian, Linux or Linux Mint:

$ sudo apt-get install unison

To install Unison on Fedora, CentOS or RHEL:

$ sudo yum install unison

Create Unison Profile

The first thing to do is to create a Unison profile on both servers which have replicas to sync. A Unison profile is a text file (with .prf extension) that specifies file sync settings such as directory roots, include/ignore paths or patterns, etc.

You can create a Unison profile anywhere in your system, in which case you must define UNISON environment variable pointing to the directory path to the profile. If UNISON variable is not defined, Unison looks for profiles in $HOME/.unison directory by default.

The following snippet is a Unison profile example.

# Two root directories to sync.
# You can use ssh:// to sync over SSH
root = /home/alice/sync_folder
root = ssh://dev@192.168.1.10//home/alice/sync_folder

# If you want one-way mirroring from one replica to the other, specify the source replica using "force" as follows.
# force = /home/alice/sync_folder

# If you want Unison to run without any user input, try "batch" mode.
batch = true

# If you don't want to be prompted, and just accept Unison's recommendation:
auto = true

# Optionally, you can sync specific sub directories only (under the root).
# path = dir1
# path = dir2

# Optionally, you can ignore specific files or directories that are matched with regular expressions.
# ignore = Name *.o
# ignore = Name *~
# ignore = Path */temp/archive_*

# If you want to ignore difference in file props:
perms = 0

The most important information in the above are the two root directories to sync. Unison supports SSH, RSH or socket to sync files over network. So you can specify the root directory on a remote host by using one of the formats below:

SSH:

root = ssh://user@remote_host//absolute/path/to/root
root = ssh://user@remote_host/relative/path/to/root

RSH:

root = rsh://user@remote_host//absolute/path/to/root
root = rsh://user@remote_host/relative/path/to/root

Socket:

socket://remote_host:port_num//absolute/path/to/root
socket://remote_host:port_num/relative/path/to/root

Sync Files with Unison

Once a Unison profile has been created on both servers, simply run Unison from either server.

$ unison

If there are multiple Unison profiles (e.g., sync1.prf, sync2.prf) in Unison directory, you can specify the profile name as follows.

$ unison sync1

If you have not set up key-based SSH authentication for the remote host, Unison will ask you to log in to the remote host via SSH before performing file sync. If you have set up passwordless SSH login, Unison will automatically start file sync over SSH.

Contacting server...
Connected [//local//home/alice/sync_folder -> //remote_host//home/alice/sync_folder]
Looking for changes
  Waiting for changes from server
Reconciling changes
new file -->            document1.pdf
         <-- new file   my.jpg  
Propagating updates
UNISON 2.40.63 started propagating changes at 21:19:13.65 on 20 Sep 2013
[BGN] Copying document1.pdf from /home/alice/sync_folder to //remote_host//home/alice/sync_folder
[BGN] Copying my.jpg from //remote_host//home/alice/sync_folder to /home/alice/sync_folder
[END] Copying my.jpg
[END] Copying document1.pdf
UNISON 2.40.63 finished propagating changes at 21:19:13.68 on 20 Sep 2013
Saving synchronizer state
Synchronization complete at 21:19:13  (2 items transferred, 0 skipped, 0 failed)

Troubleshooting Unison

Symptom: When you run Unison, you see the following message:

  Waiting for changes from server - document1.pdf
Reconciling changes

local          remote_host             
props    <-?-> props      /  [] 

This means that Unison attempts to synchronize file properties (e.g., file permission). You can press "I" to ignore this path permanently, and proceed. Alternatively, you can permanently ignore props by adding the following line to your Unison profile file.

perms = 0

Subscribe to Xmodulo

Do you want to receive Linux FAQs, detailed tutorials and tips published at Xmodulo? Enter your email address below, and we will deliver our Linux posts straight to your email box, for free. Delivery powered by Google Feedburner.

4 thoughts on “How to synchronize files between two servers

    • rsync is UNI directional, as was pointed out in the second paragraph of the article, so will overwrite changes made on the remote systems.

      csync is bidirectional; but requires the presence of SMB or sftp

      unison does not require either, but does need ssh.

      unison can be used at the command line or with its well designed GUI, and it writes a log of the changes, as well as creating a history file of the state of all files synchronized for future synchronization runs.

      These features make unison probably the best tool available for maintaining synchronization of large number of files in different directories on different machines.

  1. I used to be a rsync guy, but now I have seen the light. I use git-annex for real good synching with much more options and control where to store backup if you have space on many places, like in the cloud (box.com, dropbox.com, drive.google.com etc).
    Web interface and directly synching of directories between computers and storage.

Leave a comment

Your email address will not be published. Required fields are marked *

Current day month ye@r *