Introduction to rsync
#
rsync is an open-source, fast, multifunctional tool for local or remote data synchronization and backup that supports both full and incremental backups.
Introduction to rsync
#
The full English name of rsync
is Remote synchronization service software
, abbreviated as rsync
.
rsync
is an open-source, fast, multifunctional tool that can achieve full and incremental local or remote data synchronization (copy) backup, and is known as the fire blue dagger of file synchronization.
rsync
is a data mirroring backup tool for Linux systems. Using the fast incremental backup tool Remote Sync
, it can synchronize remotely, support local copying, or synchronize with other SSH
or rsync
hosts.
Currently, it supports cross-platform synchronization between Windows
and Linux
.
Features of rsync
#
rsync
supports many features:
- Can mirror and save entire directory trees and filesystem files.
- Can easily maintain original file permissions, timestamps, soft and hard links, etc.
- Can be installed without special permissions.
- Fast: During the first synchronization,
rsync
copies all content, but in subsequent transfers, only modified and newly created files are transmitted.rsync
can perform compression and decompression during file transfer, thus using less bandwidth and reducing network pressure. - Secure: Files can be transferred using methods like
scp
,SSH
, or through directsocket
connections. - Supports anonymous transfers for easy website mirroring.
Summary: Onersync
command integrates the functions ofscp
,cp
, andrm
, but is more flexible thanscp
,cp
, andrm
.
- Fast: During the first synchronization,
Three Modes of Operation for rsync
#
local
local mode --cp
Access via remote shell
access via remoteshell
--scp
daemon
daemon mode -- most commonly used
Modes of File Synchronization with rsync
#
- Full: Transfer all data to overwrite.
- Incremental: Only transfer the differing parts of the data;
rsync
uses a uniquequick check
algorithm to achieve incremental data transfer.
Functionality of rsync
#
- Similar to the
cp
command — local backup transfer of data. - Similar to the
scp
command — remote backup transfer of data. - Similar to the
rm
command — achieve no-difference synchronization backup. - Similar to the
ls
command — view local file information.
Authentication Protocols for rsync
#
Before using the rsync
command to synchronize files, you need to log into the remote
host. There are two protocols used during the authentication process:
ssh
protocolrsync
protocol
In regular use, the most commonly used method isrsync-daemon
.
rsync
Authentication (rsync-daemon)#
rsync
listens on the defaultTCP
port873
under thersync-daemon
authentication method;- The
rsync-daemon
authentication method is the main authentication method forrsync
, and it is also the one we frequently use; - Only in this mode can
rsync
write the password to a file.
Note: Thersync-daemon
authentication method requires both the server and client to have thersync
service installed, and only thersync
server needs to startrsync
while configuring thersync
configuration file. The client does not need to start thersync
service, which does not affect the normal synchronization process.
ssh
Authentication#
- Under
ssh
authentication,rsync
can authenticate through system users, meaning that files are transferred through anssh
tunnel, similar to thescp
tool; - At this time, synchronization operations are not limited to the synchronization folders defined in
rsync
; - There is no need to use port
873
for transmission.
Note: Thessh
authentication method does not require the server and client to configure thersync
configuration file; both parties only need to have thersync
service installed, and neither party needs to startrsync
.
#
The rsync server does not need to start the rsync daemon process; as long as you obtain the username and password of the remote host, you can directly rsync to synchronize files.
#
Since the rsync server does not start the daemon process, there is no need for the configuration file /etc/rsyncd.conf.
The ssh
authentication protocol works similarly to scp
. If you do not want to enter a password during synchronization, use ssh-keygen -t -rsa
to establish a tunnel.
// This method omits -e ssh by default and is equivalent to the following:
rsync -avz /SRC -e ssh [email protected]:/DEST
-a // File owner changes, timestamp remains unchanged
-V // Display detailed information during the process
-z // Compress data during transfer
// When needing to change the port, we can:
# Changed the ssh protocol port, default is 22
rsync -avz /SRC -e "ssh -p2222" [email protected]:/DEST
rsync
Command#
Installing the rsync
Command#
Generally, RedHat
and CentOS
come with this tool. If not, you can install it directly using yum install -y rsync
.
# Check which package provides the rsync command
[root@node1 ~]# yum provides */bin/rsync
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Last metadata expiration check: 0:00:27 ago on Mon 10 May 2021 03:27:01 PM CST.
rsync-3.1.3-9.el8.x86_64 : A program for synchronizing files over a network
Repo : @System
Matched from:
Filename : /usr/bin/rsync
rsync-3.1.3-9.el8.x86_64 : A program for synchronizing files over a network
Repo : base
Matched from:
Filename : /usr/bin/rsync
# Install the rsync command
[root@node1 ~]# yum -y install rsync
# Installation successful
[root@node1 ~]# which rsync
/usr/bin/rsync
Format of the rsync
Command#
// The common formats of the rsync command are as follows:
rsync [OPTION]... SRC DEST
rsync [OPTION]... SRC [USER@]HOST:DEST
rsync [OPTION]... [USER@]HOST:SRC DEST
// Corresponding to the above three command formats, rsync has three different working modes:
1) Copy local files. This mode is activated when neither SRC nor DEST contains a single colon ":" separator.
# Command example
[root@localhost ~]# rsync -avz abc /opt/123
2) Use a remote shell program (like rsh, ssh) to copy content from the local machine to the remote machine. This mode is activated when the DST path contains a single colon ":" separator.
# Command example
[root@localhost ~]# ssh [email protected] 'ls -l /root'
3) Use a remote shell program (like rsh, ssh) to copy content from the remote machine to the local machine. This mode is activated when the SRC path contains a single colon ":" separator.
# Command example
[root@localhost ~]# rsync -avz [email protected]:/etc/yum.repos.d /root/
Details of rsync
Command Parameters#
-v, --verbose Detailed output mode
-q, --quiet Minimal output mode
-c, --checksum Enable checksum switch, forcing file transfer verification
-a, --archive Archive mode, indicating recursive transfer of files while preserving all file attributes, equivalent to -rlptgoD
-r, --recursive Process subdirectories recursively
-R, --relative Use relative path information
-b, --backup Create backups; when a file with the same name already exists at the destination, rename the old file to ~filename. You can use the --suffix option to specify a different backup file prefix.
--backup-dir Store backup files (like ~filename) in a specified directory.
-suffix=SUFFIX Define the backup file prefix
-u, --update Only update, skipping all files that already exist in DST and have a later timestamp than the file to be backed up. (Do not overwrite updated files)
-l, --links Preserve soft links
-L, --copy-links Treat soft links as regular files
--copy-unsafe-links Only copy links pointing outside the SRC path directory tree
--safe-links Ignore links pointing outside the SRC path directory tree
-H, --hard-links Preserve hard links
-p, --perms Preserve file permissions
-o, --owner Preserve file owner information
-g, --group Preserve file group information
-D, --devices Preserve device file information
-t, --times Preserve file timestamp information
-S, --sparse Special handling of sparse files to save space in DST
-n, --dry-run Show which files will be transferred
-W, --whole-file Copy files without performing incremental checks
-x, --one-file-system Do not cross filesystem boundaries
-B, --block-size=SIZE Block size used by the checksum algorithm, default is 700 bytes
-e, --rsh=COMMAND Specify using rsh or ssh for data synchronization
--rsync-path=PATH Specify the path of the rsync command on the remote server
-C, --cvs-exclude Automatically ignore files using the same method as CVS to exclude files you do not wish to transfer
--existing Only update files that already exist in DST, and do not back up newly created files
--delete Delete files in DST that are not in SRC
--delete-excluded Also delete files on the receiving end that are specified to be excluded by this option
--delete-after Delete after the transfer is complete
--ignore-errors Delete even if there are IO errors
--max-delete=NUM Delete at most NUM files
--partial Keep partially transferred files to speed up subsequent transfers
--force Force deletion of directories, even if not empty
--numeric-ids Do not match numeric user and group IDs to usernames and group names
--timeout=TIME IP timeout, in seconds
-I, --ignore-times Do not skip files with the same timestamp and length
--size-only When deciding whether to back up files, only consider file size, not file time
--modify-window=NUM Determine the timestamp window used when files have the same time, default is 0
-T --temp-dir=DIR Create temporary files in DIR
--compare-dest=DIR Compare files in DIR to determine if a backup is needed
-P Equivalent to --partial
--progress Display the backup process
-z, --compress Compress files during transfer
--exclude=PATTERN Specify patterns for files to exclude from transfer
--include=PATTERN Specify patterns for files to include for transfer
--exclude-from=FILE Exclude files matching patterns specified in FILE
--include-from=FILE Include files matching patterns specified in FILE
--version Print version information
--address Bind to a specific address
--config=FILE Specify an alternative configuration file, not using the default rsyncd.conf file
--port=PORT Specify an alternative rsync server port
--blocking-io Use blocking IO for remote shell
-stats Provide transfer status for certain files
--progress Display transfer progress during transfer
--log-format=formAT Specify log file format
--password-file=FILE Get password from FILE
--bwlimit=KBPS Limit I/O bandwidth, KBytes per second
-h, --help Display help information
rsync
Configuration#
rsync
can be used in three ways:
- Mode One:
local
local mode - Mode Two:
Access via remote shell
access via remoteshell
- Mode Three:
daemon
daemon mode (most commonly used)
Mode One: Local Mode#
Local mode can be used directly with the command.
# Command format
rsync [OPTION...] SRC... [DEST]
Mode Two: Access via Remote shell
#
This mode generally involves pushing and pulling via remote shell
commands.
# Command format
# Pull
rsync [OPTION...] [USER]@HOST:SRC... [DEST]
# Push
rsync [OPTION] SRC... [USER]@HOST:DEST
Note: Both the accessing and accessed ends need to have the rsync
command installed.
Explanation: When transferring files, the file checksums in the source and destination directories are compared first, and only when the checksums differ will the transfer occur.
Key Point: In practical scenarios, rsync+ssh
key authentication is usually used to enable passwordless login.
Mode Three: Daemon Mode#
Daemon mode allows for real-time synchronization across locations. Compared to the first two modes, it is more complex and powerful.
Note: This mode requires the source server to install the application: rsync
+ inotify-tools
or rsync
+ sersync
; the target server only needs to install rsync
.