banner
十一

十一

Stay hungry, stay foolish.

The Fire Blue Dagger of File Synchronization: rsync

Introduction to rsync#

rsync is an open-source, fast, multifunctional tool for local or remote data synchronization and backup that supports both full and incremental backups.

Introduction to rsync#

The full English name of rsync is Remote synchronization service software, abbreviated as rsync.
rsync is an open-source, fast, multifunctional tool that can achieve full and incremental local or remote data synchronization (copy) backup, and is known as the fire blue dagger of file synchronization.
rsync is a data mirroring backup tool for Linux systems. Using the fast incremental backup tool Remote Sync, it can synchronize remotely, support local copying, or synchronize with other SSH or rsync hosts.
Currently, it supports cross-platform synchronization between Windows and Linux.

Features of rsync#

rsync supports many features:

  • Can mirror and save entire directory trees and filesystem files.
  • Can easily maintain original file permissions, timestamps, soft and hard links, etc.
  • Can be installed without special permissions.
    • Fast: During the first synchronization, rsync copies all content, but in subsequent transfers, only modified and newly created files are transmitted. rsync can perform compression and decompression during file transfer, thus using less bandwidth and reducing network pressure.
    • Secure: Files can be transferred using methods like scp, SSH, or through direct socket connections.
    • Supports anonymous transfers for easy website mirroring.
      Summary: One rsync command integrates the functions of scp, cp, and rm, but is more flexible than scp, cp, and rm.

Three Modes of Operation for rsync#

  • local local mode -- cp
  • Access via remote shell access via remote shell -- scp
  • daemon daemon mode -- most commonly used

Modes of File Synchronization with rsync#

  • Full: Transfer all data to overwrite.
  • Incremental: Only transfer the differing parts of the data; rsync uses a unique quick check algorithm to achieve incremental data transfer.

Functionality of rsync#

  • Similar to the cp command — local backup transfer of data.
  • Similar to the scp command — remote backup transfer of data.
  • Similar to the rm command — achieve no-difference synchronization backup.
  • Similar to the ls command — view local file information.

Authentication Protocols for rsync#

Before using the rsync command to synchronize files, you need to log into the remote host. There are two protocols used during the authentication process:

  • ssh protocol
  • rsync protocol
    In regular use, the most commonly used method is rsync-daemon.

rsync Authentication (rsync-daemon)#

  • rsync listens on the default TCP port 873 under the rsync-daemon authentication method;
  • The rsync-daemon authentication method is the main authentication method for rsync, and it is also the one we frequently use;
  • Only in this mode can rsync write the password to a file.
    Note: The rsync-daemon authentication method requires both the server and client to have the rsync service installed, and only the rsync server needs to start rsync while configuring the rsync configuration file. The client does not need to start the rsync service, which does not affect the normal synchronization process.

ssh Authentication#

  • Under ssh authentication, rsync can authenticate through system users, meaning that files are transferred through an ssh tunnel, similar to the scp tool;
  • At this time, synchronization operations are not limited to the synchronization folders defined in rsync;
  • There is no need to use port 873 for transmission.
    Note: The ssh authentication method does not require the server and client to configure the rsync configuration file; both parties only need to have the rsync service installed, and neither party needs to start rsync.
#
The rsync server does not need to start the rsync daemon process; as long as you obtain the username and password of the remote host, you can directly rsync to synchronize files.
#
Since the rsync server does not start the daemon process, there is no need for the configuration file /etc/rsyncd.conf.

The ssh authentication protocol works similarly to scp. If you do not want to enter a password during synchronization, use ssh-keygen -t -rsa to establish a tunnel.

// This method omits -e ssh by default and is equivalent to the following:
rsync -avz /SRC -e ssh [email protected]:/DEST 
    -a  // File owner changes, timestamp remains unchanged
    -V  // Display detailed information during the process
    -z  // Compress data during transfer

// When needing to change the port, we can:
# Changed the ssh protocol port, default is 22
rsync -avz /SRC -e "ssh -p2222" [email protected]:/DEST

rsync Command#

Installing the rsync Command#

Generally, RedHat and CentOS come with this tool. If not, you can install it directly using yum install -y rsync.

# Check which package provides the rsync command
[root@node1 ~]# yum provides */bin/rsync
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Last metadata expiration check: 0:00:27 ago on Mon 10 May 2021 03:27:01 PM CST.
rsync-3.1.3-9.el8.x86_64 : A program for synchronizing files over a network
Repo        : @System
Matched from:
Filename    : /usr/bin/rsync

rsync-3.1.3-9.el8.x86_64 : A program for synchronizing files over a network
Repo        : base
Matched from:
Filename    : /usr/bin/rsync

# Install the rsync command
[root@node1 ~]# yum -y install rsync

# Installation successful
[root@node1 ~]# which rsync
/usr/bin/rsync

Format of the rsync Command#

// The common formats of the rsync command are as follows:
    rsync [OPTION]... SRC DEST
    rsync [OPTION]... SRC [USER@]HOST:DEST
    rsync [OPTION]... [USER@]HOST:SRC DEST

// Corresponding to the above three command formats, rsync has three different working modes:
1) Copy local files. This mode is activated when neither SRC nor DEST contains a single colon ":" separator.
# Command example
[root@localhost ~]# rsync -avz abc /opt/123
2) Use a remote shell program (like rsh, ssh) to copy content from the local machine to the remote machine. This mode is activated when the DST path contains a single colon ":" separator.
# Command example
[root@localhost ~]# ssh [email protected] 'ls -l /root'
3) Use a remote shell program (like rsh, ssh) to copy content from the remote machine to the local machine. This mode is activated when the SRC path contains a single colon ":" separator.
# Command example
[root@localhost ~]# rsync -avz [email protected]:/etc/yum.repos.d /root/

Details of rsync Command Parameters#

-v, --verbose         Detailed output mode
-q, --quiet           Minimal output mode
-c, --checksum        Enable checksum switch, forcing file transfer verification
-a, --archive         Archive mode, indicating recursive transfer of files while preserving all file attributes, equivalent to -rlptgoD
-r, --recursive       Process subdirectories recursively
-R, --relative        Use relative path information
-b, --backup          Create backups; when a file with the same name already exists at the destination, rename the old file to ~filename. You can use the --suffix option to specify a different backup file prefix.
--backup-dir          Store backup files (like ~filename) in a specified directory.
-suffix=SUFFIX        Define the backup file prefix
-u, --update          Only update, skipping all files that already exist in DST and have a later timestamp than the file to be backed up. (Do not overwrite updated files)
-l, --links           Preserve soft links
-L, --copy-links      Treat soft links as regular files
--copy-unsafe-links   Only copy links pointing outside the SRC path directory tree
--safe-links          Ignore links pointing outside the SRC path directory tree
-H, --hard-links      Preserve hard links
-p, --perms           Preserve file permissions
-o, --owner           Preserve file owner information
-g, --group           Preserve file group information
-D, --devices         Preserve device file information
-t, --times           Preserve file timestamp information
-S, --sparse          Special handling of sparse files to save space in DST
-n, --dry-run         Show which files will be transferred
-W, --whole-file      Copy files without performing incremental checks
-x, --one-file-system Do not cross filesystem boundaries
-B, --block-size=SIZE Block size used by the checksum algorithm, default is 700 bytes
-e, --rsh=COMMAND     Specify using rsh or ssh for data synchronization
--rsync-path=PATH     Specify the path of the rsync command on the remote server
-C, --cvs-exclude     Automatically ignore files using the same method as CVS to exclude files you do not wish to transfer
--existing            Only update files that already exist in DST, and do not back up newly created files
--delete              Delete files in DST that are not in SRC
--delete-excluded     Also delete files on the receiving end that are specified to be excluded by this option
--delete-after        Delete after the transfer is complete
--ignore-errors       Delete even if there are IO errors
--max-delete=NUM      Delete at most NUM files
--partial             Keep partially transferred files to speed up subsequent transfers
--force               Force deletion of directories, even if not empty
--numeric-ids         Do not match numeric user and group IDs to usernames and group names
--timeout=TIME        IP timeout, in seconds
-I, --ignore-times    Do not skip files with the same timestamp and length
--size-only           When deciding whether to back up files, only consider file size, not file time
--modify-window=NUM   Determine the timestamp window used when files have the same time, default is 0
-T --temp-dir=DIR     Create temporary files in DIR
--compare-dest=DIR    Compare files in DIR to determine if a backup is needed
-P                    Equivalent to --partial
--progress            Display the backup process
-z, --compress        Compress files during transfer
--exclude=PATTERN     Specify patterns for files to exclude from transfer
--include=PATTERN     Specify patterns for files to include for transfer
--exclude-from=FILE   Exclude files matching patterns specified in FILE
--include-from=FILE   Include files matching patterns specified in FILE
--version             Print version information
--address             Bind to a specific address
--config=FILE         Specify an alternative configuration file, not using the default rsyncd.conf file
--port=PORT           Specify an alternative rsync server port
--blocking-io         Use blocking IO for remote shell
-stats                Provide transfer status for certain files
--progress            Display transfer progress during transfer
--log-format=formAT   Specify log file format
--password-file=FILE  Get password from FILE
--bwlimit=KBPS        Limit I/O bandwidth, KBytes per second
-h, --help            Display help information

rsync Configuration#

rsync can be used in three ways:

  • Mode One: local local mode
  • Mode Two: Access via remote shell access via remote shell
  • Mode Three: daemon daemon mode (most commonly used)

Mode One: Local Mode#

Local mode can be used directly with the command.

# Command format
rsync [OPTION...] SRC... [DEST]

Mode Two: Access via Remote shell#

This mode generally involves pushing and pulling via remote shell commands.

# Command format
# Pull
rsync [OPTION...] [USER]@HOST:SRC... [DEST]

# Push
rsync [OPTION] SRC... [USER]@HOST:DEST

Note: Both the accessing and accessed ends need to have the rsync command installed.
Explanation: When transferring files, the file checksums in the source and destination directories are compared first, and only when the checksums differ will the transfer occur.
Key Point: In practical scenarios, rsync+ssh key authentication is usually used to enable passwordless login.

Mode Three: Daemon Mode#

Daemon mode allows for real-time synchronization across locations. Compared to the first two modes, it is more complex and powerful.
Note: This mode requires the source server to install the application: rsync + inotify-tools or rsync + sersync; the target server only needs to install rsync.

rsync + inotify-tools Method#

rsync + sersync Method#

Loading...
Ownership of this post data is guaranteed by blockchain and smart contracts to the creator alone.