![]() ![]() rsync is widely used for backups and mirroring and as an improved copy command for everyday use. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. If you are using rsync from a remote site outside MGH, please use the gateway server .edu for your data transfer purposes.Rsync is a fast and extraordinarily versatile file copying tool. ![]() If your user name is different on the remotehost than on the computer you are running rsync, then you need to specify rather than just remotehost. You will be prompted for your password on the remote host before the copy starts. Rsync -aZP /source/dir/to/copy remotehost:/target/dir/ If you want to copy/move files to a directory over the network to another computer, you simply need to preface the destination directory with the hostname of the remote computer to copy to followed by a colon. Now any files under /target/dir/copy that are not also present under /source/dir/to/copy will be deleted. Rsync -aZP –delete /source/dir/to/copy /target/dir/ To do this you simply add the –delete option to rsync. To do that you want files on the target destination side to be deleted if they do not exist at the source. Sometimes you want to the target destination to become an exact copy of the source, aka “a mirror”. In the simple example above, if there are files in the target destination that are not present at the source, they will be left alone and not touched. Try to maintain hard links within transfered files Other common options you may which to use: The key rule here to remember is use the -Z option when you are rsyncing files INTO your group storage areas On all Martinos CentOS7 machines, we have defined a global option alias -Z that does the above. So instead you should run rsync with the following options: When you are copying data to one of your share groups areas, this can be problematic as it will ignore the sticky group bit as discussed in Understanding Group Permissions in UNIX. With the -a option the rsync will try to preserve both the exact permissions and group of the source. Normally you NEVER want a trailing slash on the source directory but you DO want a trailing slash on the target directory. GOTCHA WARNING: one thing to be careful of is trailing slashes. If your intention was to move the data instead of just copy, you would then just run Also, you can add -P to get a progress bar on each files which is helpful when you have very large files. If you want to see each file that gets copied as it rsync looks at each file and only copies over what is not present or is different at the target destination. In fact you should do this to verify the copy. You can at this point actually run the exactly same command again. The end result will be a copy of /source/dir/to/copy located at /target/dir/copy. Rsync -avP /source/dir/to/copy /target/dir/ Take a look at this example of copying /source/dir/to/copy into /target/dir Don’t let that apparent complexity scare you. If you look at the man page for rsync you will see it has a ton of options. The rsync utility is a very advanced file transfer utility that solves these issues. This same issue applies to the SSH copy utility scp. Even for a plain copy, restarting the copy is less than ideal as it will recopy everything that already got copied. When that happens on a move operation, your data will be in a inconsistant state with part of it still in the original location and part of it in the target destination. Since the operations can take a long time, there is a fair chance something will happen to interrupt or stop the copy or move. When copying or moving large numbers of files, the generic UNIX utilities cp and mv are actually dangerous. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |