Discussion:
why rsync wants to delete destination files
(too old to reply)
T o n g
2010-08-07 03:50:02 UTC
Permalink
Hi,

I believed that rsync wouldn't delete existing destination files unless
instructed so (by --delete-excluded). However, while debugging a
(mysterious) bug,
http://thread.gmane.org/gmane.linux.file-systems.aufs.user/2821/focus=2827
I now suspect that the rsync command used in the script was trying to
delete existing destination files.

Here are the (abstracted) logs:

$ find rw
rw
rw/d1
rw/d1/.wh.7
rw/d1/.wh.6
rw/.wh..wh.orph
rw/.wh..wh.plnk
rw/.wh..wh.aufs

$ find ro2
ro2
ro2/d2
ro2/d2/6
ro2/d2/5
ro2/d2/4
ro2/d2/3
ro2/d2/2
ro2/d2/1
ro2/d1
ro2/d1/.wh.2
ro2/d1/.wh.3
ro2/d1/four
ro2/d1/.wh.4

. . .
++ rsync --exclude=lost+found -aHSx --devices --specials --delete-before
--remove-source-files '--exclude=.wh..wh.*' rw/ ro2
. . .

$ find ro2
ro2
ro2/d1
ro2/d1/.wh.6
ro2/d1/.wh.7

See, all the previous content have gone.

Now, try again with the an extra '--max-delete=0' option:
(http://thread.gmane.org/gmane.linux.file-systems.aufs.user/2821/
focus=2827)

$ find rw ro_mid
rw
rw/d1
rw/d1/.wh.7
rw/d1/.wh.6
rw/.wh..wh.orph
rw/.wh..wh.plnk
rw/.wh..wh.aufs
ro_mid
ro_mid/d2
ro_mid/d2/6
ro_mid/d2/5
ro_mid/d2/4
ro_mid/d2/3
ro_mid/d2/2
ro_mid/d2/1
ro_mid/d1
ro_mid/d1/four
ro_mid/d1/4
ro_mid/d1/3
ro_mid/d1/2

++ rsync --exclude=lost+found -aHSx --devices --specials --delete-before
--remove-source-files --exclude=.wh..wh.* --max-delete=0 rw/ ro_mid
cannot delete non-empty directory: d2
Deletions stopped due to --max-delete limit (10 skipped)
rsync error: the --max-delete limit stopped deletions (code 25) at main.c
(1060) [sender=3.0.7]

I don't think the '--max-delete=0' is the correct option to choose in
this case, but apparently it reveals that rsync does trying to delete
destination files.

My brain is now blocked. Anyone can give me some hints?

Thanks
--
Tong (remove underscore(s) to reply)
http://xpt.sourceforge.net/techdocs/
http://xpt.sourceforge.net/tools/
--
To UNSUBSCRIBE, email to debian-user-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Archive: http://lists.debian.org/i3ikm4$i09$***@dough.gmane.org
Sam Leon
2010-08-07 03:50:02 UTC
Permalink
Post by T o n g
Hi,
I believed that rsync wouldn't delete existing destination files unless
instructed so (by --delete-excluded). However, while debugging a
(mysterious) bug,
http://thread.gmane.org/gmane.linux.file-systems.aufs.user/2821/focus=2827
Post by T o n g
I now suspect that the rsync command used in the script was trying to
delete existing destination files.
$ find rw
rw
rw/d1
rw/d1/.wh.7
rw/d1/.wh.6
rw/.wh..wh.orph
rw/.wh..wh.plnk
rw/.wh..wh.aufs
$ find ro2
ro2
ro2/d2
ro2/d2/6
ro2/d2/5
ro2/d2/4
ro2/d2/3
ro2/d2/2
ro2/d2/1
ro2/d1
ro2/d1/.wh.2
ro2/d1/.wh.3
ro2/d1/four
ro2/d1/.wh.4
. . .
++ rsync --exclude=lost+found -aHSx --devices --specials --delete-before
--remove-source-files '--exclude=.wh..wh.*' rw/ ro2
. . .
$ find ro2
ro2
ro2/d1
ro2/d1/.wh.6
ro2/d1/.wh.7
See, all the previous content have gone.
(http://thread.gmane.org/gmane.linux.file-systems.aufs.user/2821/
focus=2827)
$ find rw ro_mid
rw
rw/d1
rw/d1/.wh.7
rw/d1/.wh.6
rw/.wh..wh.orph
rw/.wh..wh.plnk
rw/.wh..wh.aufs
ro_mid
ro_mid/d2
ro_mid/d2/6
ro_mid/d2/5
ro_mid/d2/4
ro_mid/d2/3
ro_mid/d2/2
ro_mid/d2/1
ro_mid/d1
ro_mid/d1/four
ro_mid/d1/4
ro_mid/d1/3
ro_mid/d1/2
++ rsync --exclude=lost+found -aHSx --devices --specials --delete-before
--remove-source-files --exclude=.wh..wh.* --max-delete=0 rw/ ro_mid
cannot delete non-empty directory: d2
Deletions stopped due to --max-delete limit (10 skipped)
rsync error: the --max-delete limit stopped deletions (code 25) at main.c
(1060) [sender=3.0.7]
I don't think the '--max-delete=0' is the correct option to choose in
this case, but apparently it reveals that rsync does trying to delete
destination files.
My brain is now blocked. Anyone can give me some hints?
Thanks
--delete-before will delete everything in the destination directory that
is not in the source directory. Also using regex stuff for --exclude can
be tricky because it is a different syntax than what most people are
used to with bash shell.

Sam
--
To UNSUBSCRIBE, email to debian-user-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Archive: http://lists.debian.org/***@net153.net
T o n g
2010-08-07 04:10:01 UTC
Permalink
Thanks for your replay Sam.
Post by Sam Leon
--delete-before will delete everything in the destination directory that
is not in the source directory.
As you sure about this?
--delete-before receiver deletes before transfer (default)

I.e., "--delete-before" is the default action. It only affect the files
to be copied over. Nothing else.

--delete-before

Request that the file-deletions on the receiving side be
done before the transfer starts. See --delete (which is
implied) for more details on file-deletion.

Deleting before the transfer is helpful if the
filesystem is tight for space and removing extraneous
files would help to make the transfer possible.
However, it does introduce a delay before the start of
the transfer, and this delay might cause the transfer to
timeout (if --timeout was specified). It also forces
rsync to use the old, non-incremental recursion
algorithm that requires rsync to scan all the files in
the transfer into memory at once (see --recursive).
--
Tong (remove underscore(s) to reply)
http://xpt.sourceforge.net/techdocs/
http://xpt.sourceforge.net/tools/
--
To UNSUBSCRIBE, email to debian-user-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Archive: http://lists.debian.org/i3ilu9$jqc$***@dough.gmane.org
Henrique de Moraes Holschuh
2010-08-07 12:40:02 UTC
Permalink
Post by T o n g
I.e., "--delete-before" is the default action. It only affect the files
to be copied over. Nothing else.
rsync --delete a b c d 192.168.1.99:/tmp/

could remove a b c or d from /tmp in 192.168.1.99, if they don't exist
in the source location.

If a, b, c or d are directories, it will act on all files and
subdirectories inside them in 192.168.1.99:/tmp as well.
--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh
--
To UNSUBSCRIBE, email to debian-user-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Archive: http://lists.debian.org/***@khazad-dum.debian.net
Loading...