Stupid bash shell expansion issue - help!

[TonyRPH]

I have a bash script which runs rsync - in this script are some exclusions set as part of a variable string.

If I run the script, rsync ignores the exclusions - BUT - if I run rsync from the command line, with the very same options, the exclusions work!

I've even tried running it, and capturing the output from ps -ef I grep rsync - and then run it with the command line shown in the output of ps and it works...

I have tried various methods including single quotes, double quotes, escaping some parameters etc. to no avail.

It simply refuses to honour the exclusions when run from a shell script.

Any ideas?

This is the relevant part of the script (which I'm sure PH is going to block with a 403 error)

As expected, my script was blocked, so it's here on Pastebin

ps output said:

root 24102 24100 44 12:47 pts/0 00:00:04 rsync -aAXHv --stats --exclude={lost+found,server-2012/*} --delete-before --skip-compress=3g2/3gp/3gpp/7z/aac/ace/amr/apk/appx/appxbundle/arc/arj/asf/avi/bz2/cab/crypt5/crypt7/crypt8/deb/dmg/drc/ear/gz/flac/flv/gpg/iso/jar/jp2/jpg/jpeg/lz/lzma/lzo/m4a/m4p/m4v/mkv/msi/mov/mp3/mp4/mpeg/mpg/mpv/oga/ogg/ogv/opus/png/qt/rar/rpm/rzip/s7z/sfx/svgz/tbz/tgz/tlz/txz/vob/wim/wma/wmv/xz/z/zip /storage/vol-1/ rsync://backup/vol-1/

[Hanslow]

At a guess, is it mismatching the single quotes on line 5, so treating '-aAXHv --stats --exclude={' as a string literal, and so on?

Try having

RSYNC_EXCLUDE={'lost+found','server-2012/*'}

with

rsync_opts='-aAXHv --stats --exclude="${RSYNC_EXCLUDE}" --delete-before --skip-compress='"$RSYNC_SKIP_COMPRESS"

Edited by Hanslow on Friday 3rd March 14:40

[TonyRPH]

Hanslow said:

At a guess, is it mismatching the single quotes on line 5, so treating '-aAXHv --stats --exclude={' as a string literal, and so on?

That was my initial thought, however if I echo the string out to the console it's correct (and rsync will actually run using that string) - likewise if I echo the string to a file it's correct (removes the unnecessary quotes etc.).

string said:

cat opts.txt
rsync -aAXHv --stats --exclude={lost+found,server-2012/*} --delete-before --skip-compress=3g2/3gp/3gpp/7z/aac/ace/amr/apk/appx/appxbundle/arc/arj/asf/avi/bz2/cab/crypt5/crypt7/crypt8/deb/dmg/drc/ear/gz/flac/flv/gpg/iso/jar/jp2/jpg/jpeg/lz/lzma/lzo/m4a/m4p/m4v/mkv/msi/mov/mp3/mp4/mpeg/mpg/mpv/oga/ogg/ogv/opus/png/qt/rar/rpm/rzip/s7z/sfx/svgz/tbz/tgz/tlz/txz/vob/wim/wma/wmv/xz/z/zip /storage/vol-1/ rsync://backup/vol-1/

I can run the above directly in the console and the exclude works. It's quite bizarre.

[Hanslow]

Have you tried with two --exclude= declarations? You should be able to supply multiple rather than relying on the bracket list expansion. It's a bit uglier but if it works for you it allows you to move forward.

As to why it works from the command line but not from the script, I can't think of any obvious reasons but I have to deal with oddities like that quite regularly where things might need escaping or syntactically slightly different in a script from what you'd type at the command line but I can't find any obvious fault in what you're doing.

Edited by Hanslow on Friday 3rd March 16:37

[TonyRPH]

Hanslow said:

Have you tried with two --exclude= declarations? You should be able to supply multiple rather than relying on the bracket list expansion. It's a bit uglier but if it works for you it allows you to move forward.
<snip>

I originally had separate exclude statements, and some Googling suggested using the --exclude with braces option.

I've with with Linux for many years (former sysadmin) and I'm totally baffled by this one!

[Hanslow]

Is this relevant? https://unix.stackexchange.com/questions/317162/rs...

If it is, try just removing all the single quotes from within the exclude curly braces.

[TonyRPH]

Hanslow said:

Is this relevant? https://unix.stackexchange.com/questions/317162/rs...

I believe I came across that post in my searches - it's the exact same issue, but the suggested fix doesn't work for me (vanilla Debian 10.13).

I've even look for unusual shell options that I may have set at some time. but there's nothing apart from the usual aliases e.g. l='ls -l' etc.

[Hanslow]

Have you ran it through strace both from the command line and also the script and see if that throws up any insights to how it's being interpreted differently?

[wombleh]

Could try just excluding "server-2012" rather than "server-2012/*", those star matches sometimes behave differently in scripts. Should still do the same thing.

[TonyRPH]

Hanslow said:

Have you ran it through strace both from the command line and also the script and see if that throws up any insights to how it's being interpreted differently?

I haven't actually - I'll try that.

wombleh said:

Could try just excluding "server-2012" rather than "server-2012/*", those star matches sometimes behave differently in scripts. Should still do the same thing.

I've tried variations of server-2012 with trailing slash, leading slash. full path, wildcard, you name it! No change.

As I say - the exclusions work fine from the command line (no matter I run it on the filesystem as I'm providing full paths).

It really is bizarre.

I found a couple of posts on the Arch forums with the same issue.

Hanslow said:

At a guess, is it mismatching the single quotes on line 5, so treating '-aAXHv --stats --exclude={' as a string literal, and so on?

Try having

RSYNC_EXCLUDE={'lost+found','server-2012/*'}

with

rsync_opts='-aAXHv --stats --exclude="${RSYNC_EXCLUDE}" --delete-before --skip-compress='"$RSYNC_SKIP_COMPRESS"

Edited by Hanslow on Friday 3rd March 14:40

I missed your earlier edit - I'll try this as well.