I have written many CLI shell scripts designed to run on all *nix platforms. One script I wrote last week didn't work as expected under Mac OS X (Darwin 6.1). I tracked the problem down to a glitch that I describe below.
Because /bin/sh under Mac OS X, Linux, and certain other *nix versions is bash instead of the original Bourne shell, my scripts must work under both shells in order to be portable. Since bash is generally upward compatible from Bourne, I'm usually ok using Bourne shell conventions, i.e., the least common denominator. But not this time.
*** THE PROBLEM ***
In a Bourne-style shell script, I expected the test command expression
[ ! expr1 -o expr2 ]
to be evaluated logically as
( not expr1 ) or ( expr2 )
This assumption is correct under the Bourne shell, Korn shell, and Z shell. It is true under the GNU Bourne-Again shell (bash) when either expr1 or expr2 contains another operator such as ! or -f, e.g.,
[ ! expr1 -o ! expr2 ]
[ ! -f filename1 -o expr2 ]
[ ! expr1 -o ! -d directory2 ]
...
But the assumption is not correct under bash when expr1 and expr2 are both simple environment variable references, e.g.,
[ ! "$MYVARIABLE1" -o "$MYVARIABLE2" ]
In this last case, bash interprets the expression as
not( expr1 or expr2 )
which produces the wrong result when expr2 is TRUE. Specifically, when "$MYVARIABLE2" is nonblank the result of the expression will be FALSE instead of TRUE.
*** WHAT THE UNIX STANDARD (POSIX) SAYS ***
I checked the POSIX (IEEE Std 1003.2) rules for the test command. Earlier standards required only that -a have precedence over -o, but did not require that ! have precedence over both of them. Nevertheless, the original Bourne shell and many of its descendants were coded to follow a strict set of priorities:
first: parenthesized expressions
next: binary operations other than -a and -o
next: unary operations other than !
next: string length
next: !
next: -a
last: -o
The most recent POSIX standard, however, defines the bash-style behavior for a test expression with four operands. As I discovered, some shells behave one way and some the other way. Under Solaris 9, the man test page describes the interpretation of ! expr1 -o expr2 the new POSIX way, but the test command built into the Solaris 9 Bourne shell actually does it the old way!
*** WORKAROUNDS ***
No matter what the evolution of the problem, I needed to solve the problem. There are many workarounds available, but one seemingly obvious one doesn't work.
Workaround that doesn't work:
* Use escaped parentheses, e.g., [ \( ! expr1 \) -o expr2 ]
This works for bash but parentheses in test expressions are not supported by the Bourne shell or older versions of the Korn shell, so this solution is not portable.
Workarounds that work and are portable:
* Reverse the logic: [ expr2 -o ! expr1 ]
* Don't use the NOT operator when expr1 is a simple variable: [ "$MYVARIABLE1" = "" -o expr2 ]
* Use separate test expressions: [ ! expr1 ] || [ expr2 ]
In this last case, the semantics of evaluation are not the same and performance could actually be better or worse, since test is invoked twice but expr2 may not need to be evaluated.
*** CONCLUSION ***
I diagnosed the problem, found a solution, and tested the solution under 9 Unix versions, so I'm confidant that I have identified and solved the problem.
What an annoyance to encounter such glitches, have to find and test workarounds, and then have to change all my affected shell scripts.
Has anyone else encountered similar shell script porting problems? Such problems are not unique to Darwin, but Mac scripters who share solutions with other Mac scripters can save them time and trouble!
Because /bin/sh under Mac OS X, Linux, and certain other *nix versions is bash instead of the original Bourne shell, my scripts must work under both shells in order to be portable. Since bash is generally upward compatible from Bourne, I'm usually ok using Bourne shell conventions, i.e., the least common denominator. But not this time.
*** THE PROBLEM ***
In a Bourne-style shell script, I expected the test command expression
[ ! expr1 -o expr2 ]
to be evaluated logically as
( not expr1 ) or ( expr2 )
This assumption is correct under the Bourne shell, Korn shell, and Z shell. It is true under the GNU Bourne-Again shell (bash) when either expr1 or expr2 contains another operator such as ! or -f, e.g.,
[ ! expr1 -o ! expr2 ]
[ ! -f filename1 -o expr2 ]
[ ! expr1 -o ! -d directory2 ]
...
But the assumption is not correct under bash when expr1 and expr2 are both simple environment variable references, e.g.,
[ ! "$MYVARIABLE1" -o "$MYVARIABLE2" ]
In this last case, bash interprets the expression as
not( expr1 or expr2 )
which produces the wrong result when expr2 is TRUE. Specifically, when "$MYVARIABLE2" is nonblank the result of the expression will be FALSE instead of TRUE.
*** WHAT THE UNIX STANDARD (POSIX) SAYS ***
I checked the POSIX (IEEE Std 1003.2) rules for the test command. Earlier standards required only that -a have precedence over -o, but did not require that ! have precedence over both of them. Nevertheless, the original Bourne shell and many of its descendants were coded to follow a strict set of priorities:
first: parenthesized expressions
next: binary operations other than -a and -o
next: unary operations other than !
next: string length
next: !
next: -a
last: -o
The most recent POSIX standard, however, defines the bash-style behavior for a test expression with four operands. As I discovered, some shells behave one way and some the other way. Under Solaris 9, the man test page describes the interpretation of ! expr1 -o expr2 the new POSIX way, but the test command built into the Solaris 9 Bourne shell actually does it the old way!
*** WORKAROUNDS ***
No matter what the evolution of the problem, I needed to solve the problem. There are many workarounds available, but one seemingly obvious one doesn't work.
Workaround that doesn't work:
* Use escaped parentheses, e.g., [ \( ! expr1 \) -o expr2 ]
This works for bash but parentheses in test expressions are not supported by the Bourne shell or older versions of the Korn shell, so this solution is not portable.
Workarounds that work and are portable:
* Reverse the logic: [ expr2 -o ! expr1 ]
* Don't use the NOT operator when expr1 is a simple variable: [ "$MYVARIABLE1" = "" -o expr2 ]
* Use separate test expressions: [ ! expr1 ] || [ expr2 ]
In this last case, the semantics of evaluation are not the same and performance could actually be better or worse, since test is invoked twice but expr2 may not need to be evaluated.
*** CONCLUSION ***
I diagnosed the problem, found a solution, and tested the solution under 9 Unix versions, so I'm confidant that I have identified and solved the problem.
What an annoyance to encounter such glitches, have to find and test workarounds, and then have to change all my affected shell scripts.
Has anyone else encountered similar shell script porting problems? Such problems are not unique to Darwin, but Mac scripters who share solutions with other Mac scripters can save them time and trouble!
