The following will
approximate it (minus the visual and audio cues that it's changed):
Code:
set currentVolume to output volume of (get volume settings) -- in the range 0 to 100 (integer)
set newVolume to round (currentVolume * 64 / 100) -- convert to the range 0 to 64 (float)
if newVolume < 64 then set newVolume to newVolume + 1 -- increment the volume
set volume output volume (newVolume * 100 / 64) -- convert back to the range 0 to 100
The technical details are below, but they're not important if you're not too concerned about details.
Unfortunately it's more complicated than it should be. The actual system volume is an integer from 0 to 100, but the normal range with the up/down volume keys is 0 to 16, and when using option-shift-volume up/down the range is effectively 0 to 64 (as there are four increments for each square). As the actual volume is stored as an integer, numbers don't convert cleanly between ranges, so for example 0.25 squares is volume 1, 0.5 squares is volume 3, 0.75 squares is volume 4, etc. Apple's mapping between the two ranges doesn't seem to follow an obvious formula (a quarter of a square increase can correspond to an actual volume increase of 1, 2, or 3), so the above won't match it perfectly, but will be fairly close.