The "volume" setting in VLC is actually just a kind of a boost that lets you raise volume above 100%, but quality suffers slightly when you do this, so it's not comparable with general system volume in OS X. You can't make the maximum OS X volume level louder.
But since you are only concerned about iTunes, you can use a similar boost as in VLC - open the iTunes equalizer (under window menu), turn it on by checking the box and play with the preamp. Hope that helps.