If it's the latter, well, it isn't totally impossible, depending on how you access the app. For instance, I have GIMP on my computer (the GIMP.app version), which is an X11 app. I can launch it from Terminal using 'open gimp.app' just like any other app.....
As for laynching xapps that are direct executable files, and not .app containers, this is also possible, although you need to do a couple of extra things:
1) Set the display variable:
DISPLAY = :0.0
export DISPLAY
2) Add the bin directories to your path (optional)
3) You can then run xapps such as xcalc and xeyes, which are found in /usr/X11R6/bin/ from the Terminal just as you would from an xterm.