Obviously you can't use the system memory to extend the RAM of the graphics card, as these use different technologies and are physically connected in a completely different way. Nevertheless, the system memory is automatically used for caching etc. so the memory is being "shared" in a way.
All that besides, 512Mb should be more than enough for a flight simulator like X-plane, even on the native 27" resolution of the iMac. Judging from the screenshots, I very much doubt that it uses high-resolution multi-layer textures or other stuff modern games use which eats lots of GPU RAM. So if you have performance problems, I doubt that increasing the GRU RAM will help.