Dont assume we haven't looked into our options. Unfortunately the technically correct solution is not always the 'right' solution. This is the required spec for machines according to the vendor. If we bought less, they would not support us to the level required by our external auditors. Your right that the CPU is also underutilize and the bottleneck is the SAN as datasets are continuously reloaded. Consolidation isn't an option due to licensing terms tieing back to physical cores on the host, even under VM. This means consolidation of hardware would actually result in higher licensing costs that would exceed the value of the under utilized hardware. One month of maintenance costs for the software already exceed the hardware cost. Fighting with the vendor doesn't make financial sence since if it delayed us by even a month, the point would be moot.
The case I mentioned is a financing exception where over-provisioning hardware was the cheapest option in the end. However the reason we can't ramp up the per-server performance is because of the memory management in Java. We have worked with Oracle to attempt to improve the situation, but the heap size just doesn't scale up much past 2G in a single Java thread.
They are working to change the software to run multiple instances, but the restrictions of the code make this difficult since the datasets will need to be largely duplicated in RAM.