That’s the only downside, but both the D300 and D500 aren’t great performers anyway.
I’ve seen E5-1650 V2 on eBay for the $125 range, meaning a 6 core for $2K. Add another $250 for 64GB ECC, and $450 for a 1TB SSD (with the $13 adapter to use standard NVMe drives) and for $2700 you have a competitive system for most purposes.
CPU prices are dropping nicely. The E5-2667 V2 (8 core 3.3) is going for as low as ~$300 which is a steal.