Yes, quite expensive. L1-L3 caches all reside on the same die for the CPU's in question. The die is the piece of silicon contains the entire processor. These caches have to be really close to the CPU, since at the clock rates we're talking about (2.6 GHz), light would only travel 11.5cm (4.5 inches), and in that time, not only do the electrons have to move through lots of tiny wires, lots of gates have to turn on and off. There ain't no time to travel off the die.
Here is a a diagram of the a Sandy Bridge processor:
http://www.brightsideofnews.com/news/2009/7/5/intels-anti-amd-fusion-sandy-bridge-cpu-tapes-out.aspx
Notice that about 1/4 of the space is spent on the L3 cache (the L1 caches are even closer, and part of each Core). That's 1/4 of the space actually costs more than 1/4 of the fabrication cost of that die, because as die size increases, price increases exponentially. If you care, this is due to the fabrication yield:
http://en.wikipedia.org/wiki/Semiconductor_device_fabrication
Processors, in order to hit the high clock rate and keep electricity usage down, tend to use the smallest transistor sizes that are available. This increases costs in two ways: 1) The fabrication process is newer, and less available. 2) The silicon and fabrication process has to be even more error free, since smaller defects will cause problems.
This is all to say, yes. It's expensive. Those 32GB micro SD cards are much easier to make than that tiny 3MB of L3 cache.