Got the same problem here
i have the same problem in Tiger too.
Never occur in previous versions of OS X
Try this:
1. Reduce dock to the smallest size.
2. Turn dock's magnification on.
3. Open application from dock.
4. Minimise the windows of the application.
5. Increse the dock size to the maximum size (leaving the minimised window in the dock alone)
6. Roll over the mouse pointer over the minimised window found in the dock, using dock magnification effect.
7. Observe that the application icon is blurred (noticably poor quality).
8. To avoid this issue, one has to increase the dock size to a certain size, and the application icon will look okay when minimised in the dock.