when you actually open up your "downloads" folder into a Finder window, does it open up properly in all of the "view" modes (quick view, list, column and icon)?
It might be trying to render a pdf or another file that it is having a hard time with, and thus is closing the "window" that is actually the live preview icon. I had a PDF manual that was fine in 10.4 and opens up in preview and acrobat that would crash my finder windows when i tried to open the enclosing folder. i didn't try it, but it would probably also have killed a stack...since the stacks show live icon previews. I can't test this because I reverted to Tiger until (if?) Apple fixes all of Leopards bigger failures. I got tired of slogging around with an awkward and ineffective "feature" (stacks) and spaces just wasn't good enough to justify keeping Leo around. can't use time machine because my main external drive needs to be PC and Mac accessible, and several of my most desired features were scrapped again (resolution independence?!?)
Anyway, I'm rambling. Try my suggestion. It could be that the delay in crashing is a result of the OS trying to render a file for quick view icons that it can't digest for some reason.