terrible problem with scrolling in Lion (not reverse or natural scrolling related)

Since I side-graded to OS X Lion, my mouse behavior completely detoriated.
I have immediately set up all mouse settings, to replicate the known behavior under Snow Leopard as a start.

There remains one issue though, that drives me nuts, working quickly with Lion:

When working in several windows with Lion, moving the mouse curser to a different window and scrolling that window will in 90% of the time result in scrolling within the old window, which is totally unwanted and messes up working quickly entirely.

When working in image processing software like Photoshop or Lightroom, this in fact makes me furious, as moving from the image zoom view to a palette, wanting to scroll down the palette quickly, to grab a tool will result in panning my image completely out of view.

I have so far not found a solution and endless googling for this issue only brings up the trillion of threads about the (un)natural scrolling.

Does anybody know about a solution on this? This issue seriously slows me down.

I am using a MacBook Pro 15" Penryn (early 2008) with Magic mouse and OS X Lion 10.7.2 installed.

The problem occurs systemwide with all software - it seems, that the change in focus for scrolling from one window to another window is delayed and lags behind the visual hovering of the mouse pointer above the wanted window.