Click and drag problem

Ok, I'm still rather new to iMacs, but I have a weird problem that may just be that I don't understand how iMacs work. Say I have two finder windows open and I want to click and drag files from one finder window/directory to another finder window/directory. On a PC, I would just left-click the file and drag and drop it to the desired location. But on my iMac (with Magic Mouse) when I click the file and try to drag, the files doesn't drag. I have to click the file a second time to get it to grab the file for dragging. I can never seem to grab the file on the first click. Is this some setting somewhere or am I doing something wrong? It's not a huge deal, but it is somewhat frustrating.

 

Even when I am sure the window with the file I want is the active window, when I click a file or folder, the file for folder highlights as it should, but when I try to drag it while holding the mouse button down, it just doesn't drag the first time. I have to release the mouse key and go back and re-click the file a second time in order to drag it.

 

Well those videos look like I thought it should work but doesn't on my iMac. I'm wondering if the Magic Mouse setup has something to do with it?