Open activity Monitor, set it to show all processes, sort by RAM used, and post a screenshot that shows the list of processes and the pie chart of RAM used. Hopefully that will reveal the problem.
OS X is leaking RAM for me. If I close all my processes, Activity Monitor will still show several GB in use (not counting inactive). Logging out helps a little, but only rebooting completely resets it. I have only 46MB free and 2.33 GB inactive right now, and if I try to open something that takes a lot of RAM, some of the inactive RAM will be swapped to the disk, even though it sounds like it should just be instantly dropped based on the definition of inactive RAM. I have 6GB of RAM, and the total used by all my programs probably never exceeds 3-4GB worst-case, yet I still have to wait for stuff to swap out to the disk sometimes.
