There are a lot of things that can be using a drive. Applications, of course, but Spotlight, Time Machine can tie it up creating indexes.
If you start a terminal session, cd /Volumes, do an ls to see what is there, then fuser "Volume Name" it will give you the process name which can look up in Activity Monitor.
But as above, simple way is to shutdown and then disconnect the drive.