You don't have to keep any icons on your desktop for Firefox to run.
Double-click the .dmg file. That will open a disk image that contains the application.
Drag the application icon to the Applications folder in your sidebar. That will copy it into that folder. Once the copy is finished, you can eject the Firefox disk image by clicking the eject button next to it into the sidebar. From there, you may delete anything you've downloaded for it.
Double-click the Firefox icon in the Applications folder to start it and you may right-click on the dock icon to keep it there. If you'd like it to be set as your primary browser, you need to start Safari and go into its preferences to set that.