Can't delete "Install Xcode" icon.

So I downloaded Xcode and installed it, however, now I cannot delete the "Install Xcode" icon. I try dragging it to the trash and it just goes back to where it was, I try holding down the icon so I get the X up in the corner, I click the "X" and it asks me something like "Are you sure you want to delete this app" and I hit delete but nothing happens, it's still there.

Anyone else have this problem? Anyone help me with a fix? Thanks in advance!!!