Problem with actual magnetic tip of MagSafe?

Hey, I did a quick Google and Forum search but didn't really find my exact problem. The actual magnet to my L-Shaped MagSafe isn't really attached to the charger anymore. It just happened one day — I unplugged the charger and then realized that the magnet was still stuck to the laptop. I can get the magnet out of the jack by inserting the charger and removing it quickly (which keeps the magnet on the actual MagSafe), but this is a less than ideal situation.

Everything I found was about actual tears or rips with the charger rather than this. Perhaps someone here can point me in the right direction of how to go about fixing it. Any help is greatly appreciated!

I attached an image so you can understand what I meant.

 

Thanks! I was thinking of using glue but wasn't sure. Will have to be extra careful if/when I do it