A few hairs here and there aren't necessarily a sign of hair loss.
Are you eating well? Temporary hair loss can be a sign of certain deficiencies.
Make sure to be getting enough Biotin -- most multi-vitamins only contain about 10% of the RDA, so you might want to buy a Biotin supplement. B-vitamins in general are good for hair. In addition, you could buy a dedicated hair supplement -- many of them contain herbs (e.g., springtime horsetail) rich in silica.
In general, try to eat a balanced diet rich in protein, fruit and vegetables.
If this doesn't help, I would consult a dermatologist. And yes, chlorine is definitely not good for your hair as it dries it out. Make sure to rinse out chlorine well after swimming. You might be able to buy special after-swimming shampoo that is especially good at getting chlorine out. Condition hair after washing out chlorine.
Wear hair short.