Public health here. While you're right, getting the flu isn't a big deal for the majority of people, to some it is: the young, the elderly, and the immunocompromised. Some recent strains have even been particularly harmful to healthy adults. While its your own choice not to get the flu vaccine, please do not discourage your grandparents, or people from vaccinating their children.
The flu virus cannot get you sick since it is an inactivated (aka "dead") virus.
One reason it seems the flu vaccine gets you sick is that you get it during flu season. It'll take weeks for your body to develop antibodies after the vaccine, so if you come in contact with the virus shortly before or after getting it its onset will make it appear as though the vaccine caused it. Another likely culprit is that it could be a bad cold, since cold viruses are also very prevalent during flu season. In addition, while normally limited to a sore arm, sometimes just the immune response caused by the vaccine is rough enough to be interpreted as a sickness.