I recall as a young kid, early teenager, that I did not like being around all of my friends at the same time. I had compartmentalized my friends into smaller groups and would go hang out with each group separately. The groups were 4 people or less usually. I remember one time that all of my friends were together at one place for a party or something. Perhaps there were 10 of us altogether; I don't fully remember everyone there anymore. But I do remember that I wasn't very happy and tended to retreat a little bit during that day because I just had too much to deal with, with everyone there.
That's one of my earliest memories that I can pin down that relates to introversion. It still holds true today that I don't like being around all of my friends, or co-workers all at once. I prefer small groups over large ones every time.