Back in the '70s, I remember hearing that there were originally 10 episodes to the Star Wars saga. That may be a mis-remembering from an 8 year old mind, or my brother simply quoting something he heard at school.
One thing I like about the Harry Potter story is the continuity, and I think (I have no evidence) that the drawing out of the story of the major events from beginning to end helps with ensuring continuity. The minor events (like Hermione punching Malfoy or Snape torturing Harry in various classes) are fun "fluff" and add to the character development, but as long as the major events are true, there is a good story there.
(this paragraph is strictly opinion - I have no facts to back them up) I just get the feeling that each set of episodes (1-3, 4-6, 7-??) are set on their own terms, and borrowing from the future (1-3) or past (7-??) out of context. They're building the story as they go, and that's akin to building a tower, and drawing the plans up as they go. I know our lives are like that, but a story is a retelling of a set of known events, and that is why the new episodes seem disjointed. Heck, even the Harry Potter movies stray from the books quite a bit (I do like the final battle between Voldemort and Harry much better in the book than the movie... the movie was too much of a Darth/Luke battle...)
Lastly, I like your analysis of the why the movies are the way they are. I hadn't thought about the familiarity aspect of this, and getting a new generation in. I was an 8 year old when Star Wars first came out, and loved it! I always thought of them as a throw back to when I was that kid in the theater, and even watching them last Christmas with my 7 kids (and their husbands/boyfriends), I had that feeling again, even at the end. My 6 year old son loved it too, so I really think you have something there.