Preventing ATV from Stretching Videos...

There's 2 possibilities for "stretch" here...

1) If you mean stretching to fill your HDTV screen, that's more likely a TV (menu) setting called "stretch" or "fill" or similar. The same would be happening when watching any old non-HD channels, such as those that show classic television or old sporting events not shot in HD. If those too are filling the screen, it's likely you have a TV setting set to do that. Check your settings.

2) If you mean that the visible ratio remains the same (square instead of rectangular) but you just don't like the "blur" when 480-line video is dynamically (more than) doubled to fill a 1080-line HDTV screen top to bottom, you could re-render the video inside of a 1920X1080p "black bars" box. An easy way to do this would be to make a single slide in Keynote that is 1920X1080 and then center the 640X480 video in the center. Set the video to play and save that out as a Quicktime video.

The result is going to be the "black bars" effect on all sides (a black bars doughnut if you will), filling the spaces with blank black bars where you have less video than width or height of screen. On your TV, it is going to look like picture in picture... or like you are watching a little TV screen within a bigger TV screen. BUT, you will be seeing the 640X480 video at it's full resolution (without the TV inventing pixels to try to scale it up to fill the 1080p lines it actually has).

If you are like most people, you probably won't like the doughnut approach, which then points toward the concept of just learning to live with the concept that lower resolution video is going to show on higher resolution screens... just as it does with TV classics and/or classic sports broadcasts.

One last thing: if those home movies were shot on film, there is one other option. Even old home movie film has higher resolution than (perhaps) it's past conversions to DVD or VHS tape. There are services that will scan the film at very high resolution and basically make old films look like HD quality. This is how older movies & TV shows shot on film (instead of tape) can be remastered into HD-quality video. So if you have reels of film (not video tape) home movies, you could use a service to maximize the quality of those so they will look better at greater than 480 lines.

 

I doubt you'll be happy with the "doughnut" (black bars on all sides). While it will show the video in it's native resolution, it will just look odd to see a relatively tiny screen within a bigger screen TV. Think of all the complaining people do about the "letter boxing" effect when watching movies shot wider than 16:9 (and thus seeing black bars above & below) and/or pillar boxed video with black bars left & right of the picture... and then imagine your video with BOTH of those in play.

You might want to render just a minute of one video and have a look before you go to a lot of trouble doing many home movies. You may decide that the lack of HD-like sharpness but a fuller screen is a more desirable compromise.

And I'm assuming you've rendered your old home movies as good as possible, meaning applying stuff like Decomb to deal with interlaced video and so on. If some of what you seeing isn't solely from inventing pixels (to scale up to HD resolution), you might want to go back to your master files and work on re-rendering them for non-interlace playback. A Decomb pass can make a big difference with interlaced video.

 

It's probably not the ATV doing the stretching. It's probably you TV. On the TV remote look for a button labeled "picture" (for example) and press it a couple times to see what the result is.

 

Trying to understand here… you don't say what the videos look like when you watch them on a 16:9 or 16:10 screen directly connected to your Mac in iTunes. Do they have the correct 4:3 aspect ratio with a black bar to the left and right (pillar boxed)? If so, then something is probably set wrong on your TV, as the others have suggested.

But if they also fill the screen on the Mac, then there's a problem in the way you prepared the video. I have been working with a lot of legacy 4:3 video that I shot 15 years ago. They are properly pillarboxed on my Mac and also on my AppleTV. If you used Handbrake to process the video for iTunes, you probably had the wrong anamorphic settings.

 

As others have said before, I also have never seen aTV change the aspect ratio of my old movies or tv-shows, that are in 4:3.
About pixelation there is nothing you can do. You basically do not have the information already in your source picture, so it will be simply impossible to re-invent it afterwards.
IMHO you can do 2 main things:
a) sit back and look at greater distance (that's what we did in the SD era anyway)
b) re-encode your source without stretching at 1:1 pixel size (ie leave the picture at 640x480) and add black frame around it to fill the 1080p pixel height.

PS aTV does not switch between video modes, it keeps the output at whatever you've set in the preferences (eg. 1080p) and simply fills the height of the screen with whatever picture you give it, maintaining original aspect ratio. It handles this actually much better, than my telly with androidTV.

 

I think my legacy 4:3 looks very good on the AppleTV, but it was shot on high quality camera (Sony VX-2000). Now I am starting from the original 720x480 DV however, not 640x480. Without knowing the source of your video and how it was originally created, it's hard to make recommendations. The decomb and deinterlace setting are intended for interlaced video, which is what my source video is. I use the deinterlace slower setting in Handbrake and it does a nice job.

I would not call my 4:3 video "pixellated", however it looks a bit "out of focus" compared to HD. If your video is not interlaced, you don't need to use either the decomb or deinterlace settings.