The first macOS 26.7 Release Candidate revealed that Apple's upcoming AirPods will come with color cameras that can take 1MP pictures.
There's more to the story. Code in the same update suggests that the cameras may be used to improve head-tracked Spatial Audio by recognizing the physical playback screen. There are also hints about images being sent to Apple's servers to answer visual questions.
According to the code, during video playback with head-tracked Spatial Audio, the unreleased AirPods will use their cameras to find the physical screen you're watching on. The playback device can then compare the screen's direction with the direction estimated by the AirPods' motion sensors. If the two estimates are close enough, the playback device may make small adjustments to reduce drift.
The code includes five screen types the unreleased B790 AirPods model may be able to identify:
During the video playback, the system looks for the screen that best matches the previously selected one. It may ignore camera data when the detected screen's location seems improbable based on motion sensor data from AirPods.
Other details suggested by the code:
Visual questions about what the AirPods cameras captured will be processed by Apple's Private Cloud Compute rather than on-device models. That's how Visual Intelligence requests from iPhone or Mac are processed today.
Links to code in macOS 26.7 RC2 so you can verify the claims or explore further on your own: [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]. They're from blacktop/ipsw-diffs on GitHub. They do amazing work and you should go give them a star.
The hero image shows manually added boxes around the screens. These boxes were then passed to CoreMotion code in macOS 26.7 RC1, which selected the screen belonging to the playback device. The screens would be identified earlier in the process, probably on the AirPods themselves. Original image by KATRIN BOLOVTSOVA on Pexels.
There's more to the story. Code in the same update suggests that the cameras may be used to improve head-tracked Spatial Audio by recognizing the physical playback screen. There are also hints about images being sent to Apple's servers to answer visual questions.
Improved head tracking for Spatial Audio
Current AirPods models use motion sensors to offer dynamic head tracking. Over long viewing sessions, the estimated position of the wearer's head can gradually drift.According to the code, during video playback with head-tracked Spatial Audio, the unreleased AirPods will use their cameras to find the physical screen you're watching on. The playback device can then compare the screen's direction with the direction estimated by the AirPods' motion sensors. If the two estimates are close enough, the playback device may make small adjustments to reduce drift.
The code includes five screen types the unreleased B790 AirPods model may be able to identify:
- Laptop
- Monitor
- Smartphone
- Tablet
- TV
During the video playback, the system looks for the screen that best matches the previously selected one. It may ignore camera data when the detected screen's location seems improbable based on motion sensor data from AirPods.
Other details suggested by the code:
- Screen detection only activates for video, not music or other audio content.
- On devices with Face ID, Apple already uses front camera sensors to position the wearer's head position relative to the screen. Screen detection with AirPods cameras is preferred when available.
- When processing begins, the system may assume that you're looking at the playback screen.
- Screen detection pauses when there's a lot of movement, such as sitting, standing, or walking.
- It may also pause when more than one external display is detected in front of you.
Visual Intelligence
There are more clues that AirPods with cameras will support Visual Intelligence features. The code includes a reference to server-based model:
Code:
com.apple.fm.language.instruct_server_v1.audio_accessory_visual_reasoningVisual questions about what the AirPods cameras captured will be processed by Apple's Private Cloud Compute rather than on-device models. That's how Visual Intelligence requests from iPhone or Mac are processed today.
Smart handoff
Similar to automatic audio switching, processing of visual requests may move from device to device as audio routing changes.Links to code in macOS 26.7 RC2 so you can verify the claims or explore further on your own: [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]. They're from blacktop/ipsw-diffs on GitHub. They do amazing work and you should go give them a star.
The hero image shows manually added boxes around the screens. These boxes were then passed to CoreMotion code in macOS 26.7 RC1, which selected the screen belonging to the playback device. The screens would be identified earlier in the process, probably on the AirPods themselves. Original image by KATRIN BOLOVTSOVA on Pexels.
Last edited:
