The issue is that our hand interactions are generally with real physical objects we can touch, which don’t exist in VR. That's why we still use physical keyboards for real productivity (even the keynote highlighted the use of physical keyboards and trackpads with the Vision Pro), physical controllers for serious gaming, steering wheels and other physical controls in cars, etc. Hand gestures are inherently more ambiguous, less precise, have no physical feedback in VR, and thus only get you so far. It's also for a reason the Vision Pro has a digital crown.