This Xs is my first Face ID device and i set it up late friday night while half drunk in a dark restaurant, and it's been working fine for me ever since.
Upvoted because of your setup scenario
Nearest Genius Bar is 2 hours away. The FaceID even fails when I'm yawning, laying in bed, back down in bed, in the car, at my desk, everywhere.
Have you purged the existing FID data, and done a fresh setup? I recall when the X was first released, some people had inconsistent FID operation from their first face scan, but went back, did it again and it worked really well from that point.
FWIW, I was an early buyer for my X, one scan, I get about 98/99%, better than TID for me, works at 20-24" and reasonably off angle on a little desk stand, in bed (in near total darkness), occasionally in my car with the top down (mostly use CP).