Maybe it’s just the lighting? My iPhone X white silicone cases appears off white under fluorescent lights at my office, but it’s pure white after about a month of regular use.
Silicone can stain, so, could see oils from one's skin and other matter getting onto the case, especially white. Here's a recipe for removing stains from silicone.
Everything you own that's white becomes less white with handling. Consider white shirts or socks. That's how the world works. Just use your case. If you cease liking it, get a different case.
There are also many shades of white during manufacturing process too. But you have to really compare to notice anything. I don't think it's worth the trouble. If you notice yellow spots on the same case and can't rub it off then that might be something to concern.