True. I wouldn’t be able to significantly increase my viewing distance even if I wanted to, because I’m limited by the depth of my desk.It all comes down to sitting at a work desk. People watch any given monitor on their desk from the same distance, no matter the size of the monitor, because a desk has a constant/constrained top area, and people need to have that within physical reach. In this regard desktop monitors are not unlike notebook monitors, which are mostly used at an arm's distance. Also this is the reason why display ppi -- a linear metric, works in the first place. In contrast to TVs, where people can just move their couch back and forth to optimize their view distance, and ppi is not used as a metric at all, as angular resolution is at play with TVs.
Thus in my setup, and in the setups of many if not most desktop monitor users, as you said viewing distance is effectively a constant regardless of monitor size.