The different USB connector designs were created to reflect changing technology, the devices in question, and new use cases over the course of years. The development of USB started in 1994, with 1.0/1.1 only offering the standard type A and B ports. Mini-USB connectors came out in 2000, when cameras and other devices became more common; Micro-USB in 2007, when thinner devices demanded a smaller connector. Compared to the alternative before USB, while there might be madness today, overall it was much less so than before.
While you might be able to make USB-C connectors smaller than they are now, I wouldn't bet on it anytime soon. USB-C is a reversible 24-pin connector in a form-factor that's only slightly larger than Micro-USB. That means there are more, smaller pins in a tighter configuration. The metal sheath is especially important in that context to prevent damage. A significantly smaller connector would be more likely to be physically damaged, with fewer insertion and removal cycles for a shorter expected lifetime. So while it's certainly physically possible, it's unlikely when other engineering goals such as durability are considered.
By comparison, while Lightning connectors are slightly thinner, the actual receptacle in the device itself probably makes it a moot point for the most part as the receptacle and plug pinouts are physically reversed relative to USB-C. Meaning, the USB-C plug encloses the receptacle pinout whereas the Lightning receptacle pinout encloses the plug.
Lightning receptacle:
USB-C receptacle:
Anyhow, you can't really compare this so-called 'Ultra Accessory Connector' Apple may be working on to USB-C. It's a slightly less wide 8 pin connector that's functionally limited by design. USB-IF isn't going to publish a similar spec because if they did, it would no longer be a universal cable design, USB 3.0 or otherwise. It's a proprietary port for a specific type of accessory input. That's literally the opposite of the entire idea behind USB.
It'd be limited and specific, not universal.
The only madness here is Apple's need to replace the headphone jack with nothing, and now, a proprietary accessory connector with limited functionality in an effort to correct a problem that didn't exist. It's a return to the worst of the 90s, when peripheral use required the inclusion of everything from PS/2 to serial ports, giant parallel ports, DA-15 ports, and more in order to give consumers a bad form of flexibility.