Mac’s website says that multiple monitors are only supported over TB.
Pretty sure you can connect two up-to-4k displays directly to the MBPs Thunderbolt
ports using the approriate USB-C to DisplayPort cable- or one to Thunderbolt and one to HDMI (if your display has a HDMI input). It gets more complicated with 5k/6k displays...
You may want a dock for other reasons, but I don't think it's compulsory for dual displays.
Anyway, it's a lot cheaper to "try it and see" with a $20 USB-C to DP cable than a $300 dock!
Does this mean that any dock that has a TB 3/4 output to the MBP will support extending displays, regardless of how the monitors are connected to the dock (i.e., HDMI, USB C, DP)?
If you are talking about 4k@60Hz displays then mostly, yes.
That said, you should check the specs for the particular hub/dock you are considering - mostly there's a table on the product home page, and if that says it will support dual 4k displays on a M2 Pro Mac then you should be in business. Sometimes that will mean using the dock's HDMI connector for one display and a USB-C-to-whatever cable on the TB3 "downstream" port for the other. But
do check - some "multi display" dock/hubs aimed at PCs rely on DisplayPort daisy-chaining (which can only 'mirror' on Mac), others use DisplayLink (useful with a non-pro M1/M2 but second-class) or might describe a non-TB hub as "Thunderbolt compatible".
If you don't need things like Ethernet, SD cards, HDMI, the CalDigit Elements hub, plus a USB-C to DisplayPort cable, should do the trick.