Leave the 4GB sticks where they are - either slots 1 and 3 or slots 2 and 4. Put the 32GB in one of the other slots. Not ideal, but the extra RAM will make your system more responsive. Better would be to add a pair of identical RAM sticks.
This arrangement will result in 4GB in one channel, and 36 GB in the other channel. As a result only 8GB will run in dual channel mode, and 32GB in single channel mode (half speed). I agree, not ideal. The arrangement I suggested is also not ideal, but is slightly better.
I agree, adding a pair of identical sticks is ideal.
I have experience with 2014 and 2020 iMacs with various arrangements of sticks. 2020 iMac is the most finicky. Sorry, I do not have direct experience with all model years of iMacs. However, the principle is (mostly) the same, and knowledge and understanding of how the memory is accessed is more useful than experience by trial and error, when generalizing or extrapolating to other cases. This is just such a case, with an unusual combination of sizes and number of sticks.
There is a misconception (mostly) that sticks
have to be in matched identical pairs in each channel. Doing so ensures full dual channel mode operation, but if they are not matched (in size), then it runs in "Flex mode" where some of the memory (the portion existing in both channels) is in dual channel mode and the remainder in single channel mode. There are articles and support documentation from Intel that explain this.
Anyway, OP can test for themself. Run Novabench RAM transfer speed test several times with each arrangement, and it should become apparent which gives the best result. For reference, on my 2020 iMac with 16GB RAM in each channel I get around 29000 MB/s. 2019 iMac should be similar (same memory spec)
