That doesn't sound right. If it needs to download the full image, maybe. But if it's a photo you've literally just taken, and you have plenty of space on your phone, it shouldn't need to download anything as it shouldn't even be optimized just yet.
To help give you some perspective, I had a bunch of photos on my phone last night, but not on my iPad. When I clicked on the first one on my iPad, I could see the little circle in the bottom loading the image. But as I swiped through the photos (spending maybe 1-2 seconds per photo), the next photo was pretty much always loaded by the time I'd swipe to it. I had to swipe through several very quickly to see the loading symbol again. So 20 seconds sounds very wrong to me given the information (space on phone, good network, etc) you've shared already. I'm not sure what it could be, but that definitely doesn't sound right.