When you click the name it should bring up a tab showing "Contact" on the left and "Directions" on the right. Take your finger and drag that up till it takes up almost the whole screen of your phone. Then at the bottom you should see "Remove ____"
At least this is how it looks on my 14 PM on iOS 16.5