It's your thumb, then.
Not to sound all Jobsian, but that's really the problem.
Do you do a lot of work with your hands and/or do you get very dry hands? Do you wash them very often due to your job and/or do you tend to have sweaty palms?
Any of these could make Touch ID harder to use with consistent success. I know that when my 5s flaked out early on after I bought it, I'd traveled from NC to NorCal and the temperature change and dryness of the air, coupled with the fact that I'm a frequent hand washer and the soap in the hotel was quite drying factored into the issue. I nearly returned my 5s because of it, but after reading here and trying the trick where you enter the same finger a few times under differing conditions (dry, damp after a hand washing, etc) the 5s straightened right up and I never had problems with it after that.
I'm convinced that the 6/6+ Touch ID is better because I've not even had to do that an I've traveled since I got my phone back in September. I spend a week in San Francisco in October. No Touch ID changes needed.