a phone is not a computer that requires VM to run smoothly and will lose performance based on low memory or platter disks that go slower as memory fills to the outside bands.
if there is anything going on beyond placebo effect then it is either some piece of software you installed messing with the system (more possible now with how open the iOS has become) or you have jailbroken your phone, which tends to hinder performance.
Otherwise there is a hardware failure and you should bring it in to be replaced.
Your point of reference may have changed now that you got used to it. It is like first jumping in a fast car coming from a slower car and being amazed but after a while you get used to the speed and maybe expect more?
Without quantitative speed benchmark stats, it is hard to say if you are right or not. Try running some sunspider or geekbench tests to see if you are valid.
From my experience, I do notice when the A6 processor throttles itself to lower speeds when battery is low to conserve energy. I can confirm this by opening Geekbench 2 and seeing the CPU reported to 600-800MHz vs the 1.3GHz when under load or charging when the CPU ramps up. That may be what you are experiencing?