As Bigwaff's question implies, ideally you've allocated 2X-3X or more capacity of the total amount you want TM to backup. For example, if you have upwards of- say- 1TB to backup, you have allocated 2-3TBs or more for the TM volume on that QNAP.
If you have done that, then yes, TM should be deleting the oldest backups to make room for new backups once the max capacity is reached.
And
here's some people with this kind of issue. Perhaps a solution can be found in those posts? There are MANY pages all over the website about this kind of issue, so there very well may be some bugs with TM itself. Do some searches, read what others have tried and maybe find an action that works for you.
Are you perhaps trying to use the
same space as a prior Mac's TM backup? For example, old Mac has been backing up to this QNAP space and now you want new Mac to use the same space and/or perhaps (thinking you can use the) same backup to just takeover with a newer Mac? If an old Mac has eaten up a good chunk of free space, the new Mac doesn't have the space it needs to back up in what remains. You may find you have to delete any existing backup(s) to fully free up the TM space and then start a brand new TM backup. This may also be the ultimate remedy if the TM backup has become corrupted for some reason.
Lastly, QNAP has some configuration settings especially for TM. It's not a matter of just allocating a space and then selecting it for TM (software). If you didn't set QNAP up especially for TM, that may be the problem.