The fact that this job has to exist seems to be an admission by Apple that the ways they've been testing Siri internally don't mimic how normal people interact with Siri. They've finally realized something that most of us on this board have known for years.
As someone who also uses Google Home products, I'm amazed how I can sometimes say something to Google Home that doesn't come out exactly the way I meant to say it yet somehow, it manages to understand what I meant to say and it gives me the information I needed. Looks like Apple's trying to get there with Siri, too. I guess monitoring social media for screenshots of Siri screwups is one way to do that. It doesn't seem like a particularly efficient method, though.