I think you have to look at this on a case by case basis. For example, you might not find the configuration you want as a refurb. And here in the US, Apple has physical stores in most states so you will pay sales tax on a refurb. This can narrow the gap considerably between refurb and new in some cases. As I posted above, a new Mini from B&H Photo was only about $5 more than an Apple refurb. They also threw in a free copy of Parallels. If you live in a locale with high sales tax, the refurb might even cost more.
Like anything else, you need to do your due diligence and compare all the options. BTW, my refurb AppleTV's were packaged just like my new one but my refurb MBA was in a plain brown box. 🙂
I have bought both new and refurb and have never had any problem whatsoever with a refurb product bought from the authorised Apple site.
Again, as others have already pointed out, the trick is to find a refurb that has the specifications that you want or need. Thye offer excellent value and come with all of the warranties of a brand new computer.
I bought a refurbished 2014 MacBook Air just before Christmas from Apple. Indistinguishable from a new model, significantly lower cost, and, as an added bonus, it came with an i7 processor when it was advertised as having an i5.
Unless I can buy a new model cheaper, I usually buy refurbs. Of all the refurb products I've owned I only ever had issues with one, an iPod Mini. Otherwise, they have all been flawless.
I only consider refurb if the price difference warrants it. 9 times out of 10, it doesn't... at least for me. In some cases I've been able to find New Apple products cheaper than Apple sells the refurb equivalent. In those situations, it is foolish to buy refurb... and has nothing to do with the white box. 😛