Should Apple open up OS X Beta testing?

[darwinian]

I know developer previews have at least one specific goal at giving third party devs early access so they can get a sense of new APIs and start updating/creating apps, but they also serve for bug reporting. Do you think Apple should open up OS X beta testing to a larger community of users? Who?

 

[t0mat0]

Why should they?

 

[darwinian]

Why should they?

1) A majority of people agreed that the Leopard release was not good (wireless, BT, TM, X11, and at least 1 major Finder bug issue - the copying/moving thing)
2) Many of us would find and submit bug reports that could help improve the quality of the product upon release

Of course, more bug reports would be harder to triage. And having such a release could be confusing to some customers. You'd probably have to have a free developer account, and obviously it wouldn't be automatic update, and the recommendation would be as a secondary non production system.

I think it's a great move my MS to be doing this with Windows 7. That's why I'm thinking about it.

 

[Tallest Skil]

2) Many of us would find and submit bug reports that could help improve the quality of the product upon release

You can do that now. Then Apple applies updates in the 10.x.x releases.

 

[nutman]

1) A majority of people agreed that the Leopard release was not good (wireless, BT, TM, X11, and at least 1 major Finder bug issue - the copying/moving thing)
2) Many of us would find and submit bug reports that could help improve the quality of the product upon release

Of course, more bug reports would be harder to triage. And having such a release could be confusing to some customers. You'd probably have to have a free developer account, and obviously it wouldn't be automatic update, and the recommendation would be as a secondary non production system.

I think it's a great move my MS to be doing this with Windows 7. That's why I'm thinking about it.

If you think about it, Microsoft has been doing such public beta testings for quite awhile. And the last time they did, for Windows Vista, it was quite the travesty. Just because they do a beta doesn't mean that they will actually fix anything. It is still hard to believe how badly Vista's launch was, considering its long development phase and two betas plus a RC release, if memory serves.

 

[darwinian]

You can do that now. Then Apple applies updates in the 10.x.x releases.

Yep, and I do. But I feel like we could have caught a whole slew of problems from 10.5.0-10.5.3 that persisted. A lot of people felt 10.5.3 was what 10.5.0 should have been.

If you think about it, Microsoft has been doing such public beta testings for quite awhile. And the last time they did, for Windows Vista, it was quite the travesty. Just because they do a beta doesn't mean that they will actually fix anything. It is still hard to believe how badly Vista's launch was, considering its long development phase and two betas plus a RC release, if memory serves.

This is true, I think they did do it for Vista. So I'm actually in favor of a more limited beta, open not only to developer members but those of us who have submitted novel bug reports in the past. Many of us have an obvious interest in seeing OS X improve, but we're not going to pay for the privilege to do so. It's already our free time that's devoted to it.

 

[armoguy94]

Yes... it should definitely be open to any interested user, but a limited amount of copies can be downloaded w/ a special serial per download..just like Microsoft. Apple should open up some more, seriously.. it's quite annoying and egotistical. I love their products but they can be so much better for the price we pay.

 

[chewietobbacca]

If you think about it, Microsoft has been doing such public beta testings for quite awhile. And the last time they did, for Windows Vista, it was quite the travesty. Just because they do a beta doesn't mean that they will actually fix anything. It is still hard to believe how badly Vista's launch was, considering its long development phase and two betas plus a RC release, if memory serves.

Vista's problems were more than any beta testing could've done really.

And anyways, MS needs to do public beta's cause they support nearly every hardware piece, so unless it's released, there is no way to hit all the possible configurations out there. Macs are on set specific hardware, and are thus infinitely easier to beta test

 

[Theophany]

All software has problems. No software works perfectly. Any seasoned programmer will tell you that when you're working with code of the magnitude of an operating system, patching up some errors will undoubtedly cause errors somewhere else.

People piss and moan about Leopard's notorious (but now, thankfully, fixed) wi-fi bug that caused sporadic signal dropouts. But would that really have been fixed if Leopard had been opened up for beta testing? No, actually, because what caused that wi-fi bug was an update patch.

But to the point, with beta testing, you rely on people to take the cookie, then give you feedback on how it was. You then improve then recipe. (In the case of Vista, Microsoft handed out cookies then 'fine tuned' the recipe by defecating into the cookie dough a bit more each time and saying problems had been 'addressed'). The problem with this is that you'll never reach the audience needed to REALLY iron out the bugs unless you push the product to retail and then address the problems (not in Microsoft's way) later on with patches and software updates.

I wonder what percentage of people are actually downloading Windows 7 to beta test? I would then go on to wonder what percentage of that percentage actually have any valuable input into the beta testing stage? 30% maybe?

Apple already do beta testing. They seed to developers, i.e. the important people who actually NEED to beta test, not consumers who are looking for a play about and a sense of inflated ego because they have a pre-release copy to brag about.

AND BESIDES ALL LOGICAL ARGUMENT: Beta testing suggests a degree of openness and democracy about the software and the developer. It suggests that they give a crap about what you have to say. This is Apple, people, wake up. As I remember it, some of Leopard's most infuriatingly irritating and patently obvious bugs were never actually acknowledged properly, just fixed when they got around to it (2 patches later, or so). Personally, I love the way Apple run their affairs - like Putin runs Russia. But I wouldn't expect to see Apple opening OS X 10.7 up to beta testing or anything that follows, that's just not the way they roll. Why? "Because Steve says so."

 

[darwinian]

People piss and moan about Leopard's notorious (but now, thankfully, fixed) wi-fi bug that caused sporadic signal dropouts. But would that really have been fixed if Leopard had been opened up for beta testing? No, actually, because what caused that wi-fi bug was an update patch.

There were several wi-fi related bugs. At least one other was the failure to connect on several portables upon wake.

I wonder what percentage of people are actually downloading Windows 7 to beta test? I would then go on to wonder what percentage of that percentage actually have any valuable input into the beta testing stage? 30% maybe?

It's a good point. I'd guess it's far fewer than 30%; were there numbers on this? Two things have to occur, of course. You have to find and test a bug and then report it.

Let's say 1% of testers do this in the Windows example, and everyone else is just 'playing' (I agree there are plenty of these folks).

Even in the dev community, let's say it's 10x the reporting that it is in the general community. 10% of developers reporting bugs is still likely to be smaller than 1% of all users reporting bugs.

AND BESIDES ALL LOGICAL ARGUMENT: Beta testing suggests a degree of openness and democracy about the software and the developer. It suggests that they give a crap about what you have to say. This is Apple, people, wake up. As I remember it, some of Leopard's most infuriatingly irritating and patently obvious bugs were never actually acknowledged properly, just fixed when they got around to it (2 patches later, or so). Personally, I love the way Apple run their affairs - like Putin runs Russia. But I wouldn't expect to see Apple opening OS X 10.7 up to beta testing or anything that follows, that's just not the way they roll. Why? "Because Steve says so."

There's certainly truth in that. It should also be noted that doing proper beta testing on this scale is not a trivial problem. The proper implementation is actually quite difficult to set up completely on this scale. I'm not saying this is an excuse for not doing it, but I'm definitely not advocating Apple to just post the software on their dev site.

 

[Theophany]

Even in the dev community, let's say it's 10x the reporting that it is in the general community. 10% of developers reporting bugs is still likely to be smaller than 1% of all users reporting bugs.

That is very true, especially with Windows where the magnitude of its user base is undeniably huge. However, I would be more likely to subscribe to the thinking that both the consumer is more likely to draw attention to the bugs after they have paid for it and the developer is much more likely to fix them in a timely manner when several hundred thousand users and trolling the Internet making sure that as many people as possible are aware of the bug and are replicating it to overload MS servers with error reports

I completely appreciate that it's a dishonest, amoral practice that would likely end up in class-action lawsuits or a federal case being made, but it does make sense.

 

[hacksaw-C87]

Yes, it should but only for me so I can wallow in my own self-indulgence. Joking aside, it would be nice to see a little more transparency, Apple might win over even more users that way.

 

[lostngone]

They already do....

Its called ADC Membership.

 

[synth3tik]

Yep, and I do. But I feel like we could have caught a whole slew of problems from 10.5.0-10.5.3 that persisted. A lot of people felt 10.5.3 was what 10.5.0 should have been.

So the average user with little to no knowledge of software. Basically your "internet & email" people would be better at testing software then developers and software testers?


Also to get technical Apple did release the beta of 10. Seeing as how any current revision is going to be an extension of 10. The product (OS X) has been in production for almost a decade, a little bit outside the pre-production that "beta" refers too.

 

[NT1440]

Apple already finds it hard enough for people not to imitate/copy from their other products, why would they make it EASY for others to take their best products features?

I can only imagine how fast a copy of Time Machine or other features would have been made had the public been fully aware of them from the get go.

 

[gnasher729]

I think it's a great move my MS to be doing this with Windows 7. That's why I'm thinking about it.

Microsoft is doing that because they have an absolute stinker on their hands (named Vista) and really, really need to draw attention away from it.

Even in the dev community, let's say it's 10x the reporting that it is in the general community. 10% of developers reporting bugs is still likely to be smaller than 1% of all users reporting bugs.

A large number of bug reports doesn't help one bit. Most people can't write bug reports. Hundred people writing "OMG Mail doesn't work" and "when do you fix the so-and-so" bug are not helping one bit. If you have a good developer reporting a bug, you get a step by step description following the format "What did I do?", "What should have happened?", "What happened instead?" that can be followed step by step to get to the bug, without wasting a developers time.

Look at the questions asked in the "programming" threads. There are guys who want you to help them and they describe their problem as "I get a warning and it doesn't run". To quote Homer Simpson: "Doooh". One good bug report helps more than 100 bad ones.

 

[darwinian]

They already do....

Its called ADC Membership.

ADC membership that includes early release software is for development purposes, for people or companies that plan to release software as a part of some kind of business model, generally. Even advanced users, which constitute at least a part of this forum, would be good candidates for helping finding, testing, and reporting bugs in the correct format.

So the average user with little to no knowledge of software. Basically your "internet & email" people would be better at testing software then developers and software testers?

Also to get technical Apple did release the beta of 10. Seeing as how any current revision is going to be an extension of 10. The product (OS X) has been in production for almost a decade, a little bit outside the pre-production that "beta" refers too.

Yeah, I submitted a few bugs for that public beta. The term beta has been bandied around a bit and lost its original meaning; I think Google's one of the worst culprits of this -- Gmail has been in beta since its inception in 2004. The only reason Chrome is technically out of beta, I suspect, is that they need a "production quality" browser if they want any chance of having it ship with OEM systems. But there can also be beta software of new builds, so a beta of 10.6 for instance is not an unreasonable moniker, I don't think. Well, anyway you know what I mean!

By the way, I don't think the Email forwarders and YouTube watchers are the people who would be interested in the testing. But they're also not ones who are downloading and testing nightly builds of Camino or Safari development previews. As a gross overgeneralization, this is probably not the group of people who'd be good at testing, either, and that's fine. They want machines to work and don't care about the process. But there are so many folks willing to help test, it just seems like a great distributed work model.

Perhaps it's the ADC memberships that they feel like they'd be cannibalizing, though those ADC memberships have so much more added value, don't they, like support options and Coding Headstarts, as well as other educational resources.