I'm not saying not to "expect more". But when you're comparing how billion dollar companies with teams of hundreds of people that deal with data breaches taking a week+ to notify users, your expectation that a site administered by (perhaps jsw can answer this definitely, but my guess is) just a few people should be johnny-on-the-spot with a 24 hour notification just seems unrealistic.
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Complaint from MacRumors users about the recent security issue.
- Thread starter Orange Furball
- Start date
- Sort by reaction score
Alright maybe so, but I just feel like their standards for us are different for them.
What jsw says below is correct. We moderators do not have access to vB templates, nor do we post from the MacRumors Twitter or Facebook accounts.
Luckily I use throwaway password #37 here at MR but unluckily I also use it at a few other sites which I've also had to change.
Luck has nothing to do with it. If you use the same password at multiple sites, you will lose. If you want to be mad at somebody, be mad at yourself.
Why are you not using one of the password-management systems?
Agreed, but we are talking about a web site in the case of MR - a site which fully has the power to post the story about the breach on top of its main page and keep it there, and a site which fully has the power to post a notification on the forums, and a site which fully has the power to tweet the news. This is all easily accomplished. I think the fact that MR is a much smaller business with a much smaller paid staff fully explains why we were vulnerable, and I understand such things happen. I'm disappointed primarily in the lack of notification via means easily handled by one staff member. There are likely half a dozen people - or at the very least several - who could have done those notifications, and I find it hard to believe all of them are deeply nestled into looking at code now.
I will openly admit to being shocked how much different the site is now than it was half a decade ago, and I'll openly admit to being very disappointed in the changes. However, I'm not here to rub anyone's noses in anything, although I won't shy away from pointing out that, if you're going to expect perfect behavior from your members, you'd best be perfectly behaved yourself.
Also, to be perfectly clear: I do not blame the moderation staff for the change in tone. I do, however, blame "the management," and so, yes, I'm a bit more irked at this breach than I otherwise would be.
At what point did I say I wasn't using one? At what point did I say I used my MR password at more than a handful of sites? Since I work on multiple platforms I don't really have a password safe that is truly "cross platform" so certain "quicky" passwords might get used on another site. While I agree in principal that repeated use of passwords is a bad practice, his thread isn't about my security practices. It's about common courtesy on the part of MR staff and I prefer to focus on that aspect going forward.
I disagree. The two are intimately related to each other.
Anyone who has gotten a clue to have a unique password on each site has far less urgency to take personal action on a password-compromised site.
This presumes that common standards do exist on how to deal with such attacks. It's a very good idea, but I don't know of any...
I have confidence that MR will create some mechanism to ensure that all readers are notified of the intrusion. And, honestly, you must agree this is a pretty tiny problem for people who have compartmentalized all of their passwords, right?
----------
You told us:
Luckily I use throwaway password #37 here at MR but unluckily I also use it at a few other sites which I've also had to change.
If you are using a password-management system, you clearly don't understand the proper way to use such a system: generate a unique password for every separate login that you use.
Nope. The two are related: if a password is only used on a single site, then there is no urgency in dealing with a compromised password database on any site.
This problem should be resolved when we get a system like Secure Quick Reliable Logins.
Time to institute an emergency plan? I would think especially after this you will have some kind of protocol for handling this kind of thing. At least have someone dealing with Twitter posts + media. That (twitter) is literally one of the first places I went when the site went down.
It's really hard to get a message out to each of the 860,000+ users, but it's easier to spread the message by having it go out from something easy to access. Maybe even embed a Macrumors "Updates" twitter feed in the maintenance page (when you are actively updating the site). I know ifixit provides updates in this style when their site is down.
Just a thought!
Thomas
Last edited:
You are wrong!!!!!
aristobrat;
You are wrong! There is nothing preventing MR from posting this security breach as the top news story of the day or every hour, if need be, so that every MR reader knows about it. The fact that it wasn't is disturbing that anyone could be that foolish not to do so. If continuing news stories can be posted then warnings to all MR readers of the security breach can also be posted. That timely warning should have been the first and most timely post to this site IMO, not 24 hours after the fact.
aristobrat;
You are wrong! There is nothing preventing MR from posting this security breach as the top news story of the day or every hour, if need be, so that every MR reader knows about it. The fact that it wasn't is disturbing that anyone could be that foolish not to do so. If continuing news stories can be posted then warnings to all MR readers of the security breach can also be posted. That timely warning should have been the first and most timely post to this site IMO, not 24 hours after the fact.
Hi guys,
I don't want to get into a back and forth debate about this. I just skimmed the thread.
We are doing our best. We posted info as early as we reasonably could.
- All we knew when we took the forums down yesterday is that someone was trying something suspicious. And it was safer to shut it all down, and sort it out.
- I thought we had shut it down quick enough that nothing bad had happened, so I worked on figuring out what the exploit did. This involved going through javascript, and lots of log files.
- It wasn't until this very late last night/early this morning that it seemed more clear that they were going for user data.
- At that time both the front page and forums were up and down. Nothing was stable.
- We stabilized it, and the first thing I did was post a thread announcing, also promising more information and an email to all users.
As for the emails, they are coming. You can't just send out 800,000 emails at once. vBulletin's email system is browser based. It could never send 800,000 emails in one go. So, we're writing/testing a script to do it. We were hoping to start sending tonight, but it's now targeted as 1st thing in the am.
edit: search engines are down is an automated notice that comes up when the search engine is down.
arn
I don't want to get into a back and forth debate about this. I just skimmed the thread.
We are doing our best. We posted info as early as we reasonably could.
- All we knew when we took the forums down yesterday is that someone was trying something suspicious. And it was safer to shut it all down, and sort it out.
- I thought we had shut it down quick enough that nothing bad had happened, so I worked on figuring out what the exploit did. This involved going through javascript, and lots of log files.
- It wasn't until this very late last night/early this morning that it seemed more clear that they were going for user data.
- At that time both the front page and forums were up and down. Nothing was stable.
- We stabilized it, and the first thing I did was post a thread announcing, also promising more information and an email to all users.
As for the emails, they are coming. You can't just send out 800,000 emails at once. vBulletin's email system is browser based. It could never send 800,000 emails in one go. So, we're writing/testing a script to do it. We were hoping to start sending tonight, but it's now targeted as 1st thing in the am.
edit: search engines are down is an automated notice that comes up when the search engine is down.
arn
Last edited:
Just a note about the search engine. It's usually the first thing that is taken down in times of trouble because of the load it puts on the servers. (For example during events).
So, I suspect that Arn's keeping it down to keep the servers stable.
I have no inside knowledge that that's the case though. just glad the forums are up-ish.
B
So, I suspect that Arn's keeping it down to keep the servers stable.
I have no inside knowledge that that's the case though. just glad the forums are up-ish.
B
Hopefully the exploit was stopped in time. I don't understand the part about the pages being unstable. I thought the forums were still shut down at that time? What would cause things to be unstable? The exploit? How did you stabilize them if the exploit isn't fully known (unless you figured it out recently)? Just curious.
The thing that concerns me most is the PMs, like I asked Arn about in the other thread. I think at this point, everyone should have to reset their password before they're allowed to post again.
For us that have access to the marketplace, we all know what goes on in PMs. PayPal information, e-mail addresses, addresses, and in my case, I had someone go in and apply a coupon to one of my services so if it didn't work, he'd know it was an issue on his end and not think I was scamming him.
I think the situation was handled as best as it could be though. Because honestly, even recently, we had issues with MR related e-mails going to the spam folder in Gmail. So it was most likely that if e-mails went out, we would be in the same situation as we are now.
For us that have access to the marketplace, we all know what goes on in PMs. PayPal information, e-mail addresses, addresses, and in my case, I had someone go in and apply a coupon to one of my services so if it didn't work, he'd know it was an issue on his end and not think I was scamming him.
I think the situation was handled as best as it could be though. Because honestly, even recently, we had issues with MR related e-mails going to the spam folder in Gmail. So it was most likely that if e-mails went out, we would be in the same situation as we are now.
Hi guys,
I don't want to get into a back and forth debate about this. I just skimmed the thread.
We are doing our best. We posted info as early as we reasonably could.
- All we knew when we took the forums down yesterday is that someone was trying something suspicious. And it was safer to shut it all down, and sort it out.
- I thought we had shut it down quick enough that nothing bad had happened, so I worked on figuring out what the exploit did. This involved going through javascript, and lots of log files.
- It wasn't until this very late last night/early this morning that it seemed more clear that they were going for user data.
- At that time both the front page and forums were up and down. Nothing was stable.
- We stabilized it, and the first thing I did was post a thread announcing, also promising more information and an email to all users.
As for the emails, they are coming. You can't just send out 800,000 emails at once. vBulletin's email system is browser based. It could never send 800,000 emails in one go. So, we're writing/testing a script to do it. We were hoping to start sending tonight, but it's now targeted as 1st thing in the am.
edit: search engines are down is an automated notice that comes up when the search engine is down.
arn
We just were asking for better communication from you guys. I am sure you are trying to do your best, but you have to understand this has a lot of personal information from each user that could really hurt us. It goes well beyond this site if the hacker has personal information of ours.
All I am saying is that we are asked to abide by very, very strict rules for a forum, and we do so. We are asked to treat Mods, and other users with respect. You guys value these sets of rules, and ideals. We just expect the same in return.
It's actually piss easy. You point the vB SMTP settings to something like Mandrill or SES and it will handle the workload. Even easier - export the list of addresses and drop it directly into either of them, it'll shave off about an hour.
----------
In fairness Arn, it's a forum - the only thing any hacker would ever be interested in is the user data. It's not something to 'figure out' as it's going to be obvious that they'd be attempting to grab that information. Nobody is going to hack MacRumors to steal threads or attachments for example.
Arn, I think the complaints are valid here. The forums went down approx. Tuesday afternoon. Front page articles continued to be posted. You created a thread at 1:25am EST on Wednesday morning in probably one of the least trafficked sub-forums on the site to notify people. The only tweets from the site or you were that the forums were down and being worked on.
In reviewing your timeline on Twitter, I see you also said TA forums were down. Was that site affected also? I don't see any story on the front page there.
I couldn't care less how Sony or Adobe handled their issues. As soon as you became aware of this, it should have been posted to the front page as well as your social media outlets.
In reviewing your timeline on Twitter, I see you also said TA forums were down. Was that site affected also? I don't see any story on the front page there.
I couldn't care less how Sony or Adobe handled their issues. As soon as you became aware of this, it should have been posted to the front page as well as your social media outlets.
I've been away from much of the action for a few years, but from my observations, this is what took place. *This is not an official recount of what happened*
1) forums team took action to protect privacy as soon as they became aware of a possible breach.
2) within a 6 hours or so of initial indication of an issue - a perfectly reasonable timeframe IMO - initial notices were posted in the forums (feedback) and main macrumors page that the security may have been breached. Investigation continued to determine the scope.
3) well within 24 hours, additional notices were posted on the forums as well as the front page, and the team is working on a system to email the large quantity of users that exist.
Beyond notifying users at the expense of securing the system from further breaches, I don't know what else you would have wanted. Keep in mind that news stories that would have been posted during the initial hours of the breach were written by staff writers (Jordan and Juli) accessing a completely different system. They were likely oblivious to the issue until a few hours in.
In my opinion, this was a world class response.
MR initially appears to have updated about the breach using a forum that they had disabled so if you didn't see the thread right away then when you came to the site (forums), they were down. The complaint is fairly justified.
It is not that forum information is breached. It is that passwords and e-mails appear to have been stolen and to the point of another member, if PMs are accessed, then there is personal data in there from former market place sales. So to cover bases, treat this like a personal information breach and nothing less.
This definitely is one of the best ways to say what needs to be said.
My only issue is that I do not have this site open nearly as much as I used to. I do use twitter and I saw your posts on twitter. Nothing, not even a single word eluded me to the fact that this site was down due to a security breach. Instead, it was passed off as standard maintenance.Hi guys,
I don't want to get into a back and forth debate about this. I just skimmed the thread.
We are doing our best. We posted info as early as we reasonably could.
- All we knew when we took the forums down yesterday is that someone was trying something suspicious. And it was safer to shut it all down, and sort it out.
- I thought we had shut it down quick enough that nothing bad had happened, so I worked on figuring out what the exploit did. This involved going through javascript, and lots of log files.
- It wasn't until this very late last night/early this morning that it seemed more clear that they were going for user data.
- At that time both the front page and forums were up and down. Nothing was stable.
- We stabilized it, and the first thing I did was post a thread announcing, also promising more information and an email to all users.
As for the emails, they are coming. You can't just send out 800,000 emails at once. vBulletin's email system is browser based. It could never send 800,000 emails in one go. So, we're writing/testing a script to do it. We were hoping to start sending tonight, but it's now targeted as 1st thing in the am.
edit: search engines are down is an automated notice that comes up when the search engine is down.
arn
I'd like to know about PMs as well. Not that I gave out bank details or anything, but my home address has been shared.
I get that there was a front page article posted at about 2:48 PM Tuesday but it was buried with a bunch of other nonsense articles. If writers are posting from an entirely different system then why weren't they notified to stop posting and let the security breach article rest at the top? With the brilliant minds behind some of the really neat functions of this site, certainly posting an article and keeping it front and center is easy. I mean that pretty sincerely.
Had I known upon coming to this site and seeing the forums down and after reading a brief and vague exchange between Arn, @MacRumors, and rdowns, that there was a security issue and not something meaningless, then I would have likely started figuring out if or where I've used the same password and began changing it on any site I could still access.
These things happen and I realize there is a fire that is still likely burning. I definitely understand the amount of work that has been done and still needs to be done, but it seems like there are some lessons to be learned here. Take them as that, answer the questions that need to be answered posted by members of your community, and move on best you can.
I agree. If your monitoring or systems detect malicious activity, you would
1) Take site offline, and investigate logs/data (This takes time)
2) Evaluate findings, identify breach and plan a remedy (This takes time)
3) Implement fixes and plug the holes to prevent further damage (This takes time)
4) Determine which data may have been compromised and begin notification processes
You can't do the above in reverse order. The fact that the first 3 steps above were completed in 24 hours is pretty impressive. Step 4 is a work in progress.
Also, it's not accurate to state that "the only thing any hacker would ever be interested in is the user data" - Bot-type attacks aimed at forums often have the goal of injecting js code into posts to initiate drive-by downloads of malware etc. And spam - of course.
Bottom line, there are many potential reasons for an intrusion and finding out what (if anything) was taken takes time.
Since passwords were not stored in plain text I personally don't think the situation merits the level of alarm some feel. Just my 2 cents.
(Note using my local forum time here - GMT)
Just a minor correction - a notice was posted in the feedback section at 6:25 PM Nov 12 by Arn confirming the issue. At that point he said that as far as they were aware, the user table was compromised.
At 9:23pm, after a page and a bit of people demanding for it to go on the home page, Arn agreed to do so..
At 10:48pm the notice appeared on the homepage of MacRumors.
So the total time between reporting it, and it going on the homepage was 4 hours and 23 minutes.
The issue most people have is that the forums went offline at about 3:30pm on Nov 11 (and had been up and down for a good 6-10 hours prior to this).
That means that between them going down, and it being posted publicly on the frontpage was just shy of 17 hours. Whilst obviously it would have taken an hour or two to actually figure out what happened (your vB log files would have told all, as would the server log files) at that point people should have been informed. Be it via the homepage or a mass mail.
Bringing the forums back online should not have been the priority. If you have a car crash caused by your steering breaking, you get it looked at by an expert, you dont just dust it off and carry on driving and get it looked at later.
I wouldn't go that far
The response was 'satisfactory' however and I take no issue with the response time - Arn handled it well.The only thing I disliked was that it took a rampage of pissed off members to get Arn to actually post it on the homepage. There should have been protocol in place for such an event.
What annoys me even more is the stupidity/stubbornness from MacRumors however.
vBulletin 3.x went end of life in March 2011 - that's 2 years, 8 months ago. That's 2 years, 8 months that MacRumors has used a piece of software known to not be patched in the event of a security flaw being found.
Do you find that acceptable? I sure don't. I've seen places do this before and they always give the same lacklustre excuse of "oh we cant move to a more modern piece of software -we've made to many modifications" - the solution is to re-write them or hire someone to do it for you.
Sticking with insecure software based on laziness is inexcusable. I've been banging on to MR staff to upgrade to a better piece of software for 2 years now and nobody's even mentioned anything about considering it.
I have no sympathy for the management on this I'm afraid, and I do hold the MacRumors management team fully responsible, not vBulletin as they had warned people a long time ago that the software was EOL.
----------
Food for thought: http://codahale.com/how-to-safely-store-a-password/
----------
I can probably answer that one for you (although it'd be nice to hear officially) - anything and everything stored in the vBulletin database would have been accessible. This includes PM's.
I'm basing this on the fact that Arn confirmed the attackers gained access to the hashed passwords - the only way to do that is with a direct method of access to the database. Once you've got that every single database table would have been accessible.
Last edited:
If you mean this thread, it was created at 1:25pm EST on Tuesday. And I'm pretty sure the forums went offline on Monday afternoon.
Which rules are you referring to? These?
These are in my opinion way too few for a forum. Check out the rules on WHT to see what I would like the rules on MR to look like.