Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
I'd just like to know their ETA for coming back online. I seriously hope that they don't pay the ransom.
They probably won't. The likelihood of Garmin being sent a valid decryption key or whatever is needed is so low that paying anybody would realistically just be added losses.
I imagine if they were going the pay route, and paying actually worked, they’d be online now. More likely they are struggling.

Many IT departments today rely on highly available hardware across geographical locations. Generally speaking a corrupted source can replicate corruption.

I think that given the amount of time Garmin has been down speaks to difficulties they may be having.

Wishing them luck. I’d love to start syncing my Forerunner 935 again.
I assume most large-scale companies don't actually have backups in the traditional manner. As you said, it's commonplace for these multi-region companies to simply rely on the organic redundancy designed into their infrastructure, that is, servers of each region syncing. However, as also mentioned, this is a poor (or rather a false sense of a) failsafe because with most situations a problem quickly propagates to every node. Again, I refer to the Maersk situation, who indeed suffered from that additional fault. However, they got just enough luck.
WIRED said:
After a frantic search that entailed calling hundreds of IT admins in data centers around the world, Maersk’s desperate administrators finally found one lone surviving domain controller in a remote office—in Ghana. At some point before NotPetya struck, a blackout had knocked the Ghanaian machine offline, and the computer remained disconnected from the network. It thus contained the singular known copy of the company’s domain controller data left untouched by the malware—all thanks to a power outage. “There were a lot of joyous whoops in the office when we found it,” a Maersk administrator says.
 
I assume most large-scale companies don't actually have backups in the traditional manner. As you said, it's commonplace for these multi-region companies to simply rely on the organic redundancy designed into their infrastructure, that is, servers of each region syncing.

I’m a contract Project Manager and have been involved in a number of projects covering this, and sadly you are right.

Firms will consider business continuity in terms of hardware failure, loss of data lines etc. Lots of redundancy gets built in, lots of money spent, but little in the way of covering off a major agressive attack. They do the basics, but never do a full test over a weekend like they do with Disaster Recovery tests.

Of all the projects I’ve worked on, I can’t think of a single one that would have coped well with this scenario. However, I would like to think that all of them would have had better comms to their customers!!
 
Do they not have backup tapes? My father is the IT guy at Birds Eye (A large food manufacturer in the UK for those not aware) and I remember going in with him most Sunday evening’s as a kid to change the backup tapes. If that is the case why not format and restore, and if not then why the hell hasn‘t a massive company such as Garmin not got a robust offline backup system?

I’m pretty sure that I’ve seen recently that you can get those tapes in massive capacity’s of a terabyte or greater nowadays. I remember my dad having to change multiple of these tapes at a time back in the day too, so capacity shouldn’t be an issue just the time to dig out the tapes and restore them.
 
Do they not have backup tapes? My father is the IT guy at Birds Eye (A large food manufacturer in the UK for those not aware) and I remember going in with him most Sunday evening’s as a kid to change the backup tapes. If that is the case why not format and restore, and if not then why the hell hasn‘t a massive company such as Garmin not got a robust offline backup system?

I’m pretty sure that I’ve seen recently that you can get those tapes in massive capacity’s of a terabyte or greater nowadays. I remember my dad having to change multiple of these tapes at a time back in the day too, so capacity shouldn’t be an issue just the time to dig out the tapes and restore them.

IT Systems have pushed far further into the core infrastructure of any company. When I started work (for a frozen food firm that used to place orders with Birds Eye, strangely enough), we could operate using manual systems if the computers went down. No firm can operate for long without IT now, so they have redundant systems (one live, one spare). This means..


> The rise of having full redundancy has grown, you do hot swaps between two sets of identical hardware should the primary one go down. You can have a major hardware failure and the end user not even know anything about it. This means you rely a lot less on backups, like you did before. This is what firms focus on mainly when testing, rather than a testing for an all out attack.
> If you do restore from backup, it's a major pain to get systems back in line again. Can be done, but the way systems are highly integrated with each other means if you go to backup it's difficult to re-align.
> If your backup has the malicious code on it, having them doesn't really help. The malware can be sitting there for months before they activate it.
> As their entire system is corrupted, you can even get on to the systems to simply kick off a restore from backup. Think of it like when your iPhone is completely bricked and won't even boot up. You need some serious work to kick it back into life. Pressing and holding the power button for 10 seconds won't do a thing.

The people behind this attack knew what they were doing. I do feel for Garmin, someone somewhere accidentally left the back gate open and it's cost them dear.

That said, as a Garmin user, any sympathy I do have for the IT team does not extend to their Communications team....the lack of info, the terrible FAQ thing they sent out etc is just shocking. Complete disdain for their customers.
 
Do they not have backup tapes? My father is the IT guy at Birds Eye (A large food manufacturer in the UK for those not aware) and I remember going in with him most Sunday evening’s as a kid to change the backup tapes. If that is the case why not format and restore, and if not then why the hell hasn‘t a massive company such as Garmin not got a robust offline backup system?

I’m pretty sure that I’ve seen recently that you can get those tapes in massive capacity’s of a terabyte or greater nowadays. I remember my dad having to change multiple of these tapes at a time back in the day too, so capacity shouldn’t be an issue just the time to dig out the tapes and restore them.

Isn't this what Iron Mountain is for? Do you see the Iron Mountain vans at your place of work on a regular basis?
[automerge]1595844037[/automerge]
Garmin Connect webpage is back up and running for me. I will try a sync to see if I can see today's data. My data from Wednesday is there. It looks like things are getting better.
 
Last edited:
  • Like
Reactions: jbachandouris
Sync finished but it took a long time. I have access to my data on the phone and the webpage including this morning's data. There are some small glitches like hydration not being displayed on the phone but it does display on the web page. Some analysis is wrong as well (stress descriptions are inaccurate) but I think that that stuff has to be processed on Garmin's servers and then it provides the right descriptive annotations. I'm kind of used to doing a sync and then having to do a sync later on to get the analysis for some items.

But it looks like they are back in business. I'll go for a short run in a while to see if everything updates correctly.
 
I imagine if they were going the pay route, and paying actually worked, they’d be online now. More likely they are struggling.

The company I used to work for would do a large scale recovery of critical systems into a third party facility every year as a drill. It was a week long project and often resulted in various failures not previously encountered. That was essentially both a restore and rebuild scenario.

Many IT departments today rely on highly available hardware across geographical locations. Generally speaking a corrupted source can replicate corruption.

I think that given the amount of time Garmin has been down speaks to difficulties they may be having.

Wishing them luck. I’d love to start syncing my Forerunner 935 again.

I was a subcontractor with a small computer company in a town nearby. They had a client that had a LOT OF DATA, and this outfit's 'top guy' setup the backup system. Then they had a drive go down. They called me in to see what was going on. Well... The server drives weren't setup right, not using RAID at all. So the drive that died was dead, taking everything it had with it. The backup system that was installed was too small, and the backup was running into two tapes, and the user was told to 'ignore the tape messages' and put in the next days tape.

I get there, and they want a restore from the tape. Restore fails, can't index the tape. It wasn't closed properly (no second tape). They are freaking out. Tech support with the software company is no help. They lost a client, and almost were sued out of existence. Another client of theirs had some files accidentally deleted from the server. I was called in again, (why? I don't know). Found out all of their tapes were blank. The system was apparently doing a 'test backup' to simulate how long an actual backup would take (Stupid feature of the decade). For almost a year, they were getting no backups. Soon after that, I quit working with them. I was afraid the bad blood would spill over to my company.

Yeah, do test restores on backup systems every week, or at least ever month. Use RAID systems, at least mirroring. Install and test software to protect your infrastructure. Protect yourself...
 
I was a subcontractor with a small computer company in a town nearby. They had a client that had a LOT OF DATA, and this outfit's 'top guy' setup the backup system. Then they had a drive go down. They called me in to see what was going on. Well... The server drives weren't setup right, not using RAID at all. So the drive that died was dead, taking everything it had with it. The backup system that was installed was too small, and the backup was running into two tapes, and the user was told to 'ignore the tape messages' and put in the next days tape.

I get there, and they want a restore from the tape. Restore fails, can't index the tape. It wasn't closed properly (no second tape). They are freaking out. Tech support with the software company is no help. They lost a client, and almost were sued out of existence. Another client of theirs had some files accidentally deleted from the server. I was called in again, (why? I don't know). Found out all of their tapes were blank. The system was apparently doing a 'test backup' to simulate how long an actual backup would take (Stupid feature of the decade). For almost a year, they were getting no backups. Soon after that, I quit working with them. I was afraid the bad blood would spill over to my company.

Yeah, do test restores on backup systems every week, or at least ever month. Use RAID systems, at least mirroring. Install and test software to protect your infrastructure. Protect yourself...

Or put it in one of the big Cloud Services and pay for the automatic backup and ample storage options.
 
  • Like
Reactions: jbachandouris
Activity uploads are supposed to be happening, but I tried two different workouts, and both uploads failed. *shrug* They need more time apparently. I'm just glad to see them back.

I just tried a manual activity add and it went in. I'm in a meeting for another 30 minutes but I'll go for a short run when it's over and see if it uploads and shows up in Calendar and Daily Summary.
 
I just tried a manual activity add and it went in. I'm in a meeting for another 30 minutes but I'll go for a short run when it's over and see if it uploads and shows up in Calendar and Daily Summary.

Manual activities apparently are working, but if you think about it they *should* work. They don't require much heavy lifting at all. Syncing activities is still 'delayed'. Especially from Connect to Strava, and I'd wager from Strava to Connect as well (which was what I was trying to do). It's coming back, but it might still take a few days.

The Strava ride I did earlier DID sync to Connect. None of the past activities have, but I don't remember if they auto synced after outages before. It's looking better and better.
 
Last edited:
UPDATE!

All of my activities in Strava are in Connect now. They automagically appeared, but one activity is 'uncategorized'. I can't find out how to change the Connect category. Not that it's important I guess.

HAPPY DAYS!!!
 
UPDATE!

All of my activities in Strava are in Connect now. They automagically appeared, but one activity is 'uncategorized'. I can't find out how to change the Connect category. Not that it's important I guess.

HAPPY DAYS!!!

Sounds good.

I wish that I could set the default activity category for manual additions. I always have to select from the pulldown.
 
The webpage was down for me last night. It's back up this morning. It appears that they may be doing some catching up at night by region. The phone app displays a red warning message that they are down sometimes, even when all of the data shows up correctly. So I suspect that they still have a lot of catching up to do but the main functionality is mostly available.
 
The webpage was down for me last night. It's back up this morning. It appears that they may be doing some catching up at night by region. The phone app displays a red warning message that they are down sometimes, even when all of the data shows up correctly. So I suspect that they still have a lot of catching up to do but the main functionality is mostly available.

Yes, same for me. After having everything sync, I was shocked to see them down again, but figured it was tweaking post resurrection. This had to be a huge wake up call for corporate IT. Spend your time on protection, education, investigation, rather than have to rebuild your entire corporate IT structure because someone was phished.

I had a client confront me with an email message from 'Your corporate IT director'. They wanted to know if it was legit, they thought I was joking about our importance to them. It was a phishing expedition. I would never send them an email, and it was spoofed at that. It looked like it came from them, their 'corporate IT director' that they didn't have. I sent out an email to all of their employees that there was no 'corporate IT director', or any other corporate IT people, and that any message to them would come from OUR email address, not theirs. And also that NO ONE in any connection to supporting their companies IT infrastructure would EVER ask for their password for ANY service they have access to. Education... Encourage people to report suspicious emails they receive. I've gotten emails from OUR corporate IT department! Phishing is so stupid, but it works on too many people.

Corporate IT needs to WAKE UP!!! It's not a question of if, but WHEN they are in the same box as Garmin was. I hope Garmin has the ccourage to couple with Wired, or some other recognized IT/business publication and they share what happened, and what it took to recover. Being lax on security is easy. Recovering from it when it blows up in their faces isn't. And I'm sure it was very expensive, and (should have been) embarrassing.
 
  • Like
Reactions: MacCheetah3
Something I discovered, if a Strava event is wrong in Connect, export the event as a 'fit' file, and if you can export it that way, delete the event, then import the 'fit' file, and you should be good. One Strava activity was 'uncategorized', and much of the data was missing. Like power data...

It worked for me.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.