File sync and backup with client-side encryption?

[eggfoam]

In light of the recent security problem with Dropbox, I'm looking for a new file-sync and backup service. Sync is the priority for me, but I would also like my sync folder(s) and possibly other files to be securely backed up.

Dropbox is great from a user-experience perspective -- I can save a file at home, go to the office, and just pick up where I left off. It has good notifications of changes with the menubar icon, Growl, and overlays on file and folder icons to show you what is current and what is still updating. It handles sync conflicts well by adding "[your other computer]'s conflicted version" to the directory with the file you're working on if you happen to save different changes on different machines.

What Dropbox doesn't have is client-side encryption. If you trust them, that's probably OK. After the recent breach that potentially exposed everyone's files with no password needed for four hours (!), I don't trust them anymore. Client-side encryption, where the service never knows your password/encryption key, would ensure that even if someone screws up on their end, as Dropbox did, they can't read your files and neither can anyone who manages to gain access to their servers.

As far as I can tell, there are three options that provide both sync and backup with client-side encryption: SpiderOak, Wuala, and Jungle Disk. I've been trying SpiderOak and Wuala for the past week or so. Jungle Disk doesn't have a free trial, so I haven't tried them yet. SpiderOak and Wuala don't compare to Dropbox in terms of usability.

SpiderOak seems great from a security perspective, but the client software is ugly, un-Mac-like, and a CPU hog. Also, it was designed primarily for backup, which makes sync weird. Specifically, each computer has its own backup on their servers, and even if you sync a folder, each computer has a separate version history for things in your sync folder. There is no canonical cloud copy. I get that they're trying not to throw anything away, but this isn't really the behavior I'm looking for. Also, it does not notify you of conflicts -- it just silently replaces the file with the most recent version. If you notice something is missing, you can go dig it out through their ugly client interface, but if you don't notice, you're out of luck. There's no notification in the menubar of sync activity, and there's no overlay on file icons in the Finder to show you what is current vs. what is still updating. SpiderOak is among the cheapest of the services -- $10/month per 100GB, or $5 if you have a .edu email address.

Wuala does handle conflicts like Dropbox, which is nice, but it seems slow, both in terms of upload/download rate and in terms of recognizing changes in synced folders. (I know they are based in Europe, but do they have a North American data center? I can't tell. That might explain the speed.) It does not currently do delta uploads, so if you change a 200MB file by one character, you're uploading 200MB. SpiderOak, at least, just uploads the changed blocks. Wuala also lacks the icon overlays to show you what is current, which is extra problematic given the slow syncing. It's also substantially more expensive than SpiderOak. It does have a "network drive" that you can mount. You wouldn't want to work off of it for files of any size, but it does seem to be the canonical cloud version of your files, which is nice. If you use sync folders, you can work off your local drive and have updates sent as you save, like with Dropbox.

Has anyone used Jungle Disk? How does it work for syncing? How well does it integrate with Mac OS X?

Any other suggestions for services to check out?

 

[0dev]

Why not just buy a USB key or external HD? They're dirt cheap these days (you can get a 1TB drive for under £100) and there's no chance anyone will hack it (unless you plug it into an infected computer, in which case you'd have bigger problems on your hands anyway).

You can also set up encryption on your own external drive as long as the computers you use have the software required to decrypt it again with the password (any computer with OS X can decrypt a secure disk image if you have the key).

That way, you know your data's safe and, most importantly, under your strict control.

 

[eggfoam]

Why not just buy a USB key or external HD? They're dirt cheap these days (you can get a 1TB drive for under £100) and there's no chance anyone will hack it (unless you plug it into an infected computer, in which case you'd have bigger problems on your hands anyway).

You can also set up encryption on your own external drive as long as the computers you use have the software required to decrypt it again with the password (any computer with OS X can decrypt a secure disk image if you have the key).

That way, you know your data's safe and, most importantly, under your strict control.

Thanks, 0dev. I actually do use external HDs as backup now, but I like the auto-sync and versioning that come with an online solution. Sometimes it has come in handy to be able to access my Dropbox files from my iPhone or the web client, too. (Though I would forgo the web client access for client-side encryption. SpiderOak has one but actually discourages you from using it, because it means sending your password to their server.) Also, I've been burglarized twice in two different cities, so I don't want to keep all my backups in one place. Right now I keep one HD locked in my desk at the office, but it's enough of a hassle that I don't update it very often. Hence the desire for cloud sync/backup.

 

[0dev]

That's a fair point. I myself only bother doing backups every few months and I keep my HD hidden when I'm not using it. I should do them more regularly - and I have suffered from my laptop's HD dying without a backup in the past - but it is indeed a hassle.

I would perhaps wait to see what happens with iCloud. By default it only offers 5GB but, as I understand it, you can pay for more and it will be built right into Lion, so no CPU hogging ugly clients needed.

 

[eggfoam]

That's a fair point. I myself only bother doing backups every few months and I keep my HD hidden when I'm not using it. I should do them more regularly - and I have suffered from my laptop's HD dying without a backup in the past - but it is indeed a hassle.

I would perhaps wait to see what happens with iCloud. By default it only offers 5GB but, as I understand it, you can pay for more and it will be built right into Lion, so no CPU hogging ugly clients needed.

Yeah, that's a good point. What I haven't seen is whether iCloud will have general-purpose storage and versioning. From what I've read, it sounds like apps may have to use new APIs to take advantage of the cloud storage, and the cloud versioning may go hand-in-hand with the new auto-save/auto-versioning functionality, which I would think would also require a rewrite by application developers. Or has new info come out about this?

 

[0dev]

No new info as far as I know. I suspect we'll only get the proper details when Lion drops next month.

 

[eggfoam]

No new info as far as I know. I suspect we'll only get the proper details when Lion drops next month.

Yeah, it will be interesting to see how it works.

I suspect iCloud will not have client-side encryption, though, so any data stored there would be vulnerable to a Dropbox-style foul-up by Apple. I doubt Apple would make quite so big a mistake, but there's a lot of room for smaller ones that would still be bad. I'd like to be able to store tax returns, etc., in a cloud backup and not have to worry that anyone who works at or hacks into the provider can access them.

 

[GGJstudios]

I haven't searched, but I wonder if there is a Mac app that will enable automatic encryption of files dropped in a folder. I know such apps exist, but haven't checked to see if any work well with both Mac and Windows. While it wouldn't be as efficient as having client-side encryption built in to Dropbox (which I agree they should do), the concept would be to drag and drop files into the encryption folder, then drag them into the Dropbox folder, once encrypted. That way, sensitive files stored in Dropbox could be encrypted, but you'd still have the existing Dropbox option of adding non-encrypted files that you want to share with the public.

The lack of client-side encryption is why I've used Dropbox only for non-sensitive files. I don't entrust my sensitive data to any of the "cloud" or "virtual disk" services.

 

[0dev]

You can set up an encrypted disk image in Disk Utility which acts as an "encrypted folder". I personally have FileVault turned on, so everything in my home folder is encrypted on the fly.

 

[GGJstudios]

You can set up an encrypted disk image in Disk Utility which acts as an "encrypted folder". I personally have FileVault turned on, so everything in my home folder is encrypted on the fly.

I'm not sure how well that would work if you want to send a file to Dropbox and retrieve it from a Windows PC or another Mac.

 

[0dev]

An encrypted disk image will open up on another Mac just fine as long as you have the password. On a Windows PC you'll need to install a program for it but it would still work.

FileVault makes your home folder a disk image, so that'll work too.

 

[GGJstudios]

An encrypted disk image will open up on another Mac just fine as long as you have the password. On a Windows PC you'll need to install a program for it but it would still work.

FileVault makes your home folder a disk image, so that'll work too.

But I'm not talking about encrypting your drive or your home folder; only selected individual files that you want to upload. I could be wrong, but I don't think FileVault is designed for encrypting individual files on the fly, for the purpose of uploading them and sharing them with other computers.

 

[eggfoam]

I haven't searched, but I wonder if there is a Mac app that will enable automatic encryption of files dropped in a folder. I know such apps exist, but haven't checked to see if any work well with both Mac and Windows. While it wouldn't be as efficient as having client-side encryption built in to Dropbox (which I agree they should do), the concept would be to drag and drop files into the encryption folder, then drag them into the Dropbox folder, once encrypted. That way, sensitive files stored in Dropbox could be encrypted, but you'd still have the existing Dropbox option of adding non-encrypted files that you want to share with the public.

The lack of client-side encryption is why I've used Dropbox only for non-sensitive files. I don't entrust my sensitive data to any of the "cloud" or "virtual disk" services.

Yes, that would be great. Then you could use any service and not worry about it (unless you are concerned about people knowing the names and sizes of your files, which doesn't bother me). SecretSync does that on Windows; they claim they'll come to the Mac eventually.

There is this solution using EncFS with MacFUSE and Dropbox, but it's a bit of an awkward solution. Neither EncFS nor SecretSync would allow access from a mobile device, either. A native client like SpiderOak offers is the only way for this to work when the service doesn't have your password/encryption key.

You can set up an encrypted disk image in Disk Utility which acts as an "encrypted folder". I personally have FileVault turned on, so everything in my home folder is encrypted on the fly.

The disk image seems like another good option with Dropbox as long as it's the "sparse bundle" type of disk image (which is actually a package with a bunch of segments in it). I have used that for some particularly sensitive data. It doesn't work so well for synced stuff that you're actively using, though, because if you forget to unmount the image on one computer before moving to another, the conflicted versions of the image segments could quickly make a big mess of things.

But I'm not talking about encrypting your drive or your home folder; only selected individual files that you want to upload. I could be wrong, but I don't think FileVault is designed for encrypting individual files on the fly, for the purpose of uploading them and sharing them with other computers.

Right, using Dropbox on an account with FileVault turned on would not make your files on Dropbox's servers any more secure than usual -- they'd be decrypted by the OS before being used by Dropbox, so only the normal (inadequate) Dropbox security would apply.

 

[0dev]

But I'm not talking about encrypting your drive or your home folder; only selected individual files that you want to upload. I could be wrong, but I don't think FileVault is designed for encrypting individual files on the fly, for the purpose of uploading them and sharing them with other computers.

I know, which is why I mentioned creating a disk image in Disk Utility and sticking some files in there.

 

[eggfoam]

I know, which is why I mentioned creating a disk image in Disk Utility and sticking some files in there.

0dev, do you have any experience using/syncing your disk images with a service like Dropbox?

 

[0dev]

0dev, do you have any experience using/syncing your disk images with a service like Dropbox?

I'm afraid not, but I have carried them from computer to computer on a drive before without issue. I see you've cited some issues with Dropbox for sparsebundles, though. Have you tried using another format like .dmg?

 

[eggfoam]

I'm afraid not, but I have carried them from computer to computer on a drive before without issue. I see you've cited some issues with Dropbox for sparsebundles, though. Have you tried using another format like .dmg?

I use encrypted sparse DMGs (but not the bundle kind) for some files that I need to keep locked down (mostly research data), but I haven't tried using them on Dropbox. The problem with a monolithic encrypted DMG on Dropbox is that the whole file will change when you change anything on the disk image, so if it's 500MB, that's 500MB Dropbox will have to upload anytime you hit Save.

Encrypted sparsebundles (or the similar TrueCrypt volumes) would be better in terms of delta uploads, since I believe only one chunk would change when you save a small file. But that's where the versioning problem comes in -- if you accidentally make changes on one computer while it's mounted on another, you may end up with conflicted versions of the chunks from different computers (since Dropbox works at the level of files and doesn't care about Mac OS X bundles). If all the current chunks are from the same computer, you're fine (but may lose some changes). If the chunks are a mix from different computers, you've got a corrupted disk image.

So I haven't tried to sync DMGs myself. I just copy them manually if I need to move them.

 

[0dev]

Ahh, I see the problem then. It seems that, for syncing to work properly, you'll have to deal with reuploading the whole image, unless there's some magical format I'm missing.

 

[eggfoam]

Ahh, I see the problem then. It seems that, for syncing to work properly, you'll have to deal with reuploading the whole image, unless there's some magical format I'm missing.

Right. Or you could be really really careful about always unmounting your disk image every time you change computers, but that's something I don't want to have to think about (and don't trust that I would do with perfect regularity).

 

[0dev]

That's fair enough.

So, at the end of the day, you need the server to encrypt the files it stores. We're back at the beginning here really 😛

I'd be interested to see how iCloud does on that front. In fact, how does Google protect the data they hold? I trust them with more than I really should, I hope they're keeping it under AES-256.

 

[eggfoam]

That's fair enough.

So, at the end of the day, you need the server to encrypt the files it stores. We're back at the beginning here really 😛

I'd be interested to see how iCloud does on that front. In fact, how does Google protect the data they hold? I trust them with more than I really should, I hope they're keeping it under AES-256.

Well, I need them to be encrypted on the server. I want the client to do the encryption so the server never has the key. Dropbox does encrypt files on the server, but they also store the key. (In fact, I think it may be just one key for all Dropbox files, not a user-specific key.) The recent bug allowed anyone to access an account knowing only the email address used for it (the username, not the password). If the encryption were happening only on the client side, that wouldn't have been a problem because even an employee or an outsider who somehow gained access to all of Dropbox's internal systems wouldn't have been able to read anything.

I don't know what Google does with Docs, Gmail, etc., but even if it's encrypted, it would theoretically be vulnerable to the same sort of problem that Dropbox had. Frankly, I'm more worried about this kind of incompetence (either a botched update or inadequate protection against skilled hackers) than about employees snooping or even the government requesting files. (I'm not storing anything THAT sensitive.)

So I'm looking for the kind of thing SpiderOak does, but better executed, or a relatively transparent file-level (not volume-level) tool for encrypting the stuff I put on a less secure service like Dropbox.

 

[0dev]

Makes perfect sense, actually. Sorry I can't be of any help when it comes to recommending a service that delivers that. I'll keep a keen eye on this thread to see if anything interesting comes up though.

 

[eggfoam]

Makes perfect sense, actually. Sorry I can't be of any help when it comes to recommending a service that delivers that. I'll keep a keen eye on this thread to see if anything interesting comes up though.

Thanks -- it seems like an elusive beast so far. I'll definitely update the thread if I find anything useful. I'd still love to hear if anyone has experience using Jungle Disk on the Mac, too. It does support client-side encryption, but not without some apparent implementation flaws. Still, those flaws would represent a real vulnerability only if someone made a concerted effort to crack my password specifically (which would still take a reasonably long time with a good password and an attacker that wasn't a federal agency). I'm not paranoid enough to worry about that, and even if the linked post is correct about Jungle Disk's security flaws, their measures are enough to protect against casual or opportunistic access due to Dropbox-style incompetence.

 

[smoothisfast]

I found this thread today and decided to work something out.

Using gpgtools to PGP encrypt my files and Hazel I'm testing the following.

I Have a folder called "Dropbox PGP"

Hazel watches this folder for new files that are not .gpg files and are not Green label.

It then runs a shell command
"/usr/local/bin/gpg --recipient (email address of the private key) --encrypt "$1""
and
Labels it Green.

When Hazel sees a .gpg file it moves it to the Dropbox folder for syncing.

still working out the details.

 

[eggfoam]

I found this thread today and decided to work something out.

Using gpgtools to PGP encrypt my files and Hazel I'm testing the following.

I Have a folder called "Dropbox PGP"

Hazel watches this folder for new files that are not .gpg files and are not Green label.

It then runs a shell command
"/usr/local/bin/gpg --recipient (email address of the private key) --encrypt "$1""
and
Labels it Green.

When Hazel sees a .gpg file it moves it to the Dropbox folder for syncing.

still working out the details.

Thanks for this -- I was not familiar with Hazel. I wonder if it would work to provide notifications of changes that SpiderOak makes. That would at least address one of the Dropbox features that is missing from SO and Wuala.

You may also want to look into using Dropbox with EncFS, as it more or less does what you're trying to set up. I'm really hoping for a solution that's a little more elegant from a user-experience perspective, but this is at least a possibility now.