Size Matters?

Discussion in 'Digital Photography' started by DucatiTerminator, Jun 22, 2015.

  1. DucatiTerminator macrumors member

    DucatiTerminator

    Joined:
    Jun 11, 2015
    Location:
    Moorpark, CA
    #1
    First a confession: I'm probably the worst when it comes to organizing photos, and I have quite a few of them spread over several drives, computers, and optical (and other media) of various types. I can generally find whatever I'm looking for, but lately it's been taking a bit longer, and I've finally decided to consolidate my media and start the path towards better organization.

    At a quick glance, I have a couple of TB of images at arms reach, and I'm guessing I have double to triple that if I were to consolidate all of my images (and double+ that again if I factor in digital video). Recently, I started shooting more seriously again, and I'm starting to add images at a fairly fast rate. I shoot everything RAW, and until now, the file sizes haven't been too large, however, I am thinking of adding a Canon 5Ds(R) to the arsenal and am concerned how fast the size of my library will start growing. Maybe some of you D800/D810 users could chime in? God (and wife) willing LOL, I'm also hoping to add the new Phase One XF camera which will most likely reset my perception of storage.

    I'm curious what you hardcore enthusiasts/prosumers/pros do for storage and back up. Any of you have working libraries over a couple of terabytes, and if so, how do you manage your media? Any thoughts, advice, or recommendations would be greatly appreciated!
     
  2. The Bad Guy macrumors 6502a

    The Bad Guy

    Joined:
    Oct 2, 2007
    Location:
    Australia
    #2
    Do what I did and delete the crap you're never going to look at again. That'll free up a at least half of your storage. :D
     
  3. Apple fanboy macrumors P6

    Apple fanboy

    Joined:
    Feb 21, 2012
    Location:
    Behind the Lens, UK
    #3
    Deleting stuff you don't need is a good strategy, and one I don't use enough.
    One of the reasons I chose the D750 over the D810 was file size. 24 MP is plenty for anything I'm going to print.
    Yes storage is relatively cheap these days, but the more drives you have, the longer it takes to find stuff!
     
  4. tomnavratil macrumors 6502a

    tomnavratil

    Joined:
    Oct 2, 2013
    Location:
    Litovel, Czech Republic
    #4
    I also shoot mostly RAW so there's an increasing need of storage space. Deleting old stuff is definitely a good idea however if you think you might ever need it, convert all those old RAW files into JPGs. Also my Nikon D610 shoots at maximum of 24MP so it's not too bad storage-wise and as Apple fanboy mentioned - alright for almost any print.

    When it comes to storage itself, I use a simply external HDD, backed up through Time Machine to another one however with your needs that might not be sufficient for sure. You might want to look into more sophisticated and professional solutions such as DAS - for example some from Drobo - http://www.drobo.com/storage-products/5d/ or LaCie - https://www.lacie.com/products/product.htm?id=10607 - where you can swap disks to larger sizes if you need to.
     
  5. ChrisA macrumors G4

    Joined:
    Jan 5, 2006
    Location:
    Redondo Beach, California
    #5
    Yes. Culling your library is also a good way to improve the overall quality. It's easy to delete maybe 2 out of every 10. But most people can cut further. Removing the lest good image makes the entire collection better.

    Next, you need some kind of software to manage your library. I'd recommend Adobe Lightroom as the best choice for use on a Mac right now. Just put everything in the LR library and start entering media data as you have time

    As for how to store that much data. Get a RAID box of some kind.
    Synology makes some good ones
    https://www.synology.com/

    If you know a little about computers you can build a top-tier storage server inside a PC server chassis using this software
    http://www.freenas.org/for-home/

    But very importantly -- You need to BACK UP your data. RAID is not a backup. Ant system that follows ALL of these rules will work:
    1) The data must exist on at least THREE different physical media at all times (A raid system counts as one physical media)
    2) The data must exist at a minimum of TWO different geographical locations at all times.
    The above needs to apply especially during a backup copy operation. So if you keep an off site copy (rules #2) you need three copy to rotate.

    I use
    1) the primary storage on the iMac (physical media #1)
    2) Time Machine connected to the iMac (physical media #2)
    3) Some disconnected hard drives I keep in a fire safe in the same house (physical media #3)
    4) An on-line backup service (Backblaze for $5 per month) (geographic location #2)
     
  6. ChrisA macrumors G4

    Joined:
    Jan 5, 2006
    Location:
    Redondo Beach, California
    #6
    This is why you need some kind of library manager. I use Apple's Aperture but it is now discontinued and I plan to move to Adobe Lightroom. Using something like either of these it does not matter how many photos you have. As long as you took the time to enter basic media data you can search. Aperture and others can even search for faces.

    I tried and experiment by looking through my collection for images of George Washington and Aperture found some shots I have of a $1 bill and of Mount Rushmore. If you camera has GPS or if you take time to tag your files your can search on location. Some software now can do even better and find objects (other then faces) in images.
     
  7. tomnavratil macrumors 6502a

    tomnavratil

    Joined:
    Oct 2, 2013
    Location:
    Litovel, Czech Republic
    #7
    This.

    People don't realise how important backup is until they lose their data. There is also a lot of people who think that RAID is a backup even when it's not.
     
  8. ChrisA macrumors G4

    Joined:
    Jan 5, 2006
    Location:
    Redondo Beach, California
    #8
    The big problem with your backup system is that is is not protecting you from the most common causes of data loss

    1) Theft of the equipment (the #1 cause) because your Time Machine disk is right there with the rest of the equipment
    2) A power hit, say lightening hits a power line within a mile of your house. Anything plunged in the AC mains is going to get fried.
    3) natural disaster or a house fire. (EVERYONE who had a fire said "It probably will never happen to me.")
    4) Software bug that corrupts a file or an entire folder or drive. (This does happen)
    5) Operator error. (You do something dumb.)
     
  9. tomnavratil macrumors 6502a

    tomnavratil

    Joined:
    Oct 2, 2013
    Location:
    Litovel, Czech Republic
    #9
    Oh no, I wasn't describing my full back-up but what I simply use for images as I don't need a RAID or DAS/NAS due to the size of my library.

    I still back-up my most important files to the cloud service and also files off-site.

    I agree with your points for sure. Having all files in one place is not a great idea. However even having Time Machine enabled as a basic backup is something that many users don't even do, which really surprises me these days. So yes, a multiple backup, including an off-site one is a great idea but I know that most users don't even do the basics.
     
  10. dwig macrumors 6502

    Joined:
    Jan 4, 2015
    Location:
    Key West FL
    #10
    All good backup strategies include some form of off-site backup, period. If off-site backup is not included then it is not a good strategy.
     
  11. DucatiTerminator thread starter macrumors member

    DucatiTerminator

    Joined:
    Jun 11, 2015
    Location:
    Moorpark, CA
    #11
    Unfortunately, deletion isn't the best option for me. For example, one of my dear friends is a model who is not very computer savvy. The content of her alone is approaching a terabyte, and I do almost all of her post-processing. I'm a father of three active boys, and I'm guessing family content is approaching 2 terabytes (or more); my better half would be slightly upset if I even considered deleting any of it, lol. And I do have some commercial content which I can't delete.

    Even if I were to pare down the library, it won't address my current/future requirement for storage. I've had great experiences with LaCie in the past; I will look into both Drobo and Synology. I was hoping to delay the investment in pro level storage, but I think the writing is on the wall.
     
  12. DucatiTerminator thread starter macrumors member

    DucatiTerminator

    Joined:
    Jun 11, 2015
    Location:
    Moorpark, CA
    #12
    I've very recently acquired Lightroom through Adobe CC. Many, many years ago, I casually used Extensis Portfolio, but never really adopted it like I should have. Upon initial use, I noted that I have 10K+ images I imported into the catalog. I'm guessing I will be well into the six figures by the time all is said and done.

    I have some local drives set up in a couple of RAIDs, but it seems like a larger box like you and Tomas mention is inevitable. As far as back up is concerned, I've used Time Machine on a couple of the Macs, but I haven't yet attempted to use it for the larger drives with everything mounted to the computer. I have used older tape drives, but VXA2 is pretty old and slow and the cleaning tapes are getting difficult to source without spending a mint. I could probably make use of a LTO-5 drive I have and keep the media in a fire safe. I will look into Backblaze; the unlimited data sounds nice. I'm concerned that because of a rather slow office internet connection (dsl availability only at the moment) that backing up that much data could take a really, really long time.
     
  13. steveash macrumors 6502

    steveash

    Joined:
    Aug 7, 2008
    Location:
    UK
    #13
    My system isn't very high-tech but it works for and has done for many years. I am a commercial photographer and a graphic designer before that so have plenty of client files to look after. I also us a Hasselblad with its rather large raw files.

    My approach is to use an ordered system of multiple Lightroom catalogs. I have one per client and for personal work create a new one each year. I keep my file system in order with a folder for each catalog. I can then archive off old work onto external drives (2 copies, 1 taken off site) once the project is done. Everything is catalogued with Disk Catalog Maker so I always know where my files are.

    I aim to produce quality over quantity and so my libraries are never huge. I do still go through each shoot and delete the mistakes, duplicates and bad shots. For what is left I have never needed more than 1TB of storage for current work and so have never bothered with an expensive and potentially catastrophic (if some of the stories I've heard are correct) RAID system. I don't need instant access to work produced years ago.
     
  14. DucatiTerminator thread starter macrumors member

    DucatiTerminator

    Joined:
    Jun 11, 2015
    Location:
    Moorpark, CA
    #14
    Thank you (and everyone else!) for your input. I like your approach; I think I can use a modified version of it as a part of my workflow. I do a little bit of commercial work, and this will likely work very well.

    I don't think I will every be able to limit the quantity aspect as much as I would like. Between shooting a friend's band and my kids' sports alone (my volunteer position for years has been club photographer), I tend to fill up space pretty quickly, even after trying to be as selective as possible regarding the "keepers," and I'm only shooting at 12-16MP at the moment. As I mentioned earlier, I'm thinking of adding Canon's 50MP body (and/or hopefully a Phase One XF -- my dream) to the arsenal, so either way, my storage needs will grow significantly. Archiving old work makes sense, but I do like to keep a lot of the fun/hobbyist (majority) readily accessible as I am asked to access it by friends and family fairly often.

    Love your work, BTW!
     
  15. Robotti macrumors regular

    Robotti

    Joined:
    Oct 16, 2014
    #15
    I started using Amazon Cloud Drive for backup as soon as they published it, and have been satisfied so far. It's unlimited and a good price for that. It also has very good upload speeds (at least from where I am), which seta it apart from many other services I've tried. Their interface for accessing and cataloging the files for online access needs some work, but I think it'll get there. For backup purposes, more than sufficient already.
     
  16. OreoCookie macrumors 68030

    Joined:
    Apr 14, 2001
    Location:
    Sendai, Japan
    #16
    I'm not quite sure I understand why you think you need a 35+ MP camera, because that's a really expensive proposition. A camera like the Canon 5D R needs the best glass money can buy, and a storage system to match. These high MP RAW files are BIG, and eat up a lot of space. You will have to look into getting a RAID for sure. I recommend Synology here, they have the best software and excellent hardware. I would recommend their 5- or 8-bay systems (1515+ and 1815+ or 2015xs, respectively), and populate these with good NAS hard drives (e. g. those of HGST). However, this will just get you raw storage, and you need to up your game with organization. Perhaps it's better to just accept that your past life is a mess, start with a clean sheet, and apply all the best-practices to new photos.
     
  17. Apple fanboy macrumors P6

    Apple fanboy

    Joined:
    Feb 21, 2012
    Location:
    Behind the Lens, UK
    #17
    I use LR to catalogue all my images. However I'm not as commited to key wording etc as I should be!
     
  18. DucatiTerminator thread starter macrumors member

    DucatiTerminator

    Joined:
    Jun 11, 2015
    Location:
    Moorpark, CA
    #18
    Believe me, I don't NEED a 35+ MP camera. I simply WANT a new full frame camera that can also shoot DX should I like, shoots and focuses more quickly and accurately, and gives me the best IQ I can afford. I shoot with a 5D which meets my requirements right now, and have ready access to MKII and MKIII bodies, but I've had a lot of fun shooting with a Fuji X-Pro1 and XT-1 lately; IQ, glass quality, and portability are on par or exceed my current DSLR gear in most of my situations, and file sizes are relatively small. I do suffer from GAS sometimes, but I rarely indulge in gear on a whim. The 5D is getting tired, and it's speed can be limiting at times, so I figured it's time for an upgrade. I know I will have to upgrade some of my glass, but will likely rent or borrow a lot in the beginning.

    I'm using 4TB HGST (2 million hour MTBF) drives at the moment in my aging cMP. I will definitely look into Synology, thanks. My personal life is organized chaos at best, lol, but I have really come a long ways in improving the "organized" part. My professional life is actually a lot more organized than it appears.

    BTW, my mother is from Nagasaki. I haven't been to Fukuoka in many, many years, but really loved it there. I'm sure it's changed a lot since.
     
  19. DucatiTerminator thread starter macrumors member

    DucatiTerminator

    Joined:
    Jun 11, 2015
    Location:
    Moorpark, CA
    #19
    This is one of the things I'm trying to be very disciplined about as a LR newbie.
     
  20. Apple fanboy macrumors P6

    Apple fanboy

    Joined:
    Feb 21, 2012
    Location:
    Behind the Lens, UK
    #20
    Good idea. Too late for me 3 years in!
     
  21. Melizard macrumors 6502

    Melizard

    Joined:
    Jun 4, 2011
    Location:
    Canada/Germany
    #21
    I keep it simple: One 1 TB harddrive as my primary storage for photos only (I have only ~450 GB of photos), and one 2 TB harddrive for backup (with other data). One harddrive lives at home, the other at the office. I don't take photos regularly (just when I travel), so whenever I have a new photo folder I back it up immediately. I don't keep anything on my computer and I keep every photo I take. It's not a perfect system, but I'm not a professional photographer and I feel secure enough with this system.

    As far as archiving goes, I just keep simple folders with descriptive names (organized by year). I have subfolders that separate out the crap photos from the good ones. It's pretty easy for me to find things, but then again I've only been shooting for 5 years and all of my photos represent events.
     
  22. maflynn Moderator

    maflynn

    Staff Member

    Joined:
    May 3, 2009
    Location:
    Boston
    #22
    I need to do that more, but with storage being so cheap, I've had the mentality of just a bigger drive. I'll not go back and remove my old photos, but I think I'm starting to be a bit more aggressive in my culling process.
     
  23. OreoCookie, Jun 25, 2015
    Last edited: Jun 25, 2015

    OreoCookie macrumors 68030

    Joined:
    Apr 14, 2001
    Location:
    Sendai, Japan
    #23
    Personally, I think this is a perfectly legitimate reason, I just wanted to say that the costs you incur go beyond that of the body. Personally, I love the new Fuji cameras, I have a X100s and it's a camera I fell in love with. In a perfect world I'd probably get an X-Pro 2 next. In any case, I hardly use my dslr anymore (I have a full assortment of lenses, 2 flashes, etc.). The only problem I have with it is that my wife loves the X100s, too, and I hardly get to use it anymore ;)
    I also use HGST NAS drives in my Synology NAS (4 TB and 6 TB), according to Backblaze's stats, they are the most reliable.
     
  24. MCAsan macrumors 601

    MCAsan

    Joined:
    Jul 9, 2012
    Location:
    Atlanta
    #24
    I open my cards with Perfect Photo Suite Browser. It lets me quickly cull by seeing the jpg previews in the raw files. Then I drag and drop the keepers onto Lightroom for import. The remaining shoots are lost when I reformat the card in the camera. The Lightroom import preset fills out my copyright data and invokes a Develop preset that does all kinds of basic adjustments I do to all images (add clarity, sharpen, S curve..etc.) The preset also generates the 1:1 previews. I do not convert native raw to DNG format. The import process does not take long because there are much fewer images imported than images on the card. Once the import is done it is quick to do final edits as the 1:1 previews have already been created and to allow quick zooming.
     
  25. Miltz macrumors 6502

    Joined:
    Sep 6, 2013
    Location:
    New York
    #25
    Reading this makes me laugh a bit. My pictures all are all the place. But it's a organized mess. I can never not find something. So when I got my first Mac to use as a pro (I've had macs before but not for my photography) I was in a complete shock when the OS doesn't give me the same access to my photos like in Windows 7 Pro. I like to see my stuff at the file level and organize it that way. So long story short Mac OSX didn't work for me on a pro level, I just loved Windows "messy" file system. Storage wise I shoot RAW as well and always add internal storage to my PC which is not so easy on the newer MACS. And I also have a external 2TB which is an additional back-up. Brb... I need to go do a manual back-up right now. (Yes dragging my files across drives )And do what works for you.
     

Share This Page