Putting together grant proposal to Archive 25 yrs of VHS/DVCam

[treehorn]

The company I am doing most of my freelancing for is working on getting grant money to archive its 25+ years worth of tapes (VHS and DVCam) to hard drives, in order to A) preserve them and B) make them more available for licensing/purchase/etc.

Of course it has fallen upon my shoulders to figure out the best way to implement this and price out options (don't worry...I have plans on how to make it worth my while and I would love some input, especially from people who have done similar project.

The office has two computers at its disposal:

1) 2010 MacPro

2) 2004 Power Mac G4


Both have Final Cut Pro installed. My thought is to have one set up for the DVCam (already have a Sony deck) and one set up for VHS (already have a Broadcast-grade S-VHS machine)

As the thought is to do multiple backups, I'm thinking a 2 or 3 drive RAID would be the best option - that way the single capture would generate the backup (or two) most efficiently. Plan is to just capture the footage as raw uncompressed DV for optimal quality. Would you suggest/recommend keeping the footage on internal drive, or is there any significant benefit to having an external backup as well (the thought that an external drive would be more durable and would be easier to pull footage from if needed in the future has crossed my mind)

I'm guessing that the best RAID option would be to get something like the OWC Mercury Elite Pro RAID/Quad Interface Performance RAID (http://eshop.macsales.com/item/OWC/MEFW936AL2/) and buy a SATA card for each of the machines. If that is the case, what cards would you recommend for each machine?

Given the age of the VHS tapes, I know that they would benefit by running through an Analog/Digital converter. Was thinking of the Canopus/Grass Valley ADVC300, unless anybody has a better suggestion. I'm a bit confused, though, as to whether or not one needs or should get a Time Base Corrector in conjunction with it. Some publications say it works great as is, some forums complain that it really doesn't have a Time Base Corrector built in and that it's a misnomer.

The goal is to get everything in place to do this right from day one, as we already have funding for equipment and expenses and are close to getting funding for more 'time.' Realize that this is thousands of hours, but it's basically going to be a 'full time job' and want to make sure it's done optimally and future proofed from the get go.

Thanks for any and all help

 

[giffut]

I ...

... wouldn't rely on RAID for backup by any means - chance of failure grows with raided volumes and rebuilding huge data chunks is time consuming and error prone, too - but for a setup like this:

1. All captured video is in DV format and stored on single discs, archived accordingly to your scheme

2. Each disc is mirrored on one/two (depending on your requirementsvfor data integrity) similar discs, which are stored offsite, just for the purpose of backup. The discs need to be run from time to time. Use firewire 800/USB3/eSata port multiplier enabled cases from a established vendor to have redundancy of ports and reliability of performance

3. A sufficiently sized raid 6 array holds all the material for immediate access, a SSD drive is used for working with the material

4. If possible keep the VHS/DV Tapes as a last line of backup - and 2x capture machines for each media, too, of course

Now, the ADVC 300 makes perfekt sense, but I would think about buying up to four of these to capture several streams simultaneously and get TBC equipped SVHS machines. Your Mac Pro should have no issues at all capturing many streams at once, even some more. In that case, you best use a RAID 0 2x 2TB HDs.

The G4 could be of best use to access via 1GBit network the newly captured streams and to categorize and deliver them to their respective storage points.

Use enterprise grade HDs ans SSDs, maintain the analog capture gear as intended by the manufacturer and seriously map out your storage/categorization structure ever before you start capturing.

 

[treehorn]

Now, the ADVC 300 makes perfekt sense, but I would think about buying up to four of these to capture several streams simultaneously and get TBC equipped SVHS machines. Your Mac Pro should have no issues at all capturing many streams at once, even some more. In that case, you best use a RAID 0 2x 2TB HDs.

How would one go about doing that? I didn't think you could get that much via Firewire (and what program would you use, as FCP only would work for one stream at a time)

 

[giffut]

I ...

... did this quite some time ago, with two Dazzle Hollywood bridges. The trick was to duplicate the app one uses (Quicktime Player, MPEG streamclip or similar) as often as you have capture devices to use, then run each app dedicated to one device only.

 

[initialsBB]

You definitely should look into getting a solution to capture multiple streams at once. I don't know if you've estimated the quantity you have to capture, but seeing as it's going to be real time... Any modern RAID will offer ample bandwidth for your needs. DV is 25 megabits/s, and there is no advantage in capturing VHS above this value either (it is composite video). So at 4 simultaneous streams you are looking at 15 MB/s writes maximum.

Do the tapes have a logical naming convention in place ? Have you got a serial-number like system to match tapes to the digitized files ? And don't forget that you will likely have to deal with timecode drops and breaks in DV. It's a good idea to devise a naming scheme and folder structure that can make your life easier.

Look into getting an LTO5 tape deck as a good digital archival solution, and it's easy to ship the tapes off-site as well. LTO5 stores 1.5 TB, we use them at the end of our backup/archival workflow.

 

[treehorn]

Thank you both for your input and giving me lots of things to think about.

Here's the deal: we are, unfortunately, talking about an individual (not me) who has very VERY limited faculties in regards to computers, as is evidenced by the number of phone calls and emails I get on a daily basis to help fix the most basic things...repeatedly. Therefore this system needs to be as simple as possible. Running two different machines at once is probably stretching the limits as it is (the number of times he'd forget he was loading in a 30 minute shoot until two hours after it had been running is astounding).

So I don't think multiple decks running into multiple programs running the same computer would...be a feasible option...at least at first.

And thanks for bringing up the LTO5 tape deck option. My gut feeling on that, though, is that it is defeating the purpose of this process - to get the materials off of tapes which have a limited shelf-life and aren't that easy to access. While it would be compiling the tapes into one much larger (and more usable tape), in the long run it is still tape.

And double thank you for reminding me that all these suckers will have to be spun and run every now and then.

This is, to be honest, a nightmare project that I am not at all enthused about or wanting to be a part of beyond advising on options so thank you for giving me options!

 

[giffut]

You ...

... have no chance of success when this person you describe is in command & control. Don't do it then. You can't make it foolproof, especially not with your budget in mind.

You could try to get to your local college/university and hire students to do the rough work of capturing - because that's what it really is. It involves lots of manual work with the devices involved, while being focused onto the plan.

Don't forget, that you need to layout the definite organisation, retrival and backup structure and you need people, who understand it while implementing.

 

[initialsBB]

Wirelessly posted (iPhone: Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_5 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8L1 Safari/6533.18.5)

I agree with giffut, you don't have a snowball in hell of a chance if that person is in charge. Making a cost effective solution means getting your hands dirty.

LTO5 tape has a much longer shelf life than VHS, and it really is a great way to "compile" many tapes into one. The DVCAM stuff is digital, and with correct timecode/LTC you can batch capture. But that is impossible to do with VHS... LTO5 doesn't have to be spun up every now and then either !

 

[treehorn]

Not my business to succeed or fail, merely to suggest

And there's no worry about budget...(not sure why you think it's out of any budget as no $$$ was mentioned...). The initial investment will cover a salary of a person for a year (minus equipment costs) and more is on the way.

Agree whole heartedly that this is something that should/could be farmed out to interns, etc. But again, not my decision. Hopefully I'll be onward and upward...

I'll suggest the LT05...but honestly that's not what he is wanting - he is wanting to digitize everything onto hard drives to archive and/or be able to license it out. While I agree that it is not...necessarily...the most practical idea, I'm treating it was a full-time job for this individual for several years (without me involved...I'm just trying to implement his needs/wants and bite my tongue as often as possible)

 

[Serge88]

Don't forget, you need database software and you need to write meaningful metadata everytime you capture a tape. Otherwise it will be a mess to find something.

Sergio

 

[LethalWolfe]

My 2 cents.

1. Hard disks are designed to be active storage and can die / lose data just sitting on a shelf which is why data tape is used for back ups.

1a. The rule of thumb I'm hearing about digital archiving is be prepared to migrate your digital archives onto new media and new file formats every 5-10 years as technology changes.

2. I would suggest using a less proprietary codec than Apple's DV codec to record the footage. M-JPEG @ 75% quality, for example, is seen as a good quality, platform agnostic codec. This would necessitate using a different capture card such as the Blackmagic Design Intensity Pro.

3. Having a robust database and proxy quality videos for potential customers to download is very important. No one in their right mind is going to pay for footage sight unseen and the best way to preview footage these days is to download proxy quality versions from the web. CatDV is a popular digital asset management program.


Lethal

 

[treehorn]

Thank you all for giving me things to think about (and for making my head explode...)

 

[treehorn]

Out of insane curiosity...

Not knowing anything about the mechanics and potential possibilities of the aforementioned LTO5 tape deck option, I'm wondering...

Is it at all feasible to do a dual pronged approach, where one archives the master tapes on the LTO5 tape deck, and then backs up the data from the LTO5 tape deck to an external drive (or two), thus giving at least some aspect of 'best of both worlds' scenario?

Given that the word 'tape' is in LTO5 tape deck, I'm not sure how one extracts data from it and would be able to transfer it to a drive...but exploring options (and keeping head from exploding from all the help and advice, which I really do appreciate)

 

[LethalWolfe]

Cache-A offers LTO5 / HDD combo units and they can easily be setup on a network (if desired) and use a pretty straight forward drag and drop interface (similar to an FTP client). They also can work w/asset management software such as CatDV.


Lethal

 

[initialsBB]

Wirelessly posted (iPhone: Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_5 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8L1 Safari/6533.18.5)

But I don't think it's at all possible to dub the master video tapes to the LTO tape, if that's what you meant. You first have to offload the video to a computer file before backing up to LTO.

To that effect the cacheA solution proposed by LethalWolfe may be exactly what you need.