Best (Cheap) RAID / Storage Solution?

[usayit]

One of the nice things about mirroring externals (albeit a bit minor) is that you can split the mirror and take the other drive with you in a pinch.

Also... don't forget that redundancy doesn't protect from corruption.

Back to the OP/Vautrin, what computer do you have? If Mac, the cheapest would be mirroring of two firewire drives (LaCie is what I like) via Mac OS X's Disk Utility (built right in). If possible, make sure the disks are on seperate firewire controllers. Its not fancy but it works. The striped set can be backed via Carbon Copy Cloner to another external on a nightly basis on a automatic schedule.

Software mirroring is also possible in windows. If you spend a little more, you can also consider getting hardware based RAID controllers.

I have my two SATA drives stripped for better performance. This is my workspace, applications, O/S, and swap. In addition, I have two externals (firewire 800 LaCie) that are mirrored for short term data redundancy. Everything eventually gets backed incrementally to magnetic tape and/or archival gold DVDs.

I also have a few externals from Acomdata. They don't get good reviews but mine haven't failed yet. They are significantly cheaper than LaCie.

 

[Dao]

I am working in an environment that have over 2000 servers. And RAID1, 5, 6, 1+0, 5+0 are common. RAID 6 still offer redundancy even if 1 of the drive in the system fails. Or you can install Solaris with ZFS and have the same thing with RAID Z. I believe FreeBSD and OS X from Apple also port ZFS to its OS. I think ZFS is really the filesystem of the future.

And I do agree that the above technologies will not stop data lost. Data backup is still needed.

If you really want to go further to protect your data (or images), Distributed File System (DFS) can help. Theorically, you can save a image to a file system. The system will put that file in a storage (file server) close to you. And in the background, it will replicate the file to another file server via the LAN or WAN. The user do not know what happen. All he/she knows is a file is save in the filesystem, the system itself will do the rest. You can have 3 servers like one in USA, one in UK and one in Japan. Let say you save all you files in a filesystem under your folder called myimages. Technically, all 3 servers around the world will have a copy. Now you are in USA and you connect to the file system. Just happen at the time you connect to the file system, the USA server is missing (i.e. due to Hurricane Ike, the Houston site is off line). You still have access to all your files in myimages folder and do not even know one server is missing in action.

And I do believe there is a service out there offer storage space for customer to store their images. You paid a fee and install a program in your PC. That software will monitor your image folder in your PC for changes. And it will replicate the files to their online storage site in the background.


Anyway, for the original poster's question, a CHEAP solution is get a external HD and move the files over and burn a copy to either CD blanks (last longer) or DVD blanks (more capacity) or even BlueRay blanks ($$).

 

[Tolyk]

I dunno... I'm in with film and video producers, 3D CG animation artists, matt and comp (composite) artists, and etc. I taught CG animation & comp for 8 years and know hundreds of post production houses and boutiques.

Read the hardware support sites tho and 99% of the questions and recommendations are about 0 and 1 IDE configurations.

The LaCie RAID unit Tolyk mentioned is RAID0 btw.

Actually, most of the units are capable of switching from RAID0 to 1 or even 5. The LaCie rep I've spoken with thinks that RAID1 is what the majority of photographers would want from a raid system.

 

[Moglex]

The LaCie rep I've spoken with thinks that RAID1 is what the majority of photographers would want from a raid system.

Which is pretty daft for all the reasons already mentioned.

A photographer typically needs all the speed s/he can get from a disk and long term backup.

All they really get from RAID1 is a possible very small increase in read speed, a definite decrease in write speed and high availability.

A decent g/f/s external disk system will give them rock solid backup and still manage very high availability, since if the main drive goes down they can just plug in the latest backup and carry on.

 

[Moglex]

Then just backup to CD or DVD as others have said.

CD and DVD are not suitable for backup of irreplaceable material. They are simply too unreliable. Commercially pressed CD's and DVD's may last for decades but those burned on a computer cannot be relied upon.

You can burn them, check them and find the perfect, and then at any time find they'll refuse to load.

Some people will tell you they've never had a problem but many more will have experienced failures. Do you want that sort of risk with material that can never be replaced?

The only way to be certain of retaining data is to write it to several hard disk drives and check these regularly and replace them every 2-4 years.

It may seem an expensive option but external hard drives are cheap as chips nowadays. Only you know how important your data is to you.

If you want to be super safe use two drives: one 1TB drive and compress backup to a second drive of 320GB which you normally keep off-line.

This is nonsensical. The formats in which photographic material is stored does not compress well (if at all). Use disks of the same size (as the area on which you store your shots) for backup.

Using a single disk for backup is also a recipe for disaster. You might find that your prime drive fails and the backup chooses the next startup to konk out.

Always use at least two drives for backup and make sure one is physically separate from both the other and your computer.

 

[Bifurcator]

Actually, most of the units are capable of switching from RAID0 to 1 or even 5. The LaCie rep I've spoken with thinks that RAID1 is what the majority of photographers would want from a raid system.

Hehehe, he's a silly guy. RAID1 would be for something like Video on demand. Where you have a few hundred thousand videos across many drives (all fast enough to serve the videos). If one drive dies you pull it and replace it and the customer never knows the difference. With RAID1 you're only using 50% of your total drive space. Four 1TB HDDs in a RAID1 is 2TB with zero speed increase. Also I would be willing to bet their RAIDs can do 0 and 1 but NOT 5.

If you have four 1TB drives in a RAID0 configuration you hace 4TB of storage space with about 310% speed of a single drive.

With RAID5 three 1TB drives will give you 2TB about 160% speed of a single drive. With a RAID 5 if a drive fails you usually have to shut down to replace the bad one.

• RAID 0
  Striped set without parity/[Non-Redundant Array]. Provides improved performance and additional storage but no fault tolerance. Any disk failure destroys the array, which becomes more likely with more disks in the array. A single disk failure destroys the entire array because when data is written to a RAID 0 drive, the data is broken into fragments. The number of fragments is dictated by the number of disks in the array. The fragments are written to their respective disks simultaneously on the same sector. This allows smaller sections of the entire chunk of data to be read off the drive in parallel, giving this type of arrangement huge bandwidth. RAID 0 does not implement error checking so any error is unrecoverable. More disks in the array means higher bandwidth, but greater risk of data loss.
  Minimum number of drives needed: 2
• RAID 1
  Mirrored set without parity. Provides fault tolerance from disk errors and failure of all but one of the drives. Increased read performance occurs when using a multi-threaded operating system that supports split seeks, very small performance reduction when writing. Array continues to operate so long as at least one drive is functioning. Using RAID 1 with a separate controller for each disk is sometimes called duplexing.
  Minimum number of drives needed: 2
• RAID 2
  Redundancy through Hamming code. Disks are synchronised and striped in very small stripes, often in single bytes/words. Hamming codes error correction is calculated across corresponding bits on disks, and is stored on multiple parity disks.
  Minimum number of drives needed: 3
• RAID 3
  Striped set with dedicated parity/Bit interleaved parity. This mechanism provides an improved performance and fault tolerance similar to RAID 5, but with a dedicated parity disk rather than rotated parity stripes. The single parity disk is a bottle-neck for writing since every write requires updating the parity data. One minor benefit is the dedicated parity disk allows the parity drive to fail and operation will continue without parity or performance penalty.
  Minimum number of drives needed: 3
• RAID 4
  Block level parity. Identical to RAID 3, but does block-level striping instead of byte-level striping. In this setup, files can be distributed between multiple disks. Each disk operates independently which allows I/O requests to be performed in parallel, though data transfer speeds can suffer due to the type of parity. The error detection is achieved through dedicated parity and is stored in a separate, single disk unit.
  Minimum number of drives needed: 3
• RAID 5
  Striped set with distributed parity. Distributed parity requires all drives but one to be present to operate; drive failure requires replacement, but the array is not destroyed by a single drive failure. Upon drive failure, any subsequent reads can be calculated from the distributed parity such that the drive failure is masked from the end user. The array will have data loss in the event of a second drive failure and is vulnerable until the data that was on the failed drive is rebuilt onto a replacement drive.
  Minimum number of drives needed: 3
• RAID 6
  Striped set with dual distributed Parity. Provides fault tolerance from two drive failures; array continues to operate with up to two failed drives. This makes larger RAID groups more practical, especially for high availability systems. This becomes increasingly important because large-capacity drives lengthen the time needed to recover from the failure of a single drive. Single parity RAID levels are vulnerable to data loss until the failed drive is rebuilt: the larger the drive, the longer the rebuild will take. Dual parity gives time to rebuild the array without the data being at risk if one drive, but no more, fails before the rebuild is complete.
  Minimum number of drives needed: 4

And there's some other weird combo configurations too like RAID15. RAID0 is what you want for large fast storage. Then just backup to CD or DVD as others have said. But actually I think with 1TB HDDs being pretty common now no one needs RAID for photographs. If you want to be super safe use two drives: one 1TB drive and compress backup to a second drive of 320GB which you normally keep off-line.

 

[Moglex]

Then just backup to CD or DVD as others have said.

CD and DVD are not suitable for backup of irreplaceable material. They are simply too unreliable. Commercially pressed CD's and DVD's may last for decades but those burned on a computer cannot be relied upon.

You can burn them, check them and find them perfect, and then at any time find they'll refuse to load.

Some people will tell you they've never had a problem but many more will have experienced failures. Do you want that sort of risk with material that can never be replaced?

The only way to be certain of retaining data is to write it to several hard disk drives and check these regularly and replace them every 2-4 years.

It may seem an expensive option but external hard drives are cheap as chips nowadays. Only you know how important your data is to you.

If you want to be super safe use two drives: one 1TB drive and compress backup to a second drive of 320GB which you normally keep off-line.

This is nonsensical. The formats in which photographic material is stored does not compress well (if at all). Use disks of the same size (as the area on which you store your shots) for backup.

Using a single disk for backup is also a recipe for disaster. You might find that your prime drive fails and the backup chooses the next startup to konk out.

Always use at least two drives for backup and make sure one if physically separate from both the other and your computer.

 

[usayit]

I think you guys have gone off the deep end. A few points from the OP:

1) He never mentioned anything about business critical machines
2) His term was "computer's". Meaning 1 desktop. Not a server farm. We are not event talking about a single server.
3) He said CHEAP. We are talking best buy cheap. We are not talking about EMC frames, hot swap storage units, etc...
4) His term was "USB". At this point, it is pointless talking about high performance RAID 5.

I'm in the storage and backup/recovery business. Even with that experience, what I do at home is completely different as the purpose and final goal is different. The worst thing to do while communicating with the customer is not listening. I bet this thread is utterly useless to the OP and is now just a contest.

 

[usayit]

This is nonsensical. The formats in which photographic material is stored does not compress well (if at all). Use disks of the same size (as the area on which you store your shots) for backup.

Not true. Uncompressed Tiff just to name one.

 

[usayit]

Vautrin (if you are still following this thread). If Mac, read my previous post. If windows, I suggest getting a Firewire card and running simple RAID 1. Here is a link for you:

http://eshop.macsales.com/shop/firewire/

Yes I know it says "Mac" in their domain name but all their firewire external hard drive products should be compatible with Windows machines assuming you install a firewire card. They are easy to deal with.

 

[Moglex]

Not true. Uncompressed Tiff just to name one.

Just because there may be formats which compress does not mean it's generally true that they do.

RAW and JPG certainly don't.

If you are certain that you will continue to use something such as uncompressed TIFF, by all means audit your storage and calculate what percentage of your files will compress, then calculate by how much on average, then calculate a safety margin and finally save a small amount but in general you're probably better off just matching the storage allocated on your main drive.

 

[Moglex]

I think you guys have gone off the deep end. A few points from the OP:

1) He never mentioned anything about business critical machines
2) His term was "computer's". Meaning 1 desktop. Not a server farm. We are not event talking about a single server.
3) He said CHEAP. We are talking best buy cheap. We are not talking about EMC frames, hot swap storage units, etc...
4) His term was "USB". At this point, it is pointless talking about high performance RAID 5.

Exactly!

That is why Alex B and myself have been playing down RAID and all its complexities and asking OP to consider what is really important in an archive/backup.

Great long screeds about RAID types really aren't helpful here.

Vautrin (if you are still following this thread). If Mac, read my previous post. If windows, I suggest getting a Firewire card and running simple RAID 1.

'A' simple raid 1?

That's fine if he's prepared to lose any archived data that resides on the RAID. It doesn't really address backup considerations. Setting yourself up with huge RAID systems without very carefully considering backup is foolhardy to say the least.

As to firewire, by all means go for that if you are certain that every machine you may want to connect the array to has a firewire port.

Be aware, though, that extreme high speeds actually aren't that important for archive and backup and at 50MB/s USB2 really isn't sluggish. It may be instructional to get a disk benchmark program and find out how fast your machine actually does read and write data!

 

[Dao]

hahaha .. it seems that the OP maybe scared to come back because of the direction of this thread is heading.

Anyway, As I said, the cheap solution is just move the files to a external harddisk. And make an backup with a quality DVDs like Taiyo Yuden. If you really think the shelf life of 30 years of those quality DVDs is not enough, back it up on another cheap harddisk and store it somewhere.

 

[JerryPH]

i was implying raid 5 here. no one i know these days still uses level 0 or 1 anymore.

As for raid, if all drives are the same built, often soon after the first drive fails, one or two others die too. Very bad if you do not like data loss. seen it happening often.

Actually, if you want maximum speed AND maximum fault tolerance *both* RAID 1 and 0 in tandem are what you want (RAID10).

RAID 5 is reserved for servers, rarely for workstations, but it is done.

For photography use, I suggest people look at the WORLDBOOK 2TB setup (basically two 1TB hard drives), but mirror it down to 1TB total space.

It is either USB 2.0 or a 10/100/1000 network connection in an external unit. For home use this is about as high as 99% of the people need.

Now if you want to get fancy, I have an 18TB SAN (fiber optic) off of 40 1TB SCSI drives in my basement. Yeah, its massive overkill for my needs... but it was also free.

 

[usayit]

Exactly!
'A' simple raid 1?

That's fine if he's prepared to lose any archived data that resides on the RAID. It doesn't really address backup considerations. Setting yourself up with huge RAID systems without very carefully considering backup is foolhardy to say the least.

No kidding? I work for the market leader for enterprise DR. BUT the OP didn't ask about backup strategies. I also mentioned that redundant disks doesn't protect from data corruption.

On my Mac, setting up mirror was pretty much free and easy (all you need is the disks). Can't be any more simple or cheap.