Need some suggestions for Data storage/Backup strategy

[brandisi]

I am hoping that I could get some suggestions from anybody on this Forum.


I have a situation about how to allocate or set up a practical / easy backup strategy. I have discovered that one of my DR (RAID) is almost full. And a second DR should be full in a couple of months.


Let me show you the quasi system that I have setup:


I'm using a Mac OS 10.8.5 single user.


1 TB int HD that don't use it much since I rely mostly on Ext HD - just had it replaced a week ago after running great for 4 yrs.




DR #1
RAID 1 - 4 yrs old
965G used - 40G avail




DR#2
Partition One - 4 yrs old
775 G used - 228G avail
Use it as a Main DR, incl pix, videos and documents


Partition Two - LR Cat
900G used - 100G avail


DR#3 - 4 yrs old
Partition One
Time Mach
430G used - 572G avail


Partition Two
LR Backup 70G use -928G avail


I also have a couple of older DR with extra copies of most of my images, but they could go bust anytime since they are a bit older. I'm using them as long as they run


I just purchased a 3 TB Ext HD.


I use mostly my computer for my photography hobby as well as building up a small video MP3 Library.


I am now realizing that perhaps RAID 1 was not a great strategy for me. It cut the capacity of my DR to half. After reading online that RAID 1 is not a substitute to a regular backup strategy. I'm now wondering if I should format the RAID DR and start all over again.


However, I'm not sure how I could access all the files without capturing the mirrored files.


I would like to know if I could get some suggestions.


BTW I also have SuperDuper but I have not used it for quite sometime. I have been creating backups manually, but I think I might start using SuperDuper or something similar. Thanks for any help

 

[bratkinson]

As a former mainframe computer consultant for over 35 years and amateur photographer, I initially looked at RAID some years back and decided against it...even though one of my former coworkers and friend claims RAID is the way to go.

The problem is that RAID protects you from one risk only...a failed hard drive. It does NOT protect one from fire/theft/tornado/flood, nor does it protect you from viruses (virii if I recall my Latin of 50+ years ago) such as the one that encrypts everything on your computer and demands a ransom to decrypt it. Even though I have multiple firewalls and one of the better Internet Security products and keep it up to date, at best, it's 24 hours or more 'behind the curve' at detecting and preventing/removing a new virus.

So, for me, I have physical duplicate drives that I clone from one to another and then remove the original drive and store it on my shelf. Perhaps 3 times per year, I make a clone of everything to a portable hard drive and store that off site, in case of fire, theft, etc. As I have all my drives in slide in/out bays, adding/removing them is simply a matter of power down, slide in or out drives as needed, and power back up.

 

[JClishe]

As the previous poster said, RAID is not backup. You don't have a backup solution, and you need one.

Also, if you're using the acronym DR as Disaster Recovery, you need to understand that you don't have a DR solution either. RAID is neither DR nor backup.

 

[CameraClicker]

I have customers for whom up time is critical. Their machines must run all day, every day, reliably. Raid 5 has carried them through a couple of drive failures, so it gets my vote for that environment. As was already posted, it does nothing for fire, flood, theft, etc. So, every night the machine makes a tape backup of the whole. Tapes are taken off site the next day and a new tape is put in the drive for that night.

Doing photography, my storage is capacity oriented. I use multiple 2, 3 and 4 TB external USB drives. A local spare set, an off-site set and the live set have worked well for me so far. I used to put everything on DVD, but having hundreds of them tends to get old. Large USB drives are less time consuming and probably just as secure. You have to have the discipline to keep them up-to-date though.

 

[brandisi]

As a former mainframe computer consultant for over 35 years and amateur photographer, I initially looked at RAID some years back and decided against it...even though one of my former coworkers and friend claims RAID is the way to go.

The problem is that RAID protects you from one risk only...a failed hard drive. It does NOT protect one from fire/theft/tornado/flood, nor does it protect you from viruses (virii if I recall my Latin of 50+ years ago) such as the one that encrypts everything on your computer and demands a ransom to decrypt it. Even though I have multiple firewalls and one of the better Internet Security products and keep it up to date, at best, it's 24 hours or more 'behind the curve' at detecting and preventing/removing a new virus.

So, for me, I have physical duplicate drives that I clone from one to another and then remove the original drive and store it on my shelf. Perhaps 3 times per year, I make a clone of everything to a portable hard drive and store that off site, in case of fire, theft, etc. As I have all my drives in slide in/out bays, adding/removing them is simply a matter of power down, slide in or out drives as needed, and power back up.

Thanks for the input. I'm glad to hear that we both feel about the same about RAID 1. Please keep in mind that I have 3 backup copies. The reason that I posted my question was to get some suggestions as to the best way to save my data among all the Ext HD drives I have. If you notice that my RAID is already full so I need to make a decision fast.

I'm getting close to make a decision to use my RAID drive and transfer the data to my new drive. But here's a question if I do transfer from RAID to a new drive would I be copying a single file or the entire RAID, ie duplicate folders ets.

One thing that I was not thinking was to get portable drives because I thought they were not as fast or robust. However, I do find them very attractive from the price and size stand point.

Do you have any other suggestions about my current set up.

 

[brandisi]

Yes, I agree about using DVD as backup. I already have too many so I'll be dumping them I don't have that much space. I just realized that I could get some portable drive for backups but for daily use I think the full size drives would be better.

Again, let me ask one more time for any suggestions as to migrating to a new Ext drive since my RAID is already full. I don't think I'll be using RAID anymore.

I also got to think the impact later when my LR drives gets full and I need to replace the drive if I would need to make a new catalog

 

[usayit]

In short, RAID addresses the issues related to high availability.. the ability for a storage device to be resilient to down time (disk failure). It doesn't solve the problem of data protection nor data recovery (corruption and accidental deletion).

If you want data protection, you need duplicated data off-site (at the very least). "Recovery time objective" speaks to how quickly you need to recover the data. "Recovery point objective" speaks to how much data you are willing to loose. Both are expressed in terms of time. The ability to recover data quickly (short RTO) will result in higher costs; generally in hardware. Less exposure to data loss (short RPO) will result also higher costs; generally in logistics of producing backups more frequently. There is a place and time for both RAID and data protection/recovery.

At home, I do not use RAID. High availability is not important to me on my home computer. If disk failure denies the use of my computer for days/weeks, it is not skin off my back. I'm also less concerned over RTO but I don't want to waist a weekend recovering my data either. I am concern over RPO But I don't consider my photos "that" important... I've decided that a month's of data is "ok" to loose although I wouldn't be totally happy but that's the extent I'm willing to invest logistically. To summarize RAID (not important), RTO (several hours), RPO (1 month) for my primary. In addition, I also want the ability to archive more important data for longer term storage. My data set is only about 750GB

So what fits my needs.

* 12 hard disks. 1TB each, Internal type, SATA. Internals are cheaper and found more often on sale. Recovering from DVD/CDs, although cheaper, will take more than just a few hours and cost (time).
* Thermaltak drive dock that connects via both USB and eSATA.
* Lockable storage at work.

Each month, 1 of the 12 disks is at home connected and mounted on the dock. Before going to bed, I kick off a task to duplicate all of the data on my 750GB partition to the disk. The next morning, I unmount the disk and take it to work and lock in my desk drawer clearly marked as personal. I then take the next disk home and do the entire simply process over again the following month. The result is 12 months of recovery points at 1 month intervals which are stored off-site with a full recovery that involves a few hours of copying data. The personal time (cost investment) is small.

Once per year, I take the files I have marked for archival and I burn them to CD's and DVD. Brand is Delkin archival gold. I hate doing those.

We also have a netgear router with attached disk that serves as a time capsule. Apple time machine is used but not relied on. Basically it came as a free added bonus when I bought the new WIFI router.

There are also cloud based backup solutions. They provide off siting of data BUT at a higher cost (subscription and per byte storage) with surcharges for recovery. They do use a continuous form of base+incremental backup which results in a short RPO. RTO can be long over network. I am also skeptical of their backup strategy... the customer agreement always has disclaimers that limits liability to the cost of the service itself.... not the value of the data which is common in data protection. (The software I work on also has such limits of liability as the ultimate responsibility lies with the consumer).

Its important to test your backups.

 

[xzyragon]

As someone who runs the data server for our urine toxicology lab, our protocol is as follows:

Data stored on local computer for a year. Each machine acquiring data has 2 2TB HDDS in raid1.

All data is copied onto our in-house server. 6 3TB drives running in raid 10 (this is where we store the main working files). This protects against file corruption, hard drive failure, and accidental deletion.

All data is backed up once a week onto an out-of-house location (or any data that's been added or changed really). This protects against earthquakes, fires, or theft.

Granted we only generate 1-2TB of raw data per year, so we don't need overly large space requirements. If money was no object, I would have 3 or 4 3TB hdds in my computer in raid 5, and I would purchase a solution to back all my data up off-site weekly. If money is a concern, just run 2 3TB drives in raid0 and backup off-site weekly. Don't clear your cards until after you get a guarantee that it's backed up off site.

 

[usayit]

All data is copied onto our in-house server. 6 3TB drives running in raid 10 (this is where we store the main working files). This protects against file corruption, hard drive failure, and accidental deletion.

Assuming this is RAID 1+0 (stripe and mirror), I would like to know how this protects from file corruption and accidental deletion.

 

[CameraClicker]

All data is copied onto our in-house server. 6 3TB drives running in raid 10 (this is where we store the main working files). This protects against file corruption, hard drive failure, and accidental deletion.

Assuming this is RAID 1+0 (stripe and mirror), I would like to know how this protects from file corruption and accidental deletion.

Years ago, we had a raid system that lost the controller. It promptly corrupted both sets of drives. That was a pain! I'm not a fan of doing restores from a dozen or three tapes, so I have customers get a large enough drive to hold a full backup on one tape and it gets backed up every day. If I have to rely on a tape restore, I'm a day behind and have the whole thing on the tape. If I have a bad tape, I'm two days behind, or three at worst. If tape 6 in a 12 tape restore is bad, that's a problem. Having the whole thing on a large external drive makes restores even faster and easier if it is just required due to a hardware problem.

 

[usayit]

All data is copied onto our in-house server. 6 3TB drives running in raid 10 (this is where we store the main working files). This protects against file corruption, hard drive failure, and accidental deletion.

Assuming this is RAID 1+0 (stripe and mirror), I would like to know how this protects from file corruption and accidental deletion.

Years ago, we had a raid system that lost the controller. It promptly corrupted both sets of drives. That was a pain! I'm not a fan of doing restores from a dozen or three tapes, so I have customers get a large enough drive to hold a full backup on one tape and it gets backed up every day. If I have to rely on a tape restore, I'm a day behind and have the whole thing on the tape. If I have a bad tape, I'm two days behind, or three at worst. If tape 6 in a 12 tape restore is bad, that's a problem. Having the whole thing on a large external drive makes restores even faster and easier if it is just required due to a hardware problem.

Understood.... (I have experience in traditional tape recovery solutions). What you are describing is the difference between a simplified single tape base backup strategy versus a base+incremental strategy. Both have their advantages and disadvantages. Back when media was rather expensive, stakeholders would make the tactical decision to perform base+incremental to avoid duplication of data on media thus cutting costs down. This was at the expense of RTO (multi-tape re-stage) but it was ideal for datasets that had a low percentage of changed data (incrementals only backed changed data). Less space and less backup time. The decisions around the frequency of base vs incremental backups were around the parameters of risk (single tape failure across a set), cost (time and money), and time to recovery.

Absolutely, your experience with RAID is very common. But many administrators I have worked with will claim RAID protects them against corruption, and accidental deletion. WHen I question further what is really happening is that their RAID is working in conjunction with some sort of other technology. RAID is providing the HA but something else working behind the scenes is providing the protection from corruption and accidental deletion. Often it's a snapshotting technology of some sort via an intelligent controller.. (example NetApp).

I just wanted to ask xzyragon further to see if this was the case and to clarify. RAID stripe and mirror commonly found at consumer box stores will rarely provide the same protection against deletion and corruption as a proper backup.

 

[xzyragon]

All data is copied onto our in-house server. 6 3TB drives running in raid 10 (this is where we store the main working files). This protects against file corruption, hard drive failure, and accidental deletion.

Assuming this is RAID 1+0 (stripe and mirror), I would like to know how this protects from file corruption and accidental deletion.

Because we have the original un-tampered files that no one can touch on the machines they were analyzed on, with working files on the server. If someone deletes a file, tampers with a file, or corrupts a file on the server, we have the original as backup. The RAID 10 is just in case a HDD fails. I was responding from my phone so I probably didn't clarify that.

If you accidentally delete or corrupt anything on any raided system, odds are it's gone if you don't have an external backup. RAID is just for performance and in case an entire HDD or partition fails, at least in my experience.

 

[brandisi]

I really appreciate all the input and suggestions I got from all of you. Believe me, I am better informed now than I was before. For one thing is that your input has reinforced me not to use RAID 1. I think I might consider buying portable drives for backups since they don't take a lot of room, etc and I won't be using them on a daily basis.

However, I still I haven't received a response to my question. As I mentioned previously, I'd like to transfer all the RAID files to a temp drive. But one thing that I'm still concerned if I transfer those files would I be getting redundant files or just single files because the source being a RAID system . Please let me know before I'd make a big mistake thanks a bunch for all the help

 

[pthrift]

I really appreciate all the input and suggestions I got from all of you. Believe me, I am better informed now than I was before. For one thing is that your input has reinforced me not to use RAID 1. I think I might consider buying portable drives for backups since they don't take a lot of room, etc and I won't be using them on a daily basis.

However, I still I haven't received a response to my question. As I mentioned previously, I'd like to transfer all the RAID files to a temp drive. But one thing that I'm still concerned if I transfer those files would I be getting redundant files or just single files because the source being a RAID system . Please let me know before I'd make a big mistake thanks a bunch for all the help

Even if they are redundant couldn't you just go back and delete the duplicates after? Or is there something else I don't understand here?

 

[usayit]

I really appreciate all the input and suggestions I got from all of you. Believe me, I am better informed now than I was before. For one thing is that your input has reinforced me not to use RAID 1. I think I might consider buying portable drives for backups since they don't take a lot of room, etc and I won't be using them on a daily basis.

However, I still I haven't received a response to my question. As I mentioned previously, I'd like to transfer all the RAID files to a temp drive. But one thing that I'm still concerned if I transfer those files would I be getting redundant files or just single files because the source being a RAID system . Please let me know before I'd make a big mistake thanks a bunch for all the help

RAID 1 (mirror) duplicates at the block level across two sets of disks (or two single disks) managed by a RAID controller. The underlying controller is responsible for abstracting that complexity. As a user, a proper RAID 1 should be completely transparent; you shouldn't know whether your writes are to a single disk or to a RAID configuration.

As such a file level copy would behave identically on a RAID vs non-RAID disk. You should only get a single copy of the file.

For the Mac user (i'm sure also for Windows), you can use a variety of software solutions to make copying the data to your backup disks less complicated. Most are free or free to try... try a few, test them out, and see which tools work for you. I use SyncTwoFolders.

SyncTwoFolders for Mac - Free download and software reviews - CNET Download.com

UI could be easier to use... but I got used to it.

Another very commonly used tool is Carbon Copy. I have an old copy that I still use today. You can even schedule it to make copy files at night.