Image Organization - Storage

[Heather Koch]

Okay, the time has come people! My computer is FULL, yes full!

I purchased a MacBook in 2012 and recently have been getting "storage almost full" messages, well its now FULL.

I am here today to seek a way to organize and store my images. Of course there is the obvious SD cards, memory sticks, CD's, etc. BUT I want to know whats the best and safest (images won't get lost) way to do this?

Thanks in advance!!

 

[TCampbell]

Use external hard drives.

There are some choices... you can buy a storage array or you can buy individual external drives. For arrays... 4 drive arrays are common and these can be set up as RAID 5 (the Mac will do software-based RAID 5 on a "JBOD" (JBOD array = "Just a Bunch Of Disks), or better arrays do RAID 5 in hardware (the computer doesn't know it's 4 drives... the RAID presents to the computer as one large drive and it handles the RAID all on its own.)

RAID 1 is simple mirroring. But for four drives that means you only get the space of 2 drives (because the other 2 are "mirrors" for reliability.) For this reason, RAID 1 is usually only used when you only have 2 drives.

When you have more drives (4 is common) you use RAID 5. In RAID 5, three spindles hold data and the 4th holds a checksum data. The data is stored in such a way (using "XOR") that any single drive can fail and the array will drop into "degraded" mode and continue to function because it has the checksum drive. RAID 5 is able to 100% recover the data as long as no more than ONE drive fails (if two spindles fail, then you lose data and it cannot be recovered.) So the beauty of the system is that it's unlikely that 2 drives will fail at the same time. You just buy a spare drive (which isn't in the array). When one drive fails, you swap out the failed drive for the spare and the storage array will go into recover mode where it rebuilds the data on the replaced drive. I have three storage arrays that use this technology.

And then there's the simple single external drive option.

I have a NewerTech Voyager Pro external drive dock: Voyager by NewerTech - Hard Drive Dock for 3.5 and 2.5 SATA Devices provides high-performance and flexibility

You buy SATA disks (not in an enclosure... I think these would cost around $50 for a 1TB drive.) You plug it into the dock, and it shows up as an external disk. Copy all the data you need to it (or use it as your library). When it's full... unplug that disk and drop in another disk... and just keep going (label your disks and store them.) The downside of this approach is that you have no redundancy -- so it's not as "safe" as using a RAID 5 storage array.

If you happen to own an Apple Airport Extreme base station (one of the more recent models) then they'll have a USB port on the back. You can actually attach the storage to that port (you could hookup the storage array to that port) and it'll function as "network attached storage" (NAS). That means it can appear as a drive on your Mac that you don't actually connect physically to your Mac -- it's connected via the WiFi network. If your fairly mobile with your laptop, that's a bit more convenient because you don't have to drag the storage array with you to use your photos.

 

[Heather Koch]

Thank you Tim for the very detailed information!

I have found two very similar external hard drives on amazon.

Seagate and WD

Both have 4+ ratings and have several reviews (good reviews) and both are very similar in price.

I have a macbook pro with 500gb of storage, I currently have half of that storage full of photos, and only have 2.xx gb left!

With any external hard drive, can you pick and choose which files you want backed up? Also with external hard drives do they simply take the files off your computer and save them on the drive? OR just backed up?!

 

[luckychucky]

Me too, I'm getting there. That was some good info sharing. Thanks


Sent from my iPad using Tapatalk

 

[Dave442]

I also have a Mac Book Pro and with the SSD drive there is no room for photos. So,
1) I have a big selection of my photos on an external USB drive that I take with me with the computer.

2) I have a couple more of these drives that stay at the desk and have all my photos. All these discs are formatted with the Mac OS Extended Journal. These are set to automatically back up my Archive (RAW) directory and Developed (JPG) directory on my working portable drive.

3) Then I have are individual external hard drives that backup these main backup drives with one year of photos on each drive, those I formatted so any computer can read them. This is where you can use the Drive Dock that Tim mentions, sounds like a good option.

4) I also have about 2tb of my photos uploaded to cloud storage (not including Flickr that backs up most of my JPG files from a number of sources).

I think a good option is a network storage device to use in step 2 above (had one about five years ago but never upgraded to a higher capacity unit). I don't use the RAID1 or 5 as Tim noted (probably something a business might consider) as I have more of an archive of photos and not multiple discs with an exact copy. You can use RAID1 or 5, but then need to back that up anyway (step 3).

I did have a Mac Book and external hard drive stolen last year at the end of a trip. Lost about two months of RAW files, but had already uploaded the selected JPG files to the cloud and the last week of the trip were still on a few CF cards. I only reformat a used CF card when I put it back in the camera.

 

[TCampbell]

I think a good option is a network storage device to use in step 2 above (had one about five years ago but never upgraded to a higher capacity unit). I don't use the RAID1 or 5 as Tim noted (probably something a business might consider) as I have more of an archive of photos and not multiple discs with an exact copy. You can use RAID1 or 5, but then need to back that up anyway (step 3).

BTW, there is something called "RAID 0". RAID 0 does something called "striping" which boosts performance doesn't does not provide any redundancy. It just spread the data across as many disks as you have to increase the throughput (so reads and writes are much faster). But RAID 5 also spread the data around while also providing redundancy so you can recover as long as no more than 1 drive fails.

If you're not a storage expert... you might want to look at a Drobo: Drobo Data Storage Solutions Network Attached Storage Drobo Inc.

Drobo is self-managing storage. If you plug in 2 disks, it'll do a RAID 1 "mirror" (if it's 2 drives of 1 TB each then you only get 1 TB of storage because the 2nd 1 TB is just a mirror of the first.) BUT... plug in a 3rd drive and it'll convert your RAID 1 mirror into a small RAID 5 array... so 3x 1TB drives would give you 2 TB of storage. As you keep adding drives, it'll automatically re-organize itself to optimize space and performance while still providing redundancy. In a normal RAID 5 type storage array, the individual drives should all be the same size and ideally even the same model (if they're mis-matched then it'll treat all disks as though they only have as much space as your smallest disk -- so you waste space.) With Drobo... that doesn't actually matter. You can mix & match.

So the nice thing about them is... YOU don't have to know anything... just feed it disks and it'll do the smartest thing possible with the amount of disk space you give it. It has colored lights (green/yellow/red) to tell you everything is fine with that drive, the drive is low on space and you should add space, or the drive has failed and you should replace it.

In Drobo (as well as with most true hardware raid boxes) you can "hot swap" the disks... meaning you don't have to shut it off to replace a failed drive. You can eject a drive spindle while the storage array is active and in-use. Pop in a replacement drive and it'll immediately go into a mode where it starts recovering ... all the while STILL continuing to service your computer's storage requests (although performance is degraded while it recovers.)

If you shoot a lot of pictures... and some point you will eventually over-flow your hard drive.

My photo library is actually on an external 4 drive RAID 5 array. I don't keep any photos on my internal hard drive. You'll get much better performance out of your computer if you don't let your internal disk get more than 50% full.

 

[snowbear]

I have a crap load of these in shoe boxes

 

[Buckster]

Thank you Tim for the very detailed information!

I have found two very similar external hard drives on amazon.

Seagate and WD

Both have 4+ ratings and have several reviews (good reviews) and both are very similar in price.

I have a macbook pro with 500gb of storage, I currently have half of that storage full of photos, and only have 2.xx gb left!

With any external hard drive, can you pick and choose which files you want backed up? Also with external hard drives do they simply take the files off your computer and save them on the drive? OR just backed up?!

I have to give you a warning from my experiences over the years. I've owned and used several of those little hard drives over the years, and still use one here for other purposes. I've also given several of them away to family who needed them.

The bottom line is that they're not reliable enough on their own to be used as a backup, IMHO. They WILL fail, and I mean a LOT sooner than the more robust internal hard drives that would be placed into bays in the system or into array bays. I VERY recently had a brand new one of the new Seagate's last less than a month, then POOF - done. Amazon replaced it, no problem, even paid for return shipping on the dead one, but the data was gone - the end. Luckily, I don't trust them enough to rely on them, so I didn't actually lose anything I couldn't replace. I've learned my lesson over the years.

My regular backup storage consists of two 4-Bay enclosures each with another single-bay enclosure on top of them. In each of those 10 bays are internal SATA hard drives of 1TB to 4TB each. The left stack of 5 bays is a mirror of the right stack of 5 bays, redundant in every way.

Originally, when I put it together several years ago, they were each 1TB drives. As a hard drive died, it was replaced with a new hard drive, and its mirror in the other bank was used to make a copy. As hard drive capacities grew, I upgraded dead 1TB hard drives to larger capacity drives.

In addition, I have another 4 hard drives in the system unit, besides the operating disk itself, which has a clone up on the shelf of the OS and the base core programs I use, ready to swap out if needed, in case the operating system drive fails or gets clogged up with program crap that slows things down over time.

To answer your specific questions, yes you can pick and choose which files to back up. They just act as another drive with whatever files you put on them, whether that's another copy of the same files you have elsewhere, or the only copy you have, moved from one drive to the other. You can also use them with backup software to make backups that require the software to access the individual files for restoration, if that's what you want to do. They're just another hard drive, just like working with the contents of your current hard drive(s).

 

[Rick50]

If you have taken less than 10,000 photographs just trash those and get out there and shoot more. When you reach 10,000 then you can start to save (according to Bresson)

 

[AceCo55]

I have a crap load of these in shoe boxes

A lot of my students would ask why you keep so many "SAVE" icons in shoe boxes!!!

 

[qleak]

Thank you Tim for the very detailed information!

I have found two very similar external hard drives on amazon.

Seagate and WD

Both have 4+ ratings and have several reviews (good reviews) and both are very similar in price.

I have a macbook pro with 500gb of storage, I currently have half of that storage full of photos, and only have 2.xx gb left!

With any external hard drive, can you pick and choose which files you want backed up? Also with external hard drives do they simply take the files off your computer and save them on the drive? OR just backed up?!

I have to give you a warning from my experiences over the years. I've owned and used several of those little hard drives over the years, and still use one here for other purposes. I've also given several of them away to family who needed them.

The bottom line is that they're not reliable enough on their own to be used as a backup, IMHO. They WILL fail, and I mean a LOT sooner than the more robust internal hard drives that would be placed into bays in the system or into array bays. I VERY recently had a brand new one of the new Seagate's last less than a month, then POOF - done. Amazon replaced it, no problem, even paid for return shipping on the dead one, but the data was gone - the end. Luckily, I don't trust them enough to rely on them, so I didn't actually lose anything I couldn't replace. I've learned my lesson over the years.

My regular backup storage consists of two 4-Bay enclosures each with another single-bay enclosure on top of them. In each of those 10 bays are internal SATA hard drives of 1TB to 4TB each. The left stack of 5 bays is a mirror of the right stack of 5 bays, redundant in every way.

Originally, when I put it together several years ago, they were each 1TB drives. As a hard drive died, it was replaced with a new hard drive, and its mirror in the other bank was used to make a copy. As hard drive capacities grew, I upgraded dead 1TB hard drives to larger capacity drives.

In addition, I have another 4 hard drives in the system unit, besides the operating disk itself, which has a clone up on the shelf of the OS and the base core programs I use, ready to swap out if needed, in case the operating system drive fails or gets clogged up with program crap that slows things down over time.

To answer your specific questions, yes you can pick and choose which files to back up. They just act as another drive with whatever files you put on them, whether that's another copy of the same files you have elsewhere, or the only copy you have, moved from one drive to the other. You can also use them with backup software to make backups that require the software to access the individual files for restoration, if that's what you want to do. They're just another hard drive, just like working with the contents of your current hard drive(s).

I have to agree with this. Those WD and Seagate external USB drives deteriorate quickly and are not a reliable source of backup. You'd be better off getting a hard drive cradle and using conventional disks. Or if you can afford to buy a raid 5 array, build our buy that!

 

[The_Traveler]

Please explain RAID use in terms of these articles where the emphasis is on two things:

RAID 5 is inherently dangerous because of the likelihood of data fail in large drives
RAID is not for backup​

The RAID5 delusion ZDNet
RAIDfail Don t use RAID 5 on small arrays ZDNet
RAID 5 Data Recovery National Leaders - Affordable Recovery
Dangers of RAID 5 Array with SATA Drives xByteIT

 

[Heather Koch]

Thank you Tim for the very detailed information!

I have found two very similar external hard drives on amazon.

Seagate and WD

Both have 4+ ratings and have several reviews (good reviews) and both are very similar in price.

I have a macbook pro with 500gb of storage, I currently have half of that storage full of photos, and only have 2.xx gb left!

With any external hard drive, can you pick and choose which files you want backed up? Also with external hard drives do they simply take the files off your computer and save them on the drive? OR just backed up?!

I have to give you a warning from my experiences over the years. I've owned and used several of those little hard drives over the years, and still use one here for other purposes. I've also given several of them away to family who needed them.

The bottom line is that they're not reliable enough on their own to be used as a backup, IMHO. They WILL fail, and I mean a LOT sooner than the more robust internal hard drives that would be placed into bays in the system or into array bays. I VERY recently had a brand new one of the new Seagate's last less than a month, then POOF - done. Amazon replaced it, no problem, even paid for return shipping on the dead one, but the data was gone - the end. Luckily, I don't trust them enough to rely on them, so I didn't actually lose anything I couldn't replace. I've learned my lesson over the years.

My regular backup storage consists of two 4-Bay enclosures each with another single-bay enclosure on top of them. In each of those 10 bays are internal SATA hard drives of 1TB to 4TB each. The left stack of 5 bays is a mirror of the right stack of 5 bays, redundant in every way.

Originally, when I put it together several years ago, they were each 1TB drives. As a hard drive died, it was replaced with a new hard drive, and its mirror in the other bank was used to make a copy. As hard drive capacities grew, I upgraded dead 1TB hard drives to larger capacity drives.

In addition, I have another 4 hard drives in the system unit, besides the operating disk itself, which has a clone up on the shelf of the OS and the base core programs I use, ready to swap out if needed, in case the operating system drive fails or gets clogged up with program crap that slows things down over time.

To answer your specific questions, yes you can pick and choose which files to back up. They just act as another drive with whatever files you put on them, whether that's another copy of the same files you have elsewhere, or the only copy you have, moved from one drive to the other. You can also use them with backup software to make backups that require the software to access the individual files for restoration, if that's what you want to do. They're just another hard drive, just like working with the contents of your current hard drive(s).

I have to agree with this. Those WD and Seagate external USB drives deteriorate quickly and are not a reliable source of backup. You'd be better off getting a hard drive cradle and using conventional disks. Or if you can afford to buy a raid 5 array, build our buy that!

Whats a drive cradle?

 

[qleak]

Please explain RAID use in terms of these articles where the emphasis is on two things:

RAID 5 is inherently dangerous because of the likelihood of data fail in large drives
RAID is not for backup​

The RAID5 delusion ZDNet
RAIDfail Don t use RAID 5 on small arrays ZDNet
RAID 5 Data Recovery National Leaders - Affordable Recovery
Dangers of RAID 5 Array with SATA Drives xByteIT

Good points i guess drives are big enough now that a single parity check is not enough. I'm surprised that none of the articles mention the other risk that having drives hooked up to the same power supply increases the risk of data loss. How do you propose we back up photos?

Thank you Tim for the very detailed information!

I have found two very similar external hard drives on amazon.

Seagate and WD

Both have 4+ ratings and have several reviews (good reviews) and both are very similar in price.

I have a macbook pro with 500gb of storage, I currently have half of that storage full of photos, and only have 2.xx gb left!

With any external hard drive, can you pick and choose which files you want backed up? Also with external hard drives do they simply take the files off your computer and save them on the drive? OR just backed up?!

I have to give you a warning from my experiences over the years. I've owned and used several of those little hard drives over the years, and still use one here for other purposes. I've also given several of them away to family who needed them.

The bottom line is that they're not reliable enough on their own to be used as a backup, IMHO. They WILL fail, and I mean a LOT sooner than the more robust internal hard drives that would be placed into bays in the system or into array bays. I VERY recently had a brand new one of the new Seagate's last less than a month, then POOF - done. Amazon replaced it, no problem, even paid for return shipping on the dead one, but the data was gone - the end. Luckily, I don't trust them enough to rely on them, so I didn't actually lose anything I couldn't replace. I've learned my lesson over the years.

My regular backup storage consists of two 4-Bay enclosures each with another single-bay enclosure on top of them. In each of those 10 bays are internal SATA hard drives of 1TB to 4TB each. The left stack of 5 bays is a mirror of the right stack of 5 bays, redundant in every way.

Originally, when I put it together several years ago, they were each 1TB drives. As a hard drive died, it was replaced with a new hard drive, and its mirror in the other bank was used to make a copy. As hard drive capacities grew, I upgraded dead 1TB hard drives to larger capacity drives.

In addition, I have another 4 hard drives in the system unit, besides the operating disk itself, which has a clone up on the shelf of the OS and the base core programs I use, ready to swap out if needed, in case the operating system drive fails or gets clogged up with program crap that slows things down over time.

To answer your specific questions, yes you can pick and choose which files to back up. They just act as another drive with whatever files you put on them, whether that's another copy of the same files you have elsewhere, or the only copy you have, moved from one drive to the other. You can also use them with backup software to make backups that require the software to access the individual files for restoration, if that's what you want to do. They're just another hard drive, just like working with the contents of your current hard drive(s).

I have to agree with this. Those WD and Seagate external USB drives deteriorate quickly and are not a reliable source of backup. You'd be better off getting a hard drive cradle and using conventional disks. Or if you can afford to buy a raid 5 array, build our buy that!

Whats a drive cradle?

Something like this:

Amazon.com Cable Matters SuperSpeed USB 3.0 2.0 to 2.5 3.5 SATA Hard Drive Docking Station - Supports 3TB Hard Drives Computers Accessories

 

[KenC]

I have two WD 500 GB drives. One is a desktop which sits behind my computer. It's 4 1/2 years old, but is only on for about 30 minutes every two weeks or so, sometimes longer if I decide to reorganize folders. The other is a 3 year old portable which sits in a drawer at work and comes home with me about once every two months. Both are working just fine. Very soon I will replace both, probably with their 2 TB WD equivalents. I do all the backup manually. Almost all of it is photo files, with occasional documents. I also have some old CD's/DVD's which was my backup method before the drives. Recently I pulled a file from about 2000 off one of the CD's without any problem. I have taken some key images from the disks and put them on HD's just for redundancy. No matter how fancy a backup system is, if you don't have redundancy you're taking a big chance.