Archiving one's pictures and other data - December 2021

[TWX]

Pardon if my tone is rather curt, I've just spent the better part of two days trying to copy files from the server to a USB hard drive, only for the USB hard drive filesystem to go corrupt on me around fifteen minutes ago.

I have a RAID array in an old server set up with 4TB of storage capacity, but copying my recent completely full Micro-SD card off to it was AGONIZINGLY slow like 5MB/second, I'm concerned that there's something wrong with the box. As such I spent the last couple of days performing another AGONIZINGLY slow copy of the entire RAID array to a 5TB external USB drive over the network, only to find temporarily stopping the rsync and trying to move it to another computer that the filesystem corrupted in the process. And all of the other computers (admittedly I'm running all Linux here and this is that exfat filesystem) also call it corrupt.

What do you do, here in December 2021, for archiving your photos and other important data? We're not into an era of SSDs for most consumer-grade computers, and while the lack of moving parts means we won't see dreaded mechanical faults like we saw with platter drives, SSDs and other forms of non-volitale memory for storage have their own potential issues. Getting almost 600GB of the way through a terabyte with various problems along the way has left me very tired and disillusioned.

I admit that I'm not terribly enthused about cloud-based solutions, because of the recurring costs and the possibility of them just closing up shop leaving one's data inaccessible, along with the possibiltiy of unauthorized access to that data by others, but if you have a provider that you like I'm listening.

 

[ac12]

I copy my data to either
- Removable SATA 3.5" drive. I have a RAID drive cage installed in the computer, that lets me easily do this.
- External USB drive 2.5".

I have not had any problem doing either.
The only thing is the LONG amount of time it takes to copy hundreds of GB of data files.
I usually copy in batches, rather than the entire drive in one copy. For me, this way, if there is a copy failure, it is easy to delete the bad copy and redo the copy.

You have to look for your "weak link," and probable cause.
Examples:
- Is your network 10Mb/sec or 100Mb/sec or 1000Mb/sec?
- Are ALL the network components at your network speed? A 100Mb/sec switch in a 1000Mb/sec network will choke down the data flow to 100Mb/sec.
> my laptop has a 1000Mb/sec LAN port, but if I put the computer into the cradle, the data out of the cradle's LAN port drops down to 100Mb/sec.
> Are ALL your cables rated for your network speed?
- What is the drive interface on the source and target drives?
- What is the "sustained" data transfer rate of your source and target drives? Sometimes this spec is not easy to find.
- What is the read speed of the micro-SD card?
- What is the data xfer rate of the micro-SD reader?
- Are you using a slow USB2 reader on a fast USB3 port? Or using a fast USB3 reader on a slow USB2 port?

- Are you shoving large amounts of other data over your network at the same time? This would result in data collision, slowing down the transfer.
- Are you shoving other data to/from your server at the same time? The drive in the server will be trying to service two data streams, so the theoretical data rate to each of the two streams is HALF of what a single stream would get.
- What is the size of the files? My experience is that small files copy slower than large files. I think this is because there is X amount of overhead for each file. So 100 small files have 100 overhead vs. a single large file with a single overhead. You can see this if you monitor the data read or write rates.
> Do NOT interrupt a data transfer once it starts. Unless your data transfer tool has a "pause" or "cancel" button, the interruption could mess up the file system. Think of it like you pulled the plug as it was in the middle of writing the file index, so the index is now half written and thus corrupted.

> I have had the data xfer rate from my SD card reader DROP down to about 2MB/sec. Rebooting the computer gets me back up to 40+MB/sec. I don't know why.

 

[TWX]

I'm actually a network engineer by trade, and I cabled the entire house in Cat5e foiled twisted pair, the foil-shield type. Everything's gigabit.

I'm concerned about the server, I did not expect throughput to be that bad with the copy, but I was using an older laptop, a Dell M6400, as its SD reader likes the size while the reader in my XPS-13 for some reason doesn't.

I have had wireless performance issues with the XPS-13, which was why I was using the M6400 on cabled connection. But the M6400 gave me its own problems, the 150W AC adapter died on me today, and I wonder if the sustained extra power draw for the file copy to the USB drive was more than it could handle.

 

[ac12]

To ME, a laptop is a light use computer. Especially the older ones.
My experience, in the past, was that there were a bunch of compromises to make it as small as it is.
And they did not seem up to HEAVY use.

One of the biggest issue was cooling. Using a laptop on your lap was not comfortable, some of them got HOT. And I've run into laptops where the fan was constantly running, and the exhaust air was almost hot. There was constantly more stuff being added, that generated heat: bigger/faster processor, graphics processor, more memory, more components (WiFi, BlueTooth, USB, etc. ), gigabit LAN, etc. And all being jammed into smaller and smaller cases.

And the drives back then could not keep up with heavy demand. Those 2.5" drives were pretty slow. Today that is not the case with SSDs. But if you are using an old laptop, is that a SATA-1 port the SSD is connected to?

Today with everything connecting via USB, there sometimes isn't enough USB ports. You have to use an external hub.

As for the power supply. Possible. The laptop, including the power supply, may not have been designed for heavy use for 24+ HOURS. So you could have exceeded the current/thermal design.

I would rather use an old desktop/mini-tower for a server, primarily for the air flow.

 

[Rickbb]

File copy is the slowest way possible. For an archive I use a bootable imaging package, usually it will do a couple of TB to a USB HDD in an hour or so. Now I use a 4TB USB SSD, it images a 2 TB data drive in a half the time.

I always make 2 seperate images on seperate SSD‘s.

i usually only do a full image set once a year or whenever the 1TB cloud drive is full and I need to reduce the size. I use MS One Drive, comes with my day job so no cost to me.

If you are worried about someone accessing the files you can set up an automatic encryption/de-cryption process using a third party pacakge.