Digital Media Storage Methodologies

[Nuluvius]

I have a mess of digital media to sort out, photos and videos, and I want a robust system to move forward with. I've been doing some research and want to offer up what I've found so far for discussion. I think I like the three tier system but need a final shove to decide. To be clear the top most root would be Pictures or Videos, these types are separate.

Three Tier
[TABLE="class: brtb_item_table"][TBODY][TR][TD]YYYY[/TD][TD]YYYY-MM-DD Event[/TD][TD]Photos[/TD][/TR][/TBODY][/TABLE]

• Flattest, most denormalized methodology
• Feels like year directories could get very bloated
• Special events i.e. holidays, that span monthly boundaries would be visually contiguous
• Complex day events could have further sub directories:
  • Those with many sources such as other people and camera types
  • Events that are long like graduations may be divided into sections

Four Tier

1. [TABLE="class: brtb_item_table"][TBODY][TR][TD]YYYY[/TD]
   [TD]YYYY-MM[/TD]
   [TD]YYYY-MM-DD Event[/TD]
   [TD]Photos[/TD][/TR][/TBODY][/TABLE]
2. [TABLE="class: brtb_item_table"][TBODY][TR][TD]YYYY[/TD]
   [TD]MM Name[/TD]
   [TD]DD Event[/TD]
   [TD]Photos[/TD][/TR][/TBODY][/TABLE]
   
3. [TABLE="class: brtb_item_table"][TBODY][TR][TD]YYYY[/TD]
   [TD]MM[/TD]
   [TD]DD Event[/TD]
   [TD]Photos[/TD][/TR][/TBODY][/TABLE]

• Breaks things down further by month under the year level
• More nested than Three Tier methodology
• Holiday events that span month boundaries would be broken up by month so not visually contiguous
• Same rules possible for complex events and multiple sources as Three Tier

Five Tier

1. [TABLE="class: brtb_item_table"][TBODY][TR][TD]YYYY[/TD]
   [TD]YYYY-MM[/TD]
   [TD]YYYY-MM-DD[/TD]
   [TD]YYYY-MM-DD Event[/TD]
   [TD]Photos[/TD][/TR][/TBODY][/TABLE]
2. Variants as above

• Plus, all the naming variants as in Four Tier
• Deep nesting
• Could potentially have one or two directories in days
  • These could contain many or few photos
• Same rules possible for complex events and multiple sources as Three Tier

General Comments

• With the Three Tier methodology the concern of ‘keys’ is not there because everything is flat under the year directory.
• > Three tiers then we might start thinking in terms of a relational database i.e. primary and secondary keys:
  • YYYY is a primary key [PK] at level one but is a foreign key at level two and so on
  • YYYY[FK]-MM[PK]
  • YYYY[FK]-MM[FK]-DD[PK]
• Do we even want such a bloated naming convention with more than three tiers?
• It might be unwieldy to deal with anything beyond four tiers, we are humans and not computers after all
• It is undesirable to rely on additional software such as Lightroom
• The system must be as future proof as possible
• It should be intuitive for all that may encounter it – this will be handed down and added to through the generations
• Metadata is nice but it should be an independent concern that can be dealt with later/separately

I would greatly appreciate your thoughts and input.

 

[Derrel]

You need to look much more at metadata, and keywords and tags...YYYY/MM/DD is fine but you need to be able to FIND images...metadata and simple tags are very useful. A FEW generic tags must be attached to files...each and every birthday, birth, milestone needs to be either tagged. Keyworded, etc--- on only 1 or 2 files is enough.

After 19 years of digital shooting, This idea has proven itself to me. Your YYYY/MM/DD idea is like talking about how to organize a library by talking about how many shelves you need, rather than using the Dewey Decimal System to card catalog each and every book in the library, so you can find the book you need out of a million volumes.

 

[Nuluvius]

Your YYYY/MM/DD idea is like talking about how to organize a library by talking about how many shelves you need, rather than using the Dewey Decimal System to card catalog each and every book in the library, so you can find the book you need out of a million volumes

I honestly understand what you are saying. What I am trying to do here is to break the problem up into smaller and more manageable pieces. The first step is getting a storage methodology in place such that I don't have to revise it at 5 months or 5 years down the line when a scenario suddenly doesn't fit it and I have hundreds of thousands of images embedded into it.

You need to look much more at metadata

Indeed, I fully intend to layer this in at some point in the near future.

 

[tirediron]

My methodology is something like your three-tier system. I have a "Photos" partition on the HDD of my working computer. Each shoot is contained in its own directory labeled thusly:

01_Jan_19_Jones_Birthday_Party

inside that are at a minimum, of three sub-directories: "RAW", "TIF" and "JPG", as well as additional ones which are situation-dependent. In addition, I also use a full complement of key words when the images are imported in to LR. This allows me to search quickly on my computer (or back up drives, depending on how long ago the images were created), for a particular event, or with great granularity by searching LR.

 

[Derrel]

You might also wish to consider that it is advisable to renumber files upon import. It does not take much more than one year, or even less to have the same DSC_ 1001. NEF.... after two years you will have the same file number repeated, and after 10 years you might have 15 to 20 identically numbered files. If you use an automated renaming program you can have the year the month and the date and the file number automatedly assigned to the file.... a search of my main file server will reveal several (sometimes 8 to 10)identically named files often one or two years apart. Had I known about renaming my files in a different fashion it would be easier now.

Different camera brands use different file names. Fuji for example creates dscf for digital still capture Fuji, Canon IMG or _IMG, etc..

 

[Derrel]

Metadata is one thing: but I have found that it is also quite valuable to actually append A keyword to the actual file name, so that you can visually see/search for a particular file, rather than be forced into searching through metadata using an application.

If for example you actually append the words " Melanie_birthday" after The DSC_1678.NEF, YOU HAVE a search aid, both visually by human eye, and also one that can be used with a number of computer programs or even a system-level search function. Remember, we are building a system for the future. If in 15 years a family member calls you up and asks for a birthday photo from so and so's birthday all you have to do is enter the term birthday in any number of search programs and it will find multiple "birthday" files. With metadata, you must have a system that is compatible with files which are up to 15 to 30 years old, and this is not guaranteed. I would also advise you that a small notebook with a log of your dates and shooting events is worth its weight in gold as time goes by.

 

[Nuluvius]

You might also wish to consider that it is advisable to renumber files upon import. It does not take much more than one year, or even less to have the same DSC_ 1001. NEF.... after two years you will have the same file number repeated, and after 10 years you might have 15 to 20 identically numbered files.

Yes I have indeed encountered this within some parts of the older archive that I will have to deal with retrospectively. Interestingly it's not a problem for my future material, at least from my current equipment since the filename is a datetime stamp i.e. IMG_ YYYYMMDD_HHmmss.ext

IMG_20190324_141855.jpg

If two shots are rapid, within the same second then _# is appended i.e. IMG_ YYYYMMDD_HHmmss_#.ext

IMG_20190324_141855_1.jpg

Metadata is one thing: but I have found that it is also quite valuable to actually append A keyword to the actual file name, so that you can visually see/search for a particular file, rather than be forced into searching through metadata using an application.

Even if it is a timestamp as above? Is not locating by datetime good enough i.e YYYY -> YYYY-MM-DD Event -> Photo (timestamp) I suppose if there's a great many...

I would also advise you that a small notebook with a log of your dates and shooting events is worth its weight in gold as time goes by.

Except I would want to digitize that too!

 

[Derrel]

You must not well and truly understand the value of paper backup data. Hard drives fail. Operating systems are updated. Computer systems are switched.

If you actually append a descriptive keyword to a file's name, you visually add a description which can be easily seen by the human eye when a directory or folder is perused. There is no substitute for doing this. You may know the year and the month and the date, the hour, the minute and down to the second.



How does one discern a file that is an actual birthday file from hundreds of thousands of other files named within the same directory?

I have given you some very valuable bits of information based on twenty years of shooting digital files. If you think you have a better idea or think that a simple matter is not worth the time then by all means go ahead. I wish you luck in 10 years when someone calls you up looking for a file and you tell them it will take you 3 weeks to locate it.

I have given you some advice based upon hundreds of file searches that I have done over 20 years. Good luck with your bookshelves.

 

[Nuluvius]

You must not well and truly understand the value of paper backup data. Hard drives fail. Operating systems are updated. Computer systems are switched.

I experienced such a catastrophic failure quit recently - it caused some feelings that I don't wish on anyone. I have since implemented:

• Centralised storage on a Btrfs filesystem with monthly maintenance cycles to protect against bit rot and corruption
• File system snapshots
• Enterprise grade hard drives in RAID 5 with daily and weekly SMART scan cycles
• Data replication
• Backups to numerous cloud and local drives

How does one discern a file that is an actual birthday file from hundreds of thousands of other files named within the same directory?

Would this be a use case for the metadata?

Perhaps also using AI such as is available in Google Photos, or some other application to pattern match and then index into the file system - literally as an indexing mechanism - this came up in another topic.

I have given you some very valuable bits of information based on twenty years of shooting digital files. If you think you have a better idea or think that a simple matter is not worth the time then by all means go ahead. I wish you luck in 10 years when someone calls you up looking for a file and you tell them it will take you 3 weeks to locate it.

I have given you some advice based upon hundreds of file searches that I have done over 20 years. Good luck with your bookshelves.

Your insight is beyond valuable to me, I am very grateful that you have taken the time to share it. Sorry if my approach to information gathering is... unusual, it's fundamentally the product of a lack of confidence and wanting to get as much data as possible before making a decision.

 

[Derrel]

Paper backup in a ledger. Works without electricity. Can last for decades. Can be perused without a computer.

Actually opening a directory and seeing DSC_1996.NEF_Melanie_birthday....the file will visually " jump out" from hundreds of others.


In Mac OS, you can color- code files and search by color code...but I am pretty certain you are not a Mac OS user.

You want to future- proof your setup, but are not looking far enough into the future, when whatever system you develop now will face a world with potentially vastly different computer systems. Metadata is fine, but it depends upon being readable by Future computer systems. As I said,actually appending a keyword to one or two files per event is invaluable. That gives you both a visual clue and allows searching with different search programs which do not depend upon metadata.

Just this AM I found a compact disc from the 1990s. It was labeled Umax s900 system backup...it was for a Mac OS that cannot be read by today's most advanced Macintosh computers, those that are fully OSX based and have no Rosetta translation ability... and this disc is only about 25 years old.

 

[GimmeAnother1]

For future I believe metadata by itself is just a start. I see it more like hyperlinks where the file or metadata will be connected spatially like synapse of the brain versus a linear progression. When I think of something it may be a base of say a card catalogue system but from there it is just a starting point of reference. Metadata allows you to wrap xtra bits of info around something to make it more identifiable but you still need some kind of actionable intelligence behind it. Linking it together across all boundaries to me is the next way to move and scale.

 

[Nuluvius]

In Mac OS, you can color- code files and search by color code...but I am pretty certain you are not a Mac OS user.

Not as a primary system no.

You want to future- proof your setup, but are not looking far enough into the future...

That is the point. I'm trying to look ahead by not looking ahead i.e. by making a simple standalone system where everything else is a separate bolt on concern. If at all possible.

...metadata will be connected spatially like synapse of the brain versus a linear progression.

That is a very good analogy.

I tried Google Photos and Amazon Photos today and did not at all like the usability. I can't help but get the feeling that those are meant for a more simple audience and use case. Delete a photo and that seemed to be it, couldn't upload it again and moreover the sync applications try to annihilate it from whatever device they are installed on! Deadly.

 

[Derrel]

Sync applications can be risky...some will delete files...another case for unique file names being very valuable.

 

[Nuluvius]

Sync applications can be risky...some will delete files...another case for unique file names being very valuable.

Very scary and a shame really because of the advanced features and possibility to use it as an indexing mechanism.

 

[Derrel]

Metadata is one thing. A color-coding and an actual file-NAME amendment make a file really stand out!

Metadata is hidden.. a file name amendment is humanly visible....if you have a paper shooting log, you can look through it and find the file easily, and even if you have no log, the file has a name amendment which tells you what the file "is".

Imagine a library with 800,000 books, each titled " book" on the spine. You have a computer to keep track of them, each book in its place. Imagine if you added an amendment to the generic titles of "book" to a few important books.Sound like a benefit?