Digital Media Storage Methodologies

Discussion in 'Digital Discussion & Q&A' started by Nuluvius, May 15, 2020.

  1. Nuluvius

    Nuluvius TPF Noob!

    Joined:
    May 15, 2020
    Messages:
    8
    Likes Received:
    0
    Can others edit my Photos:
    Photos OK to edit
    I have a mess of digital media to sort out, photos and videos, and I want a robust system to move forward with. I've been doing some research and want to offer up what I've found so far for discussion. I think I like the three tier system but need a final shove to decide. To be clear the top most root would be Pictures or Videos, these types are separate.

    Three Tier
    YYYYYYYY-MM-DD EventPhotos

    • Flattest, most denormalized methodology
    • Feels like year directories could get very bloated
    • Special events i.e. holidays, that span monthly boundaries would be visually contiguous
    • Complex day events could have further sub directories:
      • Those with many sources such as other people and camera types
      • Events that are long like graduations may be divided into sections
    Four Tier
    1. YYYYYYYY-MMYYYY-MM-DD EventPhotos
    2. YYYYMM NameDD EventPhotos
    3. YYYYMMDD EventPhotos
    • Breaks things down further by month under the year level
    • More nested than Three Tier methodology
    • Holiday events that span month boundaries would be broken up by month so not visually contiguous
    • Same rules possible for complex events and multiple sources as Three Tier
    Five Tier
    1. YYYYYYYY-MMYYYY-MM-DDYYYY-MM-DD EventPhotos
    2. Variants as above
    • Plus, all the naming variants as in Four Tier
    • Deep nesting
    • Could potentially have one or two directories in days
      • These could contain many or few photos
    • Same rules possible for complex events and multiple sources as Three Tier
    General Comments
    • With the Three Tier methodology the concern of ‘keys’ is not there because everything is flat under the year directory.
    • > Three tiers then we might start thinking in terms of a relational database i.e. primary and secondary keys:
      • YYYY is a primary key [PK] at level one but is a foreign key at level two and so on
      • YYYY[FK]-MM[PK]
      • YYYY[FK]-MM[FK]-DD[PK]
    • Do we even want such a bloated naming convention with more than three tiers?
    • It might be unwieldy to deal with anything beyond four tiers, we are humans and not computers after all
    • It is undesirable to rely on additional software such as Lightroom
    • The system must be as future proof as possible
    • It should be intuitive for all that may encounter it – this will be handed down and added to through the generations
    • Metadata is nice but it should be an independent concern that can be dealt with later/separately
    I would greatly appreciate your thoughts and input.


     
  2. Derrel

    Derrel Mr. Rain Cloud

    Joined:
    Jul 23, 2009
    Messages:
    47,358
    Likes Received:
    18,385
    Location:
    USA
    Can others edit my Photos:
    Photos OK to edit
    You need to look much more at metadata, and keywords and tags...YYYY/MM/DD is fine but you need to be able to FIND images...metadata and simple tags are very useful. A FEW generic tags must be attached to files...each and every birthday, birth, milestone needs to be either tagged. Keyworded, etc--- on only 1 or 2 files is enough.

    After 19 years of digital shooting, This idea has proven itself to me. Your YYYY/MM/DD idea is like talking about how to organize a library by talking about how many shelves you need, rather than using the Dewey Decimal System to card catalog each and every book in the library, so you can find the book you need out of a million volumes.
     
    Last edited: May 15, 2020
  3. Nuluvius

    Nuluvius TPF Noob!

    Joined:
    May 15, 2020
    Messages:
    8
    Likes Received:
    0
    Can others edit my Photos:
    Photos OK to edit
    I honestly understand what you are saying. What I am trying to do here is to break the problem up into smaller and more manageable pieces. The first step is getting a storage methodology in place such that I don't have to revise it at 5 months or 5 years down the line when a scenario suddenly doesn't fit it and I have hundreds of thousands of images embedded into it.

    Indeed, I fully intend to layer this in at some point in the near future.
     
  4. tirediron

    tirediron Watch the Birdy! Staff Member Supporting Member

    Joined:
    Jul 8, 2005
    Messages:
    45,547
    Likes Received:
    14,688
    Location:
    Victoria, BC
    Can others edit my Photos:
    Photos OK to edit
    My methodology is something like your three-tier system. I have a "Photos" partition on the HDD of my working computer. Each shoot is contained in its own directory labeled thusly:

    01_Jan_19_Jones_Birthday_Party

    inside that are at a minimum, of three sub-directories: "RAW", "TIF" and "JPG", as well as additional ones which are situation-dependent. In addition, I also use a full complement of key words when the images are imported in to LR. This allows me to search quickly on my computer (or back up drives, depending on how long ago the images were created), for a particular event, or with great granularity by searching LR.
     
    • Like Like x 1
  5. Derrel

    Derrel Mr. Rain Cloud

    Joined:
    Jul 23, 2009
    Messages:
    47,358
    Likes Received:
    18,385
    Location:
    USA
    Can others edit my Photos:
    Photos OK to edit
    You might also wish to consider that it is advisable to renumber files upon import. It does not take much more than one year, or even less to have the same DSC_ 1001. NEF.... after two years you will have the same file number repeated, and after 10 years you might have 15 to 20 identically numbered files. If you use an automated renaming program you can have the year the month and the date and the file number automatedly assigned to the file.... a search of my main file server will reveal several (sometimes 8 to 10)identically named files often one or two years apart. Had I known about renaming my files in a different fashion it would be easier now.

    Different camera brands use different file names. Fuji for example creates dscf for digital still capture Fuji, Canon IMG or _IMG, etc..
     
    • Like Like x 1
  6. Derrel

    Derrel Mr. Rain Cloud

    Joined:
    Jul 23, 2009
    Messages:
    47,358
    Likes Received:
    18,385
    Location:
    USA
    Can others edit my Photos:
    Photos OK to edit
    Metadata is one thing: but I have found that it is also quite valuable to actually append A keyword to the actual file name, so that you can visually see/search for a particular file, rather than be forced into searching through metadata using an application.

    If for example you actually append the words " Melanie_birthday" after The DSC_1678.NEF, YOU HAVE a search aid, both visually by human eye, and also one that can be used with a number of computer programs or even a system-level search function. Remember, we are building a system for the future. If in 15 years a family member calls you up and asks for a birthday photo from so and so's birthday all you have to do is enter the term birthday in any number of search programs and it will find multiple "birthday" files. With metadata, you must have a system that is compatible with files which are up to 15 to 30 years old, and this is not guaranteed. I would also advise you that a small notebook with a log of your dates and shooting events is worth its weight in gold as time goes by.
     
    Last edited: May 15, 2020
    • Agree Agree x 1
  7. Nuluvius

    Nuluvius TPF Noob!

    Joined:
    May 15, 2020
    Messages:
    8
    Likes Received:
    0
    Can others edit my Photos:
    Photos OK to edit
    Yes I have indeed encountered this within some parts of the older archive that I will have to deal with retrospectively. Interestingly it's not a problem for my future material, at least from my current equipment since the filename is a datetime stamp i.e. IMG_ YYYYMMDD_HHmmss.ext

    IMG_20190324_141855.jpg

    If two shots are rapid, within the same second then _# is appended i.e. IMG_ YYYYMMDD_HHmmss_#.ext

    IMG_20190324_141855_1.jpg

    Even if it is a timestamp as above? Is not locating by datetime good enough i.e YYYY -> YYYY-MM-DD Event -> Photo (timestamp) I suppose if there's a great many...

    Except I would want to digitize that too!
     
  8. Derrel

    Derrel Mr. Rain Cloud

    Joined:
    Jul 23, 2009
    Messages:
    47,358
    Likes Received:
    18,385
    Location:
    USA
    Can others edit my Photos:
    Photos OK to edit
    You must not well and truly understand the value of paper backup data. Hard drives fail. Operating systems are updated. Computer systems are switched.

    If you actually append a descriptive keyword to a file's name, you visually add a description which can be easily seen by the human eye when a directory or folder is perused. There is no substitute for doing this. You may know the year and the month and the date, the hour, the minute and down to the second.



    How does one discern a file that is an actual birthday file from hundreds of thousands of other files named within the same directory?

    I have given you some very valuable bits of information based on twenty years of shooting digital files. If you think you have a better idea or think that a simple matter is not worth the time then by all means go ahead. I wish you luck in 10 years when someone calls you up looking for a file and you tell them it will take you 3 weeks to locate it.

    I have given you some advice based upon hundreds of file searches that I have done over 20 years. Good luck with your bookshelves.
     
    • Like Like x 3
  9. Nuluvius

    Nuluvius TPF Noob!

    Joined:
    May 15, 2020
    Messages:
    8
    Likes Received:
    0
    Can others edit my Photos:
    Photos OK to edit
    I experienced such a catastrophic failure quit recently - it caused some feelings that I don't wish on anyone. I have since implemented:
    • Centralised storage on a Btrfs filesystem with monthly maintenance cycles to protect against bit rot and corruption
    • File system snapshots
    • Enterprise grade hard drives in RAID 5 with daily and weekly SMART scan cycles
    • Data replication
    • Backups to numerous cloud and local drives
    Would this be a use case for the metadata?

    Perhaps also using AI such as is available in Google Photos, or some other application to pattern match and then index into the file system - literally as an indexing mechanism - this came up in another topic.

    Your insight is beyond valuable to me, I am very grateful that you have taken the time to share it. Sorry if my approach to information gathering is... unusual, it's fundamentally the product of a lack of confidence and wanting to get as much data as possible before making a decision.
     
  10. Derrel

    Derrel Mr. Rain Cloud

    Joined:
    Jul 23, 2009
    Messages:
    47,358
    Likes Received:
    18,385
    Location:
    USA
    Can others edit my Photos:
    Photos OK to edit
    Paper backup in a ledger. Works without electricity. Can last for decades. Can be perused without a computer.

    Actually opening a directory and seeing DSC_1996.NEF_Melanie_birthday....the file will visually " jump out" from hundreds of others.


    In Mac OS, you can color- code files and search by color code...but I am pretty certain you are not a Mac OS user.

    You want to future- proof your setup, but are not looking far enough into the future, when whatever system you develop now will face a world with potentially vastly different computer systems. Metadata is fine, but it depends upon being readable by Future computer systems. As I said,actually appending a keyword to one or two files per event is invaluable. That gives you both a visual clue and allows searching with different search programs which do not depend upon metadata.

    Just this AM I found a compact disc from the 1990s. It was labeled Umax s900 system backup...it was for a Mac OS that cannot be read by today's most advanced Macintosh computers, those that are fully OSX based and have no Rosetta translation ability... and this disc is only about 25 years old.
     
    Last edited: May 17, 2020
    • Agree Agree x 1
  11. GimmeAnother1

    GimmeAnother1 No longer a newbie, moving up!

    Joined:
    Oct 15, 2019
    Messages:
    87
    Likes Received:
    111
    For future I believe metadata by itself is just a start. I see it more like hyperlinks where the file or metadata will be connected spatially like synapse of the brain versus a linear progression. When I think of something it may be a base of say a card catalogue system but from there it is just a starting point of reference. Metadata allows you to wrap xtra bits of info around something to make it more identifiable but you still need some kind of actionable intelligence behind it. Linking it together across all boundaries to me is the next way to move and scale.
     
    • Like Like x 1
  12. Nuluvius

    Nuluvius TPF Noob!

    Joined:
    May 15, 2020
    Messages:
    8
    Likes Received:
    0
    Can others edit my Photos:
    Photos OK to edit


    Not as a primary system no.

    That is the point. I'm trying to look ahead by not looking ahead i.e. by making a simple standalone system where everything else is a separate bolt on concern. If at all possible.

    That is a very good analogy.

    I tried Google Photos and Amazon Photos today and did not at all like the usability. I can't help but get the feeling that those are meant for a more simple audience and use case. Delete a photo and that seemed to be it, couldn't upload it again and moreover the sync applications try to annihilate it from whatever device they are installed on! Deadly.
     

Share This Page