Solving the Digital Photographer's Archiving Dilemma? (Digital Asset Management)

[nwoods]

As an amateur photographer (or perhaps more accurately, "avid trip chronicler"), I have a ton of photos on my various computers, external drives, and cloud services. Every few years I poke my head up above water and look around, evaluating the current state of the industry with regards to prosumer level digital asset management. The buzz about Apple's new Photos app has prompted me to once again take an inventory in what's available for folks like me, struggling to deal with a hodge podge of digital images stored all over the place. What I really, REALLY want is to solve the problem of connecting my various photo capturing devices into an integrated library, stored on a hard drive I own, and accessible in flexible ways via the cloud.

Apple's new app called "Photos" showed some promise initially, but a closer looks reveals that its more regressive than progressive in terms of asset management. Apple doesn't seem interested in photos taken on my dSLR, or my scanned film shots from years earlier. It relies 100% on the cloud, and significantly, there is no easy way to select which images appear on which devices. It is very much a All or Nothing sort of solution, which presumes that I would be content to allow Apple to manage all my content, and that I'd be willing to pay them for the privilege. No.

So what else is out there? Adobe Lightroom is pretty good of course, but still doesn't seamlessly bridge the gap between photos taken on my iPhone verses photos taken on my dSLR. It's a hassle to export everything from iPhoto into conventional folders for Lightroom to catelogue.

Adobe Bridge is still powerful in its simplicity and speed, but suffers the same lack of connectivity to iDevices.

My search for new solutions has lead me to Mylio. Have you heard of this? Here is the site: http://mylio.com/
It sounds pretty intriguing. There is a good PDF with more details on it: http://www.outdoorphotographer.com/portals/mylio/mylio.pdf
Page 11 of this PDF (magazine page 22) lists some pretty darn impressive functions, particularly with regards to choosing what shows up where, how it gets there, and where its stored.

Has anyone out there in ExPo-land played with this software yet? If not, why not? What else are you using? How are you solving the problem of connecting your various photo capturing devices into an integrated library?

 

[workerdrone]

Lightroom is pretty robust for organization, not sure why you'd put iphone pix into iphoto before moving them to lightroom? I just put everything into LR first then use folders and search functions / metadata to find stuff by date, lens, camera, ratings, etc..

Then for photos that need it or I want to take extra care in processing, I'll export from LR to photoshop or another raw converter temporarily, then right back into LR.

I back up a couple of times a month and rotate external hard drives between my office, home safe, and bank safe deposit box. Tried online backups briefly and just didn't like it, ymmv

I also feel pretty confident that LR will continue to be supported and developed for some time, so I won't have to learn new software all over again.

 

[Chazz Layne]

I'm happy with Lightroom as well. I can't see ever taking enough photos on my phone, or caring enough about the photos on my phone, to ever need more than a once-a-month manual dump into the backups from it. To put it in perspective, there are maybe 5 photos a year taken with my phone that are worth keeping past the time it takes to post to social media, and I always have at least an S100 on me for photos I'd want to keep or use professionally.

Everything else (all 3 cameras) stays on the camera card + a backup to my field notebook while on a trip, then dumped into Lightroom on my workstation when I get home. Lightroom plays nice with plain old files so monthly backups are easy enough.


Also, in fairness the problem with your scenario is Apple/iPhoto. Photos should already be stored flat on the hard drive as conventional files/folders, Apple's approach is why everything iPhoto/Aperture is completely screwed up. It's why I use Lightroom for my business, and why Overland is also switching. Nothing will ever beat plain old conventional files.

 

[sickchilly]

My RAW files from my Fuji X-Pro1 and my scanned film from various 35mm and medium format cameras all goes into Lightroom. I'm religious about meta data making sure dates, GPS, and keywords are all accurate. I have many smart albums based on the meta data and state of editing (new, in progress, edited, shared).

I have all of my Lightroom images in the cloud (proxy JPG versions) by using Mosaic Archive. It automatically syncs my entire library to my Google Drive. I then use their iOS app to view my Lightroom library on the go.

For photos that I publish various places or otherwise "share", I used to export albums to iPhoto and use it to upload to various services and social sites, email, use in other projects, etc. Now, since signing up for the OS X beta, I'm using Photos instead of iPhoto. I also use photos to sync anything taken on my iPhone via iCloud.

It's not ideal, but the work flow has been working for me over the last year or two. I especially like having access to anything from my iPhone for a quick share or view. And I like having my Lightroom library finally organized. I can find anything back to 1998, when I had a 1.3 MP point and shoot!

 

[DiploStrat]

… Photos should already be stored flat on the hard drive as conventional files/folders, Apple's approach is why everything iPhoto/Aperture is completely screwed up. It's why I use Lightroom for my business, and why Overland is also switching. Nothing will ever beat plain old conventional files.

To be a bit pedantic, neither Lightroom nor Aperture treat photos as flat files; with these applications a photo is a logical combination of several files which only work together within the application. (Which is a data base.)

You have:

-- The Original file as it came out of the camera. No matter what the format, RAW, JPEG, etc., these applications never edit or rewrite these files. (Of course, a RAW file cannot be edited in any case.)
-- The Edits or Adjustments file. (Called a Version in Aperture.) Initially empty, this file contains any and all of the changes you may make to an image - white balance, sharpening, etc.
-- Miscellaneous thumbs and previews.

Whenever you look at an image you are seeing the original (or sometimes the preview) through the edits file, combined on the fly by your computer.

This non-destructive workflow has advantages - you can always undo, JPEG originals are never recompressed, etc., but it also means that you cannot see your edits outside of the original application. This is different from working with Photoshop, where you actively rewrite the files. (Of course, to use a RAW file with Photoshop or any other destructive editor, you must first read that file with a RAW converter and write a new file in an editable format like TIFF, PSD, or JPEG. It was to automate this process, and save some disk space, that Aperture and Lightroom were created.)

So the REALLY big question is whether you want to trust your images to a data base application like Lightroom (since Aperture is no longer in the running) or go back to a pure, flat file world where you have to track the original and the copies. One of the great advantages of data base applications is that you are not limited to keeping your images in one, fixed configuration, but you have the option of viewing them many different ways using the various data base view tools like "Smart Collections" and key words.

FWIW: I have used Aperture since 2006 and never, ever lost an image. I have about 20,000 images, ranging from TIFF (scanned slides) to JPEG to RAW. The loss of Aperture is going to be a real PITA for me.
I will probably end up with Lightroom (although I find it annoyingly clunky) because there is no way I am going to go back to managing a pile of files on my own.

I use multiple backups:

-- Time Machine every hour
-- SuperDuper! every night
-- CrashPlan for remote.

 

[MojaveKJ]

Film is the best archive format for digital pictures.

 

[workerdrone]

^ come again?

 

[Chazz Layne]

To be a bit pedantic, neither Lightroom nor Aperture treat photos as flat files; with these applications a photo is a logical combination of several files which only work together within the application. (Which is a data base.)

Nope, Lightroom stores everything as plain old flat files. There are sidecar XML files as well, which are open source and readable even by Windows, where Lightroom stores changes that are made, but even they are just regular files sitting in folders. That's why it's so much easier to share out a Lightroom library, toss part of it on a USB stick to give someone (without even opening LR or exporting), or manually back up bits and pieces at will.

By contrast, Aperture stores everything inside a single gigantic multi-gigabyte database. This is why Aperture takes forever to open, is frustratingly slow, and why you have to export before you can share (network) or copy to a USB stick. You can back up the massive database file, but not in parts. Unless you nerd-out on it there's no easy way into that database except through Aperture. Heck, even the folks that make vBulletin (the forum you're on) finally figured out it's a very bad idea to store images inside a database. :********:

 

[DiploStrat]

Dating Ourselves

Chazz,

Thanks for the update on XMP files; I have never worked with them. I agree, the ability to simply send someone your original and an edits file, in a reasonably open format, it slick. Aperture cannot do that.

But trust me me, Lightroom is a database system, hence the importance of the catalog files. (A DBMS does not require that the files managed be in a proprietary format.) Importing to Lightroom is the indexing process. (And you have the degree to understand all of this even better than I - although I did spend years in systems development.)

While it is moot at this point, your comments about Aperture file management are a bit outdated. While Aperture does offer the default option of placing your masters/originals inside a folder, you can place them anywhere you want. (While not officially supported, you can also spread them across a network. At one time, I even had some on a CD-ROM.) I have some on my system drive but most on a slower HD. Backups are on a file by file basis, whether with Time Machine or SuperDuper! and very fast unless you have just imported a lot of new images. (This was not the case with Aperture 1 and that may be the cause of your misconception.)

N.B. A performance tip worth noting for those with really big libraries. Your master/original file is strictly write-once/read-many. This means that a HD dedicated to your image files will never fragment. Your catalog file and your version/XMP files, along with your thumbs and previews, on the other hand, are read and rewritten constantly. This argues for placing these on the fastest drive you can. (I use an SSD.) The other dirty secret is that your originals are actually read very little, typically only for full res (zoomed in) editing, printing, and exporting. Most of the rest of the time, these programs simply work from a preview. I don't actually use a dedicated image HD, but a quick defrag after import assures that when the images are read they read fast. (And I am running on an ancient 2006 MacPro.)

Sadly, I will have to look at alternatives to Aperture in the coming year. I doubt that Photos will be the answer unless Apple delivers on their promise to allow plugins to work in the RAW pipe. If I have read correctly, release 1.0 of Photos does not even support an external editor option. (I used to find the combination of iPhoto and PS Elements to be quite powerful when I shot a lot of JPEG. I went to Aperture for the better image management and the ability to spread across multiple HD.) There is no doubt, however, that Lightroom has grown to be the industry standard and is a very safe bet. I just wish they could get beyond the modular layout.

EDITED TO ADD: My real point is that with RAW and with any non-destructive workflow, you no longer have a simple one-to-one relationship between images and files. This means that you may have to decide whom you trust to link them up again.

 

[DaveInDenver]

Film is the best archive format for digital pictures.

Beat me to it. Film images, the first flat(ish) file! They seem to always adapt to the newest imaging technology because you have a platform independent original. In the case of silver-halide black and white it will also last for centuries as long as the fixer used was good.

 

[Chazz Layne]

But trust me me, Lightroom is a database system, hence the importance of the catalog files. (A DBMS does not require that the files managed be in a proprietary format.) Importing to Lightroom is the indexing process. (And you have the degree to understand all of this even better than I - although I did spend years in systems development.)

I never said it wasn't. I expect any program of this nature to require a database under the hood, it's the best way to manage the data. The key difference is that the LR database is merely an index of files and changes. The big imagery data is left as standard files in regular folders, which are both far easier for the user to access and much faster for the program to load. Most of the basic user-made changes to the images are saved the same way.

What this means is if Adobe pulled the plug on Lightroom, the only thing lost would be changes at the pixel level (such as brushes or red-eye, which shouldn't be done in the asset management app anyhow). If you woke up one morning and found Lightroom was gone, all of the tagging, filing, organizational, and even ACR edits would be completely intact waiting for the next new management program to take them over. You can demonstrate this by making changes to an image in Lightroom, tossing that image (RAW) and it's .xmp onto another computer that doesn't have LR, and opening it up in a photo editor to find all the exposure, color, noise reduction, etc. changes intact (along with rating, location, and keyword tags). I run into this all the time from photos emailed by contributors, sometimes to an extremely frustrating extent...

When it comes to future-proofing against one of the big guys suddenly discontinuing a product, it's hard to imagine a better solution than that.

(I do know that Aperture can store photos in regular folders, but it's a very obscure feature few will ever find, thus my nerding-out comment.)

I doubt that Photos will be the answer unless Apple delivers on their promise to allow plugins to work in the RAW pipe. If I have read correctly, release 1.0 of Photos does not even support an external editor option. (I used to find the combination of iPhoto and PS Elements to be quite powerful when I shot a lot of JPEG.

Apple seems to be headed in the right direction with Photos at least, though it does seem a bit beta-esque (high praise coming from an Apple hater). Their goal was to 1-up Adobe with a program that was even more open than Lightroom. If they take the time to actually complete, test, and fully debug it before they release it, and make it an open system so third-parties can read it (like Photoshop), Photos has the potential to be a real winner. Unfortunately, it took them 4 painful years to fix the file-eating Lions (with Yosemite) and they still haven't gotten it quite right, so I doubt we'll be looking at a production-ready solution for many years.

 

[DiploStrat]

... The big imagery data is left as standard files in regular folders, which are both far easier for the user to access and much faster for the program to load.

Aperture does the same. Those files can be anywhere you want. The default is to put them in a folder on the system drive. (Apple uses a folder called a "Package" which does not immediately show its contents in the finder, but they are all there.) Initially, with Aperture 1, each image consisted of a folder that contained the original/thumbs/previews/edits in one folder. With Aperture 2, they realized that that was not the way people wanted to find their files, and put all of the originals in one folder, all of the previews in another, etc. (This reformatting caused no end of agony to those who didn't have enough disk space for the transition, etc.) This is not exactly an obscure feature as there are any number of tools to support this. Ironically, one of the slickest uses of Aperture is to import a plethora of old images into the library, arrange/rename/etc. as desired, and then use the "Relocate" command to arrange them out on disk(s) as you desire.

Most of the basic user-made changes to the images are saved the same way.

Had to research this after your post. The XMP option is kind of neat. (Or is it the default setting now?) As far as I know, Aperture never did this, probably because XMP is an Adobe licensed product.

Apple seems to be headed in the right direction with Photos at least, though it does seem a bit beta-esque (high praise coming from an Apple hater). Their goal was to 1-up Adobe with a program that was even more open than Lightroom. If they take the time to actually complete, test, and fully debug it before they release it, and make it an open system so third-parties can read it (like Photoshop), Photos has the potential to be a real winner. Unfortunately, it took them 4 painful years to fix the file-eating Lions (with Yosemite) and they still haven't gotten it quite right, so I doubt we'll be looking at a production-ready solution for many years.

I am not quite as optimistic, yet. (And I am an Apple lover/Windoze hater, since 1984.) Apple is putting all of its photo manipulation, including RAW development into the OS and making it video capable. The big Apple conference of a year or so ago had some amazing demos of keystone correction, etc., being done in real time. At the same time, they promised that plugins would work directly in the RAW pipe, without the need for an intermediate TIFF, etc. If so, then Photos would need to do only the data base work, basic exposure/color, and all of the pixel level/layers type stuff would be done with the various plugins that so many of us have anyway. As you may remember, back in the day, Aperture was specifically announced not as a "Photoshop Killer" but rather as a form of Photo Mechanic on steroids. It was always assumed that you would finish a certain number of images in Photoshop. But I must agree with the "beta" comment. As I understand it, Photos 1.0 doesn't have an external editor option of any sort.

With time, most of us moved away from Photoshop and both Aperture and especially Lightroom became capable of doing basically everything except layers, text, and graphics artist, as opposed to photographer, stuff. I could certainly live with that again, but I will have to play with Photos to see if the cataloging features measure up. (I am one of those geeks who doesn't use a date based filing system.) I suspect that, in the end, I will end up with Lightroom and a ton of Smart Collections.

But all of this is in the future; now I have to get my presentations in order for OEXPO West.

 

[Scott Brady]

I am still gutted that Aperture is done for. Nearly 600,000 images in 12 libraries. The migration is going to be biblical. . .

Lightroom is ok, but neither elegant or intuitive.

Honestly, I am just avoiding the inevitable in the hope that something worthwhile presents itself

 

[DiploStrat]

600,000!

600,000! :Wow1:

Actually, it may not be that bad. If your masters are not referenced, use the relocate tool to arrange them on your disks as you wish. Then simply let Lightroom index them. (You CAN use XMP side cars to export camera data if you want.)

Scholars differ about saving the edits, but my view would be to simply redo as needed in Lightroom. After all, Lightroom is said to have better tools, lens correction, etc. In the interim, you can still use Aperture as needed.

But, you are right, Lightroom certainly lacks the elegance of Aperture when it comes to actually using the program.

(However you do it, it will still be forty days and forty nights - at least.)

 

[Ryanmb21]

We have 100,000 photos in 2 aperture libraries, it works perfect and every image is stored in plain folders. Set it up about 5 years ago and works great.

I use time machine to make two backups of the iMac and the current year images drive(s). Prior year images drives are backed up with Carbon Copy Cloner.

We will transition to Lightroom, as Apple **** the bed and quit aperture. The cool thing, the library transition will not have to move a single referenced file. My fingers are crossed that it works.

Interested to hear what else is out there as I'm just having to get up to speed with aperture's demise.