Doug,
Wow, that is a lot of work. I'm still going through the images. Thanks for putting that up. I'm thinking that my PNS review (woah Nellie!), if it happens, will require me to be very systematic. I want to attain the standard you have set.
That was about one day's work, doing everything manually.
If you set up an automated workflow you could probably shave hours off the time investment.
Things that would help:
- documented processes (enables training and process improvement/optimization)
- Logical and trackable tagging/ID system for cameras under test. This may sound redundant considering that theoretically each camera has a unique mfgr & model, but when you start looking at the insane product line nomenclature of the Canon SD line, things can get really crazy really quickly. From the data standpoint, you definitely want what is called a unique ID for each camera, but I think it would help to have an additional logical ID, somewhat more informal, just to aid the testing process, e.g. mfg/mod/rez/class (DSLR, EV, compact, etc.)
- Dedicated handling bins / boxes / containers for each camera under test. Helps when checking them in, inventory control, moving from test station to test station to table-top sweep (for glamour shots), post-test return shipping, etc.
- Documented, standardized processes for testing (to aid the production of the "how we tested" editorial section)
- Documented standardized camera settings for test shots (to aid apples to apples comparison)
- Documented standardized testing criteria (resolution, frame rate, IQ, ergonomics, etc.) all within the overall framework of an overlanding / expedition perspective, meaning "the exterior surface was rated a 1 out of 5 because it is too slippery when your hands are covered in fish guts" or "the construction and durability was rated a 5 out of 5 because it survived the drop test from the rooftop tent."
- Automated workflow for download, image file naming, tagging, meta data population and cropping.
- Database for test result data entry, compilation and ranking
- Dedicated testing setups that could remain consistent across a long series of tests (months/years). This could be accomplished with a setup subject package that could be put into a case and stored away and retrieved when necessary. The lighting could be documented and recreated, although you would need to account for falloff in the output of the heads over time, etc.