Adobe: why Lightroom image export isn't faster

Updated 3:04 p.m. PDT with further Adobe remarks. I misunderstood the company's position: Lightroom's export behavior reflects engineering priorities.

Earlier this month, I encountered an Adobe Photoshop Lightroom analysis by consultant Lloyd Chambers that expressed surprise with a facet of the image editing and cataloging software: it didn't export photos as fast as possible.

Chambers found that if a photographer wants to produce JPEG or TIF images from the originals in the program, the fastest way is to divide the batch into thirds and export each third separately. Using a modern Mac Pro system, exporting a test set of photos took 351 seconds as one batch and 189 seconds divided into three batches running at the same time.

"The big disappointment is the sluggish performance importing and exporting files, which are tasks that are key to efficient workflow--tasks one has to do over and over. Most of the 'juice' of a Mac Pro goes untapped," Chambers concluded. "You have to load it up with more than one job to force more of the available CPU cores to be used. Lightroom should do this automatically!"

The study caught the attention of others, including Scott Kelby, head of the National Association of Photoshop Professionals. I was intrigued, too, because although many programming chores are difficult to spread across multiple processor cores, exporting photos is trivially easy since it breaks conveniently into independent bite-sized pieces. So I thought I'd see what Adobe had to say for itself.

According to Lightroom Product Manager Tom Hogarty, it's a consequence of Adobe's design priorities:

Lightroom receives numerous compliments because it does indeed stay responsive when background tasks are running. Now as Lloyd's article points out there is headroom in our export processing, but in our prioritization of performance enhancements there is a focus on interactive performance improvements like scrolling through your images during a review process or developing slider responsiveness. That's not to say that the "set it and forget it" tasks like batch exporting aren't important. We just need to balance our efforts at performing well on a wide range of hardware and in a number of workflow scenarios without focusing too much on just utilizing every available resource of a specific configuration.

Initially I misunderstood that to mean Lightroom doesn't max out the machine so that the software is still responsive, but he wrote back with this clarification:

What we haven't done is optimized the multiprocessor performance of that code because there are other, higher priority areas of performance that have needed attention during previous product development cycles. Utilizing more system resources without impacting the responsiveness of the software or other activities on a photographer's computer is not a simple task, and as you mentioned in one of your comments, you couldn't remember being annoyed by the performance of the export process. Similar feedback from our customers is what informs our prioritization process. For some customers this lack of export processing optimization is a "sin of omission" but what other area of performance or functionality would they trade for that improvement?

I for one often use Lightroom to edit, tag, and otherwise handle photos while it's importing or exporting, so I appreciate a balance between interactive tasks and background tasks. But often I go away and come back when a job is finished, because interactive performance does take a hit.

Perhaps in the next update, Adobe could add a "maximum performance" option to the export or import dialog boxes--or even better, detect automatically those times when I left the computer on its own to finish up.

[basraw]

I can see Adobe's reasoning.. another thing would be to put a slider of some sort .. 0/25/50/75/100%.. for enabling/disabling background responsiveness.

[ricklabanca]

Marketspeak! They haven't heard of prioritized threads?

[T_Hoff]

That's a lame excuse on Adobe's part. The days of single-core processors are history, any application that doesn't efficiently utilize multiple cores is not tapping the full potential of the system. Even the Rawshooter Essentials and Rawshooter Premium programs from Pixmantec which they acquired and promptly killed off did a far better and faster job on multi-processor systems, and those programs were last updated more than three years ago.

Adobe should assign the photos in the queue to separate threads, one for each logical processor in the system. If they want the application -- and indeed the rest of the system -- to remain responsive during the export/import process, run those threads at below normal or idle priority.

[fazalmajid]

Lightroom is a decent product but dynamic languages that support C extensions like Lua, Python or Ruby (Lightroom is written in Lua) often have limitations in how much concurrency they can support, and usually one process cannot exploit more than one core. The solution is to fork a new process for each export task, which complicates the programmer's job because he must now coordinate and monitor sub-processes from the main process that handles the GUI. I'm sure Adobe will get around to it eventually, unless they plan to dither as they are doing with 64-bit support in Photoshop.

That said, for image processing with large files you might just move the bottleneck. I wrote some shell scripts using GNU xargs to run multiple TIFF to JPEG conversions in another part of my workflow (for slide scans from my Nikon Coolscan 5000 and 9000):

http://www.majid.info/mylos/weblog/2008/09/27-1.html

Each scan is about 100MB in size. On my old PowerMac G5 dual 2GHz, going from single thread to dual threads doubled the performance. There was little point going beyond. When I upgraded to an eight-core Mac Pro, I modified the script to run 8 jobs in parallel, but I found it was not going any faster than with 2 jobs. The reason was that disk I/O became the bottleneck and a 7200 rpm Samsun SpinPoint F1 could not handle more than two concurrent ImageMagick TIFF to JPEG conversions.

Conclusion: if you have a high-performance machine like the Mac Pro and need to extract the maximum power out of it, you need to get a fast SSD like the Intel X25-E or the Samsung PB22-J to use as your primary storage.

[kyle5434]

"Really user, it's not a shortcoming of the application, it's for your own good."

Priceless!

[terminalblue]

Seems silly to me but not detrimental in logic. I never even thought it was taking along time. i mean i know it could be faster but i never really thought about it. When i am moving RAW's off the camera that are 16-20MB in size i just assumed it was the speed of the USB bus and the size of the files. but yeah, i generally start editing right away, it seems to relax the process and makes it FEEL like i get more done (or maybe just spend more time on the first pictures i take instead of skipping to the one i remember most vividly)

I dont know how i feel about this really. I guess it doesnt really affect anything i do. lightroom is reliable and i shouldnt be touching up pictures in a hurry anyway. So i guess that it doesnt really matter for everybody except people that just want to brag about there file transfer speed or people that really need to do touch ups on the fly.

[Shankland]

It might be difficult or easy to change thread priority to make a background task execute fast or get out of the way, depending on whether I'm doing something else--I'd hope that Adobe of all companies would know how. But I can't remember being annoyed at how long it was taking to export JPEGs, which indicates to me that for me at least it makes little difference.

Importing photos is a different scenario for me: I am sometimes concerned about performance. The very cases in which I want the 1:1 previews to render fast also happen to be the cases in which I start editing photos before the preview-rendering process is finished, which means I don't mind a lower priority to the background task as much. But if Adobe coded well, perhaps it could ramp up the background task priority better during all those little pauses in my work.

[terminalblue]

@shankland

I suppose that Adobe's interpretation of it makes sense if every picture you take is perfect, requires touch up, or needs tagging. imports and exports for me are hiccup free, but it does seem like they could be faster. if they are improving the user experience by not bogging down the CPU and USB then maybe its not such a bad thing. i would rather have a smooth usable interface then a glitchy locky-uppy one.

I think that the "experience" is a big part of making lightroom work so well. I cant tell you how many time i have been in photoshop and experienced a day-killing crash. I will take the less risky code of lightroom over photoshop's "good-luck-with-that-plugin" experience anyday.

[frankwick]

I often say that Apple is the new Microsoft in "bug vs feature" speak, but this little gem puts Adobe on the map also!

[T_Hoff]

IMHO, it is self-evident that the reason cited by Tom Hogarty -- namely UI responsiveness -- is not the true reason why Lightroom doesn't take advantage of modern processors and exports more than one photo in a batch at one time.

If it were, Lightroom would not permit more than one batch of exports to be processed at one time. It would simply queue all the export batches and process them sequentially.

[jonkun227]

Adobe makes some fine products, but they also make some remarkably poor decisions in their programming. A prime example here is the fact that their software won't install on case-sensitive file systems. Okay, Mr. Hogarty, Mr.s Knoll, etc.: Explain to me how that's a give-take situation that somehow benefits the users on the flip side of that coin.

You can't. It doesn't benefit users. It's just proof of lazy coding. This way the code can reference ".../photoshop/", ".../Photoshop/", ".../PhotoShop/", or ".../PHOTOSHOP/" and it all goes to the same place. There is no legitimate reason to favor case-insensitivity except that they don't have to spend as much time scrubbing their code.

Another example: While Creative Suite (and I assume others as well) installers allow you to set custom installation locations, choosing to do so creates tremendous headaches when you actually try to use the software. It's coded so poorly that choosing a custom install location can cause fatal errors. Again, this is not an engineering decision, it's just laziness and greed. Adobe's products are at the top of the food chain for applications with broad market appeal (assuming the food chain is dictated by sales price), yet they make "engineering decisions" that many shareware developers would sooner die than commit.

I like Lightroom. It's a very important part of my workflow. But I hope that this response from Adobe doesn't lead people to automatically assume that "it's not a bug, it's a feature."

Personally (and I know I'm not alone here; just google around a bit for innumerably many others) I can't wait until someone comes along with a decent alternative to Photoshop & Illustrator. I'll say goodbye to Adobe forever, just as I have to Blockbuster, GM, and Microsoft.

[jtehero]

I love Lightroom, but I have been a victim of the slow exports. I still prefer it over Aperture any day (http://blog.jontehero.com/2008/02/aperture-2-review-lightroom-vs-aperture.html). But it is really challenging to deal with some of the exporting speed issues.

I have a MacBook Pro and a MacPro. I shoot with the 5D Mark II in RAW. I prefer to shoot tethered and allow clients to see images right away. Nowadays, my clients want me to quickly spit them out a quick DVD. Exporting a couple hundred shots on the spot (even with my Quad Core MacPro) can be very very slow. I have been able to speed it up exporting multiple smaller batches, but I don't feel like that should be necessary.

Lightroom is still my favorite, but I would love some focus on hardware utilization. I have eliminated every other bottleneck I can...it's down to this.