Why performance testing and public relations don't mix

(Credit: Apple Inc.)

Performance testing might not be the first thing you think of when you read product reviews, but it often plays an important role in helping to evaluate how some products stand up against their competition. CNET Labs takes performance testing very seriously, and it's sometimes a lot more difficult than you might think... When we encounter testing problems in the lab, we often turn first to the manufacturers to help us troubleshoot the issue. We like to give the vendors the opportunity to address the issues and offer explanations or solutions, in part to make sure we're not overlooking something or doing something wrong. If relevant, we'll mention the problems we encounter during testing within the respective product reviews.

One problem we've been struggling with for more than a month now is how Windows Vista manages to break one of our benchmarks--but only on some systems. CNET Labs' Multimedia multitasking test is a robust benchmark test that uses QuickTime Pro to encode a high-definition video, while iTunes encodes a bunch of audio files in the background. While this test works without a hitch on most of the Vista systems we've seen, a number of systems couldn't perform the QuickTime video encode. Curiously, some of these were rather high-end systems. We soon discovered these problem systems had trouble even just playing back standard-definition MOV files in QuickTime. Some couldn't even open the QuickTime application! We are well aware that the current versions of QuickTime and iTunes are not officially compatible with Vista, but our test had been running fine on the vast majority of Vista systems. We had a mystery on our hands!

We queried the Apple, Intel, and Microsoft public relations reps we regularly work with, to try to get to the bottom of this. Apple asked for more information and then we didn't hear back from them for an entire month! When we finally heard back from Apple, it was only to ask us for more information. (Ugh.) We've also been playing e-mail ping-pong with Microsoft for a month, but have yet to make any progress with the company on the problem. (Sigh.) Intel at least was quick to respond to our queries and to throw resources toward looking into the problem. (Yeah!) Of course, Intel had perhaps more motivation than the others, for as it turns out, much of the blame could be traced back to an Intel driver. Early on in our investigation, we discovered that all of the systems that exhibited this problem had RAID arrays using the Intel Matrix Storage Manager driver. Intel soon confirmed our suspicions that there was an incompatibility with its Intel Matrix Storage Manager driver for Vista and QuickTime 7.1.

Intel offered to send us an updated, prerelease version of the driver, so we could confirm the fix for ourselves. Before they would send us this new, prerelease driver, however, Intel wanted us to sign an NDA. The wording of the NDA was so ambiguous, however, that it could potentially be interpreted so as to limit CNET from publicly stating anything about Intel for the next five years without first getting Intel's permission. It is unlikely that censorship was Intel's intent; the apparent primary focus of the document was to protect Intel's intellectual property. But the boilerplate nature of the document opened the door to ambiguous legal interpretation. We didn't sign the document, and Intel didn't send us the driver.

Where does this leave us? With a test that still won't run on some systems and three major technology players still leaving CNET Labs hanging. We're hoping the problem will be resolved once Apple releases a version of QuickTime that is officially Vista compatible. Chances are, that's what Apple, Intel, and Microsoft are hoping for too. Rumor has it that we should see such a release from Apple very soon.

[tatatijo]

Why performance testing and public relations don

I am having the same problem with Quicktime on my Vista Home Premium... I get a buffer overload error!!! nice QA Apple

[techie]

I'm suffering with the same problem

I have Vista Home Premium and i often videos with my digital camera. My camera records in quicktime so when i copy them over to my system and i try to play them the sound is fine but the picture breaks up into different pixel colours most of the time green and then crashes after 5 seconds of play. The funny thing is that some videos recorded on other cameras or other apps play just fine. Any idea as to when they are gonna release the new version of quicktime?

[sumit2661]

<a href=http://www.PRCrossing.com /> "Public Relations jobs" </a>
Hi, I have searched for Public Relations jobs in may sited <a href=?http://www.PRCrossing.coml? > PRCrossing </a> and I want to state that <a href=?http://www.PRCrossing.coml? > '' PRCrossing ?</a> is a good source of jobs because it consolidates jobs from employer websites and every other job board out there--while other job boards concentrate primarily on charging employers to post jobs on their site. It also help in giving guidance to Job Seekers. It has jobs from the following fields. It also gives you a good <a href="http://www.AHarrisonBarnes.com/">'' carrer advice ?</a>.
Latest jobs are given on this site as soon they are avaible in a particular area or before any other site posts that as It <a href=?http://www.PRCrossing.coml? > shows the jobs from direct Recruiters</a> and no third party is involved in between for making money.
The number of jobs given is also guite big as compared to other sites. Jobs are included from most of the fortune 500 companies.
When I was jobless and <a href=?http://www.PRCrossing.coml? > searching for job</a> on every second site only this site helped me to <a href=?http://www.PRCrossing.coml? > find a Job.</a> So do not come under influence of people who say that its a scam. Just go through it once and you will feel the difference.