How we test: Blu-ray players

Test and reference equipment

The most important piece of test equipment is a trained, expert eye. Test patterns and the latest gear are no substitute for a knowledgeable, keen-eyed evaluator with a background in reviewing home video devices. Our principal Blu-ray reviewer, Matthew Moskovciak, has handled nearly every Blu-ray player review since the format's inception.The reference and test gear in CNET's home video lab includes:

Hardware

Oppo BDP-83 Blu-ray player: The BDP-83 is our primary reference Blu-ray player, used for all comparisons with program material and test discs.

PS3 Slim: Secondary reference player.

Reference display: CNET currently does not use a consistent reference display in its Blu-ray player reviews. We typically use one of the better-performing HDTV review samples available in the lab and make note of it in the review. We are currently considering the purchase of a permanent reference display.

Monoprice HDMI cables: CNET uses low-cost Monoprice cables in both its home video and TV labs, as we find they perform identically to higher priced alternatives in most cases. For information on HDMI cables, see our Quick Guide to HDMI cables.

Software

CNET maintains multiple copies of the following discs to allow comparison between the player being reviewed and our reference.HD HQV Benchmark Version 2.0 Blu-ray Disc: One of our primary sources of Blu-ray test patterns. We also use the DVD version for all of our DVD playback quality tests.

Spears & Munsil High Definition Benchmark: Our other primary source of Blu-ray test patterns.

Qdeo HD Video Evaluation: A secondary source of Blu-ray test patterns, occasionally used to confirm the results of other discs.

Various Blu-ray movies: Includes "Mission: Impossible III," "Sunshine," "Pirates of the Caribbean: Curse of the Black Pearl" and "Nine Inch Nails Live: Beside You in Time." See below for details. CNET has a library of over 250 Blu-ray titles to choose from.

Blu-ray player review sample information

Unless noted otherwise, CNET Blu-ray player reviews are based on one reviewer's hands-on experience with a single particular sample of one model. While our experiences are usually representative of other samples with the same name by the same manufacturer, we can't always be sure of that since performance can vary somewhat from sample to sample--particularly if newer samples receive updated firmware, or if manufacturers make changes without updating the model name. We typically review models as quickly as possible, so we often receive early versions of firmware that are sometimes corrected later. However, we never review preproduction samples. All of the samples used in CNET Blu-ray player reviews represent, as far as we can tell, shipping models.

Sometimes a firmware version will have a direct effect on the performance of a Blu-ray player, and thus on its final review score. When this is the case and we're made aware of it--usually after a CNET reviewer or a reader finds a performance-related problem--we'll note the firmware version in the review and post related follow-up information in a note referenced in the review body.

It's worth noting that CNET obtains most of its review samples directly from manufacturers, typically by an editor asking a public relations representative for the desired model. This, unfortunately, could theoretically lead to manufacturers sending non-representative samples, or even tampering with the units before they are sent, to help ensure positive reviews. If were to spot a blatant case of tampering, we'd note it in the review, but it's hard to prove (and in case you're wondering, no, we've never spotted a case of tampering that we could prove enough to mention in a review). If a manufacturer cannot ship us a sample or doesn't want us to review a particular player, we sometimes buy the model in question ourselves. We'd like to move to a buy-only methodology for procuring our review samples, but unfortunately that's not possible at this time.

Test procedure

Below we outline the procedures used in CNET's Blu-ray player tests. We strive to consistently test all Blu-ray players we review using the procedure below, but in some cases we do not. In cases where not all of the test procedures are followed, we'll note the missing items in the review.

Calibration

Unlike the extensive calibration performed in our HDTV testing, CNET does not calibrate Blu-ray players or any other home video source. This is mostly due to our philosophy that picture-affecting adjustments, such as contrast, brightness or others available on some players, should typically be performed on display devices, rather than the source. There are a few procedures we follow before testing any Blu-ray player, however.

• Update firmware: We use the latest firmware available at the time of testing.
• Confirm 1080p/60 output: While 1080p at 24 frames per second (1080p/24) is the best output option if your display supports it, CNET conducts all its Blu-ray player testing at 1080p at 60 frames per second (1080p/60). For more information, see the following image quality tests section.
• Choose the best picture setting preset: We quickly cycle through the available picture setting presets and select the option that provides the purest video source (i.e. no edge enhancement, full resolution, pass both whiter-than-white and blacker-than-black signals.)

Blu-ray image quality tests

CNET's Blu-ray image quality tests focus on a player's ability to output Blu-ray movies at 1080p resolution at 60 frames per second (1080p/60). This is because 1080p/60 is much more widely supported by older HDTVs and because, in our experience, performance at 1080p/24 is essentially identical for nearly every Blu-ray player we test. If your HDTV supports and properly handles 1080p/24, and most of the Blu-ray content you watch originates on film, you can largely ignore CNET's image quality tests, as all players put out nearly identical 1080p/24 signals.

Also note that while we include our image quality results in every review, a more detailed breakdown of our testing can be seen in CNET's 2012 Blu-ray players chart.

HQV Benchmark 2.0

HQV Benchmark 2.0 is one of our two main sources for Blu-ray test patterns. While several other discs cover many of the same tests, we prefer HQV's test patterns because we find it easier to judge a definitive "pass" or "fail". We excluded tests based on noise-reduction performance or other image-quality "enhancements" that alter the original video quality of the disc, as we feel those features are generally best left off or handled by the display. The following tests are included in every Blu-ray player review.

• Film resolution: Two test patterns that check the player's ability to handle film-based content with 2:2 and 3:2 cadences. If the player passes the 3:2 cadence, we mark it as a pass in the review, as the vast majority of program material uses a 3:2 cadence. The is the most important test pattern that we do; if a player passes this test, it will usually produce excellent image quality on the vast majority of Blu-ray movies available. We do include the 2:2 cadence test in our Blu-ray players chart, but since 2:2 program material is rare, we don't mark it as a fail in the review.
• Video resolution: Four consecutive test patterns that check the player's ability to handle video-based content. If the player fails any of the four tests, we mark it as a fail in the review. There is much less video-based content than film-based content on Blu-ray, so buyers should place less importance on the results of this test.
• Text overlays on film: Two test patterns that check the player's ability to handle video-based scrolling text, both horizontally and vertically. If the player fails either test, we mark it as a fail in the review.
• Video cadences: Eight test patterns that check the player's ability to handle various video cadences. This is scored as a fraction, i.e. 1/8. Program material that uses a cadence other than 3:2 is rare, so buyers shouldn't place much importance on these tests unless they intend to watch a lot of nonstandard Blu-ray programming.

Spears & Munsil High Definition Benchmark

CNET also includes several test patterns from the "Spears & Munsil" disc as part of our usual testing procedure. These tests focus on basic Blu-ray output capabilities that we expect every player to pass.

• Dynamic range high: Test pattern that verifies a player passes all whiter-than-white video signals. Contrast on the display is decreased as much as possible to make this easier to view. While this test shouldn't make a difference with regular program material, passing whiter-than-white signals is helpful with calibration.
• Dynamic range low: Test pattern that verifies a player passes all blacker-than-blacker video signals. Contrast on the display is increased as much as possible to make this easier to view. While this test shouldn't make a difference with regular program material, passing whiter-than-white signals is helpful with calibration
• Luma multiburst: Test pattern that checks the player's ability to pass full luma resolution. This test is passed by nearly every player we test; if a player fails this test, we consider it a major problem.
• Chroma multiburst: Test pattern that checks the player's ability to pass full chroma resolution. If the high frequency section is lower in intensity, we mark it as a fail in the review.
• Chroma upsampling error: Test pattern that check the player's ability to handle the "chroma upsampling error". (For more information, consult Spears and Munsil's excellent guide on the issue.) Although this issue is considerably more difficult to see on Blu-ray than DVD, it's still visible, especially on very large screens. We test this at both 24 frames per second and 30 frames per second; we mark it as a fail in the review if the player fails either test.

Program Material

Test patterns are excellent ways of isolating video quality issues, but they're largely academic exercises unless you can see the flaws in actual program material. CNET editors have identified several scenes that are known to reveal video processing problems in Blu-ray players.

• Ghost Rider (Chapter 6, 41:50-42:02): In this test, we're looking for moire in grille of the RV in the background as the camera pulls away. This test gives an indication of how well a player handles standard, film-based movies.
• Mission: Impossible III (Chapter 8, 46:56-47:05): In this test, we're looking for moire in the stairs in the background. This test generally indicates how well a player handles standard, film-based movies.
• Sunshine (Chapter 2, 04:40-05:08): In this test, we're looking for jaggies on the edges of the glasses on the table. This test generally indicates how well a player handles standard, film-based movies.
• Tony Bennett (Chapter 7, 14:19-15:20): In this test, we're looking for jaggies in the graphics at the beginning of the clip, and also in the dancers shirts toward the end. This test gives an indication of how well a player handles video-based movies.
• Nine Inch Nails Live: Beside You in Time (Chapter 3, 08:10-08:16): In this test, we're looking for jaggies in the guitar strings. This test gives an indication of how well a player handles video-based movies.
• Nine Inch Nails Live: Beside You in Time (Chapter 4, 15:15-15:42): In this test, we're looking for comb artifacts in the red background. This test gives an indication of whether or not the player suffers from the chroma bug.

Operational speed tests

Disc-loading and navigation speed are big contributors to how enjoyable a Blu-ray player is to use. We've come up with a several disc-loading tests for player's operational speed in a variety of scenarios. We run each test three times, then average the scores.

• Disc load; player on; simple movie ("Mission: Impossible III"):With the player already on, time from hitting "disc close" until the Paramount logo shows up on the screen.
• Disc load; player off; simple movie ("Mission: Impossible III"); quick start disabled: With the player powered off, time from hitting "power" until the Paramount logo shows up on the screen. If the player supports automatic disc loading, this feature is enabled; otherwise we attempt to get the movie playing as quickly as possible using the remote. If a "quick start" or similar mode exists, the feature is disabled.
• Disc load; player off; simple movie ("Mission: Impossible III"); quick start enabled: Same as the previous test, except any "quick start" mode is enabled.
• Disc load; player on; complex movie ("Pirates of the Caribbean: Curse of the Black Pearl"); until previews: With the player powered off from the time until the "Walt Disney" logo shows up on the screen. This tests both the players ability to load movies with more complex menus.
• Disc load; player on; complex movie ("Pirates of the Caribbean: Curse of the Black Pearl"); until movie: With the player powered off from the time until the first glowing embers show up on the screen. We attempt to skip by any previews as quickly as possible; if chapter skip doesn't work, we use fast-forward. This tests both the players ability to load movies with more complex menus, its operational speed and how long it takes to "get to the movie".
• Disc load; player on; complex movie ("Spider-Man 3"); until movie: Same as the previous test, with a different movie.
• Chapter skip; skip from chapter 1 to chapter 15; ("Sunshine"): While "Chapter 1" is playing, we start the timer when we start hitting the chapter skip button on the remote until we get to "Chapter 15", and end the timer when the movie resumes playing. This tests the player's operational speed.

CNET's speed rating

This is a composite score made up of all CNET's operational speed tests. The score is based on how far the player's speed deviates from the reference, which is an average player from 2011. The reference gets a score of 100. It's possible for players to score above 100 if they're faster than last year's average player; the reference is just a baseline to compare new Blu-ray players. While we find it to be a useful simplification of our operational speed tests, it's important to recognize it's limitations--i.e., a player with a "50" score won't necessarily feel twice as slow as a player with a "100 score".

DVD image quality tests

Every Blu-ray player is also capable of upscaling standard DVDs. Our DVD testing is admittedly less comprehensive than our Blu-ray tests, but gives a good indication of what you can expect with most film-based DVDs. DVD test patterns also tend to be more subjective than Blu-ray test patterns and we use our best judgment on borderline pass/fail decisions.

HQV Benchmark 2.0 DVD

• Film resolution: Two test patterns that check the player's ability to handle film-based content with 2:2 and 3:2 cadences. If the player passes the 3:2 cadence, we mark it as a pass in the review, as the vast majority of program material uses a 3:2 cadence. We do include the 2:2 cadence test in our Blu-ray players chart, but since 2:2 program material is rare, we don't mark it as a fail in our reviews.
• Video resolution: Four consecutive test patterns that check the player's ability to handle video-based content. If the player fails any of the four tests, we mark it as a fail in the review. There is much less video-based content than film-based content, so buyers should place less importance on the results of this test.
• Text overlays on film: Two test patterns that check the player's ability to handle video-based scrolling text, both horizontally and vertically. If the player fails either test, we mark it as a fail in the review.
• Video cadences: Eight test patterns that check the player's ability to handle various video cadences. This is scored as a fraction, i.e. 1/8. Program material that uses a cadence other than 3:2 is rare, so buyers shouldn't place much important on these tests unless they intend to watch a lot of nonstandard DVD programming.

Program Material

Just like with our Blu-ray tests, we include real program material in our DVD image quality tests. Again, our testing is less comprehensive for DVDs, but gives a good indication of what to expect with the majority of DVD titles.

• Seabiscuit (Chapter 1, 01:06-02:34): In this test, we're looking for jaggies and moire in the black-and-white photographs in the opening sequence. This is a difficult test for most players; if a player passes, it's likely to handle most film-based movies well.
• Star Trek: Insurrection (Chapter 1, 01:08-01:30): In this test, we're looking for jaggies on the bridge railings and boat hulls as the camera pans over the village. This is a basic test for 2:3 pulldown; if a player fails, it indicates that most film-based movies will look mediocre.
• Invite Them Up, DVD extra ("Fuggedabuddies"; 00:00-00:34): In this test, we're looking for any comb-like artifacts on the host's face. This test indicates how the player will handle niche video-based content and should be considered much less important that the previous two tests.

Streaming video testing

The majority of Blu-ray players now include a full suite of streaming media services, such as Netflix, Amazon Video On-Demand and Vudu. Because there are now so many services, it's not feasible to test the performance of every service on every player. It also wouldn't be a wise investment of time; we generally find image quality performance of a particular service to be nearly identical on different devices. Instead, CNET focuses its effort on the most popular streaming video service, Netflix, and runs a simple test to catch any major problems.

• Lost (Season 5; "Some Like it Hoth"; 00:00-05:00): In this test, we're looking at general sharpness as well as compression artifacts like pixilation (macroblocking), especially in the flat backgrounds. This is one of the better-quality HD titles on Netflix and doesn't normally exhibit these problems in the service's highest-bandwidth setting. This is a relatively subjective test and we score the scene as either "Good", "Average" or "Poor", based on our previous experience with these services.

For the more information about the performance of any specific service on a particular player, CNET recommends looking at user opinions to get a sense of any major issues that may arise with more long-term testing.