DVD Benchmark - Special Report - The Chroma Upsampling Error in DVD Players - April, 2001
Don Munsil and Stacey Spears
Since the launch of DVD back in March of 1997, many of us have been enjoying the amazing image that only DVD can deliver. However, back in September of 1999, I (Stacey Spears) was introduced to a new artifact that, at the time, only seemed to exist in a couple of high-end DVD players. I was standing in line at the Snell & Wilcox booth (at CES) waiting to see their demonstration when I started up a conversation with one of their employees. I had asked which DVD player they were using in their booth, and I was surprised to learn that it was a standard Toshiba. He began to tell me why, and then we went into a side room in their booth where they had a Proceed PMDT DVD player plugged into the S&W Interpolator that was connected to a 30" Princeton graphics 16x9 monitor. He spun up "The Fifth Element", and we went to the reconstruction sequence. He showed me the streaks that were running through the big red panic button.
Fast-forward two years
While some mention of this bug - which we defined as the chroma upsampling error - did get around, it was not until our Progressive DVD Player Shootout that the bug's existence become general knowledge. In fact, as we learned in the shootout, all but a very small number of DVD players actually have the bug. This includes software DVD players found on PCs. Below is what the chroma error looks like on the "Toy Story" menu that can be found on the 3-disc Toy Story box set. Notice that in the left hand photo, the blue border around the word "Toy" is clean, while in the right hand photo, it has horizontal streaks. This is caused by an error in chroma upsampling, hence the name of the bug.
We were not the first to see or even talk about the problem, but the current article is the first to be published that actually explains the problem in detail. It is a bit on the technical side, but then it is a technical problem, and a technical fix has to be worked out. Don Munsil has performed all of the legwork to understand the problem and has done an excellent job below explaining the ins and outs.
Our purpose for this article is to make everyone aware of the problem. We want to be clear up front that in most cases there is nothing your DVD player manufacturer can do because they depend on a 3rd party to make the MPEG decoder. The bug exists in the MPEG decoder used inside of the DVD player and there are only a small number of companies who actually make MPEG decoders. We just want to ensure that all manufactures are aware of the problem, so that they will motivate the MPEG decoder makers to fix the problem in future silicon.
Except for a very few rare situations, there is nothing that can be done with the current players on the market or those just now coming on the market. It will probably be a while for this to get fixed and to actually make its way into new DVD players. It would be great to see new DVD players introduced at CES 2002 without the problem, but that may be a little optimistic. By the end of 2002, though, we hope that no new player will have the bug.
Should you put off buying a DVD player today because of this bug? Absolutely not! This bug has gone virtually unnoticed for 4 years, so if you do not see it, then you are one of the lucky ones. Of course, if you continue reading this report, you may start seeing it, so be warned. If you are happy with your DVD player's performance, stop reading now. Of course we also warned you about this in the Progressive DVD Player Shootout and you read it anyway :-)
- Stacey Spears -
A new and surprisingly common video problem specific to MPEG has recently come to light. It's been seen in nearly every DVD player that we have looked at, as well as many MPEG-based digital video decoders, such as the ones used for satellite broadcasting. The effect is visible on screen as strange jagged edges on brightly colored objects. It's most visible on bright red, but it can be seen on other colors as well. We at Secrets first became aware of this wide spread problem while doing research for our recent progressive DVD player reports. At the time we wrote the reports, we had only a sketchy idea of what the problem really was. Now, after further research, we're ready to report exactly what the problem is, why it's been around so long without being noticed or fixed, and what players have or don't have the problem.
The problem, stated simply, is this: the MPEG decoders in DVD players are not, in many cases, properly converting the 4:2:0 format chroma information on the DVD to the 4:2:2 or 4:4:4 format that video encoders need.
For many, many years, video engineers have known that the eye is much less sensitive to color information than grayscale information. This is physiological in nature: the retina of the eye has more gray-sensitive cells, called "rods," than color-sensitive cells, called "cones." Your eyes can resolve much finer details in black and white than in color. For this reason, it's not useful to store full color resolution on a video recording system. Your eye can't resolve it, so why waste space (money) for it?
Consumer videotape formats like Beta and VHS take advantage of this fact by devoting most of the tape bandwidth to the black-and-white signal, called "luma" or "luminance," and less bandwidth to the color signal, called "chroma" or "chrominance." (In fact, there are subtle differences between "luma" and "luminance" and between "chroma" and "chrominance," but they aren't important here.)
The luma signal is always called "Y." The chroma signal is actually two signals, modulated together to form a single waveform on a tape. Depending on the color system and format, the two chroma signals may be labeled "U" and "V," or "Pb" and "Pr," or "Cb" and "Cr." These are all slightly different encoding formats, but they are all essentially the same in concept. On DVD, the chroma is stored as 'Cb' and 'Cr'.
In MPEG-2 - the compression format used for DVD - the Y, 'Cb', and 'Cr' signals are stored separately. Since the Y signal is the black-and-white signal, it is always stored at full resolution. But the color signals can be optionally stored at lesser resolutions, relying on the eye's relative insensitivity to color to mask the resolution loss.
The highest resolution format is 4:4:4, which means that for every 4 samples of Y, there are also 4 samples of 'Cb' and 4 samples of 'Cr'. In other words, the color signal has the same resolution as the black and white signal. This format is generally only used internally within a device to avoid degradation during processing. When the image is recorded to a master tape like D1 or D5, it is reduced to 4:2:2 (see below).
Next highest is 4:2:2, which means for every 4 samples of Y, there are 2 samples of 'Cb' and 2 samples of 'Cr'. There are still the same number of scan lines in the luma and the chroma, but the chroma signal has half as many samples on each scan line. When a 4:2:2 signal is decoded, the "missing" samples are effectively interpolated from the samples on either side.
The lowest resolution format, and the one used for DVD, is 4:2:0. This is a confusing designation, as it suggests that for every 4 Y samples there are 2 'Cb' samples and 0 'Cr' samples, which is not the case at all. What 4:2:0 means is that there are half as many samples of 'Cb' and 'Cr' on each scan line, and half as many scan lines of 'Cb' and 'Cr' as of Y. In other words, the resolution for chroma is half that of luma in both the horizontal and vertical directions. If the full image resolution is 720 x 480, then the chroma information is only stored at 360 x 240. In 4:2:0, not only are missing samples interpolated on each scan line, entire scan lines of chroma must be interpolated from the scan lines above and below.
The tricky problem of 4:2:0 is that the method of interpreting the samples and interpolating new scan lines can be done two different ways. One way is used for interlaced images, and one for progressive images.
The images on DVD, whether sourced from interlaced video or from a film, are generally stored in full frames, not separate fields (for a refresher on fields vs. frames, please refer to Part 5 of the DVD benchmark, which explains progressive DVD). If the original source was interlaced, the MPEG encoder that writes the DVD takes pairs of fields, weaves them together into full frames, and stores those frames on the disc. If the original source was progressive, like a film or a computer image, the original frames are just written to disc as-is (in fact, under most circumstances the film is converted to interlaced somewhere along the line, but a good encoder will recognize the film cadence and reconstruct the original frames properly). The encoder sets the "progressive_frame" flag to True or False depending on whether the MPEG picture represents a single frame of film or two separate video fields.
On the player end, if the original video was interlaced, then it's important that the chroma information for one field not leak into the other field. The decoder should split the MPEG picture into two fields, and then convert the chroma to 4:2:2 separately for each field. If, however, the frame was originally progressive, then the chroma information needs to be interpolated to 4:2:2 across the whole frame, then split into fields if necessary. This is a subtle difference, but it's the root cause of the chroma upsampling problem.
Encoding 4:4:4 to 4:2:0
In Figure 1 (above), you can see the layout of a 4:4:4 encoded image. For every pixel in the image, there is one luma sample and one chroma sample (actually one pair of chroma samples, one for 'Cb' and one for 'Cr', but we'll treat them as a single unit).
Figure 2 (above) shows the layout of a 4:2:2 encoded image. Here, half the chroma samples have been thrown away, and while there are luma samples for every pixel, there are chroma samples only for every other pixel. When this is displayed, the missing chroma information will be interpolated from the samples on either side. This doesn't really cause a problem. As mentioned above, the eye is less sensitive to chroma, and most people can't tell the difference between 4:2:2 and 4:4:4.
Now it gets tricky. Figure 3 (above) shows how the luma and chroma samples are laid out, conceptually, for a progressive 4:2:0 frame. There are half as many chroma samples per line, and half as many lines as well. In total, there are one fourth as many of chroma samples as luma samples. Note that the chroma samples are defined to be located between the scan lines. How can a sample be between scan lines? It's simple: the chroma samples stored on the disc are averaged from the chroma information on the original scan lines above and below. The first line of chroma samples is created by averaging lines 1 and 2 from the original image, and the second line is created from image lines 3 and 4, and so forth.
Now look at Figure 4 (above). This is how the chroma values are laid out for two consecutive interlaced 4:2:0 fields, when they are stored together as a single MPEG picture. The chroma values are still located between the scan lines, but there's an important difference. The first line of chroma values is averaged not from lines 1 and 2, but from lines 1 and 3. And (assuming the encoder is doing the correct thing), the sample should be averaged using 75% line 1 and 25% line 3, because it's defined to be "virtually" closer to line 1 than line 3.
This is a little tricky, so look again. The first "virtual line" of chroma values is still between image lines 1 and 2. But line 2 is not used for field 1; it's part of field 2. So the chroma information is averaged from lines 1 and 3. The "virtual line" of chroma is 0.5 lines away from line 1, and 1.5 lines away from line 3. So Line 1, being 3 times closer, affects the eventual chroma sample three times as much as line 3. Or at least it should. Encoders sometimes play fast and loose with downsampling, so the encoder might just do a simple 50/50 average of lines 1 and 3. Or it might just use line 1, and throw away line 3. We'll assume, just to be charitable, that it does, in fact, do the math to generate a mathematically correct 4:2:0 sample.
Note, though, that picture line 2 is not involved in encoding the first chroma line at all. Line 2 is the first line of the next field, and you don't want color from field 2 bleeding into field 1. There might be movement between the two fields, and it could produce ugly color artifacts.
In the next field, field 2, the second chroma line affects picture lines 2 and 4, but again, the chroma sample is closer to line 4 than line 2, so the sample should be calculated as 75% line 4's chroma and 25% line 2's chroma.
(In practice, more sophisticated algorithms than a simple 75/25 average, using more than 2 data points, are often used by MPEG encoders, but the point remains that the chroma information is not supposed to be calculated from a simple 50/50 average.)
Converting 4:2:0 back to 4:2:2 or 4:4:4
It's not terrifically important whether the output device converts the 4:2:0 information to 4:2:2 or 4:4:4. Many video encoders can accept either format. But it has to be in one of those two formats. The video encoder needs to get a line of chroma for every line of luma, at the same time, because it's converting the digital video to analog video on the fly. It can't "remember" what the chroma information was for the previous scan line, so it can't interpolate missing chroma information. Only 4:2:2 and 4:4:4 have a 1:1 ratio of chroma and luma scan lines, so the MPEG decoder needs to output one or the other.
The MPEG pictures are stored in the same binary format whether they represent a progressive frame or two interlaced fields woven together. The only difference is the "progressive_frame" flag. That flag tells the decoder whether the chroma information should be interpolated across the whole picture, or separated into fields and interpolated separately for each one. Unfortunately, it appears that most MPEG decoders ignore the "progressive_frame" flag, and only do one kind of interpolation. It makes the chip simpler, which reduces cost.
The MPEG decoders that use only one algorithm tend to use the interlaced algorithm, most likely because it's possible to implement it with less buffer memory, as only one field at a time must be upsampled. When the interlaced algorithm is applied to progressive images, chroma samples that were supposed to be used for scan lines 1 and 2 are instead interpolated to scan lines 1 and 3, and the chroma samples for scan lines 3 and 4 instead are affecting 2 and 4. In the simplest case, where the chroma lines are just copied to the adjacent image lines, the end result is that scan line 2 gets the chroma information for scan line 3, and vice versa, all the way down the screen. Effectively, adjacent pairs of chroma scan lines get switched, which produces the characteristic jagged/streaky effect we see in the chroma upsampling error.
Look at figure 5 (above). This is a simple red shape with a diagonal side on the left. In Figure 6 (also above), it has been converted to 4:2:0, and back to 4:2:2, correctly. The left edge is more jagged, because there are only half as many samples as before, in both directions. But it still looks pretty much like the original shape.
Now consider Figure 7 (above). The shape has been converted back incorrectly, using the simplest "copy each chroma line twice" algorithm. The chroma for line 2 has been swapped with line 3, and the chroma for line 6 has been swapped with line 7. Since the original line 7, which became line 6, didn't have any red in it, there's a gap between finished lines 5 and 7. This is the reason you will often see gaps on the top and bottom of colored objects and "ghost" lines floating above or below the objects, if your DVD player has this chroma upsampling problem.
Why This Varies From Player To Player
Different DVD players show the problem differently because they all use slightly different interpolation algorithms. In the simplest case, they just copy the data from the 4:2:0 samples twice to create the 4:2:2 data, as we show in Figures 5 through 7. This tends to make the chroma channel look blocky, and accentuates the jaggedness.
Other players do more sophisticated interpolation, using bilinear filtering, or sin(x)/x filtering, which smoothes the chroma channel somewhat and can make the upsampling error less visible. But it can't be completely hidden. It will just look like a smoother version of Figure 7, with the jaggedness and streaks on the edges less noticeable.
In order to completely fix this issue, the MPEG decoder needs to have two different chroma upsampling algorithms, and switch between them on the fly, as the "progressive_frame" flag changes from True to False. Truly advanced decoder chips will recognize the common "alternating progressive_frame" encoding error we discussed in the progressive player report, and treat all the frames as progressive. This error crops up on a lot of movies, like "Titanic", "Austin Powers", and the first releases of "Terminator 2" and "Fargo", among many, many others.
In a pinch, it would probably be preferable to decode all frames as though they were progressive instead of the current choice of decoding all frames as though they were interlaced. Most of the content that people view on DVD players is progressive. The interlaced content would potentially look wrong under certain conditions in this case, but it's not clear that the artifacts would be any more objectionable than the current artifacts.
It would also be possible to fix this problem with a de-interlacing chip. All that would be required is to resample the whole chroma channel vertically, smoothing out the errors and jaggedness. Swapping chroma lines 2 & 3, lines 6 & 7, and so on, then applying a smoothing filter, should produce excellent results.
In some cases, the flaw is not in the MPEG decoder, but in the firmware driving the decoder. We know of at least one chip company (LSI) that does include the proper interpolation algorithms on the chip, but relies on the firmware to detect the progressive_frame flag and choose the proper upsampling algorithm. In such cases, it's possible that the DVD player manufacturer could release updated firmware that would address this issue. There may be other decoders with similar features that are not being properly used, though most of the chipsets we've looked at don't offer these kinds of controls in firmware. Generally, either the chip works right or it doesn't.
We here at Secrets would like to see movement on this issue. If you don't fully understand the technology that we have explained here, don't worry about it, because (1) all you need to know is that the problem does exist and what it looks like, and (2) we are really just putting the problem on the table for everyone to see, including (especially) player manufacturers, who should make sure it is resolved in future players. We, at Secrets, are dedicated not simply to reviewing products, but to improving the industry. This kind of article won't be found anywhere else, and somebody has to do it. Since a lot of our readers are engineers who design these products . . . here you go.
When DVD players were being attached to 27" direct view TVs, the chroma upsampling error was difficult to see. As more and more people are buying high-resolution HDTVs and projection units, this problem grows ever more annoying. Please, if this problem bothers you (now that you know what to look for), complain to your equipment manufacturer. In most cases, there is nothing they can do other than put pressure on the MPEG decoder manufacturers, but in some cases, the player manufacturer can eliminate or alleviate this problem in firmware. Certainly, the more people complain about this problem, the faster we'll see fixed chips appear. There is no excuse for this error: the MPEG spec is very clear about how the chroma information should be interpreted. Kudos to the MPEG decoder manufacturers whose chips already work correctly. If only there were more of them.
- Don Munsil -
Now that the details have been explained, we have chosen three reference DVDs and a couple of scenes on each DVD where you may see the problem. Note: You will not see the chroma upsampling error in the actual images below because we used WinDVD software on our computer to capture the screen shots, and this software program does not have the chroma upsampling error.
"The Fifth Element"
"North by Northwest"
"The Fifth Element"
In the first Fifth Element example shown above is the red panic button that is seen during the reconstruction sequence (above, left). This was the material used when we first encountered the bug. While you can't see it in the image above, what you end up seeing, if the artifact is severe enough, is little streaks along the outer curve of the red button. This next shot from the Fifth Element (above, right) is outside of the Zorg facility. If the artifact is severe enough, you will see the streaks along outer edges of all the blue letters.
"North by Northwest"
This first sequence in North by Northwest is of a brightly colored taxi cab (above, left). You can see the error on all of the color transitions of the cab, and it's noticeable along the seats at the top and all over the steering wheel. The severity of the bug will depend on how much you see it in this shot, and it depends on your player. The next shot from North by Northwest (above, right) is in the hotel room just after Cary Grant escapes from the crop-duster. Here you see Eva Marie Saint in a red and black dress. This dress will show the artifact all through it and it will take on the appearance of a corduroy type fabric. She really is a walking chroma upsampling test pattern.
This next scene (above, left) will show up on any DVD player with the chroma bug, no matter how mild. If you look at the checker board itself, you will see a corduroy type pattern all throughout it. You can really see the artifact anywhere in Toy Story if the chroma problem is bad enough in your particular DVD player. We chose chapter 4 because there is so much there to see it in. This particular scene (above, right) is where Woody is standing behind the Tinkertoy box, giving a speech. You will see the streaks along the top of the Tinkertoy box and on Woody's microphone.
Players that do not have the problem:
Manufacturer Model Number MPEG Decoder Camelot Roundtable Panasonic Intervideo WinDVD 2.55 (Software) Software based JVC XV-723GD Mediamatics Panasonic A110 Panasonic Panasonic H1000 Panasonic Panasonic RV80 Panasonic Sony DVPS7000 Sony
Players that do have the problem:
Manufacturer Model Number MPEG Decoder Apex A600 ESS Cyberlink PowerDVD 3.0 (Software) Software based Denon DVD-2800 ESS Meridian 800 C-Cube Mitsubishi DD-6000 Zoran Onkyo DV-S939 Zoran Pioneer DV-05 Fujitsu Pioneer DV-09 Fujitsu Pioneer DV-37 Mitsubishi Pioneer DV-38A Mitsubishi Pioneer DVL-909 Fujitsu Proceed PMDT LSI Sony DVPS360 LSI Sony DVPS7700 LSI Sony DVPS9000ES LSI Theta Voyager Fujitsu Toshiba SD-1600 Zoran Toshiba SD-5109 Zoran Toshiba SD-6200 Zoran Toshiba SD-9100 Zoran Toshiba SD-9200 Zoran
We have only included DVD players that we have confirmed having the chroma upsampling error. We will update the list as we test new players. If a player isn't on this list, but you know it uses the same decoder chip as one that is on this list, you can provisionally assume that it will act the same. But it's still worthwhile to check the mystery player. Some of these chips have software-controlled upsampling, so it's possible that a player manufacturer could be using different firmware. In the case of software DVD players such as those that run on Windows, this can be fixed with a software update.
- Don Munsil & Stacey Spears -
"Toy Story" Images Copyright 1995, Pixar and Disney
"The Fifth Element" Images Copyright 1997, Columbia TriStar
"North by Northwest" Images Copyright 1959, MGM
Go to Part 3 - Functionality Part 4 - Usability Part 5 - Progressive Scan
© Copyright 2001 Secrets of Home Theater & High Fidelity
Return to Table of Contents for this Issue.