SWE2 perhaps? hehe

the link doesn't work for me either

Lobes vs. Taps. The short version.

Any signal-processing filter has "taps" which are the discrete input values that are used to calculate each output value. In audio processing it's pretty straightforward - a 20-tap FIR resampling filter takes 20 audio samples, multiplies each by one value in a 20-element filter "kernel," adds all the results together, and spits out a single output sample. Lather, rinse, repeat. In video it's a little different, but that's the rough idea.

When a filter is based on a windowed sinc function (sinc is (sin(x)/x)), it's convenient to talk about the number of lobes of the sinc function that are being used. X=0 to X=pi is one lobe. X=pi to X=2pi is another lobe. X=2pi to X=3pi is another lobe and so forth. Because all windowed FIR sinc filters are symmetrical around 0, we only talk about the number of lobes on one side of 0. There are an equal number of lobes on the the other side.

So when people talk about Lanczos2, they mean a 2-lobe Lanczos-windowed sinc function. There are actually 4 lobes -- 2 on each side. The function, graphed in the range -2pi to 2pi looks kinda like a sombrero. I've attached a graph of Lanczos2. Catmull-Rom looks almost identical, but is not a windowed sinc filter, or in fact any kind of sinc filter. The fact that two resizing filters created using completely different mathematical principles look almost identical is not a coincidence, but understanding why it turned out that way was certainly a big ah-ha! moment for me. And way too complicated to explain here, sadly.

For upsampling (making the image larger), the filter is sized such that the entire equation falls across 4 input samples, making it a 4-tap filter. It doesn't matter how big the output image is going to be - it's still just 4 taps. For downsampling (making the image smaller), the equation is sized so it will fall across 4 *destination* samples, which obviously are spaced at wider intervals than the source samples. So for downsampling by a factor of 2 (making the image half as big), the filter covers 8 input samples, and thus 8 taps. For 3X downsampling, you need 12 taps, and so forth.

The total number of taps you need for downsampling is the downsampling ratio times the number of lobes, times 2. And practically, one needs to round that up to the next even integer. For upsampling, it's always 4 taps.

So a 2-lobe (which is really a 4-lobe) Lanczos filter could require any number of taps. To downsample a 1080p image down to 108 pixels high would require 40 taps.

All of these numbers are, of course, just in one dimension. To get the full number of taps required for an output pixel, you would theoretically calculate the number of taps required in each dimension (height & width) and then multiply them together. So upsampling would require 4x4=16 taps. By convention, though, we still call that a 4-tap filter (unless we're in marketing, and then we call it a 48-tap filter: 16 taps times 3 components R, G, and B ).

Practically speaking, for performance you would always scale the image in two passes, which is perfectly kosher (or at least it is with these filters, which are *separable*, meaning that we can apply them separately in X and Y and get the same result as applying the combined filter). Conceptually you scale just in the width dimension into a new buffer that is as wide as the target image and as high as the original. Then you scale that intermediate image just in the height dimension into a final target buffer. Thus in the upsampling case you are applying two 4-tap filters instead of a single 16-tap filter.

Again, there are a huge number of practical considerations. You end up pipelining things very differently than the "conceptual" algorithm, mainly to maximize cache coherency. In order to do any kind of arbitrary scaling, you need to calculate a large number of filter kernels, as the phase between the input and output samples changes as you work your way across the image.

Man, if that's the short version I hope I never have to write the long version. Again, the thing to do if you want to understand practical scaling algorithms is to read the Turkowski paper I referenced earlier in the thread and read the code for Paul Heckbert's "Zoom" (it's not the cleanest code in the world, but understanding why he does various things is a real education). Start there and then branch out.

And if you have questions, you can always email me. My email address is not hard to find. I can't always promise I can answer every question, as among other things some of the stuff I'm doing right now with scaling is about to be patented, but I'm happy to steer folks in the right direction.

As for the questions about the best ways to use ffdshow, I have to admit that I don't know. At home I use the scaler built into the VW10HT, which looks to me like a basic bicubic, which is perfectly fine. At work I use my own scaler. I don't really have time to evaluate ffdshow extensively.

I will say that I am adamantly against filters with more than 2 lobes when scaling images. Their advantage is extra sharpness, but they ring like crazy. In my own scaler, I adjust sharpness by combining a modified sharpening kernel with the basic scaling kernel. It makes the image just as sharp as a multi-lobe Lanczos, but without the multi-lobe ringing.

Don

Here's Lanczos3.

Here's Lanczos10. The extra-deep first negative lobe (from 1 tp 2 and -1 to -2) gives you the extra sharpness, but all the other lobes (beyond 2 and -2) produce annoying fringes around sharp edge transitions. Bleah.

The perception of sharpness in an image scaling filter is almost entirely related to the depth of the first negative lobe. The more lobes you put on a Lanczos (or any other windowed sinc) the deeper the first negative lobe gets. But that's an incredibly pricey way to deepen the negative lobe, and has ugly side effects. There are lots of other ways.

Don,

Thank you very much for your 'simple/short' explaination... it is much appreciated

Wow, great thread. Especially thanks Don.



That shouldn't be happening. But when working with integer arithmatic it is possible when you aren't careful to have a slight downward truncation bias. And when working in YUV color spaces this will appear as a greenish shift.

Don -

You've suggested, and I believe, that different algorithms may be appropriate depending upon whether you are upscaling or downscaling. Any further comment?

And if you are willing, any comments on downscaling when you expect to upscale again later? (very often the case, so it could be optimized with that intent)

- Tom

A little comparison that also features some lanczos vs. bicubic.
It is slightly OT though:

http://www.avsforum.com/avs-vb/show...209#post4752209

I am very excited to see how this thread matures, hopefully it will gain as much momentum as some of the other monster threads, this has been very enlightening indeed.

Cheers...
Duy-Khang Hoang

We are using Catull-Rom for both up and down scaling.

I was hoping to include some split-screen resize demos (2-lobe vs. 4-lobe vs. 10-lobe) on the upcoming DVE Pro (WMV HD discs), but I did not have time. There is a really cool color space demo that I am proud of.



Not sure I understand. What is the scenario?

Personally, keep the image as big as possible for as long as possible. We recently started off with some 4096x1714 content. It was 16-bit per channel (4:4:4) or RGB48. The end results was 1280x536 8-bit 4:2:0.

In another thread that I forget Chalres Black posted some scope pictures of ffdshow scaling and the ringing on all Lanczos options was pretty obvious...

The ringing is not added by Lanczos, Bjoern Roy has good evidence on his site.

Stacy -

I was actually talking about the funnel video typically goes through when compressed for delivery. We start with an oversampled video and then maybe make a nice processed compressed video for delivery on some medium.

But the story doesn't stop there as this thread shows. At display time it is often desirable to display at greater than the delivered resolution, if only to hide the raster or pixel structure.

The idea of having 3 separate capture, delivery, and display resolutions was actually best presented to me by a Microsoft paper. See section 2 of Converging Computer and Television Image Portrayal by John Watkinson.

But to me this means that as we choose or design our downscaling method we should probably be considering the eventual optimal upscaling when we evaluate the possible quality.

For instance Don above mentioned customizing the scaling a bit to add sharpening. But should that be done (if at all) during the downscaling, before compression? Or should something like that be best deferred to the later stage, sharpening while upscaling? (better compression but sharpened compression artifacts)

- Tom

Please direct me there...

Actually its in the ffdshow thread page 161...

Filters that ring

Exactly. It's really best to do all the calculations in floating point, even if the input and output samples are integers. Careful rounding on the conversion back to integer is also recommended.



In a sense you can think of upsampling as interpolation and downsampling as lowpass filtering plus interpolation. For speed and convenience most scalers tend to use the same filter kernel for both, and honestly I think that works out fine.

When I first started working on scalers, a video expert told me that downsampling required very long (many lobe) filters in order to get reasonable frequency response. I went down that road for a long time, but long filters produce lousy-looking video, IMO. It took me a long time to really understand why. The key insight is this: frequency response in digital image scaling filters is irrelevant. Completely and totally irrelevant. You will find many "experts" who disagree, but they're wrong and I have the math (and the practical experience) to prove it.

The very best image scaling filters combine smooth interpolation with edge sharpening. The smooth interpolation is important so no false edges are added to the image. The edge sharpening is important because our visual system is very sensitive to edges, and smooth interpolation without sharpening reduces the slope of edges, making the image look soft.

The "perfect" scaling filter would produce a lovely smooth interpolation of objects in the scene while keeping the edges perfectly sharp. Such a filter does not (to date) exist, so we make do with compromises. Catmull-Rom is such a compromise, and it works pretty well. The one downside to it is that it's not quite sharp enough for most people. It's sharper than bilinear, and also a better interpolator, but many folks prefer a sharper image.

Thus the big practical issue with a scaling algorithm is how to get a sharper image while retaining the smooth interpolator. Many-lobed windowed sincs (like Lanczos3 and above) are one answer, but they really aren't appropriate for image scaling because they produce ripples around edges. Mathematically speaking, they don't at all reproduce the scaling of "reality."



Don't ever downsample if you don't have to. If you do have to downsample, use the best algorithm you've got. If you are resampling and you know you'll be resampling again later, don't do any extra sharpening on the first pass. Sharpening of any kind should only be applied at the very end, because sharpening must be tailored to the display environment. The appropriate amount of sharpening for a 32" display viewed at 8 feet is completely different from the amount of sharpening appropriate for a 96" screen viewed at 12 feet.

Don

"Again, the reason you see a bit more ringing for example in the Lanczos Unbreakable sample is NOT because it 'adds' that ringing. Its because the transfered frequency response is much better with that filter, thus those higher frequencies of which the ringing consists, are less subdued, thus more visible. So the bilinear filter 'blurrs away' some of that ringing. The problem is, that 'real detail' in that freqency range are blurred away as well."


http://www.videophile.info/Editoria..._EE/Page_01.htm

Bjoern is comparing bicubic and bilinear filters to 2-lobe Lanczos. Two-lobe Lanczos is fine. It's Lanczos (or any other sinc-based filter) with more than two lobes that gets problematic.

However, if you're going to use 2-lobe Lanczos, you might as well use Catmull-Rom since as mentioned previously the two are virtually identical. The reason you see any difference between the Lanczos filter and the "bicubic" filter in various image processing apps is that the bicubic they are using is generally not Catmull-Rom. Typically it's a Mitchell-Netravali filter with the coefficients tweaked to be "sharper."

Can you tell that I like Catmull-Rom a lot? I do. I like Lanczos2 as well, but again, the two are practically identical.

Don

In the best of all possible worlds (IMO), all consumer DVDs would be carefully scaled from a 1080p master using straight unadulterated Catmull-Rom with no additional sharpening at all. Then the displays would sharpen the image appropriately as a final step.

However, the reality is that people like sharp video, and thus the people making DVDs tend to sharpen the image so it "snaps" on a 20" monitor. That level of sharpening looks really excessive on a 80" screen, or even a 50" screen. And once the image has been sharpened, it's practically impossible to unsharpen it. In addition, the sharpening has been applied at the resolution of the DVD, not at the resolution of the display device. If you have a 1280x720 display, you can do a much better sharpen than the DVD maker can do at 720x480.

Sharpening is not a bad thing. Not a bad thing at all. But it really should be done as a final process, at the final resolution of the display device.

Don

Well, I'm certainly not an "expert" and I'm not sure I even disagree. But there is one part of this I at least worry about.

To me there seem to be at least 2 aspects of sharpness. One is the sharpness of edges. Our eyes like nice sharp edges and they likely help us a lot with eye motion tracking. Sharp edges certainly make the image look less soft and I think are the basis for your comments above. And sharpness of edges is probably not well reflected in frequency domain measurements.

But it does seem to me that at least my own eyes also measure sharpness by the amount of visible "texture". Textured objects also can look either sharp or soft and I don't think it is just an edge phenomena. Instead it does seem to me more a matter of frequency response, and MTF. That is, maybe texture is repeated higher frequency info over some area of the image, possibly perceived by the eye without much phase info. This is just my own perception and otherwise just quack science but if I'm at all right then preserving this texture info might be important, over and above sharp edges.

Comments?

- Tom

You're absolutely right about the sharpness of texture being important to perception of sharpness. It's of secondary importance to the big edges we tend to think of - the outlines of objects - but it's important.

However, texture is also an edge issue. Texture in reality is all about the juxtaposition of unlike-colored bits. Keeping those bits sharply contrasting up to the Nyquist limit (or as close as we can get) is what sharpness is all about.

However, it's worth noting that when talking about digital images the Nyquist limit is misleading. In audio, I really can represent a 75% amplitude sine wave that is just under (say, 15% under) the Nyquist limit and it will come out of the speakers sounding just fine assuming I've got a good D/A converter. In video, a pattern that has a frequency just under the sampling limit of the CCD in the camera will look absolutely awful.

One key insight is that audio is evaluated by our sensory apparatus in frequency space. The ear is a frequency analyzer. Vision is evaluated by our sensory apparatus in amplitude space. Our eyes are not sensitive to spacial frequency per se. Edges in audio are largely meaningless, wheras in video they are everything.

The other key insight is that the most frequency accurate image scaling filters have terrible ringing that does not represent any kind of real-world image artifact that exists. The blurring that happens as you move away from an object in real space is not analagous to a sinc function. If anything, it's closer to a gaussian, but it's not even that.

When you think about what "real" scaling represents, it becomes clear that spacial frequencies are not relevant at all. Take two possible examples of real scaling: look at a scene through a window at 3 meters, then walk to 6 meters and look at the same scene. Or imagine talking a digital photo of a scene with a 6 megapixel camera, then switching to a 3 megapixel camera and taking the same picture. Your goal with a scaling algorithm is to as closely as possible replicate those scenarios.

One thing becomes clear: you don't have enough information in the image to replicate either of those scenarios. The best you can do is make a guess and try to come up with an image that is a reasonable approximation of the image you'd really get if you were to shoot that same scene with a lower-resolution device.

Several researchers have attempted to develop scaling algorithms that try to use an iterative heuristic to actually physically model the lower-res CCD, but it's so far quite computationally expensive, and in the end you get just one of many possible image results, and the overall quality of the scaling isn't really significantly better than Catmull-Rom. It's still promising as an area of research, though.

For now, the three most important things to keep in mind for image scaling:

1. Preserve edges.
2. Preserve amplitude locality (i.e. don't distribute image information too far from the original pixel)
3. Interpolate smoothly.

- Bilinear is bad at 1, good at 2, and not great at 3.
- Straight Catmull-Rom is pretty good at 1, pretty good at 2, and great at 3.
- Many-lobed sincs are good at 1, terrible at 2, and great at 3.
- Catmull-Rom plus sharpening is great at 1, very good at 2, and great at 3.

Don

Okay, lots to think about. More later.

- Tom

It is amazing people will believe in atoms and molecules moving in a solid object like wood, yet they can't see it. Ask them about faith or belief in a being greater than themselves and...NOT!

It is one thing to accept a postulate based on reports of multiple experiments (most designed to DISPROVE the theory) by multiple independent people. (Besides, a good tunnelling microscope can 'see' molecules.)

It is quite another to accept a postulate based on the premise that "it must be so" as defined by some particular 'interest group'.

You're far better off to start with Godel's Incompleteness Theorem.

Am I right in assuming ffdshow has no Catmull-Rom resizing?

Also, anybody know what kind of algorithm current video cards use? Bicubic sticks in my mind for some reason.

Current video cards use a variety of scaling solutions, ranging from some form of bicubic to trilinear. What algorithm is used depends on whether you're using the overlay or the VMR, whether YUV mixing mode is turned on, what generation of the hardware you've got, the size of the image, and so forth.

I just talked to one of our experts on video cards here, and all the different caveats and exceptions made my head swim.

One thing to consider is that typically hardware image processors have a fixed number of available taps. So if the theoretically correct bicubic scaling algorithm requires 10 taps to do a particular scaling operation, and the hardware only has 6, they have to make do. Thus really large downscaling operations tend not to be "correct" bicubic, even if the card has the capability of doing bicubic.

Around here, we tend to prefer using VMR in YUV mixing mode, but even that won't guarantee what scaling algorithm you're going to get. Sorry I don't have a simple answer.

Don

Boy am I glad we have this thread. It was always assumed that upscaling requires the more accurate algo with the video cards downscaling being more than enough to handle the downscale to display device res. Our eyes tend to confirm this and we were happy. Now we have all these other proofs and theories to think about... which is always nice. I do hope we can come to some conclusions or at least some general guidelines... for instance if it is established that Lanczos2 is all that is needed for good upscaling then that would make many people with moderate hardware happy.

Cheers...
Duy-Khang Hoang

if this is true, then I won`t have to spend $1000CAN for a new system I was planning to purchase!!!!

Right now the only Lanczos2 is in SSE2 optimized version of FFDShow and it definitely has issues. Lanczos3 as well. Once you get to Lanczos4 it looks good. Beyond Lanczos 4 it rings excessively. Avisynth has Lanczos3 and Lanczos4 and that's about it.

It would be great if Andy could implement at least current FFDShow SSE2 algos with 32 bit floating point precision. That would tighten up the rounding errors and potentially improve image quality. It would be even better to implement 32 bit floating point on a new and improved algo, but the obvious question is "what algo".

As far as what hardware will you need depends on your preferences. If you want to run filters after 720p resize, you will still need a P4 3.0, which is a moderate machine nowadays. I still like to run a filter after Lanczos4 resize and ASharp is IMO the best. It gives you ability to balance between texture sharpening and edge enhancement which is unique. Also can improve appearance of contrast in the image, which is great for a number of digital displays.

As far as the video card, if you are planning to use VMR9, you will need either a solid card like Nvidia FX 5700 Ultra, or a very good card like Nvidia GeForce 6600 or 6800 series. If you want the best, you will need to go for Quadro FX 3400+. Although Quadro cards are (almost) identical to their GeForce counterparts, Quadro drivers provide greater precision in image assembling and rendering. With VMR9 renderer Video card does much more than just scaling and driver support is much more important than with overlay. It's the entire 3D pipeline that assembles and renders the image, not a dedicated portion of the chip like in overlay mode.