Shedding some new light on 3DMark 03 and Nvidia

[Demigod]

I was browsing Tom's Hardware Guide today and found this article:

3DMark 2003 - Talking back to Nvidia

The column basically details the rebuttal by Futuremark on Nvidia's harsh criticism of their DX9 benchmarking tool 3DMark 03. The main points (summarized, not quoted) are highlighted below.

Nvidia: 3DMark 03 does not accurately reflect gaming performance. It does not portray a true gaming experience that gaming enthusiasts expect on their hardware and deceives the consumer.

Futuremark rebuttal: 3DMark 03 is a benchmark that's meant to test the upcoming generation of DX9 games (6-12 months ahead). Currently there are no other significant DirectX 9 application published. 3DMark 03 is a forward-looking tool for measuring performance on future games and informs consumers on the kinds of impacts their choices of hardware will make.

Nvidia: The choice of pixel shaders are odd in game 2 and 3, which use pixel shader v1.4. Fallback versions are provided for PS v1.1 but strangely absent are PS v1.3 and v1.2. Pixel shader v1.4 is hardly used in any game and the game demos do not reflect current DX8 games.

Futuremark rebuttal: There are 4 pixel shader models (v1.1, v1.2, v1.3, v1.4) in the DX8 specifications. To do a fair benchmark we wanted the hardware to do minimum number of passes required, thus the choice for v1.4 (v1.1 = 4 texture lookups, v1.4 = 12 [6 * 2] texture lookups). We've also concluded from our tests that PS v1.2 and v1.3 do not provide any additional capabilities or performance over PS v1.1 (PS v1.4 specs are available on ATI's site for a comparison of the various models).

ATI has also published this (partial) list of games that use PS v1.4:

UT 2003
Madden 2003
Tiger Woods 2003
Nascar 2003
NeverWinter Nights (OpenGL equivalent)
Doom 3 (OpenGL equivalent)

Nvidia: The mother nature test is hardly DX9 at all. Only 2 of the 7 pixel shaders used are v2.0 while the rest are still v1.4 from DX8.

Futuremark rebuttal: Once again a good application should draw a scene as efficiently as possible. Game 4 uses PS v2.0 in some areas and PS v1.1 and 1.4 when complex shaders are not required.

The article also points out some possibly inherent flaws in the NV30 design. Nvidia's primary distaste of the benchmark seems to stem from the fact that the GeForce FX does not perform well on it but this may be due to the way the NV30 operates. John Carmack already pointed out that the NV30 performs twice as slow as the R300 in an apples-to-apples comparison (using ARB2) and only pulls ahead when using its own vendor-specific modes. The performance issue may be due to the NV30 using 32-bit precision while the R300 uses 24-bit precision (although some other factors may also be involved). In such a case the GeForce FX's higher precision is actually a major drawback, as 32-bit precision offers no noticeable graphical improvement over the 24-bit precision that the R300 uses but has a much larger performance hit (reminiscent of the GeForce 4's fixed higher-quality AF vs the Radeon 9000's adaptive AF). With regard to better-performaing drivers for the GeForce FX they may actually be scaling back the precision to 16-bit, which will allow the card to outperform the R300. More harrowing is the likelihood that the NV31 and NV34 will follow suit with the NV30 and also require vendor-specific routines to perform optimally.

The GeForce FX only seems to perform well in favorable situations that use its own vendor-specific rendering modes while the Radeon 9700 performs better in standard DirectX 9 routines. 3DMark03 is meant to give an objective view of performance in DirectX 9 applications using strictly standard DirectX 9 code. It is not vendor-specific and does not look favorably upon either the R300 or the NV30 or any other card, but its performance will depent wholly on how the hardware is able to handle standard DX9 calls (including all previous pixel shader subsets: v1.1, v1.2, v1.3, v1.4).

Nvidia also seems to disfavor pixel shader v1.4, claiming that it's ATI's specific extension and that DirectX 8.1 only came about to support it when in fact it's Nvidia who's failed to support it in their GeForce 4 cards. Pixel shader v1.4 is much closer to pixel shader v2.0 than any other previous version and any developer would be smart to support it, since it can increase performance significantly (there are benchmark results using v1.1 vs v1.4 on the column). Each pixel shader model is also a superset of previous models so all DirectX 9 cards MUST have support for v1.4, which is why they use it in game 2 and 3. This also goes for the GeForce FX (to which support for PS v1.4 is unclear).

I believe this column sheds some light on the strategy that Futuremark took with 3DMark03 and the basis for Nvidia's criticism. Nvidia is taking a dangerous route with the GeForce FX as they're not optimizing their cards for the DX9 standard but may try to force developers to use their own vendor-specific routines. As it stands they are the only major chip manufacturer not to support the benchmarking software, suggesting that they want to go on a different path in their bid for GPU market dominance.

Futuremark rebuttal: There are 4 pixel shader models (v1.1, v1.2, v1.3, v1.4) in the DX8 specifications. To do a fair benchmark we wanted the hardware to do minimum number of passes required, thus the choice for v1.4 (v1.1 = 4 texture lookups, v1.4 = 12 [6 * 2] texture lookups). We've also concluded from our tests that PS v1.2 and v1.3 do not provide any additional capabilities or performance over PS v1.1 (PS v1.4 specs are available on ATI's site for a comparison of the various models).

No But they provide more instructions and better optimisations for the Pixel Shader 1.3 Routine higher programability. Thery missed the point about the instruction capabilities.

Since its the its the standard, I see no reason not to use it. If they wanted compatibility. They could have fallenback on 1.1 and 1.3 and 1.4

Futuremark rebuttal: Once again a good application should draw a scene as efficiently as possible. Game 4 uses PS v2.0 in some areas and PS v1.1 and 1.4 when complex shaders are not required.

In Other Words. The game uses Pixel Shader 2.0 on the on the water. Vertex Shader 2.0 on some of the leaves. And Pixel Shader 1.1 (For Non ATI cards on everything else) And Vertex Shader 1.3 on the grass.

While it is DX 9.0 tech. They still make the mistake of using 1.1 Pixel Shading. With less optimised instructions. I don't want to hear for "Compatibility sake" 1.3 Pixel Shader is the standard. If they were worried about compatibility they could have chosen to use 1.3 1.1 and 1.4

The article also points out some possibly inherent flaws in the NV30 design. Nvidia's primary distaste of the benchmark seems to stem from the fact that the GeForce FX does not perform well on it but this may be due to the way the NV30 operates. John Carmack already pointed out that the NV30 performs twice as slow as the R300 in an apples-to-apples comparison (using ARB2) and only pulls ahead when using its own vendor-specific modes. The performance issue may be due to the NV30 using 32-bit precision while the R300 uses 24-bit precision (although some other factors may also be involved). In such a case the GeForce FX's higher precision is actually a major drawback, as 32-bit precision offers no noticeable graphical improvement over the 24-bit precision that the R300 uses but has a much larger performance hit (reminiscent of the GeForce 4's fixed higher-quality AF vs the Radeon 9000's adaptive AF). With regard to better-performaing drivers for the GeForce FX they may actually be scaling back the precision to 16-bit, which will allow the card to outperform the R300. More harrowing is the likelihood that the NV31 and NV34 will follow suit with the NV30 and also require vendor-specific routines to perform optimally

This based on an API that is not released yet. The thing you have also neglected to mention is that carmack said with driver improvements the GeforceFX will also be faster.

Carmack also said that the Extensions are the best way to go right now. We are talking about Programmable OpenGL extensions. Which in fact are the only thing that provides life to the OpenGL APi right now.

You can't call it Nvidia forcing DX instructions on anyone because its "not" DirectX. Carmack chose a dangerous route choosing OpenGL as his API for future game engines. Considering the API is dangerously out dated. If not for Nvidiia's Ati extensions. He could not make the game what it is to today.

Another thing I found amusing about ATi's Pixel Shader comments is that half those games didn't even work with ATI's Pixel Shader drivers at first. Neverwinter Nights Being an example. Which took 8 months to even support Pixel Shader, The only reason it supports Pixel Shader 1.4 is because that Pixel Shader 1.4 is the only extension supportable by ATI's OpenGL driver extensions. Which would force the developers to use 1.4 in that game.

The Same thing for Doom 3. As they require ATi's Pixel Shader 1.4 extension because. They never provided 1.3/1.1 ect. This is why Bioware had so much trouble getting Shiny Water working in this game.

Another thing that has been neglected to menton is Nvidia And ATI both mutually agreed on the Arb2 pathway. Which in fact is because both companies want a standard to be created.

The Drivers for OpenGL 2.0 on the Nvidia card aren't matured. In Time (when the API actually isn't Vaporware) maybe this will be important. So As Carmack Said. In time the performance will improve.

Nvidia also seems to disfavor pixel shader v1.4, claiming that it's ATI's specific extension and that DirectX 8.1 only came about to support it when in fact it's Nvidia who's failed to support it in their GeForce 4 cards. Pixel shader v1.4 is much closer to pixel shader v2.0 than any other previous version and any developer would be smart to support it, since it can increase performance significantly (there are benchmark results using v1.1 vs v1.4 on the column). Each pixel shader model is also a superset of previous models so all DirectX 9 cards MUST have support for v1.4, which is why they use it in game 2 and 3. This also goes for the GeForce FX (to which support for PS v1.4 is unclear).

Considering they are DirectX 8.0 tests. It destroys futuremarks entire argument. And once again. They could have used a more optimised Pixel Shader 1.3 Code, Which is the standard.

Now to my cirticisim of 3dmarks Shader. It calls for 6 Raster Shadings. 6. You will never need more than 2, Or Possibly 3 under any ideal occasion. And Three would be pushing the candle on overkill.

The Reason being. Is You are shading a single object multiple times to provide an effect, Like Skin definition. But beyond the 3rd pass. There will be nothing noticable to the human eye,

They did 6 pixel shader passes on things like the walls. To What Purpose? I mean other than slamming hardware and providing their benchmark future proof?

How will slamming hardware needlessly ever reflect gaming performance? Which Brings it back to my original conclusion. This is no gamers benchmark as futuremark has claimed.

I believe this column sheds some light on the strategy that Futuremark took with 3DMark03 and the basis for Nvidia's criticism. Nvidia is taking a dangerous route with the GeForce FX as they're not optimizing their cards for the DX9 standard but may try to force developers to use their own vendor-specific routines. As it stands they are the only major chip manufacturer not to support the benchmarking software, suggesting that they want to go on a different path in their bid for GPU market dominance.

See, Tom also mentioned that. Without 3dmark2003 being optimised by both companies (Which it will no longer be) And Nvidia being out of the partnership. The Benchmark will "never" be a fair analyst of performance on either cards now. Because ATI will always be in the in the loop. And Nvidia will not. Without a fair representation on performance (Even tho I will never use this benchmark again)

How can it ever reflect gaming performance?