Futuremark backs down on nvidia's optimisations

Futuremark Statement

For the first time in 6 months, as a result of Futuremark's White Paper last Friday, Futuremark and NVIDIA have had detailed discussions regarding NVIDIA GPUs and Futuremark's 3DMark03 benchmark.

Futuremark now has a deeper understanding of the situation and NVIDIA's optimization strategy. In the light of this, Futuremark now states that NVIDIA's driver design is an application specific optimization and not a cheat .

The world of 3D Graphics has changed dramatically with the latest generation of highly programmable GPUs. Much like the world of CPUs, each GPU has a different architecture and a unique optimal code path. For example, Futuremark's PCMark2002 has different CPU test compilations for AMD's AthlonXP and Intel's Pentium4 CPUs.

3DMark03 is designed as an un-optimized DirectX test and it provides performance comparisons accordingly. It does not contain manufacturer specific optimized code paths. Because all modifications that change the workload in 3DMark03 are forbidden, we were obliged to update the product to eliminate the effect of optimizations identified in different drivers so that 3DMark03 continued to produce comparable results.

However, recent developments in the graphics industry and game development suggest that a different approach for game performance benchmarking might be needed, where manufacturer-specific code path optimization is directly in the code source. Futuremark will consider whether this approach is needed in its future benchmarks.

NVIDIA Statement

NVIDIA works closely with developers to optimize games for GeForceFX. These optimizations (including shader optimizations) are the result of the co-development process. This is the approach NVIDIA would have preferred also for 3DMark03.

Joint NVIDIA-Futuremark Statement

Both NVIDIA and Futuremark want to define clear rules with the industry about how benchmarks should be developed and how they should be used. We believe that common rules will prevent these types of unfortunate situations moving forward

I'm not quite sure what to make of this issue., I don't find Nvidias Optimisations to be very legitimate aproach to tweaking a synthetic benchmark.

Then again, I don't see 3dmark anywhere close to being an actual representation of Nvidias performance in Future games.


I'll post a bigger comment on this later

[Gamer1]

according to HardOCP... nVidia was going to take legal action against 3dmark...

Translation VIA HardOCP: FutureMark reneges on previous statements and confirms NVIDIA was not cheating on their benchmark and NVIDIA will not take a legal action against FutureMark that would bankrupt them. All about the $, but that is just our opinion. Still this does not change our thoughts on the FutureMark and their benchmark. Editorial coming...

i guess 3dmark backed down and succumed to nVidia's muscle.

[kairi00]

Hard-coding clipping planes is hardly what I call a 'legit' optimization method. But then again, in a world obsessed with extra 5 fps or 100 more 3DMarks, I guess anything goes.

Originally posted by kairi00
Hard-coding clipping planes is hardly what I call a 'legit' optimization method. But then again, in a world obsessed with extra 5 fps or 100 more 3DMarks, I guess anything goes.

Well ATI is adding them too now :eek:


Anyway, I do have a huge post about this Issue, But atm I hurt too bad cuz my tooth pulled at the dentist

so it'll have to wait.


The Hard code clip planes aren't really the issue that bothers me. As they won't be needed with the future patch from futuremark supporting partial precision. Which "is" a part of DirectX standards

[kairi00]

Originally posted by ChrisRay



Well ATI is adding them too now :eek:

Wasn't ATI's boost in 3DMark scores in CAT3.4 due to re-ordering of vertex shader instructions, and not because of static clip planes? Or are they adding it too? :eek:

I'd rather take 30 more fps rather than 1000 more 3DMarks, thank you very much

Originally posted by kairi00


Wasn't ATI's boost in 3DMark scores in CAT3.4 due to re-ordering of vertex shader instructions, and not because of static clip planes? Or are they adding it too? :eek:

I'd rather take 30 more fps rather than 1000 more 3DMarks, thank you very much

This is from terry the head of the Catalyst Team

Originally posted by CATALYST maker
OK time to get in the fun.

ITS CM TIME!!!!

First of all as a favor to me, leave Kyle and Brent out of this. This has nothing to do with them. Period

Secondly I am a little upset tonight so I wont say much until tomorrow.

Third I guarantee you that I will ask for an investigation for optimized drivers tomorrow such that has never happened in ATI's history. I am prepared to put a hold on all new features I have in the pipeline so our top engineers can see how much we can optmize by not rendering the whole scene. I am guessing we can gain 25% at this point.

Fourth I am not commiting to do these optimizations ever in a released driver but I think its time for apples to apples comparison.

Fifth I am sorry to hear you (the end consumers) so dissapointed in the state of the industry. I feel for you.

Sixth, you all have my personal guarantee that if you continue to support ATI the way you have so far, I will always be here to help out and be one of the boyz on the forums. (I hope that means something to at least some of you)]

Have a good night everyone and lets talk more tomorrow
Terry

[Vahid]

mmmmmmmm betty.....yes....hehehhehehe......nviDia ownz...hehehhehehe...betty....from now on i am known as betty...hehheheheh

[kairi00]

Vahid: umm.... and the point of that post is...?

ChrisRay: ah.. I never read that before.. heh. Why can't they just spend the time to do proper Hidden Surface Removal (which I presume would be more generic and useful) rather than hard coding static clip planes? sigh

[Kane]

>ChrisRay: ah.. I never read that before.. heh. Why can't they just spend the time to do proper Hidden Surface Removal (which I presume would be more generic and useful) rather than hard coding static clip planes? sigh

Because this scene is rendered back to front, rather than the traditional front to back IIRC.

[D.D.]

>Because this scene is rendered back to front, rather than the traditional front to back IIRC.

That's weird...what's the point of that? Do any games render like that?

[Kane]

Not AFAIK.

[kairi00]

Which proves, yet again, the sheer uselessness of 3DMark as a benchmark tool.

[Badaro]

This whole thing is most likely a legal/pr move from Futuremark and NVidia, Futuremark re-words their statements in a way that doesn't look bad for NVidia, and they get back on NVidia's good side.

About ATI doing the same kind of "optimizations", there may be some hope yet, this was posted further below in the same thread:

Originally posted by TII
1st post here for this > that was a knee-jerk statement by CAT MAKER. He assured me (on DH) that ATI would lead the way honestly & ethically.

He's got work in the morning & I know a lot of the European community is just reading this, so I wanted to post what I know (since CAT is sleeping) to alleviate your concerns.

Of course, we can't be 100% sure of this until Terry posts again.

[]s Badaro

[Shadow Lady]

3dmark03 sux anyway

[Demigod]

Yet there's no better benchmark for DirectX 9. Other DX9 "benchmarks" perform nearly the same as 3DMark03.

[Aleksei (St)]

but it should test the system more generally like the cpu and other things on the machine other than the video card

[GiGaHeRz]

3DMark is designed to test the 3D Hardware, not the cpu and that other things.

[Noxious Ninja]

[i]ChrisRay: ah.. I never read that before.. heh. Why can't they just spend the time to do proper Hidden Surface Removal (which I presume would be more generic and useful) rather than hard coding static clip planes? sigh [/b]

Fourth I am not commiting to do these optimizations ever in a released driver but I think its time for apples to apples comparison.

They just want to have some fun and see if they can beat nVidia with their own cheats. Not actually put them in a public driver. At least, that's how I read it.

[Demigod]

Originally posted by Aleksei (St)
but it should test the system more generally like the cpu and other things on the machine other than the video card

Heh, it's called 3DMark for a reason, you know. It's designed to specifically test DirectX capabilities of 3D accelerators. If you want a benchmark for determining overall system performance then get PCMark. The only reason it's called a "gamer's benchmark" is because DirectX is used almost exclusively for games. Of course most games have vendor-specific optimizations or happen to be too CPU-bound so 3DMark cannot be anything of a representative of actual games, but if you were to develop an unbiased, pure DX benchmark what else would you do?

Nvidia's just a sour loser in my estimation. They just didn't like how the original GeForce FX performed on the benchmark and how well the Radeon did. But I bet they were very happy when 3DMark2K1 came out, as it heavily favoured DX8 cards (to which the GeForce 3 was the only one then).

[kairi00]

On a rather off-topic note, this is kinda cute