Unless you have been out of the loop recently, you probably have heard of a new rise in the battle between nVidia graphics drivers and their use with 3DMark03. Let me give you the brief story:
On January 19th, Martin Luther King Jr. Day, I received an email from FutureMark asking that I kindly remove my 3DMark03 scores because I used a driver that was not listed on their "Approved" Drivers List. More information to follow on the "Approved" Drivers List. Nate and I did some quick scrambling to find out why FutureMark would ask such a question and what grounds they had to do so.
The first place we looked: the EULA for 3DMark03.
The End-User License Agreement:
When thinking about the situation legally, we decided to first look at the End-User License Agreement (EULA) to see if we had any leg to stand on if we decided to refuse. Low and behold, the second paragraph in the EULA states the following:
"PLEASE NOTE that Section 11 of this Agreement contains terms, which restrict Your right to publish any Results obtained by You using drivers that do not fulfil Futuremark Optimization Guidelines requirements. Updated Optimization Guidelines and list of drivers verified by Futuremark can be found on Futuremark’s website at www.futuremark.com."
Since everyone who uses 3DMark03 agrees to this EULA, we really had no way to refuse their request. So five minutes later, FutureMark's request was completed, but our quest to find out why had just begun. To conclude our findings within the EULA, the following is information pertaining to the above statement from Section 11:
"Notwithstanding foregoing, You are allowed to publish only such Results that are obtained using drivers that fulfil requirements of Futuremark Optimization Guidelines. Updated Optimization Guidelines and list of drivers verified by Futuremark can be found on Futuremark’s website at www.futuremark.com."
The "Approved" Drivers List:
In case you never pay attention to the EULA and never heard of FutureMark's Optimization Guidelines, here is the link to FutureMark's "Approved" Drivers List. Here is the description of this page from FutureMark's website:
"Here we have listed all WHQL drivers which are approved to be used only with Build 340 of 3DMark®03. If you haven't got Build 340 of 3DMark03, you can download the patch here. By using these drivers & Build 340 you will get a valid and fully comparable 3DMark03 result. We will update the driver information continuously as new WHQL certified drivers are reviewed. Please note that the approved drivers listed below have been tested only with the 3DMark03 default benchmark settings!"
Another interesting point is the "note" given after showing that the nVidia 52.16's are the only Approved Driver for use.
"The NVIDIA ForceWare 52.16 Drivers have been tested with the GeForce3 series, GeForce4 series and the GeForceFX series. The drivers have not been tested with the MX-series! There are no Win9x or WinME WHQL drivers available supporting all the GeForce series. The 52.16 drivers have 3DMark03 specific optimization for the Pixel Shader 2.0 test and that score is solely comparable between nvidia cards."
So, FutureMark is actually informing everyone that you can use these drivers for an accurate 3DMark Score, but not when comparing tests using just the Pixel Shader 2.0 test. According to a response from FutureMark given in Adrian's editorial on the matter, the usage of Pixel Shader 2.0 in the Nature game test is not affected by this known optimization in the 52.16 driver.
Since it seems that FutureMark is legitimately attempting to keep their benchmark fair and unbiased, let's next look towards nVidia's drivers to find out why the 52.16's are approved and the 53.03's are not.
nVidia's Forceware 52.16:
What we know about nVidia's 52.16 drivers is that the build 340 patch for FutureMark's 3DMark03 disabled the cheats present in this driver. This is just like when FutureMark released the build 330 patch to disable cheats found in nVidia's 44.03 driver. So build 340 is FutureMark's second attempt to keep their product's results consistent and optimization-free.
Since build 340 was released, no other drivers from nVidia have been approved. Granted, only the 53.03's have been released since the 52.16's, but this means that nVidia has already figured out how to implement their optimizations in build 340 of 3DMark2003 in their drivers.
Now let's look into why the most recent nVidia Driver has not been approved.
nVidia's Forceware 53.03:
The Forceware 53.03 driver is currently nVidia's most recent WHQL certified driver. This driver fixes a list of known bugs that overall improve the customer's experience with an nVidia graphics card. A list of those fixes can be found in the Release Notes for the 53.03 driver.
Now what I thought was interesting in this document is that under the heading "Issues Resolved in Version 53.03," nVidia states very clearly the following:
"Re-enabled NVIDIA's compiler technology for 3DMark03 Patch 340 performance."
Well, there is not much to argue about here. nVidia clearly states that they re-enabled their optimized code for 3DMark03 for build 340. This is why FutureMark's 3DMark03 is having so much trouble giving valid results. This is just a repeat of what happened right after build 330 was released.
So what should FutureMark do next? I will touch on that in my conclusion.
But next, let's see what, exactly, the difference is between nVidia's 52.16 and 53.03 drivers. In this recent revival of the FutureMark vs. nVidia debate, numerous users have posted results showing no difference between the 52.16 and 53.03 drivers. Our results will help explain why.
- AMD Athlon FX-51
- ASUS SK8N
- Kingston PC-3200 Registered Memory
Video Cards Tested:
- Chaintech GeForce FX 5600 Ultra - 128mb
- nVidia Reference 5900 Ultra - 256mb
On the graph you will see that Vertical Sync was also tested. We included the test with Vertical Sync on in order to lay to rest the theory that V-Sync was one of the reasons for noticing a performance difference.
5600 Ultra Results
As I mentioned on the previous page, we have noticed no performance difference on our 5600 Ultra between the 52.16 and 53.03 drivers. Take note that V-Sync had no effect on the results. When Adrian from Adrian's Rojak Pot emailed FutureMark about this, Adrian received this in the reply:
"...with some display cards [the 53.03 display driver] does not fulfill the run rules. Please note that drivers may behave differently according to which display card they are used for."
So it seems that the 5600 Ultra is not one of the cards that use the 3DMark03 optimizations nVidia has included in the 53.03 display driver. What about a 5900 Ultra?
5900 Ultra Results
Now we see the reason FutureMark asked us to remove our results using the 53.03 driver. The 5900 Ultra is a card which uses nVidia's 3DMark03 optimizations and in return causes invalid results.
With some further research on the net, I have seen cases where GFFX 5200's do not show a difference in 3DMark03 scores, and I have also seen cases on the net where GFFX 5950's DO exhibit a difference in 3DMark03 scores. This observation leads me to believe that nVidia is purposely making their top of the line graphics cards look better than they really are in this benchmark. So not only is nVidia writing optimized code for 3DMark, but they are only doing it for certain graphics cards!
On to the conclusion where I will try to tie up all the loose ends!