Are NVIDIA PhysX Drivers Cheating on 3DMark Vantage?

Is Running Physics On The GPU Cheating?

Ever since we published the article on the NVIDIA GeForce 9800 GTX+ graphics cards last week Legit Reviews has been getting a ton of e-mail complaining about NVIDIA’s paper launch spoiler and the fact that the drivers we used for are not fair. As the days followed the e-mails kept pouring and became more focused on one test in particular – 3DMark Vantage. Some good friends of ours at various companies also tipped us off that something wasn’t quite right and that we need to start digging deep to see what we could come up with. We couldn’t get any responses over the weekend and those willing to speak about the subject wouldn’t do so on the record. We’ve managed to talk to AMD, NVIDIA and Futuremark over the past week and have figured out some interesting pieces of information.

The Inquirer posted something up about driver cheating this week and that got the industry buzzing. The Inq claimed that NVIDIA was using in-house PhysX API’s and that they were able to manipulate the score in 3DMark Vantage since they can make the graphics card, drivers and physics API that is used for the benchmark. Our test scores showed the significant performance increase that everyone was up in arms about, but from what we could tell it was just off-loading the workload from the CPU to the GPU. The new NVIDIA drivers allow GPUs to do what once took a dedicated AGEIA PhysX card! The days of running a PPU and a GPU in a system are soon to be long gone!

AMD and a number of other companies are worried that running the CPU physics test on the GPU will sku results and made the point that GPU physics testing is already done in the normal GPU tests. 3DMark does currently includes physics as simulation collision map rendering in the game tests, so their arguement is to leave things be.

We though it would be interesting to get comments from NVIDIA and Futuremark on the cheating claims that The Inq. brought to light, so we got official on the record answers from both companies. First, we got in touch with Oliver Baltuch, the President of Futuremark and asked him to answer a few questions. We figured that it would be best to go straight to the source of the benchmark to see if we could figure things out.

Legit Reviews: Did NVIDIA optimize their Forceware 177.39 drivers for 3DMark Vantage in an illegal way? A recent story on The Inq. claims that NVIDIA usues a different workload on the benchmark with the new drivers.

Oliver Baltuch: The driver in question has not been submitted for authorization and is only for demo purposes only. NVIDIA has followed the correct rules for driver authorization and the BDP by sending us the 177.35 published driver (the same as AMD has now sent us the 8.6 published driver), both of which are currently undergoing the Authorization process in our Quality Assurance area at this moment. Only drivers that have passed WHQL and our driver Authorization Process have comparable results that will be allowed for use in our ORB database and hall of fame. Other drivers which have not been submitted will not be commented on. Otherwise, we would have to inspect every Beta and press driver that is released.

Legit Reviews: Rumor has it that an OEM has asked for the rules to be ratified to include GPU PhysX for 3DMark Vantage testing. Is this true? If so, how does this work? Could someone suggest a DirectX 10.1 addition if they wanted to bring it to a vote for the benchmark?

Oliver Baltuch:As usual I cant really comment on any rumors, but, our Benchmark Development Process does have a transparency system that allows members to request changes to the specification. Those change requests are then seen by all the other members whose written opinions are taken into account by our technology committee who then decide whether to make the change to the specification. Outside of this matter, we have been introduced to this technology from NVIDIA and it is truly innovative for future games.

Legit Reviews: Now that NVIDIA has enabled PhysX on the GPU, what will this mean in the benchmarking world? Will there be such a thing as apple to apple video card comparisons with one company using HAVOK and the other using PhysX for handling Physics?

Oliver Baltuch: ** Oliver Declined To Comment**

Oliver Baltuch returned a written statement answering our individual questions, but did not comment on everything he was asked about. He let us know that the specifications for 3DMark Vantage could be changed, but could not comment on the rumors going around about the rule change. The final question we had for Oliver was about the future of PhysX and benchmarking and again Oliver didn’t want to comment. Basically, we found out that the NVIDA ForceWare 177.39 drivers have not been submitted to become validated for Futuremark use and that the rules could be changed if needed down the road. Oliver Baltuch was careful with what he said and rightly so, his company gets funding from both AMD and NVIDIA!

After reaching a dead end with Futuremark we asked for an official statement from NVIDIA and this is what was said.

“The initial version of 3DMark Vantage shipped with a PhysX installer that only included support for CPU physics and the Ageia PPU. The initial version did not contain a GPU physics installer as nobody had that product. Once NVIDIA brought physics to the GPU through our Ageia acquisition, we updated our PhysX installer to add support for GPU physics. The same thing applies to Unreal Tournament 3. All weve done is update the installer to add support for GPU physics. The installer will be available for everyone to download off the NVIDIA.com website this week so folks can play with it and enjoy the new GPU physics effects.” – Ken Brown (PR Manager at NVIDIA Corp)

We also spoke with Brian Burke (Senior PR Manager) and Roy Taylor (Vice President of Developer Relations) on the phone and they had a ton of things to say about PhysX and were obviously all for it. They were happy to announce that 14 to 16 new PhysX game titles would be available before Christmas and double that next summer. NVIDIA purchased AGEIA Technologies just a couple weeks before 3DMark Vantage was released as they saw that the computer industry was headed towards a heterogeneous computing model, combining a flexible CPU and a massively parallel processor like the GPU to perform computationally intensive applications like real-time computer graphics. By releasing the new Forceware 177.39 drivers along with the with PhysX driver 8.06.12., NVIDIA has shown that GPU PhysX is ready for prime time. They informed us that a public driver release that will happen this week, so that everyone with a supported card will be able to try GPU PhysX out. This public driver will feature a single driver installer, which will make installation quick and easy. Brian Burke explained to us that the PhysX-based physics calculations that are used in the CPU tests on 3DMark Vantage have been to this point sent directly to the CPU to be handled. Now with the new beta GPU PhysX drivers the benchmark DLLs go directly to the GPU instead of the PPU or CPU. Since GPUs are five to twenty times faster than CPUs in physics calculations the benchmark score increases.

The reason NVIDIA is upset about the cheating claims is because they didn’t do anything other than allow the workload to run on the GPU. The way 3DMark Vantage runs the benchmark remains the same as if a dedicated PPU was installed (adding more worlds to the workload). With parallel computing headed to the GPU it only makes sense that benchmarks take this into account. The problem here it seems is that NVIDIA came out with the drivers before the industry had the benchmarks ready. When Futuremark Vantage was designed the AGEIA PhysX SDK was used and a discrete Physical Processing Unit was needed to offload the benchmark DLLs from the CPU to the PPU. We showed this in our article about 3DMark Vantage the day the benchmark came out. We dedicated an entire page to offloading PhysX to a PPU and no one cried that it was cheating moving the DLLs from the CPU to the PPU. Now that NVIDIA had moved the workloads from the CPU to the GPU they are accused of cheating!

NVIDIA is not using driver optimizations to cheat on benchmarks, they are just doing what someone with a PhysX card could do months ago. This is an evolutionary step in the technology industry and it is also a sneak peak at what the future holds and that is physics on the GPU. Other graphics cards companies (AMD, Intel and S3) are worried about the performance boost that is shown in 3DMark Vantage and rightly so, Futuremark Vantage is a very popular benchmark among OEMs and consumers. NVIDIA won’t be the only ones with physics being run on the GPU, but they are clearly raising the bar and being the innovator. In closing NVIDIA’s Roy Taylor invited AMD’s Richard Huddy to call him anytime to discuss ATI adopting CUDA and PhysX for Radeon graphics cards.

It should be noted that AMD declined to make an official comment for this article as they are having talks with Futuremark later tonight. We will update this story tomorrow with the official AMD comment.