ARM Versus Intel: Will Revised AnTuTu Benchmark Change Things?

This week there has been a debate about a popular benchmark for smartphones that could impact the battle between Intel and ARM. Things heated up on Wednesday when EE Times posted a story that basically questioned the validity of the widely-used AnTuTu Benchmark. We have used AnTuTu benchmark for smartphone reviews since 2012, so we started to pay attention to see what was going on. Many sites use the AnTuTu Benchmark to show which processor is fastest, so companies like Intel, ARM, NVIDIA, Qualcomm, Samsung and others want to be on top. It appears that some companies might be willing to pay or cheat their way to the top.

The EE Times story was written by analyst Jim McGregor of Tirias Research, who is a credible analyst that we have personally met and known for years. He recently noticed some discrepancies in a new version of the AnTuTu benchmark that made him to a double take. For years we have seen companies cheat on benchmarks when it comes to computers, so are companies cheating or making sure benchmarks are enhanced for their technology? After some digging he found that the ICC compiler introduced around version 2.9.4 of the AnTuTu benchmark was used just for the Intel processors. It appears that AnTuTu was aware that they were ‘busted’ and issued revision 3.2.2 to the benchmark Wednesday evening. Mr. McGregor went back and benchmarked AnTutu v3.3.1 versus v3.3.2 and the differences are rather alarming and it was clear that the benchmark favored Intel processors more than other companies.

His testing showed that on his Lenovo K900 smartphone and the Intel Atom Z2580 processor that his AnTuTu CPU and overall scores dropped by approximately 20 percent, while the AnTuTu RAM score plummeted by approximately 50 percent. It shouldn’t come as a shock that Samsung S4 Oct and Qualcomm Snapdragon 600, both used in the Samsung Galaxy S4, remain relatively unchanged. Ouch!! Was Intel trying to put one over on ARM? The release notes of the latest build of AnTuTu Benchmark show that changes were made that ‘Enhances the stability of scores’.

To make matters worse, we ran across this post on Berkeley Design Technology’s (BDTI) Web site found that “the ARM-based [Samsung] Exynos processor performs all the operations specified in the benchmark source code, while the Intel Z2580 processor skips some steps.” Guess what benchmark this issue was found on? The AnTuTu Benchmark once again. Double Ouch!

We all know that cheating on benchmarks has been a big deal when it comes to computers and it appears that trend will continue on the mobile side. It looks like we need better smartphone benchmarks that aren’t bought off and offer good honest real world tests!