Android-Benchmarks-cheaters

When Apple SVP Phil Schiller pointed us to a story earlier this week that Samsung was artificially inflating benchmark scores for its new Galaxy Note 3, many were quick to point out it wasn’t the first time Samsung had been caught engaged in such a practice. The same issue was discovered by AnandTech for the Galaxy S4 back in July, and today the site has an extensive report showing that almost every Android smartphone manufacturer is shipping devices that do the same.

As pictured in the chart above, that includes the HTC One, HTC One mini, LG G2, Galaxy Tab 10.1, and many others. In fact, the only companies that appear to not be using the method is Apple and Motorola, as well as Google with its Nexus 4 and Nexus 7 devices:

We started piecing this data together back in July, and even had conversations with both silicon vendors and OEMs about getting it to stop. With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung…  None of the Nexus do, which is understandable since the optimization isn’t a part of AOSP. This also helps explain why the Nexus 4 performed so slowly when we reviewed it – this mess was going on back then and Google didn’t partake.

As noted in the report, the gains that OEMs are experiencing from the inflated scores are probably not worth the press they’ve been receiving. AnandTech points out that most of the inflated scores provide under a 10% increase in GPU and CPU performance benchmarks:

The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 – 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell. I can’t stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors.

You can check out the full report here, which offers in-depth analysis on the optimizations it found for several devices across various benchmark tests.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

7 Responses to “Report finds almost all Android OEMs, not just Samsung, cheat on benchmarks”

  1. 311sie says:

    Shocking… just shocking.

    Like

  2. Won’t someone please think of the children!?

    Like

  3. Hamza Sheikh says:

    This is really shocking for everyone in smartphone industry. People may have lost their trust over Android OEMs after this post going viral on the internet. I have seen many Apple trolls getting silent on this.

    Like

  4. I love my iPad. I saw that at least 5 times a week. The first year I said it several times a day! It’s all about the fluidity and speed of iOS, not necessarily the chip.

    Like

  5. i2fun says:

    Phil “the shill” Shiller? lol… just being true to his very nature. I bet he was the kid in his grade school classes that was always telling on others for doing what he’d just done and got away with. Then getting patted on the back by the teachers, before leaving campus and getting beat to a pulp for being a RAT! …..and now he still looks like one too!

    Like