Benchmarks can be useful, whether we're talking mobile or any other computer hardware. There is a problem though -- you have to depend on everyone being honest. Let me explain. Anytime you see a video or a screenshot of a benchmark score, what you're really seeing is a measure of how the software interacts with the hardware. There are two big problems with that: No two Android phones run the same software, and there's always some joker who wants to cheat. recently our pal John from Phone Arena noted some Quadrant scores of a new handset that are just too low compared to the other phones listed. And good on John for pointing it out.
Sure, benchmarks can be fun, like see how little impact LG's UI has on the Optimus 2X compared to Stock Froyo, and they can be useful if you're running them yourself to gauge some changes you're making to your own software. But we're over it. We're going to base our judgment of performance based on performance, and not numbers.
It's just too damn easy to cheat at all benchmarks, because they are just Android apps. Look at all the other magic folks do with Android apps, and ask yourself -- are benchmarks so different that they can't be "hacked"? The answer is no, and we're going to show you how to hack at the easiest (but definitely not the only) one to manipulate, Quadrant.
Check it out after the jump.
Let's have a little look at exactly what we're about to do here, through the magic of video.
YouTube link for mobile viewing
You're going to need to be rooted for this example. There are plenty of other ways to cheat on unrooted phones by editing the code and rebuilding the app itself, but this easy way needs root. You're also going to need a few tools:
- Have adb working, with root access in the shell.
- Quadrant Standard from the Android Market. We used Quadrant Advanced in our demo, because it's so easy to see the cheating. You'll want to use Standard, because it's so easy to see the cheating.
- System Panel (or any app that can kill all processes) from the Android Market.
- SetCPU from the Android Market, or equivalent (if you have an overclocked kernel).
- Two hands, a keyboard, and no conscience.
Install all your apps and reboot your phone. Let it sit for 5 to 10minutes so all the syncing and scanning gets finished, then run Quadrant (see the "before" side in the image above) so you know the truth.
Hook your phone up to your computer, and enter the shell. As root, run the following command:
mount -t tmpfs tmpfs /data/data/com.aurorasoftworks.quadrant.ui.standard
Enter it exactly as written. What that's doing is simple -- the I/O portions of all phones are the biggest bottleneck, so we're moving the Quadrant data folder off the physical storage, and into the RAM. RAM is good. Reading and writing to it is fast. Super fast. This goes away at reboot, so it's easy to take things back to normal.
Next, we want to cancel out any CPU scaling effect. Who needs to save battery, we want high scores! Fire up SetCPU and crank the CPU clock speed as high as it will go, and choose "performance" as the CPU governor. This will keep things running at top speed all the time.
Now use whichever app you decided on (I use System Panel to monitor things, and it includes the dreaded kill-all as an extra function) to kill all your background tasks. If you can kill system processes, even better. We don't care how well the phone works, only how high the numbers are.
Fire up Quadrant, and run your benchmark. Run it two or three times to get it settled in, then start taking screenshots. Realize just how easy it was to do this, then imagine what other cheats can be done by reverse engineering the benchmark apps and changing the math used to write the scores. Add in a few other tricks like changing a line or two in the build.prop file to inflate the CPU scores, and you're golden. Before someone says "Yeah, but I use Bechmark xyz9000 Pro and you can't cheat on that one." Yes, you can. Even I can, and there's plenty of people out there way more talented than I am. Only believe benchmarks that use lots of meters and wires, with flashing lights, because anything else is just software that someone can hack.
Now the playing field is level. Congratulations, you're a cheater.
