PerfMarks

Benchmarking HTML5 Rendering Performance

Why did we make PerfMarks?

• Spaceport: Develop games with Adobe tools, publish to cross-platform mobile apps

• Native apps AND HTML5 web-apps• What should we use?– CSS2D, CSS3D, WebGL, canvas?

• Rendering Performance matters• Which is best for mobile browsers• Benchmark it

Methodology

• HTML5 games: – Translation– Scaling– Rotation

• How many images can be moved around while maintaining 30 FPS?

• Easy to measure... right?

Accurate Benchmarking is hard

• Lots of complications– JIT– latency– memory– screen updates

• Getting data takes a long time• Results require scrutiny

Try #1: Draw 100 frames – time itvar testStart = Date.now();var maxDraws = 100;var numDraws = 0;

function next() {    for (var i = 0; i < numObjects; ++i) {        renderObject(i);    }

    ++numDraws;    if (numDraws >= maxDraws) {        var testEnd = Date.now();        done(testEnd - testStart);    } else {        setTimeout(next, 0);    }}next();

Expected Results

0 20 40 60 80 100 1200

20

40

60

80

100

120

140

160

180

Frame Render Time

Object Count

mill

iseco

nds

Actual Results

0 20 40 60 80 100 1200

5

10

15

20

25

30

35

40

Frame Render Time

Object Count

mill

iseco

nds

JIT

• Just in time engine kicks in for later runs of the test – making benchmark results depend on the order the tests are run.

• Invalid results.• Solution: warm up the JIT by running the tests

twice – take data from second run.

Next Issue

var img = new Image();img.onload = function onloaded() {    // img is loaded!

    var copy = img.cloneNode(true);    // is copy loaded?};img.src = "foo.png";

DERP

Image Loading Latency

• In certain browsers image not loaded– Android Browser– Mobile Firefox– Opera– Desktop Safari

• Absurdly high benchmark scores• Solution: Add more onLoad() checks – TONS of

redundant onLoad() checks...

How many objects to run?

• First pass was [1, 3, 5, 10, 30, 100]• Next pass was add 25 until it's no longer

making 30 FPS – then increment by a smaller step size.

• Problems:

Real world data is fuzzy!

Real world data is large!

• Counting to 7000 by 25 takes a long time• Especially when you are testing 3 mobile

browsers on a dozen Android phones

A Better Algorithm

• Run Test – estimate time/object/frame• Use estimate to predict the number of objects

that will run at exactly 30 FPS• Repeat until predicted number of objects has

already been tested• Ensure that there are also data-points above

and below

Next problem - Memory

1. Generate Test Data2. Run Tests3. Destroy Test Data

• This kills the Garbage Collector• This will hard-crash most mobile browsers

A Better Approach

• Cache test data• When more data is requested than has been

generated – add on to the cached values.• Clear out all test data when the test is

completely finished.

Does the screen update?var testStart = Date.now();var numDraws = 0;

function next() {    setTimeout(next, 0);

    for (var i = 0; i < numObjects; ++i) {        renderObject(i);    }

    ++numDraws;    var frameEnd = Date.now();

    var fpsEstimate = 1000 * numDraws /        (frameEnd - testStart);}next();

Asynchronous Screen Updates

"requestAnimationFrame"

• "requestAnimationFrame" only happens in modern browsers if it's actually rendering to the screen.

• This let's us count the number of times the screen was actually updated during a test run

More Accurate Frame Countingfunction next() {    requestAnimationFrame(next, element);

    for (var i = 0; i < numObjects; ++i) {        renderObject(i);    }

    ++numDraws;    var frameEnd = Date.now();

    var fpsEstimate = 1000 * numDraws /        (frameEnd - testStart);}

requestAnimationFrame(next, element);

Results

• Modern Smartphone vs Modern Laptop• Macbook Pro vs iPhone 4S– iOS 5.1– Mobile Safari vs Safari 5.1 (Webkit)

• Macbook Pro vs Samsung Galaxy S2– Android 4.0.3– Mobile Chrome Beta vs Chrome 19

Safari

Chrome

PerfMarks

• http://spaceport.io/community/perfmarks• https://github.com/sibblingz/PerfMarks