Web App Performance Benchmarks: Share the Conditions Behind the Speedup

When sharing web app performance benchmarks, publish the test conditions beside the speed claim. Identify the two builds, the task, the metric, the device and network settings, the starting state and the repeated results. A screenshot of a better score alone does not let another developer assess what improved.

For a founder building in public, a benchmark note can be a useful release artifact. It gives peers something specific to question and helps prospective users understand which part of their task might feel different. Keep the claim no broader than the measurement.

This walkthrough designs a repeatable comparison for a fictional reporting SaaS. It does not report tests of Buildside, Uriel's products or any real customer workload. All example timings below are invented to show how to reason about a claim.

Start with the task and the finish point

“Made the app faster” leaves the reader to invent the meaning of faster. Opening a dashboard, applying a filter and exporting a report are different tasks. A change may improve one while leaving the others unchanged.

For the fictional comparison, choose opening a saved report containing a fixed set of sample rows. Define the starting action and the finish point. For example, the browser starts from a signed-in workspace page, and the task ends when the report's rows and summary are visible and usable.

If you use a custom duration, explain exactly how it was measured. A manual stopwatch, a browser trace and an instrumented application event do not automatically measure the same boundary. Do not rename your custom measurement as a standard web metric.

Keep correctness in the finish condition. A page that renders quickly but omits half the rows has not completed the same work. Record the expected row count and the visible result before collecting timings.

Name the builds that a peer can compare

Record a version or commit reference for each build and the environment where it ran. Explain whether the test used a production build, a development server or another setup. Preserve the relevant configuration with the result.

A local build of the new version and a remote production page of the old version introduce several differences at once. You may learn something from that comparison, but you cannot confidently assign the entire difference to one code change.

For the fictional SaaS, run both production builds against the same disposable sample workspace under the same hosting arrangement. Keep the report input and account role fixed. If the backend changed too, record that as part of the tested change.

Use public references only for material you are authorized to disclose. A useful note can describe the environment without exposing private repository paths, credentials, customer data or detailed infrastructure information that the comparison does not need.

Record the conditions before measuring

Write down the browser version, device, operating system, network treatment and relevant browser extensions. Note whether a throttling setting simulates a slower network or CPU. Do not describe a simulated condition as a measurement on an actual low-end phone.

Chrome's Lighthouse scoring documentation explains that performance values can vary with underlying conditions, including device differences, traffic routing and browser extensions. It also distinguishes the weighted score from the underlying metric values.

That is why a public note should retain raw measurements where they help explain the result. A movement in an overall score does not directly tell a reader how long their report will take to open.

Pick conditions relevant to your intended users, then state the limit. A desktop test on a stable connection can be a useful development comparison. It does not establish how the app performs for every mobile user, location or dataset.

Separate first visits from warmed state

A repeated attempt may begin with resources or data already available from an earlier attempt. Decide which starting condition the test needs and describe how you restored it. “Cold cache” is too vague if the browser cache was cleared but the server still retained prepared data.

List the state you control: browser cache, application storage, sign-in session, service worker behavior and any relevant backend preparation. Do not claim that every layer was reset unless you actually reset and checked it.

For the fictional saved-report task, keep two separate comparisons. One begins from the defined first-load state. The other begins after the specified warm-up action. Compare the old and new builds within the same starting condition.

This prevents a tempting mistake: presenting the old build's first attempt beside the new build's warmed attempt and calling the difference an optimization. If a useful improvement applies only to returning visits, say that plainly and show that case.

Repeat the work without selecting a favorite run

Choose a practical number of repeated attempts before viewing the result. Preserve the sequence and explain any run excluded because the procedure failed. Do not silently discard an ordinary slow result because it makes the release look less impressive.

Five repetitions can be a small rehearsal for finding obvious variability; it is not a universal sample-size rule or proof of statistical certainty. A changing environment or a high-stakes performance claim may need a more developed method.

Suppose the fictional old build takes 980, 1,020, 1,000, 990 and 1,010 milliseconds for the defined task. The fictional new build takes 790, 820, 800, 810 and 780 milliseconds. The middle values are 1,000 and 800 milliseconds.

The difference between those middle values is 200 milliseconds, or 20% of the old value. That arithmetic describes this invented controlled comparison. It does not show that the product as a whole is 20% faster or that every user saves the same amount of time.

Keep lab evidence separate from real-user experience

A controlled test is useful for investigating a change under stated conditions. Real-user measurements answer another question: what happened across actual visits with different devices, connections and behavior?

web.dev's explanation of lab and field data makes this distinction explicit. A controlled environment limits variables, while field measurements include the range of conditions and behavior encountered by actual visitors.

If you have only the controlled comparison, publish it as such. Do not borrow the language of real-user monitoring or imply that a laboratory improvement has already reached every customer. Even a shipped build needs separate evidence about actual use.

If you later collect field data, record its period, population and relevant exclusions. Keep that follow-up connected to the original benchmark note while allowing the result to disagree. A useful public record can show a promising lab change that had limited effect on real tasks.

Publish a benchmark note people can challenge

A compact note can name the task, the two versions, the controlled conditions, the repeated measurements and the practical limit. Link to a shareable method or artifact when you can do so without exposing private information.

For the fictional release, a fair headline would identify the saved-report task and the controlled comparison. The body would retain the two sets of timings and explain the first-load or warmed state. It would also say which correctness checks passed.

Then ask a focused question. A peer might review whether the finish point captures usable work, whether the cache reset is sufficient or whether the test dataset resembles the intended task. “Is this fast?” gives them much less to assess.

Buildside is a community for founders sharing work in public. A benchmark method can give a founder conversation substance there or elsewhere. Community approval is still different from a reproduced result, and a reproduced lab result remains narrower than a promise to every user.

Before posting the speedup, hand the method to another developer. If they cannot tell which build, state and finish point produced the number, improve the note before polishing the chart.

Hey I'm Uriel Bitton. I write about building in public strategies and growing startups.

Subscribe for more stories on growing your audience by building in public.

Join us on Buildside: the social network for founders building in public.

Sources

• Chrome for Developers: Lighthouse performance scoring
• web.dev: Why lab and field data can be different
• Buildside founder community