Dev.to WebDev šŸ›  Dev šŸ‘ 0 šŸ“– 4 min read

Estimating Sample Size Before Testing a Law Firm Contact Form

Law firm clients ask for A/B tests all the time. Shorten the contact form, change the button text, move the phone number. The request is reasonable, but on a typical criminal defense site the traffic to a single page is

Law firm clients ask for A/B tests all the time. Shorten the contact form, change the button text, move the phone number. The request is reasonable, but on a typical criminal defense site the traffic to a single page is small, and the number of people who actually submit a form is smaller still. Run a test without checking the math first and you will likely end up calling a winner based on noise.

A few minutes with a sample size estimate before the test starts saves weeks of pointless testing and an awkward conversation later. This post covers the inputs you need, a short script to do the arithmetic, and what to do when the answer says the test is not practical.

The inputs you need

A standard sample size estimate for comparing two conversion rates needs four numbers:

  • the baseline conversion rate of the page as it is now
  • the smallest lift you care about detecting, called the minimum detectable effect
  • the significance level, usually 0.05
  • the statistical power, usually 0.8

Get the baseline from real data. For a contact form, that is form submissions divided by unique visitors to the page over a recent, normal period. If you are not sure the form events are tracked correctly, fix that first. A test built on broken tracking is worse than no test.

Choose the minimum detectable effect based on what would matter to the firm. A change from 3 percent to 3.1 percent is not worth anyone's time to detect. A change from 3 percent to 4 percent might be.

A short script for the estimate

The formula for two proportions is short enough to keep in a utility file. This version uses only the Python standard library:

from statistics import NormalDist

def visitors_per_variant(p1, p2, alpha=0.05, power=0.8):
    z_a = NormalDist().inv_cdf(1 - alpha / 2)
    z_b = NormalDist().inv_cdf(power)
    variance = p1 * (1 - p1) + p2 * (1 - p2)
    return pow(z_a + z_b, 2) * variance / pow(p1 - p2, 2)

print(round(visitors_per_variant(0.03, 0.04)))  # about 5,300

With a hypothetical baseline of 3 percent and a target of 4 percent, the estimate comes out near 5,300 visitors per variant, or roughly 10,600 in total. A bigger expected lift, from 3 percent to 6 percent, needs only about 750 per variant. Small differences are expensive to detect.

Online calculators give similar numbers. The value of the script is that you can drop it into a planning notebook and run it for every page you are asked to test.

Turn visitors into weeks

Divide the total by the page's weekly unique visitors. If a charge page gets 400 visitors a week and needs 10,600, the test would run for more than six months. In that time the firm will change other things on the site, search rankings will move and seasonal patterns in arrests will shift traffic. The result would be hard to trust.

A useful rule: if the estimate runs past eight to ten weeks, the test as designed is not practical for that page.

Options when the numbers do not work

You still have choices when traffic is low.

  1. Test across a template. If every charge page uses the same form component, run the variant on all of them and pool the traffic. Make sure the change is the same everywhere.
  2. Test a bigger change. A redesign of the whole contact block has a better chance of producing a large effect than a single word swap.
  3. Measure a higher-volume step. Clicks on the call button or the start of form entry happen more often than submissions. Treat them as supporting evidence, though, because what the firm wants is cases.
  4. Use a before and after comparison with care. Match periods, note outside events and accept that the evidence is weaker.

For the non-technical side of that last option, a write-up on running a conversion test on a low-traffic defense website covers how to plan and read a test when traffic is thin.

Do not forget the phone

On criminal defense sites, many visitors call instead of filling in a form, especially on mobile. If a form change pushes people toward the phone, form submissions fall while total contacts rise. Track calls with a dynamic number or call tracking tool that can attribute calls to the variant, and judge the test on total contacts. Otherwise you can reject a change that actually helped.

Write the plan down before launch

Record the baseline, the minimum detectable effect, the sample size, the expected end date and the decision rule in a short document before the test starts. Agree with the firm that you will not stop early because one variant looks ahead after a week. Peeking and stopping early is one of the most common ways small tests produce false winners.

When the test ends, report the result with its uncertainty alongside the percentage lift. A firm owner can work with "probably no meaningful difference" as long as they knew in advance that this was a possible outcome.

More on testing and measurement for law firm websites is gathered in the library on conversion testing for law firm websites.

Questions about sample size for law firm tests

Can I use a smaller significance level to finish faster?

Raising alpha above 0.05 shortens tests but makes false positives more likely. Change it only if the firm understands the tradeoff.

Is a Bayesian testing tool a way around sample size?

Bayesian methods report results differently, but they do not create information from small samples. Low traffic still means uncertain results.

Should I count unique visitors or sessions?

Use whichever unit you randomize on, usually users. Keep it consistent for the baseline and the test.

SBD Marketing works only with criminal defense law firms, one firm per city. It builds websites for defense practices, handles SEO for the charges a firm defends, works on visibility in AI search tools and runs SEO audits.

šŸ“° Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.