Did that change work? Judging before and after without fooling yourself

Every owner has done this. You change something (a new budget, a new service page, someone answering the phone on Friday afternoons) and a few weeks later the numbers look better. The change worked. Or did it?

Sometimes it did. But numbers move for many reasons, and most of them have nothing to do with you. Here's a method for telling the difference that needs nothing more than a notebook and some discipline.

Write down the change before you look

Record three things the day you make a change: what you changed, the date, and what you expect to happen. "Raised the ad budget from X to Y on 4 March. Expect more leads, cost per booked job roughly flat."

Writing it down first matters for two reasons. Memory is unreliable about dates, and a change made "around the start of the month" could have been a week either side. And writing down your expectation before the results arrive stops you from deciding afterward that whatever happened is what you were hoping for.

Keep the method identical

If you count leads one way in March and another way in April, the number changes even if your business didn't. This happens more often than people think:

• Counting conversion events one month and distinct people the next. In our public demo account, a fictional roofing company with synthetic figures, the same month reads $39 per lead or $62 per person depending on which you divide by. We explain why in Google says $39. The real number is $62.
• Asking AI assistants a different set of questions this month than last, or using different providers.
• Changing which competitors you compare against, which moves share of voice on its own. See Share of voice depends on the comparison set.

The rule is simple: same questions, same providers, same definitions, same comparison set. If you must change the method, note the date and treat the before and after as two separate series.

Compare periods that are actually comparable

"Last 30 days versus the 30 days before" is a fair default, but check what's inside each window.

• Same length. A 31-day month against a 28-day month is a built-in difference of about 10%.
• Same mix of days. If weekdays bring most of your leads, a period with an extra weekend or a holiday will look weaker for no reason connected to your change.
• A clean gap. Leave out the days right around the change if it took time to settle in, and say you did.

Comparing with the same period last year can help with seasons, but it brings its own problems: prices, competitors and your own capacity may all be different a year on.

Respect small numbers

Here's a hypothetical example. A plumber books 8 jobs from ads in one month and 12 the next, after improving how calls are answered. That's a 50% increase, and it feels decisive. But at those volumes, a swing of four jobs can easily happen from one month to the next with no change at all: one busy week, one big storm, two repeat customers who happen to call in the same month.

Some rules of thumb:

• Look at the counts, not only the percentages. "Up 50%" and "up four jobs" describe the same thing, and the second is more honest.
• The smaller the count, the longer you need to watch. A pattern that holds for three months is far more convincing than one good month.
• Be most skeptical of rates built on tiny bases, such as a booking rate for one postcode with a handful of leads. In the CallRadius Signal demo, the geography screen labels its areas "6 settled reads, 3 gathering evidence," which is the right instinct: some areas simply don't have enough data yet.

Check the season and everything else that changed

Most local service businesses have a season. Roofers see storms, HVAC companies see heat waves and cold snaps, and many trades slow around holidays. If you raised your budget at the start of your busy season, more leads would have arrived anyway.

Then list everything else that changed in the same window, not just the thing you're testing:

• Did a competitor enter, leave, or change how hard they're advertising?
• Did your reviews, hours, staffing or service area change?
• Did the platform itself change? Google updates its products and AI providers change their models, and both can move results for everyone at once.
• Did you make two changes in the same week? Then you can't separate their effects, only judge them together.

A useful test for AI visibility: if your competitors' mentions rose by the same amount as yours, the provider probably changed, not your website. We cover that in Website readiness isn't AI visibility.

A change after an action is not proof of cause

A change after an action does not by itself establish that the action caused it.

That sentence appears in the ClickRadius Signal sample workspace, next to its traffic and outcomes panel, and it's the standard we use everywhere. It doesn't mean you can never conclude anything. It means you should hold conclusions loosely in proportion to the evidence. Here's how we'd grade it:

Notice what's missing: certainty. Local marketing rarely gives you a controlled experiment, so the honest answer is often "likely," not "proven." That's still far better than a guess.

Keep a change log

The single most useful habit is a running log. One line per change, kept in a spreadsheet or even a notes app:

1. Date. The day the change went live, not the day you decided on it.
2. Channel. Local Services Ads, website, Google Business Profile, phones, AI visibility.
3. What changed. Specific enough that someone else could undo it.
4. What you expect. Which number should move, and in which direction.
5. Outside events. Storms, holidays, a competitor's new ad, a staff change.
6. Verdict, later. Filled in a month or two on, using the grades above.

Over a few months the log becomes the most valuable document in your marketing. It tells you which kinds of change tend to help your business, and it stops the same failed idea being tried every spring.

For how we label measured, illustrative and proposed numbers on this site, read our evidence standard.