Measurement
Compare two headlines without fooling yourself
A headline test is only real when the audience, the offer, and the form stay still while the headline changes.
Changing a headline and watching what happens is not yet a test. A test is a comparison you can explain: these two sentences, shown to the same kind of visitor, on the same offer and the same form, over a period you chose in advance, judged by finishes and a quality mark rather than by whichever number moved. Most “we tested it” stories miss at least one of those conditions. They still get told in meetings, and they still produce confident decisions. This article is about running a comparison that would survive being reread a month later.
You do not need a testing product to start. You do need discipline. If you cannot keep the rest of the page still, do not call the result a test. Call it a change, note the date, and judge it with the weekly review in the measurement guide. Pretending a messy change was a clean experiment is how teams learn the wrong lesson and then repeat it at a larger budget.
Change one idea
A headline test should change the headline, not the headline plus the hero image plus the first field plus the button color. If finishes rise, you will not know which change did it, and you will standardize all of them, including the ones that did nothing or hurt. Pick one contrast you can say in a sentence. “Appointment in the headline versus estimate in the headline.” “Service area named versus service area omitted.” Write both sentences down before you launch. If you cannot write them, you are still drafting, and drafting is not a test.
Both sentences have to be promises you can keep. Do not test a false window against a true one to see which wins. A false promise can “win” on finishes and lose on complaints, chargebacks of attention, and a follow-up team that stops trusting the page. The quality mark from the weekly review belongs in the comparison. A headline that increases submissions of people you cannot help is a loss, even if the completion rate looks wonderful.
Hold the audience still
Send the same sources to both versions, or split one source in a way you understand. Do not send email to version A and cold ads to version B and then announce that A is the better headline. You measured the source. Time of day and day of week matter too. A version that runs only on a Sunday against a version that runs on a Tuesday is not a headline comparison if your callers behave differently on those days. Run both versions across the same kinds of days, or run them in alternating blocks you defined first.
Decide the stopping point before you look. A set number of finishes, or a set number of days, both work for a small team. What does not work is peeking every morning and stopping when your favorite is ahead. That habit crowns noise. You will remember the win and forget the times you peeked and saw nothing. Write the rule on the same sheet as the two headlines. When the rule is met, look. Not before, except to catch a broken form, which is an incident and not a result.
Read finishes, then quality
Judge the versions first by finished submissions you can open, not by clicks and not by time on page. Then apply the same quality mark to a sample from each version. If one headline brings more finishes and worse quality, say that out loud and decide which outcome the business wants. There is no mathematical law that says volume wins. A measured-window headline that produces fewer but reachable people inside the service area can be the right one to keep. The test’s job is to show the trade. Your job is to choose it on purpose.
Beware a tiny gap. If the two versions are close, the honest conclusion is that you cannot tell, not that the one with three extra finishes is the voice of the customer. Keep the clearer sentence, the one that matches the ads more tightly, and move on. Another week of the same test, under the same rules, is allowed. Inventing a story about a three-finish gap is not. Small campaigns often do not have the volume for subtle contrasts. Test contrasts a reader can notice, or do not test.
Write down the loser
Archive both headlines, the dates, the sources, the finish counts, and the quality notes. Keep the loser in the archive. The loser is evidence about language your audience did not reward, and it stops the next teammate from “refreshing” the page back to a sentence you already tried. Tie the archive to the page-record habit in the consent guide. A test that changes the promise is a new version of the page. Save it like any other change. Do not leave the unused headline only in an ad-platform draft where nobody operational will look.
After you pick a winner, stop testing for a while. Let the weekly review run on a stable page. Constant testing on a low-volume campaign means you never have a baseline, and every dip has an excuse. Stability is what lets a later change mean something. The page is a tool for the follow-up team, not a toy for the person with the login to the builder. Ship the sentence, tell the people who answer the calls which sentence is live, and leave it until a real defect appears.
A test card
- Headline A and headline B, both promises you can keep.
- What stays identical: form, consent, offer, sources.
- The stop rule, written before the first visit.
- The finish count you will use, taken from submissions you can open.
- The quality mark, applied to both versions the same way.
- A place to file the loser and the dates.
- A note to the follow-up team when the live sentence changes.
If you cannot fill in the card, you are not ready to call the work a test. Change the page deliberately, measure it with the weekly review, and try a formal comparison when the offer is stable enough to deserve one. A careful small test teaches more than a dramatic inconclusive one, and it leaves a record the next person can trust.