Skip to content
Liftwire

Measurement

verify_change proves a change is on the page. get_outcomes reports what the page's numbers did before and after it. This page says exactly what is read, over which days, how a change is matched to the numbers, and what the comparison cannot tell you.

Sources

You connect a source for a site with connect_analytics, which returns a one-time link for you to open.

Google Analytics 4. One runReport call per period, with the metrics sessions, engagementRate and conversions (conversions only where your property has them configured), filtered on pagePath for the page's path with and without a trailing slash. The sample size of a period is its sessions.

Search Console. One searchanalytics.query call per period with dataState: final, returning clicks, impressions and ctr, filtered to the page with an includingRegex page filter. The sample size of a period is its clicks.

Microsoft Clarity can be connected, and its credentials are stored encrypted, but its data is not used for outcomes, because it keeps only a few days of page history, which is too short for the windows below. GA4 sessions and Search Console clicks are the outcome sources.

Windows

For a change verified on day D (UTC):

  • the baseline period is D−28 to D−1;
  • the outcome period is D+1 to D+28;
  • day D itself belongs to neither, because the change went live part-way through it.

Outcomes for a change are computed only once the outcome period has ended and a further 3 days have passed, because GA4 and Search Console data for a day is not final straight away.

Sample floor

A relative change is computed only when both periods reach 500: sessions for GA4, clicks for Search Console. Below that, both values are reported side by side with their sample sizes, labelled insufficient_sample, and no percentage is derived. A missing value, or a baseline of zero, is labelled not_comparable.

Attribution

A change is matched to the numbers by page URL. Day D is the change's verified_at: the moment verify_change ran and found the tactic present, not the moment you deployed. Run verify_change promptly after each release; a late verify puts days that already had the change into the baseline. Only changes verified as present are measured.

What it cannot tell you

  • Several changes to one page inside one window cannot be told apart. They share the same before and after numbers.
  • There is no control group. Traffic that changed for another reason, such as a campaign, a launch or a search update, shows up as a difference.
  • There is no seasonality adjustment. A holiday or a quarter-end inside either window shows up as a difference.
  • It is a before and after comparison on one page, not an experiment. It describes what happened; it does not prove the change caused it.

This is why a tactic reaches the measured evidence grade only with outcomes from several sites, and why a measured figure is never a promise for your site.

When outcomes run

Outcomes are computed by a nightly job, which is switched on in production as an operations step. Until it is, a verified change stays "shipped, unmeasured".