Measurement & Causality
Measurement and causality analysis determines whether observed changes are caused by the experiment or by external factors.
After this lesson you can establish baselines, adjust for seasonality, assess statistical confidence, analyze ranking and conversion impact, and document test results.
This lesson covers the seven measurement areas (leaves 10.5.1–10.5.7): pre-test baseline analysis, seasonality adjustment, statistical confidence review, ranking and traffic impact analysis, conversion impact analysis, cannibalization monitoring, and test documentation.
Experiment Design Types
Not all before/after comparisons are experiments. Choose the right design for your situation:
- Controlled experiment: randomized test/control groups, simultaneous measurement — the gold standard. Random assignment to treatment and control groups, measured concurrently, eliminates most confounds.
- Pre/post analysis: comparing before vs after, no control group — observational, confounded by seasonality and external factors, NOT an experiment. Only useful as directional evidence.
- Natural experiment: leveraging an external change (algorithm update, platform change) that acts as treatment — requires careful causal reasoning. The treatment is not under your control, so you must argue convincingly that assignment is as-if random and that no other factors changed simultaneously.
Power Analysis & Minimum Detectable Effect
Before running a test, calculate what effect size you can detect:
- Define the minimum meaningful effect — what CTR/ranking/sessions change is worth rolling out? (e.g., 5% CTR improvement)
- Estimate baseline variance — from historical data, what's the standard deviation of your metric across similar pages?
- Calculate required sample size — for the desired power (typically 80%) and significance level, determine how many pages per group you need.
- If required sample > available pages — you cannot reliably run the test. Consider: wider metric scope, composite metrics, or accepting the test as directional/exploratory.
| Desired Effect | Typical Sample Needed (per group) |
|---|---|
| Large (20%+ relative change) | 30-60 pages |
| Medium (10-20%) | 60-200 pages |
| Small (5-10%) | 200-500+ pages |
| Very small (<5%) | 500+ pages or undetectable |
Pre-Test Baseline Analysis
Establish the baseline before running the test.
Baseline data points:
| Metric | Collection Period | Purpose |
|---|---|---|
| Organic sessions | 4-8 weeks before test | Traffic baseline |
| Average position | 4-8 weeks before test | Ranking baseline |
| CTR | 4-8 weeks before test | Click-through baseline |
| Engagement rate | 4-8 weeks before test | Content quality baseline |
| Conversion rate | 8-12 weeks before test (longer for low-traffic) | Conversion baseline |
Baseline analysis:
- Calculate the average and variance for each metric during the pre-test period.
- Assess whether the metric is stable or has a trend (trending metrics require different analysis).
- If the metric is trending (up or down), the trend must be accounted for in post-test analysis.
Seasonality Adjustment
Adjust for seasonal variations that could affect test results.
Seasonality adjustment methods:
| Method | Application |
|---|---|
| Year-over-year comparison | Compare test period to same period last year |
| Seasonality index | Apply monthly index to expected values |
| Control group comparison | Control group experiences same seasonality — difference-in-differences |
When seasonality adjustment is critical:
- Tests running during known seasonal peaks (holidays, industry events).
- Tests on content that has seasonal search patterns.
- Tests lasting more than 2 weeks (seasonality can shift intra-month).
Statistical Confidence Review
Review the statistical significance of test results.
Statistical concepts for SEO:
| Concept | Application |
|---|---|
| P-value | Probability that the observed effect is due to chance (p < 0.05 = statistically significant) |
| Confidence interval | Range within which the true effect likely falls |
| Sample size | Number of pages or impressions needed for reliable results |
| Power | Probability of detecting an effect if it exists (typically 0.80) |
SEO-specific statistical challenges:
| Challenge | Impact | Mitigation |
|---|---|---|
| Low-traffic pages | Cannot reach statistical significance for small changes | Aggregation across page groups |
| External factors (algorithm updates) | Can invalidate results | Document external events during test period |
| Long feedback loops | Ranking changes can take weeks | Extend test duration |
| Multiple testing | Running many tests increases chance of false positives | Apply Bonferroni correction or similar |
Ranking and Traffic Impact Analysis
Analyze how the test affected rankings and traffic.
Analysis methods:
| Method | Best For |
|---|---|
| Difference-in-differences | Test vs control group before/after comparison |
| Time series analysis | Pre/post comparison when no control is available |
| GSC position comparison | Query-level position changes |
| GA4 session comparison | Session volume and trend changes |
Causality assessment:
| Pattern | Causality Assessment |
|---|---|
| Test group improves, control stays same | Strong causal evidence |
| Both test and control improve | External factor (seasonality, algorithm) |
| Test improves, control declines slightly | Possible cannibalization — investigate |
| Test does not improve, control improves | External factor benefiting control more |
Conversion Impact Analysis
Analyze how the test affected conversion metrics.
Conversion analysis considerations:
| Consideration | Detail |
|---|---|
| Conversion lag | Conversion may not happen immediately after visit |
| Attribution model | Different models can show different conversion changes |
| Segment analysis | Did conversion change differ by traffic segment (brand/non-brand, device)? |
| Statistical significance for low-volume conversions | May require longer test periods |
Cannibalization Monitoring
Monitor whether the test change caused cannibalization.
Cannibalization detection during tests:
| Signal | Action |
|---|---|
| Test page improves, control page declines | Possible cannibalization — check query overlap |
| Multiple pages ranking for same query | Run query-to-page report in GSC |
| Traffic shifts from one page to another | Check if users are landing on the intended page |
Cannibalization threshold:
- If a test group page gains traffic at the expense of another page from the same site, the net gain is the relevant metric.
- If net gain is positive (new page gains more than old page loses), the test is still beneficial.
Test Documentation
Document each experiment for future reference and learning.
Test documentation template:
Test Name: [Descriptive name]
Test Date: [Start] — [End]
Hypothesis: [If... then... because...]
Test Group: [Pages, segment]
Control Group: [Pages, segment]
Change Applied: [Specific change]
Primary Metric: [Metric and expected direction]
Secondary Metrics: [Other metrics tracked]
Results:
- Primary Metric: [Change and statistical significance]
- Secondary Metrics: [Changes]
- Cannibalization: [Any detected]
- External Factors: [Algorithm updates, seasonality, etc.]
Decision: [Roll out / Roll back / Further investigation needed]
Learnings:
- [What we learned from this test]
- [What we would do differently next time]
Documentation best practices:
- Document every test, even failed ones (learning what does not work is valuable).
- Store in a shared location for team reference.
- Review past tests before designing new ones to avoid repeating mistakes.