The Runbook
- 1. The Most Common Question in E-Commerce Analytics
- 2. The GMV Metrics Tree
- 3. The Diagnostic Runbook — Five Steps, In Order
- 4. Rate vs Mix — The Math That Separates Seniors from Juniors
- 5. Simpson's Paradox in the Wild
- 6. The Decomposition Query
- 7. Communicating the Diagnosis
- 8. From Firefighting to Early Warning
Every e-commerce analyst gets this message weekly: "GMV is down vs LW — can you check what's going on?" Most answer it ad hoc — poking at dashboards until something looks off, then declaring that the cause. This post replaces the poking with a runbook: a fixed decomposition order that finds the real driver fast, avoids the classic statistical traps, and produces an answer leadership can act on.
1. The Most Common Question in E-Commerce Analytics
Three things make "why is GMV down?" harder than it looks:
- GMV is a product of factors, not a sum — a 5% drop can be one factor down 5% or five factors down 1%, and the fixes are completely different.
- Mix changes masquerade as rate changes — overall conversion can fall while every segment's conversion is flat, purely because traffic shifted toward low-converting segments.
- The calendar lies constantly — paydays, Ramadan/Eid, White Friday laps, weather, and competitor events make naive week-over-week comparisons misleading more often than not.
2. The GMV Metrics Tree
The decomposition everyone should be able to write from memory:
GMV = Sessions × Conversion Rate × Average Order Value
where AOV = Units per Order × Average Selling Price
Each node then splits along its natural dimension:
| Node | Splits into | Typical owner |
|---|---|---|
| Sessions | Channel (organic, paid, CRM, direct, app push) × new/returning | Marketing / growth |
| Conversion rate | Funnel stages: session → PDP → add-to-cart → checkout → payment success | Product / commercial / payments |
| Units per order | Category mix, promo mechanics (thresholds, bundles) | Merchandising |
| Average selling price | Price changes × discount depth × category/product mix | Pricing / commercial |
For marketplaces, the demand-side tree above pairs with a supply-side tree (active sellers × live assortment × availability), and the customer view (GMV = active buyers × orders per buyer × AOV) is often more diagnostic for retention problems. Use the tree that matches the question — but always a tree, never a wall of disconnected KPIs.
The tree's real value is ownership. Every node has a name attached. "GMV is down" is everyone's problem and no one's; "payment success rate on card transactions dropped 2pts" is the payments team's problem by lunchtime.
3. The Diagnostic Runbook — Five Steps, In Order
The order matters: each step is cheaper than the next and rules out an entire class of causes.
Step 0 — Is the data even right? (5 minutes)
Before analyzing a drop, verify it exists. Tracking breaks cause more "GMV crises" than demand does: a tag release, an app version with broken events, a pipeline delay that makes yesterday look 20% down at 8am and fine by noon. Check: does the drop appear in the transactional source of truth (orders table), not just the analytics layer? Is the pipeline complete for the period? Does finance's number agree?
Step 1 — Calendar & comparability (5 minutes)
Is the comparison fair? Payday weeks vs lean weeks, Ramadan shifting ~11 days annually (a WoW disaster and a YoY disaster in different years), a promo event in the base period you're lapping, weekends vs weekdays in a partial-week comparison. The most useful habit: keep an events calendar table in the warehouse (promos, holidays, paydays, price changes, app releases, competitor events) and join it to every trend query. Half of all "anomalies" die at this step.
Step 2 — Top-level decomposition (15 minutes)
Split ΔGMV into sessions, CVR, and AOV contributions (math in section 4). This tells you which meeting to book: traffic problem → marketing; conversion problem → product/payments/pricing; AOV problem → merchandising/mix. Don't skip to segment drilling before this — you'll drown in cuts.
Step 3 — Segment drill on the broken factor (30–60 minutes)
Take the factor that moved and cut it by its natural dimensions: channel and new/returning for sessions; device, funnel stage, payment method, and city for conversion; category and discount depth for AOV. You're looking for concentration: is the drop everywhere (macro/systemic) or concentrated (specific and fixable)? A conversion drop isolated to Android app v3.2.1 is a bug. A conversion drop everywhere but concentrated in one category is assortment or pricing. A drop across all segments equally is usually demand-side or measurement.
Step 4 — External checks (as needed)
If nothing internal explains it: competitor events (their flash sale is your traffic dip), price index movements (see the pricing playbook), weather, school calendars, sports finals, app store ranking changes. This is where the events calendar pays off again — log external events too.
4. Rate vs Mix — The Math That Separates Seniors from Juniors
Because GMV is multiplicative, contributions don't add up naturally. The practical fix is the log decomposition: since GMV = S × C × A,
Δlog(GMV) = Δlog(Sessions) + Δlog(CVR) + Δlog(AOV)
Each term's share of Δlog(GMV) is that factor's contribution — clean, additive, and it works for any number of factors. For small changes (<10%), percentage changes approximate the same thing: a 5% GMV drop splitting into −3% sessions, −1.5% CVR, −0.5% AOV.
The second layer is rate vs mix within a factor. When overall CVR drops, there are two possible worlds:
- Rate effect: segments actually convert worse (something broke or worsened)
- Mix effect: every segment converts the same, but traffic shifted toward low-converting segments (e.g., a viral social campaign flooding mobile web with cold traffic)
The standard two-part split for overall CVR change between period 0 and period 1, with segments i:
# w_i = segment share of sessions, c_i = segment conversion rate
rate_effect = sum( w1_i * (c1_i - c0_i) ) # conversion changed within segments
mix_effect = sum( (w1_i - w0_i) * c0_i ) # traffic moved between segments
# rate_effect + mix_effect = total CVR change (exactly)decomposition
The actions are opposite: a rate effect sends you hunting for what broke; a mix effect sends you to marketing to ask what changed in acquisition — and often the answer is "nothing's wrong, we bought cheaper traffic on purpose." Reporting a mix effect as a conversion problem is the fastest way to lose a product team's trust.
5. Simpson's Paradox in the Wild
The extreme version of the mix effect has a name, and it appears in e-commerce constantly. Real-shaped example:
| Segment | Last week | This week | Segment CVR trend |
|---|---|---|---|
| App | 100k sessions @ 5.0% CVR | 80k sessions @ 5.2% CVR | ✅ Up |
| Mobile web | 100k sessions @ 1.5% CVR | 160k sessions @ 1.6% CVR | ✅ Up |
| Overall | 200k @ 3.25% | 240k @ 2.80% | ❌ Down 45bps |
Every segment improved; the total fell — because the session mix shifted from a 5% surface to a 1.6% surface. If you report "conversion is down, we have a site problem," you've just sent three teams debugging a site that got better. The rule: never report an aggregate rate change without checking the segment-level rates and the mix. This applies to CVR, AOV, return rates, delivery SLA, payment success — every ratio metric in the business.
6. The Decomposition Query
The workhorse query — top-level factors, this period vs last, with log contributions:
WITH daily AS (
SELECT
CASE WHEN session_date BETWEEN '2026-07-04' AND '2026-07-10'
THEN 'this_week' ELSE 'last_week' END AS period,
COUNT(DISTINCT session_id) AS sessions,
COUNT(DISTINCT order_id) AS orders,
SUM(gmv) AS gmv
FROM sessions_orders
WHERE session_date BETWEEN '2026-06-27' AND '2026-07-10'
GROUP BY period
)
SELECT
MAX(IF(period='this_week', gmv, NULL))
/ MAX(IF(period='last_week', gmv, NULL)) - 1 AS gmv_change,
-- log contributions: these three sum to log(gmv ratio)
LN(MAX(IF(period='this_week', sessions, NULL))
/ MAX(IF(period='last_week', sessions, NULL))) AS contrib_sessions,
LN( (MAX(IF(period='this_week', orders, NULL))
/MAX(IF(period='this_week', sessions, NULL)))
/ (MAX(IF(period='last_week', orders, NULL))
/MAX(IF(period='last_week', sessions, NULL)))) AS contrib_cvr,
LN( (MAX(IF(period='this_week', gmv, NULL))
/MAX(IF(period='this_week', orders, NULL)))
/ (MAX(IF(period='last_week', gmv, NULL))
/MAX(IF(period='last_week', orders, NULL)))) AS contrib_aov
FROM dailybigquery sql
Then re-run the same shape per segment on whichever factor moved (see the SQL cookbook for the conditional-aggregation pattern this uses). Two disciplines: compare full weeks only, and always show the contributions next to the raw factor changes — "sessions −3.1% (62% of the drop)" reads instantly.
7. Communicating the Diagnosis
The analysis is half the job; the delivery decides whether it becomes action. The format that works:
"GMV −5.2% WoW. ~60% of it is paid traffic (campaign X ended Tuesday — expected), ~30% is Android checkout conversion (drop began with v3.2.1 release Wednesday — bug ticket filed, fix ETA tomorrow), remainder within normal variation. Nothing suggests a demand problem. Watch: Android fix validation in tomorrow's numbers."
The structure: total → contributions with causes → expected-vs-unexpected labeling → what happens next. Three practices that build the reputation of always having the answer:
- Label expected vs unexpected. A drop from ending a campaign is a decision playing out, not an incident. Mixing the two erodes trust in every alert.
- Quantify or say "unexplained." "Probably weather" is worse than "1.2% unexplained — within normal weekly variation." Normal variation exists; teach leadership its size (a control-chart view of daily GMV does this beautifully).
- Never ship a cause you haven't cross-checked in a second source. The runbook exists to protect you from the plausible-but-wrong first answer.
8. From Firefighting to Early Warning
- Automate the tree: a daily job computing the log decomposition and flagging any factor contributing more than a threshold — the 7am question answers itself before 7am.
- Alert on the tree's leaves, not the root: payment success by method, zero-result rate, app crash rate, stockout rate on A-SKUs. Leaf alerts fire a day before the GMV line notices (the leaves are covered across this series — search, payments, inventory).
- Maintain the events calendar — it converts anomaly investigation from archaeology into a join.
- Publish "normal variation" bands so a −1.5% Tuesday stops generating meetings.
Decompose before you drill, split rate from mix before you blame, check the calendar before you panic — and automate all three so the question is answered before it's asked.
Sources & Further Reading:
KPI Tree: Metric Trees for E-Commerce •
Metric Trees for Marketplaces •
Simpson's Paradox