What a Growth System Audit Actually Looks For

This series has walked all nine stages and the research behind each. This post is about the instrument that tells you which one is yours.

The logic of a diagnostic

From the constraints post: throughput in a serial system is set by one stage. So the diagnostic question is not "what could be improved" — nearly everything could — but "which single stage is currently binding."

That is a search problem, and the efficient search is one that maximises information per unit of effort. Concretely: measure the arrows, not the boxes. A stage that looks weak in isolation may be fine; what matters is where the throughput drops.

In that example the 3.1% session-to-inquiry rate is the number that draws the eye, and it is the wrong one to act on: it is normal for the category. The 44% qualified-to-contacted rate is the anomaly. Fifty-six percent of people you already qualified never got a conversation. That is stage 04, it is recoverable this month, and it does not require a single additional visitor.

The seven measurements

What I actually collect in five business days.

1. Per-stage yield, from records. Sessions, inquiries, inquiries actually received, qualified, contacted, met, proposed, closed. From the systems, not from recollection. This is the arithmetic above and it is usually the whole diagnosis.

2. Time-to-first-touch distribution. Not the mean — the mean is dominated by the tail. Median, 90th percentile, and the proportion never touched. Recall the 2014 audit where median first phone response was 3 hours 8 minutes while the mean was 61 hours Phillips et al. 2014. Two very different businesses can share a mean.

3. Synthetic transactions through every channel. A test inquiry through the form, the phone, the chat and the published email address. This is the only way to detect silent failures. In the 14,061-company audit, 4,031 companies had no working form and 183 had a broken one — failures invisible from inside the business, because nothing errors Phillips et al. 2014.

4. Offered load against capacity. Compute $R = \lambda/\mu$ from real handling time. If $R$ exceeds available attention, the queue is unstable and no process change will fix it Little 1961.

5. Handling-time variability. The coefficient of variation on the constraint stage. Through Kingman's V term this often predicts more delay than utilisation does Kingman 1961.

6. Identity integrity. Can a single record be followed from first touch to closed deal? If not, every attribution question is unanswerable and should be declared so rather than modelled.

7. Account and code ownership. Who owns the domain, analytics, ad accounts, listings and automation logic. This does not affect throughput; it affects whether any of the fixes are yours to keep.

What the diagnostic explicitly refuses to do

No modelled attribution at low volume. Given Lewis and Rao's finding that million-user randomised experiments produce ROI confidence intervals over 100 percentage points wide Lewis & Rao 2015, a multi-touch model on 400 journeys is decoration. If the honest answer is "this cannot be determined at your volume," that is the finding.

No experiment that cannot conclude. If the required sample exceeds a quarter of volume, I say so rather than building the infrastructure Kohavi et al. 2009.

No ranked backlog of forty items. A backlog transfers the hard decision back to you. The point of the exercise is to make that decision, with evidence, and to be wrong in a way you can check.

The deliverable

Three artifacts, and a test for each.

The map. Your nine stages on one page with a measured number on every arrow. Test: you can point at the constraint without me in the room.

The constraint, with its evidence. Which stage binds, what it costs per month in units you care about, and how that number was computed. Test: you could argue with it, because the derivation is shown.

The intervention, scoped. What to change, what it should move, and how you will know. Test: an engineer who is not me could execute it from the document.

That last test is deliberate. A diagnostic whose recommendations only its author can act on is not a diagnostic — it is a sales mechanism with a report attached. The same logic as leaving the keys: if the work is only valuable because you cannot take it elsewhere, it was not valuable.

What happens if I find nothing

It happens. Sometimes the system is sound and the business is simply not generating enough demand into a functioning path — which is a real finding, and a different project.

The failure mode I want to avoid is manufacturing a constraint because a diagnostic that reports "your system is fine" feels like a poor return on $950. Twyman's law applies to me as much as to anyone: any figure that looks interesting or different is usually wrong Kohavi & Longbotham 2010. If the striking number in your audit is an artifact, my job is to say so before you spend money on it.

References

1. Phillips, J. W., Krogue, K., & Elkington, D. (2014). Annual 2014 Lead Response Report. InsideSales.com / XANT research report. https://resources.insidesales.com/wp-content/uploads/2019/11/2014-Lead-Response-Report.pdfVendor research with a documented secret-shopper instrument and full sample accounting.
2. Little, J. D. C. (1961). A proof for the queuing formula: L = λW. Operations Research, 9(3), 383–387. https://doi.org/10.1287/opre.9.3.383
3. Kingman, J. F. C. (1961). The single server queue in heavy traffic. Mathematical Proceedings of the Cambridge Philosophical Society, 57(4), 902–904. https://doi.org/10.1017/S0305004100036094
4. Lewis, R. A., & Rao, J. M. (2015). The unfavorable economics of measuring the returns to advertising. The Quarterly Journal of Economics, 130(4), 1941–1973. https://doi.org/10.1093/qje/qjv023
5. Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 140–181. https://doi.org/10.1007/s10618-008-0114-1
6. Kohavi, R., & Longbotham, R. (2010). Unexpected results in online controlled experiments. ACM SIGKDD Explorations Newsletter, 12(2), 31–35. https://doi.org/10.1145/1964897.1964905