TrueLift: Honest Value-Weighted Experiment Decisioning#

From conversion winners to uncertainty-gated value policies

A technical white paper

Abstract#

Conversion-rate optimization often declares the variant with the highest conversion rate the winner. That rule is only aligned with growth when every conversion has the same downstream value. In practice, variants can change who converts, what those customers spend, whether they retain, and the costs they create. A conversion winner can therefore be a value loser.

TrueLift is a research-oriented system for making experiment decisions on value per randomized unit, with conversion retained as a diagnostic rather than the objective. It estimates average value effects, exposes conversion–value disagreement, controls a declared family of comparisons, measures value-outcome design sensitivity, and permits a hard recommendation only when the corresponding uncertainty gate is satisfied. For prespecified categorical segments, the browser tier learns and evaluates a targeting rule on separate folds. The optional backend applies deterministic cross-fitting, treatment-specific outcome models, and doubly robust policy scores to the same categorical feature surface.

The implementation is evaluated in three layers: known-answer and adversarial regression tests; seeded Monte Carlo experiments with known data-generating processes; and a reproducible external benchmark on Kevin Hillstrom's 64,000-customer randomized email experiment. It is not a production experimentation platform. It assumes a valid randomized experiment, a prespecified outcome horizon, stable treatment versions, and a value definition whose zeroes and negative values have substantive meaning.

The contribution is not a new causal estimator. It is an auditable decision architecture that combines established experimental, multiple-testing, honest heterogeneity, cross-fitting, and doubly robust policy-evaluation ideas into one small reference system while refusing to turn weak evidence into a confident product recommendation.

Table of Contents#

1. Introduction & Motivation
2. Research Questions, Goals & Maturity
3. Estimands & Decision Problem
4. Data Contract & Causal Assumptions
5. System Architecture
6. Aggregate Inference & Value-Based Power
7. Honest Heterogeneous Policy Learning
8. Uncertainty-Gated Recommendations
9. Monte Carlo Calibration
10. External Hillstrom Benchmark
11. Empirical Results
12. Correctness & Quality Assurance
13. Privacy, Security & Reproducibility
14. Comparison to Prior Art
15. Limitations & Future Work
16. Conclusion
17. Appendices

• References

1. Introduction & Motivation#

1.1 A conversion is not the outcome#

An online experiment randomizes an experience, not a conversion. The unit assigned to a variant may convert or not convert; either way, it belongs in the analysis. If value denotes net revenue, contribution margin, retained lifetime value, or qualified pipeline value measured over a fixed horizon, the primary experiment outcome is the mean value across all randomized units:

Conversion rate remains useful because it explains how value moved. It is not, by itself, the decision objective. TrueLift's signature diagnostic is a disagreement between the highest-conversion arm (including control) and the highest-value arm. That disagreement is an investigation trigger, not sufficient evidence for a recommendation.

1.2 Why the original prototype was insufficient#

The first TrueLift prototype demonstrated the product thesis, but several choices were too optimistic for research use:

• value reliability reused a conversion-rate power calculation;
• a multi-arm branch could ship an observed value winner without passing an uncertainty gate;
• segment rules were selected and valued on the same observations;
• the backend labelled an agreement-weighted plug-in calculation as IPW;
• the external Hillstrom validation was asserted but not reproducible; and
• a bespoke number was described as a probability of replication without a calibrated sampling model.

Version 0.2.0 treats those as research defects, not copy problems, and changes the implementation and its evidence artifacts together.

1.3 The core idea: decision honesty#

TrueLift uses four evidence classes:

1. Descriptive: observed conversion and value means.
2. Inferential: uncertainty for prespecified aggregate value effects.
3. Policy: out-of-fold value of a rule that chooses treatment by covariates.
4. Decision: a recommendation permitted only when its own corrected uncertainty criterion passes.

The system never upgrades a descriptive winner into a decision winner merely because its point estimate is largest.

2. Research Questions, Goals & Maturity#

2.1 Research questions#

TrueLift is evaluated against four questions.

RQ1: Objective alignment. Can a value-weighted readout distinguish a conversion winner from a value winner and quantify the opportunity cost of following the conversion metric?

RQ2: Decision calibration. Under known null, positive, negative, and heterogeneous data-generating processes, how often does the system make a hard recommendation, cover the true value effect, and choose the correct action?

RQ3: Honest targeting. Can a segment or model-based policy improve value out of sample relative to both uniform treatment and uniform control without reusing the same outcome information for discovery and evaluation?

RQ4: Reproducibility. Can an independent reviewer regenerate the unit tests, simulations, external benchmark, checksums, and reported tables from versioned code?

2.2 Success criteria#

2.3 Hard constraints#

2.4 Anti-goals#

TrueLift is not:

• a replacement for experiment randomization or instrumentation;
• an observational causal-inference system;
• a sequential-testing or always-valid inference engine;
• a general multi-treatment policy learner in the browser;
• an automatic definition of customer lifetime value;
• evidence that any uploaded segment is safe, fair, or operationally appropriate for targeting; or
• a production deployment approval system without scientific review.

2.5 Maturity#

The browser path is suitable for demonstrations and controlled methodological evaluation. The backend demonstrates a cross-fitted binary-treatment policy pipeline. Production readiness would additionally require identity-level randomization checks, outcome maturity controls, sequential monitoring rules, stronger heavy-tail inference, access control, observability, model governance, and repeated validation across independent experiments.

3. Estimands & Decision Problem#

3.1 Notation#

For randomized unit $i$:

• $A_i \in \{0,1\}$ is assigned control or treatment;
• $X_i$ contains pre-treatment covariates or a prespecified segment;
• $C_i$ is a conversion indicator used diagnostically;
• $Y_i$ is value measured over a prespecified horizon; and
• $Y_i(a)$ is the potential value under assignment $a$.

3.2 Primary aggregate estimand#

The primary estimand is the intention-to-treat average treatment effect on value:

For a randomized experiment, the unadjusted difference in arm means estimates $\tau$:

The conversion-rate effect is secondary:

The trap diagnostic is true when the arm with the highest observed conversion, including control, is not the arm with the highest observed value. Because both winners are selected from noisy estimates, trap alone never authorizes shipping or targeting. The focus contrast still uses the highest-conversion challenger when control wins, so a proposed change can be evaluated without relabelling control as a treatment.

3.3 Policy estimand#

A binary policy $\pi(X)\in\{0,1\}$ assigns treatment as a function of pretreatment covariates. Its value is

The actionable policy contrast is defined against the better uniform action:

This baseline prevents a targeted policy from looking valuable merely because it beats a poor blanket rollout while remaining inferior to control.

3.4 Decision loss#

For calibration, the oracle action is the action with highest true expected value under the simulation data-generating process. Policy regret is

The benchmark reports value and regret; it does not grade an estimator by how often it reproduces a hand-written demo label.

4. Data Contract & Causal Assumptions#

4.1 Browser data contract#

Each row represents one randomized unit:

variant, converted, value, segment?

variant, converted, and value are required. segment is optional and must be determined before treatment. The current browser contract requires value to equal zero for non-converters and rejects malformed, missing, non-finite, or inconsistent numeric cells rather than rewriting them silently. That contract is appropriate only when a non-converter truly contributes zero on the specified net-value outcome. If exposure itself creates costs, retention effects, or other nonzero outcomes, the outcome contract must be extended rather than discarding those effects.

Aggregate hard actions require at least 20 rows in each analyzed arm. Categorical-policy evaluation requires at least 10 observations in each arm-by-segment cell of each evaluation fold. These are numerical safety floors, not guarantees that a zero-inflated or heavy-tailed value outcome has reached an adequate effective sample size.

4.2 Identification assumptions#

The causal interpretation requires:

1. Random assignment: treatment assignment is independent of potential outcomes.
2. Consistency: the observed outcome equals the potential outcome under the assigned treatment.
3. No interference: one unit's assignment does not change another unit's outcome.
4. Positivity: every policy-relevant covariate region has nonzero probability of each treatment.
5. Pretreatment segmentation: policy features are not caused by treatment.
6. Stable outcome horizon: all arms have the same matured measurement window.
7. No differential missingness: value capture is not selectively absent by arm.
8. Correct randomization unit: repeated rows, clusters, or household spillovers are handled before ingestion.

TrueLift cannot recover these assumptions from a CSV. A research or production deployment must establish them from experiment design and data provenance.

4.3 Value definition#

A value metric should be declared before analysis and should specify:

• currency or business unit;
• measurement horizon;
• gross versus net value;
• margin, refunds, chargebacks, discounts, servicing cost, and churn treatment;
• censoring and late-arriving outcomes;
• winsorization or other robustification, if any; and
• the analysis population and randomization unit.

Changing the value definition after inspecting arm results is outcome selection and invalidates the nominal inference.

4.4 Experiment integrity checks outside this POC#

Before using the engine, a full platform should check sample-ratio mismatch, duplicate units, covariate balance, bot/internal traffic, instrumentation parity, missing outcomes, and overlapping experiments. The absence of these checks in the reference UI is a stated limitation, not evidence that the checks are unnecessary.

5. System Architecture#

TrueLift has three independently testable evidence layers.

5.1 Browser reference engine#

The analysis engine is pure JavaScript with no DOM dependency. Tests load the marked engine region in a Node vm, which keeps the shipped HTML as the source of truth.

5.2 Backend model-based tier#

The backend is binary for the chosen focus arm versus control. When several challengers are present, it selects the highest-conversion challenger on the full request and does not adjust the subsequent pairwise inference for that outcome-based selection. Prespecify control and a two-arm input for confirmatory use; multi-arm selection-aware policy optimization remains future work.

5.3 Research harness#

The research/ directory contains:

• seeded Monte Carlo calibration with known true effects;
• a downloader/runner for the Hillstrom randomized experiment;
• source provenance and SHA-256 verification;
• JSON results intended for machines; and
• Markdown summaries intended for review.

Every result in this paper should trace to an executable artifact rather than exist only as manually copied prose.

6. Aggregate Inference & Value-Based Power#

6.1 Value effect and standard error#

For treatment and control sample variances $s_1^2$ and $s_0^2$, the browser uses the unequal-variance standard error

The implementation reports the point estimate, p-value, and confidence interval together. Revenue-like outcomes can be zero-inflated and heavy-tailed; the large-sample approximation is therefore a declared limitation for small uploads. The backend's cross-fitted policy uncertainty is computed from the empirical variance of out-of-fold influence scores.

6.2 Value-based MDE and sensitivity#

Design sensitivity is computed for the value estimand, not conversion. With two-sided type-I error $\alpha$ and target power $1-\beta$, the current-design minimum detectable absolute value effect is approximated by

The UI also reports two-sided normal sensitivity at the observed absolute effect. If $\lambda=|\widehat{\tau}|/\widehat{SE}$ and $z_c=z_{1-\alpha/2}$, the displayed quantity is

This post-hoc sensitivity is a deterministic function of the observed effect and standard error. It is neither independent confirmatory evidence nor a probability that the result will replicate. Version 0.2 retains an 80% sensitivity threshold as a conservative decision-resolution rule layered on top of corrected, sign-definite uncertainty. A production protocol should instead prespecify a minimum economically important effect and estimate its variance from blinded historical data or a pre-experiment period; the current browser has no such planning input.

6.3 Multiplicity#

The comparison family is constructed explicitly from the value and diagnostic tests generated by the analysis. Benjamini–Hochberg controls the false discovery rate under its stated conditions. A test can authorize a hard recommendation only if it is rejected in the adjusted family; an unadjusted segment p-value cannot independently create a TARGET decision.

For high-stakes deployments, controlling family-wise error or using hierarchical testing may be preferable. The paper does not claim that FDR control is always the correct business loss function.

6.4 Why “replication probability” was removed#

The original UI transformed a confidence-bound heuristic into a percentage labelled “chance of replicating the direction.” That label implied a calibrated predictive probability without defining a future sample size, variance, design, or replication event. Version 0.2.0 instead reports directly interpretable evidence: effect, interval, adjusted significance, MDE/sensitivity, and policy-improvement uncertainty.

7. Honest Heterogeneous Policy Learning#

7.1 Why in-sample segment targeting is optimistic#

If segments are chosen because their observed treatment effects are positive and the same observations are then used to value the resulting rule, the policy estimate includes selection noise. Requiring an unadjusted p-value does not remove this bias. Honest estimation separates the information used to construct the rule from the information used to evaluate it, following the general principle developed for heterogeneous treatment-effect estimation by Athey and Imbens.

7.2 Browser categorical policy#

The browser tier supports prespecified categorical segments. It uses deterministic, arm-and-segment-stratified folds. For each fold:

1. estimate segment treatment effects on the complementary folds;
2. apply the declared multiplicity rule to the training-fold segment family;
3. define treatment eligibility using training data only;
4. apply that rule to the held-out fold; and
5. compute held-out policy, control, and blanket-treatment value contributions.

The process rotates through folds so every row is evaluated out of fold. This estimates the value of the learning procedure, whose selected segments can differ across folds. After evaluation, the displayed deployable rule is refit on the full dataset; its selected segments and displayed deployment share are therefore not themselves held out. The UI reports the full-data deployment share separately from the average out-of-fold treat share. TARGET requires positive cross-fitted evidence versus both uniform actions and also requires the final full-data rule to select a nonempty, non-universal subset. The interval should be interpreted as uncertainty for the cross-fitted learning procedure, not as an independently validated interval for that exact final fixed rule.

7.3 Backend T-learner nuisance models#

The backend estimates

with one regression model per arm. The conditional treatment-effect prediction is

A fold-specific candidate policy assigns treatment where the training-only prediction is positive, subject to any operational constraints. T-learners are a standard meta-learning construction; they are not intrinsically causal without the experiment assumptions in §4.

In version 0.2.0, $X$ is the optional categorical segment field (with an explicit missing level), encoded inside each training fold. Numeric or multi-column covariates are not yet part of the API contract.

7.4 Cross-fitted doubly robust value#

For known or estimated propensity $e_a(X_i)=\Pr(A_i=a\mid X_i)$, the out-of-fold doubly robust score for policy $\pi$ is

The estimated policy value and standard error are

Policy improvement versus a uniform baseline uses the paired score difference. Cross-fitting keeps each outcome out of the nuisance-model and policy-learning step that produces its own score. Doubly robust estimation remains vulnerable when both outcome and propensity models are poor, positivity is weak, or the policy itself is repeatedly tuned against the final benchmark.

8. Uncertainty-Gated Recommendations#

8.1 Principle#

The output is a decision aid, not a leaderboard. Point-estimate rank determines which contrast to examine; it does not determine the action.

8.2 Decision table#

Operationally, EXPLORE means hold the current policy while collecting the evidence or redesigning the bet. It is not a recommendation to expose users to an uncertain winner.

8.3 Multi-arm selection#

For multiple challengers, each treatment-versus-control value comparison enters the adjustment family. If the conversion winner differs from the observed value winner, the engine examines the value winner's own corrected contrast. It cannot ship that arm merely because it has the largest observed value.

8.4 Practical significance#

Statistical significance is not sufficient. For TARGET, the browser uses the fixed adaptive threshold $\epsilon=\max(0.001,0.005\max(|\widehat V(0)|,|\widehat V(1)|))$ per visitor against each uniform baseline. Aggregate SHIP and DON'T SHIP currently use zero rather than a configurable minimum economically important difference. A production protocol should prespecify economically meaningful thresholds that include implementation, targeting, risk, and downside costs; the current hard-coded policy epsilon is only a POC safeguard.

9. Monte Carlo Calibration#

9.1 Purpose#

Known-answer unit tests show that formulas return expected numbers. They do not show that the full decision procedure behaves well under repeated sampling. The calibration harness therefore generates experiments where the true aggregate and segment effects are known.

9.2 Data-generating processes#

The suite includes at least:

• a global null with identical arm distributions;
• a uniform positive value effect;
• a uniform harmful value effect;
• genuine segment heterogeneity with a policy better than either uniform arm;
• zero-inflated, skewed value where practical.

Each design uses a fixed master seed and records the number of replications, arm sizes, effect sizes, and distribution parameters. Unequal arm sizes are covered by a deterministic browser regression test, not by the current Monte Carlo scenario set.

9.3 Reported metrics#

The calibration output reports:

• confidence-interval coverage of the true aggregate value effect;
• unsupported hard-action rate under the null;
• correct-action rate where the oracle action is separated from alternatives;
• TARGET false-positive and true-positive rates;
• mean and tail policy regret; and
• the seed, scenario parameters, schema, and implementation hash needed for replay.

The research test suite reruns small fixed-seed cases to verify deterministic replay, while release verification compares hashes of the full generated artifacts. No single simulation setting establishes universal calibration. The suite is an executable failure detector and should expand with every new supported design.

10. External Hillstrom Benchmark#

10.1 Dataset#

The external benchmark uses Kevin Hillstrom's 2008 MineThatData Email Analytics and Data Mining Challenge. It contains 64,000 customers randomized approximately equally among Mens Email, Womens Email, and No Email. The two-week outcomes include visit, conversion, and spend. Pretreatment features include purchase history, recency, prior category behavior, channel, geography, and new-customer status.

The dataset is downloaded at benchmark time and is not redistributed by this repository.

10.2 Protocol#

The benchmark treats No Email as control and evaluates each email campaign as a binary challenger. Aggregate effects are computed on the full randomized contrast. A deterministic assignment-stratified 60/40 split learns a categorical rule from history_segment on the training rows and evaluates it only on held-out rows. Reported policy and uniform-action values use Horvitz–Thompson inverse-propensity scores with the empirical randomized propensity; policy-minus-baseline uncertainty is computed from paired held-out scores. This protocol validates a narrow honest categorical-policy path. It does not validate the optional gradient-boosted backend or claim double robustness.

10.3 Leakage controls#

visit, conversion, and spend are outcomes and cannot be policy features. Treatment label is used only to estimate training-arm means and construct held-out scores. The split and seed are fixed. The checksum is verified before analysis. Benchmark tuning after seeing final scores must create a new protocol revision rather than overwrite the existing result.

10.4 Scope#

Hillstrom is one historical email experiment with a short-horizon spend outcome. It demonstrates reproducibility on real randomized data; it does not establish transportability to SaaS trials, lead qualification, contemporary ecommerce, long-term LTV, or any client's population.

11. Empirical Results#

11.1 Seeded product scenarios#

The bundled scenarios are deterministic demonstrations with known construction. They are useful regression fixtures, not estimates of client impact.

11.2 Calibration results#

The versioned calibration summary and its JSON companion are the source of truth. Each of six scenarios uses 500 fixed-seed replications; original-HEAD and current rules receive identical simulated draws.

• false hard actions under a two-arm global null fell from 47.6% to 0.2%;
• under a 20-segment global null they fell from 58.0% to 0.2%, while false TARGET fell from 11.4% to 0%;
• harmful SHIP in the noisy multi-arm adversary fell from 41.6% to 0%; and
• focus-effect 95% CI coverage was 94.6% for the two-arm null and 94.2% for the 20-segment null.

These are Monte Carlo operating characteristics for declared synthetic designs, not production lift or a universal guarantee. The honest gate trades some action rate for safety: correct TARGET in the heterogeneous scenario was 79.8% under the current rule versus 99.6% under the optimistic original rule.

11.3 Hillstrom results#

The versioned Hillstrom summary and its JSON companion are the source of truth. On the full randomized contrasts, Men's Email increased spend by $0.7698 per customer (95% CI [$0.4851, $1.0545]) and Women's Email by $0.4244 (95% CI [$0.1690, $0.6799]); both passed the two-treatment BH adjustment.

The held-out targeting result is deliberately more modest. The Men's rule treated 100% and therefore added exactly $0 over uniform Men's Email. The Women's rule treated 77.8% and was −$0.0223 per customer versus the training-selected uniform action (95% CI [−$0.1639, $0.1194]). The benchmark therefore supports aggregate campaign value in this experiment but provides no evidence that the simple history_segment policy improves on uniform rollout.

11.4 Interpretation rule#

An observed percentage is described as:

• synthetic when produced by a seeded scenario;
• calibration when averaged over a known simulation design; or
• external benchmark when estimated from Hillstrom.

No number from these artifacts is described as expected lift for a future customer deployment.

12. Correctness & Quality Assurance#

12.1 Browser regression suite#

The Node suite covers:

• known values for distribution and testing primitives;
• value-based MDE and observed-effect sensitivity;
• multiplicity behavior;
• deterministic scenario generation;
• CSV parsing and validation;
• cross-fitted categorical targeting;
• low-evidence multi-arm selection;
• null and unequal-arm adversarial cases;
• tiny-arm and zero-variance hard-action rejection;
• money-model arithmetic; and
• compilation of both inline scripts plus the bundled 4,800-row sample.

12.2 Backend suite#

The backend tests cover:

• ordinary randomized two-arm analysis;
• deterministic cross-fitting;
• finite policy scores and intervals;
• uniform-treat and uniform-control estimator identities;
• policy improvement relative to paired baselines;
• small and degenerate arm rejection;
• invalid values and treatment labels; and
• endpoint response shape.

12.3 Research artifact checks#

Research tests verify fixed seeds, result schema, data checksum handling, benchmark determinism, and minimum-sample boundaries. A dependency-free Node bridge loads the shipped engine from index.html; fixed aggregate and segmented fixtures must match the Python calibration mirror on decisions, effects, intervals, selected segments, and policy contrasts. The release process runs browser, backend, and research suites separately so an optional backend failure cannot be hidden by a green browser demo.

12.4 Evidence hierarchy#

13. Privacy, Security & Reproducibility#

13.1 Browser privacy#

The MVP reads CSV files through FileReader, holds rows in memory, and makes no application-data network request. Dynamic user strings are inserted with textContent. There is no localStorage, sessionStorage, eval, or dynamic function construction. Google Fonts remain the only optional network dependency and have a system-font fallback.

13.2 Backend boundary#

The optional API processes rows in memory and does not persist them. Production deployment still requires authenticated access, TLS at the ingress, strict CORS, rate limiting, bounded concurrency, logging that excludes row data, a verified request-body limit, dependency scanning, and a retention policy. A demonstration default is not a production security posture.

13.3 Determinism#

Determinism is provided by:

• fixed scenario and simulation seeds;
• deterministic fold assignment;
• fixed model random states;
• pinned top-level Python dependencies;
• a checksum-pinned external dataset; and
• machine-readable result artifacts.

Exact floating-point identity across different architectures is not guaranteed. Decision and metric tolerances are encoded in tests where appropriate.

13.4 Reproduction contract#

A result is reproducible only if the repository revision, software environment, dataset checksum, configuration, seed, and command are recorded. Screenshots and prose summaries are outputs, not evidence sources.

14. Comparison to Prior Art#

TrueLift's differentiator is the composition: conversion–value diagnostics, value-aligned power, adjusted uncertainty gates, honest policy evaluation, and executable evidence in a client-private system.

15. Limitations & Future Work#

• Large-sample browser inference. Value p-values and intervals rely on approximations that can be fragile for small, extremely skewed, or heavy-tailed samples. Future work: studentized randomization inference, bootstrap intervals, and robust mean estimators with simulation coverage.
• No sequential monitoring. The browser engine does not implement always-valid p-values or alpha spending. Future work: prespecified group-sequential or e-value support with explicit stopping rules.
• No built-in SRM or identity checks. The CSV lacks expected allocation, unit id, cluster id, and outcome-maturity metadata. Future work: a versioned experiment manifest and integrity gate.
• Categorical browser policy only. Segment labels must be pretreatment and prespecified. Cross-fitting evaluates the learning procedure, after which the displayed deployable rule is refit on all rows; that exact fixed rule is not independently held out. Future work: a nested final holdout, constrained policy classes, and fairness/governance review.
• Approximate policy intervals. Browser policy intervals condition on the fold-specific learned rules and use large-sample cell variances; they do not fully propagate training-rule instability or dependence induced by shared folds. The reported Monte Carlo coverage is design-specific. Future work: repeated sample splitting, nested bootstrap or randomization inference, and explicit policy-learning confidence procedures.
• Post-hoc sensitivity and no aggregate MIED. The 80% observed-effect sensitivity gate is a deterministic function of the estimate and standard error. Aggregate actions use zero as the practical threshold. Future work: a prespecified minimum economically important difference and blinded variance planning.
• Outcome-selected binary backend focus. The backend selects the highest-conversion challenger on the full request, then reports inference for that pair without selection adjustment. Future work: prespecified focus arms or multi-treatment DR policy learning with selection-aware multiplicity and overlap diagnostics.
• Model dependence. Cross-fitting and double robustness reduce specific biases; they do not rescue arbitrary model failure, weak overlap, treatment leakage, or post-treatment features.
• Value measurement. Long-term value is delayed, censored, and often proxied. Future work: survival-aware and proxy-calibrated outcomes with prespecified validation.
• One external benchmark. Hillstrom is reproducible but narrow. Future work: multiple public randomized datasets and prospective client replications.
• No transport claim. A policy learned in one experiment should not be deployed to a changed population without monitoring and revalidation.
• Operational costs omitted. Current value differences do not automatically subtract implementation, targeting, or experimentation costs unless the supplied outcome already includes them.

16. Conclusion#

TrueLift operationalizes a simple correction to conversion optimization: evaluate the value created per randomized unit, not the count of funnel events. The research contribution is the discipline around that objective. Aggregate effects are paired with their uncertainty; power is defined on value; multi-arm rank cannot bypass evidence; targeting rules are evaluated out of fold; model-based policy value uses cross-fitted doubly robust scores; and every hard recommendation has an explicit gate.

This does not make experimentation automatic. It makes the system more honest about what an experiment can support. A descriptive trap can motivate investigation. A promising positive effect can justify extending a test. A targeting pattern can become a deployable rule only after honest policy evaluation. A benchmark can support a research claim only when its source, checksum, protocol, code, and output are reproducible.

The result supports controlled research and demonstrations, remains deliberately smaller than a production experimentation platform, and lets future evidence improve the method without rewriting its claims after the fact.

17. Appendices#

Appendix A: Reproducibility#

Browser engine:

node tests/engine.test.mjs

Backend:

python3 -m venv .venv
source .venv/bin/activate
pip install -r backend/requirements.txt
pip install pytest
pytest -c backend/pytest.ini

Monte Carlo calibration:

python3 research/calibration.py \
  --trials 500 --seed 20260723 --output-dir research/results

Hillstrom benchmark:

python3 research/hillstrom_benchmark.py \
  --data /path/to/Kevin_Hillstrom_MineThatData_E-MailAnalytics_DataMiningChallenge_2008.03.20.csv \
  --output-dir research/results --seed 20260723 --train-fraction 0.60 \
  --require-completed

Research harness tests:

python3 -m unittest discover -s research/tests -v

To fetch from the publisher rather than use a verified local copy, replace the Hillstrom --data option with --attempt-download. The runner verifies the pinned checksum before parsing and fails under --require-completed if no verified dataset is available.

Appendix B: Decision pseudocode#

analyze aggregate value contrasts
adjust the declared comparison family
compute value MDE / observed-effect sensitivity

if cross-fitted learning-procedure lower bounds > policy epsilon
   versus both uniform actions
   and fold-specific segment evidence passes adjustment
   and the full-data deployable rule selects a nontrivial subset:
       TARGET
else if selected treatment value lower bound > 0
   and selected comparison survives adjustment
   and arm-size floor and 80% observed-effect sensitivity pass:
       SHIP
else if selected treatment value upper bound < 0
   and selected comparison survives adjustment
   and arm-size floor and 80% observed-effect sensitivity pass:
       DON'T SHIP
else:
       EXPLORE

Appendix C: Glossary#

• ATE: Average treatment effect on the prespecified value outcome.
• CATE: Conditional average treatment effect given pretreatment covariates.
• Cross-fitting: Rotating sample splitting so each unit is scored by models and policies trained without its outcome.
• Doubly robust (DR): An estimator combining outcome and assignment models that remains consistent when one of the two nuisance components is correctly specified, under regularity conditions.
• FDR: False discovery rate, the expected proportion of false rejections among rejections.
• IPW: Inverse propensity weighting.
• ITT: Intention to treat; analysis by randomized assignment.
• MDE: Minimum detectable effect under a declared design, error rate, and target power.
• Policy value: Expected value if a treatment-assignment rule were followed.
• Regret: Difference between oracle policy value and learned policy value.
• SRM: Sample-ratio mismatch between expected and observed assignment counts.
• Value per visitor/unit: Mean downstream value over every randomized unit in an arm, including zero-valued outcomes.

Appendix D: Document history#

References#

All URLs accessed July 2026.

1. Cloud Ambassadors / HitBase Research. Climbing the Wrong Hill: Value-Weighted Search for the Funnel You Actually Need. Version 1.0 draft, June 15, 2026. https://hitbase-website-196053730058.us-central1.run.app/research/Business-Intelligence
2. Welch, B. L. “The Generalization of ‘Student's’ Problem when Several Different Population Variances Are Involved.” Biometrika 34(1–2), 1947, 28–35. https://doi.org/10.1093/biomet/34.1-2.28
3. Benjamini, Y., and Hochberg, Y. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society: Series B 57(1), 1995, 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x
4. Athey, S., and Imbens, G. “Recursive Partitioning for Heterogeneous Causal Effects.” PNAS 113(27), 2016, 7353–7360. https://doi.org/10.1073/pnas.1510489113
5. Künzel, S. R., Sekhon, J. S., Bickel, P. J., and Yu, B. “Metalearners for Estimating Heterogeneous Treatment Effects Using Machine Learning.” PNAS 116(10), 2019, 4156–4165. https://doi.org/10.1073/pnas.1804597116
6. Dudík, M., Langford, J., and Li, L. “Doubly Robust Policy Evaluation and Learning.” Proceedings of ICML, 2011. https://arxiv.org/abs/1103.4601
7. Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. “Double/Debiased Machine Learning for Treatment and Structural Parameters.” The Econometrics Journal 21(1), 2018, C1–C68. https://doi.org/10.1111/ectj.12097
8. Athey, S., and Wager, S. “Policy Learning With Observational Data.” Econometrica 89(1), 2021, 133–161. https://doi.org/10.3982/ECTA15732
9. Wager, S., and Athey, S. “Estimation and Inference of Heterogeneous Treatment Effects Using Random Forests.” Journal of the American Statistical Association 113(523), 2018, 1228–1242. https://doi.org/10.1080/01621459.2017.1319839
10. Fabijan, A., Dmitriev, P., Olsson, H. H., and Bosch, J. “Diagnosing Sample Ratio Mismatch in Online Controlled Experiments.” Proceedings of KDD, 2019. https://doi.org/10.1145/3292500.3330722
11. Hillstrom, K. The MineThatData E-Mail Analytics and Data Mining Challenge. 2008. https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html

Context. Prepared by Cloud Ambassadors / HitBase Research. Conceptual foundation: Climbing the Wrong Hill: Value-Weighted Search for the Funnel You Actually Need.