Holdout

The holdout is a random share of users that the model does not optimize for. Levered compares them with everyone else to measure what the optimization adds (measured lift). The share is set when you create the optimization: 10% by default, anything from 0 to 50%.

Who is in the holdout

Levered decides it from the id your app passes to the SDK (anonymousId) or to /serve (anonymous_id). Nothing is stored and nothing is random at serve time:

1. Take the SHA-256 hash of <optimization id>:<id>.
2. Read its first 4 bytes as an unsigned big-endian integer, modulo 100.
3. The user is in the holdout when that number is below the holdout percentage.

So for one optimization the same id always gets the same answer, about one id in ten is in a 10% holdout, and you can't pick who is in it. Different optimizations split users independently.

Every serve response says which group the user is in: is_holdout from /serve, isHoldout in the SDK's exposure event.

What holdout users see

You choose this when you create the optimization, as its control type: with --control on levered optimizations create, or control_type in the API.

variant (the default): one of the variants is today's experience. That variant is the baseline, and the holdout is served it. By default it's the variant with every factor at its first level; if today's experience is another combination, name it with --baseline '{"title":"b","cta":"a"}' (baseline_variant in the API, one level for every factor). For a holdout user the SDK returns the baseline with isHoldout: true, and your app renders it like any other variant.

external: the holdout keeps a version that is none of the variants, for example a legacy component while every variant uses a redesign:

levered optimizations create ... --control external --holdout 10

• Your app renders the current version whenever isHoldout is true and ignores the returned variant.
• Log the exposure as usual, with is_holdout = true. Leaving the factor values empty (NULL) is optional; it keeps your own analysis clean.
• Levered leaves holdout users out of training by the hash of their id, whatever the exposure row says, so the current version never counts towards a variant.
• Measured lift compares the users the model served with the holdout, so it's the gain over the current version.
• It needs a holdout above 0%. The control type and the holdout percentage can only be changed while the optimization is a draft.

none: every variant is new. Nothing existed before, for example a new screen or flow. Create it with --control none (or --no-baseline): there is no holdout and no guardrails, and lift is measured against an even split of traffic across the variants.

Training and measured lift

• Training uses every exposure whose factor values are levels of the optimization. With control type variant, that includes holdout users: they saw the baseline, so their outcomes are real evidence for it. With external, holdout users never train the model.
• Measured lift compares users in the holdout with users the model served, one row per user, grouped by the hash above. See Measured lift for the estimator and a query that reproduces it.

Logging holdout exposures

Log an exposure for holdout users like for everyone else, and add an is_holdout column:

user_456 is a holdout user who saw the baseline (control type variant). user_789 is a holdout user of an external optimization who saw the current version; the empty factor values are optional.

Levered doesn't need is_holdout to compute lift, but it makes your own analysis easier, and levered verify uses it to check that your id column is the id the SDK was called with: almost every is_holdout = true row should hash into the holdout.