6. Test before launch

An optimization starts as a draft. Nothing is served to real users until you set it live, so this is the time to check the setup end to end. The steps below work the same with your own warehouse and with the Managed Warehouse.

1. Check the user id

Most setup problems come from the id. The id in your exposure table and in your reward metric must be exactly the id your app passes to the SDK. If your app calls the SDK with a user id but your table's id column holds a device id, nothing fails, but rewards match no exposures and the holdout comparison is wrong.

Run:

levered verify <optimization-id>

It reads the last 14 days of your exposures table (--days changes that) and reports two shares:

The second check runs only when your exposures table has an is_holdout column. Both need some exposures in the table, from another optimization or from the test events in step 3.

When representative non-test data is present, a low share or a query error must be resolved before launch. If there are no representative non-test exposures yet, including when the only exposures belong to QA users whose rewards are intentionally excluded by the reward metric, record the aggregate identity check as pending. Verify raw exposure and reward arrival for each test user below, then repeat the aggregate check when representative data arrives. A pending aggregate check does not mean overall verification passed; it is not a reason to change a valid id mapping. Query errors and failed holdout checks must still be fixed, including for QA users.

If a share is low with representative data, fix the column, then run levered verify again. To set which columns of your exposures table hold the id, the time and the variant, see Use your own column names.

2. Find out what a test user gets

Use fresh test user ids for each run so earlier events cannot satisfy the arrival check. A draft doesn't serve real traffic, but a dry run does everything a real serve does (holdout assignment and variant selection) and records nothing on Levered's side:

levered serve <optimization-id> --anonymous-id <test-user-id> --dry-run --json

The same works over HTTP with "dry_run": true on /serve. The response shows the variant and is_holdout.

Testing the holdout. When the holdout is 0%, including --control none, use one ordinary test user and continue to step 3. With a positive holdout percentage, whether a user is in the holdout depends only on their id, so you can't put a test user there. Run the dry run for your test accounts' ids and keep the ones that come back with is_holdout: true. With a 10% holdout, about one in ten does. The answer never changes for the same id and optimization, so once you have found a holdout test user, it stays one.

3. Send test events

A dry run means Levered stores nothing. Your app's own tracking still logs as usual, and it has to, because that is what you are testing:

1. Render the variant from the dry-run response (or the holdout experience when is_holdout is true).
2. Log the exposure the way your app always does, with the same test user id, the factor values and is_holdout.
3. Trigger the conversion, so your pipeline logs the reward with the same id.

With your own warehouse, those rows land in your tables. On the Managed Warehouse, send them to the ingestion API and mark the test reward with "properties": {"test": true}.

4. Check that the events arrived

Managed Warehouse:

levered verify <optimization-id> --anonymous-id <test-user-id> --json

Require exposure: true and reward: true for event arrival. This command also runs the aggregate identity check: identity contains its result, while top-level ok combines event arrival with identity.ok. Inspect the warnings and errors even when both events arrived.

A false top-level ok or exit status 1 may be recorded as identity pending only when its sole cause is the lack of representative non-test data or reward overlap from intentionally excluded QA traffic described in step 1. Missing events, query or authentication errors, a failed holdout check, or identity failures with representative data still block launch. A pending result is not an overall pass.

For an arrival-only read, authenticated GET /api/v2/warehouse/verify with the same optimization_id and anonymous_id returns the two booleans without running identity. It does not clear a pending or failed identity check.

Your own warehouse: derive the arrival checks from the optimization's configured sources:

1. Read levered optimizations show <optimization-id> --json for the linked assignment and reward metric IDs. Fetch each metric's saved SQL and column mapping with authenticated GET /api/v2/warehouse/metrics/<metric-id>.
2. Use levered warehouse query '<SQL>' --json for two read-only checks against those actual tables and mapped visitor/timestamp columns. Scope the exposure to this optimization, the reward to the chosen goal, and both to the same test visitor and test run. Preserve the configured organization and optimization filters, use the provider's SQL syntax, and resolve time parameters from the run.
3. Read raw test rows even when training excludes them, relaxing only the deliberate QA exclusion for these targeted checks. Confirm the id and is_holdout match the dry run, timestamps belong to this run, and factors match the rendered variant. External holdout rows may have empty factors. Confirm the chosen reward arrived for that same visitor and run.

Repeat levered verify <optimization-id> when representative non-test data is available; intentionally excluded test rewards cannot establish aggregate id overlap.

5. Keep test users out of the results

Test users who keep using the app after launch are counted like everyone else. For a handful of users that doesn't matter. To leave them out completely, filter their ids (or a test marker your pipeline already sets) in both:

• the reward metric, for example ... AND USER_ID NOT IN ('qa-1', 'qa-2');
• the optimization's exposure source (its assignment source), with the same filter.

Change either with PUT /api/v2/warehouse/metrics/:id. On the Managed Warehouse, metrics that leave out properties.test already exclude test rewards.

6. Go live

Once raw event delivery passes and every runnable identity check passes, record any aggregate check still pending representative data, then go live:

levered optimizations update <optimization-id> --status live

Then deploy the version of your app that serves the optimization.