Specify the decision before the experiment.
Choose what you intend to learn: whether one lesson improves a task family, whether an update should replace existing guidance, or whether a policy change clears an agreed margin.
Define eligible traffic, stable task identities, the final outcome, and the treatment. Decide which guidance can be withheld and which mandatory instructions or protected safety memories must remain available.
Do not alter success criteria after seeing a favorable result. A change-control trial has a stronger contract when its acceptance criteria and rollback plan were fixed in advance.
UniMem promotion workflowMake assignment and delivery agree.
A memory-level holdout assigns an eligible memory to shown or withheld for an occasion. The agent must actually receive the shown set and must not recover the withheld instruction through a second path.
Use an occasion ID for a real task, not every retriever call. Retries of the same task should not be presented as independent experiments. Stable memory identities and revision tracking prevent unrelated content from being pooled under one label.
When multiple agents share state, outcomes may interfere. Task-level randomization may no longer match the causal question. Investigate clustered or other appropriate designs instead of assuming independence.
Missing outcomes are part of the experiment.
Record the result of every assigned task, including failures, timeouts, escalations, and abandoned work according to the agreed definition. If the harder arm loses more outcomes, the comparison can be biased.
A model-generated judgment is not automatically a reliable final label. Calibrate it against independent evidence, separate surrogate signals from final outcomes, and keep simulated observations separate from real ones.
The instrumented path should make it possible to reconcile assignments with outcomes. In UniMem, integrity checks examine issues such as attrition, imbalance, changed randomization, scorer drift, and edited memory revisions.
Read uncertainty as part of the result.
An effect estimate without uncertainty can look more decisive than the data supports. The interpretation also depends on the estimation mode and the assumptions behind it.
UniMem reports distinct verdicts for help, harm, no measurable effect, and underpowered evidence. Underpowered is not proof of no effect. Repeated inspection and adaptive allocation need methods appropriate to those choices; the source documents mode-specific qualifications.
Avoid turning an internal trial into a universal benchmark claim. The result concerns the measured population, memory revision, assignment process, and outcome definition. A new workflow or deployment can require a new evaluation.
Estimator modes and qualificationsUse the result to make a reviewable decision.
A useful evaluation ends with an operational choice. Continue collecting evidence when it is insufficient. Investigate and withdraw harmful guidance under the configured policy. Review a replacement rather than silently editing the treatment mid-trial.
For policy promotion, preregister the candidate and incumbent, drain outstanding outcomes, and evaluate the exact canonical trial. Keep the rollback route available after release.
Memory evaluation is ongoing engineering work. It should make the next decision better informed and easier to inspect, while leaving uncertainty visible.
Put experience to work.
Try the local loop, or bring one workflow to a pilot conversation.
Technical details link to the inspected source. Check deployment requirements before enabling optional features.