A/B Testing

Concept facts

A/B testing is a controlled experimental method in which users are assigned to different variants so researchers or product teams can compare outcomes under defined conditions.

Also known as: Split testing, online controlled experiment, randomized experiment, variant testing
Used in: Product experimentation, conversion evaluation, content comparison, interface change assessment, service optimization, digital analytics
Interpret with: Experimental design, randomization, sample size, outcome metric, statistical uncertainty, practical significance, ethics, and user impact

Plain-language explanation

A/B testing compares two or more versions of a page, interface, message, or flow. One group of users sees one version, and another group sees a different version. The team then compares a defined outcome, such as completion, selection, submission, retention, or another measurable behavior.

A/B testing can help estimate whether a change affects a measurable outcome. It is useful when the system already has enough traffic, a clear outcome metric, and a change that can be tested safely.

A/B testing is not the same as usability testing. It can show that one variant performs differently from another, but it may not explain why users behave that way or whether the experience is understandable, accessible, or trustworthy.

Why it matters

A/B testing helps teams avoid relying only on internal preference or stakeholder opinion when comparing alternatives. It can provide evidence about whether a change affects behavior at scale.

However, it can also be misused. A small measured improvement may not mean the experience is better for all users. A variant may increase clicks while creating confusion, exclusion, or long-term trust problems.

For high-impact services, experimentation must consider ethics, consent expectations, accessibility, user harm, and whether the chosen metric reflects the real service goal.

Use contexts

Product and interface experimentation
Content and message comparison
Registration or application-flow variants
Conversion and submission analysis
Feature rollout evaluation
Digital service optimization
Large-scale behavioral measurement

Application guidance

Define the hypothesis, outcome metric, population, variant, assignment method, sample size, and decision rule before running the test. Avoid changing multiple unrelated elements unless the study design can support that interpretation.

Interpret results with statistical uncertainty and practical significance. A statistically detectable change may still be too small to matter, and a large apparent difference may be unreliable if the sample or design is weak.

Use A/B testing alongside qualitative and usability evidence where understanding, accessibility, trust, or task quality matters.

Practical example

A benefits portal tests two versions of an application start page. Version A says “Apply now.” Version B says “Check eligibility before applying.”

Version A produces more immediate clicks, but Version B reduces incomplete and ineligible applications. The better result depends on the true service goal: not maximum clicks, but accurate applications, reduced staff correction, and better applicant understanding.

Interpretive boundaries

A/B testing does not explain user reasoning by itself.
A measurable difference is not automatically a meaningful improvement.
The chosen metric can distort the interpretation of success.
A/B testing may be inappropriate for low-traffic, high-risk, or ethically sensitive services.
A/B testing should not replace usability testing when comprehension, accessibility, or task quality matters.

Applied at Userhub

A/B testing is relevant when Userhub helps interpret product or service experiments in relation to usability, task quality, accessibility, and real user outcomes.

See Userhub UX Lab for applied UX research and evaluation context.

Sources and references

Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 140–181. https://doi.org/10.1007/s10618-008-0114-1
Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.
Box, G. E. P., Hunter, J. S., & Hunter, W. G. (2005). Statistics for Experimenters: Design, Innovation, and Discovery (2nd ed.). Wiley.
Montgomery, D. C. (2017). Design and Analysis of Experiments (9th ed.). Wiley.

—

Userhub. (2026). A/B Testing. UX Reference. https://userhub.com.bd/reference/ab-testing/

Reference entry content