What is A/B testing? | Oracle
Throughout the lifecycle of any A/B test, analytics is at the heart of planning, execution, and performance recommendations.
The development of a test hypothesis requires a strong foundation in analytics. You need to understand current performance and traffic levels. In terms of web analytics (for example), there are some key data points that your analytics system will provide during the planning process, including:
- Traffic (page views, unique visitors) to the page, component, or other element being reviewed for test scenarios
- Engagement (time spent, pages per visit, bounce rate)
- Conversions (clicks, registrations, fallout)
- Performance trended over time
Without this grounding in analytics, any test scenario or performance assessment will likely be based on personal preferences or impressions. Testing will often prove those assumptions to be incorrect.
Once an A/B test launches, analytics also plays a central role. A dashboard is used to monitor performance metrics in real time, to validate the test is operating as expected, and to respond to any anomalies or unexpected results. This can include stopping the test, making adjustments and restarting, and ensuring performance data reflects any changes as well as the timing of those changes. The performance dashboard helps determine how long to keep the test running and to ensuring that statistical significance is achieved.
After the test has run its course, analytics are the basis for determining next steps. For example, they can be used to decide if the test’s winner becomes the standard presentation on the website page that was tested and whether it becomes an ongoing standard. Marketers should develop a reusable analytics template to convey test results and adapt that template to reflect the specific elements of a given test.
