A/B Testing Guide

How A/B Testing Works

1. Create Variants – Design different versions of your bar or popup (e.g., different headlines, colors, or offers).
2. Set Traffic Split – Allocate a percentage of traffic to each variant (e.g., 50/50 for two variants).
3. Enable Testing – Turn on A/B testing for the campaign.
4. Collect Data – The system automatically tracks visitors, impressions, clicks, and conversions for each variant.
5. Analyze Results – View conversion rates and statistics in the Statistics tab.
6. Apply Winner – Use the highest-performing variant with one click.

The A/B Testing Dashboard

A/B Testing is available for both Announcement Bars and Popup Campaigns. Open the editor for any bar or popup and navigate to the A/B Test tab. The dashboard has two sub-tabs:

The Setup tab shows your Control variant (the original campaign) and any test variants you’ve created. The Statistics tab becomes active once the A/B test is enabled and the campaign is saved.

Setting Up an A/B Test

Step 1: Create Your Campaign

First, create your announcement bar or popup campaign as normal. Configure all settings (content, design, targeting, triggers, etc.). This becomes your Control variant — shown in the Setup tab as “Control Variant (Original)” with a green checkmark. Visitors not assigned to test variants will see this version.

Step 2: Enable A/B Testing

In the A/B Test tab, toggle Enable A/B Testing to On. This activates the testing controls and unlocks the Statistics tab.

Step 3: Create Variants

Click the “Add Variant” card to create test variants. Each variant appears as a card in the Setup tab showing:

• Variant Name – Editable name for your reference
• Traffic Split – Slider to set the percentage of visitors who see this variant
• Customized Fields – Badge showing how many fields differ from the control
• Edit button – Opens the variant settings editor
• Delete button – Removes the variant

Each variant can have different:

• Message text or headline
• CTA button text, color, and URL
• Background and text colors
• Countdown timer settings
• Coupon code settings

Step 4: Customize Variant Settings

Click “Edit” on a variant card to open the variant settings editor. The editor has two tabs:

Any field you don’t change inherits the value from the Control variant. A badge on each variant card shows how many fields have been customized.

Step 5: Configure Traffic Split

Use the traffic split slider on each variant card to set the percentage of visitors who see that variant. The system assigns visitors consistently — once assigned to a variant, they always see the same one on return visits. For example:

• 2 variants: 50% / 50%
• 3 variants: 33% / 33% / 34%

Step 6: Save and Publish

Click “Save Changes” to save your A/B test configuration, then publish your campaign. The test begins immediately once the campaign is live.

A/B Test Statistics

Switch to the Statistics tab to view real-time performance data. The statistics auto-refresh every 30 seconds, or click the Refresh button to update manually.

Overview Metrics

The top of the Statistics tab shows four summary metrics across all variants:

Variant Performance Table

Below the overview, a detailed table breaks down performance per variant:

Winner Detection

HashBar automatically identifies the best-performing variant using statistical significance testing (z-score analysis at 95% confidence).

How Winner Detection Works

When a statistically significant winner is found, the Statistics tab displays a gold Winner alert showing:

• The winning variant name with a trophy icon
• The improvement percentage over other variants
• The confidence level of the result

In the variant table, the winning variant receives a gold “Winner” badge next to its name.

Confidence Levels

Minimum Sample Size

The system requires at least 100 impressions before calculating statistical significance. Wait until you reach this threshold before making decisions.

Applying the Winner

Once a winner is identified with sufficient confidence:

1. Go to the Statistics tab in the A/B Test dashboard.
2. Review the variant performance data and winner alert.
3. Click “Apply Winner” to make the winning variant your active campaign.

Persistent Variant Assignment

When a visitor is assigned to a variant, they always see the same variant on return visits. This ensures:

• Consistent experience – Visitors don’t see different versions on each visit
• Accurate data – Results reflect true variant performance without cross-contamination
• Fair comparison – Each variant gets a clean, unbiased set of visitors

What to Test

Here are the most impactful elements to test:

Best Practices

1. Test one variable at a time. If you change the headline AND the color, you won’t know which change made the difference.
2. Wait for sufficient data. Don’t declare a winner too early. Wait for at least High confidence (1,000+ impressions) for reliable results.
3. Start with high-impact elements. Test headlines and offers first – these typically have the biggest impact on conversions.
4. Run tests for at least 1-2 weeks. This accounts for day-of-week and time-of-day variations in your traffic.
5. Keep testing. After finding a winner, create a new test with the winner as the control and a new challenger variant.
6. Document your tests. Keep notes on what you tested, the results, and what you learned for future reference.

Related Guides

• Analytics Guide – Understand your campaign metrics before running A/B tests
• Announcement Bar Editor Guide – Create announcement bar variants
• Popup Campaign Editor Guide – Create popup campaign variants
• Use Cases & Recipes – Real-world A/B testing examples

That’s it! You now have everything you need to run effective A/B tests with HashBar. If you need additional assistance, feel free to contact our support team.