A/B Testing Guide

How A/B Testing Works #

1. Create Variants – Design different versions of your bar or popup (e.g., different headlines, colors, or offers).
2. Set Traffic Split – Allocate a percentage of traffic to each variant (e.g., 50/50 for two variants).
3. Enable Testing – Turn on A/B testing for the campaign.
4. Collect Data – The system automatically tracks visitors, impressions, clicks, and conversions for each variant.
5. Analyze Results – View conversion rates and statistics in the Statistics tab.
6. Apply Winner – Use the highest-performing variant with one click.

The A/B Testing Dashboard #

A/B Testing is available for both Announcement Bars and Popup Campaigns. Open the editor for any bar or popup and navigate to the A/B Test tab. The dashboard has two sub-tabs:

Tab	Description
Setup	Configure your A/B test: enable testing, create and manage variants, set traffic splits, and customize variant settings
Statistics	View real-time performance data: overview metrics, per-variant breakdown, and winner detection

The Setup tab shows your Control variant (the original campaign) and any test variants you’ve created. The Statistics tab becomes active once the A/B test is enabled and the campaign is saved.

Setting Up an A/B Test #

Step 1: Create Your Campaign #

First, create your announcement bar or popup campaign as normal. Configure all settings (content, design, targeting, triggers, etc.). This becomes your Control variant — shown in the Setup tab as “Control Variant (Original)” with a green checkmark. Visitors not assigned to test variants will see this version.

Step 2: Enable A/B Testing #

In the A/B Test tab, toggle Enable A/B Testing to On. This activates the testing controls and unlocks the Statistics tab.

Step 3: Create Variants #

Click the “Add Variant” card to create test variants. Each variant appears as a card in the Setup tab showing:

• Variant Name – Editable name for your reference
• Traffic Split – Slider to set the percentage of visitors who see this variant
• Customized Fields – Badge showing how many fields differ from the control
• Edit button – Opens the variant settings editor
• Delete button – Removes the variant

Each variant can have different:

• Message text or headline
• CTA button text, color, and URL
• Background and text colors
• Countdown timer settings
• Coupon code settings

Step 4: Customize Variant Settings #

Click “Edit” on a variant card to open the variant settings editor. The editor has two tabs:

Tab	Settings
Content	Message text, CTA button (text, URL, target), multiple messages support
Design	Background color, text color, countdown timer style, coupon code toggle

Any field you don’t change inherits the value from the Control variant. A badge on each variant card shows how many fields have been customized.

Step 5: Configure Traffic Split #

Use the traffic split slider on each variant card to set the percentage of visitors who see that variant. The system assigns visitors consistently — once assigned to a variant, they always see the same one on return visits. For example:

• 2 variants: 50% / 50%
• 3 variants: 33% / 33% / 34%

Step 6: Save and Publish #

Click “Save Changes” to save your A/B test configuration, then publish your campaign. The test begins immediately once the campaign is live.

A/B Test Statistics #

Switch to the Statistics tab to view real-time performance data. The statistics auto-refresh every 30 seconds, or click the Refresh button to update manually.

Overview Metrics #

The top of the Statistics tab shows four summary metrics across all variants:

Metric	Description
Total Visitors	Unique visitor sessions assigned to any variant
Impressions	Total number of times variants were displayed
Clicks	Total clicks/interactions across all variants
Conversions	Total form submissions or goal completions

Variant Performance Table #

Below the overview, a detailed table breaks down performance per variant:

Column	Description
Variant	Variant name with badges — Control (gray) for the original, Winner (gold, with trophy icon) for the statistically significant winner
Impressions	Number of times this variant was shown
Clicks	Number of clicks/interactions for this variant
Conversions	Form submissions or goal completions for this variant
Click-Through Rate	Percentage of impressions that resulted in clicks
Conv. Rate	Percentage of impressions that resulted in conversions

Winner Detection #

HashBar automatically identifies the best-performing variant using statistical significance testing (z-score analysis at 95% confidence).

How Winner Detection Works #

When a statistically significant winner is found, the Statistics tab displays a gold Winner alert showing:

• The winning variant name with a trophy icon
• The improvement percentage over other variants
• The confidence level of the result

In the variant table, the winning variant receives a gold “Winner” badge next to its name.

Confidence Levels #

Level	Requirement	Meaning
High	1,000+ total impressions	Strong statistical confidence — safe to apply winner
Medium	500–999 total impressions	Moderate confidence — results are likely reliable
Low	Under 500 impressions	More data needed — wait for higher confidence

Minimum Sample Size #

The system requires at least 100 impressions before calculating statistical significance. Wait until you reach this threshold before making decisions.

Applying the Winner #

Once a winner is identified with sufficient confidence:

1. Go to the Statistics tab in the A/B Test dashboard.
2. Review the variant performance data and winner alert.
3. Click “Apply Winner” to make the winning variant your active campaign.

Persistent Variant Assignment #

When a visitor is assigned to a variant, they always see the same variant on return visits. This ensures:

• Consistent experience – Visitors don’t see different versions on each visit
• Accurate data – Results reflect true variant performance without cross-contamination
• Fair comparison – Each variant gets a clean, unbiased set of visitors

What to Test #

Here are the most impactful elements to test:

Element	Example	Impact
Headlines	“Save 20%” vs “Get 20% Off”	High – headlines are the first thing visitors see
CTA Button Text	“Shop Now” vs “Get the Deal”	High – directly affects click rates
Colors	Blue button vs Red button	Medium – affects visual attention
Images	Product photo vs Lifestyle photo	Medium – affects emotional response
Offers	“20% off” vs “Free Shipping”	High – directly affects perceived value
Timer Style	Digital vs Box countdown	Low-Medium – affects urgency perception

Best Practices #

1. Test one variable at a time. If you change the headline AND the color, you won’t know which change made the difference.
2. Wait for sufficient data. Don’t declare a winner too early. Wait for at least High confidence (1,000+ impressions) for reliable results.
3. Start with high-impact elements. Test headlines and offers first – these typically have the biggest impact on conversions.
4. Run tests for at least 1-2 weeks. This accounts for day-of-week and time-of-day variations in your traffic.
5. Keep testing. After finding a winner, create a new test with the winner as the control and a new challenger variant.
6. Document your tests. Keep notes on what you tested, the results, and what you learned for future reference.

Related Guides #

• Analytics Guide – Understand your campaign metrics before running A/B tests
• Announcement Bar Editor Guide – Create announcement bar variants
• Popup Campaign Editor Guide – Create popup campaign variants
• Use Cases & Recipes – Real-world A/B testing examples

That’s it! You now have everything you need to run effective A/B tests with HashBar. If you need additional assistance, feel free to contact our support team.