Oct, 13th, 2015

7 ways to call A/B tests with confidence

Calling A/B tests with confidence
We all know that A/B testing is one of the best ways to improve your conversion rate. By applying the scientific method – exposing your variation to a subset of your website visitors while maintaining a control group – you can determine what the impact of your change is likely to be.

At FIRST we’ve run hundreds of A/B tests and improved conversion rates for dozens of clients. We also subscribe to many of the leading conversion publications to keep track of the latest research and case studies of winning tests. However, we often find that these tests are declared ‘winners’ even when they might not be.

To illustrate this, here’s an example from a test we recently ran for one of our clients…

Graph displaying results from one week of A/B testing
This test ran on one of New Zealand’s highest traffic websites and consisted of the original web page plus three variations. After one week the test had been shown to thousands of visitors and each variation had resulted in hundreds of conversions. The variation shown in the red line was winning by 6.9% and our A/B testing tool gave this result a statistical confidence of 99%. Clear winner, right?

Wrong. We continued running this test for a few more weeks and look at what happened to the results…

Graph displaying A/B testing results for six weeks
The ‘winning’ variation, represented by the red line, continued to converge with the other variations until there was almost no difference between them. After six weeks the variation we initially thought was the winner was only 0.9% up on the original at 76% confidence.

The implications of this are important to realise. Significant agency and client resources can be invested in uncovering customer insights, designing new variations, running tests and hard coding winning variations, yet if results are not interpreted properly then it all falls over.

Part of the problem lies in the time it can take to realise a true, statistically valid outcome. We’re all impatient to see results, particularly for a test we’re excited about. However, most tests require more time than they are given to reach their conclusion. This is particularly the case in a country like New Zealand where a smaller population means that many sites have limited traffic.

Use a sample size calculator

To see how long your test needs to run, use a tool like Optimizely’s sample size calculator. Then use your web analytics system to determine how many visitors are likely to be exposed to your test and how many days it will take to reach your required sample size.

Sample size calculator
If the required sample size is too large, don’t worry. There are ways to design your tests that will help you to get results much, much faster.

1. Test big changes

Instead of tweaking one thing on your page, create a brand new page that incorporates many changes at once. Big, bold changes will generate results much faster and can improve your conversion rates in leaps instead of increments.

2. Test on pages closer to your conversion end point

In many cases the home page is simply too far away from your end conversion point for your variation to make a big enough difference. Instead, test pages in your purchase funnel such as your shopping cart or payment page.

3. Test things that are more likely to influence conversion

Try to find those elements of your website that really drive conversion. For example, visitors are probably more concerned about ‘free delivery’ than the colour of your button, so find and test these elements instead.

4. Track micro-conversions

Instead of basing an experiment on how the variation ultimately influences purchases, why not base it on how well it moves the visitor one step further down the funnel? You are more likely to see the influence of your variation on goals that are closer to where the test takes place.

5. Track groups of goals together

If you have a contact page with a contact form, email addresses and social media buttons, don’t track these individually. Instead create a single goal that is triggered by clicks on any of these elements. That way, you’ll consolidate the impact of the test on one single goal instead of diluting it among many.

6. Filter out visitors not exposed to your variation

Testing a pop-up window? Then make sure visitors are only opted into the test when they click the button that opens the window – not before. This will stop you diluting your results with visitors that aren’t actually get influenced by your variation. Use precise URL targeting or specific techniques provided by your A/B testing platform to activate experiments exactly when you need.

7. Accept more risk

A confidence level of 95% is considered the industry benchmark. However, decreasing the confidence level will reduce the time it takes to run a test. Consider whether this increased risk is compensated by the increased speed at which you can run and call tests.

FIRST are conversion experts

Regardless of the amount of traffic on your site, a smart conversion optimisation consultant will be able to design tests that maximise impact with minimal time. If you’re interested in taking your A/B testing program to the next level, contact the team at FIRST.

No responses yet

Oct, 9th, 2015

Ad blockers: a change in power in the digital advertising ecosystem?

Ad blocking has been around for over 10 years but has made its way into the spotlight recently – when Apple announced that it will integrate ad blocking capability on mobile devices in its latest operating system.

Various apps, such as Purify and Ad Block, shot to the top of the Most Downloads Apps list and caused quite a stir in the industry about what exactly ad blocking will mean to digital advertising – for both publishers and ad serving companies. After all, mobile advertising is continually rising with mobile adoption increasing – and such a move will impact their revenue stream.

A recent report by PageFair (PDF), however, has brought some facts that give perspective to the state of ad blocking.

Users of ad blocking is estimated to be around 200 million worldwide


Growth in ad blocking

Image source: PageFair

While adoption was relatively steady in the early part of this decade, it is in the past two years where adoption of ad blocking has seen a significant rise. To further put this number into perspective, the report estimates that the cost of ad blocking will be US $41.4 billion in 2016!

Ad blocking is actually being driven by desktop adoption, not mobile – and it’s not even close (for now)

According to the report, ad blocking on desktop PCs has a 98% share compared to that of mobile (including tablet) devices. So while the latest buzz is around the iOS adoption, it is important to keep in mind that it only accounts for a tiny fraction of ad blocked devices in use at the moment. (Editors note: it is also acknowledged in this report that mobile share will only increase.)

Makers of ad blocking software are now a powerful party in the digital advertising ecosystem – forcing larger advertisers like Google to ‘play ball’

Popular Chrome and Safari ad block extension, AdBlock – with a reported user base of 40 million – was bought by Adblock Plus (no relation to AdBlock). If you were a user of the extension, as of October 2, you would’ve been presented with the following screen:


Acceptable ads program

Image source: AdBlock

The main takeaway from such a message is that ad blocking is now being commoditised.

There are a few interesting points with this acquisition:

  • Larger ad blockers have recognised how allowing “acceptable ads” from certain publishers is a major revenue stream

Advertisers that are part of the program include Amazon and Google. Keep in mind that in Q2 2015, Google websites (which includes Search) generated US$12.4 billion (that’s approx. US$138 million a day!).

Adblock Plus states that being part of the Acceptable Ads program is free….for small to medium businesses. Their website states that they are “paid for supportive services by some larger entities”.

Hypothetically, let’s assume Google pays Adblock Plus 0.0001% of its revenue stream as above. That still equates to US$1.24 million per quarter!  And that’s only from Google.

  •  Ad blocking providers, while currently largely fragmented, will become consolidated

With such a (potential) revenue stream becoming more evident, it will only lead to more consolidation in the digital ecosystem. This could potentially be a big issue for large advertisers (such as Google).  Consolidation means more leverage and membership into programs, such as “Acceptable Ads”, could prove to be very expensive for such advertisers.

Final thoughts

At a user level, it would seem as if the idea of having their web experience free of advertising is a thing of the past with the emergence of such programs as the Acceptable Ads. The real impact will “more likely than not” be felt at the advertiser/publisher level and they’re taking action to overcome this. If so, it represents quite a shift in who holds the most power (and perhaps, leverage) in the digital advertising ecosystem. It will be interesting to see how the relationship between digital advertisers and ad blocker develops in the medium to long term.

No responses yet

Oct, 8th, 2015

The missing piece to your digital strategy is your customer

Published by under Digital Strategy,Only AU

Focus on your customer

The customer experience component of any brand simply cannot be ignored. Gone were the days wherein online marketing was relatively straightforward where you simply had a good enough website and a basic email marketing strategy in place to survive in the marketplace. However, the proliferation of devices and online platforms have radically changed the way customers interact with your brand.

Today’s digitally empowered customers go through multiple touchpoints and complex paths to purchase. As the customers’ journey becomes increasingly complex, managing customer experiences across multiple channels and devices is a big challenge for most brands. While most brands recognise the need for customer-centricity – not all understand what that actually means.

What does it mean to become customer focused?

A brand that focuses on the customer builds a business model around a deep understanding of its customers, what they value and the contribution each makes to growing the bottom line. This entails:

  • Fostering a culture that places the customer at the heart of the decision-making process – you need to ground your brand’s value proposition firmly on your customers.
  • Maintaining an active dialogue with customers and responding to feedback – following the customer journey from pre-engagement to post-purchase, constantly listening to, and learning how, different customers interact with your brand can unlock new ways to interact with customers.
  • Developing strategies and processes that recognise the needs of different customer segments – integrating multiple data sources and continuously refining customer segmentation approaches can identify and serve the customer segments that matter to you.
  • Providing a positive and seamless customer experience at every touch point across the customer’s journey – engaging customers across multiple channels, devices and platforms, gaining deep insights, and seamlessly acting on them can create differentiated customer experiences that build your brand, gain market share and competitive advantage.

The next question is: “How can you leverage your digital platforms to better engage with your customers – better and faster than your competitors?”


Harnessing the power of digital

Data analytics has become integral in designing a seamless end-to-end customer experience. It is important for you to understand what drives customer satisfaction for every customer segment. What attributes of their experience (price, accessibility, availability, response times, etc.) will have the most impact on providing a positive experience. Similarly, you need to know what is not driving value, to make the needed changes and improvements and effectively reduce cost.

For true omni-channel success, brands need a 360-degree view of the customers’ interactions across all channels, devices and platforms to monitor channel preference, customer behaviour and journeys from the customers’ point of view. Much of the analysis can now occur in real time, giving you actionable insights at the right moment.

Below is a diagram from Smart Insights that shows six steps that put the customer at the heart of every stage of your digital strategy. The article starts by emphasising the importance of understanding who your target customers are, how they interact with your brand and what they want from you. It then explains the steps you need to ultimately change how you do business.

Customer-centric marketing

Most customers want access to a mix of online and personal interaction throughout the customer journey. Digital is a critical driving force to deliver the experience customers want. As pointed out in the diagram, an effective integration of channels across marketing, sales and service activities is important in providing a positive, brand-building customer experience at every touch point in the customer relationship journey. As digital channels continue to grow in importance, you need to define a strategy for integrating these touch points as part of your brand’s overall business model.


Any strategy, digital or not, is driven by your customers’ needs. Smart business decisions are made by looking through the customer experience lens. Since digital experiences are becoming the primary type of experience for most customers, it is imperative to drive investment in getting the needed data, insights and technologies to connect brands and customers more intimately.

No responses yet

Oct, 2nd, 2015

How Google Analytics can boost your marketing efforts [infographic]

Published by under Google Analytics,Only AU

Google Analytics is one of the widely used and best free analytics tool for tracking your website’s performance.

There are 29,635,742 live websites using Google Analytics. Chances are you own one of those sites.

With a comprehensive set of features and functionalities, Google Analytics can provide you with the fresh data and insights you need to optimise your marketing efforts. But are you using this powerful tool correctly?

80% of retailers are using Google Analytics incorrectly.

Unfortunately, most marketers don’t use Google Analytics to its full potential. Looking at traffic stats, sources and pageviews doesn’t give you the whole picture of what you need to improve your website’s overall performance, generate good content, fine tune your SEO structure, and convert your visitors into paying customers.

The primary result of spending time looking at dashboards and statistics is an actionable change to improve your overall marketing strategy. Below is a good infographic from Quick Sprout that walks you through how to take the data from Google Analytics and make adjustments to your site to get more traffic and conversions.

How to improve your marketing with Google Analytics data


By using Google Analytics correctly, you can dig deeper into your marketing data and gain valuable insights you can use to meet your business goals, understand your customers’ behaviour on your site and provide a seamless end-to-end online experience on your site.

No responses yet

Sep, 28th, 2015

16 tips to increase your email open rates with smart subject lines

How to write smart subject lines

Even though the subject line is a small part of an email, it is the first impression you have on your recipients. If the email subject line doesn’t catch their attention, you won’t go any further.

Subject line writing has truly evolved over the years. Today, people need quick and easily digestible content. With the overwhelming amount of content online trying to get people’s attention, a good subject line can make or break your email content.

Some interesting numbers for you from Neil Patel:

  • 8 out of 10 people will read headline copy
  • But only 2 out of 10 will read the rest
  • A writer should spend half the entire time it takes to write a piece of persuasive content on the headline

We all know that writing email subject lines is not an exact science. There are so many factors that can affect your open rates. Below are 16 tips on crafting a click-worthy email subject line that will increase the number of people who will actually read your email.

Let’s begin!

Tip #1: KISS – Keep it Short and Straightforward

Most people quickly scan subject lines to decide if they’ll open or ignore the email. Keep your subject line to 50 characters or fewer. Some fun facts:

  • People scan headlines – take in only the first and last three words
  • Less to read so higher chance of retaining attention

I would go on and recommend to write concise and no-nonsense email subject lines. If you’re email has a specific purpose then your subject lines should be specific as well. Be careful not to go overboard with promotional and cheesy phrases.

Tip #2: Give it a sense of urgency

Using deadlines like “5 Hours Only” or “24-Hour Giveaway” will encourage your reader to act now, instead of putting it off until later when there’s a chance of them forgetting it. Email subject lines that use this scarcity appeal compel people to act or make a decision because of that inherent fear of missing out. This is a particular human behaviour most evident back in the days where people were always on survival mode. Hang on, on second thought, that’s also evident in today’s times.

A word of caution in using this scarcity appeal, make sure what you’re offering in your email is something that your recipients care about, otherwise, the time and availability limitation would be irrelevant to them.

Tip#3: Appeal to your audience’s interests

Tailor the subject lines to connect on a more personal level with your target audience. This is pretty much one of the guiding principles of online marketing. By knowing your audience, you can cater to their specific needs and tickle their fancies with the right style and tone in your subject lines.

For example, let’s say you have two primary segments in your email list – Marketing Managers and Business Owners. Each segment has different goals, interests and challenges. When you send out emails to them, make sure you use different subject lines that best appeal to their interests or address their challenges.

Marketing Managers are interested more in how-to guides and specific tactics, while Business Owners are interested more in the end result – the ROI. Here are two different subject lines that show this concept:

Marketing Manager: “Five ways to improve email open rates by writing better subject lines”

Business Owner: “Improve email marketing ROI by understanding how subject lines affect email engagement”

Tip#4: Use personalisation

It’s said that a person’s favourite word is their own name. By adding the recipient’s name, you make the person feel connected to your business. It’s a good way to start building the relationship. Say something like this: “Hi David, don’t miss out on our latest analytics update…”

Additional tip, if you’re not comfortable using names, use “you” or “your”. This gives them a feeling of being addressed directly.

Tip #5: Use localisation

You don’t have to end with incorporating the recipient’s name into the subject line. You can try personalising a message with location-specific offers and language. MailChimp research suggests including a city name is even better.

If you’re an eCommerce business, sending emails that promote discounts and deals based on the recipient’s local area or based on past purchases would be a great way to attract more opens. Generally, it’s always good practice to know and understand your target audience to be able to use the right language and offers that best resonates with them.

Tip #6: Don’t make false promises

You don’t want people clicking on ‘Unsubcribe’, right? Best way to do this is to deliver on your promise in your subject line.

For example, you created an email workflow for people who download files from your website. If your email subject line says, ‘”Your download is ready”, better make sure there is a clickable link inside where they can access what they downloaded from your website. Sounds pretty standard right? But there are still some who make false promises just to get their emails opened. Tsk… tsk.

Additional tip, say something along the lines of, “Access your case study now”, instead of a generic “Thank you”. This goes back to being concise (Tip#1); make it clear that something is inside the email waiting to be accessed.

Tip #7: Consider how your recipients will benefit from your email

Make it clear how your email will benefit your recipient. Given that you’ve done your homework of understanding your target audience – interests, goals and challenges, write a subject line that best communicates the value of your email to them. Below is an email I received from Quick Sprout that illustrates my point.

Sample of Quick Sprout's email subject line

 Tip #8: Use questions

Asking a question in your email subject line can also draw your recipients in. These types of subject lines arouse a sense of curiosity to know more about the subject and see whether it applies to them. By asking questions, you engage with your recipients in an instant dialogue and make them more likely to open your email.

Possible questions:

  • Are your landing pages converting?
  • Is your website providing the best user experience?

Tip #9: Use numbers and lists

People just love numbered lists. Similar to using questions, these types of subject lines make your emails stand out and create curiosity. Email subject lines with numbered lists make your recipients feel that the content of your email is a quick and easy read.

Tip#10: A/B testing

Heard this a million times? Improving email subject lines is a process of testing and optimising. I can’t emphasise this enough. A/B test everything, especially your email subject lines.

Tip #11: Avoid spam words

Avoid words like: Free, Bonus, Discount, % OFF, because they trigger spam filters. These words are overused and commonly associated with sales pitches. Though some may not trigger a spam filter, many recipients will choose to ignore them.

Tip #12: Don’t overuse capitalisation.

More caps does not equal more email opens. Best to capitalise your letters following grammar rules.

Tip #13: Use free subject line testing tools

Most email marketing systems have their own built-in tool for testing subject lines. SubjectLine.com is helpful when it comes to testing your subject lines. It’s a free subject line rating tool. Simply type in your subject line and click submit. You’ll get a rating for your email subject lines and a breakdown of the results. Below is an example of a subject line I tested.

Sample of Subject Line scoring email subject lines

Tip #14: Emoji or not to emoji?

Using symbols or emoji in subject lines isn’t a new idea. However, now that many of us have emoji-friendly mobile devices and regularly read our email on them, it’s become an increasingly popular tactic for attracting attention in crowded inboxes.

Word of caution though, if a character isn’t supported in the email provider of the client, the recipient will see a ☐ (empty box) character instead of the actual emoji or symbol.

Try using emojis and symbols in your next email subject line and test! Sometimes images are better than words in grabbing people’s attention.

Tip #15: Add preview texts

Preview texts provide your recipients a glimpse of your email content. Don’t underestimate the preview texts’ power to improve your open rates. Treat them the same way you treat your subject line. Below is an example preview text from one of our newsletters here at FIRST.

example preview text from one of our newsletters

Tip #16: Use negative words

In his Infographic, Patel notes that negatives tap into our insecurities. The words “No”, “Without” and “Stop” lead to more shares.


As I mentioned in the beginning of the post, there are several factors that can affect your open rates. Other factors could be:

  • Email list segmentation – have you properly qualified and segmented your email contacts database?
  • Sending frequency – how often are you sending emails? Make sure you’re not being too annoying.
  • Sending time and day – are you sending your emails on the best time and day of the week? That would depend on your target audience’s time zone, industry and background. Check this post on the best time to send email newsletters to get an idea.
  • Purpose for sending – do you have a clear grasp of why you are sending out emails? In the case of a newsletter, does you business even need one? Find out the why before the how.

As more and more content is created everyday, each piece trying to grab your audience’s attention, combat this noise by making your subject lines brief, concise and interesting to your audience. You might not come up with the perfect subject line straight away. It will take ongoing A/B testing to find out what works best for your unique audience. As you get a better understanding of what works, you can optimise your subject lines for better performance.


Enjoyed the post? Get a curated selection of digital marketing updates and insights that deserve your attention by subscribing to our monthly newsletter.

No responses yet

« Prev - Next »

© 2016 FIRST, part of the BeyondD Digital Marketing Group a wholly owned subsidiary of Beyond International.