I stopped trusting automated A/B testing after it tanked my email campaign

At a marketing conference in Portland last fall, one of the speakers convinced me to let an AI tool run all my email subject line tests. For two weeks it kept picking these overly clever puns that got high click rates but nobody actually bought anything. I went back to manually testing straightforward headlines and my conversion rate went from 1.2% to 4.8% in one week. Has anyone else found that AI testing tools optimize for the wrong metrics?

2 comments

2 Comments

mason_jackson3d ago

@troyp37 brought up a good point with that GPS comparison. I read a study from Marketing Week last year that said AI testing often focuses on easy to measure actions like clicks instead of harder to track things like real buying intent. It managed to boost your open rate maybe but sent the wrong people to your landing page. That punny subject line stuff just gets the wrong crowd excited.

troyp373d ago

Reminds me of how GPS apps send you down tiny side streets to save two minutes but miss the bigger picture.