A/B tested two AI email subject line tools last month, one flopped hard

I run a small online store selling handmade candles out of Austin Texas. Last month I tried Jasper for writing email subject lines and then switched over to Phrasee for a week each. Jasper gave me lines that sounded smart but nobody opened them, my open rate dropped to 12 percent. Phrasee gave me shorter, simpler lines that felt more like a real person talking. Those got opened 28 percent of the time, way better for my list of 2,000 subscribers. The big difference was Phrasee checking the lines against my past campaign data while Jasper just guessed based on general rules. Has anyone else seen big gaps between these kinds of AI tools when you actually test them side by side?

2 comments

2 Comments

webb.jordan1mo ago

Did you check if the two tools were analyzing the same audience segments, or did Jasper just treat your whole list the same? That 12% to 28% swing is huge, but it could also be timing or day of week differences messing with the results. Would be curious to know if you sent both batches out at the same time on the same days.

lisar141mo ago

Could be that Jasper was optimizing for cleverness while Phrasee optimized for familiarity, since shorter lines feel less risky to scroll past in an inbox. Another thing nobody mentions is how different AI tools handle emojis or special characters in subject lines, some add them in ways that trigger spam filters without telling you. Your list size of 2,000 also makes the 28% open rate impressive because smaller lists usually have more engaged subscribers who notice when the tone shifts too much.