Just had a wild week testing AI ad copy for local businesses

I spent last Monday through Thursday running ChatGPT against Claude on 50 different ad headlines for a plumber in Austin and the results were way different than I expected. Claude actually generated click through rates that were 23% higher on Facebook ads but ChatGPT's copy brought in more direct calls to the business. Has anyone else seen that kind of split between platforms for local service ads?

2 comments

2 Comments

gavine411mo ago

Wait "Claude actually generated click through rates that were 23% higher" - that's a huge gap just from a headline test. I've messed around with both for my buddy's HVAC company and saw maybe a 5-8% difference but nothing that extreme. Did you run the same exact prompts or tweak them per platform? I'm also curious if the plumber's specific audience just responded differently to Claude's more conversational tone versus ChatGPT's punchier style.

hart.ryan1mo ago

Yeah but the thing is @gavine41 I've been wondering about that too. Did they maybe run the Claude headlines through a different targeting setup or something? Because 23% is wild. I tested both for a real estate agent friend of mine and the difference was like 12% tops. I think it really depends on what you're selling and who you're selling to. Like plumbers deal with people who are stressed and need help fast so maybe Claude's longer more reassuring tone actually worked better for that crowd. Whereas HVAC you're probably dealing with homeowners who just want the cheapest fix possible. But honestly I'm still skeptical about that big of a gap unless they cherry picked the best Claude headlines against the worst ChatGPT ones.