AI copy used to sound like robots. Then GPT-4 hit and it shifted hard.

I've been running A/B tests on landing pages for my side project. Pre-2023, AI generated copy had a 12% lower click rate than human written stuff. Now with the newer models, I'm seeing the AI copy actually beat the human version by about 3% on average. But here's the thing - it only works if you feed it really specific brand guidelines and examples first. Without that context, it still falls flat. So is the improvement the model itself, or is it just that we've gotten better at prompting? What are you guys seeing in your own tests?

2 comments

2 Comments

skylerg171mo ago

Yeah 12% drop to 3% ahead sounds about right from what I've seen. I had a similar thing happen with my own site where the old GPT-3.5 copy was just awful but now the newer stuff actually needs less editing. But I totally feel you on the prompting thing, I've probably wasted way too many hours figuring out the exact system prompt that makes it click.

the_mia1mo ago

The 12% drop to 3% gain tracks with my own testing too (I've been doing this for a newsletter signup page since last summer). It's wild how much the prompting matters, I've literally scrapped entire workflows because I realized my system prompt was way too vague. Honestly I think we're just getting scarily good at prompting, the models themselves are better but not that much better than late 2023 versions.