I remember when A/B testing meant waiting a whole week for results

Back in 2019 I would run email subject line tests and wait 7 days to see which one won. Now I use AI tools that give me predictions in like 2 hours based on past data. Does anyone else feel like the speed of this stuff is making us less patient with the old ways?

2 comments

2 Comments

karenw231mo ago

The speed of this stuff is making us less patient" - yeah but here's what nobody talks about. Fast AI predictions are great until you realize they're trained on your past winners so you just keep making the same kind of thing over and over. Real A/B tests sometimes surprise you with a weird wildcard that crushes everything. Speed kills those happy accidents.

paul_webb451mo ago

I get what you're saying about those wildcard surprises, I really do. But I think you might be giving A/B testing more credit than it deserves. I've seen plenty of times where a "weird wildcard" beats the control but it's just a fluke or it only works on a Tuesday afternoon with a specific type of customer. The problem with pure speed isn't that it kills happy accidents, it's that it makes people stop thinking about why something worked at all. If you're just running tests fast and grabbing whatever pops up, you're not really learning anything deeper about what your audience actually wants.