Overheard a dev say training on synthetic data is a dead end

I was grabbing coffee at a coworking space in Austin last week and caught two engineers arguing about synthetic data in LLM training cycles. One claimed it just amplifies model biases instead of fixing them, and the other said it's the only way to scale cheaply. Has anyone here run their own test on synthetic vs. real data for fine tuning?

2 comments

2 Comments

gavinperez1d agoMost Upvoted

Nah I gotta respectfully disagree here, I've been testing synthetic data for a few months now on some small models and honestly it's been working way better than I expected lol. Yeah you gotta be careful with it but calling it a dead end feels like missing the point - it's more about how you generate and filter it, not the concept itself.

troy4391d ago

The filtering part is the whole game honestly, if you just feed raw garbage in you get garbage out but a good pipeline with some validation steps makes synthetic data super solid for fine tuning on niche tasks.