21
Fine-tuned model beat my fancy prompt chain 3 to 1 in our support bot test
I spent 2 weeks building a multi-step prompt system for customer replies, then ran it against a simple fine-tuned Llama model on 200 real tickets from our Houston queue. The fine-tune nailed context way better, especially sarcasm and abbreviations, but it cost 40 bucks to train and update. Is the accuracy gain worth the maintenance hassle, or do you all stick with pure prompting for this kind of thing?
1 comments
Log in to join the discussion
Log In1 Comment
val22310h ago
Ran a similar test with a buddy who handles support for a shipping company. He trained a small model on like 300 old tickets and it instantly picked up on their weird shorthand like "refund ASAP pls" and angry customer jokes. His fancy prompt system kept giving robotic, super formal answers that missed the tone completely. He still updates the fine-tune every few months when things change, but says the 40 bucks saves him way more in editing time. Do you ever worry about the model getting stale when your product or policies shift?
4