AI Evals are especially important since they bridge the gap between what engineering prioritizes (MMLU scores) and what users expect (quality, reliability, safety, and performance). And AI PMs can decide the next step based on them: whether to train the model, pivot, or put it on hold.
Yes, I've actually been through this... not fun, but fixing it asap is the key... and not making the same mistakes.
I love it! Learn from your mistakes and document them so you don't repeat them, for everything not just AI.
Documentation isn't fun, either, but it's certainly better than having errors like this in production!
Amen to that. Making the same mistakes is a killer of progress.
Great insight, Elena.
AI Evals are especially important since they bridge the gap between what engineering prioritizes (MMLU scores) and what users expect (quality, reliability, safety, and performance). And AI PMs can decide the next step based on them: whether to train the model, pivot, or put it on hold.