回复打分
使用“回复打分”模板,按你的评分细则(准确性、完整性和语气)给 LLM 回复打出 0–1 的分数,并把低分回复送去审核。
目标: 逐条了解你的助手回答得有多好,并在糟糕的回答造成损失之前把它们找出来。
创建决策
模板 → 回复打分(评分细则) → 创建决策:
| 部分 | 内容 |
|---|---|
| 状态 | question、response(字符串,必填),reference(字符串——已知答案或来源文本) |
| 上下文 | 评分细则——按你的标准修改 |
accuracy | score——wrong、mostly wrong、mixed、mostly right、fully right |
completeness | score——misses it、partial、most of it、complete |
tone | score——poor、acceptable、good、excellent |
unsafe_promise | probability——它是否承诺了退款、价格或期限? |
| 复合指标 | grade = 0.5 × accuracy + 0.3 × completeness + 0.2 × tone,取值 0 到 1 |
| 策略 | unsafe_promise ≥ 0.7 → escalate;grade < 0.6 → escalate |
| 目的地 | quality_review——escalate 时调用函数 flagForQualityReview |
调用
curl -X POST https://api.dcision.io/v1/decisions/response-grading \
-H "Authorization: Bearer $DCISION_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"state": {
"question": "Can I get a refund on my annual plan after 2 months?",
"response": "Yes, of course! We'"'"'ll refund you in full anytime.",
"reference": "Annual plans can be refunded within 30 days of purchase."
}
}'状态中的 response 为 "Yes, of course! We'll refund you in full anytime."。
{
"schema": "response-grading",
"result": { "accuracy": "wrong", "completeness": "partial", "tone": "good", "unsafe_promise": 0.97 },
"composites": { "grade": 0.17 },
"action": "escalate",
"destinations": [{ "key": "quality_review", "type": "function", "function": "flagForQualityReview", "params": { "grade": 0.17 } }]
}据此行动
在回复生成之后打分——可以在发送之前打分以拦截它,也可以在发送之后打分以跟踪质量:
const d = await dcisionDecide("response-grading", { question, response, reference });
const outOfTen = Math.round(d.composites.grade * 100) / 10; // 0.17 → 1.7 / 10
if (d.action === "escalate") await flagForQualityReview({ question, response, grade: d.composites.grade });调优
- 把评分细则写进
context——它会随决策一起版本化,因此每个分数总能对应到产生它的评分细则。 - 调整
grade的权重,使之符合你最看重的方面。 - 执行记录默认保留输入和输出:执行记录页面和概览会展示分数随时间的变化。