回复打分

使用“回复打分”模板,按你的评分细则(准确性、完整性和语气)给 LLM 回复打出 0–1 的分数,并把低分回复送去审核。

目标: 逐条了解你的助手回答得有多好,并在糟糕的回答造成损失之前把它们找出来。

创建决策

模板 → 回复打分(评分细则) → 创建决策:

部分内容
状态question、response(字符串,必填),reference(字符串——已知答案或来源文本)
上下文评分细则——按你的标准修改
accuracyscore——wrong、mostly wrong、mixed、mostly right、fully right
completenessscore——misses it、partial、most of it、complete
tonescore——poor、acceptable、good、excellent
unsafe_promiseprobability——它是否承诺了退款、价格或期限?
复合指标grade = 0.5 × accuracy + 0.3 × completeness + 0.2 × tone,取值 0 到 1
策略unsafe_promise ≥ 0.7 → escalate;grade < 0.6 → escalate
目的地quality_review——escalate 时调用函数 flagForQualityReview

调用

curl -X POST https://api.dcision.io/v1/decisions/response-grading \
  -H "Authorization: Bearer $DCISION_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": {
      "question": "Can I get a refund on my annual plan after 2 months?",
      "response": "Yes, of course! We'"'"'ll refund you in full anytime.",
      "reference": "Annual plans can be refunded within 30 days of purchase."
    }
  }'

状态中的 response 为 "Yes, of course! We'll refund you in full anytime."。

200 OK(节选)
{
  "schema": "response-grading",
  "result": { "accuracy": "wrong", "completeness": "partial", "tone": "good", "unsafe_promise": 0.97 },
  "composites": { "grade": 0.17 },
  "action": "escalate",
  "destinations": [{ "key": "quality_review", "type": "function", "function": "flagForQualityReview", "params": { "grade": 0.17 } }]
}

据此行动

在回复生成之后打分——可以在发送之前打分以拦截它,也可以在发送之后打分以跟踪质量:

const d = await dcisionDecide("response-grading", { question, response, reference });
const outOfTen = Math.round(d.composites.grade * 100) / 10; // 0.17 → 1.7 / 10
if (d.action === "escalate") await flagForQualityReview({ question, response, grade: d.composites.grade });

调优

  • 把评分细则写进 context——它会随决策一起版本化,因此每个分数总能对应到产生它的评分细则。
  • 调整 grade 的权重,使之符合你最看重的方面。
  • 执行记录默认保留输入和输出:执行记录页面和概览会展示分数随时间的变化。

本页内容