Pocketsflow Penny 1
A small open model for the decisions inside software: refunds, moderation, agent actions, tickets. Ask it to choose, say yes or no, or score. It answers with a confidence you can trust, on your own machine.
Abstract
A decision, not a paragraph
Most questions inside software have a short answer: approve or hold, allow or block, which queue, how risky. General chat models answer them with paragraphs and cannot say how sure they are. Penny 1 reads a situation (JSON, a message, a table row) and a typed question, and returns one answer with a calibrated probability in a single forward pass.
It is a 596M model fine-tuned from Qwen3-0.6B on synthetic, cross-checked decisions across 22 kinds of task, from refunds to content moderation and AI-agent guardrails. On six kinds it never saw it scores 73%, against 85% for Llama 3.3 70B, and its confidence is calibrated to within 5.2%. It runs locally in Ollama, LM Studio and llama.cpp; the weights are Apache-2.0.
Results
2128 judged questions · 7 models
By question type
| Type | Penny 1 | Llama 70B |
|---|---|---|
| Choose | 73.4% | 82.5% |
| Yes or no | 75.8% | 87.4% |
| Score 1–5 | 75.7% | 85.3% |
Scores count as right within one point of the judged answer.
Thresholds
Automate the sure ones
Because the confidence is calibrated, a threshold is a dial between automation and accuracy. At 90%, Penny decides 27% of unseen-task questions on its own and is right on 94.1% of them; the rest go to a person.
| Answer if ≥ | Automated | Accuracy |
|---|---|---|
| always | 100% | 73.5% |
| 50% | 70% | 78.5% |
| 60% | 59% | 81.3% |
| 70% | 51% | 85.3% |
| 80% | 43% | 89.5% |
| 90% | 27% | 94.1% |
| 95% | 18% | 97.1% |
Compared
Same questions, every model
| Model | Params | Runs | Trained tasks | Unseen tasks | Calibration error (unseen) |
|---|---|---|---|---|---|
| gpt-oss 20B | 20.9B | API | 86.6% | 87.9% | — |
| Llama 3.3 70B | 70.6B | API | 85.1% | 84.7% | — |
| Llama 3.1 8B | 8B | API | 75.8% | 76.6% | — |
| Llama 3.2 3B | 3.2B | API | 70.7% | 74.2% | — |
| Penny 1 | 596M | Local | 75.9% | 73.5% | 5.2% |
| Qwen3 0.6B Penny's base, untuned | 596M | Local | 50.8% | 54.9% | 25.0% |
| Llama 3.2 1B | 1.2B | API | 47.3% | 43.3% | — |
Local models answer from next-token probabilities, so their calibration is measurable; API models answer one token at temperature 0 and report no probabilities. Test questions were written by a model that wrote no training data and kept only when an independent judge (gpt-oss-120b, not in this table) agreed. Unseen tasks, absent from training: invoice approval, discount approval, collections, agent tool routing, code review, document filing. Calibration error is measured on those unseen tasks.
Cost
Per million decisions
API cost = measured prompt and answer tokens × Workers AI list prices (Oct 2026). Self-hosted cost = server price ÷ measured throughput (llama-server, 4 parallel slots, Q4_K_M); your hardware, power and utilisation will differ. On a machine you already own, the marginal cost is electricity.
Quantization
| Build | Size | Unseen money tasks |
|---|---|---|
| PyTorch fp32 | 2.4 GB | 66.6% |
| GGUF Q8_0 | 639.4 MB | 66.6% |
| GGUF Q4_K_M | 396.7 MB | 65.7% |
GGUF builds were scored through the public client against llama-server, the same path Ollama and LM Studio use.
Method
- 01
Written to an answer
Three open models wrote 23,158 decisions across 28 kinds of task, money and general. The answer was fixed before writing (which option, yes or no, which score), so labels are balanced and exact. No customer data.
- 02
Judged blind
A different model answered every question without seeing the intended answer; only agreements were kept (12,890). Test questions came from a fourth model that wrote no training data.
- 03
Tuned on one token
LoRA on the answer token only, then one temperature per question type fitted on validation data, then GGUF quantization and a re-test through the client.
| Base model | Qwen/Qwen3-0.6B |
| Adapter | LoRA rank 16, top 14 of 28 layers, merged into the weights |
| Training | 10,101 questions · 3 passes, 1146 steps · batch 8 · lr 0.0002 · CPU only, 7 h |
| Calibration | T = 1.1985 (choose) · 1.0396 (yes/no) · 0.935 (score) |
| Evaluation | 1186 trained-task + 942 unseen-task questions · held-out generator · independent judge |
Where it fits
Worked examples →- Agent guardrailsRun it, ask the user first, or block the action an AI agent proposes.
- ModerationKeep, hide or remove a comment, a review or a listing.
- Ticket routingSend each message to the right team and spot the urgent ones.
- Grading AI answersScore another model's answer before it reaches a user.
- RefundsRefund, store credit or decline, from the order and the message.
- Fraud reviewScore a payment's risk before it ships; queue only the doubtful ones.
- Bug triageSeverity and owner for an incoming bug report.
- Spam and phishingIs this message safe to deliver?
Limits
It sorts. People decide.
- Trained and tested on synthetic situations. Measure it on your own data before relying on it.
- English only. One situation and one question per call; up to 8 options.
- Not for credit, lending, insurance, hiring or any decision about a person's access to services.
- Never move money or close accounts on its answer alone: route low confidence to a person.
- Scores (1–5) cluster near the middle on unfamiliar inputs, and a score counts as right within one point. Prefer choose or yes / no for decisions that matter.
- Not a security control. It can be confidently wrong about a dangerous agent command; keep a hard allow-list in front of anything destructive.
- The test labels are model-judged; a human-labelled benchmark is the next step.
Cite
BibTeX
@misc{pocketsflow2026penny,
title = {Pocketsflow Penny 1: a small open decision model},
author = {{Pocketsflow Research}},
year = {2026},
url = {https://www.pocketsflow.com/research/penny}
}Weights