Local small-model video editing: historical test details
This appendix transcribes selected results from BonFrame’s existing internal reports. It is a public evidence summary, not a new benchmark run or a release of the underlying media and raw outputs. Results describe the dated configurations below, not current general-purpose model rankings.
1. Candidate-card selection: August 3, 2026
The Chinese suite had 13 cases based on a Federal Reserve explainer. The English suite had 11 cases based on an interview. These were existing card-level keep-ID tests, not a newly locked end-to-end holdout. Retrieval and candidate construction occurred before the selector’s decision.
The evaluation entry point was eval:card-insight-keep. The historical source report is titled “What small-model results prove,” with separate Chinese and English tables. Local inference used a high-end NVIDIA setup; these results do not establish laptop or CPU latency.
| Suite | Selector | Pass | Reported quality score | Model-used fraction | Rule-fallback fraction | Paraphrase overlap |
|---|---|---|---|---|---|---|
| Chinese | Ministral 8B Q4 | 13/13 | 0.994 | 1.000 | 0 | 0.885 |
| Chinese | OpenAI GPT-5.1 | 13/13 | 0.999 | 1.000 | 0 | 0.865 |
| Chinese | DeepSeek v4-flash | 13/13 | 1.000 | 1.000 | 0 | 0.917 |
| English | Ministral 8B Q4 | 9/11 | 0.845 | 1.000 | 0 | 0.716 |
| English | OpenAI GPT-5.1 | 8/11 | 0.830 | 1.000 | 0 | 0.691 |
| English | DeepSeek v4-flash | 8/11 | 0.826 | 1.000 | 0 | 0.680 |
The quality and overlap values are suite-specific diagnostics, not universal accuracy measures. Model-use and fallback fractions help distinguish model decisions from rule substitution. Higher pass counts alone can hide excessive retention or inconsistent selections under paraphrased requests.
No complete-video ChatGPT equivalence claim follows from these results. The historical report labels the OpenAI endpoint GPT-5.1; an API model comparison is not a comparison with every feature of the ChatGPT application.
2. Complete-cut protocol: August 20, 2026
Two reports recorded the product 8B and 3B protocol runs. Both used eight transcripts held out from assembly tuning, three attempts per case, and a separate GPT-5.1 judge with labels swapped by case and repeat. Assembly gates were not retuned on those cases.
- A: product editing workflow with local selection from a retrieved block directory.
- B: DeepSeek v4-flash with full transcript and real source IDs, followed by duration handling.
- C: DeepSeek with the same short block directory as A, using the shared selection-resolution and duration assembly path.
The primary comparison is A versus B. C helps examine the role of evidence scope. These are system comparisons, not isolated checkpoint comparisons with every prompt and input held identical.
| Protocol | Path | Mechanical pass | Combined pass | Mean generation input |
|---|---|---|---|---|
| 8B | A: Ministral 8B Q4 | 24/24 | 17/24 | 860 tokens |
| 8B | B: DeepSeek full transcript | 24/24 | 15/24 | 4,600 tokens |
| 8B | C: DeepSeek same directory | 24/24 | 11/24 | 771 tokens |
| 3B | A: Ministral 3B | 24/24 | 14/24 | 860 tokens |
| 3B | B: DeepSeek full transcript | 24/24 | 15/24 | 4,600 tokens |
| 3B | C: DeepSeek same directory | 24/24 | 9/24 | 771 tokens |
Combined pass requires mechanical checks and judged grounding, coverage, exclusion and coherence. All mechanical passes do not mean all cuts met the semantic rubric.
The two B runs have the same total but different per-case outcomes. They are separate runs and must not be treated as identical observations or added together to enlarge the sample.
Per-case results: number of passing attempts out of three
| Case | 8B A | 8B B | 8B C | 3B A | 3B B | 3B C |
|---|---|---|---|---|---|---|
| ETF pricing, Chinese | 3 | 0 | 0 | 2 | 0 | 0 |
| Feynman learning, Spanish | 2 | 2 | 2 | 2 | 2 | 0 |
| Spaced repetition, Chinese | 2 | 3 | 0 | 3 | 3 | 0 |
| Hypertension, English | 2 | 1 | 3 | 2 | 1 | 3 |
| Fair use, English | 3 | 3 | 3 | 1 | 2 | 3 |
| Docker containers, English | 0 | 0 | 0 | 0 | 1 | 0 |
| Photosynthesis, English | 2 | 3 | 0 | 1 | 3 | 0 |
| Unit economics, English | 3 | 3 | 3 | 3 | 3 | 3 |
The token reduction is (4600 − 860) / 4600 = 81.3%. It counts generation input in these runs, not total pipeline compute, total tokens, wall-clock speed or translation savings.
The original reports set nonInferior=true when the observed difference exceeded a predeclared −5 percentage-point threshold. This is a protocol acceptance rule applied to point estimates. No confidence-interval-based non-inferiority analysis is supplied here. Repeated attempts on eight transcripts are not 24 independent content samples.
3. Earlier negative result: August 19, 2026
Before local block selection, retrieval plus rule assembly passed 2/8 cases; the full-transcript baseline passed 5/8. Both passed mechanical checks on all eight. This was one attempt per case, without path C.
It shows why finding relevant passages is insufficient. It should not be combined with later pass counts or used as a controlled ablation isolating the effect of one model change.
4. Failure interpretation
Docker exposed promotion and demonstration material surviving despite insufficient explanation. Photosynthesis exposed candidates ending before a complete mechanism. Fair use showed a same-evidence cloud selector could outperform 3B on an individual case.
These are meaningful failures, even when an aggregate threshold is met. Candidate boundaries, retrieval coverage and editorial intent remain important variables.
5. Evidence that is still missing
These reports do not supply a larger independent replication, professional human blind review, a full-cut GPT-5.1 baseline or a benchmark of translation and lip-sync quality. The raw trial files and complete evaluator implementation are not distributed through this page, so the public tables alone are not a fully reproducible benchmark package.
The source records retained in the project are:
06-what-small-model-results-prove.md: August 3 card-level selection tables.CLOUD_FULL_CONTEXT_AB_HOLDOUT.md: August 19 rule-assembly holdout.CLOUD_FULL_CONTEXT_AB_HOLDOUT_PRODUCT_8B_PROTOCOL.md: August 20 repeated 8B protocol.CLOUD_FULL_CONTEXT_AB_HOLDOUT_PRODUCT_3B_PROTOCOL.md: August 20 repeated 3B protocol.
These filenames identify the internal records; they are not public download links. The public benchmark method preserves the held-out protocol summary. The companion architecture article explains the task boundaries behind these results.