Release evidence
Benchmarks
Reproducible release evidence for VerifyPatch 0.2.0. This is not a correctness, security, trust, or certification score.
These results describe the tested repositories, commits, environments, and configurations. They are measured examples, not universal runtime or defect-detection guarantees.
- Release commit
- 70d0b8794a3a495ebb199b6d9ba21d99e74cb906
- Benchmark date
- 2026-08-15
Python matrix
| Python | Result |
|---|---|
| 3.10.20 | 210 passed, 1 skipped, 2 deselected |
| 3.11.15 | 210 passed, 1 skipped, 2 deselected |
| 3.12.13 | 210 passed, 1 skipped, 2 deselected |
| 3.13.13 | 210 passed, 1 skipped, 2 deselected |
| 3.14.4 | 210 passed, 1 skipped, 2 deselected |
Real-repository provenance
Complete means analysis finished. It does not mean the patch was proven correct. tomli stayed incomplete because of an empty or ambiguous coverage context. Zero changed executable lines stay n/a, not 0%.
| Repository | Changed lines | PR-untouched | Runtime | Status |
|---|---|---|---|---|
| iniconfig | 8 | 100% | 1.89 s | complete |
| idna | 3 | 100% | 9.23 s | complete |
| tomli-w | 0 | n/a | 2.08 s | complete |
| zipp | 1 | 0% | 1.79 s | complete |
| tomli | 1 | n/a | 1.82 s | incomplete (empty_context) |
| pluggy | 3 | 100% | 2.10 s | complete |
| packaging | 0 | n/a | 11.64 s | complete |
Pinned comparisons
| Repository | Base SHA | Head SHA |
|---|---|---|
| iniconfig | 6d0af4529e4375e49dc871aa3d5ce17fe1791afe | 58c08691bbb86aee8efbf73e37293dd6d65b68b4 |
| idna | f39ea903ba49eb5a0b2c6723c9a929b41ed4a0f1 | 9067b803a55441805934410b11c0899209b66785 |
| pluggy | 237edb6e8e3067c46f91ae620a652e2fb20bf68e | 54127a334d52a49d02c77b001ee998d36f7d6037 |
Performance
Three measured runs per command. Table reports the median wall-clock time. Peak RSS is the sampled process-tree sum. These small suites make fixed startup and coverage costs prominent.
| Repository | pytest | VerifyPatch | Added | Multiple | Added RSS |
|---|---|---|---|---|---|
| iniconfig | 0.233 s | 0.579 s | 0.346 s | 2.48x | 46.8 MiB |
| pluggy | 0.332 s | 0.869 s | 0.538 s | 2.62x | 48.0 MiB |
| idna | 2.746 s | 8.488 s | 5.742 s | 3.09x | 47.7 MiB |
Mutation
Internal AST backend. Only pytest exit code 1 counts as a kill. This trial set produced no genuine survivor. It does not establish a general mutation score.
| Repository | Candidates | Selected | PR-untouched kills | Other outcomes | Independent | Overall |
|---|---|---|---|---|---|---|
| iniconfig | 3 | 3 | 3 | 0 survivors | 1.000 | 1.000 |
| idna | 3 | 3 | 1 | 2 PR-touched kills, 0 survivors | 0.333 | 1.000 |
| pluggy | 3 | 3 | 3 | 0 survivors | 1.000 | 1.000 |
| zipp / tomli | 0 | 0 | n/a | no candidates | n/a | n/a |
Report scaling
| Evidence rows | JSON size | Serialize | Validate | Peak RSS |
|---|---|---|---|---|
| 1,000 | 206,602 bytes | 0.003 s | 0.028 s | 31.5 MiB |
| 10,000 | 2,060,605 bytes | 0.023 s | 0.213 s | 38.1 MiB |
| 50,000 | 10,340,605 bytes | 0.130 s | 1.093 s | 78.6 MiB |
Interruption and cleanup
Recorded release gates sent SIGTERM during v1 pytest/coverage, generated-test execution, mutation execution, and behavioral replay. All four recorded exit 143, no leftover mutation or behavior directories, no partially trusted success report, and cleanup_ok: true. A deliberately detached child that called setsid() can escape process-group termination. VerifyPatch is not a sandbox.
Packaging
| Artifact | Size | SHA-256 |
|---|---|---|
| verifypatch-0.2.0-py3-none-any.whl | 97,600 bytes | 06642d2afdd4377ce229f0ce07228369def43e49327a786e1798351182ec3357 |
| verifypatch-0.2.0.tar.gz | 108,356 bytes | f2dce540b8aad6e7ffd9d1cc0d84cd780fb54c1549e396b1f97295b664638df1 |
What remains unverified
- Live OpenAI and Anthropic API requests. SDK shapes and failure modes are tested; no provider credentials were available for live calls.
- Windows execution. Release validation covers macOS locally and targets Linux in CI. No Windows runner result is recorded.
- A consumer repository using the final tagged composite Action. Static workflow tests and local fixture smoke passed. No external tagged run exists before the tag is published.
- Fork-pull-request behavior of the two-job provider workflow. The secret boundary is statically validated. No end-to-end fork trial is recorded.
- Real-world heuristic precision/recall, broad ecosystem compatibility, and universal performance guarantees.
Reproduction
Use the exact release artifact, immutable repository SHAs, isolated environments, an empty PYTHONPATH, disabled user-site packages, and at least three measured runs after environment setup. Report medians. Never convert incomplete, zero-candidate, or null metrics into successful percentages.