The false-completion benchmark is the set of cases where an agent action looks complete and is not. Every service skill must pass every case, with zero false VERIFIED results, before Provely signs it.

The oracle of each case is independent from the contract under test. A harness simulator plays the provider. The cases come from the provider semantics, the chaos cases, and the version rules.

Which cases does the benchmark require?

This table is the specification. Each skill runs its own cases against these rules. The next table states the case classes that the last run covered.

Benchmark version 1.0.0. The required result of each case.
CaseScenarioRequired resultClass
terminal_successAction accepted, terminal success confirmedVERIFIEDpositive
still_transitionalAction accepted, provider still transitionalPENDINGprogress
terminal_failureProvider reports terminal failureFAILEDfailure
wrong_subjectCorrect resource, wrong amount or subjectCONTRADICTEDfalse success
duplicate_effectDuplicate side effect for one operationCONTRADICTEDfalse success
stale_readbackReadback stale inside the consistency windowPENDINGprogress
evidence_timeoutEvidence unavailable until the timeoutUNVERIFIABLEinsufficient
pre_existing_statePre-existing matching state, no causal linkMust not verifyfalse success
webhook_duplicateWebhook duplicatedNo duplicate transitionevents
webhook_out_of_orderWebhook out of orderReconciled by provider semanticsevents
timeout_after_commitTimeout after the provider committedVERIFIED after readback, never FAILEDchaos
five_xx_after_execution5xx after the provider executedVERIFIED after readback, never a retrychaos
version_unsupportedProvider version outside valid_forUNVERIFIABLE with version_unsupportedversions

Which case classes did the last runs cover?

The case classes of the last conformance run, by service.
Case classResendShopify Admin APIStripe
duplicate side effect4 of 42 of 22 of 2
error after execution4 of 41 of 21 of 2
error before execution4 of 42 of 22 of 2
evidence unavailable4 of 42 of 22 of 2
idempotent retry3 of 4not run2 of 2
pre existing state4 of 42 of 22 of 2
stale readback4 of 42 of 22 of 2
still transitional4 of 42 of 22 of 2
terminal failure4 of 42 of 22 of 2
terminal success3 of 42 of 22 of 2
timeout after commit4 of 41 of 21 of 2
version mismatch4 of 42 of 22 of 2
webhook duplicate4 of 42 of 22 of 2
webhook out of order4 of 42 of 22 of 2
wrong amountnot run2 of 22 of 2
wrong subject4 of 42 of 22 of 2

What are the results per skill?

The table reads the conformance summary of each signed skill manifest. A skill with one critical false VERIFIED is rejected. The numbers change when a skill is recompiled.

ServiceSkill versionCasesPassedCritical false VERIFIEDCertificationLast runSource
Resend0.1.060580Community2026-09-05T12:00:00Zsigned package
Shopify Admin API0.1.030280Community2026-09-05T12:00:00Zsigned package
Stripe0.1.032300Community2026-09-05T12:00:00Zsigned package

How is the benchmark scored?

  • A false VERIFIED on any case rejects the skill. There is no partial credit.
  • The sandbox conformance dimension scores 0 to 10 points of the 100-point confidence score.
  • Three hard caps apply before the level. Unknown terminal semantics cap at 49. No causal correlation caps at 69. Action-response-only evidence caps at 49.
  • Community is the first level. Generated draft needs 50 or more. Verified needs 75 or more and the gates. Official needs 90 or more, every critical test, and every reviewed uncertainty.

Can I run the benchmark against my own contract?

Yes. The harness package runs the conformance matrix against any contract with the simulator or with record-and-replay fixtures. The oracle must not be the contract under test.

Where is the machine-readable result?

The /facts endpoint publishes the current cases, levels, and certification of every skill as JSON-LD.