What is a ChatGPT plugin activation test?

It is a labelled evaluation that checks whether the expected plugin tool is selected for an eligible prompt, whether irrelevant prompts avoid that tool, and whether selected calls contain correct arguments. Activation is an analyst-defined test outcome, not an official OpenAI ranking metric.

Which prompts belong in a plugin golden set?

Include direct prompts that name the product, indirect prompts that express an in-scope job without the product name, and negative prompts that should use no plugin, another capability, clarification, refusal, or a safe fallback. Record expected behavior before running the tests.

How should activation recall and precision be calculated?

Activation recall divides correctly selected expected-tool cases by completed eligible positive cases. Precision divides correct expected-tool activations by all observed activations in the evaluated set. Report direct and indirect recall, false activation on negative cases, raw counts, and exclusions separately.

How many positive and negative tests does OpenAI require for plugin submission?

OpenAI's current submission documentation requires at least five positive and three negative test cases. That is a submission minimum, not proof that the suite covers every tool, ambiguity, permission boundary, or regression risk.

Can better plugin metadata guarantee directory rank, traffic, or tool calls?

No. Clear metadata can be tested for better observed selection behavior, but neither an approved listing nor a favorable activation test guarantees directory position, user traffic, recommendation frequency, or future tool calls.

ChatGPT Plugin Activation Tests: Precision and Recall

A plugin can be published, available to a test account, and connected correctly—and still fail to activate for the job it was built to do.

The opposite failure is just as important. A plugin may activate when the user asked for something outside its scope, creating an irrelevant interruption or exposing an action path that should never have been considered.

These are tool-selection problems. They should be tested with labelled prompts, explicit denominators, and versioned evidence rather than anecdotes about one successful conversation.

Short answer: Create a golden set of direct, indirect, and negative prompts. For every prompt, predeclare the expected tool or non-tool behavior. Test only under an eligible plugin state, then report activation recall, precision, false activation, argument correctness, and execution success separately. Change one metadata field at a time and rerun the same suite. Keep listing, availability, connection, activation, arguments, and completion as distinct gates.

This guide focuses on builder-side activation QA. It complements the broader SaaS and plugin AI visibility framework, which measures public mentions, recommendations, connection suggestions, invocations, and directory discovery as separate evidence lanes.

What OpenAI Actually Documents

OpenAI's current metadata optimization guide says ChatGPT and Codex use tool metadata when deciding whether to call a tool. It recommends a labelled golden prompt set with three classes:

Direct prompts name the product or data source.
Indirect prompts describe the desired result without naming the tool.
Negative prompts are requests that built-in or other tools should handle.

The guide tells developers to record the selected tool, passed arguments, and component result, then track precision and recall. It also recommends changing one metadata field at a time and repeating the evaluation after each revision.

OpenAI's plugin submission documentation requires at least five positive and three negative test cases. Positive cases need an expected tool, skill, or workflow behavior, an expected result shape, and reproducible fixture data. Negative cases need the expected refusal, clarification, or safe fallback and an explanation of why the plugin should not complete the action.

That submission requirement is a floor, not a complete production evaluation. A plugin with six tools, two authentication modes, and several write actions needs more coverage than eight cases.

OpenAI's product-design article, What makes a great ChatGPT app, adds the design principle behind the test: capabilities should be concrete, limited, and directly tied to value. Tool descriptions help map user intent to actions. A plugin should not assume that the user already knows its brand.

The metrics and stage labels below are an operational framework for applying that guidance. They are not OpenAI-provided analytics, ranking factors, or distribution guarantees.

Do Not Collapse Six Different Gates

Before calculating activation, classify the state that preceded it.

Gate	Question To Answer	Minimum Evidence	Not Proven By Passing
Listing	Was the reviewed version published and can the governed directory procedure find it?	Listing identity, published version, timestamp, directory procedure, screenshot	Availability to every user or any directory rank
Availability	Can this declared account, surface, and region load the intended plugin version and tools?	Account profile, surface, region, visible version or snapshot evidence	Authentication, permissions, or selection
Connection	If authentication is required, can the test account connect with the required fixture permissions?	Successful connection state and least-privilege fixture access	Selection for a prompt or successful action
Activation	Did the completed eligible prompt select the expected plugin tool?	Exact prompt, expected behavior, selected tool, timestamp, version	Correct arguments, output, or business result
Arguments	Did the selected call preserve the required identifiers, values, and user constraints?	Tool arguments, schema, fixture, defaults, and exact prompt	Successful execution or result quality
Completion	Did the correctly formed call return the declared result or safe error?	Result shape, component state, server response, error, or confirmation evidence	Future reliability, traffic, or conversion

A failure at one gate should not be silently recoded as another. If the plugin is unavailable, you did not observe an activation miss. If it activates with the wrong arguments, you did not observe an activation failure; you observed an argument failure after selection.

This separation also prevents inflated results. Excluding every connection failure from activation recall is valid only when connection health is reported beside the activation metric. A user-journey report should show how many attempted cases reached each gate.

OpenAI describes a review and publication flow that ends with a published plugin appearing in the universal Plugins Directory. The documentation does not promise a particular directory position, volume of visits, or frequency of model selection. Treat those as unguaranteed outcomes, not benefits implied by approval.

Define The Eligible Denominator Before Testing

The basic activation test unit is:

Golden prompt × declared product surface × eligible account state × plugin metadata version × test run

A positive case is activation-eligible when all of the following are true:

The intended reviewed or development version is loaded.
The expected tool is present and its schema can be inspected.
The declared account and surface can use the plugin.
Required authentication and fixture permissions are valid.
The prompt completes far enough for tool selection to be observed.
No predeclared platform outage, harness error, or fixture failure invalidates the attempt.

Do not delete invalid attempts. Classify them as unavailable, connection failed, platform failed, fixture failed, or another governed reason. Report attempted, eligible, completed, and excluded counts.

For example, suppose a team attempts 80 positive cases. Four cannot load the plugin version, three have expired fixture credentials, and one test runner loses the response. The activation denominator is 72 completed eligible positive cases, not 80. The report must still show all eight pre-activation failures. Otherwise the 72-case recall number disguises a broken user journey.

Keep the eligibility rules stable across versions. The general AI visibility measurement methodology uses the same principle: a visible denominator and preserved failure states make a rate reviewable.

Build Direct, Indirect, And Negative Golden Prompts

Start from the user jobs the plugin can genuinely complete. The AI search query set guide explains how to govern stable buyer Questions; an activation suite narrows that idea into labelled tool-selection cases.

Imagine a fictional support-operations plugin called Northstar with a read-only tool named support.summarize_tickets.

Class	Example Prompt	Expected Behavior	What It Diagnoses
Direct positive	“Use Northstar to summarize the themes in the 15 open priority tickets.”	Select `support.summarize_tickets` with status, priority, and limit arguments	Whether an explicit brand request maps to the correct tool
Indirect positive	“What themes are driving our currently open priority support tickets? Summarize the top three.”	Select the same tool because private fixture data is required	Whether outcome language activates the tool without brand awareness
Negative, built-in	“Draft a two-sentence apology to a customer whose delivery was late.”	Do not select Northstar; generate text without private ticket retrieval	Whether generic writing causes accidental activation
Negative, clarification	“Delete the bad tickets from last month.”	Do not run the read-only summary tool; clarify or safely decline the unsupported action	Whether a nearby but disallowed job is contained
Negative, alternative	“Find today's weather for our support team's offsite.”	Use an appropriate alternative if available, not Northstar	Whether the plugin competes outside its domain

Direct prompts are necessary but easy. A suite made entirely of “Use Northstar” cases mainly proves that the product name is recognized. Indirect prompts test whether the metadata expresses the actual job. Negative prompts protect precision.

Within each class, include boundaries that matter:

Near-neighbor tools with similar names or overlapping descriptions.
Missing identifiers that should trigger clarification instead of a guessed argument.
Allowed-value boundaries such as open, pending, and an unsupported status.
Read-only versus write actions.
Requests for data outside the connected account's permission scope.
Multi-intent prompts where only one part belongs to the plugin.
Natural paraphrases from support, sales, and user research.

Label the expected behavior before running the suite. Do not inspect a surprising call and then rewrite the label to make it pass. Keep exploratory prompts in a separate pool; promote them into the golden set through a versioned review.

When a case depends on constraints introduced across earlier turns, treat the conversation as a separate script rather than pasting only its final sentence into the golden set. The multi-turn AI visibility guide defines the history policy, request-state ledger, and paired endpoint treatment for that design.

Calculate Four Metrics, Not One Activation Score

Use raw counts beside every percentage.

Activation Recall

Correct expected-tool selections ÷ completed eligible positive cases

Report direct and indirect recall separately. A plugin can score 19/20 on direct prompts and 8/20 on indirect prompts. Combining them into 27/40 hides the no-brand-awareness failure.

If a positive case expects clarification before a tool call, do not force it into a call-only recall definition. Grade the declared workflow behavior separately or split the case into a clarification metric.

Activation Precision

Correct expected-tool activations ÷ all observed plugin activations in the evaluated set

The denominator includes unexpected plugin activations on negative prompts and wrong-tool selections inside the plugin. Precision answers: when the plugin activated, how often was that activation the labelled choice?

False Activation Rate

Eligible negative cases with any disallowed plugin activation ÷ completed eligible negative cases

Keep this visible even when precision is high. A large positive set can make a small number of harmful negative activations look statistically minor in a pooled precision result. OpenAI's metadata guide explicitly advises prioritizing strong precision on negative prompts before pursuing marginal recall gains.

Argument Correctness

Expected-tool selections with correct required arguments ÷ completed eligible expected-tool selections

Grade arguments against the tool contract:

Required fields are present.
Entity and record identifiers resolve to the requested fixture.
Allowed values and formats are valid.
User constraints such as date range, limit, locale, or status are preserved.
Unsupported information is not invented.
An omitted optional value follows the documented default.
Write or open-world boundaries are not bypassed.

A call can be selected correctly and still be operationally wrong. If the user asks for the last 15 open tickets and the call requests 50 closed tickets, activation recall passes while argument correctness fails.

Track end-to-end completion as a fifth operational outcome, but do not fold it into selection. Tool execution can fail because of the server, downstream API, permissions, latency, or fixture. Those owners need different evidence.

Test Metadata One Variable At A Time

OpenAI recommends precise metadata rather than promotional copy:

Pair the domain and action in the tool name, such as calendar.create_event.
Begin the description with when the tool should be used and include material disallowed cases.
Explain every parameter, with examples and allowed values for constrained inputs.
Mark a truly retrieval-only or compute-only tool with readOnlyHint: true.
For non-read-only tools, set destructiveHint and openWorldHint accurately according to whether the tool can overwrite data or reach beyond the user's account.

Annotations describe behavior and risk. Never set them inaccurately to make a tool seem safer or easier to select.

For a controlled metadata test, freeze:

Golden prompt version and labels.
Tool implementation and parameter schema.
Fixture data and permissions.
Plugin, account, surface, and connection state.
Test-case execution order, retry rule, and evidence capture.
Any visible model or runtime identifier; if it is not exposed, record that fact.

Then change one field. For example, revise only the description from a broad “Analyze support data” to a scoped statement that identifies open-ticket theme analysis and excludes drafting, deletion, and unrelated customer communication.

Predeclare the decision rule: perhaps indirect recall must improve in repeated matched blocks while direct recall does not regress and no additional negative case activates. Compare raw prompt-level outcomes, not only aggregate percentages.

One-variable testing improves attribution to the edited metadata within the observed test setup. It does not eliminate platform changes, model variability, order effects, or shared context. Use repeated matched blocks, keep timestamps, and phrase the result as observed association unless the design supports a stronger claim. The broader AEO content experiment protocol offers a useful model for predeclared hypotheses, controls, guardrails, and cautious causal language.

Make Version Regression A Release Gate

OpenAI says published MCP plugins use reviewed metadata and skill snapshots. Changing that snapshot requires scanning the server, submitting a new version for review, and publishing the approved version.

Your regression record should therefore identify:

Plugin release and submission version.
MCP metadata snapshot or reproducible hash.
Skill bundle version when applicable.
Tool names, schemas, descriptions, and annotations.
Golden set version and label reviewer.
Fixture version and authentication profile.
Surface, region, account profile, and test time.
Visible runtime identifiers and unavailable identifiers.
Selected tool, arguments, component state, result, and error.

Run the full suite against the candidate in an authorized development environment. After an approved version is published, run a small smoke set to verify that the intended snapshot is available, then repeat the governed regression suite under matched conditions.

Use the official five-positive and three-negative submission requirement, but maintain a larger operational set where risk justifies it. At minimum, cover every high-value tool, every write or external action, hard negatives between sibling tools, and argument boundaries that could affect user data.

Do not overwrite previous results. A version table should show which prompts moved from pass to fail, not just the new aggregate. The AI search volatility guide explains why repeated observations and preserved ranges are safer than treating one changed response as a durable trend.

Diagnose The Failure At The Right Layer

Use a decision table before editing metadata.

Observed Failure	Classification	First Evidence To Inspect	Likely Owner
Published version cannot be found through the governed directory procedure	Listing or discovery	Publication state, listing identity, procedure, account and timestamp	Publishing or platform operations
Listing exists but the declared test profile cannot load it	Availability	Surface, account, region, admin policy, version	Workspace admin or platform operations
Plugin loads but authentication or fixture access fails	Connection	OAuth state, token scope, source-system permissions, fixture	Identity, integration, or workspace admin
Eligible indirect prompt selects no plugin	Activation recall	Tool name, description, prompt label, competing capabilities	Plugin product and metadata owner
Negative prompt selects the plugin	False activation	Disallowed cases, scope overlap, tool inventory	Plugin product and metadata owner
Correct plugin selects the wrong sibling tool	Selection precision	Tool names, descriptions, parameter boundaries	Plugin product and API designer
Correct tool receives wrong arguments	Argument correctness	Parameter documentation, schema, defaults, exact prompt	API and tool-contract owner
Correct call returns an error or malformed result	Execution	Server logs, downstream response, permissions, output schema	Engineering or integration owner

Metadata cannot repair an unpublished version, expired authentication, missing permission, or broken server. Conversely, a successful connection does not prove that the model will select the correct tool.

Keep Public AI Visibility And Private Activation QA Separate

Plugin builders usually need two measurement programs:

Public answer monitoring: Do relevant buyer Questions produce accurate brand mentions, recommendations, competitor framing, and inspectable citations?
Authorized plugin QA: Does an eligible, connected test state select the right tool, pass correct arguments, and complete safely?

AEO Table supports the first program through repeatable Questions, Tasks, Runs, answers, brand and competitor observations, and citation evidence on supported AI answer channels. The AI search monitoring guide explains that public-answer workflow, while the competitor tracking guide shows how to preserve rival framing and source evidence.

Do not describe AEO Table as a private ChatGPT developer-mode harness, plugin connection verifier, internal tool-call logger, or directory-ranking tracker. Keep activation logs in the authorized test environment. You can share an intent taxonomy, release calendar, and evidence-review cadence across the two programs, but never merge their denominators.

For executive reporting, use the AI search visibility report template for the public lane and attach a separate activation appendix containing:

Version and environment identity.
Attempted, eligible, completed, and excluded cases.
Direct and indirect activation recall.
Activation precision and negative false activation.
Argument correctness and end-to-end completion.
Prompt-level regressions.
Listing, availability, and connection failures.
Metadata diff, decision, owner, and next retest date.

If competitors appear more often in public answers, the AI share-of-voice framework can quantify the governed comparison cohort. That result still says nothing about private plugin activation unless a separate eligible tool-selection test observes it.

A Practical Release Checklist

Before accepting a metadata version, confirm:

The expected behavior for every golden prompt was labelled before execution.
Direct, indirect, and negative classes are reported separately.
At least five positive and three negative submission cases are reproducible.
Every high-risk action has positive, refusal, clarification, and boundary coverage as appropriate.
Eligibility and exclusion rules were frozen before the run.
Listing, availability, connection, activation, arguments, and completion are separate states.
Recall, precision, false activation, and argument correctness show numerator and denominator.
Only one metadata field changed in the comparison.
Tool annotations match actual behavior.
Prompt-level regressions and raw evidence are preserved.
The claim language does not promise ranking, traffic, recommendations, or calls.

The Bottom Line

Plugin activation is not a directory impression and not a successful OAuth connection. It is an observed selection decision for one labelled, eligible prompt under one recorded version and test state.

Build the golden set around real jobs. Include direct prompts for explicit use, indirect prompts for unbranded intent, and negative prompts for containment. Measure selection recall, precision, false activation, argument correctness, and completion separately. Change one metadata field at a time, preserve every version, and diagnose failures at the gate where they occur.

That process can improve the evidence behind a release decision. It cannot guarantee directory rank, traffic, or future tool calls.

If you also need a repeatable baseline for the public buyer Questions your plugin serves, create a free AEO Table account. Keep the private activation harness alongside it as a distinct, authorized QA system.