Manual Prompt Checks vs Automated ChatGPT Tracking

A manual check is a person typing a buyer question into ChatGPT and reading the answer. Automated tracking sends the same questions to an engine's official API on a schedule, several times each, stores every answer and counts how often your brand is named and cited. Both can be done well or badly. The real difference is the number of samples and whether the conditions stay the same between them.

This page asks one practical question: how far can a single manual check be trusted? We answer it with our own tracker data for the 7 sites of Netbase JSC, the company behind Leadee, then look at cost and at when a manual check is all you need. For the ChatGPT basics, see the ChatGPT SEO overview.

In the 2026-10-25 round, 159 question sets were asked exactly 3 times each (a set is one question, on one engine, in one market). In 124 sets no run named the Netbase brand, in 19 every run did, and in 16 the runs disagreed: the brand was in 1 or 2 of the 3 answers.

Sixteen out of 159 sounds small, but most of the 124 empty sets are questions where the brand never had a chance (NetbaseJSC, a Netbase-owned site, on "best Vietnam software development companies", for example). Where the brand did appear at least once, 16 of 35 sets were mixed. That is the zone a business cares about: the questions where it is sometimes in the answer and sometimes not.

Disagreement was most common on Google AI Mode (10 mixed sets out of 70) and less on Perplexity (5 of 84). ChatGPT had only 5 sets with 3 runs, because the OpenAI account ran out of credit; one of them was mixed: CMSmart was named in 2 of 3 runs for "best WooCommerce product designer plugin for print on demand".

Take the 35 sets where the brand showed up at least once. If you had checked each of them only once, picking one of the 3 runs at random, you would have seen no mention in 22 of the 105 runs: 21%, with a 95% interval of 14.3-29.7%. In other words, about one manual check in five would have told you "we are not there" for a question where you sometimes are.

For questions without the brand name, the ones that show whether a buyer who does not know you yet will hear of you, it is worse: 15 of 36 runs missed the brand (42%, interval 27.1-57.8%). Only 2 of those 12 unbranded sets named the brand every time. For branded questions the miss rate was 7 of 69 (10%).

The error goes both ways. A lucky single check can also show you in an answer that leaves you out most of the time. One run cannot tell those two cases apart; three runs can start to, and a rate over many questions and weeks can. How to check if ChatGPT mentions your brand explains how to turn counts into a rate with an interval.

Beyond the run-to-run variation above, a person's check changes with the conditions. The signed-in ChatGPT app can use saved memory and chat history, the selected model, whether a web search is triggered at all, and location. Two colleagues asking the same question on the same day can get different answers for reasons that have nothing to do with your website. We have not measured that effect, so we put no number on it.

Wording drifts too. "Best WooCommerce product designer plugin" and "top product designer plugins for WooCommerce" are different questions to an engine. A manual routine that is not written down tends to change its wording from month to month, which makes comparisons meaningless. An automated tracker uses the stored wording every time.

A manual check also rarely keeps the full answer. Without the saved answer and its source links, you cannot go back and see which pages the engine relied on, which is the information you need to get cited when ChatGPT prefers other websites.

In API fees, less than people assume, but not the same on every engine. Our whole round was 492 runs for 3.95 USD: 25 ChatGPT runs (OpenAI gpt-5 with web search) for 2.39 USD, about 0.096 USD each; 252 Perplexity sonar runs for 1.30 USD; 215 Google AI Mode runs for 0.26 USD. ChatGPT is by far the most expensive engine per answer to measure.

Manual checks have no API fee, but they cost time, and we have no measured figure for how long a careful check takes, so we do not quote one. What we can say is what the volume looks like: 20 questions, 3 runs each, once a week is 240 answers a month for one engine in one market, each to be read, classified and logged. The ChatGPT monitoring cost calculator shows the API side of that volume.

Switch when you want to know whether something you changed made a difference. That question needs the same wording, the same engine settings and enough runs before and after the change for the intervals to separate. With a 30% rate, 10 runs give an interval of roughly 11-60%; 100 runs narrow it to about 22-40%. Reaching that many runs by hand every month is where most manual routines stop.

Automate also when you track more than one engine or market, or unbranded questions in a competitive niche, which is exactly where our data showed the most disagreement between runs. Leadee does this through official APIs (see pricing); whichever tool you use, check that it repeats each question, shows the sample size and keeps the answers. ChatGPT vs Gemini vs Perplexity shows why engines should only be compared on the same questions.