ChatGPT Citations Match OpenAI API Only 26% of the Time

ChatGPT citations match Google and OpenAI Responses API sources only 26% of the time—what the study really measures for AI citation tracking.

Two analysts in a sunlit coworking space reviewing documents at a shared wooden table, one holding a pen and gesturing toward an open laptop

What That 26% Figure Actually Measures

The 26.2% is a page-level overlap number, and it is worth being precise about what sits on each side of the division. Take every page ChatGPT cited on GPT-5.5 Instant across 80 questions with 20 runs each, put that set next to the union of Google organic results and the source list returned by the OpenAI Responses API, and slightly more than a quarter of the cited pages appear in both places. Move the comparison from URLs to domains and the same test covers 48.7% of the sites behind those citations. That spread, roughly double at the site level, is the part of the paper I would read first, because it tells you where the measurement problem lives.

The setup was 80 questions pulled from Search Arena, a public database of real user interactions with search-enabled chatbots. Forty were written in English and forty in Chinese, spanning recommendation, fact lookup, information synthesis and analysis requests. Each question ran 20 times on GPT-5.3 Instant and 20 times on GPT-5.5 Instant. For the Google side, SerpAPI pulled up to five pages of organic results per question, about 43 results on average. For the API side, the same questions went to gpt-5.3-chat-latest and gpt-5.5 through the Responses API with web search on, all within three days of the matching ChatGPT session. The researchers at the National University of Singapore and Xi’an Jiaotong University posted the results on October 11, and Search Engine Journal covered the findings (opens in new tab) shortly after.

One caveat belongs near the top rather than buried at the end. This is an arXiv preprint and arXiv does not peer review. The methodology is unusually careful, since the authors recorded data flow inside their own signed-in ChatGPT sessions instead of guessing at it from outside, but the numbers have not been through external review yet.

Here is why the page-versus-site gap matters for anyone building tooling around ChatGPT citations. If you track exact URLs, your dashboard will miss half the picture even when your stand-in data is as good as Google plus the API combined. If you track domains, you are measuring something a site owner can act on, and the paper’s own authors land in the same place: report sites, not URLs.

The method is the part worth understanding, because most citation studies work from the outside. You ask ChatGPT a question, you read the answer, you copy down the links, and you call that the citation set. That approach misses whatever ChatGPT looked at and then declined to cite, and it misses the searches it ran to get there.

These researchers did something different. They recorded the data flow that ChatGPT’s web application received during its own writing process, by watching their signed-in sessions. In that flow you can see the searches ChatGPT issued, the pages those searches returned, and the specific pages ChatGPT then selected as sources. Think of it as reading the network tab while the answer writes itself, rather than reading only the finished answer.

A page counted as cited if ChatGPT picked it as a source, and that includes pages bundled into a grouped source card rather than listed as their own individual link. That detail matters more than it sounds. If you scrape only the standalone links from a response, you are undercounting, and you will conclude a page was not cited when ChatGPT did in fact use it.

The stand-ins they compared against were two surfaces you can query without opening ChatGPT. Google organic results came through SerpAPI, up to five pages per question. The other was gpt-5.3-chat-latest and gpt-5.5 through the OpenAI Responses API with web search enabled, 20 queries per question. Both were run within three days of the matching ChatGPT session, so the comparison is reasonably tight in time, which is not nothing given how fast these models turn over.

That turnover caused a real limitation. GPT-5.2 Instant and GPT-5.4 Thinking got fewer runs because OpenAI retired them mid-collection. Two collection sessions switched to another model after 14 and 16 runs respectively. So the older-model comparisons rest on a thinner sample, and the authors themselves suggest checking which model generated each response before you lean on those numbers.

What’s Happening Now: ChatGPT Citations Drift Between Runs and Languages

Before you can compare citation sets, you have to know what ChatGPT actually searched for, and the paper answers that plainly. On GPT-5.3 Instant and GPT-5.5 Instant, in every run where searches were visible, ChatGPT never queried the exact wording of the original question. Not once. GPT-5.2 Instant did it in 5.5% of runs, which makes the newer models look more aggressively rewrite-happy, not less. Every prompt you write is a starting draft that the model immediately scraps.

The language behaviour is stranger. In runs where searches were visible for Chinese-language questions, 67.0% of GPT-5.3 Instant runs and 69.6% of GPT-5.5 Instant runs issued English-only search queries. A user types in Chinese, ChatGPT hunts in English, and the sources land in a completely different index. That lines up with Peec AI’s finding earlier in the year that 43% of fan-out queries for non-English prompts ran in English. Two independent groups, similar direction. If your content is only strong in a non-English market, you are being searched for in a language you may not have optimised in.

Then there is the instability, which is the part I keep coming back to. For a given question on GPT-5.5 Instant, 44.5% of the distinct pages returned across 20 runs appeared in exactly one of those runs. On GPT-5.3 Instant it was 40.8%. Nearly half the page inventory is a one-off sighting. Site-level overlap held up better than page-level, which is the paper’s recurring theme, but it does not rescue the picture.

Here is how I would explain it to a client. Imagine twenty customers walk into the same shop with the same shopping list over twenty days, and on any given day roughly one in three of the items they pick up was chosen on that day and no other. You would not build a stock plan from a single visit. A one-shot citation check is that single visit.

As for which stand-in did better, the API’s full source list covered 22.9% of cited pages on GPT-5.3 Instant and 22.5% on GPT-5.5 Instant, while Google organic alone managed 9.4% and 6.4%. Neither is close to complete, and the API’s source list is not the same thing as the pages the API itself chose to cite, which matched ChatGPT’s citations even less often. You can read the full breakdown in Search Engine Journal’s write-up (opens in new tab).

ChatGPT Citation Tracking: Direct Answers to the Obvious Questions

Can I use the OpenAI Responses API to track ChatGPT citations?

Not as a substitute, no. The API’s full source list overlapped with 22.9% of the pages cited on GPT-5.3 Instant and 22.5% on GPT-5.5 Instant. Worse for anyone building a tracker on it, the pages the API actually chose to cite in its own answers matched ChatGPT’s cited pages less often than its raw source list did. So the one field you would naturally reach for, the API’s own citations, is the weaker signal of the two.

I have had this argument with people who assume the API is the same engine underneath. It is the same model family, not the same retrieval path, and the difference is large enough to matter. Building a citation dashboard on gpt-5.5 with web search enabled gives you a view of what that endpoint retrieves, not what a signed-in user sees in the ChatGPT app.

Does ChatGPT cite the same pages Google ranks?

Roughly a tenth of them, or less. Google organic results alone accounted for 9.4% of cited pages on GPT-5.3 Instant and 6.4% on GPT-5.5 Instant, moving to 26.1% and 22.8% if you count whole sites rather than exact URLs. For a site owner, that number should be uncomfortable. You can sit at position two for a query and still be absent from the ChatGPT answer for the same question, and the reverse happens too.

The overlap is not zero, and the researchers pulled about 43 results per question through SerpAPI, which is a generous slice of the SERP. A typical rank tracker checks ten. So even the 9.4% figure is probably flattering to a rank-tracking dashboard that only looks at page one.

Do ChatGPT citations stay stable across repeated runs?

They do not, and this is the question I would put first if I were buying a tool. Around four in ten distinct pages showed up in a single run out of twenty, on both GPT-5.3 Instant and GPT-5.5 Instant. The paper does note that pages appearing across more runs were more likely to be referenced again, which is at least a usable signal. Citation frequency, not citation presence, is the thing worth measuring.

A binary cited / not cited badge, sampled once, tells you almost nothing about what the next user will see.

What It Means in Practice for AI Citation Tracking

The combined stand-ins reached 50.3% of cited sites on GPT-5.3 Instant and 48.7% on GPT-5.5 Instant, against 26.2% at the page level on GPT-5.5. That gap is where a usable tracking practice lives. A client can act on whether their domain shows up in ChatGPT answers for a set of questions. Nobody rewrites a service page because one URL was missed in a sample of twenty runs.

So track at the site and subdomain level, and make the subdomain decision on purpose. The paper counts subdomains as separate sites, which means blog.example.com and www.example.com land in different buckets. Decide whether that matches how you think about the property, write it down, and keep it consistent across reporting periods. Changing that rule mid-quarter will make a flat trend look like growth.

Rerunning prompts matters more than picking better prompts. If roughly four in ten distinct pages appear in only one of twenty runs, then a single weekly check is close to noise. Run each prompt ten to twenty times and report a citation rate: “cited in 14 of 20 runs” is a defensible statement. “Cited” alone is not, because you have no idea whether the next user sees the same thing.

Rebaseline after every model change, and date-stamp the model version alongside the results. The spread across GPT-5.2 Instant through GPT-5.5 Instant in this paper suggests citation behaviour moves with the model, not just with your content edits. A drop in citation rate the week a new snapshot ships is not necessarily a content problem. If you cannot name the model version behind a report, the report cannot be compared to the one before it.

I would also be blunt with clients about coverage. A 26.2% page-level match between the two outside stand-ins and the real app means no third-party tool, mine included, can claim it sees ChatGPT’s full citation set from the outside. Anyone selling that claim is describing the OpenAI API’s retrieval, not ChatGPT’s behaviour, and those are demonstrably different surfaces. Say “we estimate” where you would previously have said “we track.” It is a weaker sentence and a truer one, and it survives the first time a client checks ChatGPT themselves and sees a domain your dashboard never listed.

The Search Engine Journal writeup (opens in new tab) has the full set of tables if you want the per-question numbers before you rebuild a dashboard around them.

What to Expect Next from ChatGPT Search Sources and the Responses API

My expectation, and it is only an expectation, is that OpenAI eventually narrows the distance between what the web app retrieves and what the API returns with web search enabled. The reason is commercial rather than technical. The Responses API is the only surface outside ChatGPT that developers can query at any real volume, and right now its source list overlaps with 22.9% of the pages cited on GPT-5.3 Instant and 22.5% on GPT-5.5 Instant. A retrieval product that misses three quarters of what the consumer app does is hard to build a monitoring business on top of. Closing that gap costs OpenAI engineering time and gives it nothing directly, so I would not hold my breath for a fast fix, but the pressure from developers building on the API is real and it points one direction.

Expect more work on language mixing too. Both this paper and the earlier Peec AI numbers land in the same place: a Chinese-language prompt very often produces English-only search queries inside the fan-out, 67.0% and 69.6% of visible runs in this study. That is a routing behaviour with obvious consequences for anyone optimising content in a non-English market, and it has barely been measured. I would guess the next round of papers goes at it directly rather than treating it as a side observation.

The peer review question matters more than it usually does here. This is an arXiv preprint from NUS and Xi’an Jiaotong, posted October 11, and arXiv does not review submissions. The methodology, recording real session data flow instead of guessing at it, is the strongest part and the reason I am citing the numbers at all. Treat 26.2% as a well-argued estimate rather than a settled constant until somebody replicates it on a different question set.

The marketing shift is the one I am most confident about. Vendors who spent the last year saying they track ChatGPT citations will start saying they estimate citation likelihood, because the second claim is defensible and the first one is not once a client checks for themselves. That is a better conversation for everyone involved, even if it makes the dashboards look less impressive in a pitch deck.

A Concrete Next Step: Build Your Own Baseline This Week

The 26.2% number is useful for context, not for your own reporting. Borrowed benchmarks fall apart when a client asks which prompts you tested. Build a small baseline yourself, and do it before the next model release makes the question urgent.

Pick 20 prompts that look like the searches your site actually earns traffic from. Not vanity questions. If you run a VPS host, one prompt might be “cheapest VPS with cPanel license included”. If you run a recipe blog, it might be “how long to roast a spatchcock chicken at 200C”. Use the wording a customer would type, including the awkward phrasing.

Run each prompt 10 to 20 times in the ChatGPT web app. Log the cited domains, not the exact URLs. This is the part most people get wrong. A URL-level log will show you twenty variants of the same page and hide the fact that the site appears in every run. I keep a plain text file with one line per prompt, the date, the model name, and a comma-separated list of domains. There is no database and no dashboard involved. A text file survives tool migrations.

Then run the same prompts through the OpenAI Responses API with web search enabled, and through Google (SerpAPI or whatever you already pay for). Write down the overlap. You will get your own page match and site match figures. They will not be 26.2%, because your prompts are not their prompts, and that is fine. The point is having a number you can defend.

Save the file with the model version in the name, for example citation-baseline-gpt-5.5-2026-10.txt. When the next model ships, rerun the same 20 prompts with the same wording, and diff the domain lists. That diff is the only citation data you control from end to end, and it will tell you more about your visibility than any third-party tracker built on the outside looking in. Run this before you renew another LLM visibility subscription.