How LLMs Decide What to Mention

Every major LLM starts with CommonCrawl, the largest open web corpus. GPT-3 derived over 80% of its training tokens from CommonCrawl; LLaMA allocated 67% to CommonCrawl and 4.5% to Wikipedia. At least 64% of 47 LLMs reviewed (2019–2023) used at least one filtered version of CommonCrawl.

The CCNet pipeline processes raw web data through five stages: data sourcing from CommonCrawl (a single snapshot is 8.7 TiB compressed), paragraph-level deduplication using SHA1 hashing (~70% removed), language identification via FastText, quality filtering against Wikipedia perplexity scores, and reference content filtering.

NVIDIA’s Nemotron-CC pipeline processes the full CommonCrawl English dataset of 6.3 trillion tokens into 2 trillion high-quality tokens using 28 distinct heuristic filters. Training with Nemotron-CC data boosted MMLU scores by 5.6 points over baseline (59.0 vs. 53.4).

After pre-training, models are fine-tuned via Reinforcement Learning from Human Feedback (RLHF) in three stages: preference dataset creation (human annotators rate outputs pairwise), reward model training, and RL fine-tuning using PPO.

RLHF doesn’t create explicit brand preferences but shapes models toward outputs human raters consider helpful and safe. Models avoid recommending controversial brands due to safety filters. “Digital consensus” matters: when multiple authoritative sources agree on a brand’s category position, RLHF-trained models adopt this as reliable information.

Reddit is the most-cited domain by Google AI Overviews and Perplexity, and the second most-cited by ChatGPT. Google algorithm updates (Aug 2023–Apr 2024) nearly tripled Reddit’s readership from 132M to 346M monthly visitors.

Research from Wang et al. (ICLR 2025) found that factual QA showed the strongest memorization effect, increasing alongside model scaling. Pre-training data frequency directly correlates with output probability distributions.

Brands frequently mentioned in high-quality content before the training cutoff become part of the model’s neural weights and get mentioned automatically, without web search. When models cannot confidently retrieve lesser-known brands, they substitute competitors with higher probabilistic weight — the “substitution effect.”

An ACM SIGIR 2025 study found that “fresh” passages shift the Top-10’s mean publication year forward by up to 4.78 years. Individual items moved by as many as 95 ranks in reranking experiments.