Information gain is everywhere

It's time you really understand what it is

Hi 👋🏻

Two concepts have been fascinating to me in relation to AI search systems - information gain and search intent (more on the latter - in a future edition).

When people discuss how AI search differs from traditional search, they often focus on surface-level differences: links vs. summaries, single answers vs. multi-turn conversations, SERP features vs. generated content.

But there's a deeper architectural shift happening that rarely gets discussed—and it hinges on a concept Google has been quietly using for years: information gain.

Information gain is a score that measures how much new information a piece of content provides beyond what a user has already encountered on that topic.

Traditional Google search uses information gain passively. It looks at what you've already seen, notes the gap, and quietly reorders results to show you something novel. It's responsive—you search, it adapts.

AI Mode (based on patent filings) uses information gain actively. It doesn't just reorder links; it anticipates what you don't know yet, generates new queries on your behalf, and systematically guides your search trajectory toward expanded understanding. It's proactive.

From reactive to generative personalization

In traditional Google search, information gain scoring does three things:

1. It classifies pages you've viewed vs. not viewed
2. It measures novelty (actual vs. expected co-occurrence rates of concepts)
3. It deprioritizes repetitive information

The system is fundamentally defensive—protecting you from redundancy.

AI Search systems like AI Mode flip this:

1. They maintain continuous state about what you know (what you've read, which concepts you've engaged with, your depth of understanding)
2. They use that state to generate new queries that intentionally drill into areas you haven't covered
3. They actively guide your search trajectory toward information gaps you might not have thought to ask about

Instead of just showing you less repetition, the system is now asking: What would most efficiently expand this person's understanding?

The three mechanisms that make the information gain scoring come to life in AI Search

The patent describes three components of this new model:

1. "New Information" Encoding – Unlike Google's passive classification, the system continuously updates what counts as "known." If you've read a beginner overview, the next summary automatically gets more granular. The system doesn't wait for you to realize you want depth; it assumes you do.

2. Guided Search Trajectory – Synthetic queries are generated to steer you toward information that wasn't in your first result (...an SEO might call this intent drift and be totally correct; for the user it might just be what they need). If you viewed a document, the system asks: What's in the broader discourse that wasn't in this specific source? It doesn't just want to show you the results - it wants to guide you on a path.

3. Reflected Familiarity – Each interaction updates the system's understanding of where you are in your learning journey. Summaries rewrite themselves not just based on new documents, but based on your demonstrated knowledge level from previous interactions.

What this means: Personalization becomes pedagogical

In traditional search, personalization is about efficiency—show me fewer dupes, serve me results that matter to my context.

In AI Mode, personalization becomes instructional. The system is now thinking like a teacher who knows what you've learned and what gap to fill next. It's curating not just results, but your entire learning sequence.

This has massive implications for how content gets surfaced. It's not enough to write unique content anymore (though that still matters). Your content now competes on whether it credibly expands understanding given what the user has already learned in this session.

For content and SEO strategy, this escalates the stakes on several fronts

The information gain concept isn't new to Google—but its application in AI Mode makes it far more visible and consequential. Where Google's information gain operated mostly behind the scenes (reranking results you'd never see), AI search makes it explicit through query generation and intentional breadth.

This means:

• Content must have clear depth architecture. Not just "beginner vs. advanced," but what specific aspects of a topic does each piece illuminate? Writing a truly unique and narrow-targeted content piece might be the difference between being sourced versus cited for your target audience.
• First-party data becomes more valuable. AI Mode wants to guide users toward novel synthesis—and proprietary research, case studies, and original analysis are the highest-signal sources of genuine novelty.
• Content architecture is now a ranking factor. AI systems will generate queries designed to connect related pieces of your content. If those connections are weak or missing, users will encounter gaps. Seamless semantic relationships between your articles can become a competitive advantage.

Google's information gain in traditional search was always trying to do what conversational AI now does explicitly—understand what you know and show you what you don't. The old system just did it one search at a time, reactively. The new system does it across a session, proactively.

Watch this video for a deep dive on the concept of information gain ✨

​

Side note, we've started publishing lesson snippets on our YouTube channel, covering concepts like: knowledge graphs, EAV model, implicit user feedback and user behaviour, query augmentation, query paths, and more.

A lot more is on the way - subscribe if you'd like to see more of this content in your feed. 💜

---

LEarn about other advanced semantic search concepts our Featured course 🌟

Let us teach you fundamental concepts about semantic query understanding and how traditional search engines have systematically personalised search results for years, combining multiple explicit and implicit signals. We cover dozens of patents to show you exactly what you need at the keyword research phase to ensure you create meaningful content.

SEMANTIC AI-powered keyword research Two new modules are in development, to be released in November. ✨

This course features 🔥 10+ hours of content already, with a ton of practical exercises, tools, checklists, scripts, and more.

Heads-up 💸 the price of this course will increase when the new modules drop in November —lock in now.

START THIS COURSE TODAY ✨

✨ Use code COMMUNITY30 for 30% off at checkout - that's our little thank you for being part of the MLforSEO community

85+ forward-thinking marketers are already taking our courses 💜

COMMUNITY OPPORTUNITY 🌟

We've got some free tickets to give away to the TechSEO Connect Conference 💜

Complete the form to grab one - this isn't an event to miss for any technically-inclined people reading and within the vicinity of the event 👀

Join 670+ AI/ML-interested marketers on our Slack community to stay up to date with discussions on AI/ML automation in SEO and marketing.

Happy learning! ✨

Lazarina