SEO and AEO best practices.
Every claim sourced.
Search engines rank pages, and answer engines like ChatGPT quote passages. These rules cover both SEO and answer engine optimization (AEO).
View SEO expertiseWhich numbers should you remember?
Eight signals carry most of a page's technical weight, and each comes down to a number. The sections below show where each figure comes from.
| Signal | The rule | The number |
|---|---|---|
| Title element | One per page, subject first, no shared prefix across a template | 70 characters rendered |
| Meta description | One per URL, written as a snippet candidate rather than an answer | 100 to 160 characters |
| Canonical | One absolute URL, matched by every link, sitemap row and JSON-LD url | 1 URL form, slash included |
| Heading ladder | One h1, no skipped levels, question-shaped h2 headings | 1 h1 per page |
| Open Graph | og:title, og:type, og:image and og:url on every page | 4 required properties |
| XML sitemap | Canonical, indexable, 200-only URLs with a lastmod you can defend | 50,000 URLs and 50MB per file |
| robots.txt | Separate groups for training crawlers, retrieval agents and user-triggered fetchers | 500 KiB parse limit |
| JSON-LD | One graph per page, one @id per entity, identifying fields inlined | 1 script tag |
How long should a page title be, and what belongs in it?
A page title should name one subject, lead with what sets it apart, and stay short enough to survive truncation in results. Google publishes no character limit. It says it builds the title link from the title element, on-page headings and anchor text. It rewrites the tag whenever it reads as generic, stuffed or duplicated across a site. Pick a working ceiling and enforce it in the build. This site hard-fails its own build above 70 rendered characters. That leaves 53 for the page-specific half once the build appends a 17-character brand suffix.
The failure mode is rarely a long title. It is a set of titles that all open with the same six words, because the first thirty characters are the only part a person scans and the part a retriever weights hardest. Front-load the noun that separates this page from every other page on the domain, then delete the pipe-delimited keyword tail.
What makes a meta description worth writing?
A meta description is worth writing as a snippet candidate that earns the click, not as a ranking input. Google picks between it and text lifted from the page, depending on what the searcher typed. That is why a page with no description still gets a snippet. Write one per URL at 100 to 160 characters. Describe what the page decides, not what it covers, and hold the answer back so the snippet earns a click instead of replacing one.
Google asks for descriptions that are unique per page and accurate about the content. A single description repeated across a template is worse than none at all, because it teaches the engine that the tag is boilerplate and it stops using the tag anywhere on the site.
Which URL should the canonical point at, and does the trailing slash matter?
The canonical should point at one absolute URL on the production domain, the page's single official address. Every internal link, sitemap entry, og:url and JSON-LD url for that page should match it exactly. The trailing slash matters, because /guide and /guide/ are separate URLs unless one redirects to the other. Choose one form, redirect the other to it with a 301, and never list a URL in a sitemap that redirects.
Google treats rel=canonical as a strong signal rather than an instruction, and it selects its own canonical whenever the signals on a site disagree. Disagreement is the thing to remove. A page that links to itself without the slash while its sitemap lists it with the slash is arguing against itself, and the engine settles that argument on its own terms.
How many H1 elements should a page have?
One. The HTML specification permits any number and Google tolerates several. That makes this a discipline rule rather than a penalty rule, and the discipline pays off downstream. A single h1 that states the subject hands a screen reader, a summarizer and a retrieval chunker the same unambiguous document title. Every heading below it should descend one level at a time. A jump from h2 to h4 breaks the outline that assistive technology and passage-level retrieval both walk.
Treat the heading ladder as the table of contents a model reads. A retriever splits a long page on heading boundaries, so an h2 phrased as a real question and followed by a self-contained answer is the unit that gets stored, scored and quoted. A heading that reads Overview produces a chunk no query can match.
Which Open Graph tags actually matter?
Four. The Open Graph protocol, which sets link previews, names og:title, og:type, og:image and og:url as required for every page. Everything past those four is optional. Add og:description, because most platforms render it. Generate a 1200 by 630 pixel card per route instead of one site-wide image. A shared link with a generic card converts worse than one that names the page it opens.
og:url is the property teams forget and the one that matters most for consolidation, since the protocol defines it as the canonical URL that serves as the object's permanent identifier in the graph. Point it at the exact string the canonical tag carries, trailing slash included.
What does an honest XML sitemap look like?
An honest XML sitemap lists only canonical, indexable URLs that answer 200. It stays under 50,000 URLs and 50MB uncompressed per file, and its lastmod values are true. Google says it ignores priority and changefreq. It uses lastmod only when the value is consistently and verifiably accurate, for instance when it matches the page's actual last modification.
A pipeline that stamps today's date on every URL at deploy time is worse than a sitemap carrying no lastmod at all, because one dishonest column teaches the crawler to distrust the whole file. Reserve lastmod for a real change to the main content, the structured data or the links, and keep the copyright year out of it.
What does robots.txt control, and what does it not?
robots.txt controls crawling and nothing else. Search engines can still index a disallowed URL from links elsewhere on the web. To keep a page out of the index, use a noindex robots meta tag, not a Disallow line. The file must be UTF-8 at the root of the origin. Google enforces a 500 kibibyte size limit and ignores every rule past it. RFC 9309 requires any conforming parser to read at least that same 500 kibibytes.
The snippet controls belong to the same family and deserve a deliberate setting. max-snippet, max-image-preview and nosnippet govern how much of a page an engine may display, and Google's own guidance points site owners at those controls rather than at robots.txt when they want to limit what shows up inside AI features on Search.
Which AI crawler tokens should robots.txt name?
Ten crawler names, called tokens, cover the engines that matter today. They sort into three groups, not the usual two. GPTBot, ClaudeBot and CCBot collect content that may feed model training, so disallowing them costs a site nothing on any live answer surface. OAI-SearchBot, PerplexityBot, Claude-SearchBot and Claude-User fetch pages to build or answer from a live index. They honor robots.txt, and disallowing them removes the site from the answer itself. Google-Extended, ChatGPT-User and Perplexity-User each behave unlike both groups, and each one gets read wrong in a different direction.
Google-Extended is the token most often filed under costs nothing, and that filing drops half the story. Google documents it as a standalone token that governs whether crawled content trains future Gemini models, and whether Gemini uses it for grounding. Google defines grounding as supplying Search index content to the model at prompt time. That happens in Gemini Apps and in Grounding with Google Search on Vertex AI. Google also states that Google-Extended does not impact a site's inclusion in Google Search and is not a ranking signal there. Both halves are true at once. Blocking it costs nothing in Search but forfeits the grounding citation in Gemini. Treat one token as two decisions, not one free opt-out.
The user-triggered fetchers carry the opposite error, because a disallow line there may never be enforced. Perplexity states that Perplexity-User generally ignores robots.txt rules since a person asked for the fetch. OpenAI says the same of ChatGPT-User. A user starts those actions, so robots.txt rules may not apply, and OpenAI points site owners at OAI-SearchBot for managing what appears in ChatGPT search. Anthropic takes the other route and documents that its bots, Claude-User included, honor robots.txt directives. You do control one piece of this, and OpenAI says so outright. Allow OAI-SearchBot and you can appear in search results. Disallow GPTBot and you stay out of foundation model training. OpenAI adds that each setting works independently of the others.
| robots.txt token | Operator | What it does | What blocking costs |
|---|---|---|---|
| Google-Extended | Gemini model training and grounding, controlled separately from Search | Nothing in Google Search. Forfeits grounding in Gemini Apps and on Vertex AI | |
| GPTBot | OpenAI | Crawls content for foundation model training | Training exclusion only |
| OAI-SearchBot | OpenAI | Builds the index behind ChatGPT search results | Removes you from ChatGPT search |
| ChatGPT-User | OpenAI | Fetches a page when a person asks ChatGPT about it | Little. OpenAI documents that robots.txt rules may not apply to this fetcher |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search results | Removes you from Perplexity results |
| Perplexity-User | Perplexity | Visits a page to answer a question a person just asked | Little. Perplexity documents that this fetcher generally ignores robots.txt |
| ClaudeBot | Anthropic | Collects web content that may contribute to model training | Training exclusion only |
| Claude-User | Anthropic | Retrieves a page in response to a Claude user's question | Blocks live answers about your site |
| Claude-SearchBot | Anthropic | Crawls pages to improve the quality of Claude's search results | Weakens your place in Claude's search answers |
| CCBot | Common Crawl | Builds the open corpus most public models train on | Training exclusion only |
How should a JSON-LD entity graph be structured?
One script tag per page, one @graph inside it, and one @id per real-world entity that every other node references. The point of the graph is identity rather than decoration. Put a Person node in one script and an Article pointing at it in a second, and a validator reads two separate documents. It then reports a thin duplicate entity instead of merging the two into one.
Inline the identifying fields of anything you reference. A crawler that lands on one page cannot follow a pointer to another page. So an author declared only by a bare @id publishes no author name if the crawler skipped the home page. Google also requires that structured data represent content visible to the reader, so every claim in the markup needs a matching claim in the copy.
Do FAQPage and BreadcrumbList markup still earn anything?
BreadcrumbList still earns a result feature, and FAQPage no longer does. On desktop results, where Google documents the feature, Google prints the trail in place of a raw URL. So BreadcrumbList belongs on every page below the root. Google narrowed the FAQ feature to well-known, authoritative government and health sites on 8 August 2023. It stopped showing it in Search at all from 7 May 2026, and deleted the documentation for it on 15 June 2026. A page marking up an FAQ today gets no accordion and no extra pixels from Google, whoever runs the site.
Ship the FAQPage node anyway, for the reason that outlived the rich result. It encodes a clean question and answer pair in machine-readable form, which is the exact shape a retrieval system chunks, stores and quotes back. Google is explicit that no special schema.org markup is required to appear in its AI features, so treat this as cheap structure rather than a ticket to anything. The node costs one object inside a graph the page already emits, and it survives a feature Google can retire again.
What are E-E-A-T signals, and how do you show them?
E-E-A-T stands for Experience, Expertise, Authoritativeness and Trustworthiness. It comes from Google's Search Quality Rater Guidelines rather than from any single input in the ranking system, so no meta tag sets it. You show it by naming a real author with a linked profile and dating the last significant revision honestly. Write first-hand detail only someone who did the work would know, and cite primary sources by link, not by assertion.
The cheapest version of this is a number with a source attached. A sentence stating that sitemaps cap at 50,000 URLs and 50MB per file, credited to the sitemaps.org protocol, survives being lifted into an answer somewhere else. A sentence telling the reader to keep sitemaps small does not, because it carries nothing a model can attribute.
What is llms.txt, and does publishing one do anything?
llms.txt is a proposed convention, a markdown file at a site's root that hands an agent a curated index of the pages worth reading. Jeremy Howard published it on 3 September 2024 and it now sits at a second revision. No major engine has committed to reading it, so nobody should promise rankings from it. It costs one route and doubles as a human-auditable inventory of what a site claims to offer. Keeping it in sync is close to free.
Be honest about what it buys. Google states outright that you do not need new machine-readable files, AI text files or special schema.org markup to appear in its AI features. So nobody should sell llms.txt as the thing that gets a site cited. Pair the index with a fuller variant and keep both true. This site serves /llms.txt as the index and /llms-full.txt as the expanded corpus. A test fails the build whenever /llms.txt advertises a URL that /llms-full.txt omits. That holds the corpus to a superset of the index, not the subset it briefly was. An index that drifts away from the site it describes is worse than no index.
When do you actually need hreflang?
You need hreflang, the tag linking language versions, only when the same content exists in more than one language or regional variant. A single-language site needs none, and adding one is pure upkeep. Where it applies, every URL in a set must reference every URL in that set, itself included. The annotations must agree in both directions. An x-default entry should name the page a visitor lands on when no variant matches their language.
The usual failure is a one-way declaration, where the English page names the Arabic variant and the Arabic page names nothing back. Google may ignore annotations that lack return references, which drops the site back where it started while leaving extra markup behind to maintain.
How do you measure AI citations instead of guessing at them?
Fix a query set, run it against the live engines on a schedule, and record whether the domain appears and where. Guessing is the default because the surfaces feel opaque, yet half the data already sits in Search Console. Google states that appearances in AI features such as AI Overviews and AI Mode count inside overall search traffic. It reports them in the Performance report under the Web search type. The other half of the picture comes from running the queries yourself against real engines.
This site does that with a harness in packages/tooling/scripts/ai-visibility. It queries live ChatGPT, Gemini and Copilot through searchapi.io and records the citation rate against a fixed query set. Measurement is the whole point. Google states there are no additional requirements to appear in AI Overviews or AI Mode beyond normal SEO practice. The only honest way to know whether a change moved anything is to measure the same queries before it ships.
Where were these rules tested?
On pooyagolchian.com, with numbers recorded before and after. Nothing below comes from a case study, and the third card shows the gap.
0.32% to 0.86%
Sitewide click-through rate
Measured across the August 2026 work on flat impressions. Average position moved from 11.9 to 8.7 in the same window.
ChatGPT, Gemini, Copilot
Engines the harness queries
The harness queries live engines through searchapi.io instead of inferring from rankings. In August 2026, this site got cited more often than it ranked.
16 results
Total results for the brand phrase
This domain ranks first for pooya golchian, with no sitelinks or knowledge panel. The gap is off-site mentions, not markup.
Questions this reference gets asked
Short answers to the arguments teams raise when they start optimizing for answer engines too.
Mostly the same work with a sharper output format. Google states there are no additional requirements to appear in AI Overviews or AI Mode beyond normal SEO practice, so crawlability, canonical hygiene and honest structured data still carry the weight. What changes is the writing. An answer engine quotes a passage rather than ranking a page, so a heading phrased as a question followed by a complete self-contained answer with a number in it is the unit that gets retrieved.
Block the pure training crawlers if the policy calls for it, and leave the retrieval agents open. GPTBot, ClaudeBot and CCBot only collect content for model training, so disallowing them costs nothing on any live answer surface, and OpenAI documents that split explicitly, noting each setting works independently of the others. Google-Extended is the one most people file in that harmless group by mistake. It controls Gemini training and grounding together, so blocking it changes nothing in Google Search while giving up the grounding citation in Gemini Apps and on Vertex AI. OAI-SearchBot, PerplexityBot and Claude-User honor robots.txt and blocking them removes the site from the answer. ChatGPT-User and Perplexity-User run on a person's request, and both operators say robots.txt may not govern those fetches, so a disallow line there records a preference rather than closing a door.
Weeks rather than days, and click-through rate moves before position does. On pooyagolchian.com, the August 2026 pass moved sitewide click-through rate from 0.32% to 0.86% and average position from 11.9 to 8.7. Impressions stayed flat, which is the signature of better titles and snippets against an unchanged crawl. Split the Search Console window at the deploy date. A 90-day average blends the before and after into something that reads as a broken site.
No. A knowledge panel comes from an entity Google is confident about, and confidence is built from independent sources that describe the same entity the same way. Markup helps a crawler reconcile what it already found elsewhere. This domain ranks at position 1 for its own brand phrase with a complete Person graph and still shows no panel and no sitelinks, against 16 total results for the phrase, which is a volume and off-site density problem rather than a schema problem.
Rewriting the first paragraph under each heading so it stands on its own. Most pages open a section with a transition sentence that means nothing once it is lifted out of context, and that is the sentence a retriever stores. Name the subject, state the claim, attach a number, and cite the source. It costs nothing in build time and it is the difference between a page a model can quote and a page it has to summarize from scratch.
No, and that now holds for every site rather than most of them. Google narrowed the FAQ rich result to well-known, authoritative government and health sites on 8 August 2023, stopped showing it in Search from 7 May 2026, and removed the documentation for it on 15 June 2026. Keep shipping FAQPage markup regardless, because it encodes question and answer pairs in the exact shape a retrieval system chunks and quotes, and that is where the value sits now. Expect nothing extra in the Google result, because the feature is gone rather than restricted.