The technical SEO checklist for 2026

This checklist covers the technical foundations that decide whether Google, Bing and AI engines can crawl, render, index and understand your site. Work through it from top to bottom, because the early sections block everything after them. Most checks can be run with Google Search Console and the free tools linked throughout.

Can search engines crawl your site?

Search engines can crawl your site when robots.txt allows the pages you want found, an XML sitemap lists your canonical URLs, and each page points to a single preferred version with a canonical tag. Crawl problems come first in any audit because a page that can't be crawled can't rank on its content, however good it is.

robots.txt

• robots.txt sits at the root of each hostname (https://www.example.com/robots.txt) and returns a 200 status
• No leftover Disallow: / from a staging site or redesign
• CSS, JavaScript and image files needed to render pages aren't blocked
• robots.txt isn't used to keep pages out of the index; use noindex for that, because Google can still index a blocked URL if other pages link to it
• The sitemap location is declared with a Sitemap: line

XML sitemaps

• Only canonical, indexable URLs that return 200 are listed
• Each sitemap file stays within the protocol limits of 50,000 URLs and 50MB uncompressed; larger sites use a sitemap index
• lastmod reflects real content changes; Google's documentation says it ignores priority and changefreq
• Sitemaps are submitted in Google Search Console and Bing Webmaster Tools

If your CMS doesn't produce a clean sitemap, our XML sitemap generator builds one from a list of URLs.

Canonical tags

• Every indexable page has a self-referencing canonical tag with an absolute URL
• Canonicals point to URLs that return 200 and aren't redirected or noindexed
• HTTP and HTTPS, www and non-www, and trailing-slash variants all resolve to one version
• Parameter URLs for sorting, tracking or filtering canonicalize to the clean URL when the content is the same

Google treats rel="canonical" as a strong hint rather than a directive, so it may choose a different canonical when your other signals disagree. The URL Inspection tool in Search Console shows both the canonical you declared and the one Google selected.

Are your important pages indexed?

Check indexation in Google Search Console's Page indexing report, which shows how many URLs are indexed and why others are excluded. The goal isn't to index everything. It's to make sure every page you want to rank is indexed, and that thin, duplicate or parameter pages aren't crowding out the ones that matter.

• Review the reasons pages aren't indexed: "Crawled – currently not indexed" and "Discovered – currently not indexed" often point to quality or crawl-priority problems, while "Duplicate without user-selected canonical" points to canonical problems
• Compare the indexed count with the number of URLs in your sitemaps
• Spot-check each key template (product, service, location, article) with URL Inspection, including the rendered HTML
• Confirm no important page carries a stray noindex meta tag or X-Robots-Tag header
• Give empty category pages and "no results" pages real content, or return a 404, to avoid soft 404s
• Make sure every indexable page has a unique title and meta description; check lengths with our meta length checker

Crawl budget is rarely the problem on small and mid-sized sites. Google's own crawl budget guide is written for very large sites, such as those with a million or more unique pages, or 10,000 or more pages whose content changes daily. If your site is smaller, indexing gaps usually come down to quality, duplication or internal linking.

Is your site architecture helping or hurting?

Good architecture keeps important pages within a few clicks of the homepage, groups related pages into logical folders, and links between them with descriptive anchor text. Internal links tell search engines which pages matter most and how topics relate, so orphan pages with no internal links are hard for crawlers to find and rank.

• Revenue pages, such as core services, top categories and key products, are reachable within three clicks of the homepage
• URLs are short, lowercase, hyphenated and follow the folder structure (/services/water-heater-repair/)
• No orphan pages: every indexable page has at least one internal link from a crawlable page
• Links are standard <a href> elements; Google says it can generally only follow links built that way, not JavaScript click handlers
• Anchor text describes the target page instead of saying "click here" or "learn more"
• Deep pages have breadcrumbs, marked up with BreadcrumbList structured data
• Faceted navigation and filters don't generate thousands of crawlable URL combinations

To check the wording of those links, run key pages through our anchor text analyzer. Google explains its requirements in making links crawlable.

Do your pages pass Core Web Vitals?

Core Web Vitals measure loading, responsiveness and visual stability with three metrics: Largest Contentful Paint (LCP), Interaction to Next Paint (INP) and Cumulative Layout Shift (CLS). A page passes when at least 75% of real visits meet Google's "good" thresholds: LCP within 2.5 seconds, INP within 200 milliseconds and CLS of 0.1 or less.

INP replaced First Input Delay (FID) as a Core Web Vital in March 2024. It's a stricter test: FID only measured the delay before the first interaction was processed, while INP looks at the responsiveness of interactions throughout the visit. The thresholds below are from Google's Web Vitals documentation.

• Check field data from real users in Search Console's Core Web Vitals report and PageSpeed Insights, not just lab scores
• LCP: serve the main image at the right size in a modern format, don't lazy-load it, and preload it if the browser discovers it late
• LCP: cut server response time with caching and a CDN
• INP: break up long JavaScript tasks, defer non-critical scripts, and audit third-party tags such as chat widgets, tag managers and testing tools
• CLS: set width and height on images, video and ad slots, and reserve space for banners and embeds
• CLS: load web fonts with font-display and closely matched fallback fonts to limit text shifting

Is your site ready for mobile-first indexing and HTTPS?

Google indexes and ranks the mobile version of your pages, so the mobile site must carry the same content, internal links, structured data and metadata as desktop. Every page should also load over HTTPS with a valid certificate, with HTTP URLs permanently redirected. Both are baseline expectations now, not competitive advantages.

• Mobile and desktop show the same primary content, headings and internal links
• Content inside tabs or accordions on mobile is still present in the HTML
• Structured data and meta robots tags match on mobile and desktop
• Text is readable and tap targets are usable on a phone, with no intrusive interstitials covering content
• Images and video on mobile use crawlable URLs and descriptive alt text
• Every URL serves HTTPS with no mixed-content warnings, and HTTP and non-canonical hostnames 301 to the HTTPS canonical

Google's mobile-first indexing best practices cover the details.

Is your structured data valid and useful?

Structured data is valid when it uses a supported schema.org type, matches content visible on the page, and passes Google's Rich Results Test without errors. It's useful when it describes your business, products, articles or people clearly enough for search engines and AI systems to connect your pages to real-world entities.

• Markup uses JSON-LD, the format Google recommends
• The homepage has Organization or LocalBusiness markup with name, logo, URL and sameAs links to official profiles
• Deep pages carry BreadcrumbList markup
• Article, Product, Service or Event markup is used on the templates where it fits
• Every property reflects content users can see, with no markup for hidden text or invented reviews
• No errors in the Rich Results Test or Search Console's enhancement reports
• Markup validates against schema.org in the Schema Markup Validator

Some rich results have narrowed. Since 2023, Google has shown FAQ rich results mainly for well-known government and health sites and has retired HowTo rich results, so don't add FAQ markup expecting the expanded snippet.

For a deeper walkthrough of which types to use, see our schema markup guide.

Can crawlers see your JavaScript-rendered content?

Google can render JavaScript, but rendering can be deferred, and many other crawlers, including most AI crawlers, fetch only the raw HTML. If your main content, links or metadata appear only after JavaScript runs, some systems will see an empty page. Server-side rendering or static generation is the safest approach for content you want found.

Google describes its process in three phases: crawling, rendering and indexing, with pages queued for rendering before their JavaScript runs. Other crawlers are less capable. An analysis Vercel published in late 2024 found that the major AI crawlers it studied, including OpenAI's and Anthropic's, did not execute JavaScript at all.

• View source (not the browser inspector) on key templates and confirm the main content, title, meta description, canonical and internal links are in the raw HTML
• Compare raw and rendered HTML with URL Inspection in Search Console
• Avoid relying on JavaScript to inject canonical tags, noindex directives or structured data
• Give each view a real URL rather than a # fragment
• Return proper HTTP status codes; a single-page app that shows a "not found" view with a 200 status creates soft 404s
• Don't block the JavaScript or CSS files needed for rendering in robots.txt

Is hreflang set up correctly for international sites?

Hreflang is set up correctly when every language or regional version of a page lists all its alternates, including itself, using valid language and optional region codes, and every alternate links back. Missing return links are a common failure. Sites that serve only US English don't need hreflang at all.

• Language codes use ISO 639-1 (en, es) and optional regions use ISO 3166-1 Alpha 2 (en-us, es-mx); "en-uk" is invalid, so use "en-gb"
• Each page references itself and every alternate
• Every alternate links back with matching return tags
• An x-default points to a language selector or fallback page
• Every hreflang URL is canonical, indexable and returns 200
• One implementation method is used consistently: HTML head, HTTP headers or the XML sitemap

A US example: a company with an English site and a Spanish site for Spanish-speaking US customers would use en-us and es-us, plus an x-default.

Are redirects, 404s and crawl logs under control?

Redirects are under control when every moved URL 301s directly to its closest equivalent in one hop, internal links point straight at final URLs, and removed pages with no replacement return 404 or 410. Server logs then confirm what crawlers actually request, which is the most direct way to see crawl waste and errors from the crawler's side.

Redirects and 404s

• Permanent moves use 301 or 308; temporary moves use 302 or 307
• No redirect chains or loops; old redirects are updated to point to the final destination
• Internal links, canonicals and sitemaps reference final URLs, not redirected ones
• Removed pages redirect only to genuinely relevant replacements; mass redirects to the homepage may be treated as soft 404s
• The 404 page returns a real 404 status and helps visitors find what they wanted
• Every migration has a tested redirect map before launch

Our redirect generator turns an old-to-new URL list into server-ready redirect rules.

Log file analysis

• Raw access logs from your host or CDN cover at least several weeks
• Googlebot hits are verified by reverse DNS or Google's published IP ranges, since user agents are easy to fake
• You know which sections get crawled most and which important pages are rarely crawled
• Crawl waste is identified: parameters, redirects, 404s and retired URLs that crawlers keep requesting
• AI crawler activity (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) is tracked separately

Can AI crawlers access your site?

AI crawlers can access your site only if robots.txt, your firewall and your CDN all allow them. Decide deliberately which to allow: search-focused bots like OAI-SearchBot and PerplexityBot help you appear in AI answers, while training-focused tokens like GPTBot and Google-Extended govern whether your content is used to train models.

Google's AI Overviews and AI Mode are part of Google Search and rely on Googlebot, so Google-Extended doesn't control them. Check each operator's own documentation, such as OpenAI's crawler overview, before you write rules, because bot names and purposes change.

Here is one example policy, for a site that wants to appear in AI search but opt out of model training. Many businesses allow training bots too; it's a business decision, not a technical one.

# Appear in AI search and answers
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Opt out of model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

• A written policy decides, bot by bot, which AI crawlers to allow
• robots.txt has explicit groups for each user agent you treat differently
• Your CDN, firewall or host isn't blocking AI crawlers through default bot protection, regardless of what robots.txt says
• Key content is in the raw HTML, not injected by JavaScript
• Server logs show allowed bots receiving 200 responses
• An llms.txt file is published if you decide it's worth the small effort

llms.txt is a proposed standard for a Markdown file at your site root that points language models to your most useful pages. It's cheap to publish, but Google Search doesn't use it, so treat it as optional rather than a fix. Our llms.txt guide explains how to write one.

If you'd rather have this checklist run for you, our SEO agency can audit your site and hand your developers a prioritized fix list. Our AEO service covers AI crawler access and citation tracking.