Technical SEO: Schema, Core Web Vitals, and Crawlability

What technical SEO actually is (and isn't)

Technical SEO is the part of search that has nothing to do with your words and everything to do with a machine's ability to reach them, read them, and trust them. Content is what you say. Technical SEO is what decides if the message gets delivered at all. I have watched genuinely excellent articles sit at position 40 for a year because a stray line in a robots file told Google not to bother, and I have watched mediocre pages outrank them because the plumbing underneath was clean. The plumbing wins more often than anyone likes to admit.

The discipline breaks into three pillars, and I will spend most of this article inside them. Crawlability and indexation asks a blunt question: can a crawler find your URLs, fetch them, render them, and decide to store them in the index. Structured data asks: does the machine understand what the page is, a product, a recipe, an author, an FAQ. Core Web Vitals asks: does the page feel fast and stable to a real human on a real phone. Miss any one of the three and you cap the ceiling on everything above it.

People consistently get one part wrong. They treat technical SEO as a one-time audit, a PDF you commission, act on for two weeks, and forget. It is not a project. It is a maintenance discipline, closer to keeping a server patched than to redesigning a logo. Sites rot. A CMS upgrade quietly injects a noindex tag. A marketing team ships a landing page with a 6MB hero image. A developer adds a tracking script that blocks the main thread for 400 milliseconds. None of these are content problems, and none of them will show up in a keyword tool. They show up in your rankings three months later when nobody remembers what changed.

Technical SEO is the most learnable part of the whole field, because it is deterministic. Content is taste and judgment. Link building is relationships and luck. But a crawler either can reach a page or it cannot. A schema either validates or it throws an error. An LCP is either 2.1 seconds or 4.8 seconds. You can measure all of it, fix it with confidence, and verify the fix. For a skeptical, engineering-minded person, this is the friendliest corner of search there is.

A quick word on who this is for. If you are a marketer who has never opened a robots file, this article will make you dangerous in the good way: you will know what to ask your developers and how to tell if the answer is right. If you are a developer, it will connect the pieces you already touch (caching, redirects, image formats, render strategy) to the ranking outcomes they drive, so the work stops feeling arbitrary. And if you are a founder watching traffic and wondering why the content investment is not paying off, it will give you the three questions to ask before you spend another dollar on words: can they crawl it, can they understand it, does it feel fast.

One more framing before we go deep. Every other ranking surface I write about, answer engines, voice, visual search, knowledge graphs, inherits from these three pillars. An AI Overview will not cite a page it could not crawl. A voice assistant will not read a Speakable block that never got indexed. A shopping agent will not parse a Product schema that fails validation. Technical SEO is not one surface among many. It is the substrate the others stand on.

The three pillars, and why order matters

If you only remember one thing from this article, remember the order: crawl, understand, experience. A search engine has to reach and store the page before it can understand it, and it has to understand it before the experience metrics mean anything. Fixing your Core Web Vitals on a page Google has decided not to index is like polishing the trim on a car with no engine. Sequence your work the way a crawler experiences your site, from the outside in.

Pillar one is crawlability and indexation. This is the gate. It covers robots.txt, XML sitemaps, crawl budget, canonicalization, redirects, rendering, and the log-file evidence of what the crawler actually did. The failure modes here are binary and brutal: a page is indexed or it is not, discovered or invisible. When something goes wrong at this layer, no amount of good content saves you, because the content is never seen.

Pillar two is structured data. Once a page is in the index, structured data is how you remove ambiguity about what it contains. Left to its own devices, a crawler infers meaning from your HTML, and it is decent at it, but inference is lossy. JSON-LD schema hands the machine a labeled diagram: this string is the price, this is the author, this is the star rating, these five blocks are questions and answers. Get it right and you become eligible for rich results, the enhanced listings with stars and images and FAQs that eat a disproportionate share of clicks.

Pillar three is Core Web Vitals, Google's user-experience metrics that became a confirmed ranking signal. Largest Contentful Paint measures load speed, Interaction to Next Paint measures responsiveness, and Cumulative Layout Shift measures visual stability. These are not vanity numbers. Every extra 100 milliseconds of latency shaves roughly a percent off conversion on transactional pages, and AI Overviews quietly skip slow sites when they choose citations. Speed is content's silent partner.

The three pillars are not equal in payoff, and they are not equal in effort. Crawlability problems are rare but catastrophic, so you check them first and rarely. Structured data is medium effort and compounding value, so you build it into templates once and reap it forever. Core Web Vitals is the highest ongoing effort, because performance regresses every time anyone ships code, so it needs monitoring, not a one-time fix. A sane technical SEO program spends its time in exactly that inverse proportion.

Crawlability: robots.txt, sitemaps, and crawl budget

Crawling starts with a text file most people never open. robots.txt sits at the root of your domain and tells crawlers where they may and may not go. It is a blunt instrument and a dangerous one. A single misplaced Disallow line can wall off an entire section of your site, and because the file is so small and so rarely touched, nobody notices for weeks. The two mistakes I see constantly are blocking resources the page needs to render (CSS and JS folders that Googlebot must fetch to see the page the way a user does) and, far worse, a Disallow left in from a staging environment that shipped to production. Read your robots.txt out loud before every launch.

The critical distinction robots.txt cannot express is the difference between crawling and indexing. Disallow stops a crawler from fetching a URL, but it does not remove that URL from the index, and a blocked page can still appear in results as a bare link with no description because Google saw it referenced elsewhere. If you want a page gone, you do not block it in robots.txt; you let it be crawled and put a noindex directive in the page's meta or HTTP header. Block-to-deindex is the classic own-goal: you told the crawler not to look, so it never sees the noindex tag that would have removed the page.

XML sitemaps are the opposite tool: an invitation rather than a fence. A sitemap is a list of the URLs you consider canonical and want indexed, with optional lastmod timestamps that hint at freshness. It does not guarantee indexation, but it dramatically speeds discovery, especially for deep pages that few internal links reach. Keep sitemaps clean: only 200-status, canonical, indexable URLs belong in them. A sitemap stuffed with redirects, 404s, and noindexed pages teaches Google that your sitemap is noise and to trust it less. Split large sitemaps into files of at most 50,000 URLs and reference them from a sitemap index. Submit the index in Search Console and Bing Webmaster Tools.

Crawl budget is the resource nobody thinks about until a site gets large. Googlebot allocates a finite amount of crawling to each site, a function of how fast your server responds and how much it wants your content. On a 200-page brochure site, crawl budget is irrelevant; Google will happily recrawl everything daily. On a 2-million-URL ecommerce site, it is the whole game, and the enemy is waste. Faceted navigation that generates infinite parameter combinations, session IDs in URLs, endless calendar pages, and soft-404s all burn budget on garbage instead of your money pages. The fix is disciplined: block crawl-trap parameters, flatten redirect chains, return proper 404s for dead pages, and keep the important pages within three clicks of the root.

There is a hierarchy of directives worth memorizing because people constantly reach for the wrong one. robots.txt Disallow controls crawling only. The meta robots noindex tag (or the X-Robots-Tag HTTP header, which does the same job for non-HTML files like PDFs) controls indexing. rel=canonical controls which duplicate is preferred. These are three different levers for three different jobs, and using one to do another's work is the root of most indexation bugs. Want a page crawled but not indexed? noindex, and do not block it. Want to consolidate duplicates? canonical. Want to save crawl budget on truly worthless URLs? Disallow. Match the lever to the job.

The signal you are optimizing at this layer is efficiency: the ratio of crawls that hit valuable, indexable pages to crawls wasted on duplicates and dead ends. You do not measure that with a keyword tool. You measure it in server logs, which we will come back to, and in the Search Console crawl-stats report, where a healthy site shows steady crawling of 200-status HTML and a sick one shows Googlebot drowning in redirects and not-found responses.

Canonicalization and redirects: telling Google which URL is real

Duplicate content is rarely plagiarism. It is almost always the same page reachable through five different URLs: with and without www, http and https, trailing slash and none, with tracking parameters appended, and the printer-friendly version nobody remembers building. Each variant is, to a crawler, a separate URL competing with the others for the same ranking. Canonicalization is how you consolidate that split, telling Google 'these are all the same thing, and this one is the real one.'

The primary tool is the canonical link element, rel=canonical, placed in the head of every page and pointing at the preferred URL. On a page with no duplicates it should be self-referential, pointing at its own clean URL, which sounds redundant but resolves a surprising number of parameter-based duplication issues automatically. The rule that trips people up: a canonical is a hint, not a command. Google treats it as a strong suggestion and usually honors it, but if your signals conflict (canonical says one thing, the sitemap says another, internal links point somewhere else, a redirect contradicts all of them) Google picks the canonical it finds most credible, which may not be the one you declared. Consistency across every signal is what makes the canonical stick.

Redirects are the other half of URL management, and the type matters more than people realize. A 301 is permanent and passes the vast majority of ranking signal to the destination; use it whenever a URL moves for good. A 302 is temporary and tells Google to keep indexing the original; use it only when the move truly is temporary, like a seasonal promotion. Mixing these up is a slow leak: 302s where you meant 301 keep the old URL alive in the index and dilute the new one for months.

The redirect problem that quietly wrecks large migrations is chains and loops. A chain is URL A redirecting to B redirecting to C redirecting to D. Every hop adds latency, and every hop bleeds a little signal; crawlers may give up before reaching the end, and users watch a spinner. Keep chains to a single hop wherever you can, and never let a redirect point at a URL that itself redirects. A loop, where A sends you to B and B sends you back to A, is worse: it is an infinite trap that returns nothing and burns crawl budget until the crawler bails. After any migration, crawl the whole site and hunt for chains longer than one hop and any loops at all.

There is a subtle interaction worth internalizing. Canonicals, redirects, sitemaps, and internal links are four voices telling Google which URL is real, and they must agree. The most common indexation bug I diagnose is not one broken signal, it is four signals disagreeing: the sitemap lists the trailing-slash version, the canonical points at the non-slash version, internal links use both, and a redirect flips between them. Google throws up its hands and picks one you did not want. Pick one canonical form for your whole site, http-to-https and non-www to www or the reverse, and enforce it everywhere, mechanically, in every template.

Rendering, JavaScript SEO, and log-file evidence

Modern sites are JavaScript applications, and that creates a gap between the HTML a server sends and the HTML a user sees. When Googlebot fetches your page, it gets the initial HTML response first. If your content is baked into that response (server-side rendering or static generation), the crawler has everything immediately. If your content is assembled in the browser by JavaScript (client-side rendering), the crawler sees an empty shell and has to come back later, render the JavaScript in a headless Chrome, and only then discover your content. Google can do this, but rendering is expensive, so it is queued and delayed, sometimes by days, and it is not guaranteed to complete.

This is the heart of JavaScript SEO. The safest architecture is to send meaningful HTML in the initial response. Server-side rendering and static site generation both do this; they render the page on the server and ship real content, then hydrate interactivity on top. Client-side-only rendering is the risky pattern: critical content, internal links, canonical tags, and structured data that exist only after JavaScript runs may be seen late or not at all. If your money content depends on a fetch that fires after page load, assume some crawlers and most AI answer engines will never wait for it.

The practical test is simple and I run it on every audit. Fetch the page with JavaScript disabled, or use the URL Inspection tool in Search Console and look at the rendered HTML Google actually captured. Is your headline there? Your body copy, your links, your JSON-LD? If the raw response is an empty div and everything appears only in the rendered version, you are betting your rankings on Google's render queue clearing in your favor. Sometimes it does. On large or slow-to-render sites, often it does not, and pages sit half-indexed with title tags but no body.

A few rendering rules that save real pain. Do not lazy-load content that matters for SEO behind user interaction; if it is important, it should be in the DOM at load. Make sure internal links are real anchor tags with href attributes, not click handlers on divs, because a crawler follows hrefs and ignores onclick. Keep your structured data server-rendered so it is present in the initial response. And never block your JavaScript bundles in robots.txt, because if Google cannot fetch the JS, it cannot render the page and will index the empty shell.

Server logs are the ground truth that cuts through all speculation, and almost nobody looks at them. A log file records every request Googlebot actually made: which URLs, how often, what status code came back, how long the server took. This is not a tool's estimate of what Google might do; it is the receipt of what Google did. Log analysis answers questions no other tool can: which of my pages has Google never crawled, how much of my crawl budget is being spent on parameter junk, is Googlebot hammering a redirect chain, did crawling of my new section actually pick up after launch. Pull a month of logs, filter to verified Googlebot, and you will find surprises every time: whole product categories uncrawled, thousands of hits on a faceted-navigation trap, a spike of 500 errors during a deploy window that torched your crawl rate.

The quieter foundations: HTTPS, mobile-first, and hreflang

Three technical foundations do not fit neatly into crawl, schema, or speed, but they gate everything above them, and skipping them undoes a lot of good work. They are HTTPS and security, mobile-first indexing, and internationalization with hreflang. None of them is glamorous. All of them will quietly cost you if they are wrong.

HTTPS is not optional and has not been for years; it is a confirmed ranking signal and, more practically, browsers now flag plain HTTP pages as 'not secure,' which torches trust and conversion. But installing a certificate is only step one. The failure I find most often is mixed content: an HTTPS page that loads an image, script, or stylesheet over HTTP, which browsers block or warn on and which breaks the padlock. Audit for mixed content after any migration. Make sure every HTTP URL 301-redirects to its HTTPS equivalent, that your canonical tags point at the HTTPS version, and that HSTS is enabled so browsers refuse to even try the insecure version. Half-migrated HTTPS, where some assets or some pages still serve over HTTP, is worse than either state alone because it confuses both browsers and crawlers.

Mobile-first indexing means Google predominantly uses the mobile version of your page for indexing and ranking. Not the desktop version, the mobile one. This sounds obvious in 2026, but it still catches sites that quietly serve less content, fewer internal links, or stripped-down structured data on mobile to keep the layout clean. If your mobile page hides half the body copy behind a tap, omits the breadcrumbs, or drops the schema, that is the version Google indexes, and you are ranking on the diminished page. The rule is parity: the mobile experience must contain the same primary content, the same structured data, and the same links as desktop. Responsive design gets you there by default, which is one more reason it beats separate mobile URLs.

Internationalization is where large sites bleed slowly. If you serve the same or similar content to multiple countries or languages, hreflang annotations tell Google which version to show which user: the Spanish page to Spanish speakers, the UK page with UK pricing to UK visitors. Get it right and you avoid the wrong-country result and the duplicate-content confusion of near-identical pages competing with each other. Get it wrong and you get the classic hreflang mess: annotations that do not reciprocate (page A points to B but B does not point back), missing self-references, or region and language codes that do not match the standard. hreflang is fiddly and error-prone, so validate it with a dedicated checker, and remember it works alongside canonicalization, not against it: each language version canonicalizes to itself and uses hreflang to point at its siblings, never canonicalizes across languages.

The thread connecting these three is that they are all parity-and-consistency problems, the same disease as the four-voices canonicalization bug from earlier. HTTPS half-done, mobile serving a different page than desktop, hreflang pointing in only one direction: each is a case of your signals disagreeing about what the real, canonical, correct version of a page is. Search engines resolve disagreement by guessing, and they guess wrong often enough to hurt. The fix in all three cases is the same discipline: pick the correct version, and make every signal on the site agree, mechanically, in the template, so a human error cannot reintroduce the split.

Structured data: which schema types actually matter

Schema.org defines hundreds of types, and you will use maybe a dozen. The trap is treating structured data as a checklist to max out rather than a set of claims you make to become eligible for specific search features. Google only renders rich results for a defined list of schema types, and marking up a type that has no rich-result treatment does nothing for your appearance in search. So the question is never 'what can I mark up,' it is 'what enhanced result do I want to be eligible for, and what schema makes me eligible for it.'

Start with the types that earn their keep on almost every site. Organization schema, placed on the homepage and referenced sitewide, establishes your entity: name, logo, sameAs links to your verified social and Wikidata profiles. It feeds the Knowledge Graph and is the backbone of your brand's identity to machines. BreadcrumbList on every non-root page gives Google the crumb trail it shows in results instead of a raw URL, which improves click-through for free. Article schema on editorial content, with author, datePublished, and dateModified, supports Top Stories eligibility and freshness signals. WebSite schema with a SearchAction can earn the sitelinks search box. These four are foundational, cheap, and belong in your templates.

Then come the money types, the ones tied to the high-value rich results. Product schema with an Offer, price, availability, and aggregateRating is what puts the price and star rating on a shopping listing, and in 2026 it is table stakes for ecommerce visibility. FAQPage schema turns a block of questions and answers into an expandable listing that physically occupies more space in results (Google narrowed FAQ rich-result eligibility, so treat it as a bonus, not a guarantee, and never fake Q&As to grab it). Review and AggregateRating power stars. Recipe, Event, JobPosting, Video, HowTo, and LocalBusiness each earn a specific vertical treatment, and you use the ones that match what you actually publish.

A discipline point that separates the pros. Structured data must describe what is genuinely on the page, visibly, to a user. Marking up a rating that appears nowhere on the page, adding FAQ schema for questions users never see, inflating a Product with data it does not have: these are structured-data spam, and Google issues manual actions for them. The schema is a labeled description of visible reality, not a place to smuggle in claims. If a human cannot see it on the page, it should not be in your JSON-LD.

A note on breadth versus depth, because people ask which types to add first. The answer is depth on the few types that map to your business, not breadth across the catalog. A perfectly complete Product schema on every product page, with valid offers, real ratings, and correct availability, is worth more than a dozen half-filled schema types sprinkled around. Google rewards accurate, complete markup on the types it renders rich results for, and ignores the rest. Pick the two or three types that describe what you actually sell or publish, implement them completely in your templates, validate them, and stop. Resist the urge to mark up your postal address as PostalAddress inside a Place inside a LocalBusiness if you are a pure ecommerce play with no storefront; it earns you nothing and adds surface area for bugs.

The strategic reason to invest here goes beyond blue-link rich results. Structured data is increasingly how AI answer engines and shopping agents parse your page without guessing. When an assistant needs a price, an author, an event date, or a set of specs, well-formed schema hands it a clean, unambiguous answer instead of forcing it to scrape prose and hope. As search fragments into answer engines and agents, the sites that labeled their data explicitly are the ones those systems can act on. Schema is quietly becoming the API layer of the open web.

JSON-LD patterns, validation, and the errors everyone makes

There are three ways to add structured data (JSON-LD, Microdata, and RDFa) and you should use exactly one of them: JSON-LD. Google recommends it, it is the easiest to maintain because it lives in a single script block instead of being tangled through your HTML attributes, and it is the format every modern tool expects. Microdata scattered across your markup is a maintenance nightmare that breaks the moment someone edits a template. Put a script type application/ld+json block in the head or body, write a clean JSON object, and keep your presentation HTML separate from your data.

The pattern that scales is to think in connected entities rather than isolated blobs. A well-built page does not have five unrelated schema blocks; it has a graph. The Article references its author as a Person and its publisher as an Organization. The Product references its brand and its offers and its reviews. You express this with @id references: define the Organization once, give it an @id, and point to that @id from the Article's publisher field instead of repeating the whole object. This keeps your data consistent (one source of truth for the org) and helps Google understand the relationships between entities, which is exactly what the Knowledge Graph runs on.

Validation is non-negotiable and takes thirty seconds. Two tools matter. The Schema.org validator checks that your JSON-LD is syntactically valid and conforms to the vocabulary. Google's Rich Results Test checks a stricter thing: does your markup qualify for a specific Google rich result, and what would it look like. Run both. A schema can be perfectly valid Schema.org and still miss a required property Google demands for the rich result, so the Rich Results Test is the one that tells you if you actually earned the feature. In Search Console, the Enhancements reports then show you validation errors across your whole site at scale, grouped by type, which is how you catch a template bug that broke schema on 40,000 pages at once.

The errors are boringly consistent, which means they are easy to prevent once you know them. Missing required properties is number one: Product without an offer or a review, Article without a headline, Event without a start date. Google silently drops the rich result and you never see the stars you built for. Malformed values are number two: a price with a currency symbol in it instead of a plain number, a date that is not ISO 8601, an image URL that 404s. Mismatched data is number three and the most dangerous: schema that claims a 4.8 rating the page does not show, which crosses from error into spam. And the quiet killer is stale schema after a redesign: the template changed, the fields moved, and your JSON-LD now references a price selector that no longer exists, so every product ships broken markup.

Build validation into your pipeline so it cannot regress. The professional move is to add automated structured-data testing to your deploy process, so a build fails if a template emits invalid JSON-LD, the same way you would fail a build on a broken unit test. Structured data is code; treat it like code. Version it, test it, and monitor it in Search Console's Enhancements reports for the slow drift that manual spot-checks always miss. The teams that do this ship rich results reliably; the teams that hand-check a few pages before launch ship broken schema they discover months later when the stars vanish.

Core Web Vitals: what LCP, INP, and CLS actually measure

Core Web Vitals are Google's attempt to reduce 'is this page a good experience' to three numbers a machine can measure at scale. They are a confirmed ranking signal, folded into the page-experience assessment, and while they are a tiebreaker rather than a top-weight factor, they are the tiebreaker that decides close races, and there are millions of close races. More to the point, they correlate directly with money: faster pages convert better, and slow pages lose users before the content ever loads. You should care about these numbers even if Google did not rank on them.

Largest Contentful Paint measures loading. Specifically, it is the moment the largest content element in the viewport, usually the hero image or the headline block, finishes rendering. It is a proxy for 'when does the page look loaded to the user.' Under 2.5 seconds is good, over 4 seconds is poor, measured at the 75th percentile of your real users. LCP is dominated by a small number of things: server response time, render-blocking resources in the head, and the load time of the LCP element itself. When LCP is bad, the culprit is almost always a slow server, a giant unoptimized hero image, or a stylesheet blocking the render.

Interaction to Next Paint measures responsiveness, and it replaced First Input Delay in March 2024. INP looks at the latency of interactions across the whole page visit, from tap to the next frame the browser paints, and reports a value that represents the worst of your interactions. It answers 'when I tap or click, how long until something visibly happens.' Under 200 milliseconds is good, over 500 is poor. INP is a JavaScript problem almost every time: long tasks blocking the main thread, heavy event handlers, expensive hydration, or a page firing a dozen analytics calls the instant a user touches an input. If your page feels laggy and janky, INP is the metric that catches it.

Cumulative Layout Shift measures visual stability: how much the page jumps around as it loads. You know the feeling, you go to tap a button and an ad loads above it and pushes everything down and you tap the wrong thing. CLS quantifies that with a score; under 0.1 is good, over 0.25 is poor. The causes are specific and fixable: images without width and height attributes so the browser cannot reserve space, ads and embeds injected without a reserved slot, web fonts that swap and reflow the text, and content inserted above existing content after load. CLS is the most preventable of the three because every cause is a case of not reserving space for something you knew was coming.

The trap with all three is measuring them wrong, and it comes down to field data versus lab data, which deserves its own section. For now, hold this: LCP is your loading budget, INP is your JavaScript discipline, and CLS is your layout hygiene. They map cleanly to three different teams' work: LCP to infrastructure and images, INP to front-end JavaScript, CLS to design and CSS. When you get a bad score, the metric tells you which door to knock on.

Diagnosing and fixing LCP

LCP failures decompose into four sub-parts, and diagnosing which one you have is most of the battle. The four are: time to first byte (how long your server takes to respond), resource load delay (how long before the browser even starts fetching the LCP element), resource load time (how long the LCP element itself takes to download), and element render delay (how long after loading before it actually paints). Chrome DevTools and PageSpeed Insights break LCP into exactly these buckets. Look at which bucket dominates and you know where to aim.

If time to first byte is your problem, the fix is infrastructure, not front-end. A slow server response, often database queries, uncached dynamic rendering, or a distant origin, means the browser is waiting before it can do anything. Cache aggressively: put static assets and, where possible, whole pages behind a CDN with edge caching so the response comes from a server near the user instead of your origin across the ocean. Use HTTP/2 or HTTP/3. Server-side render or statically generate so the response is ready HTML, not a computation that runs on every request. TTFB fixes are unglamorous and enormously effective; shaving a slow origin from 800ms to 100ms fixes LCP before you touch a single image.

If the LCP element is a hero image, and it usually is, image optimization is your highest-impact move. Convert to a modern format: AVIF or WebP with a JPG fallback typically cuts image weight in half or more, and swapping a hero JPG for AVIF alone can pull a second off LCP. Serve responsive sizes with srcset so a phone downloads a phone-sized image instead of your 1600px desktop hero. And add fetchpriority high to the LCP image, which tells the browser to fetch it immediately instead of in the normal queue, a trivial one-attribute change that reliably shaves a few hundred milliseconds.

The most common self-inflicted LCP wound is lazy-loading the wrong thing. Lazy loading is correct for below-the-fold images, where it saves bandwidth. It is disastrous on the LCP element, because it explicitly tells the browser to delay the one image that defines your score. I have fixed pages where the entire LCP problem was a well-meaning developer adding loading lazy to every image including the hero. Rule: lazy-load below the fold, eager-load and prioritize above it, and never lazy-load the LCP element.

Render-blocking resources round out the list. A stylesheet in the head blocks rendering until it downloads and parses, and a synchronous script does the same, so the LCP element cannot paint until they clear. Inline the critical CSS the above-the-fold content needs and defer the rest. Defer or async your non-critical JavaScript so it does not block the first paint. Preconnect to the origins of critical third-party resources, and preload the LCP image and critical fonts so the browser starts fetching them early. Self-host fonts or preconnect to the font host, and use font-display swap so text renders in a fallback immediately instead of waiting invisibly for the web font. Each of these is a small win; stacked, they routinely turn a 4-second LCP into a 2-second one.

Fixing INP and CLS, plus field data vs lab data

INP is a main-thread problem, so every fix comes down to giving the browser room to respond to the user. When someone taps, the browser needs to run your event handler, update state, and paint the next frame. If the main thread is busy running a 300-millisecond JavaScript task, the tap sits in a queue and the interaction feels dead. The fixes all reduce or reschedule main-thread work. Break up long tasks into chunks under 50 milliseconds so the browser can interleave input handling between them. Defer non-critical JavaScript so it is not competing for the thread during interactions. Move heavy, non-urgent work like analytics and telemetry into requestIdleCallback so it runs in the gaps. And stop firing a barrage of analytics events on the very first input, which is the single most common INP killer I find on marketing sites.

The framework-specific INP trap is hydration. Server-rendered React and similar frameworks ship HTML that looks interactive but is not, until JavaScript downloads and hydrates it, attaching all the event handlers. During that hydration window the page looks ready and the user taps and nothing happens, or happens after a long delay, because the main thread is saturated hydrating. The mitigations are code-splitting so you ship less JavaScript, streaming or partial hydration so interactivity comes online progressively, and simply shipping less client-side JavaScript overall. If your INP is bad and you run a heavy SPA framework, hydration cost is almost certainly the reason.

CLS is the easiest of the three to fix because every cause is a failure to reserve space. Set explicit width and height attributes on every image so the browser reserves the correct box before the image loads; on responsive images use the aspect-ratio CSS property. Reserve a fixed slot for anything injected late: ads, embeds, lazy-loaded modules, cookie banners. Never inject content above content that is already visible, because everything below it lurches down. And handle fonts deliberately: a font swap that changes text metrics reflows the paragraph, so match your fallback font's metrics or use font-display optional to avoid the shift. Do these and CLS goes to near zero and stays there.

Now the distinction that determines if any of this counts: field data versus lab data. Lab data comes from a synthetic test, Lighthouse or PageSpeed Insights running your page once, on a simulated device, in a controlled environment. It is repeatable, great for debugging, and you can run it before you ship. But it is one load, on one simulated phone, and it is not what Google ranks on. Field data is the Core Web Vitals of your actual users, collected from real Chrome browsers on real devices and networks, aggregated in the Chrome User Experience Report (CrUX). That 75th-percentile field number is what feeds the ranking signal.

This gap bites people constantly. Your Lighthouse score is 98 and your field LCP is failing, and you cannot understand why, because Lighthouse tested on fast simulated hardware while a third of your real users are on mid-tier Android phones on congested networks who experience a 5-second LCP. Lab data tells you what could happen under ideal conditions; field data tells you what is happening to real people. Use lab data (Lighthouse, WebPageTest, DevTools) to diagnose and fix, because it is fast and repeatable. Use field data (CrUX, the Search Console Core Web Vitals report, and your own real-user monitoring) to know if you actually have a problem and if your fix reached real users. Install a real-user monitoring script with the web-vitals library so you see your own field numbers continuously instead of waiting a month for CrUX to update. Fix in the lab, verify in the field.

How technical foundations now serve AI and answer engines

Everything in this article was built for a world of blue links, and it all transfers, sometimes with more force, to the world of AI answer engines. The mechanisms differ but the dependency is the same: an AI system cannot cite, quote, or act on a page it could not crawl, could not parse, or gave up waiting on. If anything, the technical bar is higher for AI surfaces, because the crawlers and models behind them are less patient and less forgiving than classic Googlebot.

Crawlability is now a two-audience problem. Alongside Googlebot and Bingbot, you have a growing fleet of AI crawlers: the ones that gather training data and, more importantly for visibility, the ones that fetch pages live to answer a question right now. Two root-level files have emerged to talk to them. llms.txt describes your content and structure for large language models, a plain-language map of what your site offers and where the important stuff lives. ai.txt expresses your opt-in or opt-out preferences for AI training crawlers, the AI-era cousin of robots.txt. Neither is a hard standard yet, but the sites thinking about AI discovery are already emitting them, and blocking the fetch-to-answer crawlers in robots.txt is a fast way to disappear from AI citations entirely.

Rendering matters even more for AI than for Google. Many AI answer engines fetch the raw HTML and do not run a full render pass the way Googlebot eventually does. If your content only exists after client-side JavaScript executes, an answer engine may see the empty shell and move on to a competitor whose content is right there in the initial response. Server-side rendering, already the safe choice for classic SEO, becomes close to mandatory for AI visibility. The content you want quoted has to be in the HTML the moment the page is fetched.

Structured data is quietly becoming the interface AI systems prefer. When an assistant needs a fact (a price, an author, an event date, a set of specifications) parsing clean JSON-LD is faster and more reliable than inferring it from prose, and models increasingly lean on that labeled data to ground their answers and reduce hallucination. Your Organization schema tells them who you are. Your Product schema hands them the exact price and availability instead of making them guess from a page that might be showing a strikethrough sale. FAQPage schema serves up ready-made question-answer pairs, which is precisely the shape an answer engine wants. The structured data you built for rich results doubles as an API for machines that read on a user's behalf.

Core Web Vitals feed AI visibility through a blunter mechanism: speed as a citation filter. AI Overviews and similar systems assembling an answer under time pressure skip slow sites when choosing sources, because a page that takes five seconds to respond is a page not worth waiting for when three faster competitors say roughly the same thing. Your fast, stable, well-structured page is not just ranking better in the ten blue links; it is the one an answer engine can crawl, parse, and cite without friction. The through-line across every AI surface is the same as it ever was: be reachable, be parseable, be fast. Technical SEO did not become less important when search got smart. It became the price of admission.

An audit workflow and the toolset

A technical audit is not a random walk through a tool's error list. It follows the crawler's own path, outside in, so you fix gate problems before you fix polish problems. I run every audit in the same five phases, and the order is the point: there is no value in optimizing schema on pages that are blocked from crawling, so you resolve crawl and index issues first and work up the stack.

Phase one is crawl and index. Pull the Search Console Coverage and Page Indexing reports and read them like a triage nurse: how many of your pages are indexed versus excluded, and why. 'Crawled, currently not indexed' and 'Discovered, currently not indexed' are the signals of thin content or crawl-budget starvation. 'Excluded by noindex' on a page you want indexed is a five-alarm fire. Then run a full crawl with Screaming Frog or a cloud crawler, and cross-reference against server logs to see what Googlebot actually does. This phase answers the only question that matters first: are my important pages reachable and stored.

Phase two is site architecture and canonicalization. From the same crawl, check click depth (nothing important should be more than three clicks from the root), hunt for orphan pages with no internal links pointing to them, and audit every redirect for chains and loops. Verify that canonicals are consistent and that they agree with the sitemap and internal links. This is where you catch the four-voices-disagreeing bug that quietly splits your ranking signal across duplicate URLs.

Phase three is structured data. Run the Rich Results Test on a representative page of each template type, then open Search Console's Enhancements reports to see validation errors at scale across the whole site. You are looking for missing required properties, malformed values, and, most importantly, template-level bugs that break schema on thousands of pages at once. A single broken selector in a product template is worth more of your attention than ten hand-crafted errors on individual pages.

Phase four is Core Web Vitals. Start with field data: the Search Console Core Web Vitals report and CrUX tell you which URL groups are actually failing for real users. Only then drop into lab tools, PageSpeed Insights, Lighthouse, and WebPageTest, to diagnose the specific failing templates and prescribe fixes. Do not skip straight to Lighthouse; a page can score 95 in the lab and fail in the field, and the field data is what decides where you spend effort. Phase five is monitoring: install real-user monitoring with the web-vitals library, set up scheduled crawls that alert on new errors, and put the whole audit on a quarterly calendar so regressions surface in weeks, not quarters.

The toolset is smaller than the marketing suggests. Google Search Console is free, first-party, and non-negotiable; it is the source of truth for how Google sees your site. Bing Webmaster Tools is its smaller but real counterpart. Screaming Frog is the workhorse crawler for site audits. PageSpeed Insights, Lighthouse, and WebPageTest cover performance lab work; CrUX and a RUM script cover the field. The Schema.org validator and Rich Results Test cover structured data. Ahrefs, Semrush, or Moz add competitive and keyword context, and a log analyzer turns raw server logs into crawl-behavior insight. You can run a professional-grade technical program with Search Console, Screaming Frog, a Lighthouse tab, and the discipline to look at your logs.

The mistakes that quietly cost you rankings

Most technical SEO damage is not dramatic. It is a small default nobody chose, shipped in a template, multiplied across thousands of pages, and left to compound. Here are the ones I find over and over, roughly in order of how much silent damage they do.

The block-to-deindex mistake tops the list because it is so counterintuitive. Someone wants a page gone, so they Disallow it in robots.txt, and it stays in the index as a bare link forever, because Google can no longer crawl it to see the noindex tag that would have removed it. The correct removal is to allow the crawl and serve a noindex directive. Related and just as common is the accidental sitewide noindex, usually a staging setting that shipped to production, which can deindex an entire site in days. Check for a stray noindex before and after every launch, mechanically.

Lazy-loading the LCP image is the performance version of the same self-inflicted wound. A blanket loading lazy applied to every image feels responsible, but on the hero it is the single worst thing you can do to LCP, because you have told the browser to delay the exact element the metric measures. Nearby on the list: shipping multi-megabyte hero images that no one compressed, and blocking JavaScript bundles in robots.txt so Google renders an empty shell. Each is a one-line default with a large blast radius.

On structured data, the recurring sins are marking up data the user cannot see (schema for a rating that appears nowhere, FAQs that exist only in the JSON-LD) which crosses into spam and earns manual actions, and stale schema after a redesign, where the template changed and the JSON-LD now points at fields that no longer exist. Both are invisible to anyone not running the Rich Results Test, which is exactly why they persist for months. If you take one habit from this article, make it validating schema on deploy.

On crawlability and architecture, the quiet killers are redirect chains that leak signal and latency at every hop, orphan pages that Google can technically crawl but never finds because nothing links to them, and faceted-navigation traps that spawn millions of parameter URLs and drown your crawl budget so your real pages get crawled once a month. None of these throw an error. They just slowly starve your important pages of attention while you wonder why new content takes forever to index.

The meta-mistake, the one underneath all the others, is treating technical SEO as a project instead of a maintenance discipline. You commission an audit, fix everything, feel good, and stop looking. Six months later a CMS upgrade injected a noindex, a marketing push shipped a page with a 6MB image, and a developer added a script that blocks the main thread, and your rankings have quietly slid. Nothing announces itself. The fix is not heroics; it is monitoring: real-user CWV tracking, scheduled crawls that alert on new errors, and the Search Console Enhancements reports checked on a cadence. Technical SEO rewards the boring habit of looking regularly far more than it rewards the occasional dramatic audit.

A worked example: rescuing a stalled ecommerce site

Let me make this concrete with a composite of jobs I have done, the kind of situation that looks hopeless from the marketing seat and turns out to be three plumbing problems in a trench coat. A mid-size ecommerce site, roughly 40,000 product pages, had watched organic traffic slide for eight months. Content was fine. Products were good. Rankings were bleeding anyway, and the team was convinced they needed to rewrite everything. They did not. They needed an audit.

Phase one, crawl and index, found the first crime immediately. Search Console showed only 12,000 of 40,000 product pages indexed, with 22,000 sitting in 'Crawled, currently not indexed' and 'Discovered, currently not indexed.' The server logs told the story: Googlebot was spending more than half its crawl budget on faceted-navigation URLs, every combination of color, size, and price filter generating a unique crawlable URL, millions of them, none worth indexing. The real product pages were being crawled once a month because the crawler was drowning in filter permutations. Fix: block the filter parameters from crawling, add canonical tags from filtered views back to the clean category page, and prune the sitemap to only canonical product URLs.

Phase two, architecture, found the second crime. A replatform two years earlier had left a forest of redirect chains: old product URLs 301-ing to a category, which 301-ed to a new URL structure, which 302-ed (temporary, wrong) to the final page. Three and four hops deep in places, with a scattering of loops that returned nothing. Signal was leaking at every hop and Googlebot was abandoning some chains before the end. Fix: flatten every redirect to a single hop pointing straight at the final canonical URL, and convert the erroneous 302s to 301s. Boring work, a spreadsheet and a config change, and it recovered ranking signal that had been evaporating for two years.

Phase three, structured data, found the third. The Product schema had survived the replatform but the template had not: the price selector in the JSON-LD pointed at an element that no longer existed, so every one of the 40,000 products was emitting Product schema with a null price and a broken offer. The star ratings had vanished from search results months ago and nobody connected it to the redesign. The Rich Results Test flagged it in one click. Fix: repair the template's field mapping, revalidate, and let Search Console's Enhancements report confirm the errors clearing across the catalog.

Phase four, Core Web Vitals, was the slow bleed underneath. Field data in Search Console showed the product template failing LCP for most mobile users at around 4.6 seconds, driven by uncompressed hero images averaging 2.8MB and a render-blocking stylesheet. Lighthouse in the lab had shown a comfortable 91, which is exactly why nobody had worried: the lab tested on fast hardware while real users on mid-tier Android phones waited nearly five seconds. Fix: convert product images to AVIF with responsive srcset, add fetchpriority high to the main product image, remove the lazy-load that had been applied to it, inline critical CSS, and defer the rest.

Before the results, the lesson worth extracting: notice that none of the three crimes announced itself. There was no error message on the homepage, no alert in anyone's inbox, no drop that lined up with a single obvious change. The faceted-navigation trap looked like normal URLs. The redirect chains returned 200s at the end, so pages loaded fine for humans. The broken Product schema was invisible without the Rich Results Test. And the CWV failure was hidden behind a green Lighthouse score. Every one of these was diagnosable in an afternoon with free tools, and every one had been silently draining the site for one to two years because nobody was looking at crawl logs, redirect maps, schema validators, or field CWV data. The eight-month slide was not one catastrophe; it was three quiet leaks the team had no instrument pointed at.

The results landed over the following quarter, in sequence, because the fixes compound. Indexation recovered first as crawl budget shifted from filter junk to real products: within six weeks, indexed pages climbed from 12,000 toward 35,000. Rich results returned a few weeks after the schema fix as Google recrawled and revalidated, and the star ratings reappeared in listings, lifting click-through. CWV moved into the green over the following month as CrUX accumulated real-user data on the faster pages. Organic traffic, which had fallen for eight straight months, turned and was up 60% year-over-year two quarters later. Not one word of content changed. Three pieces of plumbing did.

What's next for technical SEO

Technical SEO is not going away as search gets smarter; it is spreading. The three pillars, reachable, parseable, fast, were the price of admission to the ten blue links, and they are becoming the price of admission to every AI surface layering on top. The specific tactics evolve, but the direction is clear, and a few shifts are worth preparing for now.

Crawling is becoming a multi-audience discipline. For twenty years there were basically two crawlers that mattered, Googlebot and Bingbot. Now there is a growing fleet of AI crawlers with different behaviors and different patience, and the emerging root-level files (llms.txt to describe your content for models, ai.txt to signal training preferences) are early attempts to talk to them. Expect this space to standardize over the next couple of years, and expect the decision of which AI crawlers to allow to become a real strategic choice with visibility on one side and content-licensing concerns on the other.

Structured data is on a path from nice-to-have to interface. As AI systems and shopping agents increasingly act on your data rather than just displaying links to it, the difference between a page a machine can parse cleanly and one it has to guess at becomes the difference between being actionable and being ignored. I expect schema coverage to deepen, agent-oriented types like PotentialAction to matter more as agents start transacting, and validation to move fully into the deploy pipeline as teams treat structured data like the code it is. The open web's structured data is quietly becoming its API.

Performance keeps tightening because user hardware and network variance keep widening. INP replacing FID in 2024 raised the bar from 'did the first tap register' to 'is the whole experience responsive,' and the trend is toward metrics that capture the full felt experience rather than single moments. The field-versus-lab gap will keep punishing teams that optimize for the lab, and real-user monitoring will shift from a nice-to-have to standard practice, because it is the only way to see what your actual users experience across the long tail of devices.

If I had to compress the whole future into one sentence: the fundamentals do not change, but the number of systems depending on them multiplies. A page that is reachable, understandable, and fast wins in classic search, wins in AI Overviews, wins in voice, wins with shopping agents, and wins with whatever comes next, because every one of those systems has to crawl it, parse it, and not give up waiting on it. That is a genuinely optimistic place to end. You do not need to chase every new surface with a new playbook. You need to get the plumbing right, keep it right, and let every surface above it inherit the benefit. Do the boring, deterministic, measurable work, and you build a foundation that pays off across a search landscape that keeps fragmenting. The plumbing wins. It always did.