Site setup
Technical SEO: how Google reaches your pages.
Crawling, indexing, robots.txt, sitemaps, canonical URLs and JavaScript: how search engines reach your pages, and what to check first.
Short answer
What is technical SEO?
Technical SEO is the work that lets search engines reach, read and index your pages. It covers crawling, robots.txt, sitemaps, canonical URLs and JavaScript. A page that cannot be reached or indexed cannot be ranked.
In brief
Five things to know first.
Access comes before ranking
A search engine must crawl and index a page before it can show it. Fixing site setup removes what blocks that.
Robots.txt controls crawling
Robots.txt tells crawlers which URLs they can access. A blocked page can still be indexed if other sites link to it.
Sitemaps help discovery
A sitemap helps Google crawl your site more efficiently. Small, well-linked sites may not need one.
Canonicals pick one URL
When several URLs show the same content, redirects, rel="canonical" links and sitemaps signal which one should become canonical.
JavaScript needs care
Content that appears only after scripts run depends on rendering. Keep important content easy to reach.
Definition
Technical SEO means making pages reachable.
It is the part of SEO about your site's setup rather than its words. People also call it SEO technical work or SEO tech.
The goal is simple: search engines can find a page, read it and store it. Content and links then decide how well it ranks.
| Technical setup | Content and links | |
|---|---|---|
| Main question | Can a search engine reach and index this page? | Is this page useful and trusted? |
| Typical work | Robots.txt, sitemaps, canonicals, rendering | Writing pages, earning mentions and links |
| If it fails | The page may never show in results | The page shows but ranks low |
How it works
Crawl, render, index, then rank.
Google first fetches a URL. Its documentation says that before making the request, Googlebot checks whether you allow crawling. It also reads the HTTP status code to learn if something went wrong.
Each stage can fail on its own. A page can be crawled but not indexed, so check the stage where it stops.
Crawl
Googlebot checks robots.txt, then requests the URL and reads the status code.Render
Make sure the full content shows once the page's scripts run.Index
The page is stored so it can be shown. Copies of the same content are grouped under one canonical URL.Rank
Only indexed pages can appear in results, ordered by relevance and quality.
Robots.txt
Robots.txt controls crawling, not indexing.
A robots.txt file tells search engine crawlers which URLs they can access on your site. It lives at the root of your site, and not every search engine supports every rule.
Google warns that a page disallowed in robots.txt can still be indexed if other sites link to it. To keep a page out of results, robots.txt is the wrong tool.
Place it at the root
Crawlers look for the file at the top level of your domain.
Use it to manage crawling
Block sections that waste crawl effort, such as internal search results.
Don't use it to hide pages
A blocked URL can still be indexed without its content if others link to it.
Test it
The robots.txt report in Search Console shows whether Google can process your files.
Not sure what blocks your pages?
Our free AI audit includes a technical AEO checklist for your site, reviewed live with you.
Sitemaps
Sitemaps: when you need one.
Google says search engines read a sitemap to crawl your site more efficiently. It can also carry details about videos, images and news articles.
Small sites of about 500 pages or fewer, well linked from the home page, may not need one. New sites with few external links and sites with rich media often do.
| Site type | Sitemap helps? | Why |
|---|---|---|
| Small, well linked | Maybe not | Googlebot finds pages by following links |
| New, few external links | Yes | Googlebot may not discover pages nobody links to |
| Video, image or news content | Yes | Sitemaps carry extra details about media |
Canonical URLs
Canonical URLs: choosing the main version.
When several URLs show the same content, you can signal which one should become canonical. Google says using two or more methods increases the chance your preferred canonical URL appears in search results.
A single canonical also makes it easier to get consolidated metrics for one piece of content.
| Method | Strength, per Google | |
|---|---|---|
| Redirect | Redirect to the target URL | Strong signal that the target is canonical |
| rel="canonical" | Link annotation naming the preferred URL | Strong signal that the URL should be canonical |
| Sitemap | List the preferred URL in your sitemap | Weak signal that helps it become canonical |
JavaScript
Technical SEO tips for JavaScript sites.
JavaScript sites add a step: the page must be rendered before its content is seen. Google says Googlebot caches aggressively to reduce network requests and resource usage.
Keep the signals that matter clear, such as the canonical URL, so rendering cannot muddy them.
Use real status codes
Googlebot uses HTTP status codes to learn if crawling went wrong.
Self-reference your canonical
Include a rel="canonical" link on the canonical page itself.
Skip URL fragments
Google generally doesn't support fragments as canonical URLs.
Lazy-load images
Load images only when the user is about to see them, to help performance.
Want the technical base fixed?
We fix crawler access, schema, rendering and sitemaps, directly or as a specification for your developers.
How to do it
How to do technical SEO, step by step.
Work top to bottom. Fix what blocks access first, then tidy duplicates, then check rendering. Compare each result with a baseline taken before you start.
Check crawl access
Open the robots.txt report in Search Console and confirm Google can process your file.Confirm key pages are indexed
Inspect your most important URLs and note any page Google has not indexed.Add or fix a sitemap
List your canonical URLs, and add media details if you publish video, images or news.Set one canonical per page
Use a self-referencing canonical and link to that URL consistently across the site.Review JavaScript output
Make sure the content and signals you rely on show once the page renders.Track the result
Record a baseline first, then recheck indexed pages, clicks and positions in Search Console on a schedule.
Why it matters
Why it matters now: AI answers too.
Assistants return one answer with a few sources, not ten links. For your business, a page that cannot be reached or indexed cannot be ranked or cited.
The technical layer is where many citations are lost before content or authority even come into play. Google says its AI features have no additional technical requirements.
Crawler access
Review robots.txt for the AI crawlers that matter in your category.
Raw HTML content
Check that key facts are in the HTML response, not only added by scripts.
Consistent entity signals
Describe your company the same way across your site and main profiles.
FAQ
Common questions, answered plainly.
Where does site setup fit in digital marketing?
Site setup is part of digital marketing because it decides whether a page can appear in search at all. Marketing teams plan content and campaigns, while this work makes sure search engines can reach and index the pages those efforts create.
Does robots.txt keep a page out of Google?
Robots.txt manages crawling, not indexing. A page disallowed there can still be indexed if other sites link to it, just without its content. Treat it as a crawl control, not a way to hide pages.
Does every site need a sitemap?
Not every site needs a sitemap. Google says small sites of about 500 pages or fewer, where Googlebot can find every important page by following links from the home page, may not. New sites with few external links, or sites with video, images or news, may benefit.
How do I improve my site's technical setup?
Start with access. Check the robots.txt report, confirm key pages are indexed, add a sitemap, set one canonical per page and review JavaScript output. Fix what blocks pages before polishing anything else.
Can technical SEO alone rank a website?
Technical work alone cannot rank a website. It makes pages reachable and indexable, but content quality and trust decide how well they rank. Think of it as the entry ticket, not the whole contest.
What should a beginner learn first?
Learn crawling and indexing first, since everything else depends on them. Then study robots.txt, sitemaps, canonical URLs and JavaScript rendering. Start with the official Search documentation for each topic.
How do I know if my fixes worked?
Check the robots.txt report, the number of indexed key pages, and clicks and positions in Search Console against a baseline. LLMReach monitors crawl access, schema health and citations against a baseline taken before work begins.
Free AI Audit
Find what blocks your pages in Google and AI answers.
You leave knowing where the gap is, what is causing it, and which changes would matter first, whether we work together or not.
What to expect
Audit
Before the call, we run your buyers' questions through ChatGPT, Claude, Perplexity and Gemini.
Map
On the call, we show you where competitors are cited and you are not, and what's causing it.
Decide
An honest read on whether the Citation Stack fits. If it doesn't, you'll hear it on the call.
Don't see a time that works? Email Karim directly: contact@llmreach.ai