# Sitemap, robots.txt and canonical: different jobs A sitemap lists preferred discoverable URLs, robots.txt guides crawler access and canonical identifies a preferred version among similar pages. They do not replace one another. A tracked link can point to the same page while its canonical identifies the clean URL without the campaign query. Blocking a URL in robots.txt is not a dependable method for removing all knowledge of it from search results. ## Assign one practical question to each mechanism A sitemap answers, “Which public addresses does this site want to list?” A robots file gives crawler access instructions. A canonical reference expresses a preferred version among duplicate or closely related addresses. These jobs overlap in a launch checklist but are not interchangeable. A page does not become private because it is absent from a sitemap, and a canonical reference is not a password or redirect. Consider a service page with both a clean address and a campaign-tagged address. Visitors may reach either, but you generally want a consistent preferred identity for the underlying page. Google's [canonical guidance](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) describes canonical signals and their limits. Treat the declaration as a signal, not an instruction that forces every search engine to behave identically. Now consider the admin area. Its privacy must come from authentication and authorisation. Adding it to a robots exclusion may reduce ordinary crawling, but it does not stop a person who knows the address. This simple comparison helps a business owner avoid buying a “robots security setup” that does not actually protect private records. ## Work through a concrete small-site example Suppose the public site contains a homepage, three service pages, a contact page and ten articles. List their preferred addresses in the sitemap. Keep internal navigation pointed at those same addresses. For a removed service with a clear replacement, decide whether a redirect is appropriate and document the destination. Do not automatically send every removed page to the homepage; that can be confusing for someone following an old link. If a thank-you page should stay out of search, it needs an appropriate indexing instruction that can be read. Blocking all fetching while expecting the crawler to read a page-level instruction creates a conflict. The developer should explain the chosen approach and verify the published response rather than collecting unrelated SEO tags. For a multilingual article, each complete language version has a purpose of its own. It should normally identify its own address as canonical and list the true language equivalents separately. Declaring every translation a duplicate of the English page contradicts the effort spent building independently discoverable language versions. ## Review consistency whenever routes change A renamed page affects more than the menu. Update its preferred address, sitemap entry, language references and internal links together. If Markdown or other public alternatives exist, update their links too. A central route registry reduces the chance of one copy retaining the old address, but the result still deserves verification. Use a short spot check: open the old address, open the new one, inspect the canonical reference and find the new address in the sitemap. Then try a genuinely nonexistent address and check the not-found behaviour. This distinguishes intended redirects from accidental catch-all routing. Finally, remember the visitor's perspective. If the technical declarations describe a clean structure while the actual pages repeat the same vague content, the site still needs editorial work. These mechanisms help systems understand which pages exist and how they relate. They cannot create a useful service explanation or make private data safe on their own. **Check one route from four directions.** A visitor follows an internal link, a crawler reads the sitemap, a page declares its preferred address and a language switcher points to its counterpart. Those four routes should describe one coherent page model. A mismatch is a maintenance issue worth fixing even before there is any evidence of a search impact. For example, renaming a service may leave the menu current while the sitemap still lists an old URL and the translated page points elsewhere. Correcting one visible link does not finish the change. Use the route inventory to find all references and then test the resulting addresses. Keep private access decisions outside this mechanism. An administrator's page needs actual protection even if it is omitted from every public list. Conversely, a public page meant to be understood by search systems should not be accidentally hidden behind a challenge. Naming the separate purpose of each control makes configuration reviews shorter and reduces the temptation to treat every SEO file as a general security setting. ## Practical checklist - Use absolute sitemap URLs - Review crawler rules - Set self-consistent canonicals **A concrete example:** A tracked link can point to the same page while its canonical identifies the clean URL without the campaign query. **A boundary to keep clear:** Blocking a URL in robots.txt is not a dependable method for removing all knowledge of it from search results. ## Your next step [Tell Orvunweb about your project](/en/start-a-project/) ## Related reading - [A technical SEO checklist for a new business site](/en/blog/a-technical-seo-checklist-for-a-new-business-site/) - [How Google discovers a new website](/en/blog/how-google-discovers-a-new-website/) - [Why hreflang matters on multilingual websites](/en/blog/why-hreflang-matters-on-multilingual-websites/) - [Business website service](/en/services/business-website/) - [Compare packages](/en/pricing/) ## Related services - [Business website](/en/services/business-website/) --- Locale: en Canonical: https://orvunweb.com/en/blog/sitemap-robots-txt-and-canonical-different-jobs/ Updated: 2026-09-16 - [Google Search Central — Canonical URLs](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls)