WordPress already serves a basic robots.txt and an XML sitemap for you, so you may not need to create anything. To change them, edit robots.txt through your SEO plugin or upload a physical file to your site root, then submit your sitemap in Google Search Console and Bing Webmaster Tools. This guide shows each step and the mistakes that cost sites traffic.
What robots.txt and a sitemap do
A robots.txt file is a plain text file at the root of your domain. Crawlers such as Googlebot and Bingbot read it before they crawl, and it tells them which paths they may request and which to skip. It is a request, not a lock. Well-behaved crawlers follow it, but it does not hide anything from people and it does not keep a page out of search results.
An XML sitemap does the opposite job. It is a list of the URLs you want search engines to find, with optional details such as the last modified date. Crawlers can find most pages by following links, but a sitemap helps them discover new or deep pages faster, and it gives you a clean list to check in Search Console.
| File | Job | Default WordPress address |
|---|---|---|
| robots.txt | Tells crawlers what to skip | /robots.txt |
| XML sitemap | Lists the URLs you want found | /wp-sitemap.xml |
| HTML sitemap | A page of links for human visitors | Only if you build one |
What WordPress gives you by default
If there is no physical robots.txt file in your site folder, WordPress generates a virtual one when a crawler asks for it. Open yourdomain.com/robots.txt in a browser to see it. The default output blocks the admin area, allows the one admin file that front-end features need, and points to the core sitemap.
Core WordPress has also shipped an XML sitemap since version 5.5. The index lives at /wp-sitemap.xml and links to separate sitemaps for posts, pages, categories, tags and users. You do not install anything to get it.
Two things can change this. First, if you tick the option to discourage search engines from indexing your site in the Reading settings, WordPress changes the virtual robots.txt to disallow everything and adds a noindex signal. Many launches have stalled because someone forgot to untick it after a staging period. Second, most SEO plugins replace the core sitemap with their own and move it to a different address.
How to create or edit robots.txt in WordPress
Option 1: use your SEO plugin
The easiest route is the file editor inside your SEO plugin. Yoast SEO, Rank Math and All in One SEO each include a robots.txt editor in their tools or settings screens. You type your rules, save, and the plugin serves them at /robots.txt. This is the safest way because the plugin handles file permissions and keeps working if you move hosts.
Menu names change between plugin versions, so look for a robots.txt or file editor entry in the plugin’s tools or general settings rather than relying on a path you read in an old tutorial. Our comparison of SEO plugins for WordPress covers how the three differ.
Option 2: upload a physical file
If you prefer full control, create a text file named robots.txt in any plain text editor, such as Notepad or TextEdit in plain text mode. Upload it to the folder that holds wp-config.php and the wp-content folder. Your host’s file manager or an SFTP client both work. Once a physical file exists, WordPress stops generating the virtual one, and your plugin’s editor may stop working, so pick one method and stay with it. If you hit conflicts, our guide to overwriting robots.txt manually in WordPress walks through them.
An example robots.txt
This is example code for a typical content site. Adjust the paths to match your own site before you use it.
# Example robots.txt for a WordPress site
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://example.com/wp-sitemap.xmlHere is what each line does:
- User-agent: * means the rules below apply to every crawler. You can add a separate block for one crawler by naming it.
- Disallow: /wp-admin/ asks crawlers to skip the admin area.
- Allow: /wp-admin/admin-ajax.php keeps one file reachable because themes and plugins use it for front-end requests.
- Disallow: /?s= and /search/ stop crawlers from wandering through internal search result pages, which create endless thin URLs.
- Sitemap: gives the full address of your sitemap, including https. Use the address your sitemap really has.
Replace example.com and the sitemap address with your own. If your SEO plugin makes a different sitemap address, use that one.
Rules for specific crawlers
You can address one crawler by name. For example, a block that starts with the GPTBot user agent lets you set different rules for OpenAI’s crawler than for search engines. Decide this based on your own policy for AI use of your content, because blocking a crawler also removes you from whatever that crawler feeds. Google documents its own crawlers on its Search Central pages, and each AI company publishes its user agent name.
Crawl-delay is a common request. Bing reads it, but Google ignores it. If a crawler is overloading your server, fix that at the hosting or caching level too. A faster site handles crawl traffic better, and our guide to speeding up a WordPress website is a good place to start.
How to create a sitemap in WordPress
Use the core sitemap
If you run no SEO plugin, the core sitemap is already live. Visit yourdomain.com/wp-sitemap.xml and you should see an index of sitemap files. Core lets developers exclude post types or individual posts with filters, but it has no settings screen, so you cannot tweak it without code.
Use your SEO plugin’s sitemap
Most site owners prefer a plugin sitemap because it offers switches for post types, taxonomies and individual pages. Typical addresses are:
- Yoast SEO: /sitemap_index.xml
- Rank Math: /sitemap_index.xml
- All in One SEO: /sitemap.xml
When one of these plugins turns on its sitemap, it takes over from the core one. Open the address in your browser after you switch plugins, and use whichever address the plugin shows you. Do not submit two sitemaps that list the same URLs.
If your site is not on WordPress
Site crawlers and online sitemap generators can scan a site and produce an XML file that you upload to the root folder. Keep in mind that a generated file goes stale as soon as you publish something new, so a plugin that updates itself is a better fit for a site that changes often.
Submit your sitemap to Google and Bing
Google Search Console
Add your site as a property in Google Search Console and verify ownership. Open the Sitemaps report, enter the sitemap path (for example sitemap_index.xml) and submit. Search Console shows whether Google could read the file and how many URLs it found. If you are new to the tool, our guide to Google Search Console covers setup and the main reports.
Search Console also has a robots.txt report that shows the version of the file Google last fetched and any parse problems. It replaced the older standalone tester, so use that report to confirm your changes were picked up.
Bing Webmaster Tools
Sign in to Bing Webmaster Tools. You can import a site you already verified in Google Search Console, or add and verify it directly. Then submit your sitemap address in the sitemaps section. It takes a few minutes and gives you a second set of crawl reports.
The sitemap line in robots.txt
The Sitemap line in robots.txt lets any crawler find your sitemap without you submitting it by hand. Keep it even if you also submit in the webmaster tools. If your plugin manages robots.txt it usually adds the line for you.
Best practices for both files
- List only canonical, indexable URLs in your sitemap. Leave out redirected, noindexed or broken addresses.
- Use robots.txt to manage crawling, and use a noindex tag to keep a page out of results. A blocked page can still appear in results as a bare URL if other sites link to it, and crawlers cannot see a noindex tag on a page they are not allowed to fetch.
- Do not block the CSS and JavaScript files that a page needs to render. Google reads them to understand your layout.
- Keep sitemap files within the limits in the sitemaps protocol: 50,000 URLs or 50 MB uncompressed per file. Plugins split large sites into several files for you.
- Check both files after every major change, such as a theme switch, a migration or a move from staging to live.
- Keep your pages fast too. Crawl budget matters more on large sites, and a good cache plugin reduces the load that crawlers add.
Common mistakes to avoid
- Disallow: / left in place. This blocks the whole site. It often arrives from a staging copy, so check it after every launch.
- The Reading setting still ticked. Discouraging search engines overrides everything else you do here.
- Two sitemaps competing. A plugin sitemap and a second plugin or a leftover static file confuse reporting.
- Blocking whole folders by habit. Old guides block /wp-content/ or /wp-includes/. That can stop crawlers loading images and scripts.
- Using robots.txt for privacy. The file is public, so anyone can read the paths you list. Protect private content with a login or server rules.
- Never checking the sitemap report. Errors there are an early warning for indexing problems.
A quick checklist
- Open /robots.txt and confirm it does not contain a blanket Disallow for the site.
- Open your sitemap address and confirm it loads and lists the right content.
- Confirm the Reading setting for search engine visibility is unticked on your live site.
- Add the Sitemap line to robots.txt.
- Submit the sitemap in Google Search Console and Bing Webmaster Tools.
- Recheck the Sitemaps and robots.txt reports after a week.
If you want to go further with indexing, our guide to the best WordPress SEO plugins compares tools that also handle schema, redirects and meta settings.
Frequently asked questions
Does WordPress create a robots.txt file automatically?
Yes. If no physical robots.txt exists, WordPress serves a virtual one at /robots.txt. It blocks the admin area, allows admin-ajax.php and lists the core sitemap. Uploading your own file or using an SEO plugin editor overrides it.
Where is the sitemap in WordPress?
Core WordPress serves it at /wp-sitemap.xml. If you use an SEO plugin its own sitemap replaces the core one, usually at /sitemap_index.xml or /sitemap.xml. Open the address in a browser to confirm it loads.
Do I need both robots.txt and a sitemap?
No file is mandatory, but both help. Robots.txt controls what crawlers request, and the sitemap lists what you want found. Together they make crawling cleaner, and most sites should have both.
Can robots.txt remove a page from Google?
No. It only stops crawling. To keep a page out of results, add a noindex tag and let crawlers fetch the page so they can see it, or remove it and return a proper status code.
How do I check that my robots.txt works?
Open /robots.txt in your browser and read it, then check the robots.txt report in Google Search Console. It shows the version Google fetched and any errors, and you can use the URL Inspection tool to see whether a given page is blocked.
How often should I update my sitemap?
You should not need to do it by hand. Core and plugin sitemaps update when you publish or change content. Static files from a generator go stale, so regenerate them or switch to a plugin.
Should I submit my sitemap to Bing?
Yes. Bing Webmaster Tools accepts the same sitemap address, and you can import your site from Google Search Console to save time. Submitting is free.
Can I block AI crawlers in robots.txt?
You can add a block for each crawler’s user agent name and disallow your site. Well-behaved crawlers follow it, but it is not enforced, and the right choice depends on how you want your content used.
Where to go next
Set up the files, submit the sitemap and then watch the reports for a few weeks. A fast site and clean content do the rest, so read our guide to speeding up WordPress next.