A robots.txt file is an essential file that must be included on your WordPress site. Here is an instruction that how to create, add, and link with sitemap easily.
What is the Robots.txt file?
“Robots.txt file is a file that includes in WordPress sites. This allows all bots for reading data of site”. For example, the robots txt file allows Google bots to read all site pages without an admin page.
Why we need to Create & Link with Sitemap?
For easy how to crawl pages on their site by Google bots for indexing so this file must be adding and link with the sitemap.
This can control crawler access for select pages of your site. It can be very harmful if you unintentionally disallow “Google bots” on crawling your whole site. Further, this can help in the following things:
- Avoiding duplicate content from looking in SERPs
- Protection of whole sections of a site private
- Protection internal search results pages from viewing upon a public SERP
- Postulating the location of sitemap(s)
- Stopping search engines from indexing certain files on site
How I can check Robots.txt file is Added or Not?
if you sure that your site has an RT then ok otherwise you can check in the following way: Type a domain such as blogili.com then add /robots.txt.
https://blogili.com/robots.txt
How to Create, add & link with Sitemap
Once you have found that your site has no RT file then follow the following steps. This way you can create, add & link with sitemap easily on the WordPress site:
- Open Dashboard
- Go into SEO then tools
- Click on File editor
- Generate Robots.txt
- Once you create then paste the code that is mentioned below
- Write “sitemap: your sitemap address”
- Then click on save changes
Now, this is successfully added & link with the sitemap.
Sample
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
sitemap: https://blogili.com/sitemap.xml[/su_note]
