As with anything in life, you need rules to control the main issues. For your website, you must create a robots.txt file with certain rules for search engines to follow. These robots tell the
To create a robots.txt file for your website, follow these steps:
Usually, each installed WordPress comes with a default WordPress robots.txt. This default robots.txt file is found to prevent search engines from crawling into the admin folder.
Yeah, it looks stupid, but this robots.txt file is as simple as this.
To set your own rules, we will learn how to create a robots.text file.
You can simply do it by following these steps:
As we all know, the role of creating a robot.txt file is to set your own rules. The possible rules are:
Type the words “Disallow: /your-path”.
Write the path which you want to block instead of “your-path”.
With this line, you will be blocking this path and every other path comes through it.
· Block a file type:
Type the words “Disallow: /*.filetype$.”
Write the name of the filetype which you want to block instead of “filetype”.
If you write “Disallow: /*jpg$, you will prevent the crawlers from accessing all jpg files on your website.
Additionally, if you want to prevent the crawlers from accessing a certain file, write its name instead of “*”, for example: “Imagename.jpg”.
· Allow specific sub-paths to be accessed by crawlers
Type the words “Allow: /yourpath/sub-path”..
Write the path which you want to allow the crawlers to access instead of “yourpath/sub-path”.
Let’s say you wanna block the access to pictures but allow it for pictures/public, then you should put 2 lines like this:
· Allow specific file types to be accessed by crawlers
Type the words “allow: /*.filetype$.”
Write the name of the filetype which you want to allow the crawlers to access instead of “filetype”.
· Block specific crawlers from crawling your website
To block certain crawlers, you just need to add 2 new lines under the existing lines in your robots.txt file.
Look at the image below:
After creating or editing any robots.txt file, you must save it.
You can add the robots.txt file using several ways, including:
· Through FTP or SFTP
If you are using any platform, you just need to upload robots.txt file to your website through FTP or SFTP.
Usually, this requires some developing knowledge and access to your server.
Therefore, you can ask your developer to help you with it to avoid any stability problems.
· Creating and adding robots.txt file using Yoast Plugin
If you are using WordPress, you can do it easily with the help of the Yoast plugin. Follow these steps:
Notice that Yoast SEO doesn’t consider the default robots.txt file. Therefore, if you don’t have a robots.txt file, you will see a message saying that you don’t have one as in the image below: