Right here is the measures that you’re going to follow on this Magento 2: How to configure robots.txt?:
  1. On the Admin sidebar, tap Stores
  2. Expand the Search Engine Robots section.
  3. When complete, tap Save Config

Hi friends,

As you know, the Magento configuration includes settings to generate and manage instructions for web crawlers and bots that index your site. The instructions are saved in a file called “robots.txt” that resides in the root of your Magento installation. The instructions are directives that are recognized and followed by most search engines. By default, the robots.txt file that is generated by Magento contains instructions for web crawler to avoid indexing certain parts of the site that contain files that are used internally by the system. You can use the default settings, or define your own custom instructions for all, or for specific search engines. In this post, we will show you how to configure robot.txt and introduce about some Custom Instructions.

To configure robots.txt, we follow this process: 

1 On the Admin sidebar, tap Stores
2 Expand the Search Engine Robots section.
3 When complete, tap Save Config

1. On the Admin sidebar, tap Stores.

Then under Settings, choose Configuration. Then, choose Design in the panel on the left under General.

Magento 2: How to configure robots.txt
2. Expand the Search Engine Robots section.

Then, do the following:
Firstly, set Default Robots to one of the following:

  • INDEX, FOLLOW Instructs web crawlers to index the site and to check back later for changes.
  • NOINDEX, FOLLOW Instructs web crawlers to avoid indexing the site, but to check back later or changes.
  • INDEX, NOFOLLOW Instructs web crawlers to index the site once, but to not check back later for changes.
  • NOINDEX, NOFOLLOW Instructs web crawlers to avoid indexing the site, and to not check back later for changes.

(If needed, enter custom instructions into the Edit Custom instruction of robots.txt file box: For example, while a site is in development, you might want to disallow access to all folders). After that, tap Reset to Default to restore the default instructions.

Magento 2: How to configure robots.txt
3. When complete, tap Save Config.

Magento 2: How to configure robots.txt

Besides, we will introduce some examples of Custom Instructions:

1 Allows Full Access
2 Disallows Access to All Folders
3 Default Instructions
  • Allows Full Access

User-agent:*
Disallow:

  • Disallows Access to All Folders

User-agent:*
Disallow: /

  • Default Instructions

Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=

Start with Installing Magento 2 Extensions Now!

Explore Magestore Extensions for Magento 2  >>>HERE<<<

magestore-extensions-magento-2


Ready to foster your knowledge

with MAGENTO 2 BASIC GUIDE

Is your site still a mess and Magento extensions are not enough to save your site? Save your site with Magento Web Development Services now

magento services


#4. How to Set Up Sitemap in Magento 2?
#6. How to Integrate Google Tools for Magento 2 Site?
Tagged:

Leave a Reply

Your email address will not be published. Required fields are marked *