@LearnTheNewFeb 06.2019 — #Your page will not crawl by the robot and will not indexed
follow the content which Google provided in Google support pages
Test your robots.txt file
Open the tester tool for your site, and scroll through the robots.txt code to locate the highlighted syntax warnings and logic errors. The number of syntax warnings and logic errors is shown immediately below the editor.
Type in the URL of a page on your site in the text box at the bottom of the page.
Select the user-agent you want to simulate in the dropdown list to the right of the text box.
Click the TEST button to test access.
Check to see if TEST button now reads ACCEPTED or BLOCKED to find out if the URL you entered is blocked from Google web crawlers.
Edit the file on the page and retest as necessary. Note that changes made in the page are not saved to your site! See the next step.
Copy your changes to your robots.txt file on your site. This tool does not make changes to the actual file on your site, it only tests against the copy hosted in the tool.
Limitations of the robots.txt Tester tool:
Changes you make in the tool editor are not automatically saved to your web server. You need to copy and paste the content from the editor into the robots.txt file stored on your server.
The robots.txt Tester tool only tests your robots.txt with Google user-agents or web crawlers, like Googlebot. We cannot predict how other web crawlers interpret your robots.txt file.
@dp362pradhanFeb 07.2019 — #It depends on which url & how you inserted in robot.txt. If there is any error then your all pages may be either blocked or crawled.
@swapna8Feb 07.2019 — #Each and every website must have a robot.txt file. A robot.txt is a file, in which it will be specified with what the things to be allowed to crawl by a search engine, what is not!
@rootFeb 07.2019 — #Doesn't anyone understand that the whole robots.txt is completely voluntary.
IF as you read my post, will find, the only way that you can prevent a directory being crawled is to... CHANGE that folder PERMISSIONS.
issuing a chmod( "thepath/to/folder", 0644 ) should turn anything in that directory invisible to the outside world, scripts and the server can see the folder.
SO...
IF, you want to protect a folder, THAT is the only way you can do it.