Prevent GoogleBot Crawl Errors on Glype Proxy
Posted by Vectro
0 comments
0 comments
Sometimes Googlebot tries to index Glype’s browse.php file and the proxied pages within it. These pages do not need to be indexed in order for Google to find and list your proxy site. In some cases, Google encounters an error, tries again, then repeats this and becomes stuck in a loop. This can slow down your site and cause strain on the server. The way to prevent this is by placing the following code in your robots.txt file:
User-agent: *
Disallow: /browse.php
If you have renamed your browse.php file, make sure to place the new name instead. Click here for more information about renaming browse.php.
Filed in Proxy Webmasters
0 comments
Previous Post
« Limit the Maximum Size of File Downloads in PHProxy Next Post
How to Install and Configure Suffix Proxy Script »
« Limit the Maximum Size of File Downloads in PHProxy Next Post
How to Install and Configure Suffix Proxy Script »












