robotExclusion suite

This is the robotExclusion suite.

Download: suites.robotExclusion.sit.hqx

Created by John VanDyk of Iowa State University on Sunday, Feb 23, 1997 at 8:31:25 PM.

Version 1.0.2, released March 2, 1997

Added menu items to view or clear user.robotExclusion.errorlog.
Commented out debugging scratchpad.x code.

Version 1.0.1, released Feb. 28, 1997

Now supports meta tags within pages. For example, adding the tag

<META name="robots" content="noindex, nofollow">

inside the <HEAD> element of a page should prevent a robot from indexing that page or following links on that page.

See the example at sample.politegetpage.

Description

This suite handles the process of verifying whether or not a robot/crawler/spider is allowed to retrieve a file from a web site. It is primarily directed at those writing Frontier web crawlers and robots.

Caching of the robots.txt files from the hosts you visit is handled automatically and the refresh interval is user-configurable (user.robotExclusion.robotcacheexpires). Default is to retrieve a robots.txt file only once every 24 hours.

The robotExclusion suite follows the current (Dec 4 1996) internet draft of the robots exclusion protocol, with exceptions outlined below.

The main use of this suite will be to call it to check if you may get a page before you actually retrieve it. For example:

if robotExclusion.robotsOK("http://www.ent.iastate.edu/List/complete.html", "MyCrawler/1.0", "jvandyk@iastate.edu")
    page = robotExclusion.httpgetpage("www.ent.iastate.edu", "/List/complete.html", 80, "MyCrawler/1.0", "jvandyk@iastate.edu")

This has been combined into one step at robotExclusion.samples.politegetpage.

It is very important that you send the name and version of your robot along with your request. Using robotExclusion.httpgetpage instead of netevents.examples.httpget does this. Writing a robot that does NOT send its identity, version, and contact information is downright rude.

Advanced options:

There are several user-configurable options at user.robotExclusion:

keeperrorlog: there are times when you want to have your script tell you that it couldn't connect, and there are times when you want the script to just log the errors. Setting keeperrorlog to true will keep an error log at user.robotExclusion.errorlog; setting it to false will pop up an error message when one occurs. Useful for debugging.
robotcacheenabled: default is true. Setting to false will retrieve the /robots.txt file every time a call to robotsOK is made. Only disable the cache if you are testing your own site!
robotcacheexpires: the number of seconds allowed before the cached /robots.txt file expires. Defaults to 86400 (24 hours).

Requirements:

This suite requires the NetEvents app.

The following improvements can be made to this suite:

support for "Allow:" lines in the new robots.txt specification
support for encoded characters (/%7Eyippy/file.html = /~yippy/file.html)
support for redirection (though this is not commonly done with /robots.txt),
support for HTTP cache control mechanisms

If you are interested in tackling one of these areas, please contact me at