The basic idea of the Robot Exclusion Standard is that each web server can set up a single file called "/robots.txt" which contains pathnames that robots should not look at.
The TaskList is a generic interface for a storage for RobotTask objects.