org.archive.crawler.settings

The settings framework is designed to be a flexible way to configure a crawl with special treatment for subparts of the web without adding to much performance overhead.

At it's core the settings framework is a way to keep persistent, context sensitive configuration settings for any class in the crawler.

All classes in the crawler that has configurable settings subclasses {@link org.archive.crawler.settings.ComplexType} or one of its descendants. The {@link org.archive.crawler.settings.ComplexType} implements the {@link javax.management.DynamicMBean} interface. This gives you a way to ask the object for what attributes it supports and standard methods for getting and setting these attributes.

The entry point into the settings framework is the {@link org.archive.crawler.settings.SettingsHandler}. This class is responsible for loading and saving from persistent storage and for interconnecting the different parts of the framework.

Figure 1. Schematic view of the Settings Framework

Settings hierarchy

When the settings framework is asked for an attribute for a specific host, it will first try to see if this attribute is set for this particular host. If it is, the value will be returned. If not, it will go up one level recursively until it eventually reach the order object and returns the global value. If no value is set here either (normally it would be), a hard coded default value is returned.

All per domain/host settings objects only contain those settings which are to be overridden for that particular domain/host. The convention is to name the top level object "global settings" and the objects beneath "per settings" or "overrides" (although the refinements described next, also do overriding).

To further complicate the picture, there is also settings objects called refinements. An object of this type belongs to a global or per settings object and overrides the settings in it's owners object if some criteria is met. These criteria could be that the URI in question conforms to a regular expression or that it the settings are consulted at a specific time of day limited by a time span.

ComplexType hierarchy

When an attribute is accessed from the URI (either reading or writing) you want to make sure that you are editing the attribute in the right context. When trying to override an attribute, you don't want the settings framework to traverse up to effective value for the attribute, but instead want to know that the attribute is not set on this level. To achieve this, there is {@link org.archive.crawler.settings.ComplexType#getLocalAttribute(CrawlerSettings settings, String name)} and {@link org.archive.crawler.settings.ComplexType#setAttribute(CrawlerSettings settings, Attribute attribute)} methods taking a settings object as a parameter. These methods works only on the supplied settings object. In addition the methods {@link org.archive.crawler.settings.ComplexType#getAttribute(String)} and {@link org.archive.crawler.settings.ComplexType#setAttribute(Attribute attribute)} is there for conformance to the Java JMX specification. The latter two always works on the global settings object.

Getting an attribute within a crawl is different in that you always want to get a value even if it is not set in it's context. That means that the settings framework should work its way up the settings hierarchy to find the value in effect for the context. The method {@link org.archive.crawler.settings.ComplexType#getAttribute(String name, CrawlURI uri)} should be used to make sure that the right context is used. Figure 2 shows how the settings framework finds the effective value given a context.

Figure 2. Flow of getting an attribute

The different attributes has a type. The allowed type all subclasses the {@link org.archive.crawler.settings.Type} class. There are tree main Types:

1. {@link org.archive.crawler.settings.SimpleType}
2. {@link org.archive.crawler.settings.ListType}
3. {@link org.archive.crawler.settings.ComplexType}

SimpleType

ListType

ComplexType

Constraints will be checked against attribute values.

• Added elements is placed after the elements in the map it overrides.
• You can not remove elements from the map it overrides.