Decides whether a URL was already accepted or ignored.
For this decision we take advantage of a specialty of the processing of file
URLs: Since directory are searched by tree traversing, we can be sure that
we never find the same file twice in one crawler run.