International Journal of Computer Science Issues

More than a traditional journal...

The Mercator describes, as a scalable, extensible web crawler written entirely in Java. In term of Scalable, web crawlers must be scalable and it is important component of many web services, but their design is not well-documented in the literature. In this paper, we enumerate the major components of any scalable web crawler, comment on alternatives and tradeoffs in their design, and describe the particular components used in Mercator. We also describe Mercators support for extensibility and customizability. Finally, we comment on Mercators performance, which we have found to be more efficient and comparable to that of other crawlers.