Thursday 22nd of February 2018

A Novel Malicious Web Crawler Detector: Performance and Evaluation

Dexiang Zhang, Difan Zhang and Xun Liu

Internet demands a robust and resilient protected communication and computing environment to enable information flows flawlessly with no down time. However, the Internet is exposed to the general public which will lead to loss of sensitive information as well as copyright protected content. To address this issue, in this paper, we proposed two schemes to fight against unwanted automatic web crawlers, TSSNBS (Too Simple Sometimes Naive Blocking Schema) and ABS (Adaptive Blocking Schema). We validated the effectiveness of the two schemes by implementing an advanced integrated crawler detection system and applied on a high trafficked site in the real world, exposed with real attackers.

Keywords: Crawler, Internet, Network, Algorithm.

Download Full-Text


Dexiang Zhang
Dexiang Zhang woks in the Information and Network Center of Qingdao University since 2000.He graduated from Qingdao University with bachelor¡¯s degree at 2000 and with a master¡¯s degree at 2006.His current research interest includes internet security and information security. He published six papers and involved in the preparation of one book.

Difan Zhang
Difan Zhang is a researcher on Internet Security at Information and Network Center, Qingdao University. Graduated from Towson University with a Master of Science in Information Technology, his current research interest includes Distributed Intrustion Detection Systems and Mobile Device Security.

Xun Liu
Xun Liu is a researcher on Internet Security at library of Qingdao University. He graduated from Qingdao University with a master of MPM.

IJCSI Published Papers Indexed By:





IJCSI is a refereed open access international journal for scientific papers dealing in all areas of computer science research...

Learn more »
Join Us

Read the most frequently asked questions about IJCSI.

Frequently Asked Questions (FAQs) »
Get in touch

Phone: +230 911 5482

More contact details »