A Novel Malicious Web Crawler Detector: Performance and Evaluation
Internet demands a robust and resilient protected communication and computing environment to enable information flows flawlessly with no down time. However, the Internet is exposed to the general public which will lead to loss of sensitive information as well as copyright protected content. To address this issue, in this paper, we proposed two schemes to fight against unwanted automatic web crawlers, TSSNBS (Too Simple Sometimes Naive Blocking Schema) and ABS (Adaptive Blocking Schema). We validated the effectiveness of the two schemes by implementing an advanced integrated crawler detection system and applied on a high trafficked site in the real world, exposed with real attackers.
Keywords: Crawler, Internet, Network, Algorithm.
Download Full-Text
ABOUT THE AUTHORS
Dexiang Zhang
Dexiang Zhang woks in the Information and Network Center of Qingdao University since 2000.He graduated from Qingdao University with bachelor¡¯s degree at 2000 and with a master¡¯s degree at 2006.His current research interest includes internet security and information security. He published six papers and involved in the preparation of one book.
Difan Zhang
Difan Zhang is a researcher on Internet Security at Information and Network Center, Qingdao University. Graduated from Towson University with a Master of Science in Information Technology, his current research interest includes Distributed Intrustion Detection Systems and Mobile Device Security.
Xun Liu
Xun Liu is a researcher on Internet Security at library of Qingdao University. He graduated from Qingdao University with a master of MPM.
Dexiang Zhang
Dexiang Zhang woks in the Information and Network Center of Qingdao University since 2000.He graduated from Qingdao University with bachelor¡¯s degree at 2000 and with a master¡¯s degree at 2006.His current research interest includes internet security and information security. He published six papers and involved in the preparation of one book.
Difan Zhang
Difan Zhang is a researcher on Internet Security at Information and Network Center, Qingdao University. Graduated from Towson University with a Master of Science in Information Technology, his current research interest includes Distributed Intrustion Detection Systems and Mobile Device Security.
Xun Liu
Xun Liu is a researcher on Internet Security at library of Qingdao University. He graduated from Qingdao University with a master of MPM.
