Screen Scraping Security

By John Wagley

One technical measure that can protect against many scrapers is the robot.txt command, a text file that can give instructions to Web robots, said Furtsch. It has a serious limitation, however. In most cases, screen scrapers must choose to find the file and read its instructions in order for the text file to be effective; malicious bots likely won’t seek the file.

Another measure sites should take is to provide their users with mechanisms for deleting their sensitive data whenever they choose.

Another widely used tool to protect against scrapers are known as captchas. They show squiggly letters and numbers that a computer or bot cannot decipher. Sites have people type the captcha during registration to prove that they’re human.

Captchas should also be regularly updated, Furtsch said, as some scraping tools have been known to outsmart certain types of captchas.


When deciding how to

When deciding how to implement your social media marketing strategies, it is important to take into consideration the nature of your products and services. For example, if purchasing your products is something that most of your customers would prefer to keep private, then do not put Facebook-like buttons right next to the buy buttons! Eventually, someone will click it accidentally and then get angry at your business.

Great Article but needs more security solutions


Great article! The one thing I believe missing is more effective ways to protect a website from screen scraping. Robots.txt unfortnantly doesn't work. Luckily, there are several companies out there that offer or specialize in Scraping protection.  Services such as,, or can put a barrier to entry to prevent scrapers from taking your content. 


Great article ! Many bots

Great article !

Many bots are very sophisticated, changing their IP and its hard to track them

SiteBlackBox  can help you to stop such abuse and keeps those bots away for good


The Magazine — Past Issues


Beyond Print

SM Online

See all the latest links and resources that supplement the current issue of Security Management magazine.