Screen Scraping Security

By John Wagley

Increasingly, organizations are using automated tools to scan and collect information online. They’re looking at sites such as social networks and blogs for reasons such as reputation management, public relations, market research, and background checks.

Tools that can automatically scroll for data known as screen scrapers are also becoming more advanced, but companies that use them must avoid legal pitfalls, which could include personal privacy violations as well as copyright infringement.

Social networking and other sites that collect user-generated data should also take steps to protect data on their sites, including establishing appropriate privacy policies and implementing the appropriate technical security measures.

The laws surrounding screen scraping and possible privacy and intellectual property violations are somewhat murky, said Brian Bowman, a partner at the law firm Pitblado. Bowman spoke at the Global Privacy Summit in Washington, D.C., sponsored by the Independent Association of Privacy Professionals.

In the United States, one interpretation of the law is that protected information doesn’t include information in a forum where a user voluntarily shared it, where it’s publicly available, and where users have not been led to believe that there are any technical controls limiting public access, he said. But it is fairly clear that it isn’t acceptable to collect information provided by children or from sites that are aimed at children. In other countries, such as Canada, the laws may be stricter regarding “expectations relating to publicly available information,” Bowman said.

There have been a few legal cases involving screen scraping that can be looked to for guidance. One, in Canada, involved Century 21 and Rogers Communication. The latter was accused of indexing, storing, and displaying photos and descriptions of properties that were for sale from Century 21’s Web site. Rogers had used robots to crawl the site, an action that was prohibited by the site’s terms of use. Rogers was found guilty of copyright infringement; $33,000 was awarded to the plaintiff.


When deciding how to

When deciding how to implement your social media marketing strategies, it is important to take into consideration the nature of your products and services. For example, if purchasing your products is something that most of your customers would prefer to keep private, then do not put Facebook-like buttons right next to the buy buttons! Eventually, someone will click it accidentally and then get angry at your business.

Great Article but needs more security solutions


Great article! The one thing I believe missing is more effective ways to protect a website from screen scraping. Robots.txt unfortnantly doesn't work. Luckily, there are several companies out there that offer or specialize in Scraping protection.  Services such as,, or can put a barrier to entry to prevent scrapers from taking your content. 


Great article ! Many bots

Great article !

Many bots are very sophisticated, changing their IP and its hard to track them

SiteBlackBox  can help you to stop such abuse and keeps those bots away for good


The Magazine — Past Issues


Beyond Print

SM Online

See all the latest links and resources that supplement the current issue of Security Management magazine.