Vertical aggregation & Pattern matching crawlers

Posted on November 27, 2014 by Ran Geva

read the article

After bashing various crawling techniques, I would like to describe the technique we use here, at webhose.io, a technology that was developed over the past 8 years. Our crawlers were developed with the following demands in mind: Efficient on server resources, i.e CPU & bandwidth Fast in fetching and extracting content Easily add new sites...

Continue reading →

Posted in API

Crawling Horrors – Browser Scraping

Posted on November 25, 2014 by Ran Geva

read the article

In my previous blog post, I wrote about RSS crawlers, and why they don’t really work. In this post I want to discuss the technique of using a headless browser to parse a website and extract its content. A headless browser is a web browser without a graphical user interface. The logic behind using a...

Continue reading →

Posted in Technology

Crawling Horrors – RSS Crawlers

Posted on November 24, 2014 by Ran Geva

read the article

One of the fastest, simplest and unfortunately wrong ways of extracting content out of a website, is by reading its RSS feeds. I will show you how its done and why it’s useless. Each RSS feed already contains the data, structured and ready for harvesting, so content extraction is indeed simple and fast. Let’s take...

Continue reading →

Posted in Technology