Nutch plugin - selective crawling based on keywords

I need a NUTCH plugin that enables NUTCH (version 1.7) to crawl ONLY webpages that contains specific words that can be set/reset by the NUTCH administrator:

for example I want to crawl only webpages that contain a set of keywords like: ("job" & "apply") or ("job" & "submit") or webpages that contain

words like: ("education" & "requirements" & "experience" & "benefits").

The point of this is to achieve selective crawling (at source) and thus significantly reducing the amount of webpages that we have to crawl. The selection of the webpages by comparison to these words will be made at the source, before fetching the pages - as the goal of the plugin is to significantly decrease the amount of the webpages that will actually be crawled. (this is a MUST HAVE!!!)

The plugin should not slow down the NUTCH crawler, as speed is an important objective that has to be achieved.

The plugin should be instaled on my NUTCH crawler and the construction has to be stable on a large amount of data crawled daily

(more than 5000000 selected&crawled webpages daily).

Apache

Proje NO: #5382795

Proje hakkında