org.apache.nutch.crawl
Class DefaultFetchSchedule
java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.nutch.crawl.AbstractFetchSchedule
org.apache.nutch.crawl.DefaultFetchSchedule
- All Implemented Interfaces:
- Configurable, FetchSchedule
public class DefaultFetchSchedule
- extends AbstractFetchSchedule
This class implements the default re-fetch schedule. That is, no matter
if the page was changed or not, the fetchInterval
remains
unchanged, and the updated page fetchTime will always be set to
fetchTime + fetchInterval * 1000
.
- Author:
- Andrzej Bialecki
Method Summary |
void |
setFetchSchedule(String url,
WebPage page,
long prevFetchTime,
long prevModifiedTime,
long fetchTime,
long modifiedTime,
int state)
Sets the fetchInterval and fetchTime on a
successfully fetched page. |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
DefaultFetchSchedule
public DefaultFetchSchedule()
setFetchSchedule
public void setFetchSchedule(String url,
WebPage page,
long prevFetchTime,
long prevModifiedTime,
long fetchTime,
long modifiedTime,
int state)
- Description copied from class:
AbstractFetchSchedule
- Sets the
fetchInterval
and fetchTime
on a
successfully fetched page. NOTE: this implementation resets the
retry counter - extending classes should call super.setFetchSchedule() to
preserve this behavior.
- Specified by:
setFetchSchedule
in interface FetchSchedule
- Overrides:
setFetchSchedule
in class AbstractFetchSchedule
- Parameters:
url
- url of the pageprevFetchTime
- previous value of fetch time, or -1 if not availableprevModifiedTime
- previous value of modifiedTime, or -1 if not availablefetchTime
- the latest time, when the page was recently re-fetched. Most FetchSchedule
implementations should update the value in to something greater than this value.modifiedTime
- last time the content was modified. This information comes from
the protocol implementations, or is set to < 0 if not available. Most FetchSchedule
implementations should update the value in to this value.state
- if FetchSchedule.STATUS_MODIFIED
, then the content is considered to be "changed" before the
fetchTime
, if FetchSchedule.STATUS_NOTMODIFIED
then the content is known to be unchanged.
This information may be obtained by comparing page signatures before and after fetching. If this
is set to FetchSchedule.STATUS_UNKNOWN
, then it is unknown whether the page was changed; implementations
are free to follow a sensible default behavior.
Copyright © 2012 The Apache Software Foundation