j  � ht://Dig: htdig � 0  

 htdig



W ht://Dig Copyright © 1995-2002 The ht://Dig Group
8 Please see the file COPYING for license information.






Synopsis



 htdig [options]





 Description



< Htdig retrieves HTML documents using the HTTP protocol and= gathers information from these documents which can later be5 used to search these documents. This program can be" referred to as the search robot.





Options





 -a

6 Use alternate work files. Tells htdig to append 8 .work to database files, causing a second copy of5 the database to be built. This allows the original= files to be used by htsearch during the indexing run. When; used without the "-i" flag for an update dig, htdig will< use any existing .work files for the databases to update.

 -c configfile

: Use the specified configfile file instead of the default.

 -h maxhops

6 Restrict the dig to documents that are at most 6 maxhops links away from the starting document.8 This only works if -i is also given.

 -i

1 Initial. Do not use any old databases. This is/ accomplished by first erasing the databases.

 -m [url_file]

8 Minimal. Only index the URLs in the file provided and6 no others. The url_file can be a "-", causing htdig# to read the URLs from the STDIN.

 -s

3 Print statistics about the dig after completion.

 -t

9 Create an ASCII version of the document database. This8 database is easy to parse with other programs so that: information can be extracted from it for purposes other4 than searching. One could gather some interesting! statistics from this database.8

Each line in the file starts with the document id  followed by a list of8 \tfieldname:value.6 The fields always appear in the order listed below:

 - # % 9 8 - ' 0 6 8 8 9 8 1 5 7 8 9
fieldnamevalue
uURL
tTitle
aState (0 = normal, 1 = not found, 2) = not indexed, 3 = obsolete)
mLast modification time as reported by the server
sSize in bytes
HExcerpt
hMeta description
lTime of last retrieval
LCount of the links in the document (outgoing links)
bCount of the links to the document* (incoming links or backlinks)
cHopCount of this document
gSignature of the document used for duplicate-detection
eE-mail address to use for a/ notification message from htnotify
nDate to send out a notification e-mail message
SSubject for a notification e-mail message
dThe text of links pointing to this document. (e.g. <a> href="docURL">description</a>)
AAnchors in the document (i.e. <A NAME=...)

-u username:password

9 Tells htdig to send the supplied username and password: with each HTTP request. The credentials will be encoded: using the 'Basic' authentication scheme. There : HAS to be a colon (:) between the username and password.

 -v

4 Verbose mode. This increases the verbosity of the9 program. Using more than 2 is probably only useful for6 debugging purposes. The default verbose mode (using2 only one -v) gives a nice progress report while- digging. This progress report can be a bit2 cryptic, so here is a brief explanation. A line3 is shown for each URL, with 3 numbers before the0 URL and some symbols after the URL. The first. number is the number of documents parsed so2 far, the second is the DocID for this document,1 and the third is the hop count of the document, (number of hops from one of the start_url0 documents). After the URL, it shows a "*" for2 a link in the document that it already visited,1 a "+" for a new link it just queued, and a "-"0 for a link it rejected for any of a number of/ reasons. To find out what those reasons are,4 you need to run htdig with at least 3 -v options,5 i.e. -vvv. If there are no "*", "+" or "-" symbols2 after the URL, it doesn't mean the document was2 not parsed or was empty, but only that no links0 to other documents were found within it. With. more verbose output, these symbols will get5 interspersed in several lines of debugging output.

0 url_file (at end of arguments, after options)

4 Get the list URLs to start indexing from the file6 provided. This will override the default start_url.3 The url_file can be a "-", causing htdig to read the URLs from the STDIN.







Files





 CONFIG_DIR/htdig.conf

" The default configuration file.







See Also



% htmerge,5 htsearch,9 Configuration file format, andI % A Standard for Robot Exclusion.


, Last modified: $Date: 2002/01/27 05:33:20 $ ÿÿ