Site Clustering

It is often undesirable to have many results listed from the same site. Site Clustering will essentially limit the number returned results from any given site to two, but it will provide a link which says "more results from this site" in case the searcher wishes it.

Duplicate Results Removal

When dup results removal is enabled Gigablast will remove results that have the exact same content as other results. The psc parameter can be used to dedup documents with similar content.

Cached Web Page Parameters

To get a cached web page from Gigablast use a url
like: http://www.gigablast.com/get?d=94555390410&ih=1&q=my+query&c=main   where:

Gigablast allows you to receive the search results in a number of formats useful for interfacing to your program. Here is an example of the XML feed.

The XML reply has the following format (but without the comments):

# The XML reply uses the Latin-1 Character Set (ISO 8859-1) when using raw=8
<?xml version="1.0" encoding="ISO-8859-1" ?>

# OR when using raw=9
<?xml version="1.0" encoding="utf-8" ?>

# It consists of one, and only one, response.
<response>

  # If any error was received in processing the request, it will be here.
  <error>Out of memory</error>
  # The numeric code of the error, if any, goes here.
  # See all the Error Codes, but the   # following errors are most likely:
  # 32771 - A cached page was not found when it should have been.
  #    12 - There was a shortage of memory to properly process the request.
  # 32863 - Queried collection does not exist.
  <errno>32790</errno>

  # Total number of documents in the collection being searched.
  <docsInCollection>2060245584</docsInCollection>
  # An APPROXIMATION of the total number of search results for the query.
  <hits>4838158</hits>
  # This is "1" if more results are available after these, "0" if not.
  <moreResultsFollow>1</moreResultsFollow>

  # If present and value is 1, some words in the query were censored for content.
  <queryCensored>1</queryCensored>
  # If present, the value is the number of results that were censored for content.
  <resultsCensored>3</resultsCensored>
  # If this tag is present, it will hold an alternate spelling recommendation 
  # for the query. The &spell=1 parameter must be present in the query url,
  # however, for you to get a spelling recommendation back.
  <spell>nose</spell>

  # If this tag is present, it contains the list of query words that were 
  # ignored as individual words, but not necessarily as part of a phrase
  <ignoredWords>the in of</ignoredWords>
  # This is how many of the search results contain ALL of the query terms.
  # It is only used for printing the "blue bar" for doing SuperRecall
  <minNumExactMatches>300</minNumExactMatches>

  # The list of related topics, each enclosed by <topic> tags. 
  # You must provide a topics parameter to the query url to get topics.
  <topic>
    # Each topic has a score. A score of 50% or more is considered pretty good.
    <score>63</score>
    # Out of the documents scanned, how many contain this topic.
    <docCount>4</docCount>
    # The docIds of the documents scanned that contain this topic.
    <docId>9030668134</docId>
    <docId>265962215563</docId>
    <docId>43940265200</docId>
    <docId>264861015824</docId>
    # The topic name.
    <name><![CDATA[Race Cars]]></name>
    # And OPTIONALLY the name of the meta tag it was derived from.
    <from>keywords</from>
  </topic>

  # The list of search results, each enclosed in <result> tags.
  <result>

    # Each result has a title. This may be empty if none was found on the page.
    <title><![CDATA[My Homepage]]></title>

    # Each result has a summary. This may be empty. The summary is generated 
    # so as to contain the query terms if possible.
    <sum><![CDATA[All about my interests and hobbies]]></sum>

    # If this result is categorized under the DMOZ Directory, data about each
    # category it is in will be enclosed in a <dmoz> tag.
    <dmoz>
      # The category ID number of this category.
      <dmozCatId>172</dmozCatId>
      # The path of this category in the directory.
      <dmozCat><![CDATA[Health: Dentistry]]></dmozCat>
      # Title of this result as listed in the directory.
      <dmozTitle><![CDATA[My Homepage]]></dmozTitle>
      # Description of this page as listed in the directory.
      <dmozDesc><![CDATA[A Dentist's Home Page]]></dmozDesc>
    </dmoz>
    # If the directory is being given along with the results, this is the number of
    # stars given to this page based on its quality.
    <stars>3</stars>

    # Each result may have a sequence of <display> tags if the feed input
    # contained a dt parameter. This allows you to extract
    # information contained in meta tags in the content of each search result.
    # To obtain the contents of the author meta tag, you would need to pass in
    # dt=author.
    <display name="author"><![CDATA[Contents of the meta author tag]]></display>

    # Each result has a URL. This should never be empty.
    <url><![CDATA[http://www.mydomain.com/mypage.html]]></url>
    # The size of the page in kilobytes. Accurate to the tenth of a kilobyte.
    <size>5.6</size>
    # The time the page was last indexed. It may not have been indexed in a 
    # long time if the page's content has not changed. The time is expressed 
    # in seconds since the epoch. (Jan 1, 1969)
    <indexed>1064367311</indexed> 
    # The time the page was last modified. This is taken from the HTTP reply 
    # of the web server when downloading the page. It is 0 if unknown. The time
    # is expressed in seconds since the epoch. (Jan 1, 1969)
    <lastMod>1058477041</lastMod>

    # The assigned docid for this page. This number is unique and used 
    # internally by Gigablast to identify this page. It is used to retrieve the
    # "cached copy" of the page.
    <docId>65990704587</docId>

    # When doing site clustering, this tag will be present if the result is 
    # from the same hostname as a previous result for the same query. It 
    # indicates that you might want to indent the result. Any further results 
    # from this same hostname will be stripped from the feed.
    <clustered>1</clustered>

    # When Topic Clustering is being used, these will display results which 
    # are considered similar to this result and have been clustered under it. 
    # Each similar result is enclosed in a <similar> tag. 
    <similar>
      # The url for the similar result.
      <url><![CDATA[http://www.similar.com/]]></url>
      # The title of the similar result.
      <title><![CDATA[A similar topic]]></title>
    </similar>
    # If this is present and set to 1, there are more similar results beyond 
    # those given here. 
    <moreSimilar>1</moreSimilar>

    # This is a standard HTTP MIME content classification of the result. It is 
    # not present if the page is text/html. Otherwise, it will be one of the
    # following: text/plain
    #            text/xml
    #            application/pdf
    #            application/msword
    #            application/vnd.ms-excel
    #            application/mspowerpoint
    #            application/postscript
    <contentType>text/plain</contentType>

    # This is the language the page was detected as.
    <language><![CDATA[English]]></language>
    # The quality of the document as determined by Gigablast. Ranges from 0 to 100.
    <quality>80</quality>
    # The character set this page was originally encoded in. 
    <charset><![CDATA[utf-8]]></charset>
    
  </result>

  <result>

  ...
  </result>

  ...

  # If the directory has been requested, this node will include the directory
  # structure for the requested category.  Typically this is above the results.
  <directory>
    # Category ID for the displayed directory structure.
    <dirId>172</dirId>
    # Directory path of this category listing.
    <dirName>Health: Dentistry</dirName>

    # Specifies if the directory listing is displayed in a Right-To-Left format.
    <dirIsRTL>1</dirIsRTL>
    # Sub-Categories listed as letters meant to be displayed as a letter bar.
    # Each sub-category will be enclosed in a <letterbar> tag.
    <letterbar><![CDATA[Health/Dentistry/A]]>    
    # Every sub category will include a count of how many urls are listed under it.
      <urlcount>5<urlcount>

    </letterbar>
    # Normal sub-categories listed in groups.  These are listed in order of group
    # and alphabetically within each group. Each sub-category is enclosed in a
    # <narrow2>, <narrow1>, or <narrow> tag.
    <narrow2><![CDATA[Health/Dentistry/Regional]]>
      <urlcount>0<urlcount>

    </narrow2>
    <narrow1><![CDATA[Health/Dentistry/Association]]>
      <urlcount>122<urlcount>
    </narrow1>
    <narrow><![CDATA[Health/Dentistry/Children]]>

      <urlcount>24<urlcount>
    </narrow>
    # Symbolically linked sub-categories physically under a different category.
    # These will be interwoven alphabetically within the respective narrow groups.
    # The name listed before the path is the symbolic name.  
    # Each symbolically linked
    # sub-category is enclosed in a <symbolic2>, <symbolic1>, or 
    # <symbolic> tag.
    <symbolic2><![CDATA[Dentophobia:Health/Mental_Health/Disorders/Anxiety/Phobias/Dentophobia]]>

      <urlcount>2<urlcount>
    </symbolic2>
    <symbolic1><![CDATA[Dental_Laboratories:Buisness/Healthcare/Products_and_Services/Dentistry/]]>
      <urlcount>71<urlcount>

    </symbolic1>
    <symbolic><![CDATA[Products:Shopping/Health/Dental]]>
      <urlcount>71<urlcount>
    </symbolic>
    # Seperate categories in the directory which are related to this one.
    <related><![CDATA[Society/Issues/Health/Dentistry]]>

      <urlcount>4</urlcount>
    </related>
    # This category in other languages in the directory.
    <altlang><![CDATA[Basque:World/Euskara/Osasuna/Odontologia]]>
      <urlcount>7</urlcount>

    </altlang>
  </directory>

</response>