Gigablast Help

Site Clustering

It is often undesirable to have many results listed from the same site. Site Clustering will essentially limit the number of returned results from any given site or two, but it will provide a link which says "more results from this site" in case the searcher wishes it.

Duplicate Results Removal

When dup results removal is enabled Gigablast will remove results that have the exact same content as other results. The psc parameter can be used to dedup documents with similar content.

Gigablast allows you to receive the search results in a number of formats useful for interfacing to your program. Here is an example of the XML feed.

The XML reply has the following format(but without the comments):

# The XML reply uses the Latin-1 Character Set (ISO 8859-1) when using raw=8
<?xml version="1.0" encoding="ISO-8859-1" ?>

# OR when using raw=9
<?xml version="1.0" encoding="utf-8" ?>

# It consists of one, and only one, response.
<response>

  # If any error was received in processing the request, it will be here.
  <error>Out of memory</error>
  # The numeric code of the error, if any, goes here.
  # See all the Error Codes, but the   # following errors are most likely:
  # 32771 - A cached page was not found when it should have been.
  #    12 - There was a shortage of memory to properly process the request.
  # 32863 - Queried collection does not exist.
  <errno>32790</errno>

  # Total number of documents in the collection being searched.
  <docsInCollection>2060245584</docsInCollection>
  # An APPROXIMATION of the total number of search results for the query.
  <hits>4838158</hits>
  # This is "1" if more results are available after these, "0" if not.
  <moreResultsFollow>1</moreResultsFollow>

  # If present and value is 1, some words in the query were censored for content.
  <queryCensored>1</queryCensored>
  # If present, the value is the number of results that were censored for content.
  <resultsCensored>3</resultsCensored>
  # If this tag is present, it will hold an alternate spelling recommendation 
  # for the query. The &spell=1 parameter must be present in the query url,
  # however, for you to get a spelling recommendation back.
  <spell>nose</spell>

  # If this tag is present, it contains the list of query words that were 
  # ignored as individual words, but not necessarily as part of a phrase
  <ignoredWords>the in of</ignoredWords>
  # This is how many of the search results contain ALL of the query terms.
  # It is only used for printing the "blue bar" for doing SuperRecall
  <minNumExactMatches>300</minNumExactMatches>

  # The list of related topics, each enclosed by <topic> tags. 
  # You must provide a topics parameter to the query url to get topics.
  <topic>
    # Each topic has a score. A score of 50% or more is considered pretty good.
    <score>63</score>
    # Out of the documents scanned, how many contain this topic.
    <docCount>4</docCount>
    # The topic popularity. A measure of how popular the word or phrase is
    # based on how many web pages contain it overall. Ranges from 0 to 1000.
    # 1000 being the most popular.
    <popularity>16</popularity>
    # The docIds of the documents scanned that contain this topic.
    <docId>9030668134</docId>
    <docId>265962215563</docId>
    <docId>43940265200</docId>
    <docId>264861015824</docId>
    # The topic name.
    <name><![CDATA[Race Cars]]></name>
    # And OPTIONALLY the name of the meta tag it was derived from.
    <from>keywords</from>
  </topic>

  # The list of reference pages for the search results.  Each reference is
  # enclosed in <reference> tags.
  <reference>
    # Each reference has a score based on its relevance to the query.
    <score>93</score>
    # Title of the reference page
    <title></title>
    # Url of the reference page
    <url><![CDATA[http://www.greatreference.com/]]></url>
  </reference>

  # The list of related pages for the search results.  Each related page is
  # enclosed in <related> tags.
  <related>
    # Each related page has a score based on its relevance to the query.
    <score>91</score>
    # Title of the related page.
    <title></title>
    # Url of the related page.
    <url><![CDATA[http://www.similar.com/]]></url>
    # Summary of the related page.
    <sum><![CDATA[This page is similar to the results]]></sum>
  </related>

  # The list of search results, each enclosed in <result> tags.
  <result>

    # Each result has a title. This may be empty if none was found on the page.
    <title><![CDATA[My Homepage]]></title>

    # Each result has a summary. This may be empty. The summary is generated 
    # so as to contain the query terms if possible.
    <sum><![CDATA[All about my interests and hobbies]]></sum>

    # If this result is categorized under the DMOZ Directory, data about each
    # category it is in will be enclosed in a <dmoz> tag.
    <dmoz>
      # The category ID number of this category.
      <dmozCatId>172</dmozCatId>
      # The path of this category in the directory.
      <dmozCat><![CDATA[Health: Dentistry]]></dmozCat>
      # Title of this result as listed in the directory.
      <dmozTitle><![CDATA[My Homepage]]></dmozTitle>
      # Description of this page as listed in the directory.
      <dmozDesc><![CDATA[A Dentist's Home Page]]></dmozDesc>
    </dmoz>
    # If the directory is being given along with the results, this is the number of
    # stars given to this page based on its quality.
    <stars>3</stars>

    # Each result may have a sequence of <display> tags if the feed input
    # contained a dt parameter. This allows you to extract
    # information contained in meta tags in the content of each search result.
    # To obtain the contents of the author meta tag, you would need to pass in
    # dt=author.
    <display name="author"><![CDATA[Contents of the meta author tag]]></display>

    # Each result has a URL. This should never be empty.
    <url><![CDATA[http://www.mydomain.com/mypage.html]]></url>
    # The size of the page in kilobytes. Accurate to the tenth of a kilobyte.
    <size>5.6</size>
    # The time the page was last INDEXED. It may not have been indexed in a 
    # long time if the page's content has not changed. The time is expressed 
    # in seconds since the epoch. (Jan 1, 1969)
    <spidered>1064367311</spidered>
    # The time the page was last modified. This is taken from the HTTP reply 
    # of the web server when downloading the page. It is 0 if unknown. The time
    # is expressed in seconds since the epoch. (Jan 1, 1969)
    <lastMod>1058477041</lastMod>

    # The assigned docid for this page. This number is unique and used 
    # internally by Gigablast to identify this page. It is used to retrieve the
    # "cached copy" of the page.
    <docId>65990704587</docId>
    # When doing site clustering, this tag will be present if the result is 
    # from the same hostname as a previous result for the same query. It 
    # indicates that you might want to indent the result. Any further results 
    # from this same hostname will be stripped from the feed.
    <clustered>1</clustered>

    # When Topic Clustering is being used, these will display results which 
    # are considered similar to this result and have been clustered under it. 
    # Each similar result is enclosed in a <similar> tag. 
    <similar>
      # The url for the similar result.
      <url><![CDATA[http://www.similar.com/]]></url>
      # The title of the similar result.
      <title><![CDATA[A similar topic]]></title>
    </similar>
    # If this is present and set to 1, there are more similar results beyond 
    # those given here. 
    <moreSimilar>1</moreSimilar>

    # This is a standard HTTP MIME content classification of the result. It is 
    # not present if the page is text/html. Otherwise, it will be one of the
    # following: text/plain
    #            text/xml
    #            application/pdf
    #            application/msword
    #            application/vnd.ms-excel
    #            application/mspowerpoint
    #            application/postscript
    <contentType>text/plain</contentType>
    # The documents are all sorted by this score. This score is a generally a
    # product of the WEIGHT of the query term and the COUNT of the query term
    # in this document. The WEIGHT is usually influenced by them term frequency
    # of the query term (rarer terms get more WEIGHT), by the additional weight
    # received by phrases which can be adjusted in the Master Controls, and,
    # possibly, by any user-defined weight in the query (See Weighting Query Terms).
    # This score is normalized by dividing by the maximum
    # score for all documents in the search results and then making it into a
    # percentage, so the score ranges from 0 to 100, and the first result
    # should always have score 100.
    <score>100</score>
    # This is the absolute score. Useful for merging results from other
    # collections or other search engines.
    <absScore>5132</absScore>

    # This is the language the page was detected as.
    <language><![CDATA[English]]></language>
    # The quality of the document as determined by Gigablast. Ranges from 0 to 100.
    <quality>80</quality>
    # The character set this page was originally encoded in. 
    <charset><![CDATA[utf-8]]></charset>

  </result>

  <result>

  ...
  </result>

  ...

  # If the directory has been requested, this node will include the directory
  # structure for the requested category.  Typically this is above the results.
  <directory>
    # Category ID for the displayed directory structure.
    <dirId>172</dirId>
    # Directory path of this category listing.
    <dirName>Health: Dentistry</dirName>

    # Specifies if the directory listing is displayed in a Right-To-Left format.
    <dirIsRTL>1</dirIsRTL>
    # Sub-Categories listed as letters meant to be displayed as a letter bar.
    # Each sub-category will be enclosed in a <letterbar> tag.
    <letterbar><![CDATA[Health/Dentistry/A]]>    
    # Every sub category will include a count of how many urls are listed under it.
      <urlcount>5<urlcount>

    </letterbar>
    # Normal sub-categories listed in groups.  These are listed in order of group
    # and alphabetically within each group. Each sub-category is enclosed in a
    # <narrow2>, <narrow1>, or <narrow> tag.
    <narrow2><![CDATA[Health/Dentistry/Regional]]>
      <urlcount>0<urlcount>

    </narrow2>
    <narrow1><![CDATA[Health/Dentistry/Association]]>
      <urlcount>122<urlcount>
    </narrow1>
    <narrow><![CDATA[Health/Dentistry/Children]]>

      <urlcount>24<urlcount>
    </narrow>
    # Symbolically linked sub-categories physically under a different category.
    # These will be interwoven alphabetically within the respective narrow groups.
    # The name listed before the path is the symbolic name.  Each symbolically linked
    # sub-category is enclosed in a <symbolic2>, <symbolic1>, or 
    # <symbolic> tag.
    <symbolic2><![CDATA[Dentophobia:Health/Mental_Health/Disorders/Anxiety/Phobias/Dentophobia]]>

      <urlcount>2<urlcount>
    </symbolic2>
    <symbolic1><![CDATA[Dental_Laboratories:Buisness/Healthcare/Products_and_Services/Dentistry/]]>
      <urlcount>71<urlcount>

    </symbolic1>
    <symbolic><![CDATA[Products:Shopping/Health/Dental]]>
      <urlcount>71<urlcount>
    </symbolic>
    # Seperate categories in the directory which are related to this one.
    <related><![CDATA[Society/Issues/Health/Dentistry]]>

      <urlcount>4</urlcount>
    </related>
    # This category in other languages in the directory.
    <altlang><![CDATA[Basque:World/Euskara/Osasuna/Odontologia]]>
      <urlcount>7</urlcount>

    </altlang>
  </directory>

</response>

Copy and paste the following html code into your web page to display a search box that searchs the web via Gigablast:

 <form method="get"
 action="http://www.gigablast.com/search">
 <input type="text" name="q" size="40">
 <input type="submit" value="search">
 </form>

See below for instructions on how to add a search box to your web site.

• Copy and paste the following html code into your web page to display your site search box:

  
    <form method="get" action="http://www.gigablast.com/search">
    <input type="hidden" name="sc" value="0">
    <input type="hidden" name="sites" value="www.mydomain.com">
    <input type="hidden" name="iu" value="http://www.mydomain.com/logo.gif">
    <input type="hidden" name="iw" value="200">
    <input type="hidden" name="ih" value="50">
    <input type="hidden" name="ix" value="http://www.mylink.com/">
    <input type="text" name="q" size="40">
    <input type="submit" value="search">
    </form>
    

• Replace www.mydomain.com above with your site name, or a space separated list of up to 500 of the sites you want to search. Sites need not be only domains, but can contain file paths as well.
• Replace www.mydomain.com/logo.gif above with the url of your logo.
• Replace 200 and 50 with the width and height of your logo in pixels. Alternatively eliminate the lines and the image will default to its native size.
• Replace www.mylink.com above with the url to visit when your logo is clicked.
• If you are searching a long list of sites you may need to replace method=get above with method=post since some browsers can not submit long get requests.
• If you are using the site search index, then be sure to add your root url(s) via the addurl page. If your root url page uses frames and does not have any outgoing hyperlinks then you will need to add some other pages in order to spider your site. Wait a few minutes for Gigablast to download your pages so your search box actually has something to search.

To make your own custom topic engine, follow the steps below: