Gigabot is the name of Gigablast's indexing agent, also known as a spider. Gigabot is like a thousand internet users busily surfing the web. But it moves from page to page indexing the content it finds. Gigablast serves millions of queries per day to its clients.
Gigabot obeys the robots.txt standard. robots.txt is a file that you place in your web server's root html directory that tells search engines what pages you do not want indexed. This page [external link] talks more about it.
Gigabot now supports the Crawl-delay directive.
Gigabot also obeys the following nofollow, noarchive and noindex meta tags:
<meta name="robots" content="noindex"> // Gigabot will not index a page that has this tag <meta name="gigabot" content="noindex"> // Gigabot will not index a page that has this tag <meta name="robots" content="noarchive"> // Gigabot will not cache a page that has this tag <meta name="gigabot" content="noarchive"> // Gigabot will not cache a page that has this tag <meta name="robots" content="nofollow"> // Gigabot will not follow links on a page that has this tag <meta name="gigabot" content="nofollow"> // Gigabot will not follow links on a page that has this tag