Funnelback robots support
Managed by | Updated .
The Funnelback web crawler supports the following:
Robots.txt
Funnelback honours robots.txt directives as outlined at http://www.robotstxt.org/robotstxt.html
The FunnelBack user agent can be used to provide Funnelback specific robots.txt directives
e.g. Prevent access to /search* and /login* for all crawlers but allow Funnelback to access everything.
User-agent: *
Disallow: /search
Disallow: /login
User-agent: FunnelBack
Disallow:Robots meta tags
Funnelback honours robots meta tags as outlined at http://www.robotstxt.org/meta.html as well as the nosnippet and noarchive directives.
The following directives can appear within a <meta name="robots"> tag:
- follow / nofollow
- index / noindex
- nosnippet / noarchive
e.g.
<!-- index this page but don't follow links -->
<meta name="robots" content="index, nofollow" />
<!-- index this page but don't follow links and don't allow caching or snippets -->
<meta name="robots" content="index, nofollow, nosnippet" />HTML <a rel="nofollow">
Funnelback honours nofollow directives provided in rel attribute of a HTML anchor <a> tag.
e.g.
<!-- don't follow this link -->
<a href="mylink.html" rel="nofollow" />Googleoff/Googleon directives
Google HTML comment tags that are equivalent to the Funnelback noindex/endnoindex tags. The following are aliases of Funnelback's native tags:
<!-- noindex --> == <!-- googleoff: index --> == <!-- googleoff: all -->
<!-- endnoindex --> == <!-- googleon: index --> == <!-- googleon: all -->Note: other googleoff/on tags are not supported.
Comments