Managed by | Updated .
The Funnelback web crawler supports the following:
Funnelback honours robots.txt directives as outlined at http://www.robotstxt.org/robotstxt.html
The FunnelBack user agent can be used to provide Funnelback specific robots.txt directives
e.g. Prevent access to /search* and /login* for all crawlers but allow Funnelback to access everything.
User-agent: * Disallow: /search Disallow: /login User-agent: FunnelBack Disallow:
Robots meta tags
Funnelback honours robots meta tags as outlined at http://www.robotstxt.org/meta.html as well as the nosnippet and noarchive directives.
The following directives can appear within a <meta name="robots"> tag:
- follow / nofollow
- index / noindex
- nosnippet / noarchive
<!-- index this page but don't follow links --> <meta name="robots" content="index, nofollow" /> <!-- index this page but don't follow links and don't allow caching or snippets --> <meta name="robots" content="index, nofollow, nosnippet" />
HTML <a rel="nofollow">
Funnelback honours nofollow directives provided in rel attribute of a HTML anchor <a> tag.
<!-- don't follow this link --> <a href="mylink.html" rel="nofollow" />
Google HTML comment tags that are equivalent to the Funnelback noindex/endnoindex tags. The following are aliases of Funnelback's native tags:
<!-- noindex --> == <!-- googleoff: index --> == <!-- googleoff: all -->
<!-- endnoindex --> == <!-- googleon: index --> == <!-- googleon: all -->
Note: other googleoff/on tags are not supported.