Funnelback robots support

Managed by | Updated .

The Funnelback web crawler supports the following:

Robots.txt

Funnelback honours robots.txt directives as outlined at http://www.robotstxt.org/robotstxt.html

The FunnelBack user agent can be used to provide Funnelback specific robots.txt directives

e.g. Prevent access to /search* and /login* for all crawlers but allow Funnelback to access everything.

User-agent: *
Disallow: /search
Disallow: /login
User-agent: FunnelBack
Disallow:

Robots meta tags

Funnelback honours robots meta tags as outlined at http://www.robotstxt.org/meta.html as well as the nosnippet and noarchive directives.

The following directives can appear within a <meta name="robots"> tag:

  • follow / nofollow
  • index / noindex
  • nosnippet / noarchive

e.g.

<!-- index this page but don't follow links -->
<meta name="robots" content="index, nofollow" />
<!-- index this page but don't follow links and don't allow caching or snippets -->
<meta name="robots" content="index, nofollow, nosnippet" />

HTML <a rel="nofollow">

Funnelback honours nofollow directives provided in rel attribute of a HTML anchor <a> tag.

e.g.

<!-- don't follow this link -->
<a href="mylink.html" rel="nofollow" />

Googleoff/Googleon directives

Google HTML comment tags that are equivalent to the Funnelback noindex/endnoindex tags.  The following are aliases of Funnelback's native tags:

<!-- noindex --> == <!-- googleoff: index --> == <!-- googleoff: all -->
<!-- endnoindex --> == <!-- googleon: index --> == <!-- googleon: all -->

Note: other googleoff/on tags are not supported.

Was this artcle helpful?

Tags
Type: Keywords:
XML

Comments