Ignore canonical links in web pages
Managed by | Updated .
Canonical links are great when used correctly - they allow a web administrator to define the URL that should be used to reference the page (especially when there may be lots of aliases that all resolve to the same page). By default Funnelback will always use a canonical link if it is defined, replacing the URL of the page that was crawled with whatever was defined in the canonical link tag.
However, it is fairly common for the canonical link tag to be used incorrectly, with an incorrect URL being placed within the tag.
For example if every page on a website had a canonical link that was the site's homepage then Funnelback would only end up with a single document in the index as all the other pages would be marked as duplicates.
This article outlines the steps required to ignore canonical links that are embedded in web pages.
There are a couple of ways to ignore the canonical links defined in web pages.
Ignore all canonical links
The easiest method is to set an indexer option that instructs the indexer to ignore the canonical link references.
Add the following to the indexer_options in collection.cfg
Update or reindex the collection
Ignore some canonical links
The only way to achieve this is to write a jsoup filter that removes the canonical links from the desired pages.