Using search.html as auto-completion source

Managed by | Updated 3 years ago.

Background

Funnelback has a native auto-completion system (suggest.json) that suggests completions as the user types. This auto-completion system uses a JSON REST API to provide suggestions based on the user's partially entered query using a lightweight search algorithm. This ensures that suggestions are returned quickly and with minimal load on the server.

In some cases however, there may be a need to use the full text search algorithm to provide suggestions to the user. In doing this however, we face a couple of challenges:

• The user has not entered the full query they are looking for. We may only have queries like "con" or "dav" not giving a lot of context to what the user is searching for yet.
• The time it takes to run a full query using the standard search algorithm is slower than using the native autocompletion system

To overcome these challenges we can take the following steps:

• Utilise the top suggestions from the native autocomplete and perform a full text search based on these suggestions
• Limit the data returned by the result packet
• Turn off unnecessary search features or settings

Step-by-step configuration changes

Step 1. Add a custom profile

Setup a profile for the query completion. Lets call the profile auto-completion.

Step 2. Configure profile display and ranking options

Create a padre_opts.cfg file in the auto-completion profile folder with the following content:

-daat=5000 -num_ranks=1 -vsimple=1 -contextual_navigation=0 -spelling=0 -collapsing=0 -SQE=1 -SBL=1 -SHLM=0

Step 3. Add a post-process hook script to handle partial query expansion

Create a post process hook script:

// imports required for access to the padre suggest service
import java.io.File;
import java.util.List;
import com.funnelback.common.config.DefaultValues;
import com.funnelback.dataapi.connector.padre.PadreConnector;
import com.funnelback.dataapi.connector.padre.suggest.Suggestion;
import com.funnelback.dataapi.connector.padre.suggest.Suggestion.ActionType;
import com.funnelback.dataapi.connector.padre.suggest.Suggestion.DisplayType;

if(transaction.question.inputParameterMap['profile'].equals("auto-completion") || transaction.question.inputParameterMap['profile'].equals("auto-completion_preview")) {
  def q = transaction.question
  q.form = "qc"

  // convert a partial query into a set of query terms
  // maximum number of query terms to expand partial query to - read from collection.cfg partial_query_expansion_index parameter
  // eg. partial_query=com might expand to query=[commerce commercial common computing]
  def partial_query_expansion_index = 5
  if ((q.collection.configuration.value(["partial_query_expansion_index"]) != null) && (q.collection.configuration.value(["partial_query_expansion_index"]).isInteger())) {
    partial_query_expansion_index = q.collection.configuration.value(["partial_query_expansion_index"])
  }

  def profile = "_default"
  if (q.inputParameterMap["profile"] != null) {
    profile = q.inputParameterMap["profile"]
  }

  if (q.inputParameterMap["partial_query"] != null) {
    File searchHome = new File("/opt/funnelback")
    File indexStem = new File(q.collection.configuration.value(["collection_root"]) + File.separator + "live" + File.separator + "idx","index")

    // NOTE: CONSTRUCTOR depends on the version of Funnelback being run
    List<Suggestion> suggestions = new PadreConnector(searchHome, indexStem, q.collection.id) //15.16+
    // List<Suggestion> suggestions = new PadreConnector(searchHome, indexStem) //15.0-15.14
    // List<Suggestion> suggestions = new PadreConnector(indexStem) //14.0 or earlier
    .suggest(q.inputParameterMap["partial_query"])
    .suggestionCount(partial_query_expansion_index)
    .fetch();

    def expanded_query = ""

    suggestions.each {
      expanded_query += '"'+it.key+'" '
    }

    // set the number of suggestions to the value of the configured auto-completion.show
    q.additionalParameters["num_ranks"] = [q.inputParameterMap["show"]]

    // set the query to the expanded set of query terms ORed together 
    if (expanded_query != "") {
      q.query = "["+expanded_query+"]"

  }    
}

Step 4. Update the auto-completion program

Change the autocomplete program to ../s/search.html. You can do this via the collection settings in the administration interface (administer > edit collection settings > autocomplete > completion program)

Step 5. Configure the amount of expansion

Add a collection.cfg parameter called partial_query_expansion_index. This determines the maximum number of query terms to expand the partial query to. By default the hook script will set this to 5, but you can adjust this through the collection so this one is optional.

Step 6. Create a template to return search results in the auto-completion JSON format

Add a template to the profile and call it qc.ftl. The template will render the suggestions in the same format as the native system. Give it the following content:

<#ftl encoding="utf-8" />
<#import "/web/templates/modernui/funnelback_classic.ftl" as s/>
<#import "/web/templates/modernui/funnelback.ftl" as fb/>
<#if question.inputParameterMap["callback"]?exists>${question.inputParameterMap["callback"]}([<#else>[</#if>
<@s.AfterSearchOnly>
<#if response.resultPacket.resultsSummary.totalMatching != 0>
<@s.Results>
<#if s.result.class.simpleName != "TierBar">
{
"key" : "<#if question.inputParameterMap["partial_query"]?exists>${question.inputParameterMap["partial_query"]?json_string}</#if>",
"disp" : "${s.result.title}",
"disp_t" : "H",
"wt" : "${s.result.score}",
"cat" : "",
"cat_t" : "1",
"action" : "${s.result.clickTrackingUrl}",
"action_t" : "U"
}<#if (s.result.rank < response.resultPacket.resultsSummary.currEnd)>,</#if>
</#if>
</@s.Results>
</#if>
</@s.AfterSearchOnly>
<#if question.inputParameterMap["callback"]?exists>])<#else>]</#if>

Step 7. Configure Funnelback to return the JSON content headers

Add a collection.cfg parameter that tells Funnelback to return a header indicating the response is Javascript (JSON) when accessing the qc template.

ui.modern.form.qc.content_type=application/javascript