Create a failover administration server using mediator and WedDAV

Home System & Server Adminstration Server Administration

Managed by | Updated .

Background

This article describes how to setup a hot-standby, failover administration server. It uses the same building blocks as supporting multiple query processor, but with additional automation to keep the failover administration server in sync with the main one.

Primary administration server

Configure the primary administration server as per the official instructions for configuring multiple query processors.

A number of additional additional steps should be configured as outlined below.

Optionally disable the WebDAV service

The WebDAV service used to transfer files between Funnelback servers. As no other Funnelback server is supposed to push files to the main admin server, the WebDAV service can be disabled on the main admin server.

It is optional, and is just a safety measure to ensure that any misconfiguration will not result in other servers pushing files to the main admin server.

To disable the WebDAV service, locate the daemon.services line from $SEARCH_HOME/conf/global.cfg.default. It will be similar to daemon.services=FilterService,WebDavService. Copy this line to $SEARCH_HOME/conf/global.cfg, but remove the WebDavService item from the comma delimited list, e.g.:

daemon.services=FilterService

After changing this setting the Funnelback daemon will need to be restarted.

Please note: disabling the Funnelback daemon webdav service will have no impact on the ability to access template and associated files via webdav for 15.20 or newer servers. The Jetty webserver is responsible for providing this (which just happens to also use the webdav protocol).

Ensure that the list of query processors contain the additional failover admin

Ensure the query processors and failover admin servers are listed as query processors in global.cfg, as per the original instructions to support multiple query processors:

query_processors=qp1.mycluster.com,qp2.mycluster.com,ad2.mycluster.com

This will ensure that new crawls and indexes are pushed to the failover admin servers whenever a collection updates, identically to what happens for every query processor.

Configure periodic sync between admin servers

The previous configuration will ensure that the data of each collection is updated on the failover admin whenever a collection updates. An additional script must be used to keep collection and server configuration in sync, so that configuration changes on the main admin server are reflected on the failover server.

The following folders need to be kept in sync between servers:

  • conf/ - Contains the global and collection configuration. Note: The global.cfg file must be excluded as it contains server-specific settings that will differ on the failover admin server
  • admin/ - Contains the user accounts
  • databases/ - Contains internal, embedded databases

The synchronization of folders can be done via any mean (e.g. rsync or lsyncd on Linux, or other means on Windows). An example Groovy script taking advantage of the Funnelback WebDAV facility is provided below to achieve this synchronization without other external tools.

Save the code below to $SEARCH_HOME/conf/sync_ad.groovy

import com.funnelback.common.config.*
import com.funnelback.mediator.core.webdav.*
import java.nio.file.*

/*
 * Command line tool to transfer arbitrary files to a remote Funnelback server
 * using the WebDAV protocol. It assumes that:
 *
 * - Both Funnelback instances share the same server_secret in global.cfg
 * - The WebDAV service is up and running on the remote side
 * - Access to the WebDAV port (usually 8076) has been opened in the firewall
 *
 * Only files from within the local Funnelback install can be transferred.
 * conf/global.cfg is always skipped.
 */

// Port of the remote WebDAV service
def final DEFAULT_WEBDAV_PORT = 8076;

// Setup command line options
def cli = new CliBuilder(usage: getClass().simpleName+' -h <remote_host> <file or folder> [file or folder] ...')
cli.with {
    header = "Transfer a file to a remote Funnelback server.\n" + "Both Funnnelback servers must have the same server_secret"
    h longOpt: "host", argName: 'hostname', args:1, type: String.class, "Remote host name"
    p longOpt: "port", argName: 'port', args:1, type: Number.class, "Remote host WebDAV port (Defaults to $DEFAULT_WEBDAV_PORT)"
    d longOpt: "delete", "Delete remote files that don't exist locally (Default: false)"
    s longOpt: "smart", "Do not transfer file that already exist remotely, based on the filesize (Default: false)"
    r longOpt: "recursive", "Transfer folders recursively (Default: false)"
    v longOpt: "verbose", "Verbose mode"
}

def options = cli.parse(args)
if (!options) return;
if (!options.h || !options.arguments()) {
    // No host name, or no file to transfer
    cli.usage()
    return;
}

def port = DEFAULT_WEBDAV_PORT
if (options.p) port = options.p

def fs = FileSystems.getDefault()
def searchHome = fs.getPath(new File(System.getenv("SEARCH_HOME")).absolutePath)

def config = new GlobalOnlyConfig(searchHome.toFile())               // Global config data, for the server_secret
def webdav = new FunnelbackSardine(config.value("server_secret"))    // WebDAV client, to push files
def urlHelper = new WebDAVUrlFactory(options.'host', port)           // Helper to build URL to the remote machine

// For each file or folder...
options.arguments().each {
    // The actual file
    def file = new File(it)
    // A path object makes it easer to manipulate
    def path = fs.getPath(file.absolutePath)

    if (! file.exists()) {
        println "Skipping '$path' as it doesn't exist"
    } else if (file =~ /global\.cfg$/) {
        println "Skipping '$path' as it's global.cfg"
    } else if (! path.startsWith(searchHome)) {
        println "Skipping '$path' as it doesn't belong to SEARCH_HOME ($searchHome)"
    } else {
        def targetUrl = urlHelper.root() + searchHome.relativize(path).toString().replace("\\", "/")
        println "Transferring '$path' to '$targetUrl'"

        if (file.isDirectory()) {
            def transferred;
            if (options.r) {
                transferred = webdav.recursiveSync(
                    targetUrl,
                    file,
                    options.d,
                    options.s);
            } else {
                transferred = webdav.sync(
                    targetUrl,
                    file,
                    options.d,
                    options.s);
            }

            // Display transferred files
            if (options.v) {
                println "${transferred.size} files transferred"

                transferred.each {
                    System.out << ((it.success) ? "OK   " : "FAIL ")
                    System.out << it.filename.padRight(40) << " "
                    System.out << ((it.success) ? "" : it.statusCode.toString())
                    System.out << ((it.errorMessage) ? " " + it.errorMessage : "")
                    System.out << "\n"
                }
            }
        } else {
            webdav.put(targetUrl, file);
        }
    }
}

Schedule a periodic sync

The above script (or alternate syncing process using rsync, lsyncd etc.) needs to be called on the 3 folders mentioned previously (conf/, admin/ and databases/). This can be achieved via the OS scheduler (crontab on Linux, Scheduled Tasks control panel on Windows). Running the script every 10min is recommended (depending how often the main admin server configuration changes).

The above script will need to be invoked via Groovy with the correct classpath and options:

  • -h: To specify the target hostname
  • -s: To enable a "smart" mode that will not re-transfer identical files
  • -d: To delete files on the failover admin that were deleted on the main admin server
  • -r: To recursively transfer all files
  • -v; Optional, for verbose mode

Example of a Linux wrapper script

The following bash script can be saved to $SEARCH_HOME/conf/sync_ad_wrapper.sh. Ensure that the script is made executable after it is saved.

#!/bin/bash

$SEARCH_HOME/linbin/java/bin/java -Djna.library.path=$SEARCH_HOME/bin -cp $SEARCH_HOME/lib/java/all/*:$SEARCH_HOME/lib/java/groovy -Xms32m -Xmx350m -Dfile.encoding=UTF-8 groovy.ui.GroovyMain $SEARCH_HOME/conf/sync_ad.groovy -h ad2.mycluster.com -s -d -v -r $SEARCH_HOME/conf $SEARCH_HOME/admin $SEARCH_HOME/databases

Example of a Windows wrapper script

The following batch file can be saved to \conf\sync_ad_wapper.bat

@echo off

REM Note the caret to prevent the shell from interpreting the star (*)
\tools\groovy\bin\groovy -cp \lib\java\all\^\* \conf\sync_ad.groovy -h ad2.mycluster.com -s -d -v -r \conf %SEARCH_HOME\admin %SEARCH_HOME\databases

Failover administration server

The following instructions cover server configuration for a failover administration server that provides limited failover services:

  • Read only access to server configuration
  • Access to analytics
  • Updates are disabled

Configure the failover administration server as per the official instructions for configuring a query processor server.

The following additional steps should be configured:

  1. Ensure the server shares the same server_secret as other servers in the cluster (as the other query processors - refer to the official instructions)
  2. Ensure the WebDAV port is configured correctly and accessible (default is port 8076)

  3. Configure the administration UI to operate in read only mode by editing global.cfg. This prevents changes being made on the failover administration server via the admin UI.

    admin.read-only-mode=true

  4. Disable scheduled tasks/cron jobs for Funnelback related updates (analytics and trend alerts updates)

  5. (optional) Configure load balancer rules as appropriate to failover users to this server.

Failover admin service to the hot standby server

The following instructions can be used to convert a failover admin server into the primary admin server:

If the primary admin server is still available the following tasks need to be performed on the main admin server to prevent it from continuing to run updates and push new configuration and data files to the failover admin and query processors:

One the primary admin server:

  1. Stop any running updates
  2. Disable scheduled tasks/cron jobs (Sync configuration, collection updates, report updates, trend alerts updates)
  3. Enable Admin UI read only mode
  4. Run the sync configuration scheduled task manually to ensure the failover admin has the latest configuration.
  5. Comment out the query_processors line in global.cfg to prevent any further pushing of files.

Perform the following operations on the failover admin server to convert it to the primary admin server:

  1. Disable Admin UI read only mode.
  2. Disable the WebDAV service (prevents the main admin server for erroneously pushing files to the failover one, while the failover has the service)
  3. Configure the query_processors line in global.cfg so that the failover admin server can push new configuration and data to them
  4. Schedule collection updates, analytics and trend alerts updates
  5. (optional) Setup and configure a sync configuration task to clone configuration to any other failover administration server.
Was this artcle helpful?
Tags
Type: Keywords:
Features:
Similar
See all Similar
Latest In this section
See all Latest In this section