Download configuration files via workflow

Managed by | Updated 5 years ago.

Background

There is often a need to download configuration files as part of the Funnelback workflow. This can involve downloading an external file (such as an external_metdata.cfg from Squiz Matrix), or accessing Funnelback to produce a configuration file such as auto-completion.csv.

Procedure

curl is the recommended program to use when downloading via workflow. Please note this is preferred to using wget or other custom perl or python scripts that use internal libraries.

The following curl command should be used when downloading as part of workflow. This command sets a longer timeout as well as retries, and exits if there is an error.

# Linux - use single quotes
curl --connect-timeout 60 --retry 3 --retry-delay 20 '<URL_TO_DOWNLOAD>' -o <OUTPUT_FILE> || exit 1

# Windows - use double quotes (requires cygwin)
c:\cygwin\bin\curl.exe --connect-timeout 60 --retry 3 --retry-delay 20 "<URL_TO_DOWNLOAD>" -o <OUTPUT_FILE> || exit 1
# Windows - with wget.exe  Note: only use this if curl is unavailable, or you are having problems with the curl commend
c:\funnelback\wbin\wget.exe -T 60 -t 3 -w 20 "<URL_TO_DOWNLOAD>" -O <OUTPUT_FILE> || exit 1

Note:

• Funnelback under Windows ships with a wget binary that can be called - it's located at \winbin\wget.exe
• If required under Windows curl can be used by installing Cygwin and using the curl binary that is part of that product (note the native Windows/DOS version of curl is too old and doesn't support a lot of options).
• curl support various forms of authentication (eg. http. Windows integrated) which is sometimes required when downloading files.
• When downloading from a local Funnelback instance use the localhost address e.g. http://localhost/s/search.html?collection=COLLECTION&query=QUERY (as opposed to a fully qualified address such as http://FUNNELBACK-SERVER.com/s/search.html?collection=COLLECTION&query=QUERY. Note the ports should be adjusted to be whatever the Funnelback http port is.
• If you are downloading external metadata please make use of the external metadata validator that is built in to Mediator.

• Wget command is equivalent to curl, but curl is preferred for consistency (and also because has options for session cookies etc). Some versions of cygwin have a bug with curl.exe that results in errors like:

  cygwin curl.exe: *** fatal error - couldn't initialize fd 0 /dev/cons0
  

  If this is happening then fall back to the wget command.

Example

#Download external metadata from Matrix
curl --connect-timeout 60 --retry 3 --retry-delay 20 'http://matrixsite.com/resources/external_metadata' -o $SEARCH_HOME/conf/matrixsite/external_metadata.cfg || exit 1

#Download query completion CSV from Funnelback
curl --connect-timeout 60 --retry 3 --retry-delay 20 'http://localhost:8080/s/search.html?collection=myfaqs&query=!showall&num_ranks=1000&form=query_completion' -o $SEARCH_HOME/conf/myfaqs/query_completion.csv || exit 1