Commit 58d5da8b authored by Markus Heiser's avatar Markus Heiser

nginx: normalize installation (docs and script)s over all distros

This is the revision of the documentation about the varous nginx installation
variants.  It also implements the nginx installation scripts for morty and
filtron.
Signed-off-by: default avatarMarkus Heiser <markus.heiser@darmarit.de>
parent 67afa6b1
......@@ -158,6 +158,12 @@ of:
Route request through filtron
=============================
.. sidebar:: further reading
- :ref:`filtron.sh overview`
- :ref:`installation nginx`
- :ref:`installation apache`
Filtron can be started using the following command:
.. code:: sh
......@@ -171,15 +177,24 @@ Use it along with ``nginx`` with the following example configuration.
.. code:: nginx
location / {
# https://example.org/searx
location /searx {
proxy_pass http://127.0.0.1:4004/;
proxy_set_header Host $http_host;
proxy_set_header Connection $http_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Script-Name /searx;
}
location /searx/static {
/usr/local/searx/searx-src/searx/static;
}
Requests are coming from port 4004 going through filtron and then forwarded to
port 8888 where a searx is being run. For a complete setup see: :ref:`nginx
searx site`.
......@@ -159,14 +159,22 @@ Started wiki`_ is always a good resource *to keep in the pocket*.
.. code:: nginx
location / {
proxy_pass http://127.0.0.1:4004/;
# https://example.org/searx
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
}
location /searx {
proxy_pass http://127.0.0.1:4004/;
proxy_set_header Host $http_host;
proxy_set_header Connection $http_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Script-Name /searx;
}
location /searx/static {
/usr/local/searx/searx-src/searx/static;
}
2. Configure reverse proxy for :ref:`morty <searx morty>`, listening on
......@@ -174,10 +182,13 @@ Started wiki`_ is always a good resource *to keep in the pocket*.
.. code:: nginx
location /morty {
# https://example.org/morty
location /morty {
proxy_pass http://127.0.0.1:3000/;
proxy_set_header Host $http_host;
proxy_set_header Connection $http_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
......@@ -197,7 +208,7 @@ Started wiki`_ is always a good resource *to keep in the pocket*.
image_proxy : True
.. group-tab:: proxy or uWSGI
.. group-tab:: proxy or uWSGI
Be warned, with this setup, your instance isn't :ref:`protected <searx
filtron>`. Nevertheless it is good enough for intranet usage and it is a
......@@ -208,15 +219,17 @@ Started wiki`_ is always a good resource *to keep in the pocket*.
.. code:: nginx
location / {
proxy_pass http://127.0.0.1:8888;
# https://hostname.local/
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Script-Name /searx;
proxy_buffering off;
}
location / {
proxy_pass http://127.0.0.1:8888;
proxy_set_header Host $host;
proxy_set_header Connection $http_connection;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_buffering off;
}
Alternatively you can use the `uWSGI support from nginx`_ via unix
sockets. For socket communication, you have to activate ``socket =
......@@ -234,8 +247,8 @@ Started wiki`_ is always a good resource *to keep in the pocket*.
.. code:: nginx
server {
# replace example.org with your server's public name
server_name example.org;
# replace hostname.local with your server's name
server_name hostname.local;
listen 80;
listen [::]:80;
......@@ -245,7 +258,7 @@ Started wiki`_ is always a good resource *to keep in the pocket*.
uwsgi_pass unix:/run/uwsgi/app/searx/socket;
}
root /usr/local/searx/searx-src/searx;
root /usr/local/searx/searx-src/searx;
location /static { }
}
......@@ -254,10 +267,10 @@ Started wiki`_ is always a good resource *to keep in the pocket*.
.. code:: bash
mkdir -p /run/uwsgi/app/searx/
sudo -H chown -R searx:searx /run/uwsgi/app/searx/
mkdir -p /run/uwsgi/app/searx/
sudo -H chown -R searx:searx /run/uwsgi/app/searx/
.. group-tab:: proxy at subdir URL
.. group-tab:: \.\. at subdir URL
Be warned, with these setups, your instance isn't :ref:`protected <searx
filtron>`. The examples are just here to demonstrate how to export the
......@@ -265,19 +278,22 @@ Started wiki`_ is always a good resource *to keep in the pocket*.
.. code:: nginx
location /searx {
proxy_pass http://127.0.0.1:8888;
# https://hostname.local/searx
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Script-Name /searx;
proxy_buffering off;
}
location /searx {
proxy_pass http://127.0.0.1:8888;
proxy_set_header Host $host;
proxy_set_header Connection $http_connection;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Script-Name /searx;
proxy_buffering off;
}
location /searx/static {
alias /usr/local/searx/searx-src/searx/static;
}
location /searx/static {
alias /usr/local/searx/searx-src/searx/static;
}
The ``X-Script-Name /searx`` is needed by the searx implementation to
calculate relative URLs correct. The next example shows a uWSGI
......@@ -286,15 +302,17 @@ Started wiki`_ is always a good resource *to keep in the pocket*.
.. code:: nginx
location /searx/static {
alias /usr/local/searx/searx-src/searx;
}
# https://hostname.local/searx
location /searx {
uwsgi_param SCRIPT_NAME /searx;
include uwsgi_params;
uwsgi_pass unix:/run/uwsgi/app/searx/socket;
}
location /searx {
uwsgi_param SCRIPT_NAME /searx;
include uwsgi_params;
uwsgi_pass unix:/run/uwsgi/app/searx/socket;
}
location /searx/static {
alias /usr/local/searx/searx-src/searx;
}
For searx to work correctly the ``base_url`` must be set in the
:origin:`searx/settings.yml`.
......
......@@ -44,9 +44,8 @@ GO_ENV="${SERVICE_HOME}/.go_env"
GO_PKG_URL="https://dl.google.com/go/go1.13.5.linux-amd64.tar.gz"
GO_TAR=$(basename "$GO_PKG_URL")
# Apache Settings
APACHE_FILTRON_SITE="searx.conf"
NGINX_FILTRON_SITE="searx.conf"
# shellcheck disable=SC2034
CONFIG_FILES=(
......@@ -60,9 +59,7 @@ usage() {
# shellcheck disable=SC1117
cat <<EOF
usage::
$(basename "$0") shell
$(basename "$0") install [all|user|rules]
$(basename "$0") update [filtron]
......@@ -72,6 +69,7 @@ usage::
$(basename "$0") inspect [service]
$(basename "$0") option [debug-on|debug-off]
$(basename "$0") apache [install|remove]
$(basename "$0") nginx [install|remove]
shell
start interactive shell from user ${SERVICE_USER}
......@@ -92,6 +90,9 @@ option
apache (${PUBLIC_URL})
:install: apache site with a reverse proxy (ProxyPass)
:remove: apache site ${APACHE_FILTRON_SITE}
nginx (${PUBLIC_URL})
:install: nginx site with a reverse proxy (ProxyPass)
:remove: nginx site ${NGINX_FILTRON_SITE}
filtron rules: ${FILTRON_RULES}
......@@ -118,8 +119,6 @@ EOF
}
main() {
rst_title "$SERVICE_NAME" part
required_commands \
sudo install git wget curl \
|| exit
......@@ -127,7 +126,7 @@ main() {
local _usage="unknown or missing $1 command $2"
case $1 in
--source-only) ;;
--getenv) var="$2"; echo "${!var}"; exit 0;;
-h|--help) usage; exit 0;;
shell)
......@@ -143,6 +142,7 @@ main() {
*) usage "$_usage"; exit 42;;
esac ;;
install)
rst_title "$SERVICE_NAME" part
sudo_or_exit
case $2 in
all) install_all ;;
......@@ -187,6 +187,13 @@ main() {
remove) remove_apache_site ;;
*) usage "$_usage"; exit 42;;
esac ;;
nginx)
sudo_or_exit
case $2 in
install) install_nginx_site ;;
remove) remove_nginx_site ;;
*) usage "$_usage"; exit 42;;
esac ;;
option)
sudo_or_exit
case $2 in
......@@ -473,6 +480,51 @@ This removes apache site ${APACHE_FILTRON_SITE}."
}
install_nginx_site() {
rst_title "Install nginx site $NGINX_FILTRON_SITE"
rst_para "\
This installs a reverse proxy (ProxyPass) into nginx site (${NGINX_FILTRON_SITE})"
! nginx_is_installed && err_msg "nginx is not installed."
if ! ask_yn "Do you really want to continue?" Yn; then
return
else
install_nginx
fi
"${REPO_ROOT}/utils/searx.sh" install uwsgi
SEARX_SRC=$("${REPO_ROOT}/utils/searx.sh" --getenv SEARX_SRC)
SEARX_URL_PATH=$("${REPO_ROOT}/utils/searx.sh" --getenv SEARX_URL_PATH)
nginx_install_app --variant=filtron "${NGINX_FILTRON_SITE}"
info_msg "testing public url .."
if ! service_is_available "${PUBLIC_URL}"; then
err_msg "Public service at ${PUBLIC_URL} is not available!"
fi
}
remove_nginx_site() {
rst_title "Remove nginx site $NGINX_FILTRON_SITE"
rst_para "\
This removes nginx site ${NGINX_FILTRON_SITE}."
! nginx_is_installed && err_msg "nginx is not installed."
if ! ask_yn "Do you really want to continue?" Yn; then
return
fi
nginx_remove_site "$FILTRON_FILTRON_SITE"
}
rst-doc() {
eval "echo \"$(< "${REPO_ROOT}/docs/build-templates/filtron.rst")\""
......
......@@ -138,6 +138,16 @@ rst_para() {
fi
}
die() {
echo -e "${_BRed}ERROR:${_creset} ${BASH_SOURCE[1]}: line ${BASH_LINENO[0]}: ${2-died ${1-1}}" >&2;
exit "${1-1}"
}
die_caller() {
echo -e "${_BRed}ERROR:${_creset} ${BASH_SOURCE[2]}: line ${BASH_LINENO[1]}: ${FUNCNAME[1]}(): ${2-died ${1-1}}" >&2;
exit "${1-1}"
}
err_msg() { echo -e "${_BRed}ERROR:${_creset} $*" >&2; }
warn_msg() { echo -e "${_BBlue}WARN:${_creset} $*" >&2; }
info_msg() { echo -e "${_BYellow}INFO:${_creset} $*" >&2; }
......@@ -471,11 +481,7 @@ service_is_available() {
# usage: service_is_available <URL>
local URL="$1"
if [[ -z $URL ]]; then
err_msg "service_is_available: missing arguments"
return 42
fi
[[ -z $1 ]] && die_caller 42 "missing argument <URL>"
http_code=$(curl -H 'Cache-Control: no-cache' \
--silent -o /dev/null --head --write-out '%{http_code}' --insecure \
......@@ -624,6 +630,175 @@ EOF
}
# nginx
# -----
nginx_distro_setup() {
# shellcheck disable=SC2034
NGINX_DEFAULT_SERVER=/etc/nginx/nginx.conf
# Including *location* directives from a dedicated config-folder into the
# server directive is, what what fedora (already) does.
NGINX_APPS_ENABLED="/etc/nginx/default.d"
# We add a apps-available folder and linking configurations into the
# NGINX_APPS_ENABLED folder. See also nginx_include_apps_enabled().
NGINX_APPS_AVAILABLE="/etc/nginx/default.apps-available"
case $DIST_ID-$DIST_VERS in
ubuntu-*|debian-*)
NGINX_PACKAGES="nginx"
NGINX_DEFAULT_SERVER=/etc/nginx/sites-available/default
;;
arch-*)
NGINX_PACKAGES="nginx-mainline"
;;
fedora-*)
NGINX_PACKAGES="nginx"
;;
*)
err_msg "$DIST_ID-$DIST_VERS: nginx not yet implemented"
;;
esac
}
nginx_distro_setup
install_nginx(){
info_msg "installing nginx ..."
pkg_install "${NGINX_PACKAGES}"
case $DIST_ID-$DIST_VERS in
arch-*|fedora-*)
systemctl enable nginx
systemctl start nginx
;;
esac
}
nginx_is_installed() {
command -v nginx &>/dev/null
}
nginx_reload() {
info_msg "reload nginx .."
echo
if ! nginx -t; then
err_msg "testing nginx configuration failed"
return 42
fi
systemctl restart nginx
}
nginx_install_app() {
# usage: nginx_install_app [<template option> ...] <myapp>
#
# <template option>: see install_template
local template_opts=()
local pos_args=("$0")
for i in "$@"; do
case $i in
-*) template_opts+=("$i");;
*) pos_args+=("$i");;
esac
done
nginx_include_apps_enabled "${NGINX_DEFAULT_SERVER}"
install_template "${template_opts[@]}" \
"${NGINX_APPS_AVAILABLE}/${pos_args[1]}" \
root root 644
nginx_enable_app "${pos_args[1]}"
info_msg "installed nginx app: ${pos_args[1]}"
}
nginx_include_apps_enabled() {
# Add the *NGINX_APPS_ENABLED* infrastruture to a nginx server block. Such
# infrastruture is already known from fedora, including apps (location
# directives) from the /etc/nginx/default.d folder into the *default* nginx
# server.
# usage: nginx_include_apps_enabled <config-file>
#
# config-file: Config file with server directive in.
[[ -z $1 ]] && die_caller 42 "missing argument <config-file>"
local server_conf="$1"
# include /etc/nginx/default.d/*.conf;
local include_directive="include ${NGINX_APPS_ENABLED}/*.conf;"
local include_directive_re="^\s*include ${NGINX_APPS_ENABLED}/\*\.conf;"
info_msg "checking existence: '${include_directive}' in file ${server_conf}"
if grep "${include_directive_re}" "${server_conf}"; then
info_msg "OK, already exists."
return
fi
info_msg "add missing directive: '${include_directive}'"
cp "${server_conf}" "${server_conf}.bak"
(
local line
local stage=0
while IFS= read -r line
do
echo "$line"
if [[ $stage = 0 ]]; then
if [[ $line =~ ^[[:space:]]*server*[[:space:]]*\{ ]]; then
stage=1
fi
fi
if [[ $stage = 1 ]]; then
echo " # Load configuration files for the default server block."
echo " $include_directive"
echo ""
stage=2
fi
done < "${server_conf}.bak"
) > "${server_conf}"
}
nginx_remove_app() {
# usage: nginx_remove_app <myapp.conf>
info_msg "remove nginx app: $1"
nginx_dissable_app "$1"
rm -f "${NGINX_APPS_AVAILABLE}/$1"
}
nginx_enable_app() {
# usage: nginx_enable_app <myapp.conf>
local CONF="$1"
info_msg "enable nginx app: ${CONF}"
mkdir -p "${NGINX_APPS_ENABLED}"
rm -f "${NGINX_APPS_ENABLED}/${CONF}"
ln -s "${NGINX_APPS_AVAILABLE}/${CONF}" "${NGINX_APPS_ENABLED}/${CONF}"
nginx_reload
}
nginx_dissable_app() {
# usage: nginx_disable_app <myapp.conf>
local CONF="$1"
info_msg "disable nginx app: ${CONF}"
rm -f "${NGINX_APPS_ENABLED}/${CONF}"
nginx_reload
}
# Apache
# ------
......
......@@ -73,9 +73,7 @@ usage() {
# ----------------------------------------------------------------------------
_cmd="$(basename "$0")"
cat <<EOF
usage::
$_cmd build [containers|<name>]
$_cmd copy [images]
$_cmd remove [containers|<name>|images]
......@@ -101,7 +99,7 @@ show
:suite: show services of all (or <name>) containers from the LXC suite
:images: show information of local images
cmd
use single qoutes to evaluate in container's bash, e.g. 'echo $(hostname)'
use single qoutes to evaluate in container's bash, e.g.: 'echo \$(hostname)'
-- run command '...' in all containers of the LXC suite
:<name>: run command '...' in container <name>
install
......@@ -151,14 +149,14 @@ main() {
fi
case $1 in
--source-only) ;;
--getenv) var="$2"; echo "${!var}"; exit 0;;
-h|--help) usage; exit 0;;
build)
sudo_or_exit
case $2 in
${LXC_HOST_PREFIX}-*) build_container "$2" ;;
''|containers) build_all_containers ;;
''|--|containers) build_all_containers ;;
*) usage "$_usage"; exit 42;;
esac
;;
......@@ -171,7 +169,7 @@ main() {
remove)
sudo_or_exit
case $2 in
''|containers) remove_containers ;;
''|--|containers) remove_containers ;;
images) lxc_delete_images_localy ;;
${LXC_HOST_PREFIX}-*)
! lxc_exists "$2" && usage_containers "unknown container: $2" && exit 42
......@@ -185,7 +183,7 @@ main() {
start|stop)
sudo_or_exit
case $2 in
''|containers) lxc_cmd "$1" ;;
''|--|containers) lxc_cmd "$1" ;;
${LXC_HOST_PREFIX}-*)
! lxc_exists "$2" && usage_containers "unknown container: $2" && exit 42
info_msg "lxc $1 $2"
......@@ -203,7 +201,7 @@ main() {
lxc exec -t "$3" -- "${LXC_REPO_ROOT}/utils/lxc.sh" __show suite \
| prefix_stdout "[${_BBlue}$3${_creset}] "
;;
*) show_suite;;
*|--) show_suite;;
esac
;;
images) show_images ;;
......@@ -213,7 +211,7 @@ main() {
! lxc_exists "$3" && usage_containers "unknown container: $3" && exit 42
lxc config show "$3" | prefix_stdout "[${_BBlue}${3}${_creset}] "
;;
*)
*|--)
rst_title "container configurations"
echo
lxc list "$LXC_HOST_PREFIX-"
......@@ -228,7 +226,7 @@ main() {
! lxc_exists "$3" && usage_containers "unknown container: $3" && exit 42
lxc info "$3" | prefix_stdout "[${_BBlue}${3}${_creset}] "
;;
*)
*|--)
rst_title "container info"
echo
lxc_cmd info
......@@ -267,7 +265,7 @@ main() {
! lxc_exists "$3" && usage_containers "unknown container: $3" && exit 42
lxc_exec_cmd "$3" "${LXC_REPO_ROOT}/utils/lxc.sh" __install "$2"
;;
'') lxc_exec "${LXC_REPO_ROOT}/utils/lxc.sh" __install "$2" ;;
''|--) lxc_exec "${LXC_REPO_ROOT}/utils/lxc.sh" __install "$2" ;;
*) usage_containers "unknown container: $3" && exit 42
esac
;;
......
......@@ -47,6 +47,7 @@ CONFIG_FILES=()
# Apache Settings
APACHE_MORTY_SITE="morty.conf"
NGINX_MORTY_SITE="morty.conf"
# ----------------------------------------------------------------------------
usage() {
......@@ -54,9 +55,7 @@ usage() {
# shellcheck disable=SC1117
cat <<EOF
usage::
$(basename "$0") shell
$(basename "$0") install [all|user]
$(basename "$0") update [morty]
......@@ -66,6 +65,7 @@ usage::
$(basename "$0") inspect [service]
$(basename "$0") option [debug-on|debug-off]
$(basename "$0") apache [install|remove]
$(basename "$0") nginx [install|remove]
$(basename "$0") info [searx]
shell
......@@ -86,6 +86,9 @@ option
apache : ${PUBLIC_URL_MORTY}
:install: apache site with a reverse proxy (ProxyPass)
:remove: apache site ${APACHE_MORTY_SITE}
nginx (${PUBLIC_URL_MORTY})
:install: nginx site with a reverse proxy (ProxyPass)
:remove: nginx site ${NGINX_MORTY_SITE}
If needed, set the environment variables in the '${DOT_CONFIG#"$REPO_ROOT/"}' file::
PUBLIC_URL_MORTY: ${PUBLIC_URL_MORTY}
......@@ -122,8 +125,6 @@ EOF
}
main() {
rst_title "$SERVICE_NAME" part
required_commands \
sudo install git wget curl \
|| exit
......@@ -131,7 +132,7 @@ main() {
local _usage="ERROR: unknown or missing $1 command $2"
case $1 in
--source-only) ;;
--getenv) var="$2"; echo "${!var}"; exit 0;;
-h|--help) usage; exit 0;;
shell)
......@@ -147,6 +148,7 @@ main() {
*) usage "$_usage"; exit 42;;
esac ;;
install)
rst_title "$SERVICE_NAME" part
sudo_or_exit
case $2 in
all) install_all ;;
......@@ -185,6 +187,13 @@ main() {
remove) remove_apache_site ;;
*) usage "$_usage"; exit 42;;
esac ;;
nginx)
sudo_or_exit
case $2 in
install) install_nginx_site ;;
remove) remove_nginx_site ;;
*) usage "$_usage"; exit 42;;
esac ;;
info)
case $2 in
searx) info_searx ;;
......@@ -348,8 +357,8 @@ EOF
wait_key
fi
if ! service_is_available "${PUBLIC_URL}"; then
warn_msg "Public service at ${PUBLIC_URL} is not available!"
if ! service_is_available "${PUBLIC_URL_MORTY}"; then
warn_msg "Public service at ${PUBLIC_URL_MORTY} is not available!"
if ! in_container; then
warn_msg "Check if public name is correct and routed or use the public IP from above."
fi
......@@ -363,7 +372,7 @@ EOF
fi
local _debug_on
if ask_yn "Enable filtron debug mode (needs reinstall of systemd service)?"; then
if ask_yn "Enable morty debug mode (needs reinstall of systemd service)?"; then
enable_debug
_debug_on=1
else
......@@ -436,6 +445,50 @@ This removes apache site ${APACHE_MORTY_SITE}."
apache_remove_site "$APACHE_MORTY_SITE"
}
install_nginx_site() {
rst_title "Install nginx site $NGINX_MORTY_SITE"
rst_para "\
This installs a reverse proxy (ProxyPass) into nginx site (${NGINX_MORTY_SITE})"
! nginx_is_installed && err_msg "nginx is not installed."
if ! ask_yn "Do you really want to continue?" Yn; then
return
else
install_nginx
fi
"${REPO_ROOT}/utils/searx.sh" install uwsgi
SEARX_SRC=$("${REPO_ROOT}/utils/searx.sh" --getenv SEARX_SRC)
SEARX_URL_PATH=$("${REPO_ROOT}/utils/searx.sh" --getenv SEARX_URL_PATH)
nginx_install_app "${NGINX_MORTY_SITE}"
info_msg "testing public url .."
if ! service_is_available "${PUBLIC_URL_MORTY}"; then
err_msg "Public service at ${PUBLIC_URL_MORTY} is not available!"
fi
}
remove_nginx_site() {
rst_title "Remove nginx site $NGINX_MORTY_SITE"
rst_para "\