Loading docs/admin/installation-uwsgi.rst +6 −5 Original line number Original line Diff line number Diff line Loading @@ -108,6 +108,7 @@ restart the uwsgi application. :start-after: START searx uwsgi-description ubuntu-20.04 :start-after: START searx uwsgi-description ubuntu-20.04 :end-before: END searx uwsgi-description ubuntu-20.04 :end-before: END searx uwsgi-description ubuntu-20.04 .. hotfix: a bug group-tab need this comment .. group-tab:: Arch Linux .. group-tab:: Arch Linux Loading @@ -115,6 +116,7 @@ restart the uwsgi application. :start-after: START searx uwsgi-description arch :start-after: START searx uwsgi-description arch :end-before: END searx uwsgi-description arch :end-before: END searx uwsgi-description arch .. hotfix: a bug group-tab need this comment .. group-tab:: Fedora / RHEL .. group-tab:: Fedora / RHEL Loading @@ -128,22 +130,21 @@ restart the uwsgi application. .. group-tab:: Ubuntu / debian .. group-tab:: Ubuntu / debian .. kernel-include:: $DOCS_BUILD/includes/searx.rst .. kernel-include:: $DOCS_BUILD/includes/searx.rst :code: ini :start-after: START searx uwsgi-appini ubuntu-20.04 :start-after: START searx uwsgi-appini ubuntu-20.04 :end-before: END searx uwsgi-appini ubuntu-20.04 :end-before: END searx uwsgi-appini ubuntu-20.04 .. hotfix: a bug group-tab need this comment .. group-tab:: Arch Linux .. group-tab:: Arch Linux .. kernel-include:: $DOCS_BUILD/includes/searx.rst .. kernel-include:: $DOCS_BUILD/includes/searx.rst :code: ini :start-after: START searx uwsgi-appini arch :start-after: START searx uwsgi-appini arch :end-before: END searx uwsgi-appini arch :end-before: END searx uwsgi-appini arch .. hotfix: a bug group-tab need this comment .. group-tab:: Fedora / RHEL .. group-tab:: Fedora / RHEL .. kernel-include:: $DOCS_BUILD/includes/searx.rst .. kernel-include:: $DOCS_BUILD/includes/searx.rst :code: ini :start-after: START searx uwsgi-appini fedora :start-after: START searx uwsgi-appini fedora :end-before: END searx uwsgi-appini fedora :end-before: END searx uwsgi-appini fedora docs/blog/index.rst +1 −0 Original line number Original line Diff line number Diff line Loading @@ -6,6 +6,7 @@ Blog :maxdepth: 2 :maxdepth: 2 :caption: Contents :caption: Contents lxcdev-202006 python3 python3 admin admin intro-offline intro-offline Loading docs/blog/lxcdev-202006.rst 0 → 100644 +413 −0 Original line number Original line Diff line number Diff line .. _blog-lxcdev-202006: ======================================= Developing in Linux containers [202006] ======================================= .. _LXC: https://linuxcontainers.org/lxc/introduction/ .. sidebar:: Audience This blog post is written for experienced admins and developers / readers should have a serious meaning about: *distributed*, *merge* and *linux container*. .. contents:: Contents :depth: 2 :local: :backlinks: entry In PR :PR:`1803` we added a lot of scripts to Searx's boilerplate. In this blog post I will show you, how you can make use of them in *distributed and heterogeneous development cycles* (TL;DR; jump to the :ref:`blog-lxcdev-202006 abstract`). Motivation ========== Normally in our development cycle, we edit the sources and run some test and/or builds by using ``make`` before we commit. This cycle is simple and perfect but might fail in some aspects we should not overlook. The environment in which we run all our development processes matters! The :ref:`makefile` and the :ref:`make pyenv` encapsulate a lot for us, but they do not have access to all prerequisites. For example, there may have dependencies on packages that are installed on the developer's desktop, but usually are not preinstalled on a server or client system. Another examples are; settings have been made to the software on the developer's host that would never be set on a *production* system. *Linux Containers* (LXC_) are isolate environments and not to mix up on developer's all the prerequisites of all the projects he contribute to, is always a good choice. The scripts from PR :PR:`1803` can divide in those to install and maintain software: - :ref:`searx.sh` - :ref:`filtron.sh` - :ref:`morty.sh` and the script :ref:`lxc.sh`, with we can scale our installation, maintenance or even development tasks over a stack of containers, what we call: *Searx's lxc suite*. Gentlemen, start your engines! ============================== .. _LXD: https://linuxcontainers.org/lxd/introduction/ .. _archlinux: https://www.archlinux.org/ Before you can start with containers, you need to install and initiate LXD_ once: .. tabs:: .. group-tab:: desktop .. code:: sh $ snap install lxd $ lxd init --auto And you need to clone from origin or if you have your own fork, clone from your fork: .. tabs:: .. group-tab:: desktop .. code:: sh $ cd ~/Downloads $ git clone https://github.com/asciimoo/searx.git $ cd searx The :ref:`lxc-searx.env` consists of several images, see ``export LXC_SUITE=(...`` near by :origin:`utils/lxc-searx.env#L19`. For this blog post we exercise on a archlinux_ image. The container of this image is named ``searx-archlinux``. Lets build the container, but be sure that this container does not already exists, so first lets remove possible old one: .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh remove searx-archlinux $ sudo -H ./utils/lxc.sh build searx-archlinux .. sidebar:: The ``searx-archlinux`` container is the base of all our exercises here. In this container we install all services :ref:`including searx, morty & filtron <lxc.sh install suite>` in once: .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh install suite searx-archlinux To proxy HTTP from filtron and morty in the container to the outside of the container, install nginx into the container. Once for the bot blocker filtron: .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ ./utils/filtron.sh nginx install ... INFO: got 429 from http://10.174.184.156/searx and once for the content sanitizer (content proxy morty): .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ ./utils/morty.sh nginx install ... INFO: got 200 from http://10.174.184.156/morty/ .. sidebar:: Fully functional searx suite From here on you have a fully functional searx suite running with bot blocker (filtron) and Web content sanitizer (content proxy morty) needed for a *privacy protecting* search engine. On your system, the IP of your ``searx-archlinux`` container differs from http://10.174.184.156/searx, just open the URL reported in your installation protocol in your WEB browser from the desktop to test the instance from outside of the container. In such a searx suite admins can maintain and access the debug log of the different services quite easy. .. _working in containers: In containers, work as usual ============================ Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers open the root-bash in the container using ``./utils/lxc.sh cmd searx-archlinux``: .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux bash INFO: [searx-archlinux] bash [root@searx-archlinux searx]# pwd /share/searx The prompt ``[root@searx-archlinux ...]`` signals, that you are the root user in the searx-container. To debug the running searx instance use: .. tabs:: .. group-tab:: root@searx-archlinux .. code:: sh $ ./utils/searx.sh inspect service ... use [CTRL-C] to stop monitoring the log ... Back in the browser on your desktop open the service http://10.174.184.156/searx and run your application tests while the debug log is shown in the terminal from above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug option"* in searx's settings file and restarts the searx uwsgi application. To debug services from filtron and morty analogous use: .. tabs:: .. group-tab:: root@searx-archlinux .. code:: sh $ ./utils/filtron.sh inspect service $ ./utils/morty.sh inspect service Another point we have to notice is that each service (:ref:`searx <searx.sh>`, :ref:`filtron <filtron.sh>` and :ref:`morty <morty.sh>`) runs under dedicated system user account with the same name (compare :ref:`create searx user`). To get a shell from theses accounts, simply call one of the scripts: .. tabs:: .. group-tab:: root@searx-archlinux .. code:: sh $ ./utils/searx.sh shell $ ./utils/filtron.sh shell $ ./utils/morty.sh shell To get in touch, open a shell from the service user (searx@searx-archlinux): .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ ./utils/searx.sh shell // exit with [CTRL-D] (searx-pyenv) [searx@searx-archlinux ~]$ ... The prompt ``[searx@searx-archlinux]`` signals that you are logged in as system user ``searx`` in the ``searx-archlinux`` container and the python *virtualenv* ``(searx-pyenv)`` environment is activated. .. tabs:: .. group-tab:: searx@searx-archlinux .. code:: sh (searx-pyenv) [searx@searx-archlinux ~]$ pwd /usr/local/searx Wrap production into developer suite ==================================== In this section we will see how to change the *"Fully functional searx suite"* from a LXC container (which is quite ready for production) into a developer suite. For this, we have to keep an eye on the :ref:`installation basic`: - searx setup in: ``/etc/searx/settings.yml`` - searx user's home: ``/usr/local/searx`` - virtualenv in: ``/usr/local/searx/searx-pyenv`` - searx software in: ``/usr/local/searx/searx-src`` The searx software is a clone of the ``GIT_URL`` (see :ref:`makefile setup`) and the working tree is checked out from the ``GIT_BRANCH``. With the use of the :ref:`searx.sh` the searx service was installed as :ref:`uWSGI application <searx uwsgi>`. To maintain this service, we can use ``systemctl`` (compare :ref:`service architectures on distributions <uwsgi configuration>`). .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ systemctl stop uwsgi@searx With the command above, we stopped the searx uWSGI-App in the archlinux container. The uWSGI-App for the archlinux dsitros is configured in :origin:`utils/templates/etc/uwsgi/apps-archlinux/searx.ini`, from where at least you should attend the settings of ``uid``, ``chdir``, ``env`` and ``http``:: env = SEARX_SETTINGS_PATH=/etc/searx/settings.yml http = 127.0.0.1:8888 chdir = /usr/local/searx/searx-src/searx virtualenv = /usr/local/searx/searx-pyenv pythonpath = /usr/local/searx/searx-src If you have read the :ref:`"Good to know section" <lxc.sh>` you remember, that each container shares the root folder of the repository and the command ``utils/lxc.sh cmd`` handles relative path names **transparent**. To wrap the searx installation into a developer one, we simple have to create a smylink to the **transparent** reposetory from the desktop. Now lets replace the repository at ``searx-src`` in the container with the working tree from outside of the container: .. tabs:: .. group-tab:: container becomes a developer suite .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ ln -s /share/searx/ /usr/local/searx/searx-src Now we can develop as usual in the working tree of our desktop system. Every time the software was changed, you have to restart the searx service (in the conatiner): .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ systemctl restart uwsgi@searx Remember: :ref:`working in containers` .. here are just some examples from my daily usage: .. tabs:: .. group-tab:: desktop To *inspect* the searx instance (already described above): .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ ./utils/searx.sh inspect service Run :ref:`makefile`, e.g. to test inside the container: .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ make test To install all prerequisites needed for a :ref:`buildhosts`: .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ ./utils/searx.sh install buildhost To build the docs on a buildhost :ref:`buildhosts`: .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ make docs .. _blog-lxcdev-202006 abstract: Abstract ======== We build up a fully functional searx suite in a archlinux container: .. code:: sh $ sudo -H ./utils/lxc.sh install suite searx-archlinux To access HTTP from the desktop we installed nginx for the services inside the conatiner: .. tabs:: .. group-tab:: [root@searx-archlinux] .. code:: sh $ ./utils/filtron.sh nginx install $ ./utils/morty.sh nginx install To wrap the suite into a developer one, we created a symbolic link to the repository which is shared **transparent** from the desktop's file system into the container : .. tabs:: .. group-tab:: [root@searx-archlinux] .. code:: sh $ mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old $ ln -s /share/searx/ /usr/local/searx/searx-src $ systemctl restart uwsgi@searx To get remarks from the suite of the archlinux container we can use: .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh show suite searx-archlinux ... [searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx [searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/ [searx-archlinux] INFO: (eth0) docs-live: http://10.174.184.156:8080/ [searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65] ... docs/utils/lxc.sh.rst +4 −3 Original line number Original line Diff line number Diff line Loading @@ -47,9 +47,9 @@ one**:: *Good to know ...* *Good to know ...* Eeach container shares the root folder of the repository and the Each container shares the root folder of the repository and the command command ``utils/lxc.sh cmd`` **handles relative path names transparent**, ``utils/lxc.sh cmd`` **handles relative path names transparent**, compare output compare output of:: of:: $ sudo -H ./utils/lxc.sh cmd -- ls -la Makefile $ sudo -H ./utils/lxc.sh cmd -- ls -la Makefile ... ... Loading @@ -66,6 +66,7 @@ If there comes the time you want to **get rid off all** the containers and $ sudo -H ./utils/lxc.sh remove $ sudo -H ./utils/lxc.sh remove $ sudo -H ./utils/lxc.sh remove images $ sudo -H ./utils/lxc.sh remove images .. _lxc.sh install suite: Install suite Install suite ============= ============= Loading utils/searx.sh +42 −28 Original line number Original line Diff line number Diff line Loading @@ -816,6 +816,9 @@ rst-doc() { case $DIST_ID-$DIST_VERS in case $DIST_ID-$DIST_VERS in ubuntu-*|debian-*) cat <<EOF ubuntu-*|debian-*) cat <<EOF .. code:: bash # init.d --> /usr/share/doc/uwsgi/README.Debian.gz # init.d --> /usr/share/doc/uwsgi/README.Debian.gz # For uWSGI debian uses the LSB init process, this might be changed # For uWSGI debian uses the LSB init process, this might be changed # one day, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833067 # one day, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833067 Loading @@ -826,9 +829,13 @@ start: sudo -H service uwsgi start ${SEARX_UWSGI_APP%.*} restart: sudo -H service uwsgi restart ${SEARX_UWSGI_APP%.*} restart: sudo -H service uwsgi restart ${SEARX_UWSGI_APP%.*} stop: sudo -H service uwsgi stop ${SEARX_UWSGI_APP%.*} stop: sudo -H service uwsgi stop ${SEARX_UWSGI_APP%.*} disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} EOF EOF ;; ;; arch-*) cat <<EOF arch-*) cat <<EOF .. code:: bash # systemd --> /usr/lib/systemd/system/uwsgi@.service # systemd --> /usr/lib/systemd/system/uwsgi@.service # For uWSGI archlinux uses systemd template units, see # For uWSGI archlinux uses systemd template units, see # - http://0pointer.de/blog/projects/instances.html # - http://0pointer.de/blog/projects/instances.html Loading @@ -840,9 +847,13 @@ start: sudo -H systemctl start uwsgi@${SEARX_UWSGI_APP%.*} restart: sudo -H systemctl restart uwsgi@${SEARX_UWSGI_APP%.*} restart: sudo -H systemctl restart uwsgi@${SEARX_UWSGI_APP%.*} stop: sudo -H systemctl stop uwsgi@${SEARX_UWSGI_APP%.*} stop: sudo -H systemctl stop uwsgi@${SEARX_UWSGI_APP%.*} disable: sudo -H systemctl disable uwsgi@${SEARX_UWSGI_APP%.*} disable: sudo -H systemctl disable uwsgi@${SEARX_UWSGI_APP%.*} EOF EOF ;; ;; fedora-*) cat <<EOF fedora-*) cat <<EOF .. code:: bash # systemd --> /usr/lib/systemd/system/uwsgi.service # systemd --> /usr/lib/systemd/system/uwsgi.service # The unit file starts uWSGI in emperor mode (/etc/uwsgi.ini), see # The unit file starts uWSGI in emperor mode (/etc/uwsgi.ini), see # - https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html # - https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html Loading @@ -850,13 +861,16 @@ EOF create: ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} create: ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} restart: sudo -H touch ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} restart: sudo -H touch ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} EOF EOF ;; ;; esac esac echo -e ".. END searx uwsgi-description $DIST_NAME" echo -e ".. END searx uwsgi-description $DIST_NAME" echo -e "\n.. START searx uwsgi-appini $DIST_NAME" echo -e "\n.. START searx uwsgi-appini $DIST_NAME" eval "echo \"$(< "${TEMPLATES}/${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP}")\"" echo ".. code:: bash" echo eval "echo \"$(< "${TEMPLATES}/${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP}")\"" | prefix_stdout " " echo -e "\n.. END searx uwsgi-appini $DIST_NAME" echo -e "\n.. END searx uwsgi-appini $DIST_NAME" ) ) Loading Loading
docs/admin/installation-uwsgi.rst +6 −5 Original line number Original line Diff line number Diff line Loading @@ -108,6 +108,7 @@ restart the uwsgi application. :start-after: START searx uwsgi-description ubuntu-20.04 :start-after: START searx uwsgi-description ubuntu-20.04 :end-before: END searx uwsgi-description ubuntu-20.04 :end-before: END searx uwsgi-description ubuntu-20.04 .. hotfix: a bug group-tab need this comment .. group-tab:: Arch Linux .. group-tab:: Arch Linux Loading @@ -115,6 +116,7 @@ restart the uwsgi application. :start-after: START searx uwsgi-description arch :start-after: START searx uwsgi-description arch :end-before: END searx uwsgi-description arch :end-before: END searx uwsgi-description arch .. hotfix: a bug group-tab need this comment .. group-tab:: Fedora / RHEL .. group-tab:: Fedora / RHEL Loading @@ -128,22 +130,21 @@ restart the uwsgi application. .. group-tab:: Ubuntu / debian .. group-tab:: Ubuntu / debian .. kernel-include:: $DOCS_BUILD/includes/searx.rst .. kernel-include:: $DOCS_BUILD/includes/searx.rst :code: ini :start-after: START searx uwsgi-appini ubuntu-20.04 :start-after: START searx uwsgi-appini ubuntu-20.04 :end-before: END searx uwsgi-appini ubuntu-20.04 :end-before: END searx uwsgi-appini ubuntu-20.04 .. hotfix: a bug group-tab need this comment .. group-tab:: Arch Linux .. group-tab:: Arch Linux .. kernel-include:: $DOCS_BUILD/includes/searx.rst .. kernel-include:: $DOCS_BUILD/includes/searx.rst :code: ini :start-after: START searx uwsgi-appini arch :start-after: START searx uwsgi-appini arch :end-before: END searx uwsgi-appini arch :end-before: END searx uwsgi-appini arch .. hotfix: a bug group-tab need this comment .. group-tab:: Fedora / RHEL .. group-tab:: Fedora / RHEL .. kernel-include:: $DOCS_BUILD/includes/searx.rst .. kernel-include:: $DOCS_BUILD/includes/searx.rst :code: ini :start-after: START searx uwsgi-appini fedora :start-after: START searx uwsgi-appini fedora :end-before: END searx uwsgi-appini fedora :end-before: END searx uwsgi-appini fedora
docs/blog/index.rst +1 −0 Original line number Original line Diff line number Diff line Loading @@ -6,6 +6,7 @@ Blog :maxdepth: 2 :maxdepth: 2 :caption: Contents :caption: Contents lxcdev-202006 python3 python3 admin admin intro-offline intro-offline Loading
docs/blog/lxcdev-202006.rst 0 → 100644 +413 −0 Original line number Original line Diff line number Diff line .. _blog-lxcdev-202006: ======================================= Developing in Linux containers [202006] ======================================= .. _LXC: https://linuxcontainers.org/lxc/introduction/ .. sidebar:: Audience This blog post is written for experienced admins and developers / readers should have a serious meaning about: *distributed*, *merge* and *linux container*. .. contents:: Contents :depth: 2 :local: :backlinks: entry In PR :PR:`1803` we added a lot of scripts to Searx's boilerplate. In this blog post I will show you, how you can make use of them in *distributed and heterogeneous development cycles* (TL;DR; jump to the :ref:`blog-lxcdev-202006 abstract`). Motivation ========== Normally in our development cycle, we edit the sources and run some test and/or builds by using ``make`` before we commit. This cycle is simple and perfect but might fail in some aspects we should not overlook. The environment in which we run all our development processes matters! The :ref:`makefile` and the :ref:`make pyenv` encapsulate a lot for us, but they do not have access to all prerequisites. For example, there may have dependencies on packages that are installed on the developer's desktop, but usually are not preinstalled on a server or client system. Another examples are; settings have been made to the software on the developer's host that would never be set on a *production* system. *Linux Containers* (LXC_) are isolate environments and not to mix up on developer's all the prerequisites of all the projects he contribute to, is always a good choice. The scripts from PR :PR:`1803` can divide in those to install and maintain software: - :ref:`searx.sh` - :ref:`filtron.sh` - :ref:`morty.sh` and the script :ref:`lxc.sh`, with we can scale our installation, maintenance or even development tasks over a stack of containers, what we call: *Searx's lxc suite*. Gentlemen, start your engines! ============================== .. _LXD: https://linuxcontainers.org/lxd/introduction/ .. _archlinux: https://www.archlinux.org/ Before you can start with containers, you need to install and initiate LXD_ once: .. tabs:: .. group-tab:: desktop .. code:: sh $ snap install lxd $ lxd init --auto And you need to clone from origin or if you have your own fork, clone from your fork: .. tabs:: .. group-tab:: desktop .. code:: sh $ cd ~/Downloads $ git clone https://github.com/asciimoo/searx.git $ cd searx The :ref:`lxc-searx.env` consists of several images, see ``export LXC_SUITE=(...`` near by :origin:`utils/lxc-searx.env#L19`. For this blog post we exercise on a archlinux_ image. The container of this image is named ``searx-archlinux``. Lets build the container, but be sure that this container does not already exists, so first lets remove possible old one: .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh remove searx-archlinux $ sudo -H ./utils/lxc.sh build searx-archlinux .. sidebar:: The ``searx-archlinux`` container is the base of all our exercises here. In this container we install all services :ref:`including searx, morty & filtron <lxc.sh install suite>` in once: .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh install suite searx-archlinux To proxy HTTP from filtron and morty in the container to the outside of the container, install nginx into the container. Once for the bot blocker filtron: .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ ./utils/filtron.sh nginx install ... INFO: got 429 from http://10.174.184.156/searx and once for the content sanitizer (content proxy morty): .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ ./utils/morty.sh nginx install ... INFO: got 200 from http://10.174.184.156/morty/ .. sidebar:: Fully functional searx suite From here on you have a fully functional searx suite running with bot blocker (filtron) and Web content sanitizer (content proxy morty) needed for a *privacy protecting* search engine. On your system, the IP of your ``searx-archlinux`` container differs from http://10.174.184.156/searx, just open the URL reported in your installation protocol in your WEB browser from the desktop to test the instance from outside of the container. In such a searx suite admins can maintain and access the debug log of the different services quite easy. .. _working in containers: In containers, work as usual ============================ Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers open the root-bash in the container using ``./utils/lxc.sh cmd searx-archlinux``: .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux bash INFO: [searx-archlinux] bash [root@searx-archlinux searx]# pwd /share/searx The prompt ``[root@searx-archlinux ...]`` signals, that you are the root user in the searx-container. To debug the running searx instance use: .. tabs:: .. group-tab:: root@searx-archlinux .. code:: sh $ ./utils/searx.sh inspect service ... use [CTRL-C] to stop monitoring the log ... Back in the browser on your desktop open the service http://10.174.184.156/searx and run your application tests while the debug log is shown in the terminal from above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug option"* in searx's settings file and restarts the searx uwsgi application. To debug services from filtron and morty analogous use: .. tabs:: .. group-tab:: root@searx-archlinux .. code:: sh $ ./utils/filtron.sh inspect service $ ./utils/morty.sh inspect service Another point we have to notice is that each service (:ref:`searx <searx.sh>`, :ref:`filtron <filtron.sh>` and :ref:`morty <morty.sh>`) runs under dedicated system user account with the same name (compare :ref:`create searx user`). To get a shell from theses accounts, simply call one of the scripts: .. tabs:: .. group-tab:: root@searx-archlinux .. code:: sh $ ./utils/searx.sh shell $ ./utils/filtron.sh shell $ ./utils/morty.sh shell To get in touch, open a shell from the service user (searx@searx-archlinux): .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ ./utils/searx.sh shell // exit with [CTRL-D] (searx-pyenv) [searx@searx-archlinux ~]$ ... The prompt ``[searx@searx-archlinux]`` signals that you are logged in as system user ``searx`` in the ``searx-archlinux`` container and the python *virtualenv* ``(searx-pyenv)`` environment is activated. .. tabs:: .. group-tab:: searx@searx-archlinux .. code:: sh (searx-pyenv) [searx@searx-archlinux ~]$ pwd /usr/local/searx Wrap production into developer suite ==================================== In this section we will see how to change the *"Fully functional searx suite"* from a LXC container (which is quite ready for production) into a developer suite. For this, we have to keep an eye on the :ref:`installation basic`: - searx setup in: ``/etc/searx/settings.yml`` - searx user's home: ``/usr/local/searx`` - virtualenv in: ``/usr/local/searx/searx-pyenv`` - searx software in: ``/usr/local/searx/searx-src`` The searx software is a clone of the ``GIT_URL`` (see :ref:`makefile setup`) and the working tree is checked out from the ``GIT_BRANCH``. With the use of the :ref:`searx.sh` the searx service was installed as :ref:`uWSGI application <searx uwsgi>`. To maintain this service, we can use ``systemctl`` (compare :ref:`service architectures on distributions <uwsgi configuration>`). .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ systemctl stop uwsgi@searx With the command above, we stopped the searx uWSGI-App in the archlinux container. The uWSGI-App for the archlinux dsitros is configured in :origin:`utils/templates/etc/uwsgi/apps-archlinux/searx.ini`, from where at least you should attend the settings of ``uid``, ``chdir``, ``env`` and ``http``:: env = SEARX_SETTINGS_PATH=/etc/searx/settings.yml http = 127.0.0.1:8888 chdir = /usr/local/searx/searx-src/searx virtualenv = /usr/local/searx/searx-pyenv pythonpath = /usr/local/searx/searx-src If you have read the :ref:`"Good to know section" <lxc.sh>` you remember, that each container shares the root folder of the repository and the command ``utils/lxc.sh cmd`` handles relative path names **transparent**. To wrap the searx installation into a developer one, we simple have to create a smylink to the **transparent** reposetory from the desktop. Now lets replace the repository at ``searx-src`` in the container with the working tree from outside of the container: .. tabs:: .. group-tab:: container becomes a developer suite .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ ln -s /share/searx/ /usr/local/searx/searx-src Now we can develop as usual in the working tree of our desktop system. Every time the software was changed, you have to restart the searx service (in the conatiner): .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ systemctl restart uwsgi@searx Remember: :ref:`working in containers` .. here are just some examples from my daily usage: .. tabs:: .. group-tab:: desktop To *inspect* the searx instance (already described above): .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ ./utils/searx.sh inspect service Run :ref:`makefile`, e.g. to test inside the container: .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ make test To install all prerequisites needed for a :ref:`buildhosts`: .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ ./utils/searx.sh install buildhost To build the docs on a buildhost :ref:`buildhosts`: .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ make docs .. _blog-lxcdev-202006 abstract: Abstract ======== We build up a fully functional searx suite in a archlinux container: .. code:: sh $ sudo -H ./utils/lxc.sh install suite searx-archlinux To access HTTP from the desktop we installed nginx for the services inside the conatiner: .. tabs:: .. group-tab:: [root@searx-archlinux] .. code:: sh $ ./utils/filtron.sh nginx install $ ./utils/morty.sh nginx install To wrap the suite into a developer one, we created a symbolic link to the repository which is shared **transparent** from the desktop's file system into the container : .. tabs:: .. group-tab:: [root@searx-archlinux] .. code:: sh $ mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old $ ln -s /share/searx/ /usr/local/searx/searx-src $ systemctl restart uwsgi@searx To get remarks from the suite of the archlinux container we can use: .. tabs:: .. group-tab:: desktop .. code:: sh $ sudo -H ./utils/lxc.sh show suite searx-archlinux ... [searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx [searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/ [searx-archlinux] INFO: (eth0) docs-live: http://10.174.184.156:8080/ [searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65] ...
docs/utils/lxc.sh.rst +4 −3 Original line number Original line Diff line number Diff line Loading @@ -47,9 +47,9 @@ one**:: *Good to know ...* *Good to know ...* Eeach container shares the root folder of the repository and the Each container shares the root folder of the repository and the command command ``utils/lxc.sh cmd`` **handles relative path names transparent**, ``utils/lxc.sh cmd`` **handles relative path names transparent**, compare output compare output of:: of:: $ sudo -H ./utils/lxc.sh cmd -- ls -la Makefile $ sudo -H ./utils/lxc.sh cmd -- ls -la Makefile ... ... Loading @@ -66,6 +66,7 @@ If there comes the time you want to **get rid off all** the containers and $ sudo -H ./utils/lxc.sh remove $ sudo -H ./utils/lxc.sh remove $ sudo -H ./utils/lxc.sh remove images $ sudo -H ./utils/lxc.sh remove images .. _lxc.sh install suite: Install suite Install suite ============= ============= Loading
utils/searx.sh +42 −28 Original line number Original line Diff line number Diff line Loading @@ -816,6 +816,9 @@ rst-doc() { case $DIST_ID-$DIST_VERS in case $DIST_ID-$DIST_VERS in ubuntu-*|debian-*) cat <<EOF ubuntu-*|debian-*) cat <<EOF .. code:: bash # init.d --> /usr/share/doc/uwsgi/README.Debian.gz # init.d --> /usr/share/doc/uwsgi/README.Debian.gz # For uWSGI debian uses the LSB init process, this might be changed # For uWSGI debian uses the LSB init process, this might be changed # one day, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833067 # one day, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833067 Loading @@ -826,9 +829,13 @@ start: sudo -H service uwsgi start ${SEARX_UWSGI_APP%.*} restart: sudo -H service uwsgi restart ${SEARX_UWSGI_APP%.*} restart: sudo -H service uwsgi restart ${SEARX_UWSGI_APP%.*} stop: sudo -H service uwsgi stop ${SEARX_UWSGI_APP%.*} stop: sudo -H service uwsgi stop ${SEARX_UWSGI_APP%.*} disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} EOF EOF ;; ;; arch-*) cat <<EOF arch-*) cat <<EOF .. code:: bash # systemd --> /usr/lib/systemd/system/uwsgi@.service # systemd --> /usr/lib/systemd/system/uwsgi@.service # For uWSGI archlinux uses systemd template units, see # For uWSGI archlinux uses systemd template units, see # - http://0pointer.de/blog/projects/instances.html # - http://0pointer.de/blog/projects/instances.html Loading @@ -840,9 +847,13 @@ start: sudo -H systemctl start uwsgi@${SEARX_UWSGI_APP%.*} restart: sudo -H systemctl restart uwsgi@${SEARX_UWSGI_APP%.*} restart: sudo -H systemctl restart uwsgi@${SEARX_UWSGI_APP%.*} stop: sudo -H systemctl stop uwsgi@${SEARX_UWSGI_APP%.*} stop: sudo -H systemctl stop uwsgi@${SEARX_UWSGI_APP%.*} disable: sudo -H systemctl disable uwsgi@${SEARX_UWSGI_APP%.*} disable: sudo -H systemctl disable uwsgi@${SEARX_UWSGI_APP%.*} EOF EOF ;; ;; fedora-*) cat <<EOF fedora-*) cat <<EOF .. code:: bash # systemd --> /usr/lib/systemd/system/uwsgi.service # systemd --> /usr/lib/systemd/system/uwsgi.service # The unit file starts uWSGI in emperor mode (/etc/uwsgi.ini), see # The unit file starts uWSGI in emperor mode (/etc/uwsgi.ini), see # - https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html # - https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html Loading @@ -850,13 +861,16 @@ EOF create: ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} create: ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} restart: sudo -H touch ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} restart: sudo -H touch ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} EOF EOF ;; ;; esac esac echo -e ".. END searx uwsgi-description $DIST_NAME" echo -e ".. END searx uwsgi-description $DIST_NAME" echo -e "\n.. START searx uwsgi-appini $DIST_NAME" echo -e "\n.. START searx uwsgi-appini $DIST_NAME" eval "echo \"$(< "${TEMPLATES}/${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP}")\"" echo ".. code:: bash" echo eval "echo \"$(< "${TEMPLATES}/${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP}")\"" | prefix_stdout " " echo -e "\n.. END searx uwsgi-appini $DIST_NAME" echo -e "\n.. END searx uwsgi-appini $DIST_NAME" ) ) Loading
