Commit b1f76640 authored by Nicolas Gelot's avatar Nicolas Gelot
Browse files

Merge branch 'sync_upstream_v0.17.0' into 'master'

Sync upstream v0.17.0

See merge request e/cloud/my-spot!60
parents 8c261be7 3627eb4a
......@@ -28,8 +28,6 @@ stages:
jobs:
include:
- python: "2.7"
env: PY=2
- python: "3.5"
- python: "3.6"
- python: "3.7"
......
Searx was created by Adam Tauber and is maintained by Adam Tauber, Alexandre Flament, Noémi Ványi, @pofilo and Markus Heiser.
Searx was created by Adam Tauber and is maintained by Adam Tauber, Alexandre Flament, Noémi Ványi, @pofilo, Gaspard d'Hautefeuille and Markus Heiser.
Major contributing authors:
......@@ -124,3 +124,12 @@ generally made searx better:
- @CaffeinatedTech
- Robin Schneider @ypid
- @splintah
- Lukas van den Berk @lukasvdberk
- @piplongrun
- Jason Kaltsikis @jjasonkal
- Sion Kazama @KazamaSion
- @resynth1943
- Mostafa Ahangarha @ahangarha
- @gordon-quad
- Sophie Tauchert @999eagle
- @bauruine
0.17.0 2020.07.09
=================
- New engines
- eTools
- Wikibooks
- Wikinews
- Wikiquote
- Wikisource
- Wiktionary
- Wikiversity
- Wikivoyage
- Rubygems
- Engine fixes (google, google images, startpage, gigablast, yacy)
- Private engines introduced - more details: https://asciimoo.github.io/searx/blog/private-engines.html
- Greatly improved documentation - check it at https://asciimoo.github.io/searx
- Added autofocus to all search inputs
- CSP friendly oscar theme
- Added option to hide engine errors with `display_error_messages` engine option (true/false values, default is true)
- Tons of accessibility fixes - see https://github.com/asciimoo/searx/issues/350 for details
- More flexible branding options: configurable vcs/issue tracker links
- Added "disable all" & "allow all" options to preferences engine select
- Autocomplete keyboard navigation fixes
- Configurable category order
- Wrap long lines in infoboxes
- Added RSS subscribtion link
- Added routing directions to OSM results
- Added author and length attributes to youtube videos
- Fixed image stretch with mobile viewport in oscar theme
- Added translatable JS strings
- Better HTML annotations - engine names and endpoints are available as classes
- RTL text fixes in oscar theme
- Handle weights in accept-language HTTP headers
- Added answerer results to rss/csv output
- Added new autocomplete backends to settings.yml
- Updated opensearch.xml
- Fixed custom locale setting from settings.yml
- Translation updates
- Removed engines: faroo
Special thanks to `NLNet <https://nlnet.nl>`__ for sponsoring multiple features of this release.
Special thanks to https://www.accessibility.nl/english for making accessibilty audit.
News
~~~~
- @HLFH joined the maintainer team
- Dropped Python2 support
0.16.0 2020.01.30
=================
......
......@@ -166,29 +166,29 @@ quiet_cmd_grunt = GRUNT $2
cmd_grunt = PATH="$$(npm bin):$$PATH" \
grunt --gruntfile "$2"
themes.oscar:
themes.oscar: node.env
$(Q)echo '[!] build oscar theme'
$(call cmd,grunt,searx/static/themes/oscar/gruntfile.js)
themes.simple:
themes.simple: node.env
$(Q)echo '[!] build simple theme'
$(call cmd,grunt,searx/static/themes/simple/gruntfile.js)
themes.legacy:
themes.legacy: node.env
$(Q)echo '[!] build legacy theme'
$(call cmd,lessc,themes/legacy/less/style-rtl.less,themes/legacy/css/style-rtl.css)
$(call cmd,lessc,themes/legacy/less/style.less,themes/legacy/css/style.css)
themes.courgette:
themes.courgette: node.env
$(Q)echo '[!] build courgette theme'
$(call cmd,lessc,themes/courgette/less/style.less,themes/courgette/css/style.css)
$(call cmd,lessc,themes/courgette/less/style-rtl.less,themes/courgette/css/style-rtl.css)
themes.pixart:
themes.pixart: node.env
$(Q)echo '[!] build pixart theme'
$(call cmd,lessc,themes/pix-art/less/style.less,themes/pix-art/css/style.css)
themes.bootstrap:
themes.bootstrap: node.env
$(call cmd,lessc,less/bootstrap/bootstrap.less,css/bootstrap.min.css)
themes.eelo:
......@@ -228,6 +228,7 @@ test.pylint: pyenvinstall
$(call cmd,pylint,\
searx/preferences.py \
searx/testing.py \
searx/engines/gigablast.py \
)
endif
......@@ -248,7 +249,7 @@ test.sh:
test.pep8: pyenvinstall
@echo "TEST pep8"
$(Q)$(PY_ENV_ACT); pep8 --exclude=searx/static --max-line-length=120 --ignore "E402,W503" searx tests
$(Q)$(PY_ENV_ACT); pep8 --exclude='searx/static, searx/engines/gigablast.py' --max-line-length=120 --ignore "E402,W503" searx tests
test.unit: pyenvinstall
@echo "TEST tests/unit"
......
......@@ -11,7 +11,6 @@ Spot was forked from searx: read [documentation](https://asciimoo.github.io/sear
* eelo theme
* redis cache on http requests (TTL 1 day)
* docker packaging thinking to be production ready
* better locale support
## Architecture
......
......@@ -108,6 +108,7 @@ restart the uwsgi application.
:start-after: START searx uwsgi-description ubuntu-20.04
:end-before: END searx uwsgi-description ubuntu-20.04
.. hotfix: a bug group-tab need this comment
.. group-tab:: Arch Linux
......@@ -115,6 +116,7 @@ restart the uwsgi application.
:start-after: START searx uwsgi-description arch
:end-before: END searx uwsgi-description arch
.. hotfix: a bug group-tab need this comment
.. group-tab:: Fedora / RHEL
......@@ -128,22 +130,21 @@ restart the uwsgi application.
.. group-tab:: Ubuntu / debian
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:code: ini
:start-after: START searx uwsgi-appini ubuntu-20.04
:end-before: END searx uwsgi-appini ubuntu-20.04
.. hotfix: a bug group-tab need this comment
.. group-tab:: Arch Linux
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:code: ini
:start-after: START searx uwsgi-appini arch
:end-before: END searx uwsgi-appini arch
.. hotfix: a bug group-tab need this comment
.. group-tab:: Fedora / RHEL
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:code: ini
:start-after: START searx uwsgi-appini fedora
:end-before: END searx uwsgi-appini fedora
......@@ -6,6 +6,7 @@ Blog
:maxdepth: 2
:caption: Contents
lxcdev-202006
python3
admin
intro-offline
......
.. _blog-lxcdev-202006:
=======================================
Developing in Linux containers [202006]
=======================================
.. _LXC: https://linuxcontainers.org/lxc/introduction/
.. sidebar:: Audience
This blog post is written for experienced admins and developers / readers
should have a serious meaning about: *distributed*, *merge* and *linux
container*.
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
In PR :PR:`1803` we added a lot of scripts to Searx's boilerplate. In this blog
post I will show you, how you can make use of them in *distributed and
heterogeneous development cycles* (TL;DR; jump to the :ref:`blog-lxcdev-202006
abstract`).
Motivation
==========
Normally in our development cycle, we edit the sources and run some test and/or
builds by using ``make`` before we commit. This cycle is simple and perfect but
might fail in some aspects we should not overlook.
The environment in which we run all our development processes matters!
The :ref:`makefile` and the :ref:`make pyenv` encapsulate a lot for us, but they
do not have access to all prerequisites. For example, there may have
dependencies on packages that are installed on the developer's desktop, but
usually are not preinstalled on a server or client system. Another examples
are; settings have been made to the software on the developer's host that would
never be set on a *production* system.
*Linux Containers* (LXC_) are isolate environments and not to mix up on
developer's all the prerequisites of all the projects he contribute to, is
always a good choice.
The scripts from PR :PR:`1803` can divide in those to install and maintain
software:
- :ref:`searx.sh`
- :ref:`filtron.sh`
- :ref:`morty.sh`
and the script :ref:`lxc.sh`, with we can scale our installation, maintenance or
even development tasks over a stack of containers, what we call: *Searx's lxc
suite*.
Gentlemen, start your engines!
==============================
.. _LXD: https://linuxcontainers.org/lxd/introduction/
.. _archlinux: https://www.archlinux.org/
Before you can start with containers, you need to install and initiate LXD_
once:
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ snap install lxd
$ lxd init --auto
And you need to clone from origin or if you have your own fork, clone from your
fork:
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ cd ~/Downloads
$ git clone https://github.com/asciimoo/searx.git
$ cd searx
The :ref:`lxc-searx.env` consists of several images, see ``export
LXC_SUITE=(...`` near by :origin:`utils/lxc-searx.env#L19`. For this blog post
we exercise on a archlinux_ image. The container of this image is named
``searx-archlinux``. Lets build the container, but be sure that this container
does not already exists, so first lets remove possible old one:
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh remove searx-archlinux
$ sudo -H ./utils/lxc.sh build searx-archlinux
.. sidebar:: The ``searx-archlinux`` container
is the base of all our exercises here.
In this container we install all services :ref:`including searx, morty & filtron
<lxc.sh install suite>` in once:
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh install suite searx-archlinux
To proxy HTTP from filtron and morty in the container to the outside of the
container, install nginx into the container. Once for the bot blocker filtron:
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
./utils/filtron.sh nginx install
...
INFO: got 429 from http://10.174.184.156/searx
and once for the content sanitizer (content proxy morty):
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
./utils/morty.sh nginx install
...
INFO: got 200 from http://10.174.184.156/morty/
.. sidebar:: Fully functional searx suite
From here on you have a fully functional searx suite running with bot blocker
(filtron) and Web content sanitizer (content proxy morty) needed for a
*privacy protecting* search engine.
On your system, the IP of your ``searx-archlinux`` container differs from
http://10.174.184.156/searx, just open the URL reported in your installation
protocol in your WEB browser from the desktop to test the instance from outside
of the container.
In such a searx suite admins can maintain and access the debug log of the
different services quite easy.
.. _working in containers:
In containers, work as usual
============================
Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers
open the root-bash in the container using ``./utils/lxc.sh cmd
searx-archlinux``:
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux bash
INFO: [searx-archlinux] bash
[root@searx-archlinux searx]# pwd
/share/searx
The prompt ``[root@searx-archlinux ...]`` signals, that you are the root user in
the searx-container. To debug the running searx instance use:
.. tabs::
.. group-tab:: root@searx-archlinux
.. code:: sh
$ ./utils/searx.sh inspect service
...
use [CTRL-C] to stop monitoring the log
...
Back in the browser on your desktop open the service http://10.174.184.156/searx
and run your application tests while the debug log is shown in the terminal from
above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug
option"* in searx's settings file and restarts the searx uwsgi application. To
debug services from filtron and morty analogous use:
.. tabs::
.. group-tab:: root@searx-archlinux
.. code:: sh
$ ./utils/filtron.sh inspect service
$ ./utils/morty.sh inspect service
Another point we have to notice is that each service (:ref:`searx <searx.sh>`,
:ref:`filtron <filtron.sh>` and :ref:`morty <morty.sh>`) runs under dedicated
system user account with the same name (compare :ref:`create searx user`). To
get a shell from theses accounts, simply call one of the scripts:
.. tabs::
.. group-tab:: root@searx-archlinux
.. code:: sh
$ ./utils/searx.sh shell
$ ./utils/filtron.sh shell
$ ./utils/morty.sh shell
To get in touch, open a shell from the service user (searx@searx-archlinux):
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
./utils/searx.sh shell
// exit with [CTRL-D]
(searx-pyenv) [searx@searx-archlinux ~]$ ...
The prompt ``[searx@searx-archlinux]`` signals that you are logged in as system
user ``searx`` in the ``searx-archlinux`` container and the python *virtualenv*
``(searx-pyenv)`` environment is activated.
.. tabs::
.. group-tab:: searx@searx-archlinux
.. code:: sh
(searx-pyenv) [searx@searx-archlinux ~]$ pwd
/usr/local/searx
Wrap production into developer suite
====================================
In this section we will see how to change the *"Fully functional searx suite"*
from a LXC container (which is quite ready for production) into a developer
suite. For this, we have to keep an eye on the :ref:`installation basic`:
- searx setup in: ``/etc/searx/settings.yml``
- searx user's home: ``/usr/local/searx``
- virtualenv in: ``/usr/local/searx/searx-pyenv``
- searx software in: ``/usr/local/searx/searx-src``
The searx software is a clone of the ``GIT_URL`` (see :ref:`makefile setup`) and
the working tree is checked out from the ``GIT_BRANCH``. With the use of the
:ref:`searx.sh` the searx service was installed as :ref:`uWSGI application
<searx uwsgi>`. To maintain this service, we can use ``systemctl`` (compare
:ref:`service architectures on distributions <uwsgi configuration>`).
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
systemctl stop uwsgi@searx
With the command above, we stopped the searx uWSGI-App in the archlinux
container.
The uWSGI-App for the archlinux dsitros is configured in
:origin:`utils/templates/etc/uwsgi/apps-archlinux/searx.ini`, from where at
least you should attend the settings of ``uid``, ``chdir``, ``env`` and
``http``::
env = SEARX_SETTINGS_PATH=/etc/searx/settings.yml
http = 127.0.0.1:8888
chdir = /usr/local/searx/searx-src/searx
virtualenv = /usr/local/searx/searx-pyenv
pythonpath = /usr/local/searx/searx-src
If you have read the :ref:`"Good to know section" <lxc.sh>` you remember, that
each container shares the root folder of the repository and the command
``utils/lxc.sh cmd`` handles relative path names **transparent**. To wrap the
searx installation into a developer one, we simple have to create a smylink to
the **transparent** reposetory from the desktop. Now lets replace the
repository at ``searx-src`` in the container with the working tree from outside
of the container:
.. tabs::
.. group-tab:: container becomes a developer suite
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
ln -s /share/searx/ /usr/local/searx/searx-src
Now we can develop as usual in the working tree of our desktop system. Every
time the software was changed, you have to restart the searx service (in the
conatiner):
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
systemctl restart uwsgi@searx
Remember: :ref:`working in containers` .. here are just some examples from my
daily usage:
.. tabs::
.. group-tab:: desktop
To *inspect* the searx instance (already described above):
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
./utils/searx.sh inspect service
Run :ref:`makefile`, e.g. to test inside the container:
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
make test
To install all prerequisites needed for a :ref:`buildhosts`:
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
./utils/searx.sh install buildhost
To build the docs on a buildhost :ref:`buildhosts`:
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
make docs
.. _blog-lxcdev-202006 abstract:
Abstract
========
We build up a fully functional searx suite in a archlinux container:
.. code:: sh
$ sudo -H ./utils/lxc.sh install suite searx-archlinux
To access HTTP from the desktop we installed nginx for the services inside the
conatiner:
.. tabs::
.. group-tab:: [root@searx-archlinux]
.. code:: sh
$ ./utils/filtron.sh nginx install
$ ./utils/morty.sh nginx install
To wrap the suite into a developer one, we created a symbolic link to the
repository which is shared **transparent** from the desktop's file system into
the container :
.. tabs::
.. group-tab:: [root@searx-archlinux]
.. code:: sh
$ mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old
$ ln -s /share/searx/ /usr/local/searx/searx-src
$ systemctl restart uwsgi@searx
To get remarks from the suite of the archlinux container we can use:
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh show suite searx-archlinux
...
[searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx
[searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/
[searx-archlinux] INFO: (eth0) docs-live: http://10.174.184.156:8080/
[searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65]
...
......@@ -13,7 +13,7 @@ Private engines
To solve this issue private engines were introduced in :pull:`1823`.
A new option was added to engines named `tokens`. It expects a list
of strings. If the user making a request presents one of the tokens
of an engine, he/she is able to access information about the engine
of an engine, they can access information about the engine
and make search requests.
Example configuration to restrict access to the Arch Linux Wiki engine:
......
......@@ -30,6 +30,14 @@ Example plugin
ctx['search'].suggestions.add('example')
return True
Register your plugin
====================
To enable your plugin register your plugin in
searx > plugin > __init__.py.
And at the bottom of the file add your plugin like.
``plugins.register(name_of_python_file)``
Plugin entry points
===================
......
......@@ -81,7 +81,7 @@ Parameters
Theme of instance.
Please note, available themes depend on an instance. It is possible that an
instance administrator deleted, created or renamed themes on his/her instance.
instance administrator deleted, created or renamed themes on their instance.
See the available options in the preferences page of the instance.
``oscar-style`` : default ``logicodev``
......@@ -91,7 +91,7 @@ Parameters
``oscar``.
Please note, available styles depend on an instance. It is possible that an
instance administrator deleted, created or renamed styles on his/her
instance administrator deleted, created or renamed styles on their
instance. See the available options in the preferences page of the instance.
``enabled_plugins`` : optional
......