Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 1ec325ad authored by Markus Heiser's avatar Markus Heiser
Browse files

[mod] limiter -> botdetection: modularization and documentation 

In order to be able to meet the outstanding requirements, the implementation is
modularized and supplemented with documentation.

This patch does not contain functional change, except it fixes issue #2455

----

Aktivate limiter in the settings.yml and simulate a bot request by::

    curl -H 'Accept-Language: de-DE,en-US;q=0.7,en;q=0.3' \
         -H 'Accept: text/html'
         -H 'User-Agent: xyz' \
         -H 'Accept-Encoding: gzip' \
         'http://127.0.0.1:8888/search?q=foo'

In the LOG:

    DEBUG   searx.botdetection.link_token : missing ping for this request: .....

Since ``BURST_MAX_SUSPICIOUS = 2`` you can repeat the query above two time
before you get a "Too Many Requests" response.

Closes: https://github.com/searxng/searxng/issues/2455


Signed-off-by: default avatarMarkus Heiser <markus.heiser@darmarit.de>
parent 5226044c

docs/admin/engines/settings.rst

+1 −1
Original line number Diff line number Diff line
@@ -235,7 +235,7 @@ Global Settings 


``limiter`` :

  Rate limit the number of request on the instance, block some bots.  The

  :ref:`limiter plugin` requires a :ref:`settings redis` database.

  :ref:`limiter src` requires a :ref:`settings redis` database.



.. _image_proxy:

docs/src/searx.botdetection.rst

0 → 100644
+45 −0
Original line number Diff line number Diff line 
.. _botdetection:



=============

Bot Detection

=============



.. contents:: Contents

   :depth: 2

   :local:

   :backlinks: entry



.. automodule:: searx.botdetection

  :members:



.. automodule:: searx.botdetection.limiter

  :members:





Rate limit

==========



.. automodule:: searx.botdetection.ip_limit

  :members:



.. automodule:: searx.botdetection.link_token

  :members:





Probe HTTP headers

==================



.. automodule:: searx.botdetection.http_accept

  :members:



.. automodule:: searx.botdetection.http_accept_encoding

  :members:



.. automodule:: searx.botdetection.http_accept_language

  :members:



.. automodule:: searx.botdetection.http_connection

  :members:



.. automodule:: searx.botdetection.http_user_agent

  :members:

docs/src/searx.plugins.limiter.rst

deleted100644 → 0
+0 −13
Original line number Diff line number Diff line 
.. _limiter plugin:



==============

Limiter Plugin

==============



.. sidebar:: info



   The :ref:`limiter plugin` requires a :ref:`Redis <settings redis>` database.



.. automodule:: searx.plugins.limiter

  :members:

searx/botdetection/__init__.py

0 → 100644
+26 −0
Original line number Diff line number Diff line 
# SPDX-License-Identifier: AGPL-3.0-or-later

# lint: pylint

""".. _botdetection src:



Bot detection methods

---------------------



The methods implemented in this python package are use by the :ref:`limiter src`.



"""



import flask





def dump_request(request: flask.Request):

    return (

        "%s: '%s'" % (request.headers.get('X-Forwarded-For'), request.path)

        + " || form: %s" % request.form

        + " || Accept: %s" % request.headers.get('Accept')

        + " || Accept-Language: %s" % request.headers.get('Accept-Language')

        + " || Accept-Encoding: %s" % request.headers.get('Accept-Encoding')

        + " || Content-Type: %s" % request.headers.get('Content-Type')

        + " || Content-Length: %s" % request.headers.get('Content-Length')

        + " || Connection: %s" % request.headers.get('Connection')

        + " || User-Agent: %s" % request.headers.get('User-Agent')

    )

searx/botdetection/http_accept.py

0 → 100644
+24 −0
Original line number Diff line number Diff line 
# SPDX-License-Identifier: AGPL-3.0-or-later

# lint: pylint

"""

Method ``http_accept``

----------------------



The ``http_accept`` method evaluates a request as the request of a bot if the

Accept_ header ..



- did not contain ``text/html``



.. _Accept:

   https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept



"""



from typing import Optional, Tuple

import flask





def filter_request(request: flask.Request) -> Optional[Tuple[int, str]]:

    if 'text/html' not in request.accept_mimetypes:

        return 429, "bot detected, HTTP header Accept did not contain text/html"

    return None