Merge branch 'tc-testing-plugin-architecture'

*  The kernel must have veth support available, as a veth pair is created

   prior to running the tests.

*  All tc-related features must be built in or available as modules.

   To check what is required in current setup run:

*  All tc-related features being tested must be built in or available as

   modules.  To check what is required in current setup run:

   ./tdc.py -c

   Note:

RUNNING TDC

-----------

To use tdc, root privileges are required. tdc will not run otherwise.

To use tdc, root privileges are required.  This is because the

commands being tested must be run as root.  The code that enforces

execution by root uid has been moved into a plugin (see PLUGIN

ARCHITECTURE, below).

All tests are executed inside a network namespace to prevent conflicts

within the host.

If nsPlugin is linked, all tests are executed inside a network

namespace to prevent conflicts within the host.

Running tdc without any arguments will run all tests. Refer to the section

on command line arguments for more information, or run:

the failed test in the TAP output.

OVERVIEW OF TDC EXECUTION

-------------------------

One run of tests is considered a "test suite" (this will be refined in the

future).  A test suite has one or more test cases in it.

A test case has four stages:

  - setup

  - execute

  - verify

  - teardown

The setup and teardown stages can run zero or more commands.  The setup

stage does some setup if the test needs it.  The teardown stage undoes

the setup and returns the system to a "neutral" state so any other test

can be run next.  These two stages require any commands run to return

success, but do not otherwise verify the results.

The execute and verify stages each run one command.  The execute stage

tests the return code against one or more acceptable values.  The

verify stage checks the return code for success, and also compares

the stdout with a regular expression.

Each of the commands in any stage will run in a shell instance.

USER-DEFINED CONSTANTS

----------------------

Example:

	$TC qdisc add dev $DEV1 ingress

The NAMES values are used to substitute into the commands in the test cases.

COMMAND LINE ARGUMENTS

----------------------

Run tdc.py -h to see the full list of available arguments.

-p PATH           Specify the tc executable located at PATH to be used on this

                  test run

-c                Show the available test case categories in this test file

-c CATEGORY       Run only tests that belong to CATEGORY

-f FILE           Read test cases from the JSON file named FILE

-l [CATEGORY]     List all test cases in the JSON file. If CATEGORY is

                  specified, list test cases matching that category.

-s ID             Show the test case matching ID

-e ID             Execute the test case identified by ID

-i                Generate unique ID numbers for test cases with no existing

                  ID number

usage: tdc.py [-h] [-p PATH] [-D DIR [DIR ...]] [-f FILE [FILE ...]]

              [-c [CATG [CATG ...]]] [-e ID [ID ...]] [-l] [-s] [-i] [-v]

              [-d DEVICE] [-n NS] [-V]

Linux TC unit tests

optional arguments:

  -h, --help            show this help message and exit

  -p PATH, --path PATH  The full path to the tc executable to use

  -v, --verbose         Show the commands that are being run

  -d DEVICE, --device DEVICE

                        Execute the test case in flower category

selection:

  select which test cases: files plus directories; filtered by categories

  plus testids

  -D DIR [DIR ...], --directory DIR [DIR ...]

                        Collect tests from the specified directory(ies)

                        (default [tc-tests])

  -f FILE [FILE ...], --file FILE [FILE ...]

                        Run tests from the specified file(s)

  -c [CATG [CATG ...]], --category [CATG [CATG ...]]

                        Run tests only from the specified category/ies, or if

                        no category/ies is/are specified, list known

                        categories.

  -e ID [ID ...], --execute ID [ID ...]

                        Execute the specified test cases with specified IDs

action:

  select action to perform on selected test cases

  -l, --list            List all test cases, or those only within the

                        specified category

  -s, --show            Display the selected test cases

  -i, --id              Generate ID numbers for new test cases

netns:

  options for nsPlugin(run commands in net namespace)

  -n NS, --namespace NS

                        Run commands in namespace NS

valgrind:

  options for valgrindPlugin (run command under test under Valgrind)

  -V, --valgrind        Run commands under valgrind

PLUGIN ARCHITECTURE

-------------------

There is now a plugin architecture, and some of the functionality that

was in the tdc.py script has been moved into the plugins.

The plugins are in the directory plugin-lib.  The are executed from

directory plugins.  Put symbolic links from plugins to plugin-lib,

and name them according to the order you want them to run.

Example:

bjb@bee:~/work/tc-testing$ ls -l plugins

total 4

lrwxrwxrwx  1 bjb  bjb    27 Oct  4 16:12 10-rootPlugin.py -> ../plugin-lib/rootPlugin.py

lrwxrwxrwx  1 bjb  bjb    25 Oct 12 17:55 20-nsPlugin.py -> ../plugin-lib/nsPlugin.py

-rwxr-xr-x  1 bjb  bjb     0 Sep 29 15:56 __init__.py

The plugins are a subclass of TdcPlugin, defined in TdcPlugin.py and

must be called "SubPlugin" so tdc can find them.  They are

distinguished from each other in the python program by their module

name.

This base class supplies "hooks" to run extra functions.  These hooks are as follows:

pre- and post-suite

pre- and post-case

pre- and post-execute stage

adjust-command (runs in all stages and receives the stage name)

The pre-suite hook receives the number of tests and an array of test ids.

This allows you to dump out the list of skipped tests in the event of a

failure during setup or teardown stage.

The pre-case hook receives the ordinal number and test id of the current test.

The adjust-command hook receives the stage id (see list below) and the

full command to be executed.  This allows for last-minute adjustment

of the command.

The stages are identified by the following strings:

  - pre  (pre-suite)

  - setup

  - command

  - verify

  - teardown

  - post (post-suite)

To write a plugin, you need to inherit from TdcPlugin in

TdcPlugin.py.  To use the plugin, you have to put the

implementation file in plugin-lib, and add a symbolic link to it from

plugins.  It will be detected at run time and invoked at the

appropriate times.  There are a few examples in the plugin-lib

directory:

  - rootPlugin.py:

      implements the enforcement of running as root

  - nsPlugin.py:

      sets up a network namespace and runs all commands in that namespace

  - valgrindPlugin.py

      runs each command in the execute stage under valgrind,

      and checks for leaks.

      This plugin will output an extra test for each test in the test file,

      one is the existing output as to whether the test passed or failed,

      and the other is a test whether the command leaked memory or not.

      (This one is a preliminary version, it may not work quite right yet,

      but the overall template is there and it should only need tweaks.)

ACKNOWLEDGEMENTS

- Add support for multiple versions of tc to run successively

- Improve error messages when tdc aborts its run

- Improve error messages when tdc aborts its run.  Partially done - still

  need to better handle problems in pre- and post-suite.

- Allow tdc to write its results to file

- Use python logger module for debug/verbose output

- Allow tdc to write its results to file.

  Maybe use python logger module for this too.

- A better implementation of the "hooks".  Currently, every plugin

  will attempt to run a function at every hook point.  Could be

  changed so that plugin __init__ methods will register functions to

  be run in the various predefined times.  Then if a plugin does not

  require action at a specific point, no penalty will be paid for

  trying to run a function that will do nothing.

- Proper exception handling - make an exception class and use it

- a TestCase class, for easier testcase handling, searching, comparison

- a TestSuite class

  and a way to configure a test suite,

  to automate running multiple "test suites" with different requirements

- super simple test case example using ls, touch, etc

#!/usr/bin/env python3

class TdcPlugin:

    def __init__(self):

        super().__init__()

        print(' -- {}.__init__'.format(self.sub_class))

    def pre_suite(self, testcount, testidlist):

        '''run commands before test_runner goes into a test loop'''

        self.testcount = testcount

        self.testidlist = testidlist

        if self.args.verbose > 1:

            print(' -- {}.pre_suite'.format(self.sub_class))

    def post_suite(self, index):

        '''run commands after test_runner completes the test loop

        index is the last ordinal number of test that was attempted'''

        if self.args.verbose > 1:

            print(' -- {}.post_suite'.format(self.sub_class))

    def pre_case(self, test_ordinal, testid):

        '''run commands before test_runner does one test'''

        if self.args.verbose > 1:

            print(' -- {}.pre_case'.format(self.sub_class))

        self.args.testid = testid

        self.args.test_ordinal = test_ordinal

    def post_case(self):

        '''run commands after test_runner does one test'''

        if self.args.verbose > 1:

            print(' -- {}.post_case'.format(self.sub_class))

    def pre_execute(self):

        '''run command before test-runner does the execute step'''

        if self.args.verbose > 1:

            print(' -- {}.pre_execute'.format(self.sub_class))

    def post_execute(self):

        '''run command after test-runner does the execute step'''

        if self.args.verbose > 1:

            print(' -- {}.post_execute'.format(self.sub_class))

    def adjust_command(self, stage, command):

        '''adjust the command'''

        if self.args.verbose > 1:

            print(' -- {}.adjust_command {}'.format(self.sub_class, stage))

        # if stage == 'pre':

        #     pass

        # elif stage == 'setup':

        #     pass

        # elif stage == 'execute':

        #     pass

        # elif stage == 'verify':

        #     pass

        # elif stage == 'teardown':

        #     pass

        # elif stage == 'post':

        #     pass

        # else:

        #     pass

        return command

    def add_args(self, parser):

        '''Get the plugin args from the command line'''

        self.argparser = parser

        return self.argparser

    def check_args(self, args, remaining):

        '''Check that the args are set correctly'''

        self.args = args

        if self.args.verbose > 1:

            print(' -- {}.check_args'.format(self.sub_class))

tdc - Adding plugins for tdc

Author: Brenda J. Butler - bjb@mojatatu.com

ADDING PLUGINS

--------------

A new plugin should be written in python as a class that inherits from TdcPlugin.

There are some examples in plugin-lib.

The plugin can be used to add functionality to the test framework,

such as:

- adding commands to be run before and/or after the test suite

- adding commands to be run before and/or after the test cases

- adding commands to be run before and/or after the execute phase of the test cases

- ability to alter the command to be run in any phase:

    pre        (the pre-suite stage)

    prepare

    execute

    verify

    teardown

    post       (the post-suite stage)

- ability to add to the command line args, and use them at run time

The functions in the class should follow the following interfaces:

    def __init__(self)

    def pre_suite(self, testcount, testidlist)     # see "PRE_SUITE" below

    def post_suite(self, ordinal)                  # see "SKIPPING" below

    def pre_case(self, test_ordinal, testid)       # see "PRE_CASE" below

    def post_case(self)

    def pre_execute(self)

    def post_execute(self)

    def adjust_command(self, stage, command)       # see "ADJUST" below

    def add_args(self, parser)                     # see "ADD_ARGS" below

    def check_args(self, args, remaining)          # see "CHECK_ARGS" below

PRE_SUITE

This method takes a testcount (number of tests to be run) and

testidlist (array of test ids for tests that will be run).  This is

useful for various things, including when an exception occurs and the

rest of the tests must be skipped.  The info is stored in the object,

and the post_suite method can refer to it when dumping the "skipped"

TAP output.  The tdc.py script will do that for the test suite as

defined in the test case, but if the plugin is being used to run extra

tests on each test (eg, check for memory leaks on associated

co-processes) then that other tap output can be generated in the

post-suite method using this info passed in to the pre_suite method.

SKIPPING

The post_suite method will receive the ordinal number of the last

test to be attempted.  It can use this info when outputting

the TAP output for the extra test cases.

PRE_CASE

The pre_case method will receive the ordinal number of the test

and the test id.  Useful for outputing the extra test results.

ADJUST

The adjust_command method receives a string representing

the execution stage and a string which is the actual command to be

executed.  The plugin can adjust the command, based on the stage of

execution.

The stages are represented by the following strings:

    'pre'

    'setup'

    'command'

    'verify'

    'teardown'

    'post'

The adjust_command method must return the adjusted command so tdc

can use it.

ADD_ARGS

The add_args method receives the argparser object and can add

arguments to it.  Care should be taken that the new arguments do not

conflict with any from tdc.py or from other plugins that will be used

concurrently.

The add_args method should return the argparser object.

CHECK_ARGS

The check_args method is so that the plugin can do validation on

the args, if needed.  If there is a problem, and Exception should

be raised, with a string that explains the problem.

eg:  raise Exception('plugin xxx, arg -y is wrong, fix it')

Include the 'id' field, but do not assign a value. Running tdc with the -i

option will generate a unique ID for that test case.

tdc will recursively search the 'tc' subdirectory for .json files.  Any

test case files you create in these directories will automatically be included.

If you wish to store your custom test cases elsewhere, be sure to run tdc

with the -f argument and the path to your file.

tdc will recursively search the 'tc-tests' subdirectory (or the

directories named with the -D option) for .json files.  Any test case

files you create in these directories will automatically be included.

If you wish to store your custom test cases elsewhere, be sure to run

tdc with the -f argument and the path to your file, or the -D argument

and the path to your directory(ies).

Be aware of required escape characters in the JSON data - particularly when

defining the match pattern. Refer to the tctests.json file for examples when

in doubt.

Be aware of required escape characters in the JSON data - particularly

when defining the match pattern. Refer to the supplied json test files

for examples when in doubt.  The match pattern is written in json, and

will be used by python.  So the match pattern will be a python regular

expression, but should be written using json syntax.

TEST CASE STRUCTURE

If an error is detected during the setup/teardown process, execution of the

tests will immediately stop with an error message and the namespace in which

the tests are run will be destroyed. This is to prevent inaccurate results

in the test cases.

in the test cases.  tdc will output a series of TAP results for the skipped

tests.

Repeated failures of the setup/teardown may indicate a problem with the test

case, or possibly even a bug in one of the commands that are not being tested.

individual command into a list, with the command being first, followed by all

acceptable exit codes for the command.

Example:

A pair of setup commands.  The first can have exit code 0, 1 or 255, the

second must have exit code 0.

        "setup": [

            [

                "$TC actions flush action gact",

                0,

                1,

                255

            ],

            "$TC actions add action reclassify index 65536"

        ],