A Pluggable Framework for the Asterisk Test Suite - Part 1

Asterisk Test Suite - a (very) brief Overview

On the Asterisk development team, we attempt to write software using an Agile methodology (specifically, the "Scrum" flavor).  As part of this development process, we perform continuous integration using Atlassian Bamboo, which, in addition to doing a basic build check, also executes sets of tests against the current build. There's a myriad of reasons to do continuous integration - they're almost too many to list - but the most obvious of these is that we get a consistent sanity check as to the state of the code at all times.

The tests that the Bamboo server runs come in two forms: unit tests, as part of the Asterisk Unit Test framework, and functional tests, as part of the Asterisk Test Suite.  The Asterisk Test Suite is a freely available external tool that orchestrates these tests against one or more instances of Asterisk.  These tests cover a large range of functionality - everything from SIP compliance, to channel technology interoperability, to ensuring that Asterisk applications and functions work as intended.

Each test is essentially a stand-alone entity - the Asterisk Test Suite does not mandate what language a test is written.  Organically, the Test Suite has moved towards Python, but it still includes tests written in a host of other languages (most notably Lua, but there's also some bash ones in there that no one enjoys maintaining).  As the tests have grown, we've added to the libraries that the Asterisk Test Suite provides to try and minimize reproduction of functionality and to minimize the amount of time that it takes to write a test.

While the Test Suite is very flexible - it will run any executable file named 'run-test' - that flexibility has cost us in having a lot of repeated, boilerplate code.  In the current state of the Test Suite, the Python tests typically inherit from a common base class, TestCase, that does a lot of the heavy lifting of starting/stopping Asterisk, creating an AMI connection, and exposing the basic AMI events that allow the test logic to be inserted. Even still, the tests can get a bit repetitive.

Two Current Tests: cdr_accountcode and cdr_userfield

The following two tests - cdr_accountcode and cdr_userfield (which tests that the AccountCode and UserField values are recorded properly in CDR records) - illustrate the general problem.  The tests inherit from CDRTestCase, which in turn inherits from TestCase.  Besides the complexity of having multiple levels of inheritance, there is still some repeated elements to the meat of these two tests.

cdr_accountcode:

#!/usr/bin/env python
'''
Copyright (C) 2012, Digium, Inc.
Walter Doekes

This program is free software, distributed under the terms of
the GNU General Public License Version 2.
'''

import sys
sys.path.append("lib/python")
from asterisk.CDRTestCase import CDRTestCase
from asterisk.asterisk import Asterisk
from asterisk.cdr import AsteriskCSVCDR, AsteriskCSVCDRLine
from twisted.internet import reactor
import re

class CDRAccountCodeTest(CDRTestCase):
    def __init__(self):
        CDRTestCase.__init__(self)

        # 3@default -> (answer)
        # accountcode: third
        self.add_expectation("Master", AsteriskCSVCDRLine(source="",
            accountcode="third",
            destination="3", dcontext="default", callerid="",
            channel="Local/3@default-.*", dchannel="",
            lastapp="Hangup", lastarg="",
            disposition="ANSWERED", amaflags="DOCUMENTATION",
        ))
        # NOTE: I removed some of the additional expectations
        # for brevity

def main():
    test = CDRAccountCodeTest()
    test.start_asterisk()
    reactor.run()
    test.stop_asterisk()
    return test.results()

if __name__ == '__main__':
    sys.exit(main())

# vim: set ts=8 sw=4 sts=4 et ai:


cdr_userfield:

#!/usr/bin/env python
'''
Copyright (C) 2010, Digium, Inc.
Terry Wilson

This program is free software, distributed under the terms of
the GNU General Public License Version 2.
'''

import sys
sys.path.append("lib/python")
from asterisk.CDRTestCase import CDRTestCase
from asterisk.asterisk import Asterisk
from asterisk.cdr import AsteriskCSVCDR, AsteriskCSVCDRLine
from twisted.internet import reactor
import re

class CDRUserFieldTest(CDRTestCase):
    def __init__(self):
        CDRTestCase.__init__(self)

        self.add_expectation('cdrtest_local',AsteriskCSVCDRLine(source="", 
            destination="1", dcontext="default", callerid="",
            channel="Local/1@default-.*", dchannel="", lastapp="Hangup", lastarg="",
            disposition="ANSWERED", amaflags="DOCUMENTATION", userfield="bazinga"
        ))


def main():
    test = CDRUserFieldTest()
    test.start_asterisk()
    reactor.run()
    test.stop_asterisk()
    return test.results()

if __name__ == '__main__':
    sys.exit(main())

Some obvious things in this comparison:

• The act of setting up the test case class, starting Asterisk, running the twisted reactor, stopping Asterisk, and returning a result is all functionality that a single entry point could do, if it knew what object to instantiate
• The object being instantiated is identical, save for what CDR information it expects to match on
• The CDR information that a match must be made on can be configuration driven

Since common classes and modules can provide the functionality these tests use, and the differences between the tests can be driven by configuration, why not ... do that?

A Pluggable Framework

Since tests already define their configuration in a YAML file, the idea is to use that configuration data to drive  all of the test's behavior.  This includes what modules are instantiated to support test execution, what data is needed to configure them, and what the expected results of the test should be.

Since all tests are spawned as a separate process, a new module - TestRunner (what a creative name...) - is responsible for having an entry point and building the test objects specified in the configuration.  TestRunner parses the test's YAML file, looking for modules to locate and instantiate.  In the next post I'll go more into how that happens - there's one aspect of this that I think is pretty cool and gets around some of the requirements Python has for defining packages.

Loadable modules currently fall into one of two categories: Test Objects, and Pluggable Modules.  A Test Object is responsible for starting and stopping Asterisk and orchestrating the test activities.  Pluggable Modules add bits of functionality into the test - things like verifying CDRs/CELs, adding AMI event listening and verification, etc.  These categories may have to expand in the future - and the role of a Test Object defined more narrowly - but it works well for our purposes now.

As an example, say we have a test-configuration that defines a Test Object SimpleTestCase (in module SimpleTestCase (and yes, per PEP8, this should be lower case - Pythonic compliance has not traditionally been our strongest suit, but we're working on that)):


test-modules:
    test-object:
        config-section: test-object-config
        typename: 'SimpleTestCase.SimpleTestCase'

test-object-config:
    spawn-after-hangup: True
    test-iterations:
        -
            channel: 'Local/1@default'
            application: 'Echo'



The TestRunner parses the test-modules configuration section and determine that it needs to create a Test Object.  The YAML file itself defines the Keyword (test-object-config) that contains the configuration information for this object.  TestRunner takes the generated in memory representation of the YAML and passes that in to the constructor of the TestObject.

    # NOTE: Some code removed here for brevity
    test_object_spec = test_config['test-modules']['test-object']

    module_obj = load_and_parse_module(test_object_spec['typename'])
    if module_obj is None:
        return None

    test_object_config = None
    if ('config-section' in test_object_spec and
        test_object_spec['config-section'] in test_config):
        test_object_config = test_config[test_object_spec['config-section']]
    else:
        test_object_config = test_config

    # The test object must support injection of its location as a parameter
    # to the constructor, and its test-configuration object (or the full test
    # config object, if none is specified)
    test_obj = module_obj(test_path, test_object_config)


Note that a test object gets two things: its relative location to the current process directory (so it can find its test specific files), and the in-memory YAML object configuring it.  The method load_and_parse_module does the work of separating the typename into package/module/classname parts and importing the module.


def load_and_parse_module(type_name):
    ''' Take a qualified module/object name, load the module, and return
    the type specifying the object

    Parameters:
    type_name A fully qualified module/object to load into memory

    Returns:
    An object type to be instantiated
    None on error
    '''


    LOGGER.debug("Importing %s" % type_name)

    # Split the object typename into its constituent parts - the module name
    # and the actual type of the object in that module
    parts = type_name.split('.')
    module_name = ".".join(parts[:-1])

    if not len(module_name):
        LOGGER.error("No module specified: %s" % module_name)
        return None

    module = __import__(module_name)
    for comp in parts[1:]:
        module = getattr(module, comp)
    return module



And voila: we have an in-memory object that acts as our test!

TestRunner does the mechanics of starting the twisted reactor and telling the Test Object to run - which takes the part of the entry point mechanics that was reproduced across all tests.  So, now that we can create a Test Object and tell it to do stuff, we need to figure out how to actually verify the purpose of those tests: CDR records.

In the next post I'll go into more detail about the Pluggable Modules - specifically the CDR Module - and how they interact with the Test Object.  We can then put the pieces together to see how the CDR tests above are configured to obviate the need for the run-test scripts.