Towards a Testing Framework Architecture
by Martin May

Abstract:
While other engineering disciplines inherently provide test methodologies 
to compare technologies and mechanisms, the networking community did not 
yet propose testing environments for their ideas. In computer programming, 
unit testing is a procedure used to validate that individual units of 
source code are working properly. The key idea is to integrate self-testing 
capabilities into the tests by formally specifying the inputs to the software 
under test as well as specifying the expected results of the test in an 
executable format. 

Specifically in the context of the situated and autonomic communication (SAC), 
the testing of the proposed systems is difficult. A formal specification of 
autonomic behavior is required as well as standardized test cases. 
It is further unclear how testbeds for disruptive technologies have to 
be designed and built. The Internet is now an important production and 
business tool. ISPs would not risk taking it down to install new, untested 
routing software or risk running experimental software. Such an interruption 
or an unexpected failure that could take the whole network down could result 
in loss of a large number of existing business and resulting revenue. 

Thus, extensive experimentation is a crucial step for coming even close to 
convincing a network provider to deploy any new enhancement. On the other 
hand, deployment of the new technology might actually be the only way of 
testing it, under ``realistic'' conditions. These two facts create a vicious 
circle, where disruptive technologies are not deployed due to lack of enough 
experiments to prove that they would not take the Internet down, and new 
protocols can never be tested to the extend necessary due to the ISPs' 
averting to deploy them. The key question here is how to build testbeds 
for disruptive research.

The proposal now is to agree on a test methodology and test framework 
for novel network architectures/new communication paradigms. Why not 
adopting the test methodology as applied in software unit testing? 
Software testing requires the formal specification of inputs to the 
software under test as well as the specification of the expected results. 
We have to apply these rules in the networking community. 
As a result, when we apply a strict testing methodology we have to make 
sure that:
-       Test results are reproducible
-       We agree on the test inout. E.g., mobility models (RWP is 
        considered unrealistic  even though might be used as a worst 
        case benchmark)
-       Measurements and traces are available. Even though mobility traces 
        are difficult to obtain and share (metadata about the conditions of 
        the measurements,...)
-       What is the failure model we are interested in? Packet losses, 
        link breaks, transmission errors? 
-       The target environement is clearly defined. Is the application or
        protocol for fixed networks, intermittently connected networks, 
        or delay tolerant networks?
-       The expected result can clearly be described? How to test 
        autonomic behaviour? If the network is self-managing, how
        to define formally the expected output?

Martin May
ETH Zurich
Gloriastrasse 35
8092 Zurich
Switzerland
email: maym@tik.ee.ethz.ch