Jabber/XMPP server testing tool

What is the ideal Jabber/XMPP server testing tool?

Well, the question actually should be: What is the ideal testing tool in general? There are also helper questions: Why do programmers not test their code properly? Why do programmers not write unit tests for their code? What tool for testing would programmers want to use?

I was thinking about this subject for some time already because testing is important part of Tigase server development. Test Suite is Tigase subproject to which I dedicate significant development time and still I noticed my tests were not up to date. Some new extensions (XEPs) I added were not covered by tests even though testing framework is created by myself.

How could I expect others to use my testing framework when even I don't use it?

I asked myself all questions from the beginning of this article to rethink my approach to tests and to possibly re-design or re-implement my framework to the form which I would use on daily basis.

It seems to me that answer to above questions is quite simple:

I would use test framework on daily basis if it made my live easier. If it simplified my development work and basically if I had less to do....

The last sentence is a key: If I had less to do..... So tests should decrease development effort.

The ideal solution for Jabber/XMPP server would be a tool which could take an URL to XEP as a parameter, I press a button and it executes test against specification. As a result I could work just on server code until the test against the XEP returned success. In this way I would have really much less work to do because I wouldn't need to start a dozen Jabber clients and test the functionality in each client, possibly even use XML console to track whether the input from the server is correct.

It took just a few days for me to extend test suite...

Well, it doesn't accept an URL to XEP yet but it is almost equally simple. It accepts simple text file as an input. The file in most cases contains just stanzas directly taken from XEP - packets to send to the server and packets expected as a response. And instead of pressing button to execute test you run a single, simple command. Enough for me.

How it works:

Let's take XEP-0049 Private XML Storage. Looking into the spec we can see the first example:

Example 1. Client Stores Private Data
CLIENT:

<iq type="set" id="1001">
  <query xmlns="jabber:iq:private">
    <exodus xmlns="exodus:prefs">
      <defaultnick>Hamlet</defaultnick>
    </exodus>
  </query>
</iq>

SERVER:

<iq type="result" id="1001"/>

This is enough for the first simple test. I have to create text file JabberIqProvate.test looking like this:

send: {
<iq type="set" id="1001">
  <query xmlns="jabber:iq:private">
    <exodus xmlns="exodus:prefs">
      <defaultnick>Hamlet</defaultnick>
    </exodus>
  </query>
</iq>
}
expect: {
<iq type="result" id="1001"/>
}

And now I can execute the test:

testsuite $ ./scripts/all-tests-runner.sh --single JabberIqPrivate.test
Tigase server home directory: ../server
Version: 2.8.5-b422
Database:         xmldb
Server IP:        127.0.0.1
Extra parameters: JabberIqPrivate.test
Starting Tigase:
Tigase running pid=6751
Running: 2.8.5-b422-xmldb test, IP 127.0.0.1...
Script name: scripts/single-xmpp-test.xmpt
Common test:  Common test  ...        failure!
FAILURE,  (Received result doesn't match expected result., 
Expected one of: [<iq id="1001" type="result"/>], 
received: 
[<iq id="1001" type="error">
  <query xmlns="jabber:iq:private">
    <exodus xmlns="exodus:prefs">
      <defaultnick>Hamlet</defaultnick>
    </exodus>
  </query>
  <error type="cancel">
    <feature-not-implemented xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/>
    <text xml:lang="en" xmlns="urn:ietf:params:xml:ns:xmpp-stanzas">
      Feature not supported yet.</text>
  </error>
</iq>]),  
Total: 100ms
Test time: 00:00:02
Shutting down Tigase: 6751

If I just started working on this XEP and there is no code on the server side the result is perfectly expected although maybe this is not what we want. After a while of working on the server code I can execute the test once again:

testsuite $ ./scripts/all-tests-runner.sh --single JabberIqPrivate.test
Tigase server home directory: ../server
Version: 2.8.5-b422
Database:         xmldb
Server IP:        127.0.0.1
Extra parameters: JabberIqPrivate.test
Starting Tigase:
Tigase running pid=6984
Running: 2.8.5-b422-xmldb test, IP 127.0.0.1...
Script name: scripts/single-xmpp-test.xmpt
Common test:  Common test  ... success,  Total: 40ms
Test time: 00:00:01
Shutting down Tigase: 6984

This is it. The result we want. In simple and efficient way. We can repeat it as many times we want which is especially important in longer term. Every time we change the server code we can re-run tests to make sure we get correct responses from the server.

You can have a look in current, with more complete test cases, file for JabberIqPrivate.

Now my server tests are no longer outdated. Of course not all cases are so simple. Some XEPs require calculations to be done before stanza is sent or to compare received results. A good example for this case is user authentication like SASL and even NON-SASL. But still, there are many cases which can be covered by simple tests: roster management, privacy lists management, vCard, private data storage and so on.....

Application: 
Article type: