It looks like we have a season for load tests. High number of people contact me asking for some help and advice on the load tests running.

Most of cases are about comparing different server implementations and I am not going to talk about comparing. Comparing different server implementation is very difficult. So many factors and elements must be taken into consideration for a fair comparison. Even generating exactly the same database for each server is quite difficult task.

I would like to give a few hints about running load tests over the Tigase server. What needs to be considered, what settings are important, how to tweak the system to get best possible performance. How to find system bottlenecks and how to deal with them....

All these suggestion are put in a random order and I am writing them from the top of my head. Please don't hesitate to send me your comments about it. I will try to update this guide in time.

This is pre-final release of the new version for the Tigase XMPP Server. It has been extensively tested under a different load and in different environments. Also it went through all our automated tests system. It seems to be stable. Please feel free to check it out and report all the problems you may find.

For installation instructions please refer to quick start documentation. The pre-final release is published only as JAR file - the installer. The final version will be also released on all other formats: exe, tar.gz and zip.

This is a major release with huge number of changes. Here is a brief list of what's new and what has been changed in quite random order:

1. Virtual hosts support - now virtual hosts are stored in database. Tigase can now effectively run for thousands of virtual domain. Virtual domains can be added/removed at runtime without any disruption to the working system. You can also limit number of users per domain, temporarily block registration or user login for the domain and this is all available through ad-hoc commands.
2. Scripting support - you can load scripts to the Tigase server at runtime also via ad-hoc commands. At the moment scripts are supported for administration tasks only. All JSR-223 languages compatible are supported which includes but is not limited to: Groovy, JRuby, Jython, Scala and possible a few others. 
3. Hugely improved performance on multi-cpu/multi-core systems. You can now control number of threads for almost every component of the system. Again big thank you to Sun Microsystems for offering they hardware for tests.
4. User repository connection pool added.
5. New, much more effective clustering code added.
6. Monitoring and self-monitoring functionality added. You can watch the system running and track the server statistics via SNMP, JMX, HTTP and XMPP. But you don't have to watch them. The server monitors itself and can send you notifications when something is going wrong. The monitoring framework is highly extensible and you can easily plug your own monitors too. At the moment the Tigase looks at Memory, CPU, Disk usage as well as it watches logs.
7. New installer has been improved too. It does create administrator accounts for you during installation and often requested feature - the command line mode for the installer is now supported too.
8. Alternatively - Debian/Ubuntu packages are available too. Directly from SVN so you can install and play with the most hot version of the server.
9. PubSub further improved. Now, you can optionally use PubSub with dedicated database schema instead of Tigase built-in UserRepository. The new schema greatly improved performance and was successfully tested with up to 1mln nodes and 1 thousand subscription each.

The full and detailed description of what's new will be compiled for the final release. I am awaiting your comments and please report all the problems you may find.

If you use Ubuntu and also run the most recent Tigase version from SVN you no longer have to compile the code yourself.

I have just received a word from Smoku that he has updated Ubuntu packages for the Tigase server he maintains. Fresh packages are now generated straight from the Tigase SVN every 2 hours.

For more details please refer to his announcement.

If you have any comments or suggestions on this, please don't hesitate to let us know.

Thanks to our new contributor smoku we have now the user database converter.

You can convert user database from format used by other XMPP servers to Tigase format. This is Groovy based script or rather extensible framework and you can easily 'plug' more source formats for different servers if you need a conversion from the format not yet supported.

Tigase can store users' data in different ways. The converter handles this very well. It reads the Tigase configuration you use for your installation and loads all the user data in a correct format using Tigase own database connectors. This way there is no place for incompatibility.

The converter is already available in our SVN repository for Tigase converter and the README file explains how to use it.

Please feel free to try it and send us your comments.

I have had a great opportunity and pleasure to use Sun's environment both hardware and software to run load tests on the Tigase server for a couple of last weeks. Sun also ofered me something more. Their knowledge and help during the tests allowed me to improve Tigase and fine tune the system to get the best results. I would like to say great thank you to Deepak Dhanukodi, ISV Engineering from Sun who was a huge help.

Below is a description of the tests I run, environment and results I got.

Summary

I know summary should be at the end but I realize that many people may be interested in the results while not being interested in all the details. So here we go.

• The main goal for all these tests was to run the Tigase server under the load of 500k active user connections with an empty user roster. This test was going to show how well the Tigase server handles huge number of I/O and huge number of user sessions on a single machine.

  Success! The Tigase easily handled the load with CPU usage below 30% and memory consumption at about 50% level. The test was so successful that we tried to run another similar test to get to 1mln online users. This however failed because client machines couldn't generate such a load.

• Secondary goal was to run comparison tests with different user roster size for and user connections count above 100k to see how the roster size impacts the load and resources consumption.

  This test wasn't the kind of score max but still I think it is also a great success. At the roster size of 40 and above the Tigase server started to behave unstable. Long GC activities impacted overall performance and in some cases leaded to unresponsive service. More details below. I learnt not only that default GC is not a good choice for the Tigase server under a high load but also I found the best GC and GC parameters to get a stable service with even higher load than I planed before. The CMS GC is the one which should be used to run Tigase.

• Max connections and roster with 50 elements was the last test I wanted to run. In most XMPP installations I helped to setup, the average roster size was just below 50 elements. So the goal for this test was to see how many connections the Tigase can handle with such a roster.

  300k user connections with roster size 50 is the result which is quite good. CPU usage was below 50% and memory consumption below 60%. We could certainly try to handle more connections. Unfortunately I have never expected that the system can handle more than 300k user connections with 50 elements roster so this is what I had in the database prepared for the test.

For the Tigase server plugin development it is important to understand how it all works. There are different kind of plugins responsible for processing packets at different stages of the data flow. Please read the introduction below before proceeding to the actual coding part.

Introduction

In the Tigase server plugins are pieces of code responsible for processing particular XMPP stanza. A separate plugin might be responsible for processing messages, a different one for processing presences, and there might a separate plugins responsible for iq roster, different for iq version and so on.

A plugin provides information about what exact XML element(s) name(s) with xmlns it is interested in. So you can create a plugin which is interested in all packets containing caps child.

There might be no plugin for a particular stanza element and then a default actions is used which is simple forwarding stanza to a destination address. There might be also more than one plugin for a specific XML element and then they all process the same stanza simultaneously in separate threads so there is no guarantee on the order in which the stanza is processed by a different plugins.

Each stanza goes through the Session Manager component which processes packets in a few steps. Have a look at the picture below:
The picture shows that each stanza is processed by the session manager in 4 steps:

I am happy to announce we now have Ubuntu packages for the Tigase server 4.1.4. The packages have been prepared by Tomasz Sterna and are hosted on his server.

The archive is available at the address: http://codex.xiaoka.com/apt/, instructions in the README file. To install it just run:

apt-get install tigase-server

Note! This is not official release of the Tigase server version 4.1.4 yet.

If you have any comments or suggestions please don't hesitate to contact us.

If you don't want to install Tigase using manual method, you can use the GUI installer. It not only copies server files to preferred place, but also assists with configuration of the most important parameters and database setup. Therefore it is the preferred way to install Tigase.

Most projects try to make sure that the SVN trunk code always compiles correctly. In the Tigase server case this is not enough however. Lots of installations out there we know of run just straight from out SVN trunk line. This puts on us an extra responsibility. Therefore our general approach is to run all functional tests before each code commit to ensure it works correctly. Of course this does not guarantee it will work correctly and efficiently in all cases and on all systems.

Some people like to be on the bleeding edge and always use the source code repository trunk some others prefer to stick to stable public releases. There is however a lot of other who would like to be able to use something from the middle - have the most recent features and new bug fixes but at least in "Beta" or "Release-Candidate" state.

If you look at the Monitor on the right hand side of the Tigase website you can see that the currently running version is always higher then the version available for download. If you are interested to update your server to the more recent version without much hassle please continue to read.

Installing Tigase from the public binary packages is pretty straightforward especially if you use the installer. Using SVN trunk sources to compile them for your own installation is not simple thing, however people who decide to do so normally don't need much help or instructions. This document describes how to update your system using so called Betas or RC versions we release from time to time.

The Tigase server can be remotely monitored over following protocols: JMX/RMI, SNMP and HTTP. Even though JMX offers the biggest control and visibility to the server states all of the monitoring services give the same basic set of the server statistics:

• Number of network connections for s2s, c2s and Bosh
• Last second, last minute and last hour load for all main components: SM, MR, c2s, s2s, Bosh, MUC and PubSub
• System statistics - memory usage (heap and non heap) and the server uptime in milliseconds and human readable text.
• Users statistics - number of registered users and number of online user session.

JMX/RMI and SNMP servers offer basic security and can restrict access based and the HTTP server doesn't offer any access restriction mechanisms. Therefore HTTP monitoring is recommended to work behind a firewall.

The monitoring itself causes very low overhead in terms of the resources and CPU consumption on top of the normal Tigase processing requirements so it can be left always on without worrying about performance degradation.

Note. This works with the Tigase server from version 4.2.0 or SVN revision 1418.

This is a chat from yesterday:

17:37:37 (wsharp): hey kobit, just wanted to give you a big Thank You for an awesome XMPP platform
17:37:46 kobit: hehe, thank you too
17:38:00 kobit: it is very nice to hear that someone likes it
17:38:08 (wsharp): I briefly chatted w/ you a year about using it for a real-time trading system
17:38:25 (wsharp): I did end up creating that system, and tigase is the critical messaging piece
17:38:38 kobit: ah, really nice, good to know
17:38:42 kobit: great news
17:38:44 (wsharp): its just great, now I'm back and I see all this new work you've done
17:39:01 kobit: yes, Tigase is under an intensive development all the time
18:12:12 (wsharp): oh one more thing, your bosh implementation was the best a year ago and it still is.. it's just rock solid
18:12:35 kobit: thank a lot, good to know
18:12:39 kobit: it is even better now
18:12:41 (wsharp): I've not had to touch, restart, or administer tigase since I've setup the platform 9 months ago
18:12:45 kobit: I have done lot's of improvement
18:12:57 kobit: wow, that's good indeed
18:13:21 (wsharp): I better get the new version! I have another new system I"m creating that'll use some real-time messaging  too

It is really nice to have a chat like that. It does really help in the development effort and gives extra motivation to continue work.

There is however something more you (users) can do. No, no I am not going to talk about money. Something different: Spread the word, let others know about the Tigase. Chat is nice and helps a lot but consider doing a bit, this bit more....

A few weeks ago I have written the article about active service monitoring in the Tigase server via XMPP. This mechanism allows for sending alerts to a selected set of JIDs. It allows also to pull statistics from the server on demand. It is very convenient way because you can track the server activity using any XMPP client. You can also receive warnings while you are online.

I received a good feedback about this feature but also some people pointed me out that it would be extremely useful to do remote monitoring of the service via different protocols as well. This would allow to integrate the Tigase server monitoring with external automated systems.

I have been working hard for a few last weeks to fulfill those requests and now all the basic server statistics are accessible via JMX framework as MBeans.  As MBeans they are automatically available to any JMX server. There are 3 servers available with the Tigase server: JMX/RMI default Java server. This protocol gives you the most flexibility and most control over the server. It allows not only to passively watch the server activity but also you can change the server parameters and interact with the system.

Two more JMX servers are available as well exposing data via SNMP and HTTP.

It has happened recently that we have tried very hard to fix a few annoying problems on one of the Tigase installations. Whatever we did, however the problems still existed after uploading a new version and the server restart. It worked fine in our development environment and it just didn't on the target system.

It turned out that due to a specific environment settings on the target system an old version of the Tigase server was always started regardless updates we were uploading. When I finally started looking at the installation the first indication that something is wrong was lack of any log files in place where I expected them.

The Tigase server is very extensible through the well defined API. You can easily attach your code to any part of the server and load it at the startup time. It is quite simple to connect the Tigase to any database, add own components, plugins, packets filters, resource monitors and so on.

There are, however at least two disadvantages of this kind of extensibility. You have to write Java code and the code can not be reloaded at run-time. So basically you have to restart the server to apply your changes.

Installing Groovy script

What about a code in a scripting language which could be reloaded/added/removed at runtime without affecting normal work of the server? Which scripting language? Ideally any, ideally your preferred language. How? Of course using ad-hoc commands.

This is possible now. You can create your scripts in almost any language and load the script at runtime to the server to do some work for you. The Tigase server supports scripting through JSR-223 API in Java6. Maybe it doesn't make sense to use some of the supported languages like AWK for example in the Tigase server. Nevertheless I have learned already that users' use cases go far beyond my expectations. Therefore there are virtually no restrictions on the language you choose. Just put all required JAR files in the Tigase libs/ directory and the language is available to you.

The main part of the Tigase server development is testing. Actually I spend much more time on testing the server in different scenarios than on writing the actual code.

Most of the tests can be run locally on the development machine. Performance tests and especially load tests require more hardware but usually the development machine as a client and one mor as the server is enough.

One category of tests however require more hardware to be involved and in the best case scenario that hardware should be always on to see how the software behaves over the longer period of time. Clustering tests. They require ideally 3 machines for the server deployment. I run automated tests on the Tigase server very often and also this clustered installation is available to the public at the address: xmpp-test.tigase.org. By running it for a long time and exposing it to the public I can catch many exceptional cases and fix problems which don't appear on short automatic test runs.

At the moment I use 3 machines which are always turned on. Unfortunately 2 of them are very old hardware which have started to refuse working more and more often. Therefore I am looking at replacing them with something different. Ideally it would be low power consumption machine with at least 512MB RAM. There is even no hard disk needed as the system could be started from USB stick or DVD disk. It could be also a laptop with at least single core Pentium M 1.5GHz and 512MB RAM.

If you have such a hardware and can offer it to the project I would very much appreciate your help. Please send me a message using the contact form, or via email/xmpp at the address: kobit AT tigase.org to discuss details.