Tests

Submitted by mateusz.fiolka (not verified) on Thu, 2009-11-26 11:08

Tests are very important part of Tigase server development process.

Each release goes through fully automated testing process. All server functions are considered implemented only when they pass testing cycle. Tigase test suite is used for all our automatic tests which allows to define different test scenarios.

There is no tweaking on databases for tests. All databases are installed in standard way and run with default settings. Database is cleared each time before test cycle starts.

There are no modifications to Tigase configuration file as well. All tests are performed on default configuration generated by configuration wizards.

The server is tested in all supported environments:

  1. XMLDB - tests with built-in simple XML database. This is simple and efficient solution for small installations. I recommend it for services with up to 100 user accounts although it was successfully tested with 10,000 user accounts.
  2. MySQL - tests with MySQL database. Much slower than XMLDB but may handle much more user accounts.
  3. PostgreSQL - tests with PostgreSQL database. Again it is much slower than XMLDB but may handle much more user accounts. This is basically exactly the same code as for MySQL database (SQL Connector) but tests are executed to make sure the code is compatible with all supported SQL databases and to compare performance.
  4. Distributed - is test for distributed installation where c2s and s2s components run on separated machine which connects using external component protocol (XEP-0114) to another machine with SessionManager running.

Functional tests:

Functional tests - basic checking if all the functions work at correctly. These tests are performed every time the code is sent to source repository.

Version XMLDB MySQL PGSQL Distributed
3.3.2-b889 00:00:12 00:00:17 00:00:17 none
3.3.2-b880 00:00:13 00:00:15 00:00:15 None
3.0.2-b700 00:00:22 00:00:24 00:00:25 00:00:25
2.9.5-b606 00:00:22 00:00:24 00:00:24 00:00:24
2.9.3-b548 00:00:22 00:00:23 00:00:25 00:00:25
2.9.1-b528 00:00:21 00:00:23 00:00:24 00:00:25
2.8.6-b434 00:00:21 00:00:24 00:00:24 00:00:25
2.8.5-b422 00:00:21 00:00:24 00:00:24 00:00:26
2.8.3-b409 00:00:27 00:00:29 00:00:29 00:00:32
2.7.2-b378 00:00:30 00:00:34 00:00:33 00:00:35
2.6.4-b300 00:00:30 00:00:32 00:00:35 00:00:39
2.6.4-b295 00:00:29 00:00:32 00:00:45 00:00:36
2.6.0-b287 00:00:31 00:00:34 00:00:47 00:00:43
2.5.0-b279 00:00:30 00:00:34 00:00:45 00:00:43
2.4.0-b263 00:00:29 00:00:33 00:00:45 00:00:44
2.3.4-b226 None 00:00:48 None None

Performance tests:

Performance tests - checking whether the function performs well enough.

Version XMLDB MySQL PGSQL Distributed
3.3.2-b889 00:12:17 00:13:42 00:17:10 none
3.3.2-b880 00:13:39 00:14:09 00:17:39 None
3.0.2-b700 00:10:26 00:11:00 00:12:08 00:24:05
2.9.5-b606 00:09:54 00:11:18 00:37:08 00:16:20
2.9.3-b548 00:10:00 00:11:29 00:36:43 00:16:47
2.9.1-b528 00:09:46 00:11:15 00:36:12 00:16:36
2.8.6-b434 00:10:02 00:11:45 00:36:36 00:17:36
2.8.5-b422 00:12:37 00:14:40 00:38:59 00:21:40
2.8.3-b409 00:12:32 00:14:26 00:37:57 00:21:26
2.7.2-b378 00:12:28 00:14:57 00:37:09 00:22:20
2.6.4-b300 00:12:46 00:14:59 00:36:56 00:35:00
2.6.4-b295 00:12:23 00:14:59 00:42:24 00:30:18
2.6.0-b287 00:13:50 00:16:53 00:48:17 00:49:06
2.5.0-b279 00:13:29 00:16:58 00:47:15 00:41:52
2.4.0-b263 00:13:20 00:16:21 00:43:56 00:42:08
2.3.4-b226 None 01:23:30 None None

Stability tests:

Stability tests - checking whether the function behaves well in long term run. It must handle hundreds of requests a second in several hours server run.

Version XMLDB MySQL PGSQL Distributed
2.3.4-b226 None 16:06:31 None None