Presence queue?

Sorry, I thought you use Tigase from sources. I have just generated a new snapshot for the Tigase server. Please pick the last one and the functionality should work.

Is this available in the snapshots?

Justin, I have just committed code which allows you to use non-priority queues in the Tigase server.

Hopefully this might solve your problem with packets reordering. Unfortunately I have no easy way to replicate your problem so this would just your task to make sure that the changes I have made really help.

To activate non-priority queues you have to add following line to the init.properties file:

--nonpriority-queue=true

Please let me know if that helped. I have run all the functional automatic tests on the code with both types of queueing system and all tests passed. I do not know, however what are the performance implications yet.

I will try to find solution today on the train on my way back from fosdem. If this change is possible to code within the time I will add it to the next version release and let you know all the details.

Your explanation makes sense. I wonder if there is a workable solution that does not break spec. Stanzas only need to be processed in order between the same to and from JIDs, so you could prioritize message or iq traffice between one to+from pair over the presence of a different to+from pair. But maybe this is really difficult.

Anyway, since priority queues in tigase are not optional, it sounds like I still have to perform a workaround, right?

Back to your original problem - stanza reordering. It can in theory happen but usually it does not happen as user don't do things which can trigger this problem.

The thing is that different kind of packets cause a different load on the server and also their importance is different. For example user can normally accept that they did not receive status update from some of their friends but they won't accept if the message is lost.
And, because the status update packets can generate huge traffic on the server, they can overload the server queues and packets start to be dropped. This can normally happen on a large installation with a very high traffic. So to reduce the impact it can cause the Tigase has a few different queues for packets and each queue has a different priority.
All the iq, message and some presence packets have normal priority and all status update presence packets have lower priority. (There are also other queues for system and cluster packets.) Service disco and statistics have high priority.
Therefore a high volume of status update packets don't affect flow of any other data. It may even happen that the low priority queues are overloaded and packets are dropped but still all other packets are delivered very quickly without any delays. Low priority queues are processed only if all higher priority queues are empty. All what happens is that some presence updates may be missing or delivered with delay.
This is however a better than the total server overload and failure with OOM.

Of course this causes a side effect that message and iq packets may take over the presence updates. And this sometimes happens on kind of automated systems where there is a code generating traffic instead of people.

So the simplest way to make sure that there is no reordering is to send only packets with the same priority. They all get to the same queue and they all are processed sequentially. If for some reason this is not possible then it might be necessary to add an option to the Tigase to not use priority queues at all. However this is not a good solution for a high traffic installations for reasons described above.

No workorunds please. I have an idea how to solve it properly but it needs someexplanation. I am on mobile now so I will describe it tomorrow.

By the way: we mix other stanza types too. For example, an iq-set to request lots of data, which is then followed by an iq-result acknowledging the request and then a series of message stanzas carrying the real data. It is important that when the component replies with the iq-result and the message stanzas, that the clients receive iq-result first, otherwise the message stanzas won't have context and would be dropped by the client.

So really, we depend on in-order delivery across all stanza types, not just per-type.

There is no ack for the presence, no. I suppose I could have an ack to it, probably some sort of response presence. Maybe I will do this as a work around.

But... XMPP is supposed to deliver stanzas in order so I want to say this is wrong behavior by tigase.

Yes, this might be the case. Sometimes presence stanza can be taken over by other stanzas. Do you expect to receive any kind of response from the component for your presence stanza? Maybe you should wait before you send the IQ?

I am having a problem where if I send <presence to="my_component"> followed by <iq to="my_component"...>, the iq might be received by the component first. I am not sure where the fault lies, but I am wondering if it is tigase doing prioritization of stanzas and reordering them?

My component requires you to have sent it presence first, before certain iq requests are allowed.

There can be lots of the presences in the system, and by lots I mean really a lot. If the average user has a roster with 150 contacts and you have 100k online user, each of them changes status once a 1000 seconds then you may need to process something like 15k presence packets per second.
This is quite a high traffic to handle. I consider presences as the lowest priority packets which if lost no significant harm is caused. Of course this loss is possible only in case of the system overload when all queues are full and the system can not keep up with the traffic.
Thus presences have own, the lowest priority queue.

If the system is not overloaded then everything is processed normally, of course the lowest priority packets are processed when there is no higher priority packets waiting.