Logs for jdev@conference.jabber.org

[00:14:09] * Florob left the chat.
[01:53:18] * stpeter joined the chat.
[02:18:05] * Timon joined the chat.
[02:21:43] <Timon> hello, just finished the xmpp book by skr and need some help
[02:23:08] <Timon> [pkr rather than skr]
[02:23:46] <stpeter> Timon: ahoj!
[02:23:54] <stpeter> Timon: what is your question?
[02:24:31] <stpeter> /me parses the pkr acronym
[02:27:01] <Timon> ajoi stpete, how is an ejabberd cluster glued together?
[02:27:31] <stpeter> magic
[02:27:45] <stpeter> :)
[02:28:07] <Timon> haha, i mean, is it done via the federation or by other means?
[02:28:08] <stpeter> but seriously, clustering is something that is specific to each implementation and it's part of the "secret sauce" for each server
[02:28:15] <stpeter> no, not federation
[02:28:24] <stpeter> usually there is an internal API of some kind
[02:28:45] <stpeter> e.g., in ejabberd it might be message-passing of some kind using Erlang primitives
[02:28:56] <stpeter> whereas in Tigase it would be done in a totally different way
[02:29:03] <stpeter> etc.
[02:29:40] <Timon> are there opensource glues to make an ejabbered cluster?
[02:29:47] <stpeter> ah
[02:30:24] <stpeter> I don't have the answer to that, but I suspect that clustering *might* be a Process-One add-on -- you would need to do some research in the ejabberd community
[02:32:53] <Timon> can you suggest an ejabberd chat-group for this?
[02:33:48] <stpeter> there's a chatroom on jabber.ru -- I think it is ejabberd@conference.jabber.ru but I am not 100% sure
[02:33:50] <stpeter> you might try that
[02:33:57] <stpeter> if it doesn't work, let us know
[02:34:32] <stpeter> yep: http://www.ejabberd.im/chatroom
[02:34:41] <stpeter> the Internet contains such helpful information :-)
[02:36:02] <Timon> i know, but thought i'd ask in case you might know off the bat :-)
[02:36:02] * jcea left the chat.
[02:36:21] <stpeter> Timon: we know some things here, but not everything :-)
[02:37:37] <Timon> well, you did write the xmpp book ...
[02:37:48] <Timon> [co-write]
[02:37:52] <stpeter> well, I wrote about a third of it :-)
[02:38:43] <deryni> ejabberd clustering used to just be normal erlang node clustering I believe. I don't know what it is now.
[02:38:58] <stpeter> deryni: ah, that makes sense
[02:39:30] <deryni> Since all internal erlang process communication is by "address" it didn't matter what node that internal address was on since the cluster knew how to route it appropriately... or something like that. I haven't thought about it in a while and I only mostly understood it even then.
[02:44:13] <Timon> can you suggest a good tut for installing and configuring ejabberd?
[02:44:50] <stpeter> Timon: I suggest you look around on http://ejabberd.im/
[02:45:13] <stpeter> Timon: are you wedded to ejabberd for some reason? there are other servers out there, you know :-)
[02:47:07] <Timon> i know, and erlang spooks me (i come from c, c++, javascript, ...) but ejabberd appears to have the most current dev and support
[02:47:35] <stpeter> Timon: IMHO the most actively developed servers are Prosody, ejabberd, and Tigase
[02:48:11] <stpeter> Timon: there is also an actively developed fork of ejabberd called MongooseIM
[02:48:39] <deryni> Where were you seeing active ejabberd dev and support?
[02:50:39] <Timon> would rather have a server written in c so i don't have to learn erlang ... and i was under impression that process-one was the current dev and support for ejabberd
[02:51:17] <stpeter> Timon: jabberd2 is a possibility (it is somewhat active but less so, IMHO, than the other servers listed above)
[02:51:46] <stpeter> jabberd2 = C, Prosody = Lua, Tigase = Java, ejabberd/MongooseIM = Erlang
[02:53:44] <Timon> i read the jabberd2 made issues for chesspark when many concurrent users hence chesspark migrated to ejabberd resolving the issues
[02:54:17] <stpeter> that might be old information - chesspark has been offline for a long time :-)
[02:55:08] <stpeter> no software is perfect, of course
[02:55:26] <stpeter> there are undoubtedly some issues with each implementation
[02:56:19] <Timon> old is right, i think the post was 2008, but who knows if jabberd2 had those issues resolved, and for an xmpp server that's a pretty important issue
[02:56:36] <stpeter> Timon: I don't disagree
[02:56:54] <stpeter> Timon: you might post to the jabberd2 discussion list and see what the developers have to say
[02:57:23] <stpeter> Timon: what are your scalability requirements?
[02:58:20] <Timon> good idea ... want to easily scale when each node maxes whatever parms (cpu, mem, etc.)
[02:59:05] <stpeter> but what scale? 10k sessions? 100k? 1 million? etc.
[03:01:30] <Timon> who knows, one hopes 450+ million ... haha ... the important thing is that scaling the app be easy, just by adding nodes and some light configuration, without major code rewrites
[03:01:56] <stpeter> Timon: sure, you will be the next WhatsApp I am sure
[03:02:15] <stpeter> but it helps to know what you're trying to build so you can determine the right tool for the job
[03:02:29] <Timon> i did qualify with "one hopes" :-)
[03:05:07] <Timon> whatsapp is a good example of function, albeit at much smaller scale to begin, but with easy growth built in
[03:05:29] <stpeter> I am going to log off now
[03:05:42] <stpeter> Timon: good luck making $19 billion :-)
[03:05:50] <stpeter> hang around here for more answers
[03:05:52] <stpeter> bye!
[03:05:54] <Timon> ok, thanks for the chat
[03:05:56] * stpeter left the chat.
[05:11:32] * Valérian joined the chat.
[05:20:38] * Valérian left the chat.
[05:32:53] * Valérian joined the chat.
[05:40:19] * Valérian left the chat.
[05:42:24] * Valérian joined the chat.
[05:48:20] * Timon left the chat.
[06:14:00] * Alex joined the chat.
[06:17:22] * Flow joined the chat.
[06:21:43] * Timon joined the chat.
[06:30:56] * Flow left the chat.
[06:31:01] * Flow joined the chat.
[06:42:57] * Valérian left the chat.
[06:46:32] * Flow left the chat.
[06:49:15] * Dev joined the chat.
[06:50:25] * Dev left the chat.
[07:04:32] * ermine joined the chat.
[07:24:59] * Timon left the chat.
[08:49:25] * Lloyd joined the chat.
[09:46:49] * Lance joined the chat.
[10:03:47] * Asterix joined the chat.
[10:12:10] * Lance left the chat.
[10:22:52] * Valérian joined the chat.
[10:35:45] * Asterix left the chat.
[10:45:02] * Valérian left the chat.
[11:02:05] * xnyhps left the chat.
[11:21:42] * Valérian joined the chat.
[11:22:48] * xnyhps joined the chat.
[11:35:01] * Valérian left the chat.
[11:55:08] * Lance joined the chat.
[12:09:41] * adfsdf joined the chat.
[12:10:47] * adfsdf left the chat.
[12:12:46] * Lance left the chat.
[12:25:45] * Valérian joined the chat.
[12:39:01] * Asterix joined the chat.
[13:04:07] * Florob joined the chat.
[13:06:40] * Tobias joined the chat.
[13:07:31] * Asterix left the chat.
[13:19:06] * Asterix joined the chat.
[13:24:01] * Asterix left the chat.
[13:30:55] * jcea joined the chat.
[13:45:37] * Valérian left the chat.
[13:49:24] * naw joined the chat.
[14:18:46] * Sam Whited joined the chat.
[14:37:57] * naw left the chat.
[14:47:09] * 0xAFFE joined the chat.
[15:21:48] * deryni left the chat.
[15:25:14] * jcea left the chat.
[15:45:33] * stpeter joined the chat.
[15:48:10] * jkj joined the chat.
[15:48:13] * jkj left the chat.
[15:54:01] * Sam Whited left the chat.
[15:56:28] * MattJ joined the chat.
[16:08:58] * 0xAFFE joined the chat.
[16:22:33] * deryni joined the chat.
[16:53:59] * Valérian joined the chat.
[17:00:01] * Lloyd left the chat.
[17:09:44] * intosi left the chat.
[17:10:45] * intosi joined the chat.
[17:25:29] * ermine left the chat.
[17:39:21] * Valérian left the chat.
[17:42:33] * Valérian joined the chat.
[18:00:39] * Valérian left the chat.
[18:01:15] * Valérian joined the chat.
[18:09:38] * ermine joined the chat.
[18:14:37] * Link Mauve left the chat.
[18:17:29] * Timon joined the chat.
[18:18:20] * Link Mauve joined the chat.
[18:21:30] * Timon left the chat.
[18:25:58] * Timon joined the chat.
[18:27:36] * Valérian left the chat.
[18:39:30] * ermine left the chat.
[18:44:19] * Alex left the chat.
[18:47:11] * intosi left the chat.
[18:55:22] <Timon> i'm trying to join ejabberd@conference.jabber.ru but there's a captcha block and the captcha link is broken, is there a proper ejabberd chat somewhere else?
[18:55:31] * Philonous left the chat.
[18:56:55] <stpeter> Timon: that is the proper chat :-)
[18:57:08] * Philonous joined the chat.
[18:57:51] <stpeter> they just try to make sure that random bots don't join
[18:57:58] <stpeter> Timon: they have an email discussion list, too
[18:59:01] <Timon> i'm on psi and used the same method to join this chat as for ejabberd@conference.jabber.ru but cannot get in, any ideas?
[18:59:26] <deryni> Hold on a second.
[18:59:54] * coyo joined the chat.
[19:00:05] <deryni> Try joining the room again.
[19:00:17] <Timon> hold on let me try
[19:01:09] <Timon> now i get this ... *** 2014-02-26 [10:59:39] *** ejabberd@conference.jabber.ru is Offline
[19:01:31] <Timon> but at least the captcha is not blocking me
[19:02:16] <dezant> Timon: enable captcha plugin in your psi
[19:02:57] <deryni> Is offline? That's odd.
[19:04:44] <Timon> i've just looked for the captcha plugin in the options|advanced but can't find it, where is it?
[19:07:51] <ralphm> I can join the ejabberd room without issues from Gajim. Surprisingly it also showed the captcha thing in the room tab. Pretty neat.
[19:11:40] <Timon> i'm on uprod.biz ... could that be the problem?
[19:20:37] * Valérian joined the chat.
[19:20:38] <coyo> in swift, ejabberd's captcha showed up in a separate server tab
[19:24:55] <Tobias> swift doesn't have native captcha supporr
[19:24:57] <Tobias> swift doesn't have native captcha support
[19:25:39] <Timon> on the psi control panel there's a drop down titled "Not in List (2/2)" and below it a red X with "ejabberd@conference.jabber.ru", any ideas?
[19:27:13] <deryni> Did you add that as a buddy to your roster?
[19:27:31] * Flow joined the chat.
[19:28:11] <Timon> i may have done so when i was fiddling trying to join ejabberd@conference.jabber.ru
[19:29:33] * Valérian left the chat.
[19:32:28] <Timon> hovering over the red X shows this ... "ejabberd@conference.jabber.ru ... Subscription: none ... Presence Error: Not authorized ... The sender must provide proper credentials before being allowed to perform the action, or has provided improper credentials."
[19:33:53] * naw joined the chat.
[19:36:03] <deryni> Yeah. That looks sort of like you tried to add it as a buddy.
[19:38:16] <coyo> i've seen stranger things in xmpp than adding the server as a buddy.
[19:38:27] <coyo> most clients interpret server messages as a potential buddy
[19:38:47] <coyo> most servers do not agree that it is, itself, a potential buddy
[19:39:07] <Florob> Oh, servers is fine. MUCs however are a bit problematic.
[19:39:13] <coyo> haha.
[19:39:34] <Florob> /me has 2 servers in his roster
[19:39:45] <coyo> well, MUCs are treated similarly to users in what little conference server code i've read, so small extensions like MUC avatars are possible
[19:40:09] <coyo> clients may actually render avatars for MUCs without modification
[19:40:19] <coyo> some clients, anyway
[19:48:40] * Alex joined the chat.
[19:50:31] <Timon> i deleted the ejabberd@conference.jabber.ru buddy from the psi roster ... now when i try to join ejabberd@conference.jabber.ru i get this ... "Unable to join groutchat ... Reason: Conflict ... Access cannot be granted because an existing resource or session exists with the same name or address ... That nickname is registered by another person"
[19:50:56] <Timon> [groupchat]
[19:53:28] * 0xAFFE left the chat.
[19:53:33] * 0xAFFE joined the chat.
[19:53:46] * 0xAFFE left the chat.
[19:53:54] * 0xAFFE joined the chat.
[19:57:13] <deryni> Try with a different nickname? I don't know that I can see what the registered nicknames are.
[19:57:35] * MattJ left the chat.
[20:01:49] <Timon> deryni: yay ! ... i got in, thank you !
[20:04:56] <Timon> oddly, there's no Timon in ejabberd, but i got in as Timon2 ... maybe my psi is polluted with all the fiddling i've been doing, is there a way to reset it?
[20:06:35] <Timon> i didn't mean to post in bold red text, how does one do that anyway?
[20:10:17] <mathieui> like this
[20:16:48] * Tobias left the chat.
[20:17:59] * Tobias joined the chat.
[20:23:23] <dwd> Timon, Whenever someone says your nickname -- it's probably your client highlighting.
[20:27:23] <Timon> dwd: so my post above that begins with "oddly" appears as normal text to you? ... it's my client that's highlighting it for me?
[20:27:56] <dwd> Timon, That's probably it. Assuming this one is in red too.
[20:28:12] <dwd> Timon, Your message starting "dwd:" was highlighted for me.
[20:28:39] <Timon> dwd: yes it is, thanks for clearing that up
[20:32:20] <Timon> i've general questions about clustering, and have posted on ejabberd but so far no reply there, perhaps someone here can help ...
[20:32:58] <deryni> Clustering is really a server specific area of detail and the ejabberd muc is very very quiet most of the time.
[20:34:42] <Timon> may i post some of them here anyway? as they are of a general nature perhaps there is a general answer
[20:36:15] <deryni> You are welcome to try.
[20:37:37] * Lance joined the chat.
[20:38:17] * Lance left the chat.
[20:38:43] <Timon> when clustering xmpp servers is the DB generally on a separate server, distinct from all the xmpp servers? or does each xmpp server maintain an updated copy of the DB?
[20:44:53] <Kev> I imagine each cluster node maintains at least a shard of the data.
[20:45:01] <Kev> Otherwise you're at the mercy of the DB for consistency.
[20:45:12] <Kev> And reliability.
[20:45:51] <Kev> I only know one server implementation intimately, and in M-Link the nodes don't have external dependencies (unless you go out of your way to make it so, like using external AD for auth or such).
[20:50:19] <dwd> I understood that most rely on a clustered data store of some form (ejabberd relies on either Mnesia or MySQL for example).
[20:50:48] <dwd> But I'm only familiar with M-Link, really, and that as Kev says maintains its own internal datastore which it clusters itself.
[20:50:51] <Timon> Kev: that makes a lot of sense (for reliability you'd want each node to have a full copy of the DB), but doesn't the imply heavy inter-node traffic to keep all the nodes updated?
[20:51:25] <Kev> Not as much as you might think.
[20:51:37] <Kev> This isn't a naive 'share all stanzas between all nodes'.
[20:52:01] <Kev> It's only state changes that need to sync.
[20:53:37] <Timon> no, of course not, but the larger the cluster the more inter-node state changes that need to be synchronized ... what is a real-world limit on cluster size, in M-Link for example?
[20:54:18] <Kev> It would depend on the real-world scenario (and I don't know the answer for any).
[20:54:52] <Kev> If it's a mostly chat service, then presence is almost the only thing getting shared with any frequency.
[20:55:09] <Kev> If it's mostly MUC, then much more.
[20:56:05] <Kev> So...we haven't hit it. I expect that some day someone will manage to come up with a scenario where there can put more data through than it'll cope with, and then we'll presumably start sharding data - but there's been no need yet.
[20:56:20] <Kev> *they can
[20:57:42] <dwd> Timon, A key factor is what consistency vs availability requirements are on the data, too.
[20:59:03] <Timon> i'm a little fuzzy about how a given user's requests are handled in an xmpp cluster ... is a given user assigned to one-and-only-one xmpp server node or can that user's requests be handled by any node, via some form of load balancing?
[20:59:27] <Kev> Timon: You have a TCP session with one node.
[21:00:01] <Kev> Traffic that's heading to you (e.g. from other users, MUCs) can come in from anywhere in the cluster. It then is routed to your node, and out to your client.
[21:01:47] * Neustradamus left the chat.
[21:01:51] * Neustradamus joined the chat.
[21:06:10] <Timon> for example, if Today user Arnold is served by node-1 in an xmpp cluster of 10 xmpp servers, and he logs out ... Tomorrow is Arnold guaranteed to be served by node-1 or could he be served by any of the 10 nodes, based upon some form of load balancing?
[21:07:00] <dwd> Timon, At least one cluster system does, I think, operate by "putting" Arnold back onto the same node. And indeed putting Arnold's friend on the same node.
[21:08:00] <Kev> With M-Link you could connect anywhere, though.
[21:08:20] <Kev> And I don't think any servers do what dwd says out of the box.
[21:08:43] <dwd> Kev, No. I *think* I'm right in saying it's part of the commercial fork of Tigase.
[21:09:05] <dwd> Kev, But as you know, I've not had any experience with that server.
[21:09:32] <Kev> I think Artur does customised clustering strategies based on what his customers need.
[21:09:52] <Kev> Which is no bad thing, of course.
[21:12:04] <Timon> kev's model implies that an exact copy of all of Arnold's state data resides in the DB of each xmpp node, which also implies that state data must be synchronized in real-time among all the nodes in the cluster
[21:12:24] <Kev> Most of the state data, yes.
[21:12:53] <Timon> what portion of the state data would not need to be synchronized?
[21:13:35] <Kev> TLS/Compression data.
[21:14:50] <dwd> Timon, Well, a roster, for example - it might be surprising to a user if their roster "rolls back", but it's not a disaster, and the clster might be able to re-synch it on the fly without any apparent loss if done right. That's not to say one wouldn't try to keep rosters in synch, of course.
[21:15:33] <dwd> Timon, Similarly, offline messages - you could actually harvest those from each available cluster node when the user comes online. Not *ideal*, but it wouldn't hurt.
[21:15:59] <dwd> Timon, I'd stress that I'm not implying any server does either of those suggestions.
[21:18:42] <Timon> it makes sense that state data related to the session is not synchronized, because it needn't be as it will change with a new session ... but anything related to message or membership states would need to be synchronized, ideally in real-time
[21:19:08] <Kev> Yes.
[21:19:24] <Kev> Fortunately, XMPP servers are really quite good at shipping data around in realtime.
[21:21:46] <Timon> this implies that inter-node synchronization traffic will increase exponentially with N, the number of nodes in the cluster, and also implies an upper bound on the N, for a given M (the max number of users that each node can support concurrently)
[21:22:05] <Kev> For complete replication, yes.
[21:22:14] <Kev> I'm not pretending for a moment that it doesn't.
[21:23:19] * naw left the chat.
[21:24:35] <Timon> just trying to understand what are the upper bounds for a general xmpp cluster ... we do have some real-world examples of what that bound may be, at present anyway
[21:26:11] <Kev> I assume the answer is pretty much 'whatever you want it to be, as your vendor will make it work'.
[21:27:56] <Timon> google appears to be migrating to a binary protocol for xmpp traffic, or may have already done so ... i wonder if that is related to this upper bound we've been discussing?
[21:28:29] <Kev> I couldn't say why.
[21:29:20] * Neustradamus joined the chat.
[21:30:22] <Timon> in other words, to push N and M to their absolute limits one may need to minimize xmpp traffic and one way is to not use xml
[21:31:08] <Kev> It's important not to confuse the C2S/S2S protocols and clustering protocols here.
[21:32:19] * Neustradamus joined the chat.
[21:32:29] <Timon> well i'm talking about minimizing both, if you control the servers and the clients, you can do that, but you also break xmpp compatibility in the process
[21:32:29] * Neustradamus left the chat.
[21:33:09] <Kev> You're assuming that it being XML is the biggest scalability issue with a server, which I'm not sure is necessarily the case.
[21:34:47] <dwd> Timon, I think you may be making unfounded assumptions.
[21:34:56] * Neustradamus left the chat.
[21:36:13] <dwd> Timon, In particular, Google's Hangouts has just the same internal state as XMPP, and so (by Shannon's rules) would have the same minimal bandwidth.
[21:36:39] <dwd> I should say "as traditional XMPP", since much of it is still XMPP.
[21:37:18] * Neustradamus left the chat.
[21:38:46] * Neustradamus joined the chat.
[21:39:42] <Timon> i'm not sure xml is THE limiting factor, but it is a contributing factor to scalability, is it not? i mean, one could envision a more compact protocol, which would certainly reduce bandwidth in a large N cluster, given that we've determined that bandwidth increases exponentially with N in an xmpp cluster
[21:42:49] * deryni left the chat.
[21:44:16] <dwd> Timon, But the clustering protocol is internal, and *that* doesn't need to use XML.
[21:44:40] <Timon> btw, the PKR book (i think) mentions that google was leaning towards a binary protocol for its xmpp implementations, did that ever come to fruition, or is google still using xml for its xmpp?
[21:46:41] <Kev> Did we say that?
[21:49:49] * Alex left the chat.
[21:50:02] <Timon> i was assuming (probably wrongly) that the clustering protocol also used xml, but you're right, using xml for clustering is probably not generally done ... however, given that presence stanzas comprise 90% of the xmpp network bandwidth (from what i've read), a more compact protocol would likely help to push N and M to higher limits.
[21:51:39] <Timon> Kev: i'm not sure if the PKR book said that about google, but i did read that somewhere during the past day or two, and part of that reading was the PKR book, so i may be mixing my sources
[21:51:52] <Kev> I'm assuming by PKR you mean TDG.
[21:52:24] <Timon> Kev: yes indeed, sorry, i didn't know the proper acronym
[21:52:37] <Kev> Just 'the book' usually suffices ;)
[21:53:51] <Kev> I don't /think/ we said anything about Google wanting to use binary encodings in there, but it was a number of years ago, so I could have forgotten.
[22:06:48] * stpeter left the chat.
[22:07:50] * Flow left the chat.
[22:08:59] <Timon> Kev: not in "the book", but on the web: "The com.google.android.xmppService package has been replaced by the com.google.android.gtalkservice package. This is driven by the fact that the GTalk API is not XMPP compliant, and will be less so going forward. The reason is that XMPP is too verbose and inefficient for mobile network connection, and the GTalk API will be moving to a binary encoding for the protocol between the client and the server."
[22:09:46] * 0xAFFE left the chat.
[22:09:49] <mathieui> yuck
[22:11:15] <Timon> The above is from: 14 Feb 2008 by Anders Conbere, "A post floated by the XMPP standards mailing list today about a change to the android spec involving their use of XMPP."
[22:11:47] <dwd> That was 6 years ago. :-)
[22:12:18] <dwd> And turned out to relate to a remote debugging thing, they claimed at the time.
[22:12:46] <dwd> BTW, what does "PKR" stand for?
[22:12:58] <Kev> Peter Kevin Remko, I'm assuming.
[22:13:11] <dwd> Oh. Yes, that makes sense now.
[22:13:17] <Kev> Took me a while, too.
[22:13:29] <dwd> I couldn't think what the "K" might be. :-)
[22:13:33] <Kev> But I have an excuse, I wasn't one of the reviewers :p
[22:13:40] <dwd> Well, indeed.
[22:13:42] <Timon> dwd: does google no longer use the binary protocol for their xmpp? ... and yes on PKR, sorry if it caused confusion
[22:14:54] <dwd> Timon, So I think they used a proprietary protocol between Android and GTalk in the end. But as far as I know, you can talk to any Android device over Google Cloud Messaging using XMPP to the device, so it's all a bit confusing.
[22:20:15] <Timon> maybe android supports both, but uses binary "in-band" and xml "out-of-band" ... where "band" is their proprietary mobile "xmpp" protocol
[22:28:40] <dwd> Timon, BTW, the measurements I looked at a few years ago suggested that IQ, not presence, accounted for the most traffic. It's possible that's changed, of course.
[22:28:59] <Kev> That was including pings, mind.
[22:30:54] <dwd> Yes, true. I'd forgotten that. I think I decided much of it was disco, too.
[22:39:36] * Tobias left the chat.
[22:39:54] * 0xAFFE left the chat.
[22:45:32] * Tobias joined the chat.
[23:00:28] * deryni joined the chat.
[23:12:46] * Florob left the chat.
[23:29:25] * stpeter joined the chat.