Some good questions there. To answer your question about caching, it's currently not possible due to the fact that whispers are present on the main dicussion pages and as such they differ between users.

Though I guess Vanilla could cache things like total post counts for each topic so it doesn't have to query that as often. I know that vBulletin does that (well at least has the option to do that).

!!! I don't know anything about load balancing,

Vanilla use the php built-in session manager. By default php save the session data in local files but you can set php to save them in a database..

Thank you all for your answers.

It's a shame about caching. After looking into Vanilla code more, I see it would be really difficult to get right - especially given the extensible nature of the product. Plus it's not really aimed for large forums. Fortunately, throwing more hardware at the problem suffices.

Dinoboff, I think you are talking about the session_set_save_handler() routine to define user-defined session storage routines, right? Thanks for the pointer. You're right that, provided the callbacks are established before session_start(), I can easily override the storage of the PHP Session ID, and redirect storage to (for instance) a database. PHP itself still creates that Session ID (with extreme low-prob that it will conflict with another server) - but you can simply ignore it. You can use custom cookies and implement everything yourself.

It may be a shock to Vanilla users ... but Databases are slow man. You wanna avoid them - esp on high traffic sites where load-balancing is required. In my case, I'd like to integrate Vanilla on a high-traffic site that already has an efficient client-side session mechanism. Retrieving session variables from the Vanilla database on every page (to enable a user access to other parts of the site) isn't acceptable. Most discussions here relate to molding a site to Vanilla, whereas I'd like to do the opposite.

So - for others wishing to do the same, check out session_set_save_handler(). I haven't tried, but I think it's the cleanest way to alter Vanilla's session handling.

A see a quick'n'dirty way also : achieved by using Vanilla's "PersistentSession" (remember me) logic, and assigning different session names (the default is "PHPSESSID") on each servers. A persistent session is stored in the database, but only retrieved once for each web-server (which creates its own session, with a unique session cookie name). The "remember me" would have to be forced on, and the logout code needs to clear all the web-servers' session cookies .. and session data can't change. But - it's quick.

Hey Stash - yes, you're right. You'd just have to delete all the session cookies and the database would be read again using the same "PersistentSession" mechanism. Damn you guys are smart. :)

@tomtester : I have a few suggetions for you.

I found apples and oranges in the forum world. let's call them "bad apples" and oranges. The bad apples all look alike - I find them unacceptable from a user-interface perspective (and not easily corrected). These are written by the engineer, so a lot of them are built to scale (phpBB for instance). For the oranges, I found only two : Vanilla and bbPress. I stopped looking when I found Vanilla, there may be more. So suggestion 1 is : look for more here - http://en.wikipedia.org/wiki/Comparison_of_Internet_forum_software

Some form of caching is built into bbPress (I'm not sure how much) - but from the little I know, the caching is broken right now. bbPress doesn't index keywords, but does use the "fulltext" feature of mysql. It is very likely than bbPress will scale much better than Vanilla. But the main problem with bbPress is that it is still being developed. It isn't as slick as Vanilla, but that may also change down the road. So suggestion 2 : check bbPress out.

Another is to build caching in Vanilla. I'm real new here, so take what I say with a grain of salt (everything below is 'my opinion', not fact) :

While I respect and consider what others say, I actually think building a caching mechanism in Vanilla is possible - just pretty darn hard. I'll stick to discussing database query caching, since that interests you. So, there's smart caching and dumb caching :

Dumb Caching : As you point out, building a "dumb cache" is dirty but very effective. it involves caching data for a page and blindly using it for a set period of time (you used 5 seconds). It said it can't be done because of "whispering". i think it just changes the implementation of the cache. Take the main discussions page. When Bob is signed-in, he sees all the discussions like most people see them, but popular Sally (who gets a lot of whispers), sees a different list. To do effective caching, you need to cache per user. Imagine the cache is a directory structure, the first directory is the user_id and cached data held under them. So when Bob accesses the main page, a cache file is generated and put into his cache directory. When he refreshes the page, his cache file is fed back to him. Same for Sally. You could argue that dumb caching per user doesn't help, but visitors will count as one user - so all visitors share the same cache file. Still - it's pretty weak. If you turn off whispering, maybe you don't need per user caching (but I still think you do - see my note below about this).

Smart Caching : This is where you cache data and use it until the data changes. This is done by purging the cache file (aka invalidating the cache). The trick is to detect when the cache should be invalidated. If whispering is turned on, and caching is required per user, invalidating everyone's cache files becomes a challenge. It's not acceptable to go remove a thousand cache files (one per user). So a global cache directory should be made, and timestamps compared as part of cache fetching. Per user caching would be very effective in this case. But if whispering is turned off, per user caching *may* not be required (but I still think so) - and would greatly simplify the implementation.

Up till now, I've been talking about caching data for 'pages' - when in fact the code gets data in terms of a database query. For dumb or smart caching, the cache has to be keyed by the inputs to the database query. Fortunately, the code is already geared for this. Database queries are constructed by calls to the SqlBuilder class. The Select() and GetRow() calls of the Database class are the points to implement the caching mechanism. The cache filename is a construction of all the inputs to the database query. This all sounds easy until you look at the vast number of queries that take place in Vanilla. And some of these inputs may be things like the current time or others that constantly change. So it has to be somewhat smart. It's kinda scary. Cache invalidation would have to happen in the same place - calls in the database to update the data. I think only Mark could write the algorithm to match select inputs with update inputs to do cache invalidation. It looks way too complicated. I doubt he would be interested to do this, it looks really hard.

The other option is to cache data at certain places outside the Database class. But this requires invalidation to be done at certain places too. Since it is not a generic cache, it'll likely break and be difficult to maintain.

so, the note about whispering : The other thing I see unique per user is the highlighting showing which entries have been read and which are new. This is done via the LUM_UserDiscussionWatch table. When a user views a discussion, a timestamp entry into this table is updated. So the main discussions page will still be a unique database fetch per user, even without whispering activated. So I think that feature makes per-user caching mandatory also.

So my suggestion 3 : don't even try to implement a cache in Vanilla. In writing this up, I've convinced myself that it's just too hard man.

Suggestion 4 : search for "FARM_DATABASE_HOST" - I don't know much about it, but it looks like Vanilla somehow has built-in support for a database farm. Combine that with web-server load-balancing, and your forum scales - though at a cost.

WallPhone - can't wait to hear what you come up with.

I thought of something else :

It seems the two features of Vanilla that hold back (global - not per user) caching is whispering and the visual cues that depict whether or not you've read a discussion. If you disable both of these features, there's nothing to stop caching the data for the discussions page for both guests and users. Am I right? Everyone knows how to disable whispering. The quick way to disable the "have I read this" cue is to change the style-sheet. Your users will learn to use the forum without that feature.

In this case, caching shouldn't be so hard. It would be better to invalidate the cache, rather than have a short-timeout. You just need to identify the points of code where invalidation needs to take place ... e.g. when a comment is added, a sink is done... etc. Grep for Update - can't be too hard, but there are probably a lot of these.

The same technique may work for the data associated with viewing each discussion - but I haven't looked into this. It may involve user-specific database queries (even with whispering disabled?). not sure.

With the visual cues disabled, it would also be beneficial to disable the database update to the LUM_UserDiscussionWatch table that takes place every time a user opens a discussion. In a replicated database environment, write rates are the limiting factor. I'm guessing this would help a lot.

"Some good questions there. To answer your question about caching, it's currently not possible due to the fact that whispers are present on the main dicussion pages and as such they differ between users."

Present whispers as a new tab\page?

silly comment - self-censorship activated

Actually, this is the UK, the Ukraine, Australia, Germany... but hey ;)

aah,

but the strength of whispers also comes from the fact that these look like ordinary comments and you post them just like ordinary comments (same look, same logic, as clean as)?

I know/agree y2/Toiv *but* if the present method prevents caching of forum pages, would you prefer to:

A. - Live without Whispers altogether
B. - Live with Whispers on a Tab (private messages)
C. - Live with *another* implementation of Whispers that looks the same but is implemented differently (e.g. via browser-level 'comment/whisper-interleaving')
D. - Live with the current Whisper stuff and screw all those speed-freaks

Of course I'm being facetious here... but I really believe the present Whisper implementation (comments
marked as private) is sub-optimal... I *think* it limits page 'cache-ability' and also at the root of several
'positioning' errors of various extensions.

Also note: initially I did *not* get the benefit of the Whispers over Private messages. I think Whispers
are one of the more difficult things to convey ton Vanilla-'noobs' and experienced 'other forum'-users.

In that context it may make sense to re-think the Whispers too.

It may be a shock to Vanilla users ... but Databases are slow man. You wanna avoid them - esp on high traffic sites where load-balancing is required. In my case, I'd like to integrate Vanilla on a high-traffic site that already has an efficient client-side session mechanism. Retrieving session variables from the Vanilla database on every page (to enable a user access to other parts of the site) isn't acceptable.

Too right! I've been telling people that about databases for a long time, but they just don't want to hear it. All of Vanilla's configuration settings are in php-editable files for precisely that reason: I don't want to have to pull those out of the database on every page load. I think you'll be hard-pressed to find any database driven forum that doesn't call the database for some data on every page - if it's not user related information, it will be general configuration settings.

I made Vanilla pull the user's id and role information on every page load so that permissions are always up to date. If I ban you, you won't be able to see the forum the next time you load a page. Other forum packages have problems with things like this - you ban someone and they stay signed in until their session expires and they have to re-authenticate (either by re-signing in or by cookie).

Certainly sounds interesting. So this basically puts the strain on the end user rather than the server? I wonder how this affects mobile devices?

(PS) Of course I *never* even tried Vanilla on my BlackBerry... Color me impressed,
it's darn fast *and* pretty. The only problem I see is a repeat of the USER ICON
causing a whole line of 'stashes'

What kind of performance are others getting out of Vanilla? When you're talking large-medium-small scale, what sizes/levels of activity are you actually talking? It would be nice to get at least a ballpark estimate of the volume of traffic, concurrent users, other measures of performance that you're talking about.

Maybe a bit about the processor/memory/OS you're running as well.

I too am evaluating Vanilla for a larger installation. It would be nice to know if we're on the same scale with one another.

I'm assuming you were using a shared server? How many users did you have? And how many posts a day?

Do you have access to your slow query log?

How many concurrent active users/pageviews do you have on vanilla? This site runs on a far less powerful machine than yours (nice boxes the G5's, we have a few at work for virtualisation. lots of shiny blue lights too :)) and obviously is running fine. Since it has double the number of discussions and presumably comments that seems a bit strange - though yours could be a LOT more active just newer, if you see what I mean. I know this box also hosts 5+ other sites but I'm not sure exactly how many.

The only extensions I can see on there which might be worth looking at (i.e. disabling, see if it makes a difference) are ParticipatedThreads (though i believe this runs on a seperate page so shouldn't be too bad unless users are using it a lot), maybe PrivateMessages (havn't checked out how this works but generally speaking whispers put a lot of load on DB calls - i guess in theory PM should reduce that load but I'm not 100%).

Which queries are they that are taking up all the power? I'm guessing probably the one for the all discussions page?

Yeah I suspect having a 27k comment discussion would hurt a bit.... Can you try closing that discussion for a while and see if it helps?

Is there a way to auto limit threads ,or shut them down when they get too large?

Must have been a helluva lot of robot activity! Glad you got it sorted though...