The secret sauce files

For all software that I write, I am an immense fan of the “Eat your own dog food” dogma. It tells that you should be willing to run your own software on your own systems. It’s the perfect way of finding out if this new dog food that you invented really is as good as you think it is. As long as you do not want to eat your own dog food, then there is probably something wrong with it.

Applying this rule to Phorum development, it means that I always try to keep up with the bleeding edge development tree for my live forum web site. This is of course not for the faint of heart and I do not recommend anybody to start doing this, unless:

• You are an experienced Phorum developer;
• You do not panic when things go “boom!“;
• You have a user base which is prepared for the worst.

Especially the users are an important factor here, since forums involve both a lot of users and a lot of user functionality. If the users are not prepared well for the fact that occasional bugs can occur, they might get angry with you and leave. To keep this from happening, here are a few things that you can do:

• Let your users know that they are used as lab rats.
  Before I switched my forums from some old Phorum 3 version to the active 5.2 development tree, I explained to my users that they would become an important part of Phorum development, because they would always be using the most up-to-date Phorum system in the world. Therefore they would be the first ones to run into development bugs.

• Let your users know that being a lab rat is cool.
  Users will not only be the first ones to run into bugs, they will also be the ones to run into brand new features. Since development is spread over time, these new features will not arrive all at once. We all know that in the end it is more fun to slowly eat a bag of candy than to eat a full bag in one go. The users will also be the ones to be using new features, long before they become available through the stable software releases.

• Always inform about upgrades that you have done.
  I always let my users know when I upgrade some part of the system. If the upgrade involves a visible change I tell them what changes for them. In my experience, users tend to report more bugs if they can respond to an upgrade notification. I often have received bug reports as response to an upgrade notification, which did not at all relate to the specific upgrade.

• Always respond to bug reports.
  Do not let your users linger after they reported a bug to you. Users that report bugs are valuable and you should treat them as such. Try to solve bugs on short notice and report back about the progress. If you cannot solve a bug immediately, be sure to let the users know that you are working on it.

A lot of this info might seem rather obvious. I have seen projects however, where user care was not really high on the agenda, causing all kinds of user frustration. I hope this info might be useful to others who want to involve their user base in their software development process.

The last couple of days, I have been working on the “sphinx_search” module. This is a Phorum module which implements the Sphinx search engine as the backend for the Phorum search interface. This search engine is truly amazing for its search speed. Its query parser however, is not yet fully matured and has some shortcomings therefore. This caused us some troubles when we tried to use the extended query syntax for defining searches on both the body and subject of the forum messages. In this article I will show what work arounds were implemented in the “sphinx_search” module to get around this. Maybe these examples are useful for other who are using Spinx as well.

Search on all words in the query

We started out with the following format for defining a query “search for bodies that have both word1 and word2 in it or subjects that have both word1 and word2 in it”:

@body word1 word2 | @subject word1 word2

This appeared to be based on a wrong assumption. We found out that the OR symbol “|” takes precedence, making this query look like this for Sphinx:

@body word1 & (@body word2 | @subject word1) & @subject word2

This is not what we were looking for. So to make the query work like we wanted it to, we needed to group the query terms in the query like this:

(@body word1 word2) | (@subject word1 word2)

Unfortunately, this query syntax is not allowed by Sphinx (yet), so we had to find a different solution. After some experimenting, we ended up with the following query:

@body word1 | @subject word1 & @body word2 | @subject word2

This query looks like this for Shpinx:

(@body word1 | @subject word1) & (@body word2 | @subject word2)

Using this query we were able to get some good results. Note that this query is a little bit different from the query that we tried at first. For our application, this difference is no problem however. In fact, this latter query syntax makes the “sphinx_search” module behave the same as the standard Phorum search (which searches globally over the subject and body), so it is actually better in our case.

Search on any words in the query

When searching on any word in the query, things are easier. For this query it is possible to simply use one of these two syntaxes:

@body word1 | word2 | @subject word1 | word2

@body word1 | @subject word1 | @body word2 | @subject word2

Although the first notation is somewhat more elegant, I settled for the second one. That one is closest to the syntax that I ended up using for the “all words” query, which made it easier to program the quiry builder in the module.

Search on negated words

Using negated words, some words can be excluded from the search. So you could for example search for “word1 -word2″ in the Phorum search. Again, we missed the option to group parts of the query, disallowing us to do the following query:

(@body word1 -word2) | (@subject word1 -word2)

To make this query work, we finally came up with the following syntax:

@body -word2 & @subject -word2 & @body word1 | @subject word1

This query will be seen by Sphinx as:

@body -word2 & @subject -word2 & (@body word1 | @subject word1)

Again, not a really elegant query syntax, but at least it can be parsed correctly by Sphinx this way and it delivers the required results.

Hello. My name is Maurice.
I am addicted to Phorum programming.
“Hi Maurice!”.

Finally, as the last one of the Phorum core development team (you can now stop pointing at me and calling me names like “troglodyte” and “cavemen” guys), I started a blog. This is where I will be blogging about my Phorum addiction and about my attempts to stay addicted as much as possible.

I am a huge fan of the Phorum module system and have built many modules that make use of it already. So one of the things that I want to talk about here, is the way in which I solved certain module writing issues, to make other module writers benefit from it. Also, if you are a module writer that is running into troubles with implementing a certain feature as a module, then please drop me a line and explain your problem to me. If your problem is a general module design issue, then I will be most happy to write an article about it to help you out.

Secret sauce?

So what is “Secret Sauce”? Well, it’s a concept that I picked up last year when the Phorum team was able to meet up at the MySQL conference in the U.S. One night we went out to grab a bite and I was totally stuck at choosing something to eat from the menu. Just when I was prepared to order anything at random, my eyes were drawn towards two words that seemed to be highlighted by some devine glow: SECRET SAUCE. For no other reason than me liking the idea of secret sauce, I ordered the sandwich that went with it and later on adopted the name for referring to my Phorum development activities.