So, after a few minutes of googling I came across an entry at answers.launchpad.net[0] and a blog post, but I cannot seem to remember the URL to that one. I can imagine that some of you might be having the exact same problem, so the solution is holding down the ALT, whilst dragging as usual.

[0] https://answers.launchpad.net/ubuntu/+source/gnome-panel/+question/264

The root of the problem was my setup running PHP as a separate FastCGI process. Unfortunatly it seems as if PHP can only handle 500 requests per FastCGI process and then seems to lock up.
The old setup of this site didn’t cause such problems and it seems the problem lies in not setting the PHP_FCGI_CHILDREN and PHP_FCGI_MAX_REQUESTS environment variables with the new setup.

I initially thought that the default values of those environment variables were safe, but they are not. As I already wrote PHP seems to lock up after 500 requests and the solution lies in changing PHP_FCGI_CHILDREN, which defaults to 0 (no additional processes) to something bigger than 0 (I am using 2 children to make sure I have at least one PHP process reading for answering requests at any time).
Why? Quite simple, if you increase the value the PHP root process becomes some sort of manager and delegates requests to the children, as expected. However, using PHP_FCGI_MAX_REQUESTS it only forwards the specified number of requests to a child process before killing it and starting a new one. Problem solved.

Information on this behaviour can not be found in the PHP online manual, but rather at http://lxr.php.net/source/php-src/sapi/cgi/README.FastCGI.

Now you probably expect me to come up with a pure technical reason showing superiority of Free Software, but I am taking another path this time: let’s talk about user trust.

Software updates on pure Free Software operating systems

Firstly, let’s take a look at how security updates are deployed on pure Free Software operating systems.
All major GNU/Linux and *BSD distributions come with built-in update systems for all Free Software packages provided by the distribution. Once a security update to any piece of software is released you will get a notification saying updates are available and that you should install them and after doing so your system should be in a quite secure state.

Updater-applications of proprietary software

Okay, proprietary software does often come with its own update mechanisms, giving you pretty much the same functionality. But here is my point: users tend to block these updater-applications from accessing the network with personal firewalls and similar tools, which basically disables the updater and most likely will cause the system to be more vulnerable rather sooner than later.

But why do people even consider blocking an updater-application?

When I asked people why they blocked these applications a lot of reasons came up, but it usually boils down to lack of trust for proprietary software vendors. People seem not to trust software vendors for a variety of reasons. The most common reasons seem to be that they are either worried about the disclosure of private information or the lack of a license for a piece of software, combined with the first reason.

So people actually seem to be scared by what a piece proprietary software could do, and when not having their software licensed people seem to be even more scared. When I ask people why they are running these programs, even though they do not trust their vendors they usually shrug and I get replies that can be summed up as “it just works”, “but I don’t have a choice” and “I am used to software X”.

The solution is Free Software

Free Software can be a solution to all these problems. The most important thing people should be aware of that they do not have to fear Free Software vendors. After all, when using Free Software you do have a valid license and you can, at least in theory, check exactly a program is doing to your system. Also, if you are not capable of doing such checks yourself, you can rest assured that other people are doing such checks and give back to the project in question, improving your favorite Free Software applications.

Free Software is a choice, and there is hardly any proprietary piece of software which cannot be replaced by a free equivalent. Free Software is my first choice, not only for me but for my family too, with great results so far.

Rest assured you can trust in what your software is doing, free yourself today – use Free Software.

So I guessed the lighttpd process itself must be experiencing problems of some sort, but after doing a “netstat -nat” I  knew what was going on:

tcp6       1      1 83.65.62.72:80          61.135.190.248:12474    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.234:39671    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.253:39211    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.234:55160    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.230:25836    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.231:16865    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.232:24266    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.240:38441    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.243:17726    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.241:38206    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.251:23892    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.225:29675    LAST_ACK

Plus “a few” more of those. Now I’m not entirely sure whether it’s just some systems misbehaving or actually an attack, but my feelings told me this could have been intentional after all.
I did a quick whois on one of those IP addresses and came up with the 61.135.0.0/16 network which is owned by China Network Communications Group Corporation.

As the connections were made from pretty much every host in that network I had two choices: sit it out or block it.

I came to the conclusion that blocking the entire subnet from connecting to this system, at least temporarily, might be a viable solution and so I did.
However, afterwards I am asking myself whether I really had to block an entire 16-Bit network, so I am asking you: how do you handle such situations usually?

The AC_CHECK_PYTHON_MODULE macro takes two arguments: The module name and optionally the variable name holding version information. This way it is not only possible to determine whether a module is installed (ie. loads in Python) on the current system, but also retrieve version information from that module.

The following examples checks whether the Crypto module is installed and retrieves its version information from Crypto.__version__:

AC_CHECK_PYTHON_MODULE(Crypto, __version__)

The macro itself does never report and error, but rather only a found/not found result. Error checking is up to the user and can be done via these two Autoconf variables:

• PYTHON_<MODULE_NAME>
• PYTHON_<MODULE_NAME>_VERSION

PYTHON_<MODULE_NAME> is set to “1″ if the module is present and “0″ if not present.
PYTHON_<MODULE_NAME>_VERSION is only set when the version variable argument has been set and contains the version information of the module, if the module been found. If the module is not present this variable is also set to “0″.

The version variable argument is optional as I wrote, so the following invocation works too and only checks whether the distutils module is present:

AC_CHECK_PYTHON_MODULE(distutils)

As I wrote earlier in this article I would like to share this macro with you. You can download it here.

The first article can be found here and points out flaws and problems in the current systems.

Before going into detail about how the infrastructure could look like I would like to point out the goals of my proposal:

• security through end-to-end encryption
• security through sender and server authentication
• integrity of message contents
• built-in load-balancing support
• getting rid of email forwards

These five major points should be covered directly by a new infrastructure and should be mandatory. There is no point in making any of these optional as the rest of this article should point out.

security through end-to-end encryption

Even though both SSL and TLS support exist for (E)SMTP these features are optional. In fact this means that it is possible that even though one submits his or her email over a secure channel the message could be transferred in plain-text somewhere on the way to its destination.
This enables an attacker to snoop at your message somewhere along its way. Whilst some people believe this is okay I strongly oppose to anyone being able to read either my private or business emails.

The solution to this problem is end-to-end encryption. The new infrastructure should make encryption of all message exchanged mandatory and further provide a way of encrypting the message contents. This way only the intended recipient can actually read the message (as in not even a server administrator having direct access to a user’s mailbox). End-to-end encryption of the communication channels should be done by using TLS for all communication between all clients and servers and for server-to-server communication.

Encrypting of the message payload could be done in a similar (if not even the same) way OpenPGP (RFC 4880) works.

security through sender and server authentication

The next feature a possible SMTP successor should provide is sender and server authentication. As TLS should be mandatory for the implementation the easiest way to achieve this is using a public key infrastructure. This could then in turn be used for multiple things, including message integrity checking, encryption of message contents, authentication of the sender and authentication of the server.

Integrating a public key infrastructure could be done by having special DNS (maybe TXT) records that contain the address of key servers. These key servers would store not only a domain-root certificate which would allow user and server authentication but also all user and server certificates themselves.

A receiving server could then check the sending domain’s key server for both the domain-root certificate and the sending server and thus verify that the message is legitimate and actually originated from the specified domain.

Sender authentication works together with message integrity. Basically the receiving server opens the message, gets the client’s message signature from the message, and asks the sending domain’s key server for the public key of the sender. The receiving server then checks the signature and this way verifies the sender.

integrity of message contents

Integrity checking is closely related to sender verification. As the receiving server checks the sender’s message signature in the sender-verification process the message is automatically checked for integrity too.

built-in load-balancing support

Load-balancing is also closely related to the PKI approach. The sending server could use the receiving domain’s key server to locate the server to send the message to. This way load-balancing of receiving servers can easily be implemented. Furthermore load-balancing of multiple key servers for a single domain is possible using DNS round-robin records.

getting rid of email forwards

Forwards can also be gotten rid of by using the receiving domain’s key server similarily to the load-balancing approach. Instead of pointing the sender to a domain-local receiving server the key server could simply point the sender to another domain’s receiving server. This way the message would not really be forwarded or relayed anymore but rather a pointer to where the message should be stored could be provided.

Putting everything together

Making all the mentioned features mandatory for a possible successor of SMTP should make users benefit in a few ways. Firstly, users could rely on both then integrity of the message, that the sender actually the person he or she pretends to be and the fact that snooping on the contents of the messages they send is hard to impossible.
Furthermore this infrastructure should make sending SPAM messages a lot harder as domains for sending spam would have to be bought, DNS servers and key servers would need to be operated and blocking unwanted messages could be as easy as blocking either a domain or a single user using the information provided through their message signature.

ISPs would benefit from the built-in load-balancing mechanisms and the mailbox alias feature (forwarder). Whilst the load-balancing technique simplifies set-up and operating of a load-balanced infrastructure the mailbox alias feature should help cutting down on traffic generated by email forwarders.

Please be aware that I intentionally left out all implementation specific details, such as the message exchange protocols. More technical aspects of a possible implementation are to be covered in the next parts of this series. As always, comments are highly appreciated.

Just think about it for a moment. About 20 years ago you got specifications for pretty much every piece of hardware you bought. You were given exact instructions on how to use the hardware you just bought, not only how to install it. Things have changed since then.

If you buy any piece of hardware today you actually have to expect not to get any documentation on how to “talk” to your new toy. You are only given a CD (sometimes even only a link to a homepage) containing drivers for a few specific operating systems, usually only Microsoft Windows.

Now I am no driver hacker and so I probably wouldn’t be able to implement a driver for anything on my own anyways, but the Free Software community would largely benefit from hardware documentation, as there are a lot of capable driver hackers out there.

This is not a problem that only affects the Free Software community though. There are a lot of pieces of hardware which do not work on recent proprietary operating systems anymore due to lack of support by its manufacturers.
At least this problem would not exist for Free Software operating systems, such as GNU/Linux, if hardware makers would publish documentation of their hardware. The people still using devices which are well beyond their end-of-life could implement drivers on their own, not being dependent on anyone.

What I am really wondering about in this case is why hardware companies are unable to coin standards for accessing devices of the same class. It works perfectly well for USB (take USB mass storage devices as an example) and I do not understand why there can’t be standardized interfaces to other hardware, such as network adapters, as well. On a very-low level these standardized interfaces do work. Just think of PCI, PCI Express or AGP.

Actually, if you think about this for a few more seconds you should realize one thing: Having standardized interfaces for devices of the same class would cut a lot of costs for hardware makers. Why? Oh well, if they design a brand new networking chip and still implement the given standard there would be no need of writing a new driver. Wait, there would be no need for per-device drivers at all. Implementing a common driver that accesses the standardized interface would be enough, for a whole range of devices.

So what am I asking of hardware makers? I would love to see companies creating devices of the same class to get together, create standardized interfaces, publish them and implement them in their new devices.
I know, this is not likely to happen anytime soon, so a more realistic approach is asking for Free Software drivers and/or documentation.

Personally I have stopped buying hardware which “works” with GNU/Linux, I have come to the point where I try only to buy hardware which either comes with Free Software drivers from the manufacturer or documentation which allows implementation of Free Software drivers.
This is probably the best way of showing these companies what you demand: Freedom.

However, few know how old SMTP actually is, and that, even though it serves everyone well, it has been designed in a time when everyone was thinking of Spam as canned meat. Back in 1982 SMTP was a great achievement and a lot of kudos should go to its creators, but now, in 2008, SMTP has become more of a liability than a great tool.

Originally, I wanted to write a single article covering all shortcomings of SMTP and possible solutions to these problems, but while writing the article a lot of text came up, so this is the first of two articles I am going to write on this topic. The first part is about the problems with SMTP and how fix-ups for SMTP are, even though they do work to some extent, a proper solutions to today’s issues.

Due to the way SMTP was designed and the way the Internet was back then it is prone to various things, like SPAM messages, sender spoofing, data manipulation and so forth. A few attempts have been made at fixing some of the shortcomings of SMTP, like ESMTPA (SMTP-AUTH) or SPF, Callback Verification, and DKIM, but none of them has really fixed all problems that exist and all of these modifications are in my opinion mere workarounds. Let us have a look at why both SPF and DKIM fail to fix the all problems SMTP has right now.

SPF

First let’s have a quick look at how SPF works: When an email is received a special TXT DNS record at the sender’s domain is used to verify that the sending computer (using its IP address) of an email is actually allowed to send email for a domain. A great mechanism that in theory would work perfectly well. Reality is a bit different though.

There are some domains which are actually using SPF and do have valid SPF records on their DNS servers. However, those are only some of the millions of domains on the Internet. How should one treat emails coming from a domain without SPF record? The messages could be real, non-SPAM messages, that should be delivered, but on the other hand these could be SPAM messages. Also, the more people start using SPF, the more likely it becomes that spammers are simply going to use sender-domains which do not have SPF records.
Also, there are some organizations that have domains with improperly configured SPF records and there are even well-known ones, such as Technorati (I covered this in one of my articles). So one cannot even trust SPF records and valuable messages could be lost if a mail server is configured to drop all messages for which SPF authentication fails.

And there is a third problem: Sending emails from other places than your default one (office, home, etc.) and ISPs not allowing external users to use their SMTP servers (even not with authentication). A good example of this would be Austrian ISP UPC (their SMTP server tells me that the AUTH extension was not advertised, even though it was; long story short, I cannot log in from outside) and I am quite sure there are a lot of others.

And I can come up with yet another problem: What about email relaying? Think about downloading all messages from all your email accounts into a single one, using fetchmail for example. This makes SPF useless, as no checks can be done anymore, due to the sending system’s IP address not being the original sender. If one assumes that every mail server uses SPF this is not a problem, but I like doing my checks on my server rather than relying on some other server.

Maybe there are even more problems with SPF, such as what to do when an email is received from a nonexistant domain or when there is a temporary DNS failure on either side, but the ones listed above are those I am confronted with most often.

Callback Verification

On to the next topic. Callback verification is a simple method used by mail servers to try verifying that the sender actually exists. Whilst this works for some SPAM messages which use non-existent senders, it does not help much as soon as the sender address does exist, and it does not even matter if the message was actually sent by the user owning the address. I guess there is nothing to add, even though it is a nice method to get rid of some spam it does not help with a lot of such messages.

DKIM

DKIM, or DomainKeys Identified Mail (originally named DomainKeys) is a method that is not meant to prevent abuse (such as SPAM), but rather to make tracking abuse easier. It works by the sender (usually the MSA of the sender on behalf of him) adding another header, “DKIM-Signature”, which contains a cryptographic signature of the message body. The signature is generated using public-key cryptography, where the public key is stored in a DNS TXT record and can thus be used by the receiving end to verify that the message contents have not been tampered with during transport and that the mail actually originated from that domain.

This method, even though being one of the most advanced ones today, is prone to replay attacks and does not protect from tampering with message headers. In short this means that even though the message body cannot be modified without the receiving end detecting the modification, the headers can, and thus the message can be redirected. Also, it is possible to intercept the transmission of a message, generate a thousand messages with the same content but a different recipient and this way flood a mailbox with a message that would stand DKIM verification.

The email relaying problem that’s present in SPF is not a problem here anymore, but the mobile-mail, the nonexistent domain and the DKIM-not-in-use problem still exist.
Also, DKIM seems not to be used by a lot of email servers on the Internet. Thinking about it for a second I can just come up with two names of well-known organizations using DKIM: Google and Yahoo.

The message format

The next part of this article is about the SMTP message format. This part is not directly related to the SPAM problem, but should provide you with some more information that verifies that SMTP is outdated nowadays.

RFC822 messages (or emails) usually consist of two parts: a message header and a message body. Originally these messages were designed to contain 7bit-encoded ASCII data, which is plain text. This means that there were only 128 different characters which could be transferred via email, without support of special characters, like German “Umlaut” characters. A solution has then be developed, not only to support special characters in emails, but also to support transferring of binary data (such as images).

MIME is the name of this solution, and it enables every one of us to send binary attachments and special characters via email today. MIME allows the email client to include more than just 7-bit plain text messages, including attachments. This is achieved by special header, “MIME-Version“, which indicates that the contents of a message are MIME encoded. This header is then followed by a “Content-Type” header, identifying the type of content. For simple messages just consisting of a message body this would be “text/plain”, telling the client that there is just text in the mail.

However, how can emails then consist of both text and attachments? Well, there is a special value for the “Content-Type” header: “multipart/mixed“. This one indicates that there are several parts of a message, and every part comes with a separate “Content-Type” header. This way contents of a message can be organized in a tree, for example, containing the message body and a forwarded message.

An attachment is added by specifying an additional part of the message, usually with a “Content-Transfer-Encoding: base64” header, that says that the data has been base64 encoded. This way binary data can be represented using 7-bit ASCII.

But what does that mean? First of all, even though a message is split into several parts, there is only one body. Now if you are downloading a message via POP3 for example, there is no way of only downloading the actual text. You always need to download the whole message. Everyone knows this situation: You are downloading a message and have to wait for all attachments to be downloaded, even though you might not be interested in those attachments at all.

Also, encoding binary data using base64 creates a lot of overhead, as every byte (which can have 256 different values and corresponds to a single letter of text in 7-bit ASCII) is represented using only 64 possible values. Talking numbers here this means that messages encoded using base64 are usually 137% the size of the data they contain.

Conclusion

I hope that I have shown you what the problems with SMTP are right now. SMTP initially was designed to transport only text and had no way of verifying either the sender of a message or the integrity of data. Some workarounds have been created to get rid of these issues, but even though some helped a lot, none has really fixed any of those problems.

Also, one should never forget how much time and money has been used to try fixing SMTP, whilst a lot less money might have been sufficient for creating something new, something better, something that is built for the needs of the Internet today, and not for the needs of the Internet back in 1982.

Personally I believe that the days of SMTP are long over and that there is need of a proper replacement. I do understand that SMTP and the current email infrastructure are still in use because an infrastructure exists, but SMTP really deserves being retired, after serving us pretty well for more than 25 years.

The next part in this series of articles will be about what my idea of a successor of SMTP and the whole email infrastructure is, what it could look like and also how it could work. So stay tuned.

To sum the problem I was experiencing up: My anti-spam system (namely Spamassassin) did not detect spam mails anymore.

Now here is the reason it did not: After some more investigation of the problem I noticed that spam emails were received via a local connection (forwarded from fetchmail). However, one of my Exim ACLs says not to scan emails from localhost for spam.

So, the solution might be a hack, but it worked out perfectly. Starting fetchmail with the -S <servername> argument causes it to send emails to the given SMTP server rather than localhost. Using the real hostname of my server caused the “do not scan local mails” not to kick in and all mails received via fetchmail to be scanned again.

Problem fixed.

Now I am wondering why someone sets up SPF for his mail domain when he is in fact sending emails from other IP addresses as well. Time to update your SPF rules Technorati…