After noticing this problem I became curious on how Android detects that it is roaming and I found the GsmServiceStateTracker.isRoamingBetweenOperators method to be responsible for that magic, but soon noticed that the method is not only inefficient, but also doesn’t work as intended. This is hardly related to the bug mentioned above, but let’s have a look at the code in question:

/**
* Set roaming state when gsmRoaming is true and, if operator mcc is the
* same as sim mcc, ons is different from spn
* @param gsmRoaming TS 27.007 7.2 CREG registered roaming
* @param s ServiceState hold current ons
* @return true for roaming state set
*/
    private
    boolean isRoamingBetweenOperators(boolean gsmRoaming, ServiceState s) {
        String spn = SystemProperties.get(PROPERTY_ICC_OPERATOR_ALPHA, "empty");

        String onsl = s.getOperatorAlphaLong();
        String onss = s.getOperatorAlphaShort();

        boolean equalsOnsl = onsl != null && spn.equals(onsl);
        boolean equalsOnss = onss != null && spn.equals(onss);

        String simNumeric = SystemProperties.get(PROPERTY_ICC_OPERATOR_NUMERIC, "");
        String operatorNumeric = s.getOperatorNumeric();

        boolean equalsMcc = true;
        try {
            equalsMcc = simNumeric.substring(0, 3).
                    equals(operatorNumeric.substring(0, 3));
        } catch (Exception e){
        }

        return gsmRoaming && !(equalsMcc && (equalsOnsl || equalsOnss));
    }

Okay, let me summarize what this piece of code does wrong, at least from my understanding:

• It takes both the network operator alphanumeric identifier and alphanumeric long identifier and compares both to the alphanumeric identifier coming from the SIM card, whilst…
• … it could simply use the network and SIM card numeric identifiers and compare those, which should be a lot cheaper than comparing those strings
• Then it takes the first three characters/digits of the numeric identifiers (which indicate the country) and compares those

Now in my case my SIM card doesn’t seem to provide the phone with a alphanumeric identifier, so the first two comparisons always fail for obvious reasons and, looking at the inline-if in the last line of that method my phone will always indicate that I am in roaming mode, even when I am not.

The problem is not only the logic which seems to be wrong, but I rather see the inefficient comparisons used there to be a major problem in embedded systems like mobile phones. This is the first piece of Android code I have had a look at, but if all other code is as ugly and inefficient as these few lines Android really needs some major fixes. Related to this I have reported bug #4590 and forked the git repository in question over at github, to fix this method, should be a matter of 5 minutes.

Matt Porter’s Android Mythbusters presentation and Harald Welte’s reaction

I haven’t seen the presentation live, but I had a look at the slides. Impressing work done by Matt putting all this information together. However, we all knew that Android only (ab-)uses Linux, without making use of the GNU userland for a long time, didn’t we?

In his presentation Matt has shown things such as Android’s udev “replacement” that uses hardcoded values for device node creation and (on his blog) Harald has then come up with a statement I have found to be very strong:

The presentation shows how Google has simply thrown 5-10 years of Linux userspace evolution into the trashcan and re-implemented it partially for no reason. Things like hard-coded device lists/permissions in object code rather than config files, the lack of support for hot-plugging devices (udev), the lack of kernel headers. A libc that throws away System V IPC that every unix/Linux software developer takes for granted. The lack of complete POSIX threads. I could continue this list, but hey, you should read those slides. now!

Now both of these statements target technical details, but the root of the problem seems to be elsewhere.

Where is my Android 2.0?

Okay, that heading might not be making any sense in the context of this post at a first glance, but let me elaborate on that. Google and the Open Handset Alliance refer to Android as being an “Open Source” operating system, but the project is different from “real” Free Software projects: development takes place in a closed group and the results are shared with the community later on, when they are deemed to be ready.

This means that innovation also takes place behind closed curtains and that the community is not involved in the actual development process at all. Lately we have seen the result of that, as Motorola is bragging about working close with Google on Android 2.0 (”Eclair”), but the AOSP source trees, open for everyone to have a look at, show no signs of version 2.0. In fact no changes that might even remotely suggest the release of a new major version have been made public in the past few weeks. So where is the openess there?
Actually, the Motorola Droid has already shipped with Eclair on 6th, but still, there is no indication that Eclair will be made available to the broader public.

In short Android seems to be developed behind closed curtains, with hardly (read no) community input whatsoever and is sometimes released as Free Software, not what I would describe as an open development process.

The Android Market problem

As we have seen in the past Google is enforcing their copyright on proprietary applications that ship with pretty much every Android device, such as the Android Market. This has become really clear when Steve Kondik received  a cease and desist letter when packing the Google-proprietary applications into his ROMs. Okay, it’s Google’s right to enforce their copyright and there is nothing wrong with actually doing so, the thing I really have a problem with is something else: the Market is proprietary.

Now what this means should become rather clear. You can have an Android device without Google’s proprietary bits, but with default settings you just do not have any way of installing additional software. In my opinion the Market should be freed by Google themselves, or the community has to react and come up with a free replacement to overcome the vendor lock-in. Oh, you might know a replacement called SlideMe (or Mobentoo) already. Well, that bugger is proprietary too, so not a solution at all.

Nokia and Maemo to the rescue

In most discussions about the openness of Android someone throws in Nokia and Maemo, as a solution to the dilemma. Reading all those positive comments I simply had to give it a try, but all my hopes were destroyed within a few minutes.

Let’s start with the good news and let alone the reason why my hopes were destroyed for another minute or two. Maemo is based on Debian GNU/Linux and various Free Software components, such as GTK+, gstreamer, esd and friends. Most of the system is Free Software which is a good thing(tm) and reading all of this really got me into Maemo. Okay, some applications seem to be proprietary, but I am sure that could be fixed rather easily, so I could once for all use a truly open phone.

…and then came the SDK installer shell script:

#!/bin/sh
# Copyright (C) 2006-2009 Nokia Corporation
#
# This is proprietary software owned by Nokia Corporation.
#
# Contact: Maemo Integration <integration@maemo.org>
# Version: $Revision: 1110 $

Now there is one question you should ask yourself: Why would someone trying to promote his platform as being open make the *installer* script for its SDK proprietary? Come on, it’s an installer script, how much of your secret juice could be in there? What’s the problem with people modifying it and working on this installer script in an open development environment?

I had high hopes for Nokia actually doing a bit better than Google, but it seems they’ve failed to do so. It may be me overreacting, but a proprietary SDK installer shell script scares me enough not to install the SDK and have a look at it for now nor to think about buying a Maemo-based device in the near future. Please Nokia, either get the facts straight or provide us with a free SDK to your free & open platform.

So, in short, Google is bad at working with the community and creating a truly open development process, and Nokia simply fails in terms of not scaring off prospective developers for their open platform with the proprietary SDK installer. Do you have any solutions in terms of an open phone environment, apart from what OpenMoko has come up with?

So, after a few minutes of googling I came across an entry at answers.launchpad.net[0] and a blog post, but I cannot seem to remember the URL to that one. I can imagine that some of you might be having the exact same problem, so the solution is holding down the ALT, whilst dragging as usual.

[0] https://answers.launchpad.net/ubuntu/+source/gnome-panel/+question/264

As PyGame provides a nice library for writing games purely in Python it is becoming more common to use Python for this task too. The book “Beginning Game Development with Python and Pygame” is linked directly from the PyGame homepage, and thus is probably a good resource if you want to start writing games in Python.

However, I do not want to go into detail on how this library works, but rather provide you with a few examples of games written in Python. To provide you with a few examples I had a look at the PyWeek homepage. PyWeek is a Python Game Programming Challenge which invites everyone to participate, so the winners of this contest are of high-quality, and I’m showing you the latest two winners.

There are always two winners of PyWeek in for indivduals who have created games and teams. The latest winners are “Team Rambo” in the individual effort category and “Midnight Sun” with their two-man team.

PyWeek: Team Rambo’s Stringrolled (individual)

Stringrolled makes use of the pygame library I mentioned earlier and is a platform game. In a mere 2377 lines of code, including comments and blank lines, Team Rambo created an impressive game, coming with a story, easy-to-learn controls and nice 2D-graphics, screenshot below.

PyWeek: Midnight Sun’s Kite Story

Kite Story is yet another interesting game, with game mechanics I have not seen ever before. You are controlling a kite with your mouse and are trying to catch objects, such as bees and birds, with the kite’s rope. So what you basically do you draw a loop around an object with your mouse and that way catch it. Every third cought object you advance to the next level, but keep in mind not to collide with the objects, because you will lose them and in turn be doing the previous level again, screenshot below. It should be noted that this game does not make use of PyGame at all, but rather relies on pyglet, and is 1997 lines of code in length, again counting blank lines and comments too.

Games using Python

You have seen now that it is possible to write a game completely in Python, but there’s another use-case of Python in games: scripting.
Some (proprietary) games, such as Civilization IV, offer Python support in their editors and SDKs. This quote from the article at 2kgames.com should give you a good idea of what can be done using Python in Civilization IV:

The next level offers Python and XML support, letting modders with more experience manipulate the game world and everything in it. XML (eXtensible Markup Language) files can be edited in standard text editors or in special XML file editors that have ease-of-use features like a grid view. Editing these files will allow players to tweak simple game rules and change or add content. For instance, they can add new unit or building types, change the cost of wonders, or add new civilizations. Players can also change the sounds played at certain times or edit the play list for your soundtrack. NOTE: You can have custom soundtracks simply by adding music to the custom folder. You only need to edit the XML in order to assign certain pieces to specific eras or remove certain pieces.

The Python scripting language is fully integrated throughout the game and offers experienced modders a chance to really strut their stuff! People with some programming skills will be able to do things to alter the game in interesting and extraordinary ways. For instance, all of the game interface screens are exposed to Python, so modders will be able to change the information that’s displayed, as well as how it’s positioned on the screen. We also use Python to create and generate all of the random map scripts that are included in the game. So, players will now have the ability to add scripted events to the game like automatically generating units when a tile is reached, having specific situations trigger automatic war, or get this, bringing back Civil Wars caused by unrest, Civ II style!

EVE Online is another game making use of Python, as an article over at eveonline.com points out.

Python everywhere – also in compuater games

Even though I am sure you can come up with a lot more examples of Python being used in computer games I think I have proven my point. Python is being used not only to create computer games, but sometimes also to provide developers with a way of extending games. To me personally it feels as if adoption of Python for this very task is increasing too, and I expect Python to be used even more by the game development community in the future.

You can expect the third part of this series to be released in about a week, so please check back regularly if you like the series.

The root of the problem was my setup running PHP as a separate FastCGI process. Unfortunatly it seems as if PHP can only handle 500 requests per FastCGI process and then seems to lock up.
The old setup of this site didn’t cause such problems and it seems the problem lies in not setting the PHP_FCGI_CHILDREN and PHP_FCGI_MAX_REQUESTS environment variables with the new setup.

I initially thought that the default values of those environment variables were safe, but they are not. As I already wrote PHP seems to lock up after 500 requests and the solution lies in changing PHP_FCGI_CHILDREN, which defaults to 0 (no additional processes) to something bigger than 0 (I am using 2 children to make sure I have at least one PHP process reading for answering requests at any time).
Why? Quite simple, if you increase the value the PHP root process becomes some sort of manager and delegates requests to the children, as expected. However, using PHP_FCGI_MAX_REQUESTS it only forwards the specified number of requests to a child process before killing it and starting a new one. Problem solved.

Information on this behaviour can not be found in the PHP online manual, but rather at http://lxr.php.net/source/php-src/sapi/cgi/README.FastCGI.

As this is the first article in this series I would like to explain what the series is all about.
As an avid Python user and developer I want to share my observations whenever I find Python applications doing not-so-unusual things, Python applications running on embedded devices. In the end I want to point out just what the name of this series suggests: Python is everywhere and can be used for everything.

So, straight ahead to the first issue: the conficker scanner.

When reading an article about a detection mechanism for the conficker worm on heise Security [german] I was myself wondering a few things, but wanted to give it a try. So I followed the link to the article Detecting Conficker, by Tillmann Werner. Before clicking the link I was wondering whether I could get this tool running on GNU/Linux using wine, or another method.

After downloading the ZIP file and unpacking it I thought I was dreaming. There were two Python files, along with a COPYING file.
So, even though before having a look at the code I wanted to know the COPYING conditions, and again I saw something unexpected: it’s licensed under the GPLv3, great!

As there are some computers running a proprietary operating system from Redmond on this network I immediately gave it a shot. I started the script (scs.py), and after fulfilling its requirements (namely the impacket Python module) I ran it on the local network and it worked without any problems. No conficker found on this network, after all my flatmates have their systems secured – good.

So there you have another use-case for Python: detecting malware over the network.
Kudos should go to Tillmann Werner, not only for this piece of Python code, but also for his work on the honeynet project and, together with Felix Leder, the great analysis of conficker. Keep up the good work, and thanks for proving Python can also be used for this task.

pyttpd is my effort of implementing a webserver in Python, with a focus on security (through privilege separation), extensibility and scalability.

I started this project because I was not entirely happy with the lack of flexibility and support for privilege separation by popular webservers. Whilst both lighttpd and Apache httpd provide means of running processes under different users these usually require hacks like suexec. Additionally I am somehow curious about how a fully-fledged webserver implemented in Python would perform compared to the mentioned daemons.

Security through extensive use of Privilege Separation

Whilst it is common for daemons to initially run as a privileged user and drop privileges as soon as possible it is possible to make more extensive use of setuid and friends.
pyttpd’s design aims at creating one privileged process, which only binds to privileged ports and spawns subprocesses.
All subprocesses have specific tasks, such as routing between all processes, protocol-specific parsing of incoming requests and handling processing of those requests.
The point is that all these processes do not run as “www-data” or another common account, but that a logic separation takes place on a per-host basis. This means that if the webserver is hosting www.example.org and webapp.example.org those will be running under different system accounts, making it hard to interfere with each other. This method should also enable the use of MAC mechanisms such as SELinux or SMACK more efficiently.

The design choice of having separate processes for each vhost comes with another benefit: users (or customers) “owning” a vhost could potentionally be allowed to modify parts of the vhost’s configuration (excluding UID, GID, and other security-relevant options) on their own.

Early status

Right now pyttpd is in a very early planning stage, with no code to show yet. I am still in the middle of the process of writing down all ideas that come to my mind, weeding some out and documenting the others.
So what do I have to show you then? Well, the the concept section of the documentation is online now and I am planning on extending it in the next few days and eventually start writing code rather sooner than later.

Your ideas…

…and opinions are what I am really interested in. If you are interested in this project I would love if you got involved in some way. Feel free to create tickets at pyttpd’s project page if you have an idea you feel is worth adding or if one of my ideas is flawed, create a comment here or send me an email.

Now you probably expect me to come up with a pure technical reason showing superiority of Free Software, but I am taking another path this time: let’s talk about user trust.

Software updates on pure Free Software operating systems

Firstly, let’s take a look at how security updates are deployed on pure Free Software operating systems.
All major GNU/Linux and *BSD distributions come with built-in update systems for all Free Software packages provided by the distribution. Once a security update to any piece of software is released you will get a notification saying updates are available and that you should install them and after doing so your system should be in a quite secure state.

Updater-applications of proprietary software

Okay, proprietary software does often come with its own update mechanisms, giving you pretty much the same functionality. But here is my point: users tend to block these updater-applications from accessing the network with personal firewalls and similar tools, which basically disables the updater and most likely will cause the system to be more vulnerable rather sooner than later.

But why do people even consider blocking an updater-application?

When I asked people why they blocked these applications a lot of reasons came up, but it usually boils down to lack of trust for proprietary software vendors. People seem not to trust software vendors for a variety of reasons. The most common reasons seem to be that they are either worried about the disclosure of private information or the lack of a license for a piece of software, combined with the first reason.

So people actually seem to be scared by what a piece proprietary software could do, and when not having their software licensed people seem to be even more scared. When I ask people why they are running these programs, even though they do not trust their vendors they usually shrug and I get replies that can be summed up as “it just works”, “but I don’t have a choice” and “I am used to software X”.

The solution is Free Software

Free Software can be a solution to all these problems. The most important thing people should be aware of that they do not have to fear Free Software vendors. After all, when using Free Software you do have a valid license and you can, at least in theory, check exactly a program is doing to your system. Also, if you are not capable of doing such checks yourself, you can rest assured that other people are doing such checks and give back to the project in question, improving your favorite Free Software applications.

Free Software is a choice, and there is hardly any proprietary piece of software which cannot be replaced by a free equivalent. Free Software is my first choice, not only for me but for my family too, with great results so far.

Rest assured you can trust in what your software is doing, free yourself today – use Free Software.

The reason for not everyone using GPG/PGP for encrypting their emails might be that, even though GPG/PGP have become a lot more usable for the end-user in the last few years, these programs are probably still too technical and thus hard to understand for non-technical users.

This is when I thought a little about how people could be made using public key encryption for E-Mails. After a bit of brain-storming an idea came to my mind, an idea I would like to present you with.

Basic idea

What about creating a program acting as both SMTP and POP3/IMAP proxy server that included all the logic to do encryption and would encrypt/decrypt messages transparently?
If this logic was moved out of Email clients we could get a solution working universally for each and every Email client out there.

How this could work

Imagine you sending an email to someone you’ve never sent an email to. You write the message in your Email client as you are used to and hit the send button. Now, instead of connecting to your SMTP server the E-Mail client would connect to the Email proxy program and submit the message there.

At this point the program would check the email sender and recipient. If the sender does have a public/private key pair and the recipient’s public key is known the program would prompt you for the passphrase to your encryption key. After entering the passphrase and hitting a button (send, sign, encrypt, I guess you can think of a more appropriate name) again the message would be encrypted and then forwarded to your SMTP server.

On the other hand, if the public key of the recipient is not known (and cannot be fetched off key servers) the program could send a message informing the recipient that you wanted to encrypt your email, but were unable to do so, explain that this program exists, where to get it from, how to set it up, why encryption is important, and so on. I can imagine having a hard-fail mode, sending only this message and a soft-fail mode, attaching or including the automatically generated message somehow (attach it, inline it, etc.) to the original message. Either way, the generated message should be cryptographically signed.

Receiving mail would work the other way around. The proxy would try to fetch messages off all configured IMAP/POP3 servers on its own, check if they are signed. If a signed message arrives the public key should be, if not already done, be imported into the local keyring. As for encrypted messages this should happen the same way, plus decrypting the message.
The Email client would connect to the IMAP/POP3 proxy server and fetch (the decrypted) messages from it. Both unencrypted and unsigned messages should be marked somehow (think subject re-writing here and maybe adding an X- header). However, no automatic sending of emails should happen when receiving messages as the From header could be forged (spam anyone?).

Features of the program

The program I have in mind should include the following features:

• GPG key management (creating, distribution to keyservers, etc).
• Automatic encryption/decryption and signing/checking signatures.
• Non-technical, so everyone can use it.
• Support multiple IMAP/POP3 and SMTP servers, so it can act as a central point for storing all Emails a user could receive.
• Cross-platform functionality (Java? Python?)
• Free Software

Plans

I would love to implement this program, but fear that this could be way too much work for a single person. If you are interested in helping with the implementation or simply have any comments feel free to either drop me an email at blog at sp dot or dot at or use the blog’s comment function.

I hope I did explain my idea clear enough and did not miss anything.

Happy hacking!

After I first read these news over at heise (german) I was impressed, but started to fear that yet again some sort of government has invested in proprietary software and is able to bring its services only to users of such software. Seconds later my fears became reality.

EuroparlTV seems to work only for users of either Adobe’s proprietary Flash player (via the proprietary Adobe Flash file format) or users of Microsoft’s Windows Media Player (via the proprietary WMV file format).

What this means to an open web, that is usable for everyone, should be clear.

Basically this is a service all citizens of the European Union pay for, but some cannot use. Is this really how governments (and the EP is some sort of government) should treat their citizens? Rather not.

On the one hand the European Commission is fighting vendor lock-in and monopoles, but on the other hand it directly helps these vendors by creating such services. Not a smart move in my opinion, neither is it understandable.

What I am asking myself though is why the EP was unable to create such a service, which itself could be quite interesting, without having all users of that service use proprietary software?
Is it so hard to deliver the service in a free (as in freedom), standardized format?
I will let answering these questions to you, but keep in mind that there are alternatives to this whole proprietary mess, like Ogg, which are completly free.

Personally I am pretty disappointed by this move. However, I hope that I at least informed people that there is a problem with EuroparlTV.
Putting it simple and short this way the EP does a great deal with helping vendor lock-in whilst fighting the freedom of its own citizens. Even though it should be the other way round.