After noticing this problem I became curious on how Android detects that it is roaming and I found the GsmServiceStateTracker.isRoamingBetweenOperators method to be responsible for that magic, but soon noticed that the method is not only inefficient, but also doesn’t work as intended. This is hardly related to the bug mentioned above, but let’s have a look at the code in question:

/**
* Set roaming state when gsmRoaming is true and, if operator mcc is the
* same as sim mcc, ons is different from spn
* @param gsmRoaming TS 27.007 7.2 CREG registered roaming
* @param s ServiceState hold current ons
* @return true for roaming state set
*/
    private
    boolean isRoamingBetweenOperators(boolean gsmRoaming, ServiceState s) {
        String spn = SystemProperties.get(PROPERTY_ICC_OPERATOR_ALPHA, "empty");

        String onsl = s.getOperatorAlphaLong();
        String onss = s.getOperatorAlphaShort();

        boolean equalsOnsl = onsl != null && spn.equals(onsl);
        boolean equalsOnss = onss != null && spn.equals(onss);

        String simNumeric = SystemProperties.get(PROPERTY_ICC_OPERATOR_NUMERIC, "");
        String operatorNumeric = s.getOperatorNumeric();

        boolean equalsMcc = true;
        try {
            equalsMcc = simNumeric.substring(0, 3).
                    equals(operatorNumeric.substring(0, 3));
        } catch (Exception e){
        }

        return gsmRoaming && !(equalsMcc && (equalsOnsl || equalsOnss));
    }

Okay, let me summarize what this piece of code does wrong, at least from my understanding:

• It takes both the network operator alphanumeric identifier and alphanumeric long identifier and compares both to the alphanumeric identifier coming from the SIM card, whilst…
• … it could simply use the network and SIM card numeric identifiers and compare those, which should be a lot cheaper than comparing those strings
• Then it takes the first three characters/digits of the numeric identifiers (which indicate the country) and compares those

Now in my case my SIM card doesn’t seem to provide the phone with a alphanumeric identifier, so the first two comparisons always fail for obvious reasons and, looking at the inline-if in the last line of that method my phone will always indicate that I am in roaming mode, even when I am not.

The problem is not only the logic which seems to be wrong, but I rather see the inefficient comparisons used there to be a major problem in embedded systems like mobile phones. This is the first piece of Android code I have had a look at, but if all other code is as ugly and inefficient as these few lines Android really needs some major fixes. Related to this I have reported bug #4590 and forked the git repository in question over at github, to fix this method, should be a matter of 5 minutes.

Matt Porter’s Android Mythbusters presentation and Harald Welte’s reaction

I haven’t seen the presentation live, but I had a look at the slides. Impressing work done by Matt putting all this information together. However, we all knew that Android only (ab-)uses Linux, without making use of the GNU userland for a long time, didn’t we?

In his presentation Matt has shown things such as Android’s udev “replacement” that uses hardcoded values for device node creation and (on his blog) Harald has then come up with a statement I have found to be very strong:

The presentation shows how Google has simply thrown 5-10 years of Linux userspace evolution into the trashcan and re-implemented it partially for no reason. Things like hard-coded device lists/permissions in object code rather than config files, the lack of support for hot-plugging devices (udev), the lack of kernel headers. A libc that throws away System V IPC that every unix/Linux software developer takes for granted. The lack of complete POSIX threads. I could continue this list, but hey, you should read those slides. now!

Now both of these statements target technical details, but the root of the problem seems to be elsewhere.

Where is my Android 2.0?

Okay, that heading might not be making any sense in the context of this post at a first glance, but let me elaborate on that. Google and the Open Handset Alliance refer to Android as being an “Open Source” operating system, but the project is different from “real” Free Software projects: development takes place in a closed group and the results are shared with the community later on, when they are deemed to be ready.

This means that innovation also takes place behind closed curtains and that the community is not involved in the actual development process at all. Lately we have seen the result of that, as Motorola is bragging about working close with Google on Android 2.0 (”Eclair”), but the AOSP source trees, open for everyone to have a look at, show no signs of version 2.0. In fact no changes that might even remotely suggest the release of a new major version have been made public in the past few weeks. So where is the openess there?
Actually, the Motorola Droid has already shipped with Eclair on 6th, but still, there is no indication that Eclair will be made available to the broader public.

In short Android seems to be developed behind closed curtains, with hardly (read no) community input whatsoever and is sometimes released as Free Software, not what I would describe as an open development process.

The Android Market problem

As we have seen in the past Google is enforcing their copyright on proprietary applications that ship with pretty much every Android device, such as the Android Market. This has become really clear when Steve Kondik received  a cease and desist letter when packing the Google-proprietary applications into his ROMs. Okay, it’s Google’s right to enforce their copyright and there is nothing wrong with actually doing so, the thing I really have a problem with is something else: the Market is proprietary.

Now what this means should become rather clear. You can have an Android device without Google’s proprietary bits, but with default settings you just do not have any way of installing additional software. In my opinion the Market should be freed by Google themselves, or the community has to react and come up with a free replacement to overcome the vendor lock-in. Oh, you might know a replacement called SlideMe (or Mobentoo) already. Well, that bugger is proprietary too, so not a solution at all.

Nokia and Maemo to the rescue

In most discussions about the openness of Android someone throws in Nokia and Maemo, as a solution to the dilemma. Reading all those positive comments I simply had to give it a try, but all my hopes were destroyed within a few minutes.

Let’s start with the good news and let alone the reason why my hopes were destroyed for another minute or two. Maemo is based on Debian GNU/Linux and various Free Software components, such as GTK+, gstreamer, esd and friends. Most of the system is Free Software which is a good thing(tm) and reading all of this really got me into Maemo. Okay, some applications seem to be proprietary, but I am sure that could be fixed rather easily, so I could once for all use a truly open phone.

…and then came the SDK installer shell script:

#!/bin/sh
# Copyright (C) 2006-2009 Nokia Corporation
#
# This is proprietary software owned by Nokia Corporation.
#
# Contact: Maemo Integration <integration@maemo.org>
# Version: $Revision: 1110 $

Now there is one question you should ask yourself: Why would someone trying to promote his platform as being open make the *installer* script for its SDK proprietary? Come on, it’s an installer script, how much of your secret juice could be in there? What’s the problem with people modifying it and working on this installer script in an open development environment?

I had high hopes for Nokia actually doing a bit better than Google, but it seems they’ve failed to do so. It may be me overreacting, but a proprietary SDK installer shell script scares me enough not to install the SDK and have a look at it for now nor to think about buying a Maemo-based device in the near future. Please Nokia, either get the facts straight or provide us with a free SDK to your free & open platform.

So, in short, Google is bad at working with the community and creating a truly open development process, and Nokia simply fails in terms of not scaring off prospective developers for their open platform with the proprietary SDK installer. Do you have any solutions in terms of an open phone environment, apart from what OpenMoko has come up with?

So, after a few minutes of googling I came across an entry at answers.launchpad.net[0] and a blog post, but I cannot seem to remember the URL to that one. I can imagine that some of you might be having the exact same problem, so the solution is holding down the ALT, whilst dragging as usual.

[0] https://answers.launchpad.net/ubuntu/+source/gnome-panel/+question/264

So I guess it’s about time to introduce my project to you: Distribution-independent update manager, mentored by Michael Vogt (mvo).

Okay, I believe some of you might wonder what this project is all about, as update-manager is in the Debian package archive already. There is a problem with update-manager though. As you see in the package’s version number (it contains “.debian”) update-manager has been adapted for use in Debian. Also, Debian contains update-manager 0.68 right now, whilst upstream (Ubuntu in this case) has released 0.111.6 (actually there were quite a few upstream versions meanwhile). The reason Debian is nowhere near being up-to-date with upstream is that right now a lot of effort has to be put into porting update-manager to Debian every time a new upstream release is made, because certain Ubuntu-specific functionality breaks update-manager in more or less severe ways on Debian.

This leads me directly to what my project is about: making update-manager (Ubuntu-) distribution-independent, but not package manager independent.
There are 6 main goals for this project, which I will be working on in the order below.

• Analyzing the code and identifying Ubuntu-specific parts.
• Creating a distribution-plugin interface and moving the Ubuntu-specific parts into a distribution-plugin, creating a core package that is distribution-independent.
• Creating a special notification for important/security related updates and providing the code that handles updates from security.debian.org as such.
• Creating a backend-plugin interface, moving the synaptics backend into a backend-plugin and optionally create a python-apt based plugin.
• UI redesign, providing a simpler interface to average joe, whilst allowing more experienced users to optionally display more information.
• Automatic downloading & installation of updates. This is still up to discussion, as automatic downloading is already provided by software-properties (-gtk and -kde) and automatic installation can be handled by unattended-upgrades. Both packages are part of Debian already.

Please note that this list should not be considered final and may be extended or modified over time. It exists to give you an overview of what exactly my project is about and how I am planning on carrying out the tasks.

Finally I wanted to let you know that I will keep you posted on the progress I am making, via this blog. Alternatively a blog aggregator for Debian’s GSoC students has been set up over at http://soc.alioth.debian.org/feeds/blogs/, where you can not only find my posts, but those of all of Debian’s students.

As PyGame provides a nice library for writing games purely in Python it is becoming more common to use Python for this task too. The book “Beginning Game Development with Python and Pygame” is linked directly from the PyGame homepage, and thus is probably a good resource if you want to start writing games in Python.

However, I do not want to go into detail on how this library works, but rather provide you with a few examples of games written in Python. To provide you with a few examples I had a look at the PyWeek homepage. PyWeek is a Python Game Programming Challenge which invites everyone to participate, so the winners of this contest are of high-quality, and I’m showing you the latest two winners.

There are always two winners of PyWeek in for indivduals who have created games and teams. The latest winners are “Team Rambo” in the individual effort category and “Midnight Sun” with their two-man team.

PyWeek: Team Rambo’s Stringrolled (individual)

Stringrolled makes use of the pygame library I mentioned earlier and is a platform game. In a mere 2377 lines of code, including comments and blank lines, Team Rambo created an impressive game, coming with a story, easy-to-learn controls and nice 2D-graphics, screenshot below.

PyWeek: Midnight Sun’s Kite Story

Kite Story is yet another interesting game, with game mechanics I have not seen ever before. You are controlling a kite with your mouse and are trying to catch objects, such as bees and birds, with the kite’s rope. So what you basically do you draw a loop around an object with your mouse and that way catch it. Every third cought object you advance to the next level, but keep in mind not to collide with the objects, because you will lose them and in turn be doing the previous level again, screenshot below. It should be noted that this game does not make use of PyGame at all, but rather relies on pyglet, and is 1997 lines of code in length, again counting blank lines and comments too.

Games using Python

You have seen now that it is possible to write a game completely in Python, but there’s another use-case of Python in games: scripting.
Some (proprietary) games, such as Civilization IV, offer Python support in their editors and SDKs. This quote from the article at 2kgames.com should give you a good idea of what can be done using Python in Civilization IV:

The next level offers Python and XML support, letting modders with more experience manipulate the game world and everything in it. XML (eXtensible Markup Language) files can be edited in standard text editors or in special XML file editors that have ease-of-use features like a grid view. Editing these files will allow players to tweak simple game rules and change or add content. For instance, they can add new unit or building types, change the cost of wonders, or add new civilizations. Players can also change the sounds played at certain times or edit the play list for your soundtrack. NOTE: You can have custom soundtracks simply by adding music to the custom folder. You only need to edit the XML in order to assign certain pieces to specific eras or remove certain pieces.

The Python scripting language is fully integrated throughout the game and offers experienced modders a chance to really strut their stuff! People with some programming skills will be able to do things to alter the game in interesting and extraordinary ways. For instance, all of the game interface screens are exposed to Python, so modders will be able to change the information that’s displayed, as well as how it’s positioned on the screen. We also use Python to create and generate all of the random map scripts that are included in the game. So, players will now have the ability to add scripted events to the game like automatically generating units when a tile is reached, having specific situations trigger automatic war, or get this, bringing back Civil Wars caused by unrest, Civ II style!

EVE Online is another game making use of Python, as an article over at eveonline.com points out.

Python everywhere – also in compuater games

Even though I am sure you can come up with a lot more examples of Python being used in computer games I think I have proven my point. Python is being used not only to create computer games, but sometimes also to provide developers with a way of extending games. To me personally it feels as if adoption of Python for this very task is increasing too, and I expect Python to be used even more by the game development community in the future.

You can expect the third part of this series to be released in about a week, so please check back regularly if you like the series.

As this is the first article in this series I would like to explain what the series is all about.
As an avid Python user and developer I want to share my observations whenever I find Python applications doing not-so-unusual things, Python applications running on embedded devices. In the end I want to point out just what the name of this series suggests: Python is everywhere and can be used for everything.

So, straight ahead to the first issue: the conficker scanner.

When reading an article about a detection mechanism for the conficker worm on heise Security [german] I was myself wondering a few things, but wanted to give it a try. So I followed the link to the article Detecting Conficker, by Tillmann Werner. Before clicking the link I was wondering whether I could get this tool running on GNU/Linux using wine, or another method.

After downloading the ZIP file and unpacking it I thought I was dreaming. There were two Python files, along with a COPYING file.
So, even though before having a look at the code I wanted to know the COPYING conditions, and again I saw something unexpected: it’s licensed under the GPLv3, great!

As there are some computers running a proprietary operating system from Redmond on this network I immediately gave it a shot. I started the script (scs.py), and after fulfilling its requirements (namely the impacket Python module) I ran it on the local network and it worked without any problems. No conficker found on this network, after all my flatmates have their systems secured – good.

So there you have another use-case for Python: detecting malware over the network.
Kudos should go to Tillmann Werner, not only for this piece of Python code, but also for his work on the honeynet project and, together with Felix Leder, the great analysis of conficker. Keep up the good work, and thanks for proving Python can also be used for this task.

pyttpd is my effort of implementing a webserver in Python, with a focus on security (through privilege separation), extensibility and scalability.

I started this project because I was not entirely happy with the lack of flexibility and support for privilege separation by popular webservers. Whilst both lighttpd and Apache httpd provide means of running processes under different users these usually require hacks like suexec. Additionally I am somehow curious about how a fully-fledged webserver implemented in Python would perform compared to the mentioned daemons.

Security through extensive use of Privilege Separation

Whilst it is common for daemons to initially run as a privileged user and drop privileges as soon as possible it is possible to make more extensive use of setuid and friends.
pyttpd’s design aims at creating one privileged process, which only binds to privileged ports and spawns subprocesses.
All subprocesses have specific tasks, such as routing between all processes, protocol-specific parsing of incoming requests and handling processing of those requests.
The point is that all these processes do not run as “www-data” or another common account, but that a logic separation takes place on a per-host basis. This means that if the webserver is hosting www.example.org and webapp.example.org those will be running under different system accounts, making it hard to interfere with each other. This method should also enable the use of MAC mechanisms such as SELinux or SMACK more efficiently.

The design choice of having separate processes for each vhost comes with another benefit: users (or customers) “owning” a vhost could potentionally be allowed to modify parts of the vhost’s configuration (excluding UID, GID, and other security-relevant options) on their own.

Early status

Right now pyttpd is in a very early planning stage, with no code to show yet. I am still in the middle of the process of writing down all ideas that come to my mind, weeding some out and documenting the others.
So what do I have to show you then? Well, the the concept section of the documentation is online now and I am planning on extending it in the next few days and eventually start writing code rather sooner than later.

Your ideas…

…and opinions are what I am really interested in. If you are interested in this project I would love if you got involved in some way. Feel free to create tickets at pyttpd’s project page if you have an idea you feel is worth adding or if one of my ideas is flawed, create a comment here or send me an email.

Now you probably expect me to come up with a pure technical reason showing superiority of Free Software, but I am taking another path this time: let’s talk about user trust.

Software updates on pure Free Software operating systems

Firstly, let’s take a look at how security updates are deployed on pure Free Software operating systems.
All major GNU/Linux and *BSD distributions come with built-in update systems for all Free Software packages provided by the distribution. Once a security update to any piece of software is released you will get a notification saying updates are available and that you should install them and after doing so your system should be in a quite secure state.

Updater-applications of proprietary software

Okay, proprietary software does often come with its own update mechanisms, giving you pretty much the same functionality. But here is my point: users tend to block these updater-applications from accessing the network with personal firewalls and similar tools, which basically disables the updater and most likely will cause the system to be more vulnerable rather sooner than later.

But why do people even consider blocking an updater-application?

When I asked people why they blocked these applications a lot of reasons came up, but it usually boils down to lack of trust for proprietary software vendors. People seem not to trust software vendors for a variety of reasons. The most common reasons seem to be that they are either worried about the disclosure of private information or the lack of a license for a piece of software, combined with the first reason.

So people actually seem to be scared by what a piece proprietary software could do, and when not having their software licensed people seem to be even more scared. When I ask people why they are running these programs, even though they do not trust their vendors they usually shrug and I get replies that can be summed up as “it just works”, “but I don’t have a choice” and “I am used to software X”.

The solution is Free Software

Free Software can be a solution to all these problems. The most important thing people should be aware of that they do not have to fear Free Software vendors. After all, when using Free Software you do have a valid license and you can, at least in theory, check exactly a program is doing to your system. Also, if you are not capable of doing such checks yourself, you can rest assured that other people are doing such checks and give back to the project in question, improving your favorite Free Software applications.

Free Software is a choice, and there is hardly any proprietary piece of software which cannot be replaced by a free equivalent. Free Software is my first choice, not only for me but for my family too, with great results so far.

Rest assured you can trust in what your software is doing, free yourself today – use Free Software.

So what’s has changed compared to 0.8.2

Basically a lot of code has been rewritten or reviewed and modified. The most important change lies in argvalidate’s internals though.
Instead of relying on Python internals, those are the func_* and func_code.* attributes of every function argvalidate uses another mechanism for obtaining the information about functions now, the inspect Module. In short this means that argvalidate should be more resistant to possible changes in the format of these attributes, because the inspect Module that ships with Python should be changed accordingly and it is unlikely that its public interface is going to break.

Which leads me straight to the next point regarding argvalidate: the public interface. The most visible change was the deprecation of the method_args, func_args and return_value decorators. These decorators are still present, but will be removed in one of the next upcoming releases of argvalidate. The functionality of method_args and func_args has been combined into the accepts decorator, whilst the returns decorator replaces return_value.

accepts now includes some magic for automatically determining whether a function or method has been decorated. Initially the plan was on using inspect’s ismethod function, which did not work out as when the decorator is called for methods the method has not been bound to the class yet and looks like a simple function to ismethod. However, the magic added checks the first argument’s name and decides that functions having either “self” or “cls” as their first parameter are methods, whilst everything else is a simple function.

There is more to say about accepts. Internally some checks which were done at every call to the decorated function in the past are now done only once, when the decorator is invoked. This should reduce the overhead argvalidate adds when the function (and thus the internal wrapper doing the checks) is called, but most likely will slow down decoration itself. However, decoration is only done once, whilst a function is likely to be called several times in its lifetime, which should give us an overall improvement of speed.

The documentation has been updated accordingly and can be found at the usual place.

What’s up next?

There is no real further roadmap for argvalidate right now. No issues were left open, except for Python 3 testing & support. Further releases are going to be made when needed, that is when you submit a bug report and it is fixed.

After I first read these news over at heise (german) I was impressed, but started to fear that yet again some sort of government has invested in proprietary software and is able to bring its services only to users of such software. Seconds later my fears became reality.

EuroparlTV seems to work only for users of either Adobe’s proprietary Flash player (via the proprietary Adobe Flash file format) or users of Microsoft’s Windows Media Player (via the proprietary WMV file format).

What this means to an open web, that is usable for everyone, should be clear.

Basically this is a service all citizens of the European Union pay for, but some cannot use. Is this really how governments (and the EP is some sort of government) should treat their citizens? Rather not.

On the one hand the European Commission is fighting vendor lock-in and monopoles, but on the other hand it directly helps these vendors by creating such services. Not a smart move in my opinion, neither is it understandable.

What I am asking myself though is why the EP was unable to create such a service, which itself could be quite interesting, without having all users of that service use proprietary software?
Is it so hard to deliver the service in a free (as in freedom), standardized format?
I will let answering these questions to you, but keep in mind that there are alternatives to this whole proprietary mess, like Ogg, which are completly free.

Personally I am pretty disappointed by this move. However, I hope that I at least informed people that there is a problem with EuroparlTV.
Putting it simple and short this way the EP does a great deal with helping vendor lock-in whilst fighting the freedom of its own citizens. Even though it should be the other way round.