Because we can: MySQL talks with Johan Wikman, Father of MySQL on Symbian/S60. (part 1 of 3)
By Henrik Ingo on Oct 26, 2008
By the end of 2007, to the surprise of many of us, a guy at Nokia Research Center announced that they had ported and were about to publish the full LAMP stack running on the Symbian/S60 platform of Nokia mobile phones. They dubbed this the Personal AMP stack: PAMP, and you can run most of the popular PHP apps like Wordpress, Drupal, phpMyAdmin... out of the box on a Nokia phone now.
Today we had the opportunity to have a chat with Johan Wikman, the man leading the efforts of porting the AMP stack to Symbian. Johan works as Principal Research Engineer at Nokia Research Center and as such has also previously participated in porting other interesting things to Nokia phones, such as the Linux kernel, eventually leading to what maemo is today.
There is an interesting "it's a small world" aspect in that Johan used to study at Helsinki University of Technology about the same time as Monty, Mårten and Kaj were there.
Q: Hi Johan. I was thinking, for our US readers it might be necessary to start with the basic questions: What is Nokia? What is Symbian? And what is S60?
Well, Nokia is the world's biggest mobile phone manufacturer with a global market share of just below 40%. In the US, however, the market share is significantly smaller. Nokia has its roots in Finland and a great deal of research & development is done over here and the headquarter is here as well, but otherwise Nokia really is a global company with activities all over the globe.
Nokia manufactures phones in all categories, from low-end phones sold in developing countries to high-end smartphone containing all the goodies you can think of, and then some. And these high-end phones run the Symbian operating system. So, just like you can talk about a Windows phone or a Linux phone, you can talk about a Symbian phone. S60 is basically a UI and look&feel layer on top of Symbian, but at times it is a bit difficult to say where Symbian ends and S60 begins.
Q: Also, maybe it is worth talking about development on Symbian a little more. For those who have never done it, could you explain how it differs from, say, creating a Windows app or web development?
Symbian as a programming environment is basically quite different from pretty much everything else. It has a C++ API that takes time getting used to and although it has processes and threads like most other OSs, you usually handle concurrent processing within an application using so called active objects and an active scheduler. Basically its cooperative multitasking between different activities of the application, where you must ensure that you frequently enough give the control back to the scheduler, so that it can let other active objects run.
It may sound primitive, but it actually works quite well, since the operating system itself is well suited for this model, and in practice you don't really have to worry about keeping the CPU for too long. Most OS functionality can be invoked asynchronously, so that you instead of making a call and then block while waiting for it to finish, just initiate the call and are then later called back when operation has finished. While the asynchronous call is executing other active objects can run. But obviously this model has an effect on how you structure your program.
However, from high enough vantage point, creating an application in Symbian is similar to creating an application in other environments. You just have to learn and get used to the required API frameworks. Admittedly, the initial threshold in native Symbian development is higher than in other environments. But nowadays you can also do application development using Python and that really makes things easy.
Q: And I haven't tried it myself, but I've read that you can now program on Symbian pretty much like a Linux system? That is, any POSIX-like system, to be precise.
Yes, recently, with the introduction of the Open-C libraries, the situation has changed significantly. Open-C brings a very large set of functions, typically available in a Linux/Unix environment, to Symbian. So now it is possible to program for Symbian the way you program for Linux. That applies mostly to non-UI stuff, but usually, regardless of the environment, you anyway have to learn the particular UI framework you intend to use. But just a short time ago it was announced that the Qt framework will be made available for S60. That is, provided you are familiar with Qt, you will be able to create software for Nokia smartphones without learning the native APIs.
Although Open-C makes the porting of existing open source software to Symbian much easier than it used to be, there are still a couple of things still that can cause headache. First of all, the security model of Symbian is completely different from that of Linux. In Linux everything revolves around the concept of a user (not necessarily a real one); files and other resources are are owned by a user and processes are executed using the credentials of a particular user. In Symbian there is no concept of a user and prior Symbian version 9 there was basically no security. At runtime processes were protected from each other, but any process could access any file anywhere and also use any system resource, for instance, initiate a phone call. You could say that all processes run as root.
In Symbian 9, platform security was introduced and now the situation is quite different. At build-time binaries are assigned capabilities and those capabilities decide what the process can do at runtime. What capabilities you can assign a binary, depends upon what kind of certificate you are going to sign the installation package with. If you manage with a limited set of capabilities, then a self-signed certificate - one that you create yourself - is sufficient. However, if you need a wider set of capabilities, then a proper certificate is needed. Anyway, trust is effectively given to applications, not to the person who runs them.
But because of the signing process there is traceability, so if some malicious software turns up, you know who is responsible for it. Except for the self-signed applications, of course, but they can't do that much harm anyway and there are lots of warnings when you install them. Further, every application now also has a private directory that no other application, except one with a very hefty set of capabilities, can access.
A more practical problem is that on Symbian all symbols of a DLL (shared library) are exported by ordinal and not by name. So, loading a function explicitly by name from a DLL simply is not possible. So, if your open source software needs that functionality, then you need to work around the limitation. A closely related problem is that it is not possible to export data, but only functions, from a DLL. Since a lot of open source software export data from DLLs, that's something that also must be worked around, and it may not always be trivial to do that without having to touch the code too much.
Q: Which brings us nicely into what we really wanted to know... Was it a lot of work to port MySQL? What about Apache and PHP? Did you use the Open_C library for this, or maybe that's the whole point of the excercise?
Apache was the first component that was ported. And that was originally made when Open-C was not available. At that point there were only a rather limited and buggy C-library, that originally (to the best of my knowledge) was created in order to port the Java VM to Symbian. So, a fair amount of modifications were necessary and, for instance, the dynamic loading of Apache modules was not supported, but everything had to be built into the main executable.
Then came the already mentioned Symbian 9 and Open-C. At that point I decided to throw the existing port away, take the latest Apache httpd version into use and redo the port. Obviously that was much simpler this time, not only could a number of workarounds be thrown out but a lot of things just worked. However, when Apache loads a dynamic module, it looks up a data structure by name from the DLL, and that, as I earlier mentioned, is not possible on Symbian. But I could work around that in such a way that now only a single line must be added to a module in order to make it dynamically loadable.
The next component that was ported was PHP. Initially it was made as an experiment by a Hannu Kankaanpää, a smart summertrainee we had. Later when we really started working on our PAMP stack, the PHP port was updated by Petteri Muilu, Markus Heikelä and Markus Grönholm from Futurice (http://www.futurice.com). From a Symbian perspective PHP is quite problematic, because there are loads of global variables that not only are used by other DLLs, but stored in static structures in other DLLs. Namely, the usual way of working around the data export limitation is to export a function that returns the address of the variable and then define a macro that calls that function and dereferences it. That way client code need not be modified. However, when the addresses of those variables are stored in static structures it no longer works, but somehow the structures must explicitly be initialized at runtime.
Primarily of this reason, PHP has been built as a monolith where all extensions are linked in. However, I'm currenly working on redoing this so that the dynamic loading of PHP extensions will be possible. That way you don't have to link in every PHP extension just in case.
Q: And how about MySQL then?
MySQL was also initially ported by Petteri from Futurice. For sake of simplicity everything is built as static libraries, which currently makes PHP very large, since a lot of MySQL code is linked into PHP. I'm currently also working on changing this so that the client library of MySQL is exposed as a DLL. But it was quite straightforward to port MySQL; if memory serves me, the first version run after just a few weeks, which tells about the portability of MySQL and also about the "Linuxness" of Open-C.
Getting the first version of an open source software to run on top of Open-C would be even faster if it only were possible to run the configure script that open source software typically use for configuring themselves, according to what the current environment provides. Unfortunately, that is not possible at the moment, so what configure usually does, must be done manually on Symbian/Open-C.
But the devil is in the details. Even if you can make the software run quickly, there will be rough edges that it may take a lot more time to fix. Especially if the edges are related to some strange problem that you first must be able to pinpoint.
Q: Did you get - or even need - any help from MySQL employees or community members, or our online resources?
There is some information about porting MySQL, but not much when the target is a completely new environment. So the port was made without any external help. After we had released the PAMP stack there was a report that it is not possible to create indeces after a table has been created, but only as part of the table creation. So I posted a question in the MySQL forum, but before I got a reply (that said that the MySQL developers hang out on IRC), I had already figured out what the problem
The problem was caused by one fundamental difference between Unix and Symbian filesystems. On Unix you can open a file and then remove the file, but still continue reading from and writing to the file. Only when you close the file it is lost. This works, because in Unix a file and its name are actually quite distinct. If that sounds strange, just think about Unix hard links that allow you to have the very same file in two or more places under different names. You can "remove" the file in one place but it continues to exist in all others. So "remove" just removes the directory entry, and only when there are no entries left, the file itself is lost.
Anyway, on Symbian you simply do not have this behaviour. If a file has been opened, it cannot be removed, not by another process and not by you. And because of this, the index creation fails. And this is quite hard to fix without modifying the code extensively, which you don't want to do. So for the time being you have to create the index at table creation time or live without it.
Q: I believe Windows has this same behavior! At least I remember a lot of times I tried to delete a file or folder and couldn't do it, and it is more or less impossible to trace down which application might be holding some files open. And Symbian also has C:, D: and E: drives :-)
Yes, Windows is somewhere in between, but still closer to Unix. You can delete a file that someone else has opened, but the directory entry will not disappear until all handles to that file has been closed.