Indie web movement

So it seems that I’m not only into open and free software but also somehow belong to the Indie web movement. But then again, this seems like a natural reaction against all this cloud centralization thats going on. Fewer and fewer actors owning more and more of your personal data.

What sparked this post was that I read an article on Huginn which is an agent system much like iftt (If this then that) or a scriptable RSS reader on steroids. Agent systems that where all the buzz in the late dot com era (~2000) when i where involved in that.

Huginn seems like a really interesting project that i really have to take a deeper look into later. If only it weren’t written in ruby then 🙁

But then to return to the main issue for this short post. Iftt, even if it seems like a great thing. Are people really trusting one  cloud service with access to “all” their other accounts on the web? Like Dropbox, all their google accounts, Facebook, Instagram and Evernote to mention a few.

First off I cant imagine that ifttt can get a working business model by just looking at the recipes you do and since there seems to be no advertising they somehow has to get money out of the information they gather. Second its obvious that this would be a goldmine for any authority requesting information out of cloud providers. Finally losing control over this account would send you chasing application password revocations all over the place.

With all that said Huginn still looks really interesting and perhaps one could modify it to be language agnostic 🙂

Posted in opensource | Leave a comment

Avahi and local domains

If you like me have a crappy ISP that for some strange reason has a record for the “local” domain in their DNS.  (This is some thing that the avahi daemon is not fond of since it wants to handle that one by itself.) And you run dnsmasq on your server, like fx the Excito B3, there is a simple central solution to this.
Add the following line to your dnsmasq config file:


(/etc/dnsmasq.d/bubba.conf on Bubba)

Which tells dnsmasq that it should forward any queries regarding the local domain to Google’s DNS which does not contain any “local” record.

Posted in Bubba, Linux | Leave a comment


Today’s useful command dmidecode

From the man page:
dmidecode is a tool for dumping a computer’s DMI (some say SMBIOS) table contents in a human-readable format.”

Very useful if you fx want to know which motherboard that resides inside your computer.  I wanted to know what memory to get when upgrading RAM on my workstation. Dmidecode told me which motherboard i had and which memory slots where used and with what memory size these had installed. (Intriguing how many useful utilities there are out there that you never heard of. 🙂 )

On similar topics. Just the other day i learned on a new Linux device file that I completely missed, even if I have been working with Linux for the last 10+ yearsm had never paid attention/noticed.  /dev/full the special device file that always returns ENOSPC when written to 🙂

Posted in Linux | Leave a comment

Corrupted initramfs during boot

What would you do when your embedded Linux suddenly starts to break in mysterious ways? I was faced by this problem “recently” and this is a compilation of my notes when faced with this somewhat strange problem.

I worked for a customer who has a small Arm system booting of a NAND flash and then serving a few peripherals over TCP/IP, RS232/485 etc. The system also runs a small web server to allow easy configuration of the system.

The system is developed by people no longer involved in the project and I was hired to do mainly bug fixing and minor adjustments to the system. During this I stumbled on a problem so interesting that I think it’s worth a blurb.

When the system booted it was noted that occasionally the system would not show the login page when accessing it via web interface. Instead the user was greeted with an error message when trying to log in.

At first i thought that this was just a simple CGI script that bugged out for some reason. Man, was i wrong.

Doing some initial investigation indicated that it was the login script that for some reason produced “illegal” output making the web server throw up. But then when running the CGI on the command line the first surprise showed up:

/root # export QUERY_STRING="xxx-deleted-xxxx"
/root # python /usr/lib/cgi-bin/
/root #

The script halted with an illegal instruction! This is nothing you see everyday with a python script. My first suspicion was that for some reason the pre-compilation of modules failed, which is done during boot up of the system, and I tried manually doing this. Everything worked.

Manually starting python also worked and produced a working prompt. But launching the CGI would not work. Reducing the content of the script trying to deduce what made it crash. It turned out that importing urllib or the CGI module made the interpreter crash with an illegal instruction whilst other imports worked nicely.

/root # python -c "import cgi"
/root # python -c "import os"
/root #

Rerunning the import with verbose output:

/root # python -v -c "import cgi"
# /usr/lib/python2.4/socket.pyc matches /usr/lib/python2.4/
import socket # precompiled from /usr/lib/python2.4/socket.pyc
dlopen("/usr/lib/python2.4/lib-dynload/", 2);
/root #

So, it was when python tried to dlopen a native module that everything went wrong. Further testing confirmed this. Loading a module with native code indeed crashed the interpreter whilst working nicely otherwise.

On a side note, doing this work was very time consuming because the system behaved nicely most of the time. In my setup I sometimes could reboot the system hundreds of times before the problem showed up. Thank good for Selenium helping me automating this. I wrote a short python script that would try to log in to the system and if successful then would reboot it through web interface and try again. (If you are into serious web development and haven’t checked Selenium out i seriously recommend doing so 😉 )

Another strange behavior of the system was that when the problem occurred it was permanent for that boot. I.e. the bug persisted until a reboot. Something in the boot process seemed to trigger the bug under rare circumstances.

Armed with the knowledge above that something regarding dlopen might trigger the bug I went on writing a short C-program which loaded a shared library, resolved a symbol to a function in the same library and called it.  This to isolate if it was Python that was responsible for the error. To my big surprise this program, as well, terminated with an illegal instruction! The problem was not located in Python but on a lower level.

Further reducing the C-program removing all but the dlopen and dlclose functions. The problem still persisted. Thus it seemed like the problem might be located within in the libc, uClibc, performing the dlopen function.

This was a big problem because since this was an embedded system there where no debug symbols available on this level. All libraries where stripped to save space on the target system and the project provided toolchain provided no binaries with debug symbols neither.

So I had to do the best of the situation. I recompiled the libc on my host computer with debug symbols enabled and started a remote debug session against my test application running on target. (To be able to run the remote debugging on the target system i loaded a USB key with the target debug tools and mounted these on the target system)

Then on target i ran:

/ # /mnt/cg/gdbserver /mnt/dlopentest
Process /mnt/dlopentest created; pid = 529
Listening on port 2222

and then connecting to this from my desktop:

arm-none-linux-uclibcgnueabi-gdb dlopentest
(gdb) set solib-search-path /home/tor/build/uclibc/testinstall/lib
(gdb) target remote
Remote debugging using

The key here being the set solib-search-path that tells gdb to load my debug versions of the libraries instead of the toolchain provided ones. (Note that this is most likely not optimal since this is not the original libraries running on target)

Running gdb on the test application loading different libraries gave some strange findings. First off data seemed garbled in gdb when entering library code, libdl and

Program received signal SIGILL, Illegal instruction.
0x4000f494 in dlopen (libname=0xbee5ed88 "h���\b \001", flag=74072)
    at ldso/libdl/libdl.c:434

But not being sure if this was due to my separately compiled or the code itself was a setback. But I had also noted that during some runs I got a segmentation fault instead of the usual illegal operation. When I got the segmentation fault this occurred before main was entered.

Program received signal SIGSEGV, Segmentation fault.
0x40004df0 in check_match (sym=0x40013304, strtab=0x40013504 "",
    undef_name=0x40026b0d "error_message_count", type_class=0)
    at ./ldso/include/dl-string.h:78

The crash then was during the initial shared library loading performed by and not during dlopen as in the other situation. was clearly not even able to load libdl.

At this time i started to question and what was going on inside it. I mounted the target via ssh to my desktop and started examining the library and to my big surprise got:

arm-none-linux-uclibcgnueabi-objdump -a
BFD: invalid string offset 602752145 >= 133 for section `.shstrtab'
BFD: don't know how to handle section `' [0x 400c040]
arm-none-linux-uclibcgnueabi-objdump: File format not recognized

Objdump was unable to parse the library! Using readelf to confirm this:

There are 17 section headers, starting at offset 0x20b0:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .hash             HASH            00000134 000134 000140 04   A  2   0  4
  [ 2] .dynsym           DYNSYM          00000274 000274 000290 10   A  3   3  4
  [13] <corrupt>         <unknown>: 400c 00009f98 001f98 000068 04  WA  0   0  4
  [14] .bss              NOBITS          0000a000 002000 000004 00  WA  0   0  1
  [15] .ARM.attributes   ARM_ATTRIBUTES  00000000 002000 000028 00      0   0  1
  [16] .shstrtab         STRTAB          00000000 002028 000085 00      0   0  1

Something clearly was wrong with libdl. Using dhex, a binary diff program, I also could confirm that there actually was differences in the corrupt and the working one.

This would explain a lot of the behavior I was experiencing. The all or nothing, either works the complete boot or not at all.

When the illegal instruction happened the library that i tried to load would be corrupted and when python crashed with a segmentation fault would be corrupt.

Clearly something must corrupt the image after it has been loaded. The complete image is checksummed by the bootloader, uBoot, before the kernel is started. Further more most of the times the boot seems to complete without problems.

One further notation is that when the customer built the images the system seemed to fail more often. To examine this a bit more i built a virtual machine with almost the same specs as the customers workstation. In this i recompiled the image and started testing it.

Already om the second round the system failed. This was very repeatable approximately 50% of boots would fail on python.

Investigating this further remarkably showed that now it was no longer a problem with the shared library loading. All my previous tests would pass but the login script would not work, terminating with a segmentation fault.

Some serious debugging showed that loading the string module in python would crash the interpreter. Digging into that module revealed that it was when the module issued a map instruction the thing would explode. I could reduce code triggering the  problem to:

/ # python -c "map(abs,[1])"

The plot thickened.  Since earlier problems was caused by corrupt files it most likely was the cause here as well. To analyze the situation i gathered md5 checksums of all files in the directories bin, etc, home, lib, root, sbin, usr and var and compared these sums with one calculated on the same files on the build machine.

This “quickly” revealed that in this case it was libpython2.4 that was the bad guy this time. Running my dlopentest on this library gave me the same result SEGV.

So there seemed to be a generic file corruption problem. The question was why this was so isolated to python and libraries used by python. To verify the extent of the problem I wrote a small python script that used pySerial, pyexpect and Gnomes GVFS.

The script would use the console on serial port to mount the usb memory with debug tools and sftp server, launch the sftp server and then mount the target file system on my desktop computer. Then the script would launch the md5sum calculations on all files store the result in a file and then umount everything and reboot the system to start all over again.

The output of this operation where a lot of files all containing the md5sums of the files of every run. I then compared these sums to the original file with checksums from the build machine.

To my very big surprise none of these files yielded corrupted files. They all gave the same checksums. Thus rebooting the system via command line most likely did not trigger the problem. Still rebooting via web browser  immediately caused problems.

The conclusion then was that the system corrupted the image when interacted with over the network but not when it was operated via the serial port. This yielded that it most likely was some problem in the lower levels of the system.

This was also confirmed in parallell by one of the other members of the development  team narrowing the problem down to the ethernet controller.

Analyzing the bootloader u-boot revealed that someone from the original development team had reenabled the ethernet controller just before the system where about to launch the Linux kernel. A comment indicated NFS-boot as the cause for doing so. (Even though this should not be needed when the Linux kernel would reinitialize the network controller during boot. Before doing any NFS operations.)

Leaving the ethernet controller running during Linux boot was indeed the culprit. The controller is set up to automatically transfer ethernet frames into buffers in RAM when they arrive on the wire. But when control was handed over to Linux this memory was reserved for other use. In this case it matched the location where the initramfs where loaded.

So then when there came network traffic addressed to the unit in the time slot between the start of the kernel until Linux had reinitialized the network controller this would corrupt the file system image by overwriting parts of it with the arriving frames.

Lessons learned would be to never trust the lower level of the system when experiencing intermittent hard to track errors. I wasted a lot of debug time here due to starting from the top of the stack working my way down.

Posted in Linux, Programming | Leave a comment

Your digital rights

This morning i read an interesting article here: “The Coming Civil War over General Purpose Computing” which really hit one of my areas of interest.

It’s a good read on lock down of computing devices. Please let me stay in control of my devices.

I “daily” get irritated on devices locked down leaving me out in the cold. Be it smart phones and tablets, streaming media in general including “over the air” broadcasts etc.

All these are pushed upon me in the name of increased user experience but in reality its only meant to limit my freedom locking me into another silo.

I often have to pay more for less. Take music or videos as an example, before i had a physical copy which i could decide on how to use. Today I’m supposed stream it or store the copy on a controlled device. What if i want to see the same movie clip tomorrow? Sorry, its no longer available…

Lets say that i buy/get a shiny new PVR for my TV. You should be completely aware that you wont have control over saved content any more, even if it’s stored on your PVR hard drive. Broadcasters can decide on how long that it should be available for you. Want to make a copy of the same movie you recorded last night? Forget it… Lets say the PVR breaks down with a hard drive full of content. Lets move the hard drive or content to another device. Forget it, the new PVR wont accept that and further more content on the drive will be encrypted.

The really sad part is that i often end up paying more for the “service” because i do get a “better user experience”, better quality etc. But in reality often these argument is only used to in the same time slip in these DRM measures of which I’m of course not told.


Posted in Uncategorized | Leave a comment

“WAS” assembly, part two

This is the third article on my mission to build a Web Application Server for my home needs. The two first articles can be found here “Lets build a WAS” and ‘“WAS” assembly, part one‘.

My assembling was abruptly halted last week when i realized that the motherboard would not fit into the casing i had bought. Well i decided to most likely build a HTPC of that casing and power supply later. So how to proceed? The only way forward would be to get a new casing. After some research i settled with a 80W Morex T3500 a “small” casing with a fanless power supply. Approximately 750 SEK. (Actually a bit cheaper than the original casing but not as small 🙁 )

New casing

New bigger casing

Lets try to finish this computer building and get on to the more interesting parts.

Inside new casing

Inside new casing

Not a lot of room inside there either. Interestingly the hard drive and the optical unit is located in a space underneath the main compartment. I see no reason to put in a DVD or Blu-ray player into any computer, installing is so much easier with USB media these days and i wont play any movies on this unit, so i left that out.

New casing with SSD mounted

New casing with SSD mounted

Adding the motherboard after some serious “precision engineering” 🙂

New casing all parts mounted

New casing all parts mounted

To wrap it all up, adding the lid!

New casing, all done

All done!

Then to get started i need a base operating system on the unit. My default choice would of course have been Debian. But since Debian Squeeze starting to get slightly old i got a bit nervous on how it would play with the Intel Sandy Bridge CPU. The kernel in Squeeze seems to be something like 2.6.32. Not being adventurous enough to run testing i actually opted to  run Ubuntu Server edition.

The installation was very painless. I built a USB installer from the ISO-file for the 64 bit Ubuntu Server. Booted that and selected to install SSH-server and a LAMP stack. (Done in under 15 minutes!)

I now have a functional machine to start playing with! My main goals now are to get a solution providing (web-)mail, (shared) calendars and shared contacts. These services shall be available via browser, tablet/smart phone, and fat clients like Thunderbird or Evolution.

Posted in Hardware, Linux | Leave a comment

“WAS” assembly, part one

This is the second article outlining my work building a Web Application Server for home use. The first article is here: “Lets build a WAS”

So, with all parts on the table lets build a new computer then. First lets mount the CPU, following the directions on the scarce instructions folder. (I ended up downloading the complete manual later)

CPU fitted to motherboard

Then add DDR3-RAM

Mounted RAM

Huge DDR3 RAM mounted

Those pieces surely are tall. Measuring this against the casing revealed that my initial fear of the memory modules being to tall to be absolutely true 🙁

Tall RAM modules

Tall RAM modules

There is approximately a 5mm “lack of” space for the lid to fit with these modules mounted. (Mental note, don’t assume anything regarding sizes of  “standard” computer parts.)

So what to do now? My first thought was to swap memories with the ones i have in my workstation, also 8GB DDR3, that should work. But wisely i decided to first mount the rest of the components to make sure the rest would work.

It should turn out that this would be a double failure. This really wasn’t the right day to build a new computer 😀 When doing the first test placing the motherboard into the computer casing i quickly realized that the power supply would not fit. Intel had placed the main power connector at the absolute outer part of the main board PCB. Which would have worked if it wasn’t turned the way it was, with the plastic clip facing outwards. This resulted in it being both impossible to fit the MB into the casing and the wiring of the power supply where also impossible to fit.

Misaligned powersupply

Misaligned power supply

It was apparent that this really would not work. I had to halt work for this time working on a solution.

Posted in Hardware, Linux | 2 Comments

Lets build a WAS

Components for the new computer

This will hopefully be the first article of a few covering the building of my new home application server.

I have been toying with the idea on building a new server for home use for a while now. I need something a bit more powerful then B3 to run web applications on. No disrespect for the B3 that is a shiny piece of hardware doing what it’s supposed to very nicely, delivering the best user experience possible, per watt of power consumed. But i think that i have met the limits on what its capable of in some ways. My plan is to demote the B3 to file serving duty and use the new computer running mainly web applications.

So i started gathering data on what was available and this is what i have come up with this far.

Wanting this piece of hardware not eating my electrical bill for lunch i decided to try keeping power consumption somewhat low.

I started choosing CPU and ended up deciding to use the Intel core I3-2120T being a low power I3 with a dual core 2.6GHz Sandy bridge with a max TDP of 35W.

After some reading of tests i decided to go with an Intel DH67CF motherboard since it seems to do best with regards to power consumption and performance. I also had Asus P8H67-I as a strong contender but finally decided for the Intel brand. Both these boards come with 6GHz SATA-III and USB 3.0, hopefully, making sure that i get good IO performance.

Intel DH67CF motherboard

The Asus motherboard had two other things not being right for me. It has a Realtek® 8111E ethernet controller which at least previously had bad Linux support. Further more the Deluxe version, the one i could order when looking around, had integrated WIFI which i where uninterested in.

To further keep power consumption low and performance high i opted for a SSD drive. My plan is to only run the Linux OS, web, database and most likely mail server on this machine. All other files such as media etc will be left on the B3 server. Thus i don’t need a lot of space.  I choose the 2.5″ OCZ Vertex 3 at 60GB using a similar one in my workstation being happy with it.  (According to specs it should use ~3W when in full operation)


As primary memory i just picked up 8GB (2x4GB) of DDR3 from Corsair. This might actually be me my first problem. Opening the packaging i saw a pair of huge heat sinks. These most likely will not fit into the casing i got….

8GB of Cosair DDR3

well we just have to wait and see.

Estimating the power consumption of the above mentioned parts i would guess that it would use ~45W power when exercised a bit. With that in mind i got a 90W DC power supply to give it its DC power i got a no-name 80W brick.

To house all of the above parts i got a, butt ugly, aluminum casing from Emko

Emko alu casing

But since this piece should hopefully not be visible on a daily basis the WAF is not the primary concern here 🙂

So what does all these parts cost? I must admit that i did not hunt for the absolute lowest prices on the net. I surely could have got this cheaper than what i did. But i estimate that all parts on the first image cost would be something like 4500 SEK (Swedish “kronor”) or about Eur 480 at current exchange rates.

As a final note i have to show what came in one of the packages from one of the retailers.

Candy bar from Inet

I’m not really sure how i should decipher that. Sure its a nice gesture to have something to chew on while assembling the components, or something to comfort one self with when everything goes wrong.  Never the less, i think its quite a PR trick 🙂

Well thats all for now. In the next part of this series i will try getting all parts to fit into the minimal casing. As said above i have strong doubts that the memory modules will fit. And when thats done its time for the most interesting part, what software to put into place. There are a few new applications that I’m very interested in trying out.

Posted in Hardware, Linux | 6 Comments

Long time no blog

Hi there!

It has been quite a while since my last post. The reason for this is quite simple. I have been swamped with work.

Since I left Excito I have been working as a consultant/freelancer and “unfortunately” this have taken a lot more time than anticipated.

So what do one do when working this job or in other words what have I been up to? Among other things I have been doing some serious clean up of a project around a CPU-board used in a cash handling system. I have done some base work on a BSP for a train computer. What was interesting on those assignments was that the first project ran on the same processor as B1 and the other used the big brother to the processor used in B2 🙂

Currently I do specification work for a project on a water cleaning product. Talk about an alien environment for me used to work exclusively in a Linux workspace. I use a Windows XP based laptop to do my dirty work, mainly utilizing MS Word/Excel. When digging into their code I use proprietary CM systems and Keils uVision IDE.

And even if the assignment is interesting in many ways it has one major side effect. It makes me love my Linux Desktop even more 🙂 Lets face it we have accumulated quite some kick ass applications over the years.

I know that there is something called Win7 which is supposed to be a nice environment. But WinXP sucks big times compared to any recent Gnome-desktop and yes, I know that XP is some 10 years old. But never the less XP is still used widely in the corporate world.

(Mentioning this I however can’t resist giving the Gnome people some critique. I mean Gnome 3, come on. Talk about loosing touch with your user base. Its a wonderful architecture beneath the surface, no doubt about that and Gnome 2 being almost perfect for my needs. But Gnome 3 makes me feel like being treated like a retard.)

But lets focus a bit on the MS world and start by talking about MS Office. Talk about a bloated behemoth. Using and later LibreOffice I sometimes cursed loudly about missing or incomprehensible features. But a few weeks with sluggish responses, corrupt tables, and almost unmanageable embedding of objects I will think twice before cursing over LibreOffice again.

Then come development environments. Keils uVIsion makes you feel like you where back in the nineties again when used to Eclipse and CDT. I however can understand that a “closed” company wants a centralized CM system even if that means that you are left with all problems that are associated with that. Git, I miss you!

So to sum that up, I just wanted to say that we really have a kick ass environment in the Foss world.

But on other news. Apart from pure work I have now teamed up with PA, who also left Excito this year, and a former colleague, and close friend, from way back Håkan. Currently we all do consultancy work for our new company but I have high hopes that we also will be able to do some other cool things in a not to distant future 😉

Until next time!

Posted in Uncategorized | Leave a comment

Nautilus Open in Remote Terminal

Open in Remote Terminal

Open in Remote Terminal

How cool is this! I stumbled upon this today. I use the “Open terminal” plugin in Nautilus (Linux Gnome). And when using this on a remote ssh share in this menu it presents the alternative to open a remote terminal.

Lo and behold, choosing this pops up a terminal running a terminal session on the remote device. This is so freaking useful and I have wanted this feature for so long.

Now if I only could get the reverse function. Right clicking in the Gnome terminal and get the option to open Nautilus in that location. (Preferably on remote devices as well)

That reminds me of another useful application. ZSSH, a wrapper for SSH enabling zmodem transfers to and from remote devices. “All” that is needed on the remote device is the zmodem application suite rz/sz and thats it.

Use zssh as a replacement. I.e zssh instead of ssh.

To upload files to target just press [ctrl]+space and you will enter the zssh shell. Enter sz “local filename” to start the transfer of files to target.

To download files from target start by issue sz “remote filename” and then press [ctrl]+space to enter the zssh shell. There issue the command rz to receive the files.

The really nice part is that this works regardless on how many hops you have made. So this works equally well if you first zssh to some gateway, from there you ssh to another and possibly a third host. ZSSH still does the tricks without open up a channel home.

ZSSH is available packaged for at least Debian and Ubuntu.

Posted in Gnome, Linux | Leave a comment