Talks from Linux Conf AU 2013

For convenience, here are direct links to four of the five talks we gave at LCA2013 in Canberra:

http://mirror.linux.org.au/linux.conf.au/2013/mp4/Making_Mobile_Communications_Secure.mp4
http://mirror.linux.org.au/linux.conf.au/2013/mp4/Serval_Maps.mp4
http://mirror.linux.org.au/linux.conf.au/2013/mp4/Serval_Project_Technology_Stack.mp4
http://mirror.linux.org.au/linux.conf.au/2013/mp4/Field_Trialling_the_Serval_Project_with_New_Zealand_Red_Cross.mp4

We can’t find what happened to the other one, oh well.

Root-free operation of the Serval Mesh

We have known from the beginning that expecting people to root their Android phones to use the mesh was not the ideal solution.  This is why we have put effort into liaising with the IEEE 802.11 committee, as well as encouraging Google to fix bug 82.

Bug 82 is the bug that describes the lack of ad-hoc Wi-Fi support in Android, and was filed back in January 2008, and at the time of writing is the fourth most starred bug in the Android bug tracker.  You can see it and vote for it here.  To vote for it, just click on the star next to the issue.

While we hope that this will be fixed one day, and ad-hoc Wi-Fi will be easily available on all Android phones, we have put considerable effort into a work-around.

That work-around is to alternate Wi-Fi on a phone between Access Point and Client modes. Combined with the Serval Rhizome store-and-forward protocol, this allows data to move from phone to phone.  The downside is that you can’t have an end-to-end connection over many hops. Nonetheless, we have created text messaging and file sharing applications, amongst others, over Rhizome and using this step-by-step approach to moving data by Wi-Fi.

More recently, we have sought to mature this ability to operate without root access so that the Serval Mesh software assumes that there is no root access, and to operate as best it can in that situation. This functionality is now more or less complete, and is expected to appear in Serval Mesh 0.91 in the next while.  You can try it out by downloading and building the development branch of the Serval Mesh at http://github.com/servalproject/batphone.

To give an idea of how this works in practice, I made a few screenshots of a nightly build of Serval Mesh 0.91-pre:

First, here is the new home screen.  The main difference is that the “Switch on/Switch off” button is now called “Connect”.  This name might change before 0.91 is released.


Instead of just starting or stopping the mesh, it now takes you to a list of available networks:
The operation of the mesh software on the phone itself is now controlled by ticking or unticking “Enable Services”. Again, the label may change and indeed we expect to make this whole display much more visually appealing. 
The Wi-Fi networks that are available or might be available are listed below.  Selecting “Auto Cycle” is still an option to get the mostly-automatic behaviour of previous versions.  But it is now much easier to pick a particular Wi-Fi network.
Note also that ad-hoc shows up, even though this is on an unrooted phone.  However, it won’t be used without user intervention.  
To attempt to use ad-hoc mode you touch the “Adhoc” item.  It then asks if you would like to try to setup ad-hoc mode, and reminds you that this requires root access, and might not be supported on all phones.

If you respond with “Test”, then it tries to get root access and enable ad-hoc Wi-Fi:

In the case of this phone, it fails of course, because it is not rooted.  But I can of course choose to join another Wi-Fi network, or just enable Auto Cycle mode, and any collection of phones running the Serval Mesh should then connect to one another or other networks when available:

So while there is some refinement to go, the Serval Mesh software is now able to work easily without root access, and does not try to get root access without asking the user first.

"Multi-threaded radio" for the ISM915 band for fun and profit

This is some ideas that I plan to implement that I wanted to capture somewhere public before I forget.  I plan to implement this in the RFD900 firmware for the Mesh Extenders.

The ISM915 band requires frequency hopping in most countries where it is proclaimed.

This is a bit of a pain, because the frequency hopping means that we need to synchronise all the nodes in a mesh such that they are all listening and talking on the same frequency at the same time. That is, they need to be on the same “virtual channel”.

The virtual channel is really just a hopping pattern, and knowing your position in the pattern.

It occurred to me some time ago that this imposition actually creates some interesting opportunities that we might not have otherwise considered for cooperative shared use of a block of spectrum.

The first step is to use a fairly fast frequency hopping pattern, hopping perhaps every 1ms – 2ms.

This hopping time is purposely shorter than the time it takes to send a whole packet.

The second step is to have each station only listen on a fraction of slots.  For example:

  1. Define a bundle to be a fixed number of, say, 10 consecutive slots.
  2. Each station listens on exactly one of those slots, based, for example, on the last three bits of its’ network address.
  3. The remaining two slots are a double-sized slot for broadcast packets. All stations should listen during this slot.
  4. This means that while the logical hopping rate might be 1000Hz, the effective rate for each station is reduced to two hops per slot bundle, i.e., twice every 100Hz, i.e., 200Hz.
This reduces the number of hopping actions for each station, and to some extent allows for a little slop in the synchronisation. 
When a station sends a packet to another station, it can predict which time slot to send in, because the receiving stations listen slots can be determined from its network address.  
The sender then switches to the appropriate channel at the appropriate time, and sends the entire packet, even though it might take many slots to do so.
In effect, the transmitter and receiver switch from the shared virtual channel to another channel.  
This means that each packet transfer “forks” off from the main channel, hence my calling it “multi-threaded radio”.
This means that as many packets can be in flight as there are real channels in the hopping pattern, and that they can all be used at the same time, but not by the same stations. That is, it has an intrinsic fair-share mechanism built in.
In terms of aggregate bandwidth, the result is potentially very pleasing.  
A 50 channel 128kbit hopping scheme such as we use on the RFD900 radio goes from being 128kbit/second throughput to potentially 50x128kbit = 6.4Mbit/second!
Yet, without the range reduction and increased receiver power consumption that Wi-Fi suffers by trying to make all that bandwidth available to a single station.
Anyway, I need to implement it to see how it works in reality, and whether the accurate synchronisation is possible on a completely distributed network with no external clock.

Serval versus Infrastructure

Today I needed to charge our Mesh Helper prototypes (which we are thinking of calling Mesh Extenders instead — anyone with suggestions for better names is invited to poke me with them).  This is to get them ready for the Adelaide Mini Maker Faire

The trouble was I forgot to bring our mains charger for the LiFePO4 battery packs in the Extenders.

That left our solar charging system as the only way to charge the batteries.  Fortunately this is South Australia, and we get a lot of good sunshine, even in Autumn.  The declination of the sun is enough though, that finding a nice 30 degree angle for the solar panel to sit on is a good idea.  Fortunately we have such a slope just outside the lab.  It is even North-facing.  So I hopped outside and setup our 40W panel with the two Mesh Extenders.  They each have a 120W hour battery, so 240W hours total is needed, which should take about 6 hours, but the batteries are already partly charged, so they should be charged enough by the end of the day.

Here they are charging happily without relying on any fixed infrastructure at all — not even for energy.  A very Serval moment.

Stepping back after I took the photo, I realised that I had inadvertently captured a great contrast between infrastructure and the low-cost, portable resilience that we are championing: In the background of the photo you can see the 250KW generator (including its perimeter fence to keep it and its fuel safe) and one of the University’s data centres, together costing millions of dollars, and our few hundred dollars worth of resilience mobile communications gear lost in the vista.

Of course, the University data centre does much more than what our little buckets of radio can do, but sometimes you only need a little, and can’t afford or sustain the big alternative.

Interfacing between the Serval Mesh and Cellular Networks (Part 1)

A number of people now have asked whether a Serval Mesh can be connected to the cellular network, e.g., to allow people living just beyond the range of cellular service to be able to make and receive mobile telephone calls.

This would be best done using a cellular modem and a bit of software integration to make use of it, so that the cellular audio can be passed through digitally.  However, most people who have asked about GSM/mesh gateways have indicated strongly that the solution must use only Android phones, and not require any soldering.

This is not unexpected, since it is hard to get new equipment into areas where disaster, rural, remote or poverty are at play. The same goes for areas impacted by war or unrest.

So we started thinking about how we can do this.

Unfortunately, most models of Android phone don’t let you access the audio from the cellular call directly, which rules out the easy option of writing some clever software that can run on a phone.

However, all is not lost.  We can use two phones, with one making the cellular end of the call, and the other handling the mesh end of the call.  Then by using headsets to do some good old-fashioned acoustic coupling, we can record the cellular audio through the microphone of the mesh-connected phone, and play the audio out the speaker of the mesh-connected phone and into the microphone of the cellular-connected phone.  A picture should help illustrate how this would be arranged in practice:

Yes, it really is that simple. And that ugly.  
For extra simple and ugly, you could just tape the two phones together with one upside down so that the speaker and microphones are close enough to couple the audio.
Of course it has some drawbacks, notably the need for two phones instead of one, and the degredation in audio quality that is almost certain to occur.  However, initial tests reveal that the audio quality is better than might be expected, and quite likely to be very usable.
Where we are up to now is that we have a student working on prototyping this.  He has got to the point where a call from on the mesh can trigger an outbound call to the cellular network, and has the two ends of the audio playing.  You can see this in the video below:

The next step, and the one that will get us to a complete usable prototype for mesh to cellular calls (the other way around is a bit trickier) is to make the Serval Mesh software know about wired headsets so that it can route the audio through that path.

Testing Serval Mesh Helper Prototypes in Moores Valley

Where we have been staying in Moores Valley near Wellington, NZ, the mobile phone coverage drops out half-way along the road, and power and even the bridge are cut from time to time by fallen trees and the like — all natural hazards of living in such a beautiful location.

Despite the isolation, it is easy to understand why people choose to live in places like Moores Valley
So for communities like this, there is real value in being able to easily make a backup communications system — which is just what we designed the Mesh Helper concept to enable.  By placing a few Mesh Helpers in homes, we can provide those houses with text and file communications amongst themselves, and possibly voice later on.  With a bit of extra effort, we can then connect the local mesh to the outside world as well, but that’s something we will work on later.
So I thought I would go for a wander up the road and see how far away I could get while still having a link.
Because I wanted the test to be reasonably realistic for this use-case, I just put one Mesh Helper on a book case in the patio here.  The only special effort I made was about five seconds to generally point the antenna up the valley in the direction I was going to head — and that is only necessary in these prototypes, because the antenna and large battery are sitting together in a bucket. 
We could do much better by placing the Mesh Helper on a high point, such as on top of a building, but I really wanted to see how it would perform with the kind of placement that could be reasonably expected by non-technical people who just want to use it, without investing lots of time, money or energy.
I then proceeded to carry the other Mesh Helper up the valley road, and check the signal strength at every house number.  As most people in the valley live on 10 acre sections, the houses are actually some way away from the road.  So the results may not be 100% accurate.  However, the road is generally lined with screening vegetation, which actually makes it a more challenging environment than adjacent to the houses themselves, which are mostly surrounded by paddocks.
Typical Road-Side Environment
Taking a measurement from on a letterbox along the road.

 I plotted the results on a Google Map, and colour coded the measurement points according to signal strength, with blue being best, yellow least, and red meaning no signal:



The result? We were able to obtain a link over distances of up to 1.27km, which is far enough to reach at least 13 houses in that direction, not counting those that might be reachable on the other side of the road, or on either side in the opposite direction along the road.  This means there are perhaps 50 homes that can be reached from one site.  
This is very important, because it means that a network can be constructed in a setting like this, even with only one out of 50 homes having a Mesh Helper.
Remember also that this is 1.27km between a pair of fairly casually placed Mesh Helpers. 
With units placed in good vantage points (and some of these properties include areas close to the local ridge line) with clear line of sight, we already know that links of >3km are easily possible.
Also, once we finish rewriting the radio firmware, we will be able to mesh many of these Mesh Helpers, we will be able to create links many times the range of a single Mesh Helper link.
Thus it might well be possible to connect communities in  rural, remote and developing settings over tens of kilometres.

3KM Mesh link using Mesh Helper Prototypes

After the short-range testing of the mesh helpers, we then set about creating a longer-range test so that we could determine the maximum useful range.  
As part of the exercise, a VHF repeater was located on a prominent ridge, and this seemed like a good place to put a mesh helper to give us the chance to setup some long-range links.
The VHF repeater is on the left, and our mesh helper is hiding under the plastic bag on the right to keep out of the sun.

Here is a closer view of the mesh helper lashed to the fence post before receiving the plastic bag.  You can get an idea of the kind of vantage point where the gear was placed.

We then proceeded to drive along the road towards Pongaroa with one of the mesh helpers in the car with us.  Monitoring of link budget was using the built-in diagnostic webpage that we incorporated into servald.  
That web page provides continuous link budget, calculated as difference between RSSI and noise-floor recalibrated into dB. I am not completely confident in the reading from this, but it is a good general guide.  
For those interested, the radios we are using are transmitting at +30dBm, and using +3db antennae, that are admittedly poorly placed next to a large LiFePO4 battery and other gear in the buckets. Receiver sensitivity is around -121dB @ 1200bps, and so should be expected to be around -100dB @ 128kbps, giving a link budget of around 133dB.
Earlier, we had setup a second site nearer to the cottage, the results from which are marked as “site 2″ in the following table.  The primary site by the repeater is approximately 1km from the cottage, almost in the opposite direction from the road we travelled along, so the distances here are underestimates of the actual distances.  The 2.65km and 2.68km records almost certainly correspond to actual distances of >3km.
The road varied in altitude, and had a lot of road-side vegetation and forest in places, as well as hills completely occluding line-of-sight to the sites where the other mesh helpers were located.  This is reflected in the varying link budgets as we travelled along the road.

Link Margin versus Distance
Link Margin Distance to Cottage

Notes
11dB 0.36km Adjacent to the bridge
6dB 0.75km road side
30dB to site 2 0.76km road side
13dB to site 2 1.00km road side
12dB 1.19km On the hill side
18dB 1.33km hill side
21dB 1.38km track
10dB 1.48km Side of road, next to a gate
2dB 1.63km road side
13dB 2.00km road side
23dB 2.00km parked, antenna vertical
20dB to site 2 2.19km road side
10dB 2.5km While driving
22dB to site 2 2.65km road
3dB to site 2 2.68km
14dB 26.02km Steady 14db margin, but no lock/link.

It is nice to know that we could maintain a link over distances of around 3km. There were of course spots along the way where link was not possible due to vegetation, hills and other obstacles getting in the way.
Between 3km and the 26km there were hills blocking any line-of-sight, and so measurements were not possible there, except in one place at around 12km, where we could not detect signal.
3KM is way short of what we believe these radios to be capable of, as reflected by the strong link budget (22dB) at around that range. Certainly 12km should be plausible, if not possible.
We are exploring a number of possible reasons why we were not able to obtain links at these distances.
One of the main culprits we are exploring is the forward error correction on the radios.  Currently they can detect and correct 25% BER in every 12 bits.  However, there is no block code or convolution code to allow graceful handling of bursts of errors. The addition of such a code should allow links at greater distances. The 14db reading at 26km, which was steady over several minutes, offers tantalising hope that with improved error correction links over such distances may be possible.  It is also possible that the number of black spots nearer might be reduced by such improvements in forward error correction.  
Something like Turbo Codes would be ideal, but are too complex to implement in the radios, but could be implemented in the wireless router.
Another source of trouble is the prototype units themselves, where all the components are sitting together in a plastic bucket.  There is almost certainly interference, occlusion and other problems occurring as a result.
It is nice to know that several kilometres is possible, and that there are some obvious areas we can work on to improve the range further.
What is significant is that the mesh helpers extend the range to useful distances, compared with the native WiFi capability of smartphones, as the following table summarising our experience shows.

Link Margin versus Distance
(with omnidirectional antennae)
Setting SmartPhone WiFi Mesh Helper
Urban, through dwellings ~20m ~200m
Vegetation ~20m >300m
Rural ~200m >3KM

Note that we are concentrating on omnidirectional antennae here, because our focus is on networks that are easy for normal people to setup, without having to aim antennae or have significant radio experience.

The Mesh Helper takes the range from being about one house to potentially tens of houses in distance, depending on the setting.

Thanks to the square rule, this means that instead of covering perhaps just one house, this means that we can cover potentially hundreds of dwellings with one unit, thus reducing the density of units required to establish a network — which is precisely what we set out to achieve.

If it turns out that with improved forward error correction we can push those ranges out further, then the benefit will be even greater.

One of the next steps will be to reduce the size of the mesh helpers to mobile-phone size, to make them truly portable, which is entirely feasible as the core electronics are already small enough.

Mesh link using our prototype Mesh Helper through ~200m of forest

Yesterday we setup a test link between two buildings here on the field exercise.  The distance between the points was about 300m — in theory short enough for a good WiFi link, like between a pair of Mesh Potatoes.  However, mobile phone WiFi has poorer range for a variety of reasons.  Also, there was 200m or so of damp forest in between, so any WiFi link would be likely to have problems.  You can see the situation here:
A perfect chance to try out our prototype Mesh Helper devices with their UHF packet radios. Basically they look like this, with a OpenWRT router, UHF packet radio and a big LiFePO4 rechargeable battery in a plastic lunch bucket:
So we put one mesh helper at the wool-shed (the building on the left of the image above).  The landing of the wool-shed was ideally situated, a bit elevated and facing towards the other building:
You can see the whole of the front of the wool-shed here.  The large antenna mast is a HF radio link that is part of the exercise.
At the other end, we tried placing the other mesh helper on the roof of one of the out buildings where it was close enough to provide WiFi coverage for nearby phones, but hopefully provide a link to the wool-shed.  

Unfortunately the roof was almost completely blocking signal, and so we didn’t quite get link to the wool-shed from there.  But by moving the mesh helper to the back of an adjacent building we got a strong signal (indicated by the RSSI and link-lock from the radios), although I forgot to write down exactly what the link budget was.

We then sent a message from the wool-shed back to the operations room near the other mesh helper device, which was successfully received:

What is nice is that all we have to do is to place the boxes, and everything connects automatically, keeping with our zero-configuration ideal for Serval Mesh networks.

The next step is to do some longer-range tests with one of the units on a good vantage point, so that we can get an idea of the maximum range possible with the current configuration.

Building Serval Mesh Helper Device Prototypes

We finally have all the bits and pieces we need to assemble three prototype Serval Mesh Helper (SMH) devices, that under ideal conditions should be able to support links over tens of kilometres, instead of the tens of metres that WiFi is capable of.  

The SMH has an embedded router (TP-LINK WR703Ns running OpenWRT — see http://developer.servalproject.org/dokuwiki/doku.php?id=content:meshhelper:prototyping_on_mp2&#etc_config_wireless for more information) running the Serval Mesh software with 802.11n WiFi  running in ad-hoc mode so that the SMH’s can mesh with each other.  They will simultaneously run in access-point mode so that phones can connect to them as WiFi clients, and hence not need to be rooted or otherwise able to do ad-hoc mode. This is the core functionality of the helper device.
The other capability in the helper is a long-range UHF radio operating in an appropriate ISM band, so that the mesh helpers can communicate over long distances. We are using RFD900 radios from rfdesign.com.au.
The fundamental components draw less than 1W sustained, and it can probably be optimised down to somewhere around 0.3W on average.
Here are some pictures of the assembly of these units.
First, for the power supply for the radio and RFD900s, we are using UBECs (little 5v switch-mode power supplies that are great for running from batteries).  The RFD900s can be powered from USB to TTL serial cables (we are using FTDI ones at the moment), but at full power it is a bit too much for the WR703N’s to pass through, so we are powering them separated. 
This entailed soldering up a power connector that can plug into the UBEC and provide the correct power outlets for the WR703N and the RFD900 radio.

 

Then it was a case of plugging all the bits together.  The following shot shows the connections to the radio modules:

Basically the black wires go on the left-hand edge, with the serial cable underneath, and the power connector on top.

We need a USB memory stick for extra storage (the WR703N has only 4MB flash, much too small for a Serval Rhizome node that might want to cache giga-bytes of data), which entailed adding a passive USB hub so that both the radio and memory stick could share the single USB port on the WR703N.

We are powering the components from large 120Wh LiFePO4 batteries, that should be able to run these units for close to a week.  We opted for using car accessory power connectors as vehicle power is the most likely backup power source we expect to encounter:

One of the (many) nice things about the RFD900 radios is that they support antenna diversity.  We take advantage of this by plugging in two antennae on perpendicular axes, so that we get horizontal and vertical polarisation (assuming we hang them vertically in the box).  We also have some +6db yagis for when the need arises.

 On the charging side of the batteries we used some nice connectors that we also fitted to our RedArc solar regulator, so that we can charge the batteries easily, including while using them to run the helper devices.

With such large batteries it is really important to make sure you have everything fused and protected against short-circuit, especially since we will be flying with these and don’t want to accidently melt a hole in a plane.

Then it was time to think about a convenient container that would fit everything, be easy to carry around, and be weather proof enough for use at KiwiEx 2013 in a couple of weeks time.  Some $10 storage tubs did the trick.  We will put a small bag of rice in each when we take them on field to absorb any condensation in the tubs.  You can see everything in there happily running:

Securing Mobile Telecommunications Talk from LinuxConfAU 2013

The full video of my recent talk from Linux Conf AU 2013 is now available:

http://mirror.linux.org.au/linux.conf.au/2013/mp4/Making_Mobile_Communications_Secure.mp4

The abstract for my talk is here:  http://linux.conf.au/schedule/30058/view_talk?day=wednesday

GSM/3G are surprisingly insecure, which is sad since good cryptographic frameworks exist. During the past year the Serval Project has been working on integrating very strong security into voice, text and data transfers on a mesh network. Rather than implement a secure SIP and secure RTP combination, we have taken a fresh approach and created a light-weight but secure packet and voice transport that is designed from the ground up with mesh networking in mind. One of the key innovations is using public keys as the network address, so that no key exchange or verification is required to setup an end-to-end encrypted channel. Consideration has also been given to how to defeat man-in-the-middle attacks for peers who are not able to verify each others keys prior to connection.

The system will be demonstrated in it’s intended application in open-source Serval Mesh telephones to allow secure telephone calls.
Part of the talk will discuss the technical details of the security model, but (hopefully) in a fairly accessible manner that most developers should be able to follow, and in particular avoiding getting buried in mathematics. Feedback on the security model is invited so that any obvious vulnerabilities can be addressed before the software becomes widely distributed.”