Friday Dec 26, 2008

Experimenting with VoIP

Being part of a multinational company with distributed engineering means most meetings are conducted partly or purely over the phone. While phone companies were able to leverage cost benefits due to technology improvements, competitive pressure from new technologies are forcing some to drop their tariffs for basic services. VoIP is the prime example. Unfortunately, they didn't deliver the feature bundles and rate plans at truly competitive prices to prevent the proliferation of providers.

I pay my local “traditional” telco to deliver dial tone, but they're trying to milk every last cent out of me for this service, rather than innovate. While they're service is reliable (which is an important criteria for me), their prices keep going up and I know their costs are going down (apart from executive salaries). A$30 per month just for dial tone! No included calls, etc. I even have to pay an extra charge to have an unlisted number – I need to pay them “hush money” so they don't sell my personal details.


Now for the good news - my ISP in Australia is Internode. They regularly drop their prices, provide more download quota and greater bandwidth, which matches my ever increasing data demands. Their service is also very reliable, I can't recommend them highly enough. It's usually weeks or longer between DSLAM resets or other outages. A$90 per month for 25G of ADSL2+. This will be even cheaper when my ISP populates the exchange / CO with more of their own equipment, and I'll get more service offerings!

My Australian VoIP Provider

Apart from data, voice the is other vital component for my environment. I've experimented with a few VoIP providers, but settled on a local one (recommended by a peer) - My Net Fone. Voice is hard to do, when packets don't arrive quickly and regularly - the wheels fall off. My provider has some interesting plans, but the one I use (GlobalSaver) gives me a DID (local number) which can be anywhere in Australia and 100 free calls per month that can be within Australia or to 30 countries. Voice-mail is included, and you can even have that delivered as an email attachment. Having it delivered to my iPhone (which could be anywhere in the world) is neat, being able to forward it to my wife is even better! This costs A$15 per month.

My US toll free number!

One thing that my local VoIP provider can't do are US DID's. (local numbers). The choice of providers in the US is large, but I don't need a calling plan in the US – I'm happy with my local provider. I don't really want my local calls within Australia going via the US – the latency would hurt. I recently found a US wholesale VoIP provider called Gafachi. Being a wholesaler means the rates are good, but you don't get assistance configuring your device – you're supposed to handle that. They also don't mind having direct end users as customers, the only difference is that my volumes are lower, but it's no extra cost for them. The particular product that interested me was the inbound US toll free number. It costs US$2 per month, then US$0.0176 per minute of use! Now my colleagues and friends in the US can call me cheaply. Calling home when I'm traveling also becomes a lot easier, and cheaper.

My modem / router / VoIP & wireless device

The device I use to tie all this together and do wireless is a DrayTek 2700VG. I try to reduce the number of devices running, so this was a good one to combine all this at a reasonable price. From a VOIP perspective it provides two FXS ports, i.e. two independent lines that can be configured to send calls (by default) to different providers (SIP accounts), and can also route calls through the PSTN (the ADSL line).

Configuring VoIP should be considered in each direction independently – inbound and outbound. Configuration information tends to vary slightly between providers so experiment a little. While I have NAT enabled, the VoIP component isn't really behind the NAT, so I don't need a STUN server.

Profile Name
Register via
Call without Registration
SIP Port
Act as outbound proxy
Display Name
Account Number/Name
Authentication ID
Expiry Time
1 hour
NAT Traversal Support

Ring Port
VoIP1  VoIP2
Ring Pattern

Calls from the VoIP device:
  • The SIP server is entered into the proxy field.
  • The domain / realm specifies the domain of the authentication information. Some providers don't check it, while others want the server name.
  • The account name is the SIP login name from the provider.
  • Some providers require the authentication field, but tends to be the same as the account name.

The main SIP account list screen shows whether the Draytek has registered with the SIP server. If it has, you should be in business. There are a number of other settings like CODEC, but they tend to be obvious from the providers documentation. It's also easy to test outbound calls, but remember to use the correct long distance codes depending what provider you use.

Account Name
Ring Port




Calls to the VoIP device:

Receiving calls can be a little more complicated depending on the addressing information sent by your provider and where the call originated from. This router expects addresses in URL form. e.g. sip: Non-PSTN originated calls shouldn't be a problem as the caller would specify your SIP account, but PSTN originated are different as the caller doesn't specify a URL – your provider does a translation and routes the call to your device. This requires your provider to know where you are on the internet, hence it's important the router register with the SIP server (so it knows your IP address and port), particularly for internet connections using dynamically assigned addresses. The other important thing is the destination address your provider sends with the call. Some providers map your account name in, while others like Gafachi send the number the caller dialed. This is understandable for a wholesaler – you could could have many inbound lines from them, all with different phone numbers, and you may want to route them to different extensions on a PABX. This is where the Draytek caught me – it won't accept a call if it doesn't know about an account or number (it will reject it with a 404 response code). I needed to configure a simple SIP entry where the account name was the PSTN number (this is the yyyyy entry above, e.g. 18669536677). The entry doesn't need to register with the SIP server (as it knows where you are from the other entry), but it tells the router to accept a call to that number, and also specifies which FXS port(s) to send the call to.

The Draytek can be configured via a browser or CLI (for the brave), but the CLI will provide some debugging information.

> voip debug showmsg
This is a normal registration

 -->Send Message to <09:48:17>
Via: SIP/2.0/UDP;branch=z9hG4bK-bRg-14631;rport
From:  <>;tag=kZg-4725
To: <>
Call-ID: dSJ-21796@
Contact: <sip:wwwww@>
Authorization: Digest username="wwwww", realm="",
nonce="acc78e806fd414c4", uri="",
response="30c41178afb803cd364d41aeeaf1961a", algorithm=MD5
Max-Forwards: 70
Expires: 3600
User-Agent: DrayTek UA-1.2.3 Vigor2700 Series 2.8.1
Content-Length: 0

<--Receive Message from <09:48:17>
SIP/2.0 200 OK
Via: SIP/2.0/UDP;branch=z9hG4bK-bRg-14631;received=;rport=5060
From: <>;tag=kZg-4725
Call-ID: dSJ-21796@
Server: Gafachi UAS v110.08
Contact: <sip:wwwww@>
Date: Tue, 23 Dec 2008 04:55:03 GMT
Content-Length: 0

This is the start of an incoming call

<--Receive Message from <09:57:01>
INVITE sip:yyyyy@ SIP/2.0
Via: SIP/2.0/UDP;branch=z9hG4bKe803f015
From: "unknown"
To: <sip:yyyyy@>
Contact: <sip:unknown@>
Call-ID: 3b033fc8d0d82693d102cd76b76e4f60@
CSeq: 102 INVITE
User-Agent: Gafachi UAC v110.08
Max-Forwards: 70
Date: Tue, 23 Dec 2008 05:03:46 GMT
Content-Type: application/sdp
Content-Length: 238

o=root 2125136926 2125136926 IN IP4
c=IN IP4
t=0 0
m=audio 46896 RTP/AVP 0 101
a=rtpmap:0 PCMU/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=silenceSupp:off - - - -

-->Send Message to <09:57:01>
SIP/2.0 100 Trying
Via: SIP/2.0/UDP;branch=z9hG4bKe803f015
From: "unknown"
To: <sip:yyyyy@>
Call-ID: 3b033fc8d0d82693d102cd76b76e4f60@
CSeq: 102 INVITE
Content-Length: 0

-->Send Message to <09:57:01>
SIP/2.0 180 Ringing
Via: SIP/2.0/UDP;branch=z9hG4bKe803f015
From: "unknown"
To: <sip:yyyyy@>;tag=BEC-22495
Call-ID: 3b033fc8d0d82693d102cd76b76e4f60@
CSeq: 102 INVITE
Contact: <sip:yyyyy@>
Content-Length: 0
Seeing this debugging information gave me some insight of the interaction between the Draytek VoIP device and service provider.

Going mobile

Something I'm yet to try is using a cheap, portable VoIP device, or software phone on the laptop when I travel. I can connect to my VoIP providers from anywhere in the world, so I have true number portability. My VoIP provider lets me have two lines in the plan - I assume one for home and one for traveling!

Thursday Nov 13, 2008

Welcome to Fishworks!


My role at Fishworks has been to enhance the kernel to enable features like hot plug, multipathing, clustering and device management in the Sun Storage 7000 series, particularly the 7410. These features are the sort of thing you should just expect in an enterprise grade storage product. You need to be able to dynamically change your storage hardware and withstand component failures – transparently. Ultimately these kernel changes will go back into generic Solaris to support Sun's J4x00 storage products, but for the cool features you'll want a 7000!

The past couple of years have been an amazing time for me. I've worked with the most talented, motivated and passionate team of engineers creating a great product line. There have been many long days and nights, with all the thrills and spills you'd expect, so I need to thank a few people: the Fishworks team, our testers (particularly Michael Harsch), Javen Wu, and most of all - our families.

Now that I can tell you what I've been working on, here are some of the details.

Talking to the disks:

Fishworks appliances (as we call them) utilise a combination of SAS and SATA disks, both traditional rotating magnetic disks and Solid State Disks, with the hosts using LSI 106xE series HBA's and onboard controllers. These controllers have the ability to communicate with both SAS and SATA disks, and are capable of providing both internal and external storage connections. This line of controllers is also capable of performing on-chip hardware RAID and even multipathing, but we don't use either of those features in our product (for a number of good reasons), it's all done in Solaris.

A heavily modified version of the mpt(7d) driver is used for these controllers and the associated parallel SCSI controller - the 1030. The design goal was to enrich the driver and frameworks to handle cascaded JBOD's in a multipathed, hot pluggable environment. mpt already handled parallel SCSI, hardware RAID, SAS, SATA, and was recently modified to support multipathing for the 2530 SAS arrays, so it has a long history.

Apart from the groundbreaking use of SSD's, these appliances provide bulk data storage using large numbers of disks. The important factor being the disks are connected as JBOD's and directly attached disks, and not using complicated hardware RAID and storage controllers. The significance is the reversing of the trend where operating systems communicate with a small number of storage targets (with a number of disks hidden behind enclosure controllers) to an environment where the operating system directly manages all of the individual disks. From a driver perspective, this creates extra challenges to maintain performance, particularly when using commodity parts.

J4x00 JBOD architecture:

While SAS disks are capable of supporting multiple paths, SATA disks do not. To overcome this, transparent active/active SATA multiplexers are attached to each disk, this provides two SATA connections per disk. While the mux is technically a single point of failure for access to the disk, there are individual muxes for each disk, and the use of ZFS provides protection against individual disk and mux failures. The internal connection of disks in the JBOD is a star topology with an LSI SAS expander at the centre, these expanders can be thought of the functional equivalent of an ethernet switch. The topology is duplicated in the JBOD to provide multiple paths, with two host side connections of each disk mux connecting to different expanders. The external SAS ports also connect to these expanders. The connections between the muxes and expanders are one phy wide, while each external JBOD connections are 4 phys wide.

A “novel” aspect of SAS is STP, the SATA Tunnelling Protocol – it allows a SATA device connection to a SAS port. To the operating system, SATA disks looks similar to SAS disks. The HBA translates SCSI commands (used for communicating with SAS disks) to SATA commands and tunnels them over the SAS fabric. The expander removes the tunnel and speaks native SATA to the mux. Another role of the expander is to co-ordinate requests from multiple initiators (typical of clustered environments). The current generation of LSI expander only provides single affiliations, simplistically meaning only one initiator is allowed to communicate with each disk, attempts to access the disk from another initiator, even from the same host, will be rejected – this is per path. Different hosts can still access the same disk via different paths hence, like regular SAS disks, access needs to be co-ordinated. Keith's clustering does this for you!

Not only do the expanders provide access to the disks, they also provide SMP and SES services to interrogate and control the JBOD enclosure, this all happens in-band on the SAS channel used for data.


OS and storage musings, with a dash of other technologies for good measure.


Top Tags
« April 2014