Friday Nov 04, 2005

Silent and Cool - Home Upgrades to Solaris x86

Solaris running on a mini-ITX fanless system inside a shoebox case is quiet and stable.

It's been a long time since I blogged. I guess I just got tired of blogging. And plus, I got caught up in other things in life like my first kid starting Kindergarten and all the other things that happen - like getting to know the school, the teachers, and the other parents and getting involved in the PTA. I also switched groups internally. I'm working with Solaris x86 and hardware vendors. It's a heck of a learning experience and lots of fun.

Speaking of Solaris x86, the momentum has really taken off. Someone mentioned that we hit over 3 Million downloads the other day on S10 alone. Wow. Not bad for GA - General Availability - in March of this year. But sadly, I've somewhat of a hypocrite. Sure, my work laptop has Solaris x86 on it, but it's triple boot, and to get wireless networking in airports, I'm still booting Linux more often than not. And at home, I'm still mostly a Linux shop, even if I do have Solaris on a couple of boxes.

But that time has come every few years to upgrade. My last attempt failed last year around the time Fedora Core 3 released. I had the FC2 CDs and was excited about upgrading, frankly because some of us geeks love slapping our own boxes together :-).

I saw a sale at the local Fry's and bought 3 low-power VIA c3 mini ITX boards for about $89 each back then. And shopping online, I found a sale on some Inwin low-profile bookpc small form factor cases for $40 with free shipping. I was pretty excited waiting for the parts to arrive and when they came in, I assembled my first box and slapped the FC2 install CD into the drive, booted up and then got a big REJECTION when FC2 quit the install telling me that my hardware wasn't supported. I tried some Redhat Enterprise and it too wasn't supported, and then some Solaris 9 CDs and that wasn't supported. And that wasn't the only disappointment, the 60mm P/S fan inside the case was a 5700RPM noisy bugger that screamed at close to 40dB. It was inward facing and already muffled by the case, but still, it was pretty loud, and at night, when the kids went gone to bed, it made my house sound like a data center.

I did manage to solder two 30 Ohm resistors onto the P/S fan wire and it slowed it down a little but it still whined loudly, albeit, no longer loud enough to make me want to take a sledgehammer and kill it. I knew I could go ahead and install FC1 which I already have running in my bedroom in a quiet PC. But, but I decided to download FC3 which folks reported worked with VIA c3 systems. The install was stable, but FC3 turned out to be quite unstable in itself, locking up constantly due to problems in the graphics. I tried the same board in different cases, different power supplies, with new DIMMs, and still had instabilities that would lock up the system even though the text mode worked fine.

I shelved those boards until recently, when I bought a couple of new Antec Aria cases. Initially, I ordered 2 for work, to do some testing in low power Solaris x86 systems and compare them to Linux. The cases were so quiet that I went out and bought two for myself. I also tried out FC4, but it turned out to be somewhat disappointing and not really stable either. And by instability, I mean that it would lock up after 20 hrs of uptime. I've downloaded all the upgrades for both FC3 and FC4 and it hasn't improved the stability, even though I've spent quite a few sleepness nights swapping boxes, power supplies, and memory. And with certain brands of ps/2 KVM switches, the Linux 2.6 kernel seems to hiccup and inject a button 2 or 3 event with using scroll mice.

But enter Solaris 10 x86. The installation is still buggy and the kdmconfig core dumps when trying to bring the XFree86 installer up. The failure converts into a pastel screen text console with funky colors for text when it should be black on white. There's also a problem with reverse text visibility. But the text is actually there and the install proceeds. It only happens with certain graphics chips and BIOS sets is what I'm told, like the VIA chipsets using Unichrome or formerly called CastleRock AGP. Prior to December 2004, Solaris had never been able to recognize VIA c3 as a valid x86 cpu and would crap out during the secondary boot. Since build 69 back late last year/early this year, the kernel team said they did a put back that fixed the CPU recognition issue. But this has really opened up the market for low-power OEMs like Igologic.COM to supply the Jbox running Solaris.

Ironically, I hadn't actually tried using the 3/05 GA release bits for s10 to test my mini-ITX boards back in March. I did test using some CDs that someone had burned for me with "S10x86 GA" on the labels, Disk 1 - 4, but when they installed, they crapped out on the secondary boot with the "Unsupported architecture" error and I never bothered to follow up with my mini-ITX boards being too busy. After all, there was a workaround for the bug. Prior to S10, I had been using a hack to replace the GenuineIntel recognition string in the kernel binary and on the ISO install image with CentaurHauls (some funky name for the family of VIA c3 processors). I was thinking that the S10 bits only supported some of the cores like the Nehemiah but not Ezra or Samuel cores.

But 7 months later, with a couple of test systems at work I want to install, I looked into the CPU recognition problem and behold, the old media I had didn't actually have S10 GA bits, but build 67 bits. So I went home and actually tried my own mini-ITX systems with S10 GA and it installed. With the exception of the kdmconfig errors during initial install, the process was straight forward and I had a graphical workstation up and running in an hour or so. Xorg with JDS boots and configures itself for 1280x1024 24 bit graphics. The USB storage driver works too, as does the camera tool. One needs to restart the volume manager daemon (/etc/init.d/volmgt [stop|start] ) which is a legacy service daemon not folded yet under Solaris 10's new Service Management Framework (SMF).

I had to go online using another system to obtain VIA Rhine Ethernet network drivers and the VT 8235/8xxx audio drivers as well. But both sources of drivers had recent tests by their authors on Solaris 10 and they provided portable driver build srcs that were just 100kB to download and supported Solaris 8, 9, and 10 versions. that easily fits on a USB Jump Drive and within a few minutes, I had audio and network up.

The build size and archive for these drivers is pretty incredible when you think how big a Linux cross-platform kernel driver build environment has to be, and the compatibility issues between kernel 2.4 and 2.6 versions. I've been running Linux since the early 90's when I saw my first set of CDs walking along the streets of Akihabara, Tokyo. They were Walnut Creek CDs if I recall, and they still distribute Slackware, if I'm not mistaken. But the kernel source build environment has really bloated since then, and for companies trying to support -for-profit- proprietary software, such build systems and lack of ABI compatibility is tough. I've worked with some Linux IHVs now that are porting to Solaris x86 because their minimum support source base was 30+ Gigabytes! This is for all the cross-compilers, GCC versions, and kernel source versions for each flavour of Linux, be it SuSe, RedHat, Fedora, and their corresponding updates, and 32-bit versus AMD-64bit. The comparative build and install on S10 is well less than one GB.

The Antec Aria case is around $100, and has front USB, Firewire, and 7-in-1 USB flash reader. Older VIA EPIA-800 mini-ITX motherboard with cpu and fan will cost around $110 today and probably not have all the pin-outs you need to hook up the Mobo. But if you're like me and look out for hardware deals on quiet PC components, you can core a sale on boards like I did for $82.95 at Fry's a few months ago on clearance for VIA Eden 600MHz ME6000 fanless boards which as firewire and compliant USB pinouts. A 160GB disk will run you about $69 after rebates, and an NEC or Lite-on DVD burner 16x with bonus floppy drive OEM might run you around $50 if you catch those online, one-day sales like at NewEgg. A 512MB stick of DDR can run between $30 - $90. Since these chipsets don't support ECC, I've learned that it's probably better to get higher quality name-brand memory. But for around $400, you can build a pretty quiet Solaris 10 x86 system that won't break any speed records or nuke aliens in gaming, but it works great for vpn, email, web surfing, and office productivity. And it's beautifully quiet.

Thursday Jan 27, 2005

Spam zombies and port scans - to log or not to log

Not much happened back in December except it was cold and rainy all along the West Coast. But weather aside, I was ignorant and blissful about the security of my networks both here and up at my other place in British Columbia. I did have a chance to drive north with the family and score on some Boxing Day sales in the great, dry and friendly North, but the visit was simply too short and I only had two opportunities to enjoy great dimsum. I've been back in California for quite some time now and quite busy. However, prior to my return, I setup my B.C. WiFi network to do more logging - both on the internal interfaces as well as the external WAN interface. And I had my reasons.

It was exactly a year ago that the big ISP in B.C. offered broadband DSL along Highway 99 up to Whistler, and I was one of the first on that leg to subscribe. In just that year, I've seen some tightening up of packet filtering on the ISP's network. Within about a month after DSL became available, apparently, I picked up intrusion attempts by at least three other compromised systems from neighbours. I sent out some email to the home owners up there to be on the lookout for unusually heavy network activity on their routers when they weren't actively using their systems. And I wasn't the only one that noticed. About 6 months into our new found bandwidth, the ISP decided to shutdown free flow of port 25 SMTP traffic from any subscriber except through their mail routers.

Such action by my ISP was annoying, but easily circumvented by tunnelling packets through a virtual private channel back to my mail server back in California, so my problems were solved, and I still retained some autonomy and privacy. But, I'm sure that for quite a few customers, including some of my neighbours up there, the ISPs actions caused some grief. You see, for some of my neighbours, a significant fraction of their emails began to bounce and were no longer getting to their intended and legitimate recipients. Mail was bouncing due to the ISP's mail servers getting onto DNS blacklists as primary sources of SPAM. And the reason why the ISP's servers got blacklisted was that the spammers adapted to the block on packets destined to port 25 on non-ISP servers; they decided simply to route email through the ISP's mail gateways. And to avoid the ISP from tracking down all the compromised systems, the spammers didn't ust use a few spam-running-zombies, these folks compromised hundreds of systems and had each one send just a few thousand emails and then stop after a couple of days, until the next campaign. This caused half of the ISP's mail servers to get onto some of the major DNS blacklist servers out there, and I would guess their tech support guys had to field a lot of calls from folks that ended up with rejected emails and needed to switch SMTP gateways. Three or four out of the half dozen ISP's mail servers in the lower mainland B.C. and Alberta are currently or were as of a few weeks ago, on the top world's DNS blacklists for sending too much spam. In fact, I think more than 90% of email coming out of their network is spam, at least I block about 80 to 90 spams a day from them with no abatement and 99+% of that is spam. And while I've configured my mail server at home to block spams and return a polite spam Error 550 messag, the ISPs around the world that route spam emails often just seem to ignore, or worse, forward the problem, as opposed to aggressively dealing with the situation and solving it.

A clear example was a case of rejected spam which I tracked was originating from a poor guy in southern California, who evidently, suffered a fatal disk crash after I contacted him and told him about the problem. He ended up having to re-format and install his operating environment. How I found out his system was compromised and a spam zombie was quite a coincidence. A month and a half ago, during a 30 minute period, I received over 1000 emails from about 5 MTAs worldwide that were bouncing an undeliverable spam to me, the apparent sender. Fortunately for me, 4 of those 5 mail servers included the message with full headers, and clearly, I could tell that the first hops and last hops were not from my IP address domain. But the ISPs should have easily figured that out and just killed or dropped the email or simply denied mail routing because the mail headers and addresses were so obviously mismatched. But as I said, many ISPs are just sloppy about mail filtering and don't bother. So despite me obviously not being the true sender of the spam, these ISPs just let me have the flood of bounced emails. In fact, one ISP's automated SPAM fighting machine apparently recognized the 250+ emails it got as spam, but then decided to reply to the faked Sender address with some legal-mumbo-jumbo about abuse of terms of service. Geez. I felt like half these ISPs were just playing dumb and arrogant. Clearly, their own header information encapsulated within the email indicated it was a spam and the sender domain and MTA IP address had huge mismatches.

To stop the flood for the next few hours, I decided to simply block all emails to that address and send immediate errors messages that explained that this address was not valid. The campaign did only lasted about 2 hours, and then the numbers of messages subsided after several thousand bounces. How many were actually delivered, I don't know. It did make me think.

But the lucky coincidence for me was that all the spoofed emails were using an email alias that I publish to the network for just my fishing msgboards. And by inspecting the headers in some of the bounced emails, I quickly found a common point of origin from an IP address in southern California. And the two pieces of information led me to check my web server access logs, and I did get a match, plus a bonus piece of info: a cookie ID. This cookie is something I plant in my web pages that can help identify unique sessions, especially identifying HTTP connections for logins.

This allowed me to identify the actual user and again, luckily, he had contacted me in the past and left an email address. Unfortunately, he was a skilful angler, but not a big IT technician, and so he wasn't sure beyond running standard anti-virus software how he could stop being this spam zombie. And unfortunately for him, I guess the folks using his system for a zombie were finished and didn't want many traces of their activity. Within just a day after I notified him, his computer disk crashed and all data was lost. After a week of silence, he emailed me back and told me about the crash, and the subsequent re-format and re-install of his entire system.

All this bad network activity in the past month or so spurred me to turn on aggressive logging on most of my home server and router systems. So just two weeks ago, I started to get a rash of panic emails from my router up in B.C. Evidently, pings of death were being detected and I had set the system up to email me immediately. Again, like the spam incident I was getting copious emails, this time not as quickly, but they were averaging many per minute and they indicated that the attacks were coming from 7 separate networks in at least 3 different countries. For two days, the router logs were arriving in my mailbox here in California almost once every 30 seconds. I wrote the ISPs to politely forward abuse emails to the right folks in their network to stop the attacks on my hosts. Most did have an automatic mail responder, but only the Germans sent back a personal response to my inquiries and told me that they have identified the host and have forwarded the headers and logs to those in charge of that subnet for investigation. After 3 days, and a few megabytes of logs and emails, the pings of death and port scans finally stopped up in Canada. I'm not sure if rebooting the router and getting a new IP address assigned was the trick or if the campaign just stopped.

Being the curious kinda person I am, I couldn't just be satisfied with the status quo, so I decided to turn on aggressive packet filtering and logging on my local systems here in California. I have two servers that run 24/7 and have open interfaces to the internet. I do have firewalls turned on, but I was not logging the packet rejections or denials. So for the last few days, I decided to turn them on and observe. Just between 2:45am last night and 8am this morning, I rejected about 120 attempts on my mail/web server, and about 40 attempts on my NAT firewall box. I have funky ICMP packet requests that don't look like pings of death. I have strange UDP and TCP attempts at really weird high port numbers that don't conform to any service or standard, and by far, the 90+% of denied packets are port scans for 139 and 445 NTFS file share UDP ports. And that was just in a little over 5 hours. It's incredible to me just how many scripts set off by hackers there are out there, and how many unique attempts occur to gain unauthorized remote entry onto a system there are. The costs must be staggering for folks with systems less robust and less protected against these hackers. But just having that knowledge itself can be pretty depressing, especially seeing how it means we need to be ever more vigilant against intrusion. It's almost enough to turn off the logging, save some disk space, and just live in ignorant bliss for a (short) while.

Tuesday Aug 17, 2004

Introductions - Hello World from PostickerGuru

Welcome to my first Sun blog. I had been reading the blogs from other colleagues for a while now but just never got around to putting up one of my own. The hardest part was figuring out the type of blog and picking out a handle that represented the gist of my existence.

Well, late last night, I couldn't figure out anything else more brilliant than PotstickerGuru. Afterall, my mailserver inside Sun (yes, I'm one of the few that hosts his own mail) is called Gyoza, and my main development workstation is a SunBlade 2K called Yumcha, and I have supporting backup and hot-archive systems in the office and at home named Shumai, Wonton, and Charsiu. Not that I have a food-fetish, but I think it's more appealing that say a Star Trek theme for hostnames, or stuffed-animal-disney-character theme, or smart-alec-unix-shell-command hostname theme. And after all, I am known within certain circles as a Master Potsticker maker - I even roll my own skins and I don't need a tapered rolling pin like some of the PSG-wannabees who can't pass muster with a straight rolling pin.

As some background info, I'm considered a senior engineer at Sun, going on 9 years now. Work has taken me to some interesting places around the world. I'm in the business of helping Independent Software Vendors (ISVs) and some customers and partners architect and build software. After they build it, I help them tune it to make it run faster. I used to be a C/C++ guy, but for the last 8 years or so, I've been focusing on Java more, and mostly at the application level. Only in the last couple of years, management has been kind enough to put me closer to the OS/Kernel where I've been looking more deeply at platform/OS provider performance. Rarely a dull moment here, and it keeps my brain from going senile.

Prior to coming to Sun, I was a real geek locked up part-time at Berkeley and then Livermore designing and simulating Thermo-Nuclear blasts in Inertial Confinement Fusion (ICF - aka Laser Fusion) Reactors. My dissertation was a long tome that included Analytical, Experimental, and Numerical work in multi-dimension simulation of blast wave propagation in Gas-continuous Two-Phase Media. I wrote some open source code called TSUNAMI (later limited in access for security reasons) which was an acronym for Transient Shockwave Upwind Numerical Analysis Method for ICF. I thought the acronym alone deserved a Ph.D., but my Professor at Cal wasn't as amused. But I must say, all that coding and work with C/C++ on Macs generating post-calc movies of the 250 microseconds after blast worked wonders with the DOE. They decided to fund the National Ignition Facility ($1.8Billion) at Livermore in part due to that work, and I think there are still a number of grad students and researches trying to improve my model and computational kernel even more than a decade later. Ironically, I'm now at a company that's trying to sell those guys the computational horse-power to run those simulations, and all my Physics and Math skills aren't put to intense use.

But my interests in working at Livermore after graduate school took a turn when Bill Clinton took office. He stalled the budget that year and I wasn't assured of any openings at the National Lab, so instead, I worked in Tokyo for 2 years. I spent the first year Post-Doc'ing at Tokyo Institute of Technology (Tokyo Kogyo Daigaku) in O-okayama - a better Engineering school than Tokyo University (at least the Faculty tell me that :-)). And anyway, exceptional singer Oda Kazumasa of "Tokyo Rabu Sutori" (Tokyo Love Story) fame graduated from Tokyo Tech. I did a lot of systems and network administration while running Computational Fluid Dynamics calculations on SunOS and AIX clusters. I was trying to develop faster running multi-dimensions Turbulent models.

One day, on a train, I bumped into another Gaijin (foreigner) who was the head of IT at Solomon Bros. Tokyo Office. We got talking and somehow, I got connected with some head hunters looking for Wall Street "Quants" who could program and do sysadmin. It wasn't long before I joined a small Tokyo outfit, Fusion Systems Japan and they put me on a bunch of projects building equity and fixed income trading systems, derivate risk management systems, and new GUIs, servers, and failover mechanisms. It was long hours, but really rewarding. The internet was just starting to boom in Japan and I even had a chance to work in NTTData's Ueno ops-center when the first batch of Ciscos and Ascends showed up. I helped startup an ISP in Tokyo too. All that scientific stuff faded in a year of doing finance.

I had always known how to program in a little C/C++, and I even had some tools experience with compilers and source control/build systems, and I knew a lot of UNIX. But I was never an O-O programmer or a real Systems programmer until my 2nd year in Japan. I owe most of that credit to four guys - first a Brit named John Tumulty who was first at CSFB in Tokyo, then he jumped to Goldman-Sachs - introduced me to the NIH class libraries and taught me the elegance of O-O. Second was a guy named Joe Diperna. Originally from the Fusion NYC office, but he came out to Tokyo to head up engineering for Fusion Systems Japan - he taught me all about product and build schedules and QA test automation. And third, Finn Christensen - a Norwegian with ties to Strustrap - the original guy who wrote the book on C++ - he taught me lots of basics about coding O-O network programs and Financial engines. Last, was Gary Arakaki - a Univ. of Hawaii transplant to Tokyo with a Ph.D. in CompSci. He specialized in the SunOS kernel and taught me tonnes about threads and scheduling and OS service providers - we analyzed the early concurrency models in Perl when it first came out with O-O support. There was a fifth guy, Jason Bloomstein, from Hal Computers who was doing a stint at Fujitsu back in the early 90's. I met him at a bunch of Gaijin parties and Potsticker-Making parties (which I hosted) back then in Yokohama/Kawasaki - my 3 bdrm, living/dining/kitchen (3LDK) in Japan was just in Kawasaki on the Nambu line, close to Futako-Tamagawa-En station. He often criticized me for being a dilettante with Operating Systems. He didn't think highly of application programmers and instead loved to put me down by asking simple CompSci test questions that any upper division Stanford student would know. Well, I admit, I am a Cal Berkeley grad, and I was a world-class Nuclear Engineer and Computational Physicist, and not a CompSci major, but at least I could configure circles around his DNS and hack his systems if he wasn't careful! Sometimes, folks just gotta recognize that not everyone goes through an academic route to become a good computer scientist. Some, like me, get it from the school of hard knocks. But Jason's criticisms have made me stronger, I think, and that's good, even if he is a Stanford grad!

In early 1995, I was on site at First National Bank of Chicago in Hibiya. I can't remember if I was still on site before or after the Tokyo Subway Sarin gas attaches by Aum Shinrikyo, only that I was in Maui giving a paper on Internet, Web and Networking at the time and noticed on the news that it was the train station I usually get off to go to one of my customers' offices. I couldn't help but think it could have been me dead or paralyzed for life.

Around that time, Sun released something called Hotjava Browser and the Java Development Kit. I was caught up in the excitement and while on site at a realtime data provider, I wrote my first stock feed applet. The Java applet ran in HotJava Browser and accessed a CGI script on a system that was authorized to connect to the datafeed. I demo'd the system to others, including the folks at the data provider. To my dismay, they rejected the idea and told me I was in license violation, even if I was ready to give them the concept and source code. To make a long story short, the data provider suffered major financial losses in the follow 2 years, and no longer charges $2400/month per seat for real-time data,, and others have undercut their derivatives risk-assessment desktop tools so anyone can leverage a Java applet to get those tools from any discount online trading house. And ironically, 18 months after that demo, I was invited back to Japan and the same guys who had been chastising me about license violations were bowing and asking "James-Hakase" to provide them some architectural guidance.

It was inevitable that I joined Sun. I have used their technology since the 80's and love it. I believe that better technology should win, but I think I understand the market well enough to know that the best technologies don't always win, because some technologies aren't quite self-evident, or they come out too soon. For example, Sun produced the first diskless workstations that were flat panel mono-chrome monitors with the computer built in that worked fairly well back in the 1990. It's now almost 2005, and network computing is finally mainstream 15 years later; monitor-based and Tablet PCs are now going more mainstream. But admittedly, not everything we produce rocks the world. Lots of it fails to gain any market because the technology isn't appropriate. But that's why working with ISVs has been so rewarding. ISVs are brutally focused on the market and provide solutions to enterprise customer problems. It's cool knowing that your ideas may be impacting a big corporation that will then impact millions of people around the world, either every time they stick their Bank Card into an Automated Teller, or when folks go online to order Fishing Tackle or a Fishing License to fish, say, in the state of Florida.

But back to Potstickers. Yes, I do have an illustrated recipe. You can get it [here] . The recipe requires lots of labour. Yes, I'm a do-it-urself kinda engineer. I roll my own potsticker skins and stuff them with my own filling just because it's a self-satisfaction thing. I think of the line in Repo Man when Emilio Estevez's character comes home and whips out a can of generic food, opens it and starts to eat it out of the can, and his Mom hollers, "Put it on a plate dear... it'll taste better."

Yeah, the feeling is kind of like that. Yes, I like to fix my own cars, wrap my own fishing rods, lace my own wheels on my bicycle, and install my own French Doors. I also run my own static IP network out of my house, host my own virtual servers, build my own Whitebox PCs, and run Linux and Solaris at home. But I understand if not everyone adheres to the same standards for Do-it-urself. I cheat too. So it's okay if some of you sneak off to Costco and buy a bag of 50 potstickers and grill them up yourself. They taste okay and I won't call you a dilettante.

Happy reading. I'll be back...soon.




« July 2016