"How I got into doing Security for a living"
By davew on Nov 19, 2009
...or, if you want to put it another way, "Limited Confessions of a Security Geek".
It's been almost precisely a decade, now, since I found myself working in computer security.
I've been asked, on occasion, how I ended up doing what I do - after all, for the most part, security folk can be divided into two categories; those who started out trying to prove a point by doing things to expose inadequate practices (such as my pal Alec, who wrote Crack to expose standard 8-char Unix passwords for the insecure things they are), or my pal Darren, who came from a background of formal security training in terms of protective markings, etc, and now gives us civilians the benefit of his knowledge.
(There's also the crypto illuminati and general anomalies such as Whit, but it's to our misfortune that such folk surface so infrequently.)
I wonder whether security folk are made, or born. I note with considerable interest that a course on "thinking like a security geek" is now being taught at Washington Uni, and I was also pointed at an interesting article comparing the mindset of a security geek, to that of a mathematician skilled in formal analysis. It's also interesting to note that, of the many security folk I know, few took degrees in CompSci.
In my case, interest in security, followed from interest in networking. I may be showing my age by admitting to being a Micronet 800 subscriber, back in the early to mid '80s . My interest in communications was such that I got my first modem before I even got a floppy disk drive, for my BBC Micro - CommStar had to be modified on my behalf, to fix bugs associated with saving things to cassette tape - but once I realised that one computer could talk to another, it's wasn't long before I figured out that, where a computer had multiple levels of privilege on various accounts (hey, bear in mind that I was in my early teens, and back then and there wasn't such a thing as a remotely-affordable home computer with a time-sharing environment), then nefarious things could be done, to elevate one's privilege on the system at the other end of the 'phone. Of course, while I figured this out in theory, I was a good boy, really :-).
I didn't go hacking while at University - indeed, I was a sysadmin for the Computer Society's shared Unix box, for a while. Granted, I expect it was completely riddled with security holes and hope it has long since been removed from JANET, but it worked.
I first really got interested in security during my last couple of years in my first job at Acorn; if you remember Larry Ellison jumping up and down and getting hugely enthused about thin-client computing in the late '90s, it's a little-known fact that the Oracle Network Computer (abbreviated NC; the original post-dumb-terminal thin client) was a reference design from Acorn, based on an ARM 250 and running a cut-down and embedded RISC OS.
Guess who used to do a bunch of work on the demo-environment server end, where the servers were frequently Risc PCs running RiscBSD :-). In fact, pretty much the last thing I was doing before I was RIFfed, was working on securing NC-to-server communications.
However, I'll hold my hands up and say I'm sad to admit, that's actually the last occasion on which I did any Real Programming. After my days (5 and a half years, come to think of it) at Acorn, I haven't written any actual code. I can still readily go down to the level of designing and debugging state machines, but it's not the same.
After Acorn effectively imploded in the Autumn of 1998, I went through some doldrums and small-time contract work before being picked up by Sun, at the start of '99 - and Sun is where I've been, ever since.
I joined Sun as a general jobbing Project Engineer; after a bit of a learning curve, I was sent off doing Solaris installs, various builds and generally delivering standard services, etc, until I ended up on a gig at one of the UK national newspaper publishers, initially doing enterprise-wide Y2K patching. Their security was, frankly, appalling - some systems didn't even have root passwords, and others had root passwords known to almost the entire IT element of the organisation, and these passwords were which were never changed. It's the latter case, which nearly cost them their business.
Once I'd done all the patching - involving being in their offices at very strange hours and often at weekends, owing to the downtimes they were able to schedule for their systems (the Sunday editions have to come out, after all) - I basically became recognised as "a face in the office of the internal IT group", and started to overhear all sorts of discussions around security. When the Sun project manager was replaced, for political reasons (the replacement, Kevin, became and remains a very good pal of mine), the security agenda was escalated so Kevin had visibility of it; so, a contractor was hired in to develop a security strategy and policy, for the enterprise.
After a few weeks, the contractor submitted a draft security policy. I asked to see it, and saw it for the pile of cut-and-paste crap that it was. I told Kevin that I could give him something better by close of business that Friday, and at 16:50 on the Friday, I hit "Send" on my mail client.
Kevin, bless him, agreed I'd done a better job, got rid of the consultant, and took me on as a security proto-geek. Fundamentally, that's why I'm where I am, today.
However, much more amusement was to come.
The customer's senior management, decided that they liked the new security strategy and wanted it implemented. So, this being in the days before
JASS the Solaris Security Toolkit (SST), system hardening was done to a manual - and time-consuming - script that I'd written.
About halfway through configuration roll-out, we found that a sysadmin in the Output Services group, had resigned under "something of a cloud" when he was advised that his shift patterns were being changed to something that he considered unreasonable. He had subsequently approached a rival newspaper, with his knowledge of operating practices and access controls, and offered to disable the systems of his former employer, such that there would not be any January 1st, 2000 editions of any of their newspapers.
(A note to my readers; UK newspapers operate on extremely tight financial margins, which are governeed by their advertising revenue. In the event that a paper does not hit the news stand before or at the same time at its rivals, the contractual clauses with the advertisers are really punitive. For a group which produces several titles, a day's outage on all their titles could result in financial damages so huge, that the enterprise is effectively taken to the cleaner's.)
So, our Bad Guy (let's call him Fred, as it's not his name) was basically handing his former employer's competitor, a bankruptcy of their primary rival, on a plate.
Fortunately, newspaper editors do have some integrity. Our Man at the rival publication, notified the police.
The first I knew about any of this, was when I and the on-site Sun Technical Project Manager were summoned to the IT Director's office.
We were sworn to secrecy (even from our own Project Manager), and introduced to a couple of extremely cool gentlemen (a Detective Sergeant and Detective Inspector, respectively) from the local police force's High Tech Crime Unit.
Kevin was, therefore, shut out of everything which was going on, for a fortnight; we were still being paid for, but he couldn't know what we were doing. I consider it a measure of his character, that he trusted us to get on with the job in hand.
Forensics gigs (and I've done a few, but don't need to use more than one hand's worth of fingers to count them, even now), tend to run one of two ways; the customer either wishes to mitigate the risk of someone compromising their systems by the same attack, or they want to find the perpetrator and nail them to the fullest extent of the law.
The newspaper firm in question, wanted to do both. Usually this is next to impossible, but we had the extreme luxury of already knowing who our Bad Guy was. So, what we needed to do, was protect the systems he had access to, as best we could, and encourage him to make another attack, so we could find out where he was entering the network from and gather evidence.
Fortunately, Fred was a known entity; it was definitely known which systems he had root access to, and fortunately, these were environments which only processed transient data. For folk outside the newspaper industry, Fred had worked in Output Services; the function of this unit is to take the fully-formatted pages which emerge from Editorial, and work a little magic on them. Specifically, Output Services is where medium-resolution images, as manipulated in Editorial, are substituted for the high-resolution images from Image Services; also, it's where page pairing takes place. "Page pairing" is the process by which 4 newspaper pages are put together into a double-sided sheet a little more than twice the width of an editorially-output page; this dictates what the printers print, and also adorns the pages with the day's date and the page numbers.
So, we set about our forensics gig. We knew Fred wasn't the sharpest tool in the box, but we knew he still potentially had root access to systems vital to next-day publication. So, we got radical. As the systems Fred potentially had root access to, were both resilient system-wise - there being multiple boxes configured as failovers, to do the same job - and had mirrored disk arrays, we removed a layer of resilience by breaking the disk mirrors and offlining one set of disk arrays; even if Fred decided to do an "rm -rf /" on a system, we could potentially have it back up in the next 5 minutes and ready to receive data, in a manner which would only adversely affect data flow for less than half an hour, and would most likely confound Fred's efforts.
We couldn't apply any obvious security lock-down to the systems Fred had access to, though, as it was reckoned that he would run rampant and destroy any systems he might have access to, before we could shut him out of them, if he saw any obvious modifications. Talk about playing cat and mouse... Next, the team (comprising Muggins here, Ray the Sun Technical Project Manager, the customer's Head of Networking, the contact point for a particularly important networking application, and our two new friends from the Police) identified network segments that Fred might attack our environment through, and posted network sniffers (interesting boxes from Dolch) on them, to detect any anomalous connections.
We were also very fortunate to be able to persuade the editor of the rival paper, to contact Fred and ask for proof of his hacking capability. This seriously brave guy wore a wire, and recorded his conversation with Fred; the challenge was for Fred to hack the page pairer systems such that on a particular day, and on a particular page, "Thursday" would be changed to "Thrusday", and that would be the evidence that Fred could do what he liked, with Output Services.
The page was hacked, the change was made. The readers didn't notice.
We did, though :-).
It turned out, that Fred had got a new job with the maintainers of the newspaper printers. Now, I don't know how many of my readers, have seen a newspaper printer in action; they're pretty impressive beasts, standing up to 3 storeys high, constructed out of skyscraper-style steel I-beams and running paper at significant width, tension and velocity. Anyway, Fred had joined a firm which manufactured and maintained these beasts, and was using the dial-home fault alert line, to dial-in to the printer and hence bridge out to the enterprise IP network.
We spotted the traffic to the page pairers, once we had Output Services ring-fenced with Dolch boxes.
We had everything in Output Services not only set up for short-term service resumption in the event of system destruction, but had logging seriously cranked-up to detect Fred attempting access.
We also had an out-of-band channel detecting differences between pages submitted by Editorial and pages as paired, but that's a little bit of a trade secret, as to how we did it :-).
Anyway, Fred spent 18 months being housed by Her Majesty (for readers outside the UK, this is a euphemism for a prison sentence).
So, that's how I properly got into security. It's so interesting, I've wanted to stay there there ever since. I've pretty much succeeded, in doing so, too.