www.lewiz.org

It snowed at Lake Guillemont yesterday.  The snow was fairly white and cold.  I didn’t crash into anything on the way to work.

Anton and I decided to make constructive use of our time in the morning, and so headed out to the Lake before breakfast.

Arguably more interesting than the great big smiley face we collaborated to draw are the two letters to the very left of the photo.  Those readers that are not blind may be able to make out that they are A and N.  Clearly the start of Anton’s name.

Nice work, Anton.  What worries me most is what might have happened to Anton should he have pursued this endeavour… did he not realise that the Lake was sat fair and square at the same position as O and N.

Anyway, here’s a bigger picture to put things into perspective.

Chris would have you know that the five or so minutes spent jumping in a big circle were watched closely by at least five people stood in the corridor between SPARC and Solaris House.  I’m even told they found the whole thing rather amusing.

(P.S. thanks to David for taking the photo!)

This machine is a bit of a beast.  Fully populated it has 12 UltraSPARC III CPUs, 96GB of RAM and a weight rivalling that of your local Indian elephant.  With these impressive figures in mind, I now tell you that it is the lowest end machine in the Serengeti range of machines.  Its bigger brothers and sisters include the 4800/4900 and the 6800/6900.  In addition it shares a lot in common with the Starcat systems: the Sun Fire 15k and 25k (a herd of Indian elephants).

Over in our second lab we had a sick 1280 that was throwing all sorts of errors during the power on self tests.  Months ago I spent some time looking into this and came to the conclusion that none of the system or repeater boards were faulty.  By process of elimination (I could have said Sun Global Resolution (SGR) here; it’s the same thing) I came to the conclusion that we had a faulty baseplane.

The V1280 is quite different from the rest of the Serengeti and Starcat systems in that it was designed primarily for the telco industry (the Netra V1280).  For whatever reason, the engineers decided to orient the boards vertically and as such the V1280 is equipped with a fairly unique ratchet mechanism to support boards as they are being inserted into the chassis.

Anybody who has ever dealt with a 1280 will know exactly what I mean when I say that the first time you install a board is a memorable experience.  The ratchet system is very effective in supporting the load, so much so that to get the board down into the chassis requires a fair amount of force… all very well good until you find out (at the cost of your system board and baseplane?) that the ratchet system stops about an inch before the bottom of the chassis.  This discovery is usually accompanied by a loud crashing sound and potentially a hefty repair bill.

That said, I like this mechanism.  If your first experience isn’t a bad one and if you understand the system it works very well.  The trick is to utilise the latches at each end of the board… use these to push the board into the chassis and, when it nears the bottom, hold them in such a way that they cannot leave the vertical position and snap shut.  If this is done the board will drop just a short distance but the clasps will hold the delicate interconnect away from the internal baseplane.  Once this has been done it is a case of gently closing the clasps to properly install the system board.

Unfortunately it seems that somebody wasn’t aware of this technique and the cost was our V1280 baseplane.  Fortunately we didn’t lose a system board at the same time.

Months went by until last weekend we reworked the lab and moved this system to a rack that has an anti-tilt mechanism (it’s just a foot that sticks out).  I immediately ordered a replacement baseplane and got around to installing this yesterday.

The procedure is relatively straightforward, but the twist is that the baseplane is on the bottom of the chassis (logical, but different from the other Serengeti boxes).  After removing the power supplies, the main fan tray, the IB_SSC board and a crazy internal power board I was ready to lift out the three system boards and two repeater boards.  The proper thing to do here is to lift them up a few inches until they are properly held by the ratchet mechanism.  As I’ve already mentioned, this is strong and holds the boards well.

Next up is ten minutes lying on my back undoing the screws on the bottom of the chassis.  “Undo 31 of the 32 screws securing the baseplane.  Remove the final screw and be prepared to support the weight.  WARNING the baseplane is heavy.”

Fortunately all of this went off without a hitch.  One thing I particularly liked was the ability to lie with my head under this monstrous box and look up at the five high density interconnects above my head.  Each “pin” is so fine that it can (and often do) pierce a hole in the surround of the female high density connector they mate with… certainly not something I’d like to have land on my head!

The good news is that following the baseplane replacement the system is back up and running.  I have one issue left regarding the two onboard gigabit ethernet interfaces, but this is a very minor issue by comparison.

Another job well done :)

Last week Liam, James, Chris, Anton, Wilson and myself were all on a SunRay training course being held for OpenAnswers.  We were lucky enough to get the places in return for setting the conference room up with a whole pile of Ultra 60s, a few Ultra 10s and some SunRay 1s with attached monitors.

I had messed briefly with SunRays before the course (setting up SunRay@Home) and obviously we use and support these on a day-to-day basis.

I think that of us all it was just Chris and Anton who hadn’t quite gotten around to setting up a knock-off SunRay at home.  For those of us that had, the first day or two was very straightforward: it covered installing the packages, the very basic setup and initial setup of the DHCP server.

Beyond the first day or two a lot more useful things began to crop up: the full workings of failover groups, exactly how/what happens in multi-head and Xinerama setups (although I swear Sun haven’t quite got their definitions of multi-head and Xinerama the right way around) and fancy multi-failover groups on the same subnetwork.

All in all there was plenty of useful information, although much of it was in the form of “gap filling” and getting some actual hands-on experience.

I’ve got myself booked onto another training course starting this coming Monday morning.  I’ve opted to go for the Solaris System Performance Management (SA-400) course, which appears to cover plenty of really useful and interesting stuff.  I’m still desperate to get on the Solaris Internals course, but the big problem is finding enough other people wanting to attend.

As most people will know, I’m working for Sun on a one-year contract. My job involves setting up hardware configurations for support engineers. As part of the lab team I help run a large lab packed full of almost all of the hardware and parts Sun have sold in the last decade (or more). We get hands-on experience with pre-release, beta and even alpha hardware; we can work head-to-head with the engineers to resolve issues affecting customers.

In addition to this we get to play with some pretty funky software, for me this was playing with the Logical Domains support on the Niagara boxes way before it had even been announced to the customers.

The structure of the lab team in our office is probably quite different from many support organisations. We have the big boss (that’s Paul), David (who has an uncanny ability to remember all sorts of useful info) and then the five student “lab rats”. Paul would argue against calling us lab rats, but it does a good job to describe what our primary role is. In addition to the student guys is Wilson, an ex-student who is now back with Sun on a contract; he works on software issues and server maintenance. The reason the structure is interesting is because of us students… currently we’ve been at Sun for a little over six months, at the beginning of this period we were more or less new to all Sun hardware, software and the Way Things Work, but so too were the guys before us, and those before them, and so on.

In simple terms: every twelve months almost all of the accumulated knowledge in the lab team buggers off back to university to let a new bunch of clueless guys take over.

So, back on track: it’s six months since I started my job, and this means that it is time to interview second year university students to help find another five guys to run the shop once we’ve gone. We reviewed a whole pile of CVs before Christmas and let a number of applicants know that they had been successful in getting a telephone interview, the first of these was last Tuesday and was conducted by Paul and David (most of us students were still slacking off on holiday).

On Wednesday Anton and I sat in on a couple more telephone interviews and were given the opportunity to run the show the following Thursday.

The telephone interview structure is quite straightforward: Paul introduces the job, provides some more info as to what exactly it involves and then one of us begins the technical side of the interview by asking a number of UNIX questions. There is a short break in the middle for more non-technical questions and a chance for the interviewee to ask any questions she/he may have, before a few more technical questions to finish the interview off.

So what was it like to act as an interviewer for a job I was interviewed for just six months ago?

Surprisingly difficult. It looked and sounded very straightforward, but when you get down to it you know that the way that you phrase the questions has a direct impact on whether or not the candidate will be able to give the answer we want, and could therefore be the deciding factor in whether or not they reach the second stage of the interview—I quickly realised that if I bugger the question up and the interviewee gets the wrong end of the stick, changing their perception of the question is next to impossible.  And, as Paul says, stressing out the applicants isn’t going to help us make any sort of decision.
My general feeling was very much for the person I was interviewing: I wanted them to do well and did it was quite difficult to put myself in their position and think how best to pose the question.  Unfortunately I think I made a mess of one question, and then made it even more confusing by re-asking in a slightly different manner.  That said, after sitting in on a few interviews even the most straightforward questions can be misinterpreted, but often with a little help we can appreciate that the person on the end of the ‘phone does know the answer, and that nerves or “newness” to non-face-to-face interviews are at play.

I originally started this blog entry on January 9 2007, but I’ve managed to keep it sat in a semi-composed state for quite some time.  It is interesting to note that on January 9 2006 I was heading down from Manchester to Guillemont Park for my second interview.  Obviously I got the job (goes without saying, right? ;) and I’ve been very happy ever since.  I’ve got about five or six months left at Sun and then I’ve got to sort something else out or try and bribe Paul to give me a really good reference for somebody else in the company!  In the meantime… I’ve got plenty of stuff to be getting on with.

I’ve just purchased my first annual multi-trip insurance cover.  All past insurance has been single trip of over 30-days.

There are so many things that I need to cover before I catch my flight to Shanghai Pu Dong airport the Friday after next.  I’ve barely had chance to even think about buying Christmas presents, and time is really running out: I’m heading back home tomorrow or Saturday; this will be the last time I see my family in 2006 (and the last chance to give them presents for Christmas day—unless I involve the good old Royal Mail).

If everything goes according to plan I will be moving house next weekend (16/17th).  I’m not moving far: just up the road to Yateley, where I’ve found a room in a two-bedroomed house currently occupied by the landlord’s son.  More info (and photos) when the time comes.

Tomorrow will be a very busy day at work.  Each of us “students” have been assigned a row in our second lab; the task is to physically rewire all of the data and console cables to the machines, make sure the hosts all work, that they are properly labelled and available for booking in the labtool.  Part of the work involves moving them from small /24 subnets to the supernet, so a good deal of IP and routing work will have to be done on all of the booked machines (the rest can just be reinstalled hands-free).

My plan is to arrive at work for around 7AM (an hour earlier than usual) to allow me to make as much progress as possible.  Ideally I aim to have most of the physical work finished before lunchtime, leaving me an hour or two to update the software  configuration, with the hope of leaving work shortly after lunch so I can drive home for a lie in on Saturday morning.

After wasting a good number of hours this weekend attempting to get my SunRay connected to the SunRay failover group at work I decided to give it another shot.

I decided to ignore vmware, ignore Solaris Nevada installed natively and focus on what I am familiar with: Ubuntu (actually, I’d have preferred FreeBSD for either ipfw, ipf or pf: anything beats Linux’ iptables).

The primary ingredient in getting a SunRay unit connected to a server is DHCP. I decided to bring my laptop into the foray as a test utility. After much messing with iptables I finally figured out how to set my machine as a NAT router, routing packets between eth0 (my physical network interface, connected back-to-back with my laptop (which had the wireless card disconnected!)).

Once I could successfully ping the SunRay server from my laptop, I switched my focus to the SunRay unit, and the specific DHCP config. This turned out to be easier than I had expected. Last Thursday Chris Gerhard suggested I follow links on his blog to find a list of all of the required DHCP options to get things working… this I did, and at this point I ended up at Think Thin’s Sun blog, where I was fortunate to find the two main options: XDispMgr and TFTPsrvN.

These two options are used to communicate the IP address of the SunRay display server and firmware TFTP server to the SunRay unit. Without these two options set, you’ll receive the plenty frustrating “22 B” error code.

These two options are all very well and good, but when the blog entry covers both the Microsoft and Sun DHCP servers specifically, it’s easy to feel left out when you’re using the ISC DHCP server on Linux. More hunting and I found the following dhcpd.conf options:

option space SunRay;
option SunRay.AuthSrvr code 21 = ip-address;
option SunRay.AuthSrvr YOUR-SUNRAY-IP-HERE;
option SunRay.FWSrvr code 31 = ip-address;
option SunRay.FWSrvr YOUR-SUNRAY-IP-HERE;

I configured the two values with the IP address of our primary SunRay server: ebusy. Restart the DHCP server and the SunRay unit practically sprung into life… a few of the ordinary establishing connection OSD pictures followed by the never-before-so-beautiful “Welcome to ebusy” login prompt!

By now, I truly knew that all was good with the world… insert Sun badge and boom! The desktop I had been using for my day-to-day tasks at work a mere hour earlier!

And now, for some truly beautiful shots of the magic at play:

That’s the SunRay unit with my badge in the slot… of course I could use any other SunRay card, but that wouldn’t allow me to attach to the session I have associated with the card.

That’s my desk, here the left monitor is my regular desktop Ubuntu installation, but the right monitor is being driven by the SunRay and shows my work session. My laptop is uninvolved at this point, but is kept on so I can charge my mobile :)

The whole SunRay@Home experience is very good… of course it’s not as snappy as work, but it is absolutely usable; the main slowdowns are when switching virtual desktops, minimising/maximising lots of large windows and the initial card insert.

I’m not going to attach my DHCP server config file as it is quite specific to my setup, as is the Linux iptables stuff. If you do want to know more, feel free to drop me a line, or leave a comment and I will probably let you know the specifics.

At work today I asked Paul about taking a SunRay home to play around with.  My plan was to set up the free SunRay server software inside a Solaris Nevada VMware instance.

As I asked Paul, he asked if I was wanting one to try and get SunRay @ Home working.  This hadn’t actually occurred to me until he mentioned it, but…

by connecting the SunRay directly to my desktop via a crossover cable I could let my desktop PC route traffic between the wired interface and SWAN.  I don’t see that this would be a problem with the SWAN T&Cs as the SunRay unit can’t get to the Internet, can’t store data, etc.

For the time being I’m working on my initial plan: the Nevada install is about to start, and then I’ll look at the SunRay server software.  Unfortunately the latest Nevada DVD I have kicking around is b48, so I’ll have to either grab build 53 from work tomorrow, or perform a Live Upgrade.

On the whole today was fairly ordinary. I had an early start at work (7:55) which gave me time to sort out a V20z for a BT engineer. Getting up at 7:10 was a bit easier today, thanks to an early night yesterday—I’ll probably do the same this evening, as I’m still feeling tired.

We had two short meetings today: the first with the guys in Singapore (where David and William are for the week), and then a lab team meeting in the afternoon to discuss the small projects we’ve been working on.

Each of the students have been “assigned” a production server to look after: I’ve got enonet, which is a NIS+ replica zone server. The global zone provides no services, but each of the (roughly) five zones is configured as a NIS+ backup server, mirroring the config for each of the primary production NIS+ geos.

A quick look suggests that enonet shouldn’t be a lot of work: it’s a more or less standard Solaris 10 (latest == good) install, with a single service in each domain. The only worry is that I’ll be asked to perform a live upgrade to a newer version of Solaris: a real pain when Zones are involved.

We also discussed our “work from home” day, and with any luck we’ll be starting this in a few weeks. The big problem for me is finding a project to work on. Right now I’m more or less down to do some JLT (our system booking tool) work related to post-booking, i.e. providing suitable notification to the labstaff when a booking expires, allowing us to remove non-default cards, storage, etc.

This post-booking work should be interesting to do: it’ll involve messing around with some Java, some Sybase SQL, Perl and the API for our ticketing system, ServiceDesk. But really this isn’t what I’d like to spend the bulk of my time doing… I need to come up with a decent project to work on, hopefully something that will save us all plenty of time, make our jobs easier, etc.

All suggestions welcome.