Month: July 2011

The Alcohol Assessment

I don’t drink alcohol. If this statement is news to you, you’ve obviously never spent much time with me. No, I’m not a friend of Bill nor do I have any moral reservations about consuming alcohol (religious or otherwise). I just don’t like the taste. And while that’s all true, that’s not the entirety of the story.

Most of my peers acquired a taste for alcohol in the state-sanctioned pocket of hell called high school. I didn’t do so at that time for two reasons: 1) my personality quirk that explicitly wants to do the opposite of what everyone else is doing and 2) I hadn’t come out yet – to myself or anyone else. I was uber-paranoid that should I imbibe I’d out myself and that wasn’t something I was willing to take a risk on. When I finally came out after college the risk of outing myself went away (particularly if you’re drinking in a gay bar!) but by then the fact that I didn’t drink became almost a badge of uniqueness. That coupled with the aforementioned personality quirk didn’t lend itself to picking up the habit.

Over the years I’ve sampled many different alcoholic drinks, from beers to wines to mixed drinks. (The only time I ever recall drinking an entire alcoholic beverage was some fruity drink with Scott Gaydos at Bahama Breeze in Austin around 10 years ago. I can’t for the life of me recall why I picked that particular moment to have one, perhaps Scott can enlighten us if he remembers.) Numerous times I’ve had people say “try this, you can’t taste the alcohol in it”. More often than not I can not only taste it, but can smell it before it even gets to my mouth. I’m uncertain if I’m just sensitive to it or what, but very few alcoholic beverages make it past my nose undetected. Those that do often taste good, but no different than a similarly-flavored virgin drink. That always brings up the point: if the alcohol doesn’t improve the flavor of the beverage, why use the alcohol at all? Most people when queried bring up the physiological effects of alcohol, which really doesn’t interest me all that much. In many ways I’m a control freak and the thought of relaxing some of that control over my own person holds no interest to me. What seldom comes up as a response, but yet I think is a bigger reason in many circles, is the social aspect.

Which brings me, finally, around to the point: drinking alcohol is very much a social activity and at times I find myself frustrated that I sit outside of it. While I have no problem whatsoever with people drinking around me, I’ve had many people abstain in my presence because they “didn’t want to be the only one drinking” (this philosophy makes absolutely no sense to me, but it seems prevalent).

Internally I struggle with the implicit peer pressure of drinking alcohol as a social skill vs the desire to hang on to my quirkiness. Given that I really don’t like the smell/taste of alcohol and my intractable desire to never be less than fully in control of my facilities still has me firmly in the “not drinking” camp. That doesn’t mean I don’t feel the desire to join in the millenia-old human social tradition of sharing an adult beverage.

An authentic vacation

Last night I got back from a week-long trip out to the DC area to visit my sister Renee, her husband Robert, and my niece RG. Aside: in the past I’ve referred to Renee as my “virtual sister” or “sister by choice” but really, family is family – she’s my sister.

I had a great time visiting the R^3 crew and it was a very much needed break. We went on hikes, hung out, spent several lazy afternoons by the pool (I have a sunburn/tan!), and in general just spent time hanging out together. This morning it dawned on me that aside from spending time with people I love, the best part about the vacation was that I felt free to be Just Me. I’ve known Renee since we were 3 (or so she claims, I thought it was closer to 5 but there’s no arguing with Renee!). She’s one of a very few people who knows almost everything about me and with whom I feel I can share anything. And Robert, despite only knowing me 10 years instead of 30, accepts me as family without question. As for RG, I’m “silly” Uncle Casey end of story!

Since moving to Seattle I’ve not spent enough time with people around which I can be Just Me. I’m not the type of person who puts on a full personality costume when I go out, but it still takes me a while to open up to new friends. That process is happening but it’s slow. The situation is compounded by the worry about “what other people will think” that’s been ingrained in me by my parents which I strive to overcome on a daily basis. And while it isn’t up to the Will & Ned-level of effort, it’s still tiring.

I’m resolved to try and model ‘s MO of daily living life more authentically and spending more time in the company of the people here in Seattle I feel most authentic around already (Jeff, Jonobie, and Kevin – I’m lookin’ at you).

Mystery of the terrible throughput (or how I solved a TCP problem)

It all started out with a simple single stream reading test. Just a simple request for the entirety of an 8GB file. We do this stuff all the time. Except this time instead of 700 MB/s I was getting 130 MB/s. What?

Usually we test with jumbo frames (9000 MTU) but for this exercise we were using standard frames (1500 MTU). Still, there’s no way that was the difference. After 2 days I discover a method to consistently reproduce the problem: while the streaming test is running, toggle the LRO flag on the server’s network interface. This is just as crazy as making your car go faster by removing your soda from the cupholder. There’s no way that it has anything to do with it, but for some reason it does. Consistently. At last I have a reproducible, if ludicrous, defect.

Fast forward through 5 days of eliminating nodes, clients, switches, and NFS overcommits. Add in packet traces, kernel debugging output, and assorted analysis. Eventually Case catches the first real clue: the packet congestion window between the ‘fast’ and ‘slow’ states are distinctly different. In the ‘fast’ state, the congestion window stays fairly constant. In the ‘slow’ state, the window oscillates wildly – starting at the MTU growing really large, and starting over.

The LRO trick worked by causing enough retransmits that the stack dropped into slow start mode — one mystery solved. The reason we haven’t seen this before is that after a node-client pair get into the fast state, the slow start threshold is retained in the TCP hostcache between connections which is why we haven’t clearly identified this before — another mystery solved.

Fast forward through a few more days of slogging through TCP code down the path of blaming slow start threshold (or rather the lack of slow start in the slow state). By this time I’m way more familiar with the TCP code, and our kernel debugging framework, than I want to be. I notice that every time the congestion window drops back to the MTU it’s caused by an ENOBUFS error. It’s very unlikely we’re running out of buffer space though. Checking the called function reveals that the error would show up not only when we’re out of buffers, but also if we can’t return one immediately. We surmise the problem is some contention causing an inability to immediately get the requested buffer. So I change the code to reduce the congestion window by a single segment size (aka MTU) instead of dropping it all the way down to the segment size. The assumption being the next time we request a buffer of this size, we’re likely to get one.

And performance shoots up to 900 MB/s — even higher than the previous fast state.

The reason we’re unable to return the requested buffer immediately is unclear, and frankly above my paygrade. I’ll happily let the kernel devs work on that (it involves slabs and uma and things geekier than me).

The core of the problem remains “why aren’t we able to return the requested buffer immediately” but until the devs conquer that one we have a valid, shippable, workaround. And a lowly tester found, identified, and fixed it!