I’m embarking on something I’ve wanted to do for a long time. Coffee reviews for traveling nerds. I don’t travel extensively but I do go places and my trip logistics are often driven by my need for acceptable espresso drinks. Over the years I’ve raised the bar several times for what I want in a coffee shop. I used to just get excited that I could find Starbucks on road trips through Nebraska and Oklahoma or had a Starbucks across the street from my hotel.
I just want to take a moment to point you at an epic new Chef cookbook undertaking. John Larsen has been working on a cookbook to configure a full ATG stack with JBoss and has finished a first version. This is something I always wanted to work on but never got around to, and so I’m being extra appreciative of someone who did take the time to get it working.
This post has been percolating in my brain for a long time. I mean like, long. Ever since I read a snippet from last year’s Chef Summit discussing best practices regarding attributes, I’ve wanted to write something. But what I’ve got to say keeps changing.
This post assumes some basic familiarity with Chef and node attributes. I assume that most people are going to be bored with the subject matter unless you care about nerdy details or why strings are the official preferred notation. You can get the short version on the Chef Wiki Cookbook Attributes page. I have a longer discussion here that focuses on my discovery process and evolution of thought.
A Quick Attribute Notation Overview
When I first started working with Chef, the publicly available cookbooks and wiki documentation were written in different styles.
- Some used symbols: node[:some][:symbol]
- some used strings: node[‘some’][‘string’]
- some used method_missing: node.dot.notation
I usually refer to this last as “dot notation,” mostly because I had no idea what else to call it at first. I still call it that as it’s an intuitive reference for someone who doesn’t know much Ruby. I read up on method_missing in prep for this blog post, but it was a new term and concept to me.
O dot notation, how do I love thee?
Getting back to it, I lurved my dot notation. It was easy to type. It was easy to read. It was harder to make typos. I am the queen of typos. Anything that helps me make fewer is my favorite friend. Then Chef Summit happened and recaps happened and I was like:
At first I was outraged that they were oppressing my dot notation and I was terrified they would deprecate it, which would make me sad. For a long time after that, I was just mad and still loved my dot notation and resolved to just keep using it anyway. This was the first time I had the urge to blog about it. I was planning to tell everyone to piss off. I loved my dot notation and I didn’t want it taken away.
I mean, yeah, dot notation didn’t always work and I probably never understood why. I was like this when I first learned chef:
Today I was having a conversation about consulting and how important it is to not be emotionally attached to your solutions. As a consultant, you exist to be an enabler. You help others succeed with tasks that, without your expertise, they might fail instead. As consultants, our value comes from a long history of observing and participating in spectacular failures and fantastic successes. We know what works and what doesn’t work and we can articulate the reasons why. But our value can be compromised if we can’t interact effectively with our clients. This means checking your emotions and inner schadenfreude at the door and working with clients to discover a solution that is practical for their environment.
Back in the day, I worked on a silo’d web operations team at a big retail company. We knew how we liked things done and, although we might argue amongst ourselves, we presented a united bullying front to the dev teams. For example, we didn’t allow them to deploy apps into production when they came with the caveat of “you have to restart the app every night.” We drank developer tears with breakfast, and we liked it that way.
Guess what? Consultants don’t have that luxury. I might also argue that internal operations teams no longer have that luxury, but that’s a blog for another day.
This is partially a post for posterity because I can never remember the format of this when I want it and I have to search all over the Internets for it. And for some reason, it’s really hard to find with keywords. Possibly my google-fu sucks, but I actually think it’s a java developer conspiracy.
The first time I found this jvm arg, it was by accident. I was reading a page of comprehensive JVM args. No it wasn’t for fun, although I was enjoying it. I was on a gig to help a client get rid of 20 second garbage collections that were crashing their site. So I was reading through the page just to see what might be useful and I found this:
It was love at first sight. See, I know developers lurve their stack traces and exceptions. Truly. But I’ve met some truly horrendous apps. You know the kind. They throw 500 line stack traces every time someone types in a bad password. Or you have an overrun team that, once an app is in production, they don’t have time to tune the app and so you are stuck with 2000 lines of stack trace for every exception, even though it only takes < 10 to figure out where to look for trouble.
After I found this and used it, I had bookmarked the page of wonderful jvm options, but it was dead the next time I went there. So I ended up searching the internet for it later and found that my googling led me all over. I found a stackoverflow discussion where a harried admin was asking about it and a herd of VERY ANGRY developers jumped all over him saying things like
“How dare you truncate our wonderful stack traces!”, and
“An exception is called that because it’s exceptional. There’s a reason for it and you shouldn’t truncate it!”
Oh they were like an angry mob.
But I have to tell you, this arg is an admin’s best friend if you have apps in your care that can’t be bothered with proper tuning on the dev side. Often this was because a client had a contract dev team come in, deliver an app and then leave. There was no one to fix it.
There’s no shame in cleaning up your log files so they are readable. It’s tough to actually dig into a problem if you have thousands of lines of “benign” errors filling your world. This option allows you to limit the damage done by excessive logging while still seeing that an exception is being generated. Of course, with all the wonderful logging packages and tools with filters out there today, this sort of thing is less important if you are filtering, but there are lots of admins out there with nothing but the CLI, syslog and vi even today. This post is dedicated to them.
A friend challenged me tonight to write a brilliant blog post on tuning a technology about which I know nothing. Actually I don’t think you can really do that. I don’t think you can write a brilliant blog post unless you’ve participated in some kind of failure/stress activity with a product.Until you’ve seen real life interact with your infrastructure, it’s all just theories and beer. But after a long day debugging Chef code I wrote in a way I wish I’d never written (no really, I don’t want my name anywhere on some of this stuff!), I thought I’d take a little downtime to read about a product I’m utterly clueless about. Because that’s fun. YES IT IS.
I have a couple of half-written blog posts started this weekend. Real wordy things about scrummifying your infrastructure and my experience with it. I’ll probably get to that in a few days but I needed to vent about my weekend of yak shaving first.
Moral of my story: The story you’re about to read is not the worst yak shave ever, and the problem is not a hard problem. It’s all in a day’s work for any halfway decent sysadmin. What we’re seeing here is a small problem exacerbated by a combination of technical debt and inadequate tooling.
Technical debt is a choice and can happen to anyone. Here the client allowed their configuration management to run away from them. They haven’t been maintaining their Puppet nodes and so don’t have a good list of what servers they are managing. They also let some config files slip through unmanaged. I generally don’t point fingers about it as there’s usually a sane tradeoff involved, but the first issue makes fixing the second one harder.
However, inadequate tooling does frustrate me. They’ve gone to the trouble of automating with Puppet and managing application configs with subversion and scripting, but do not seem to have considered holistic server management. The only way to perform administrative tasks is by hand on each server or with Puppet. This seems a rather gaping hole in long term planning. 100 servers is long past what I consider manageable by hand. But you read the story and decide for yourself.
I have a few clients that I keep in touch with and take on small projects from time to time. This has been a strange week. I’m technically full time on a project at the moment and I haven’t gotten any calls for new work in a couple of months. It’s been quiet. Then all of the sudden Wednesday I had 3 different people contact me about work.
One call was actually for some awesome Chef work at a place I’ve been before that I’d love to go back to. It really pained me to tell them that I thought they had in-house resources of which they weren’t aware and that these resources would be more than sufficient to Chef their project without my help. Sadface Sascha.
Another call was from someone I do work for from time to time. Their projects source my “Puppet for Chef” series, if 2 blog posts can be called a series. In my head I have a 3rd one on templates but haven’t gotten around to it yet. This 3wk run may source some if I’m lucky. I agreed to do some ad hoc sysadmin work for these guys because they’re in a swivet, getting ready for a PCI audit in 3 weeks.
The job: There are about 150 Centos 5 servers. Caveat: some of the servers on the list may be retired already. Examine all of the configuration files for ConfigServer Firewall, bring them under Puppet control. Individual node config sets are ok (ugh). All servers are running a puppet client. I don’t know if it’s the same client. The master was at 2.7.9 last I checked, I think.I have a mini Chef project too for the weekend, but I think I’ll save that for another blog post. I actually wrote post of this post on Saturday morning and am just finishing it up. I haven’t actually started my Puppet stuff except to wonder if they have some command line tools and think I’d better get to writing some tcp testers and comparison scripts if there isn’t such a thing.
I actually spent a chunk of time working on my Chef thing which was an exercise in frustration. It’s for a Windows Ohai plugin and the actual ruby work was a piece of cake. Testing has made me crazy though. Windows servers don’t fire up as fast as Linux for one and they are a huge pain in the ass to interact with. On my todo list is an SSH server for Windows cookbook, unless my ultimate wish of hoping that all Windows servers DIAF happens prior to that. The ssh cookbook is half done, like so many things.
I also am not sure I understand the testing strategy. I didn’t write any new code so I shouldn’t need to author any tests (relief). But it turns out I can’t seen any windows testing happening in the run, or at least, I think that’s what’s not happening. It ran happily on my Mac and then I thought maybe I should run the test on the Windows box too to see if the output was different. It was different. But I think that’s because there’s no Windows plugin tests in there? Wondering if I missed them? I was planning to email the chef-dev list tomorrow to see if I can get some love. Unfortunately the time I have to work on pet projects is the weekends and the IRC channels were like graveyards this weekend.But I digress. I also ran manual tests - you know, the kind where you do stuff by hand and watch the output? My Ohai changes tested perfectly sanely on my Windows 7 64bit workstation. They tested insanely on a 2008r2 VMware VM I had available to me. And when I went into ec2 to fire up a neutral 2008r2, I discovered Ohai hangs on gathering ec2 metadata when run from the command line. I have no idea why. I hand-edited the ec2 plugin and the mixin/ec2_metadata to try and eke out some debug info but I got no joy. While the problem fascinated me, that was as far down that yak shave as I wanted to go today. So I put away the Chef and cleaned up the yard for 2 hours.
Now I guess it’s time to play with Puppet!
I enjoy independent consulting. For that matter, I enjoyed consulting for a consulting company. I’ve been doing it in one form or another since 2006. I love the variety and the challenge that comes from never knowing wtf will be thrown at me on any given day or project. Some days it’s terrifying and some days I want to slap people around, but I won’t deny that I like it.Currently my LinkedIn profile, a business card and a terse web site are all I use for self-promotion. I never lack for work. I get calls every week from larger businesses looking to recruit for FTEs. This is rarely something I’m interested in. I dislike the inevitable silo and I’m not able to engage in the cognitive dissonance required to believe in all the meaningless BS that is part of corporate baggage. So I politely tell all these recruiters that, while I appreciate the contact, I’m not looking for full time work at this time.Once in a while I get contacted by recruiters looking to hire for smaller startups. I always listen to the pitch for these because, if the right small company comes along, I’d probably go work for them. Sometimes they’re way off the mark with my skillset or with people who want a culture match - like the recruiter looking to hire someone to work at a social network sports startup. When I told her that, unless it was bicycle racing, I wasn’t interested, I could hear the awkward just floating through the phone. Obviously these guys trying to use ESPN APIs and social networking to bring people together would have nothing to say to me.I had a call in December from a recruiter from a brand new startup local to the area, using a tech stack of which I approved. They tried to set me up but it never worked out. The place was all chaos and key people kept going on vacation and one thing after another. Eventually I forgot about it. Until I got a call the other day from the same recruiting company. They left me a voice mail, telling me they had a question and would I call. So I did. I called them today.I got on the phone with the recruiter, a new one, and he literally stumbled all over himself for over a minute, trying to spit out a complete sentence. He finally did and I nearly hung up on him. The conversation went something like this:Him: “ah, um…so…you know that company we were trying to…remember you worked with my boss last year and that company….they um, were looking for…remember we tried to set you up with X company?”
Him: Would you happen to….they’re looking for…it’s really hard to find people who are into startup culture….um….ah…they’re looking for….um…developers….and so um…I was wondering….if, um
Me: I don’t know any developers looking for work
Him: Oh, um…right, well, if you happen to think of anyone…could I um…send you a followup email…just in um case?
Me: sure <click>Then I sent an email to his manager, the very nice, polite and professional person I’d been talking to in December:“Hi Harry, I would appreciate it if you could let your colleagues know not to call me for the sole purpose of asking if I know anyone looking for work. It would probably also help their delivery if they not preface their requests with “remember the job last year we were trying to get for you but couldn’t?” and follow up with “do you know anyone else who might like to work there?”Thanks,
[[posterous-content:tDmkxmhdJkvcIbHmytmJ]]In the beginning, there was no Apple…
I have an uneven history of geekiness. Most of my geek comes from my dad. When I was little, sometimes my parents would let me sit up late on the weekends and play Dungeons and Dragons with them and their friends. My dad was a bona fide war gamer. He even played them by mail, with maps hanging up on the wall, using sticky putty to hang counters on them. We didn’t have a lot of money, but I remember pong and the first computer I played on was a TRS80. I played Zork, a little. My dad took me to wargame conventions sometimes. There were no apple products in the house.
Still no Apple…
Left to my own devices, I was mostly just a bookworm and music junkie. I knew computers because they were always around at home, but I never programmed for fun. I made money in college tutoring computer subjects and took a few “computer classes,” because that was the only way to get an email address at my tiny college. But I didn’t take them seriously. I was a Medieval Lit major and only interested in computers for email, browsing and word processing. I used a unix terminal long before I mastered DOS or even saw an apple up close.
As an undergrad I took a job doing tech support because it beat the hell out of working in the library, both fun-wise and money-wise. Seriously, ever mended a book binding or filed the catalog cards that come with new books? Helping the clueless figure things out won for me every time and tech support usually means Microsoft systems. So still no apple.
Isn’t Apple going to die a horrible corporate death?
What I’m trying to say is, I never even saw a Commodore or an Apple PC until I was an adult. I never had any reason for interest in Apple products. For years to me they were just a company about to go broke. All my experience was Microsoft-based. Then the internet boom happened and I left the Lit studies behind.
Oh look, an iP….
I never wanted a Mac. I eventually bought a 4th gen iPod and really liked it. I’d never really clicked with any mobile music devices. But I loved the iPod and used it extensively for running.
Eventually, I discovered the Shuffle, when it first came in the small square. I’ve gone through 3 of them as I firmly believe they are the best music player for running ever invented. Apple got this so right. It’s tiny, clips on to my sports bra strap and has an easy-to-identify pause button and fast forward button. It shuffles my playlist too. This is all I want in a running player. Anything else complicates tactile navigation. I skipped the funny rectangle shuffle - it made things complicated.It’s a little irritating that it costs so much for the memory it come with, but it really approaches perfection for me. So I pay for it.Still, who needed a Mac? Not I.
Despite all of this, I never wanted a Mac. Among other things, I had some experience with linux touchpad drivers and some older Mac touchpad drivers and they made me want to headdesk myself bloody. I had years of Windows experience. I knew how to fix it and I understood its quirks. I liked it. There was no Mac software for certain things I love (hello Quicken) and they aren’t really made for games. And their single mouse buttons gave me the creeps. Little did I know…
I always thought of Macs the way I do Volkswagen: Nice, but overpriced for what you’re getting. I’ve always liked a high screen resolution on my laptops and Macs could never give me that. My 11” Sony had a 1366x800 screen resolution which I loved. My 13” had 1600x1024 (or something like that). It was awesome and lightweight to boot. I could even snag a unix-like command line with Cygwin. It did everything I needed it to do.
Until I started programming….
2 years ago, when I started in on Chef and Ruby, needing Rubygems and all the other disparate parts that come with managing laptop development environments, it all fell apart. I’ve heard rumors that it’s possible to get ruby environments running with Cygwin. As a matter of fact, I know you can because a couple of the Thoughtworks consultants forced to use client-supplied Windows laptops made it work. But it was painful. And I wanted no part of it.
I ad-hoc’d my way through for a while by using a Linux VM. That worked ok, but it wasn’t really flexible enough for me. Also, I ran into funky problems like some gems/tools worked on some Linux distros but not others. But one thing almost everything in the ENTIRE WORLD works on is Mac OSX. BECAUSE ALL THE DEVELOPERS ARE USING IT! And so all the Ruby and Python tools in the world work on OSX as well as common Linux distros (read: Ubuntu)
I eventually discovered that I really wanted a Mac because I was so tired of devoting countless cycles to shoe-horning (yak shaving) things into a workable state on my windows laptop when I knew it would all just work on a Mac. So I waited for the new Macbook Air to surface last July and grabbed one. I’ve had it for 8 months and am pretty used to it now. It doesn’t have the screen real estate that my last 13” had, but I’ve come to really love the touchpad and gestures. I also love hot key empowerment and being able to type at a comfortable command line. I love that, when I don’t know wtf the OS is doing, I can drop to a command line and figure it out. I love that OSX comes with awesome built in tools like iPhoto and the VPN client.
I still have days when I’m all key smashy omgwtf-fu mac, but they are rare.
And then there was iPad….
I’ve never felt the need for a tablet. I’m not on the road that much these days. My laptop is fairly mobile. I have tiny hands. No seriously, I shop for winter gloves in the boys section because all the women’s size small gloves are TOO BIG. WTF are my tiny hands going to do with a tablet?
It turns out, strain to reach the letters in the middle of the keyboard.
My recent experiences with different phones made me think I could ready for a tablet. I’d canceled my iPhone contract and was going to sell it. But I found myself continuing to read on the iPhone and use it for things at home via wifi. I read a lot at night in bed in the dark. The new Samsung was too bright even with the brightness all the way down and the battery needed to be saved for when I actually needed to use the phone. I eventually got a light filter app whose sole duty is to dim the backlight even more.
5 months later, I’m still using the iPhone. I figured I might find some use for a tablet and maybe sell the iPhone. Plus, I love the iPhone keyboards and the way iOS seems to read my mind when it’s autocorrecting (most of the time). I’ve never connected well with the Android keyboards, even with auxiliary apps like SwiftKey.
I now have a new iPad. It’s really big. I can’t really reach the letters in the center of the keyboard with my thumbs although it’s nice to type on when flat. It’s also really slippery and hard to use in bed. My case will come tomorrow. This will hopefully keep it from sliding out of my hands and give me some texture to hang on to.I like that Tweetdeck will sync across all my devices. I like that I can now read PDFs without pain on a mobile device. I don’t like that everyone seems to think that Retina means everything should stay the same size and look nicer. I want smaller stuff. SHRINK ALL THE THINGS APPLE!The iPad is not yet earning its keep. I’m on the extended work from home plan so it’s doing light duty at the moment. I am reserving judgement; although it’s possible that, if you buy the rumored 7 inch iPad later this year, I might trade you.And the paraphernalia…
I picked up a stand for my Macbook and a bluetooth keyboard. And then the Airport Express. I researched wireless print servers and couldn’t find a decent one for much less than the Airport. And the Airport will stream my iTunes too. And the iTunes...
Yeah I forgot to mention iTunes. I’ve been using it for years, even though it sucked up massive resources on my Windows boxes. It’s the easiest, most intuitive music manager I’ve ever met. And it manages my i-devices. I asked Android owners before I bought my phone, “how do you manage your phone?” They just looked at me like I was crazy. I still can’t figure out if I’ve been conditioned by Apple, but I really like having the iTunes interface to manage my phone, etc.and then there was the iTunes Store…
I loathe the iTunes store. There’s no way to filter searches. There’s no way to make it show you what you want. You get only what it wants you to see. So I don’t know, am I a fangirl?
I’ve always thought Apple owners came across rather smug. But then I kind of fell down the slippery slope of Apple product ownership. I don’t feel smug.
I still won’t buy iTunes content except for a few things here and there. I won’t buy the Apple TV. I have a Roku that streams everything I need and for less money.I don’t care to be locked into one content provider, which is why I refused to buy a Kindle. Also, I love me my backlight. Ironically I like a smartphone/tablet for reading because it allows me to have apps for all content providers. In general I tend to use Amazon, but I refuse to be required to use them through the mishap of buying one of their devices. If Apple ever started restricting other content provider apps, I would have to take a serious look at my hardware collection.I still have a Windows desktop. My Quicken is there. My (tiny) gaming collection is there. Calibre is there. Other large footprint software is there. I haven’t rid myself of my 13” Sony yet although I only open it when I need to look for something. It’s my only mobile HDMI-out device at the moment. Apple devices don’t play Flash and that’s what Amazon is streaming in at the moment.I’m a big fan of their hardware. I love all my Apple stuff. But I’m still comfortable with Windows and have nothing against it as a user (although Windows Server Administration makes me want to cut off my fingers). I wouldn’t urge someone else to get an Apple device. I might or might not gush about it, depending on what it was. I kind of wish Apple had designed my treadmill. But then, I probably couldn’t have afforded it.To an outsider, I probably look like a fangirl. But if another company made something as awesome to use as an Apple product, I’m sure I’d buy it. My foray into that arena though was my Samsung Galaxy S2 which has just not been as lovable as I’d hoped.I’ve been all “I don’t need Apple stuff” my whole life. But I kinda like it, it’s reliable and it’s all so very comfortable to use. So while I may not gush, I might be a bit of a closet fangirl. It’s possible….