Bratty Redhead

the sarcasm is free!

Shenanigans

I’ve been trying to think of a thoughtful and engaging way to tell everyone the news. I feel like I should reflect on the last two years of independent consulting life and offer up some deep thoughts and wise advice. But I got nothing.

Starting August 5th, I’ll be an Opscode employee.

The truth is, I’m super excited to go to work with my friends and eventually move out of the snow-addled state of MN. A big reason I am independent is because I was tired of being an Enterprise employee. I have always been willing to consider work with small companies but I also want a basis of friendship with my co-workers.

I’m confident enough about who I am and what I do now that I’m not interested in what I have been calling “arranged marriage” jobs - the kind where you have a few interviews, get really excited and then discover you’ve walked into a clusterfk egofest. I don’t want to have to evaluate companies and people on the basis of a few interviews. I also don’t want to have to engage in the inevitable dominance posturing and pissing contests that seem to be inevitable with many tech teams.

On the other hand, working with smart people is one of the best things in the world. Being independent means you’re often working alone and don’t get to hang with other smart people very much. It’s why I’m so enthusiastic about conferences, because that’s where I get to hang with my smart friends.

You can read in the very first post on this blog about how transformational it was for me to begin working with open source software, and Chef played a pivotal role in that. I’ve worked with the tool now for over 3 years and I love it.

While I do love the product, what draws me to Opscode is their attitude about culture, both internally and the community. HugOps is a real thing and I love that it’s a lifestyle choice. Between this and the fact that I like without reservation every Opscoder I’ve met makes this the beginning of an exciting adventure for me.

I expect shenanigans to ensue almost immediately.

Cleaning Up the Clubhouse

I’ve been a bit reluctant to enter this discussion on my blog; partly because there are some amazing, thoughtful posts already that eclipse anything I could say and also because any of these discussions seem to spark a lot of outrage and anger regardless of the actual topic content. That kind of drama is a big turnoff for me. But I did want to say a few things about a recent experience at devopsdays because it seems to be an angle we aren’t discussing.

At devopsdays Silicon Valley this year, one of the breakout sessions was “Encouraging Women in Dev/Ops,” led by Doug Ireton, one of the awesome automation engineers at Nordstrom’s. I was already dragging from Velocity and hadn’t planned to go to any sessions, much less yet another “how do we get more women in tech” discussion.

If I sound impatient, it’s because I am a bit. In general, I don’t have any really strong opinions on the “women in tech” discussions. This is because I feel we have the same discussions over and over again without covering any new ground or making any difference in how people think. This may actually not be a fair feeling as the discussion has certainly changed the way I think. Or, more accurately, it’s made me think about it. I don’t actually think we should be worrying about how to get MORE women in tech as much as how to make it a comfortable place for people who are already here. Take care of that and balance will come eventually.

Some History

I entered tech during the 2000s internet boom where any warm body would do and you didn’t have to be a decent human being to keep your job. I know this because I often was not a decent human being myself. I was grumpy and petulant, at least in my head, and folks from those early days would probably agree (Kiosk level 2 support team who I pissed off in a major way and worked hard to repair that relationship, are you out there anywhere?). I did eventually grow up and become a moderately self-aware decent human being and have even managed to develop some empathy for people outside my direct life experience.

Not only have I worked in Tech most of my adult life, my dad was a wargamer and RPGer. I was exposed to many strains of nerds and geeks growing up and some of the biggest treats of my 8 year old self involved getting to stay up late with the grownups while they played D&D and maybe even being given a character sheet.

I lived in a house with 4 other guys in college where I learned to watch Star Wars and Repo Man on repeat on one TV while playing video games on the second TV until 2am.

What I’m trying to communicate with this history is that, while there are certainly some women in all of these lifestyle choices (wargames, rpg, tech support, web ops), we make up a small percentage. So I spent my life getting used to being one of very few women in any tech or social group. Sometimes it’s fine, sometimes it’s not fine. It largely depends on the company I’m keeping.

Regardless, I was taught to communicate by social groups who probably shouldn’t be teaching anything. But I learned early that guys tease and insult when they like you and that you shouldn’t worry until they are polite, kind or ignore you. That also became my style of communicating for a long time because that’s what I knew. To this day I am comfortable with it. Dick/Boob jokes don’t bother me. Profanity doesn’t bother me. It was co-workers on the clock that introduced me to such great things in life as Triumph, the insult comic dog, South Park and all kinds of inappropriately funny youtube videos. Calling me a girl doesn’t bother me. Calling me a princess doesn’t bother me because there are plenty of boy princesses out there and I’ve called them on it.

Things that DO bother me: overt misogyny, hatefulness and guys who constantly bitch at work about how their wife makes their life miserable. People who won’t admit there is a problem with “isms” in the world because they themselves don’t perpetuate it or see it. There is a big difference between a dick joke in private with friends/colleagues and making one at a party where you don’t know people. Context is everything.

The Effects of HTFU or GTFO

To be clear, I’m not saying I’ve sailed through life happily making dick jokes. I have experienced a lot of internal turmoil and emotional anguish over all-guy teams that I have been on, especially early in my career where it was still really hard to gain acceptance and professional respect. But 10 years ago it was HTFU or GTFO. And so that’s what I did. And when you make that a condition for acceptance, you greatly shrink the available population and personality types who stick around, both men and women.

The modern discussion of sexism (specifically because it relates to me) has taken me from an attitude of “most of it doesn’t bother me and I don’t know why people care” to “A lot of perceived exclusionary sexism doesn’t bother me, but other people have different expectations from their professional lives now; I entirely respect that and will work to make my professional sphere a welcoming and safe place.” In order to attract the next generation of women into tech, everyone is going to have to come at least this far in their thinking. And it’s not just sexism; it’s ableism, genderism, homophobia, and on and on.

But to get back to why I’m bored with the women-in-tech discussion people keep wanting to have, I don’t see it making a diff and I’m still getting a big vibe from guys of “I don’t do it and I don’t see it and how is it possibly a problem?” I ended up going to the discussion because I really like Doug and wanted to be supportive of his efforts. In general, it was pretty much what I always hear and it usually boils down to “we’re really nice guys and we can’t find any women to hire.” I get that there are not a lot of us out there. BUT…

Self Awareness is the Key

My answer to everyone who feels like this, and I gave it to the Nordstrom’s guys when we talked about it later, is this: work on your own self-awareness. I’m sure you think you’re a reasonable person who never had a sexist thought in their life. I’m pretty sure this is not true, because even I have made pissy comments about women drivers, although only in the privacy of my own car by myself. But I still think it. And I’m a woman!

So work on your self awareness. If you really care, go around to every woman in your professional and personal life and ask them to please offer you feedback if they feel you’ve ever done something sexist or demeaning either to them or in their presence, no matter how trivial it seems. Because I have news for you guys, we don’t offer that stuff uninvited. We expect to be scoffed at or ignored because so many people think sexism is drama. Example: I have a friend I’ve worked with at two places now and it took me 3 minutes to spit out “can I offer you some feedback?” before calling him on the fact that he’d talked over and for me in a meeting; something I find particularly galling.

None of this is going to get MORE women in tech. But it’s going to comfort the ones who are already here and word of mouth is a particularly powerful advertiser, as every viral video ever should tell you. And honestly, I think it’s going to do more for everyone if we clean up the house first before inviting more women over and asking them to help us clean it up for them.

Screencasts: Context Is Everything

Yesterday I tweeted in frustration about being confronted with yet another screencast/video when searching for help on the internet. This has been a growing frustration for me as I branch out away from pure command line/coding tools.

I find help videos in general to be obnoxious. I read and comprehend fast and generally want to skim some text to understand the thing that I’m looking for. I find “new user feature” videos that we end up from Google and stuff to be especially annoying.

I get that not everyone wants to just figure stuff out, but I generally can figure things out way faster than you can tell me about it.

A few months ago I was writing my first public talk and also using Keynote for the first time. I kept having to search the Internet for answers to questions because (2 years later) I’m still getting used to the way OSX presents things and am never sure where to look for formatting-type stuff. Something that really set me off was trying to figure out how to make new masters for Keynote. When I looked around, all I could find were videos lasting several minutes on the topic. Seriously. All I wanted was a paragraph on the general method and maybe a line with the actual tool bar clicks to begin with. Eventually, on the second presentation I worked on, I found what I was looking for. But I was really annoyed by the whole exercise.

On the other hand. I subscribe to RubyTapas. I find that an appropriate use of a screencast. Similar to a podcast, it’s a discussion of a complex topic that doesn’t have an answer at the end. When I want to figure out how to do something, I want to read about it on stack overflow, not watch a movie. BUT, if I’m interested in the how and why of a learning topic, then I love a good screencast. I especially like the short and sweet nature of the RubyTapas because they are not a major time commitment to listen to one.

But if I have a question, and that question has a fairly simple answer like “click here, here and here and type these things” I’m not sitting through a slow moving video to find out the answer. Yesterday I was looking for the start command for a tool I’d just downloaded and, when I clicked on the “getting started with Tool” link, it took me to a screencast which is was sent me over the edge and caused that tweet.

It was interesting to me that this is probably my top ever retweeted tweet. I think there’s an assumption out there that people want video instead of words. I never watch video on news sites either. I hate it. I want to skim stuff. Maybe places making these assumptions should recalibrate. Or maybe those of us who can read and comprehend faster than someone else can talk are actually in the minority? We’ll probably never know.

Ruby Toolchaining Part 3: Walking the Package Talk

This is the third post in a series about managing your Ruby Toolchain. Expect at least one more after this. You can also read:
Pieces and Parts: Managing Your Ruby ToolChain(part 1)
Managing Your Ruby Toolchain Part 2: The Package Repo Rant

Why I rant about package repositories: Static software without a repository is the pebble that becomes an avalanche. So many undesirable behaviors stem from no one owning the package collection. Read the rant if you want details. This is a practical discussion, although I’m not telling you what to do. I’m just telling you what I’ve done and what I’ve seen over the course of my career. There are many ways to solve this problem. Do what works for you.

This post grew so much that there is one more after it that specifically deals with some toolchaining tools and ideas as well as a few anecdotes on how I’ve dealt with some of these challenges in the past.

Today we’re going to talk about:
Tarball Orphans vs Artifacts vs Inbetweeners
The Internet
An Approach to Repositories
Packaging Tarballs
Ruby Management
Ruby Toolchain Management

Tarball Orphans

What do I mean when I rant about tarballs? Basically I mean that pile of source you downloaded from the internet, or Tomcat package, JBoss tarball, Ruby source, whatever.

What is exempt from my ranting wrath? Custom code that you might tar up instead of jar/ear/war. Anything that changes often shouldn’t be in our static package repository. Those are places to keep long term, install-once per server software.

When I talk about tarballs, I mean tumbleweed off the internet. When I refer to compiled code I’ll use the word artifact. What’s an artifact? It might be a jar file you bundle with your deployed code. It might be a war file. It might be a tar of ruby code Or database artifacts. Anything that’s your stuff, we don’t put that into package repositories. We put it into artifact repositories like Artifactory and Nexus. People also use S3 storage to store and retrieve artifacts.

Some things may not be easily categorized. Newer products get frequent updates that may require frequent patches or deployments. For example, I worked with a team that implemented Basho’s Riak which they updated frequently and ended up engaged in a disagreement with Ops on how often they could update it. The ops team wanted to restrict it to quarterly and the devs wanted much more frequently. This sort of management requires a custom solution that may be package-based or not.

The Internet

If your infrastructure has free access to the internet, that’s awesome. Many infrastructures have limited or no access to the internet. Often the reasons for this seem silly. But sometimes you can’t fight city hall, or you can but you need to save your fights for the life or death kind. Your servers don’t need the internet to survive. Or they should not.

Eventually you will find yourself needing stuff from the Internet. In order to work within restrictions, you can set up a Bastion host or AWS VPC infrastructure that can call out to the internet in order to create private mirrors of any repositories you need. You can follow the slightly more difficult path of setting up an internal server that contains your repos that you need to sync in a way that doesn’t have direct access to the internet.

Internet as Troublemaker
Internet access brings its own set of problems as well. If you aren’t specific about package versions, with every server provision you could have slightly different versions of things installed. This is a problem because it can cause erratic behavior across your infrastructure. If you have 1000 nodes built over the course of six months, each build wave can introduce incremental differences. When you control the packages in your internal repos, you exert a stabilizing effect, which is a good way to control your system without making it brittle.

Even if sloppy package management doesn’t actually cause issues, it will be a suspect in any intermittent, mysterious problems. Picture an outage conference bridge including execs. Now envision yourself explaining to them how this mismatch isn’t a problem. Can you envision any explanation that doesn’t make you sound like a dolt?

Example: A few years ago JBoss 5 shipped with a bug that was a major performance blocker but difficult to track down. It was later fixed in a minor patch release. Imagine having a JBoss package block that just says package "jboss" {action :install} and using an internet repo to install it. 6 months later you provision 100 servers from your trusty internet repo and over your peak season, the bug triggers but everything is weird because it’s only a problem on some servers. Imagine tracking this down. (Yes this is a flawed example b/c as far as I know, there’s no internet repo with a JBoss rpm)

If you’re a dev and you think this isn’t a problem, try chatting with a friend in the sysadmin/ops area before you poo-poo. If you’re a sysadmin and you think this isn’t a big deal, may we never be responsible for the same infrastructure.

Be Conservative. As a rule it’s ok to use external signed key repositories for your base OS packages as those shouldn’t change. Tread carefully even with patch/update servers to ensure homogenous installs and any packages acquired from other internet repos (like EPEL) should be installed with explicit versions.

Adopt Orphans Don’t let Internet-sourced software roam around your infrastructure unsupervised. If you need a package from the internet, it should be easy enough to download and insert into your custom repository. There are several tools for managing repositories. For Enterprise Linux, Cobbler, the provisioning and infrastructure management tool has repository management and mirroring. My new favorite thing for this though is a tool called Pulp, an open source project by Red Hat. If you prefer simplicity, you can just install a web proxy and use a command line tool like createrepo to manage your yum repos.

Don’t assume I can talk to the Internet. When it comes to configuration management tools, I find it in bad taste to write a module or cookbook intended for the community with an Internet dependency. If you have to write one, label the dependency in large font.

An Approach to Repos

OS Packages

If you’re going to set up repositories, consider a collection like this list.

Base - base EL install repo - This is generally straight up ISO content
Patch - repository containing patches
EPEL - EL Extras repo mirror
Custom - repository containing all the custom packages I need that I’ve either crafted myself or downloaded

Purity of Purpose: Always keep your base and patch repos pure. Don’t let people just dump random packages into a Base OS collection. You’ll never know what was supposed to be there and what was added later.

Prioritization and Specificity: If you have two versions of a package in different repos, either be specific when installing or configure a priority for each repo.

Rubygems

This is where you get/put your gems. Gem is the package manager for Rubygems. You either need to connect to the internet to get all of these things or you need internal package repos/mirrors. See this blog post by David Moulton for how easy it is to set up a rubygems repo.

Artifacts

This is where your deployment artifacts, jar, war, ear files should be contained; a tool where it’s easy to retreive them programmatically. I have seen people use Nexus, Artifactory and even S3 to store artifacts. If you’re not a java shop, you may not need something this complex. My major experience is with Java shops.

YMMV

There are many ways to craft your infrastructure. Having package repositories is an advantage because they make it really obvious to everyone in your organization where to look for software and where they should put software they need.

The important thing to keep in mind here is that, especially if you’re in the Enterprise, you aren’t designing for yourself or just one team. This will balloon out to include dev teams using your tools. In order to deploy their app, the devs will test using VMs provisioned by your infrastructure and they should be taking advantage of your beautiful repos to deposit their extra software packages. Your team may even be administering the VM infrastructure.

A major goal to strive for is for all VMs and servers to be provisioned the same way in all environments, desktop to production. You make this easiest and more likely to happen when you make it easy to interact with software repositories. If it’s not easy, other teams will “figure something out” which will likely outrage you when you discover it.

Practical Tarball Management

Avoid Compiling From Source

Believe me when I say that you do not want to compile Ruby from scratch every time you build a server. The more automation achievements you unlock, the easier it becomes to use ephemeral VMs. In Dev, people should be popping up and killing VMs all the time. If it takes 10 minutes to build a server, they won’t be doing this, dev VMs will have major cruft buildup and dismay will ensue when people realize they can’t actually reproduce their dev VM.

Turning a Tarball into a Package

How do you get packages for your repos when all you have is a pile of tarballs? If you’re brilliant, write a package spec for it. If you’re like the rest of us, you use FPM. Even if you are in a situation where you can’t have a package repo, you can at least create and store packages which will speed up installation and simplify implementation.

FPM: Learn it, love it, live it.

Ruby Management

Ruby Source

1.8.7 is EOL
First, on the topic of existing packages, as announced in October of 2011, Ruby 1.8.7 is officially end of life and won’t even get security updates after June 2013. Just say no to the supplied 1.8.7 packages on your Enterprise Linux(EL) and the one that comes installed with OSX. No really, never touch it. If you’re using it, start seriously considering how to move on.

System Ruby
Whatever else you’re doing, your servers will all want a system Ruby. It’s possible to get along without this(see omnibus in subsequent sections), but especially in dev, people will be shocked and dismayed when there’s no easily accessible Ruby. So get you some Ruby, not from the ancient EL repos, but get the source tarball from Ruby-Lang.org.

Why do I tell you this obvious thing? We had the brilliant idea of switching out our compile-every-time system ruby install with just using the embedded Ruby that ships with Chef client. Never do this. The trouble is that your Ruby install is not just an executable, but a name space for installing gems and related software. This is is why Chef ships in an omnibus in the first place: to avoid dependency hell and version conflicts. The other problem with this idea is that you are still lacking a system Ruby. You either have symlink to the Ruby executable or people have to know where to look for. So install a system Ruby independent of any other software.

Next make a Ruby RPM. RPMs are hard. Luckily Ian Meyer is watching out for us and has a defined, working spec from which to build an RPM. I’ve made this Ruby RPM for two different clients and it compiled without a hitch.

Once you create an RPM, put it into your custom package repo so that you can yum install ruby-1.9.3 or install it via cookbook later.

I repeat this series of steps for the Rubygems tarball, except using FPM to package it. Then they go in my custom repo where the custom repo has a higher priority than my base repo and my recipe just has a package block to install Ruby and Rubygems. It takes about 30 seconds instead of the 5 minutes it takes to compile Ruby from scratch.

If you use Ubuntu, well, they are cooler than EL and have Ruby-1.9.3 packages.

Where Do We Go From Here?

An important part of toolchaining is maintaining control of the supply chain. The way we do that is by creating space for packages and actively managing that space. It should be freely accessible to both deposit and receive software. Don’t manage things by locking people out, but by making it easy to use and providing guidelines for usage. Most people want to do the right thing. When you make the right thing easy to do, more people will do it.

Managing Your Ruby Toolchain Part 2: The Package Repo Rant

TL;DR Use package repos for all your static software

Last weekend I wrote a blog post that started exploring standards for managing your Ruby toolchains. To review, I had a query from someone on how to cope with the dynamic nature of Ruby-based tooling and managing a Chef infrastructure. and the fact that you don’t always want to go to the internet to get stuff, it’s a pain to be compiling things on servers all the time AND to do that, you need a compiler which is regarding as a Bad ThingTM in Production.

The question came to me via a talk I did at Chef conf and I’m going to use Chef examples, but, unless someone tells me different, I’m going to come down on the side of you can use this advice with Puppet and CFEngine and any other CM tool out there. It works without CM too, but why would you ever?

In this blog post, I want to talk about why I am so adamant about package repositories and offer some real world examples. I pick on Tomcat a lot here. Nothing against Tomcat. It’s something most people understand and that I have also had personal experience with it not being managed as a package and creating much distributed pain.

Broken Record Much?

I can’t shut up about package repos. Why am I so passionate about the need for all package repos all the time and to have a team actively managing and grooming them?

As part of the decentralization of Ops teams and the rise of DevOps, we now see multiple autonomous teams simultaneously working on similar projects but not collaborating or communicating on infrastructure. It behooves the infrastructure managers, whether that’s ops or devops or the tools team or a central collaborative group comprised of reps from all the teams, to own and groom a collection of package repositories.

But Why?
With the rise of self-service, all you can eat virtual machines, development teams are expected and want to be independent from the corporate server supply chain. Unfortunately this also decouples them from people they depend on: infrastructure and middleware experts(Yes Ops, that’s you). They may sail gayly on and do their best or trudge forward grudgingly, resenting all of the extra work they now have to do because the Enterprise wants to be “like a startup.” Either way, they’re going to move forward, and when you don’t offer them ways to easily use software, they’ll figure out ways to [cope][]; ways that will horrify you when you discover them.

I work at a startup because Enterprise is stupid like that.
Nope. It’s not only the Enterprise that needs to fear the chaos of unmanaged, difficult to use infrastructure. Say your whole startup is fewer than 20 people? You’re the “devops engineer” and you work with a 5 person dev team. The dev team may or may not include you in everything. You might not know of actions they’ve taken to [cope][] with problems you don’t even know they have. They don’t realize there could be a simple solution to their problems because they are brilliant startup devs and they will just figure it out. Then they’ll hand you their giant shiny mud pie and you will cry because you could have saved them a bunch of work with a few well placed repos and a web proxy.

Package repos create a beautiful garden where you static software can flourish and discourages tarball weeds from springing up in the cracks everywhere.

If you establish a well known, central location for packages, people will know to look there first for an existing package that meets their needs. And they will ask to place packages they need there as well.

Ex: In the situation where someone thinks

“I need Tomcat7 and Red Hat only ships with Tomcat5”

Their correlating thought will NOT be

“I should go download that from the internet and store it on the jumphost”

but instead

“I wonder if we have that in our custom repo?”

or in an even more mature organization

“I should include our Tomcat cookbook in my runlist and override the attributes that are unique for my situation”

When you make package repos easy to access and use, you

  • Eliminate several slightly different copies of software (ex: tomcat) floating around your infrastructure.
  • Eliminate several slightly different versions of packages deployed to Production.
  • Discourage placing random tarballs in version control or Chef cookbook files.
  • Make your configuration management tool easy and intuitive to use.

The Problem With Tarballs - a Real Life Example

Config management tools are designed to be idempotent, which means they only do things once and ensure things look the same regardless of how many times you run the the same process. Tarballs subvert the use of configuration management primitives, requiring the use of non-idempotent execute blocks to manipulate them. Real world example:

Without a package repo in place, this happens:

1
2
3
4
5
6
7
8
9
10
11
remote_file "/tmp/tomcat7.tar.gz" do
  source "https://www.sometomcaturl.com"
  not_if { File.exists?("#{node[:tomcat][:home]}/bin") }
end

execute "extract_tomcat" do
  command "tar xf /tmp/tomcat7.tar.gz"
  cwd "opt"
  not_if { File.exists?("#{node[:tomcat][:home]}/bin") }
  only_if { File.exists?("/tmp/tomcat7.tar.gz") }
end

Or this utter travesty:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
bash "install_tomcat6" do
  tomcat_version_name = "apache-tomcat-#{node.tomcat.version}"
  tomcat_version_name_tgz = "#{tomcat_version_name}.tar.gz"
  user "root"
  code <<-EOH
   curl --proxy https://aproxy.com:8080/ --user user:pass https://myartifactoryurl.com/artifactory/ext-release-local/apache-tomcat/apache-tomcat/#{node.tomcat.version}/#{tomcat_version_name_tgz} -o /tmp/#{tomcat_version_name_tgz}
   tar -zxf /tmp/#{tomcat_version_name_tgz} -C /tmp
   rm /tmp/#{tomcat_version_name_tgz}
   mv /tmp/#{tomcat_version_name} #{node.tomcat.install_path}
   chown -R #{node.tomcat.run_user}:#{node.tomcat.run_group} #{node.tomcat.install_path}
   chmod -R 755 #{node.tomcat.install_path}
   rm -rf #{node.tomcat.install_path}/webapps/ROOT
    EOH
end

Notice that they very carefully delete that tarball when they are done, ensuring that it will be re-downloaded 30 minutes later on the next run. They also untar the file and then move it to the install location without checking to see if anything is already there. Again, 30 minutes later the entire process happens again despite whatever is already there.

Now look at what happens with a package repo in place:

1
2
3
package 'tomcat7' do
  action :install
end

You may have to customize config files and init scripts, but this is a far simpler and less error prone way to install the base software.

In this case, Chef knows what OS you are using, what your package manager should be, repository connection details and how to install the package. Puppet does the exact same thing. So does every other config management system in the history of ever. The less code you have to write and the less verification code you have to write, they happier you’ll be and the people who come after you will be.

The End

I’ve heard a lot of complaints from ops and tools folks that devs do “crazy” stuff and they are much affronted that the devs don’t seem to understand the unpublished rules of operationalizing apps and the best way to use tools, and omg did you see the travesty of a Tomcat cookbook they wrote?

I have spent years on ops teams, tools teams and automation teams now. I have been one of those self-righteous, affronted peeps. This behavior doesn’t scale, yo.

One of the best messages that DevOps has for all of us is, if you don’t like the way someone’s doing something, talk to them and find out why. You’ll probably learn that there were great reasons for everything, including “I didn’t know how to do it, Chef is hard, Tomcat is hard, and I was just a Java developer until you guys started making me do all this automation stuff without telling me the rules or offering guidelines, so I copied another team’s Tomcat cookbook. I had no idea it’s crap. It installs Tomcat and that’s all I care about.”

Indeed. That’s all they care about. If you care about more than just the install, maybe you should reach out and offer to partner with them on the parts you care about. Don’t like the way devs are building application infrastructure? Well maybe you should do it for them. Provide them with a consistant VM build from a stable package repo and possibly infrastructure cookbooks with sane defaults and overrideable parameters. Chances are good devs don’t like recreating that crappy Tomcat wheel any more than you like them doing it. It’s likely they will be pathetically grateful because, while they are coping with the tasks they’ve been handed, they don’t really know what to do in detail and researching JVM tuning parameters is not their idea of a good time.

Why am I telling you this story? Because I believe that a major piece of your infrastructure stability is created with package repos. Making them easy to find, use and contribute to will help smooth any number of peripheral problems you are trying to solve. In my next post I’ll talk about practical considerations for repositories, packages and toolchains.

Pieces and Parts: Managing Your Ruby Toolchain (Part 1)

I recently received an email from someone who attended my ChefConf talk and asked me to expand on a few things. As I started to answer the email, I could see it was going to turn into a dissertation, so I migrated it over into a blog post. As I started to write the blog post, it seemed to grow into a book, so now I’ve decided to break it down into multiple posts. In this post I’ll try to summarize the question and situational environment and cover the easiest answers first: scripting languages and RVM.

The Question
Given the dynamic nature of Ruby tooling and the uncertainty of internet connections, and given that I work in the Enterprise, how should I design my supporting infrastructure and should I make Ruby my default scripting language?

The Situation
Situated in the Enterprise, working with Red Hat/CentOS Enterprise Linux(EL).

Concerns
* Rapidly changing versions of gems and software on the internet
* A constant need to download and compile Ruby (also compilers in production)
* The dynamic nature of downloading an RVM script from the internet on every install
* No connection to the internet (I’m not sure if this was a concern of his, but I’m betting it’s one of yours)

Let’s look at our major parts, shall we? In this set of posts we’ll cover
* Scripting Languages
* RVM
* [Package Repositories]
* [Package Versions]
* [Ruby Management]
* [Rubygems]

Scripting Languages
The question was, should we go all in on Ruby as a scripting language? My answer to that is, unless everyone loves Ruby, that’s probably an unnecessary hardship to impose while also trying to adopt Chef. Generally you find some combination of Shell, Python, Perl or Ruby on most enterprise server systems. I think you should script in the language that works best for you, where “works best” is a combination of what you’re comfortable with and what will best suit your immediate need.

Honestly I think we’ll never move entirely away from Shell scripting and that’s ok. It’s easy to learn and suits functional tasks like cron jobs and other server-specific work. Do I think you should have a script suite in 4 different interpreted languages? Well, no. If it were me, I’d pick Shell and Ruby for my options. But there’s no reason it can’t be Shell and Python or any other combo.

Me? I love Ruby and have an irrational dislike of Perl that stems from my early days in ops dealing with travesties of unreadable scripts. So I like Ruby first and Shell second, Python third and Perl never. But it’s hard enough learning a new tool like Chef. Don’t impose artificial constraints on your team.

Also, Your devs are probably already programming in Java or C or whatever. Over time most devs pick up a smattering of shell. If you force Ruby on them too it’s likely to breed resentment.

RVM
While I’m not an expert on RVM and other Ruby managers, I have heard that it’s not meant to be a server tool. I do use RVM for workstation development and I’ve used Rbenv as well. I don’t mind compiling Ruby from scratch when installing it locally although I will love it when RVM gets precompiled rubies for OSX.

A Slight Digression on Workstation Automation

I’ve written workstation automation using Fletcher Nichol’s chef-rvm cookbook. A frustrating problem I have run into is that RVM undergoes frequent development and at first I had people coming to me because the workstation toolchain install wouldn’t work and it was almost always because the RVM version had changed and something was now wonky in the settings or install chain. So I started pinning the RVM version to install with occasional revisits to bump the version when I could.

If you want to look at workstation automation for OSX 10.8 try a couple of these repos:

  • Sprout-wrap - This used to be Pivotal Workstation and is now basically an instruction page for getting going with Soloist
  • SoloWizard - a browser based checklist that will assemble a chef solo provisioner for your workstation.
  • Joshua Timberman’s workstation-chef-repo and mac_os_x-cookbook for his personal collection of workstation automation
  • Puppet-RVM - Puppet module for installing and managing RVM
  • Boxen - A tool for automating OSX with Puppet. Links to a blog post with details.

I have written workstation automation but I don’t maintain it these days and I’ll probably try Soloist when I get my next new Mac.

TL;DR:
* Use RVM on workstations not servers
* If you’re automating for many people over time, pin your RVM version

Wrap Up To keep the blog post readable, this is all I want to discuss in this first post, but I’m about to publish Part 2 as well: a rant about package repos. I’m extremely interested in feedback on this series as I was only half joking about writing a book, though it would likely be an online doc of some kind. This type of discussion evolves too quickly to commit to paper.

Minneapolis - St Paul Infracoders Meetup Recap

Last night we had a fun second meetup of the Twin Cities Infracoders group. On the docket were presentations and demos for Vagrant and Sensu. We had about 18 people show up and a lively discussion ensued on why you’d want to use something like Vagrant and implementation strategies.

Most of us have partnered with development teams who have asked for things that are difficult to provide or unwise and one of the best use cases for Vagrant is the ability to hand someone a development environment homogenous to the team and easy to troubleshoot. It’s homogenous because everyone uses the same base OS box file and also uses the same provisioner to create the environment, whether that’s Chef, Puppet or Bash scripts or anything else. It also shortens time required to bootstrap a new team member.

Mike Goetz and Tom Duffield gave us a great compressed Wordpress install demo on a local VM with Vagrant and Chef. Then they gave us a second demo of spinning up two ec2 instances to separate the front and back end pieces of Wordpress.

After Vagrant, we ran through some Sensu slides and looked at some basic info about Sensu. There were technical difficulties around the demos we were working on so demoing was minimal but discussion around why Sensu, how Sensu and when you might switch was great and we’re looking forward to a more detailed demo at next month’s meetup.

Here is the collection of tools and links from last night:

Vagrant
Vagrant Plugins
Sensu
Sean Porter’s Slides
Wordpress Demo(this link will be available shortly)

Coming up:

Next month’s meetup, hosted again at the Nerdery, will be a presentation on using Selenium for automated testing and QA as well as our enhanced Sensu Server and client nodes deployed with Vagabond on LXCs inside a Vagrant VM. I hope we get a great turnout.

On another note, I have been talking lately about how I want to organize a Chef hackday. After last night’s discussions around tooling and workflow, I think it might be nice to instead do a tool-agnostic Workflow Tooling hack and help day. Many of us have gotten workflows configured successfully, but depending on your experience beating on the Ruby toolchain, it can be challenging the first couple times you do it. The Nerdery said they would probably be willing to host and I was thinking about asking CoCo Minneapolis if they had any interest in hosting.

I’ll post up a question in the meetup group about location and date preferences to see if I get enough interest to make it a thing. I’ll also send someone to shill at the monthly DevOps meetup that I can never attend due to my work travel schedule.

If you haven’t made it out to an Infracoder meetup yet, I hope you because we are having some great tech discussions and getting to hear about other people’s uses cases for things is fascinating. Also, the Nerdery takes super great care of us and feeds us Pizza for dinner.

Devops Consulting: Why I Don’t

I had an interesting exchange with Adam Jacob yesterday on Twitter around devops consulting. It’s a topic I’ve discussed often with friends as we work out how to spread the devops message in our consulting adventures. After the Twitter chat, I’m motivated to say a few words about devops consulting, not because there was an argument, but because it’s something I’ve thought about often since I started independent consulting and I think it’s important.

The twitterings began with me trolling Adam on a topic about which I learned my first, best lessons from him. Two years ago, my ideas around devops were still mostly unformed when I saw him give his “What is Devops” and “The No Assholes Rule” vignettes at Velocity. These two 5 minute segments clarified several nebulous thoughts I’d been having and form the base for how I approach devops today.

My trolling quickly transformed into an exchange as to whether devops consulting is a valid thing. Here’s my take:

Devops is The New Black

As far as toolmakers, tool-consumers and recruiters are all concerned, devops is the new black. I get frustrated with the “slap a devops on it and they will come” attitude we see prevailing. It makes me want to never use the word “devops” to describe anything ever because of things like this:

  • If it’s a tool for scaling or automation or monitoring or cloud, it’s now a “devops tool”.

  • If you have a job for some poor schlub to do production support for the dev team, you call it “looking for a devops engineer with build experience.”

  • A tooling company has a giant virtual sticker on the website: “Designed by devops.”

  • If you want to get some other poor schlub to join your overworked ops team you call it a “devops engineering” position but it’s really just another day in the life of duct-taping together prod.

I already hear colleagues succumbing to the terminology “devops Tool” because it’s a phrase understood by (or at least familiar to) upper management and business users. I can even see the day when I give in and start calling things devops tools. This makes me sad.

Emo Devops

Call me cynical, but this sort of devaluation discourages me and has turned me into something of a hipster goth. You know, the girl in all black with the black nail polish listening to Bauhaus and denying she’s goth because she transcends labels in her non-originality. I can make fun of this girl because I was her in high school.

But as I see tool companies and recruiters slapping devops across everything and large companies begging for the devops panacea, I become less and less inclined to talk about devops and the hipster goth in me is damned if she’s going to cash in on all the quick bucks of devops consulting.

Devops Consulting - Why I Don’t

Last year I had a company try to hire me as a devops consultant, but the very idea made me shudder. It felt utterly pretentious even though they had the best of intentions and were knowledgeable about devops. My business card says Devops Enthusiast. That’s it. I love it and I try to live it every day I walk into an office, whether it’s my home office, a client office or any other place with people in it.

But my business card will never say Devops Consultant. Why not?

Devops Isn’t a Deliverable - When I get done with you, you don’t have a devops. It’s possible that the things we’re working on will bring you closer to a lifestyle and culture that is devops. And that’s super. Working at scale demands extraordinary feats of technical prowess, but it also demands empathy and trust, key components of devops culture. I expect this to be a natural byproduct of my tenure, not the thing I craft.

Consultants Have No Moral Authority - This is the classic water and horse thing. You can hire someone to tell you how to do devops, but unless everyone is signed on and excited, it’s going to be the consultant and the person who hired the consultant drinking from the trough and a bunch of other horses standing around dehydrating. I have no desire to earn a living doing this.

I’m Not Qualified - I learn new things every day and I often remind myself that devops isn’t just me preaching DevOplyness, but also involves me actively empathizing with and trusting people I find difficult or frustrating to work with. This stuff is hard, and I’m not even close to perfect. I’d rather lead by example than preach. I feel like a hypocrite telling other people how to put their house in order when I’m still cleaning my own.

I’d Rather Be Crafting - Let’s face it. If I say Devops Consultant, that’s management consulting. As much as I’ve considered going into management because of the ignorance I see in charge of stuff, I’d rather be making things. This is a selfish thing, right? I love to CLI and hack on Ruby and play with Chef and figure out why stuff is broken, mentor newer engineers and coach devs on how to operationalize their apps. I don’t get to do that if I’m teaching senior management about devops. It’s why I like consulting. There’s always a new problem to solve.

Is Continuous Delivery a valid Euphemism for devops?

This was also touched on yesterday. Jeff Sussna wrote a well-articulated article for why we should just call it Continuous Delivery instead of devops. I think Adam and I were not actually disagreeing about anything in our chat today, unless he was postulating that CD is as nebulous as devops. I don’t think it is. I think Continuous Delivery is a measurable deliverable and a valid consulting field. If done right, it results in not just a great pipeline, but also devops culture because CD should be impossible with crappy culture.

What About devops?

I know some folks who are doing the devops consulting thing. They generally want to do the management consulting thing and they believe wholeheartedly in devops as cultural shift and in CAMS, which I think is the key to making it work, if that’s what you really want to do.
There are also some super fantastic companies with amazing cultures; some calling it devops, others not but still living the lifestyle. I’m not panning everything calling itself devops. However, the more I see vendors and recruiters latch onto the word and commoditize it without sincerity, the less I want to actually associate with the word myself and the more grains of salt I require when I get yet another email from someone telling me about their great devops job for which I’d be a perfect fit.

Coffee Nerdery: Chromatic Coffee

Chromatic Coffee
Location: Santa Clara, CA, a few miles from the Apple campus
Espresso: Emperor
Hardware: La Marzocco and heavily modded grinders of unknown origin (see picture)
Coffee Art: heck yeah
Wifi: Yes
Drink: Cappuccino
Served in: 5oz ceramic cups
Visited: Daily in February and March 2013

I first met Chromatic Coffee when it was a franchise under another name in June 2012. For the record, it was super then too. I recently acquired a client in Santa Clara, CA and knew I would be traveling out here every week. Strangely, Silicon Valley does NOT have a great artisan coffee shop on every corner. A lot of the area is actually office park and strip mall hell. When I found this place last year, I drove 8 miles from the Hyatt Convention Hotel to get some decent coffee. (For the other record, all of the espresso served at the Hyatt is utterly vile).

So I immediately thought of this location when I set up weekly trips to California. In the meantime I discovered the shop had changed its name to Chromatic Coffee but was the same crew of awesome baristas that I’d met last year. I actually managed to find a good hotel near the shop, putting me about 5 miles from the client but worth every commuting minute, knowing I’d have decent coffee at least once a day (I work in office park hell).

I am happy to say that breaking off on their own has only improved things over here. In my chats with the baristas, I’ve learned that the owner is a tech obsessed uber coffee nerd. You can see this in the fact that they are their own roastery and hey, check out the grinders. Heavily modded, the barista couldn’t remember the original brand on them, but informed me that the owner is actually working on his own equipment as well. (uber techie coffee nerds unite!).

So I can’t tell you enough how much I love Chromatic. Everyone there is super nice and super knowledgeable about the coffee and will happily nerd out over coffee talk when it’s not busy. There’s also a Hario pourover bar and they tell me they make cold brew as well, which I will be sampling next week. In addition to all the other awesome, they hold classes on everything from pour-over technique to barista art.

Bonus: They are not snooty coffee snobs here. You can order your drink with skim milk or to go and no one will look down their nose at your or inform you snottily that they only use whole milk from locally approved cows.

I wish I could keep gushing, but you should really get in here and try them for yourself. Don’t be fooled by their strip mall exterior. They are beyond awesome.

Shell Scripts Are Like Gremlins

Today I ended up in a heated discussion with some team mates over deployment strategies. As is often the case with this team, myself included when I don’t stop and think, we often leap right to arguing over which tool is best before discussing the problem we want to solve. It wasn’t the first time.

The source of this discussion was brought on while we were reviewing some chef work I’m doing with a development team. My work was mainly to assist them in getting to functioning cookbooks that also had their app logic separate from global cookbooks. I basically copied their intent wherever it made sense while shoring up design and Chef styling. One of the things they were doing was making a call to our Artifactory server for a latest snapshot of a WAR file, downloading it into the Tomcat directory and restarting Tomcat. Works for me.

Getting to this point in the review triggered a long, heated debate over

  1. whether you ever wanted this to happen,
  2. whether you should use Chef to manage deployments, use Jenkins to kick off shell scripts or use some other orchestration tool to do something else or
  3. just copy the world by hand (well, no one really believes that last is a good idea).

I was like, geez guys, I’m just mimicking the dev team’s functionality, why are we are arguing about this? But this has been a topic of discussion often recently and, with another team announcing yesterday they were writing a homegrown tool to manage jboss, deploy ATG ears and manage config files all retrieved from Artifactory, I don’t expect the subject to die soon.

I don’t think there’s one right answer to how you deploy your code, but I think there are many poorly thought out ones. I’m not here to necessarily make an argument for Chef as a deployment orchestrator. While there are people deploying with Chef at scale, I am not one of them, nor have I been. My work with Chef has been mainly in development with some provisioning and so I have a lot of theories, but that’s really all. What I do want to talk about is why I don’t like shell scripts for deployments or orchestration and what I want in a deployment system. This is the first part of at least a 2 on this topic.

Shell Scripts are like Gremlins. You start out with one adorably cute shell script. You commented it and it does one thing really well. It’s easy to read, everyone can use it. It’s awesome! Then you accidentally spill some water on it, or feed it late one night and omgwtf is happening!?

The Fixer: Someone else comes along and found an edge case your shell script doesn’t deal with. They add in some logic for the edge case, voila, problem solved.

The Refiner: Eventually someone realizes the logic for the edge case is not specific enough and is causing deployments to fail sometimes, so they refine the logic.

The Slippery Slope: After that, someone might decide it’s a good idea to automate stopping the apache server from sending traffic during deployments and decides to do it in the same script. Great idea! That’s such an awesome idea that everyone starts adding functionality to the tiny beautiful shell script, now no longer tiny nor beautiful.

OMG GREMLINS! Then you come back along and find all the extra stuff in your shell script that doesn’t belong there. You’re horrified. You might even be feeling a little bit violated (come on, we’ve all been there at least once). So what do you do? You pour some water on it. You break out the shell script into several functional bits. Now we have LOTS of gremlins instead of just one. Now you have a suite of scripts that are once again beautiful. But now the deployment is complex. You have a suite of bash perl ruby python scripts that also need a wiki page to describe intended flow, what to do if something doesn’t work and any edge cases that you haven’t gotten around to scripting yet.

The Exodus: Next up: You get a call from a buddy who is dying to have you come work for his Next Big Thing startup. So you quit your job, pack your bags and move to Silicon Valley Sitcom with ne’er a backwards glance, leaving a couple of forlorn junior sysadmins desperately reading wiki pages trying to figure out what to do with your shell scripts as a new application is launched that requires a bunch of new logic for deployment. These guys do the best they can and start tacking on if statements everywhere they need to in order to make the deployments go.

Subsistance Living: 6 months later, one of these guys leaves and the company hires 3 more guys with no understanding of the history of the deployment scripts. Sometimes they work, sometimes they don’t, people aren’t entirely sure why and just self correct by hand at the command line until things work(phew!). Everyone is afraid to touch them because they are fragile, the code connecting them is obscure and there are similar logic blocks found in several sections, sometimes commented out, sometimes used, but you’re not really sure whether it’s necessary. The original wiki page gets updated sometimes but not often and not usually by the person maintaining the scripts but by the people using them in the middle of the night.

And that’s why I hate shell scripts and think you should never use them for deployment scaffolding.

True story: my first venture with Chef involved deconstructing an organically grown Kickstart post that had been originally written for Red Hat 3 and subsequently updated for RH 4, 5 and 6. I was removing functionality from the postscript and rewriting it in Chef blocks when one of the admins came and yelled at me for omitting a block of host names from /etc/hosts and I was like, GUYS, those host names are for servers in a data center that was decommissioned when I started here 3 years ago.

You can tune in for the second half of this blog post, what I want in a decent deployment system, when it goes up next week on the Sysadvent blog. Woohoo!