TL;DR Use package repos for all your static software

Last weekend I wrote a blog post that started exploring standards for managing your Ruby toolchains. To review, I had a query from someone on how to cope with the dynamic nature of Ruby-based tooling and managing a Chef infrastructure. and the fact that you don’t always want to go to the internet to get stuff, it’s a pain to be compiling things on servers all the time AND to do that, you need a compiler which is regarding as a Bad Thing^TM in Production.

The question came to me via a talk I did at Chef conf and I’m going to use Chef examples, but, unless someone tells me different, I’m going to come down on the side of you can use this advice with Puppet and CFEngine and any other CM tool out there. It works without CM too, but why would you ever?

In this blog post, I want to talk about why I am so adamant about package repositories and offer some real world examples. I pick on Tomcat a lot here. Nothing against Tomcat. It’s something most people understand and that I have also had personal experience with it not being managed as a package and creating much distributed pain.

Broken Record Much?

I can’t shut up about package repos. Why am I so passionate about the need for all package repos all the time and to have a team actively managing and grooming them?

As part of the decentralization of Ops teams and the rise of DevOps, we now see multiple autonomous teams simultaneously working on similar projects but not collaborating or communicating on infrastructure. It behooves the infrastructure managers, whether that’s ops or devops or the tools team or a central collaborative group comprised of reps from all the teams, to own and groom a collection of package repositories.

But Why?
With the rise of self-service, all you can eat virtual machines, development teams are expected and want to be independent from the corporate server supply chain. Unfortunately this also decouples them from people they depend on: infrastructure and middleware experts(Yes Ops, that’s you). They may sail gayly on and do their best or trudge forward grudgingly, resenting all of the extra work they now have to do because the Enterprise wants to be “like a startup.” Either way, they’re going to move forward, and when you don’t offer them ways to easily use software, they’ll figure out ways to [cope][]; ways that will horrify you when you discover them.

I work at a startup because Enterprise is stupid like that.
Nope. It’s not only the Enterprise that needs to fear the chaos of unmanaged, difficult to use infrastructure. Say your whole startup is fewer than 20 people? You’re the “devops engineer” and you work with a 5 person dev team. The dev team may or may not include you in everything. You might not know of actions they’ve taken to [cope][] with problems you don’t even know they have. They don’t realize there could be a simple solution to their problems because they are brilliant startup devs and they will just figure it out. Then they’ll hand you their giant shiny mud pie and you will cry because you could have saved them a bunch of work with a few well placed repos and a web proxy.

Package repos create a beautiful garden where you static software can flourish and discourages tarball weeds from springing up in the cracks everywhere.

If you establish a well known, central location for packages, people will know to look there first for an existing package that meets their needs. And they will ask to place packages they need there as well.

Ex: In the situation where someone thinks

“I need Tomcat7 and Red Hat only ships with Tomcat5”

Their correlating thought will NOT be

“I should go download that from the internet and store it on the jumphost”

but instead

“I wonder if we have that in our custom repo?”

or in an even more mature organization

“I should include our Tomcat cookbook in my runlist and override the attributes that are unique for my situation”

When you make package repos easy to access and use, you

• Eliminate several slightly different copies of software (ex: tomcat) floating around your infrastructure.
• Eliminate several slightly different versions of packages deployed to Production.
• Discourage placing random tarballs in version control or Chef cookbook files.
• Make your configuration management tool easy and intuitive to use.

The Problem With Tarballs - a Real Life Example

Config management tools are designed to be idempotent, which means they only do things once and ensure things look the same regardless of how many times you run the the same process. Tarballs subvert the use of configuration management primitives, requiring the use of non-idempotent execute blocks to manipulate them. Real world example:

Without a package repo in place, this happens:

1
2
3
4
5
6
7
8
9
10
11

remote_file "/tmp/tomcat7.tar.gz" do
  source "https://www.sometomcaturl.com"
  not_if { File.exists?("#{node[:tomcat][:home]}/bin") }
end

execute "extract_tomcat" do
  command "tar xf /tmp/tomcat7.tar.gz"
  cwd "opt"
  not_if { File.exists?("#{node[:tomcat][:home]}/bin") }
  only_if { File.exists?("/tmp/tomcat7.tar.gz") }
end

Or this utter travesty:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

bash "install_tomcat6" do
  tomcat_version_name = "apache-tomcat-#{node.tomcat.version}"
  tomcat_version_name_tgz = "#{tomcat_version_name}.tar.gz"
  user "root"
  code <<-EOH
   curl --proxy https://aproxy.com:8080/ --user user:pass https://myartifactoryurl.com/artifactory/ext-release-local/apache-tomcat/apache-tomcat/#{node.tomcat.version}/#{tomcat_version_name_tgz} -o /tmp/#{tomcat_version_name_tgz}
   tar -zxf /tmp/#{tomcat_version_name_tgz} -C /tmp
   rm /tmp/#{tomcat_version_name_tgz}
   mv /tmp/#{tomcat_version_name} #{node.tomcat.install_path}
   chown -R #{node.tomcat.run_user}:#{node.tomcat.run_group} #{node.tomcat.install_path}
   chmod -R 755 #{node.tomcat.install_path}
   rm -rf #{node.tomcat.install_path}/webapps/ROOT
    EOH
end

Notice that they very carefully delete that tarball when they are done, ensuring that it will be re-downloaded 30 minutes later on the next run. They also untar the file and then move it to the install location without checking to see if anything is already there. Again, 30 minutes later the entire process happens again despite whatever is already there.

Now look at what happens with a package repo in place:

1
2
3

package 'tomcat7' do
  action :install
end

You may have to customize config files and init scripts, but this is a far simpler and less error prone way to install the base software.

In this case, Chef knows what OS you are using, what your package manager should be, repository connection details and how to install the package. Puppet does the exact same thing. So does every other config management system in the history of ever. The less code you have to write and the less verification code you have to write, they happier you’ll be and the people who come after you will be.

The End

I’ve heard a lot of complaints from ops and tools folks that devs do “crazy” stuff and they are much affronted that the devs don’t seem to understand the unpublished rules of operationalizing apps and the best way to use tools, and omg did you see the travesty of a Tomcat cookbook they wrote?

I have spent years on ops teams, tools teams and automation teams now. I have been one of those self-righteous, affronted peeps. This behavior doesn’t scale, yo.

One of the best messages that DevOps has for all of us is, if you don’t like the way someone’s doing something, talk to them and find out why. You’ll probably learn that there were great reasons for everything, including “I didn’t know how to do it, Chef is hard, Tomcat is hard, and I was just a Java developer until you guys started making me do all this automation stuff without telling me the rules or offering guidelines, so I copied another team’s Tomcat cookbook. I had no idea it’s crap. It installs Tomcat and that’s all I care about.”

Indeed. That’s all they care about. If you care about more than just the install, maybe you should reach out and offer to partner with them on the parts you care about. Don’t like the way devs are building application infrastructure? Well maybe you should do it for them. Provide them with a consistant VM build from a stable package repo and possibly infrastructure cookbooks with sane defaults and overrideable parameters. Chances are good devs don’t like recreating that crappy Tomcat wheel any more than you like them doing it. It’s likely they will be pathetically grateful because, while they are coping with the tasks they’ve been handed, they don’t really know what to do in detail and researching JVM tuning parameters is not their idea of a good time.

Why am I telling you this story? Because I believe that a major piece of your infrastructure stability is created with package repos. Making them easy to find, use and contribute to will help smooth any number of peripheral problems you are trying to solve. In my next post I’ll talk about practical considerations for repositories, packages and toolchains.

I recently received an email from someone who attended my ChefConf talk and asked me to expand on a few things. As I started to answer the email, I could see it was going to turn into a dissertation, so I migrated it over into a blog post. As I started to write the blog post, it seemed to grow into a book, so now I’ve decided to break it down into multiple posts. In this post I’ll try to summarize the question and situational environment and cover the easiest answers first: scripting languages and RVM.

The Question
Given the dynamic nature of Ruby tooling and the uncertainty of internet connections, and given that I work in the Enterprise, how should I design my supporting infrastructure and should I make Ruby my default scripting language?

The Situation
Situated in the Enterprise, working with Red Hat/CentOS Enterprise Linux(EL).

Concerns
* Rapidly changing versions of gems and software on the internet
* A constant need to download and compile Ruby (also compilers in production)
* The dynamic nature of downloading an RVM script from the internet on every install
* No connection to the internet (I’m not sure if this was a concern of his, but I’m betting it’s one of yours)

Let’s look at our major parts, shall we? In this set of posts we’ll cover
* Scripting Languages
* RVM
* [Package Repositories]
* [Package Versions]
* [Ruby Management]
* [Rubygems]

Scripting Languages
The question was, should we go all in on Ruby as a scripting language? My answer to that is, unless everyone loves Ruby, that’s probably an unnecessary hardship to impose while also trying to adopt Chef. Generally you find some combination of Shell, Python, Perl or Ruby on most enterprise server systems. I think you should script in the language that works best for you, where “works best” is a combination of what you’re comfortable with and what will best suit your immediate need.

Honestly I think we’ll never move entirely away from Shell scripting and that’s ok. It’s easy to learn and suits functional tasks like cron jobs and other server-specific work. Do I think you should have a script suite in 4 different interpreted languages? Well, no. If it were me, I’d pick Shell and Ruby for my options. But there’s no reason it can’t be Shell and Python or any other combo.

Me? I love Ruby and have an irrational dislike of Perl that stems from my early days in ops dealing with travesties of unreadable scripts. So I like Ruby first and Shell second, Python third and Perl never. But it’s hard enough learning a new tool like Chef. Don’t impose artificial constraints on your team.

Also, Your devs are probably already programming in Java or C or whatever. Over time most devs pick up a smattering of shell. If you force Ruby on them too it’s likely to breed resentment.

RVM
While I’m not an expert on RVM and other Ruby managers, I have heard that it’s not meant to be a server tool. I do use RVM for workstation development and I’ve used Rbenv as well. I don’t mind compiling Ruby from scratch when installing it locally although I will love it when RVM gets precompiled rubies for OSX.

A Slight Digression on Workstation Automation

I’ve written workstation automation using Fletcher Nichol’s chef-rvm cookbook. A frustrating problem I have run into is that RVM undergoes frequent development and at first I had people coming to me because the workstation toolchain install wouldn’t work and it was almost always because the RVM version had changed and something was now wonky in the settings or install chain. So I started pinning the RVM version to install with occasional revisits to bump the version when I could.

If you want to look at workstation automation for OSX 10.8 try a couple of these repos:

• Sprout-wrap - This used to be Pivotal Workstation and is now basically an instruction page for getting going with Soloist
• SoloWizard - a browser based checklist that will assemble a chef solo provisioner for your workstation.
• Joshua Timberman’s workstation-chef-repo and mac_os_x-cookbook for his personal collection of workstation automation
• Puppet-RVM - Puppet module for installing and managing RVM
• Boxen - A tool for automating OSX with Puppet. Links to a blog post with details.

I have written workstation automation but I don’t maintain it these days and I’ll probably try Soloist when I get my next new Mac.

TL;DR:
* Use RVM on workstations not servers
* If you’re automating for many people over time, pin your RVM version

Wrap Up To keep the blog post readable, this is all I want to discuss in this first post, but I’m about to publish Part 2 as well: a rant about package repos. I’m extremely interested in feedback on this series as I was only half joking about writing a book, though it would likely be an online doc of some kind. This type of discussion evolves too quickly to commit to paper.

Last night we had a fun second meetup of the Twin Cities Infracoders group. On the docket were presentations and demos for Vagrant and Sensu. We had about 18 people show up and a lively discussion ensued on why you’d want to use something like Vagrant and implementation strategies.

Most of us have partnered with development teams who have asked for things that are difficult to provide or unwise and one of the best use cases for Vagrant is the ability to hand someone a development environment homogenous to the team and easy to troubleshoot. It’s homogenous because everyone uses the same base OS box file and also uses the same provisioner to create the environment, whether that’s Chef, Puppet or Bash scripts or anything else. It also shortens time required to bootstrap a new team member.

Mike Goetz and Tom Duffield gave us a great compressed Wordpress install demo on a local VM with Vagrant and Chef. Then they gave us a second demo of spinning up two ec2 instances to separate the front and back end pieces of Wordpress.

After Vagrant, we ran through some Sensu slides and looked at some basic info about Sensu. There were technical difficulties around the demos we were working on so demoing was minimal but discussion around why Sensu, how Sensu and when you might switch was great and we’re looking forward to a more detailed demo at next month’s meetup.

Here is the collection of tools and links from last night:

Vagrant
Vagrant Plugins
Sensu
Sean Porter’s Slides
Wordpress Demo(this link will be available shortly)

Coming up:

Next month’s meetup, hosted again at the Nerdery, will be a presentation on using Selenium for automated testing and QA as well as our enhanced Sensu Server and client nodes deployed with Vagabond on LXCs inside a Vagrant VM. I hope we get a great turnout.

On another note, I have been talking lately about how I want to organize a Chef hackday. After last night’s discussions around tooling and workflow, I think it might be nice to instead do a tool-agnostic Workflow Tooling hack and help day. Many of us have gotten workflows configured successfully, but depending on your experience beating on the Ruby toolchain, it can be challenging the first couple times you do it. The Nerdery said they would probably be willing to host and I was thinking about asking CoCo Minneapolis if they had any interest in hosting.

I’ll post up a question in the meetup group about location and date preferences to see if I get enough interest to make it a thing. I’ll also send someone to shill at the monthly DevOps meetup that I can never attend due to my work travel schedule.

If you haven’t made it out to an Infracoder meetup yet, I hope you because we are having some great tech discussions and getting to hear about other people’s uses cases for things is fascinating. Also, the Nerdery takes super great care of us and feeds us Pizza for dinner.

I had an interesting exchange with Adam Jacob yesterday on Twitter around devops consulting. It’s a topic I’ve discussed often with friends as we work out how to spread the devops message in our consulting adventures. After the Twitter chat, I’m motivated to say a few words about devops consulting, not because there was an argument, but because it’s something I’ve thought about often since I started independent consulting and I think it’s important.

The twitterings began with me trolling Adam on a topic about which I learned my first, best lessons from him. Two years ago, my ideas around devops were still mostly unformed when I saw him give his “What is Devops” and “The No Assholes Rule” vignettes at Velocity. These two 5 minute segments clarified several nebulous thoughts I’d been having and form the base for how I approach devops today.

My trolling quickly transformed into an exchange as to whether devops consulting is a valid thing. Here’s my take:

Devops is The New Black

As far as toolmakers, tool-consumers and recruiters are all concerned, devops is the new black. I get frustrated with the “slap a devops on it and they will come” attitude we see prevailing. It makes me want to never use the word “devops” to describe anything ever because of things like this:

• If it’s a tool for scaling or automation or monitoring or cloud, it’s now a “devops tool”.

• If you have a job for some poor schlub to do production support for the dev team, you call it “looking for a devops engineer with build experience.”

• A tooling company has a giant virtual sticker on the website: “Designed by devops.”

• If you want to get some other poor schlub to join your overworked ops team you call it a “devops engineering” position but it’s really just another day in the life of duct-taping together prod.

I already hear colleagues succumbing to the terminology “devops Tool” because it’s a phrase understood by (or at least familiar to) upper management and business users. I can even see the day when I give in and start calling things devops tools. This makes me sad.

Emo Devops

Call me cynical, but this sort of devaluation discourages me and has turned me into something of a hipster goth. You know, the girl in all black with the black nail polish listening to Bauhaus and denying she’s goth because she transcends labels in her non-originality. I can make fun of this girl because I was her in high school.

But as I see tool companies and recruiters slapping devops across everything and large companies begging for the devops panacea, I become less and less inclined to talk about devops and the hipster goth in me is damned if she’s going to cash in on all the quick bucks of devops consulting.

Devops Consulting - Why I Don’t

Last year I had a company try to hire me as a devops consultant, but the very idea made me shudder. It felt utterly pretentious even though they had the best of intentions and were knowledgeable about devops. My business card says Devops Enthusiast. That’s it. I love it and I try to live it every day I walk into an office, whether it’s my home office, a client office or any other place with people in it.

But my business card will never say Devops Consultant. Why not?

Devops Isn’t a Deliverable - When I get done with you, you don’t have a devops. It’s possible that the things we’re working on will bring you closer to a lifestyle and culture that is devops. And that’s super. Working at scale demands extraordinary feats of technical prowess, but it also demands empathy and trust, key components of devops culture. I expect this to be a natural byproduct of my tenure, not the thing I craft.

Consultants Have No Moral Authority - This is the classic water and horse thing. You can hire someone to tell you how to do devops, but unless everyone is signed on and excited, it’s going to be the consultant and the person who hired the consultant drinking from the trough and a bunch of other horses standing around dehydrating. I have no desire to earn a living doing this.

I’m Not Qualified - I learn new things every day and I often remind myself that devops isn’t just me preaching DevOplyness, but also involves me actively empathizing with and trusting people I find difficult or frustrating to work with. This stuff is hard, and I’m not even close to perfect. I’d rather lead by example than preach. I feel like a hypocrite telling other people how to put their house in order when I’m still cleaning my own.

I’d Rather Be Crafting - Let’s face it. If I say Devops Consultant, that’s management consulting. As much as I’ve considered going into management because of the ignorance I see in charge of stuff, I’d rather be making things. This is a selfish thing, right? I love to CLI and hack on Ruby and play with Chef and figure out why stuff is broken, mentor newer engineers and coach devs on how to operationalize their apps. I don’t get to do that if I’m teaching senior management about devops. It’s why I like consulting. There’s always a new problem to solve.

Is Continuous Delivery a valid Euphemism for devops?

This was also touched on yesterday. Jeff Sussna wrote a well-articulated article for why we should just call it Continuous Delivery instead of devops. I think Adam and I were not actually disagreeing about anything in our chat today, unless he was postulating that CD is as nebulous as devops. I don’t think it is. I think Continuous Delivery is a measurable deliverable and a valid consulting field. If done right, it results in not just a great pipeline, but also devops culture because CD should be impossible with crappy culture.

What About devops?

I know some folks who are doing the devops consulting thing. They generally want to do the management consulting thing and they believe wholeheartedly in devops as cultural shift and in CAMS, which I think is the key to making it work, if that’s what you really want to do.
There are also some super fantastic companies with amazing cultures; some calling it devops, others not but still living the lifestyle. I’m not panning everything calling itself devops. However, the more I see vendors and recruiters latch onto the word and commoditize it without sincerity, the less I want to actually associate with the word myself and the more grains of salt I require when I get yet another email from someone telling me about their great devops job for which I’d be a perfect fit.

Chromatic Coffee
Location: Santa Clara, CA, a few miles from the Apple campus
Espresso: Emperor
Hardware: La Marzocco and heavily modded grinders of unknown origin (see picture)
Coffee Art: heck yeah
Wifi: Yes
Drink: Cappuccino
Served in: 5oz ceramic cups
Visited: Daily in February and March 2013

I first met Chromatic Coffee when it was a franchise under another name in June 2012. For the record, it was super then too. I recently acquired a client in Santa Clara, CA and knew I would be traveling out here every week. Strangely, Silicon Valley does NOT have a great artisan coffee shop on every corner. A lot of the area is actually office park and strip mall hell. When I found this place last year, I drove 8 miles from the Hyatt Convention Hotel to get some decent coffee. (For the other record, all of the espresso served at the Hyatt is utterly vile).

So I immediately thought of this location when I set up weekly trips to California. In the meantime I discovered the shop had changed its name to Chromatic Coffee but was the same crew of awesome baristas that I’d met last year. I actually managed to find a good hotel near the shop, putting me about 5 miles from the client but worth every commuting minute, knowing I’d have decent coffee at least once a day (I work in office park hell).

I am happy to say that breaking off on their own has only improved things over here. In my chats with the baristas, I’ve learned that the owner is a tech obsessed uber coffee nerd. You can see this in the fact that they are their own roastery and hey, check out the grinders. Heavily modded, the barista couldn’t remember the original brand on them, but informed me that the owner is actually working on his own equipment as well. (uber techie coffee nerds unite!).

So I can’t tell you enough how much I love Chromatic. Everyone there is super nice and super knowledgeable about the coffee and will happily nerd out over coffee talk when it’s not busy. There’s also a Hario pourover bar and they tell me they make cold brew as well, which I will be sampling next week. In addition to all the other awesome, they hold classes on everything from pour-over technique to barista art.

Bonus: They are not snooty coffee snobs here. You can order your drink with skim milk or to go and no one will look down their nose at your or inform you snottily that they only use whole milk from locally approved cows.

I wish I could keep gushing, but you should really get in here and try them for yourself. Don’t be fooled by their strip mall exterior. They are beyond awesome.

Today I ended up in a heated discussion with some team mates over deployment strategies. As is often the case with this team, myself included when I don’t stop and think, we often leap right to arguing over which tool is best before discussing the problem we want to solve. It wasn’t the first time.

The source of this discussion was brought on while we were reviewing some chef work I’m doing with a development team. My work was mainly to assist them in getting to functioning cookbooks that also had their app logic separate from global cookbooks. I basically copied their intent wherever it made sense while shoring up design and Chef styling. One of the things they were doing was making a call to our Artifactory server for a latest snapshot of a WAR file, downloading it into the Tomcat directory and restarting Tomcat. Works for me.

Getting to this point in the review triggered a long, heated debate over

1. whether you ever wanted this to happen,
2. whether you should use Chef to manage deployments, use Jenkins to kick off shell scripts or use some other orchestration tool to do something else or
3. just copy the world by hand (well, no one really believes that last is a good idea).

I was like, geez guys, I’m just mimicking the dev team’s functionality, why are we are arguing about this? But this has been a topic of discussion often recently and, with another team announcing yesterday they were writing a homegrown tool to manage jboss, deploy ATG ears and manage config files all retrieved from Artifactory, I don’t expect the subject to die soon.

I don’t think there’s one right answer to how you deploy your code, but I think there are many poorly thought out ones. I’m not here to necessarily make an argument for Chef as a deployment orchestrator. While there are people deploying with Chef at scale, I am not one of them, nor have I been. My work with Chef has been mainly in development with some provisioning and so I have a lot of theories, but that’s really all. What I do want to talk about is why I don’t like shell scripts for deployments or orchestration and what I want in a deployment system. This is the first part of at least a 2 on this topic.

Shell Scripts are like Gremlins. You start out with one adorably cute shell script. You commented it and it does one thing really well. It’s easy to read, everyone can use it. It’s awesome! Then you accidentally spill some water on it, or feed it late one night and omgwtf is happening!?

The Fixer: Someone else comes along and found an edge case your shell script doesn’t deal with. They add in some logic for the edge case, voila, problem solved.

The Refiner: Eventually someone realizes the logic for the edge case is not specific enough and is causing deployments to fail sometimes, so they refine the logic.

The Slippery Slope: After that, someone might decide it’s a good idea to automate stopping the apache server from sending traffic during deployments and decides to do it in the same script. Great idea! That’s such an awesome idea that everyone starts adding functionality to the tiny beautiful shell script, now no longer tiny nor beautiful.

OMG GREMLINS! Then you come back along and find all the extra stuff in your shell script that doesn’t belong there. You’re horrified. You might even be feeling a little bit violated (come on, we’ve all been there at least once). So what do you do? You pour some water on it. You break out the shell script into several functional bits. Now we have LOTS of gremlins instead of just one. Now you have a suite of scripts that are once again beautiful. But now the deployment is complex. You have a suite of bash perl ruby python scripts that also need a wiki page to describe intended flow, what to do if something doesn’t work and any edge cases that you haven’t gotten around to scripting yet.

The Exodus: Next up: You get a call from a buddy who is dying to have you come work for his Next Big Thing startup. So you quit your job, pack your bags and move to Silicon Valley Sitcom with ne’er a backwards glance, leaving a couple of forlorn junior sysadmins desperately reading wiki pages trying to figure out what to do with your shell scripts as a new application is launched that requires a bunch of new logic for deployment. These guys do the best they can and start tacking on if statements everywhere they need to in order to make the deployments go.

Subsistance Living: 6 months later, one of these guys leaves and the company hires 3 more guys with no understanding of the history of the deployment scripts. Sometimes they work, sometimes they don’t, people aren’t entirely sure why and just self correct by hand at the command line until things work(phew!). Everyone is afraid to touch them because they are fragile, the code connecting them is obscure and there are similar logic blocks found in several sections, sometimes commented out, sometimes used, but you’re not really sure whether it’s necessary. The original wiki page gets updated sometimes but not often and not usually by the person maintaining the scripts but by the people using them in the middle of the night.

And that’s why I hate shell scripts and think you should never use them for deployment scaffolding.

True story: my first venture with Chef involved deconstructing an organically grown Kickstart post that had been originally written for Red Hat 3 and subsequently updated for RH 4, 5 and 6. I was removing functionality from the postscript and rewriting it in Chef blocks when one of the admins came and yelled at me for omitting a block of host names from /etc/hosts and I was like, GUYS, those host names are for servers in a data center that was decommissioned when I started here 3 years ago.

You can tune in for the second half of this blog post, what I want in a decent deployment system, when it goes up next week on the Sysadvent blog. Woohoo!

Blue Ox
Location: Chicago and 38th St, South Minneapolis, MN
Espresso: Counter Culture
Hardware: Mazzer and La Marzocco
Coffee Art: Yes
Drink: cappuccino
Served in: 6oz ceramic cups
Visited: 2012 often

The Blue Ox coffee shop opened up about 3 blocks from my house in 2011. I love them. I love that they opened up in what I consider a marginalized area of Minneapolis, albeit one that people are working to revitalize. I love that I can walk 3 blocks to get wonderful artisan coffee. Sadly, I’m super lazy and often stay home and make my own substandard not-really-artisan coffee instead.

Blue Ox always has local art on the walls. Furnished with several second hand dinette tables and chairs, they also have a comfy futon and you’ll often find the windows open and the ceiling fans running instead of a/c.

Baristas here are always willing to talk the finer points of espresso and coffee with you. They have individually brewed coffee, pour over and they’ll make you an AeroPress Aerobie coffee if you ask. I’ve chatted with them and I know they calibrate the espresso grind and length of the pull at least every morning, sometimes more.

They’ve cycled through a few different brands of coffee and have recently started serving Counter Culture beans. When paired with the local milk used here, these produce a lovely, sweet espresso drink.

Warning, Blue Ox only serves whole milk. There is no skim or 2% here. They probably have soy, although I haven’t asked. They have locally baked pastries but no hot food. Today when I came in, they were also offering chips and hummus or salsa.

When you combine the laid back atmosphere and super yummy coffee combined with proximity to my house, Blue Ox pretty much wins my “favorite coffee shop in Minneapolis” award. That’s not to say there aren’t other places just as good and I’ll be getting to those in future posts.

Irving Farm Coffee Roasters
Location: New York, 14th St & 7th Ave, Uptownish
Espresso: Irving Farms
Machines: La Marzocco & Doge
Coffee Art: yes
WiFi: Unknown
Served in: 8oz ceramic cup
Drink: skim cap
Visited: 9/2/12

Verdict: Noms. Seems like a lot of places in NYC are serving Intelligentsia coffee. While I don’t object necessarily, it’s nice to see a coffee shop using locally roasted or even better, roasting its own. My drink was mild and reminded me a bit of Stumptown with it’s citric tinge. They heated the milk a bit more than some places but it was still drinkable and I like my milk with a smidge of heat so the drink doesn’t cool off too soon. I’m guessing they heat to 145-150.

There are four small tables and a wall bench at this location, so the lounge factor isn’t high. However, the was an elderly man enjoying his newspaper when I got here and so I’m guessing foot traffic isn’t too obnoxious. The shop is small and unassuming from the outside and I wouldn’t have noticed except I got off the subway there and was looking for friends.

Fyi, no public bathroom

Final Word: Noms.

Financier Coffee
Location: New York, Cedar & William, Downtown, Financial District
Espresso: Financier (roasted in Park Slope, Brooklyn)
Machine: La Marzocco & Mazzer
Coffee Art: No
WiFi: kind of (“It’s not working right now”)
Served in: 8oz ceramic cups after I asked twice for “here”
Drink: skim cappuccino
Visited: 9/2/2012

This drink had the potential to taste good. Unfortunately the barista had no concept of micro foam or craftsmanship. The taste is a little harsh but I can see that it could taste awesome if prepared correctly.

Financier prepares and roasts their own beans and so I may come back again and check as it has the advantage of being close to where I’m staying and the subway stations are totally horked with construction this weekend. (n.b. I didn’t. I walked to Kaffe 1668 in search of a sure thing).

The pasty case looks amazing but I won’t be able to report on that. The store has long bar seating at the window plus some table in the back. While I prefer the dark hipster coffee shops, this is still ok. The music is kind of pop/r&b dreadful. There’s no public bathroom either btw.

Also, beware, a small cappuccino is really only one shot. They divide the shots. When I saw that, I asked for the second. This annoyed me.

Pros: cap tastes good despite mangled prep, bright and airy if you like that, PASTRIES!
Cons: Inexperienced baristas, Split shots (who are you, Starbucks?), crappy wifi

Final word: Beats Starbucks and the nearby Blue Spoon

tl;dr: As a sysadmin, whenever someone tweets something snarky about sysadmins, I feel a little put down. As a result, I try to think twice about snarky stuff I might tweet about devs or other teams. And so, I think sysadmins posting snarky captioned images about devs and then adding the devops hashtag is mean and inappropriate.

I’ve seen this captioned image circulating all day with the devops hashtag on Twitter. For the record, I don’t think this picture is funny. In the right context, it might be funny. If it were, say, a slide in a deck pointing to how things are or used to be without devops, I’d laugh. My issue is seeing tweeted with the devops hashtag and having it billed as funny.

Now I love me some snark. But I think it’s too easy to be snarky and mean on Twitter, because there isn’t a lot of consequence and we tend to live within a circle of our peers who often think alike. The snark factor often goes up around conference time when many of us are congregating in once place and competing to sound smart and funny in 140 characters or less. I’ve commented before that I think people are too often mean on Twitter in the name of being witty or complaining about speakers without ever thinking about what that speaker might feel if he sees that tweet later.

So why does this caption bother me? I have a long history of unfiltered snark and smart ass remarks. Consulting since 2006 taught me a lot about active filtering and my introduction to DevOps caused me to implement additional filtering for the sake of my emotional attachment to an ideal.

As someone who’s signed on as a big fan of DevOps culture, I spend a lot of patrolling my cynicism and preventing disparaging remarks from escaping the filter. I think when you espouse certain ideals, you’re responsible all the time for representing them.

I sometimes see cynical, catty remarks about sysadmins or ops from people who can only have come from a long life of development and, even though I haven’t technically done ops in 7 years, I’ve been sysadminly all my long life and I am affronted every time by those remarks. Recently I heard a professional dev say to someone in a beginners programming workshop, “If you want to understand/master the install of the programming tools, you’re probably better off as a sysadmin not a programmer.” The guy who said is a super nice guy, but that remark still got a side-eye from me.

If DevOps is a movement that promotes collaboration, communication, respect and friendship between functional teams, I don’t think a sincere proponent would post this kind of caption. While it makes some folks feel validated, it’s just fueling the fire that walls people off from each other. I don’t think we can all get along when folks are throwing up cynical remarks perpetuating stereotypes of bad development practices.

While there are many successful DevOps teams out there, there are far more silos in transition or bitter adversaries who haven’t yet heard of DevOps. I can’t believe that only some developers or sysadmins are capable of transcending the usual barriers; I have to believe that any and all are able to transcend them or what’s the point? But it’s the more entrenched and cynical cases that will be the most difficult to move ideologically and emotionally and I feel that humor like this can only alienate.

I don’t want to pick on anyone specifically because I see this kind of humor from all disciplines all the time and it makes me increasingly uncomfortable each time. If we’re actually going to all be in this together, we need to jump in with both feet. So I thought I would speak up for a moment and say something. If a thing can only be funny by being rude about someone else, maybe it’s not actually funny at all.