cwebber.net

Random thoughts about life and systems administration

Cooking Up Something New

| Comments

After an amazing adventure with the great people at Demand Media, I on my way to cook up something new. If the cheesy pun didn’t give it away, I will be joining the community team at Chef on April 28th. While I will miss the amazing folks at Demand, I am super excited about the new adventures ahead.

My role at Chef will be working as a Community Software Engineer working to help make the experience of using and contributing to community cookbooks delightful. I am looking forward to working with all the amazing people in our community.

I am looking forward to the new adventures with Chef and the ability to focus on making the lives of others better as I work with people to make it easier to solve problems with cookbooks. The ability to focus on shared and open-source infrastructure code is something I have wanted to do for a long time. This will give me the chance to help encapsulate the years of yak shaving into code that will hopefully let others do more awesome.

Just because I want to explicitly state it, I still have lots of love for Puppet and the folks at Puppet Labs. I am excited to continue to see where the Puppet ecosystem goes. While I for sure have some definite reasons why I like Chef, Puppet is still an amazing tool.

If you are going to be at #ChefConf I will be around all day on Wednesday so hit me up.

A Year Remote

| Comments

In early December of last year, I hired on at Demand Media to be an 80% remote employee. While, in general, the culture at Demand around being in the office is strong, there are actually a fair number of remote employees at Demand. For me personally, I live about 90 miles from our office in Santa Monica and usually come in on Tuesdays.

The journey has definitely had its ups and downs but it has been amazingly positive and enjoyable. When I first considered the idea of working for Demand, I reached out to a few friends I knew that were also remote. The interesting thing about this is that they all advised against me taking the position because not only was it partially remote, but because I was going to be “the remote guy” on a team that was in the office. After having spent a year in that world, I can definitely see why they had concerns and how quickly those situations could turn sour.

Because I am a little on the crazy side, I took the job. It was a chance to get some experience working remote and the team seemed like a good fit. Needless to say, I made the right decision. I feel like I have been super successful working at Demand and owe a huge part of that to the team I work with. While tools help a lot, it is the culture of my team and the other teams I work with that allow me to be effective.

My team, the Media Infrastructure group, takes an active role in making sure I am not left out of things. They always make sure the conference calls are setup and I am there via video chat when possible. Not only that, there is an active effort to make sure conversations are had over IM so that I can be a part of them and am not left out. On the occasions where I am not able to be a part of whatever was going on in the office, they make a concerted effort to follow up with me and make sure I am aware of what is going on.

In addition, the teams I interact with have adapted to the fact that I am remote and do a great job keeping me in the loop. There are plenty of screenshots of the teams being on a video chat with me as we launched a new site, or you can walk into one of the weekly lunch meetings to see my gigantic head on the video display. Most of the developers I work with have made a point to make sure they have Jabber up so I can reach out to them when there are issues.

The impact of tools in this space cannot be understated. The conference bridges, Jabber and the Vidyo system are what allow all of this to be possible. I want to take a second to thank the amazing teams in my IT group that made the Vidyo system happen and keep things like Jabber up and running. While tools like Google Hangouts and other instant messaging systems exist, the tools that IT provide work better and have allowed for a much more seamless environment. In particular, when we were in a previous office location, I used to dread meetings where I had to call in on the conference phone. Half the time I couldn’t hear what was being said and the other half the time I couldn’t chime in when I needed to. The Vidyo systems that are now in the big conference rooms have made that problem go away almost completely, and I am now one of the loudest people in the room.

It is interesting to look at some of the benefits of being remote. Most people focus on what the remote employee gets out of it, but I think the employer gets the most benefit, especially with the industry we are in. As a result of being remote, my team regularly practices the same techniques we use during incident response. We all know how to get on the conference bridges and are comfortable working and IMing at the same time. (Ok so that is a bit of a stretch… I suck at multi-tasking.) Even more interesting, when I get a call in the middle of the night about a problem, I generally walk into my normal work environment so there is less of a need to adapt to the new situation. I turn on the display I have setup as an information radiator that displays dashboards and logs and am able to quickly start getting a feel for what is wrong, even if I still have to setup my laptop.

Another interesting thing that has happened on a few occasions is that because I am in a different location, I am not affected by local issues. There was a situation recently where the power had failed in the office and because I was remote and one of my teammates was still at home we were able to address issues that couldn’t be handled by the rest of our team in the office. It means that I am occasionally able to route around problems or see things differently because I am not in the office.

I would be doing a disservice if I didn’t talk a little about the challenges that I have faced over the last year. Most of the problems I have faced are not ones that can be overcome by my team or the company, they are the reason that remote work isn’t for everyone.

To start, 90 miles is a long way. Without stopping and without traffic I can do the drive in approximately 1hr and 20 mins. For those that have ever been to LA, the “without traffic” part is, by and large, hard to achieve. To help with that I have adjusted my schedule a bit and try to be out of the house by 4:00 am on days I am in the office, resulting in about 1hr, 30 minute drive to the office. And, because traffic in the evening is bad, I don’t head for home any earlier than 6:00 pm and it is usually closer to 7:00 pm by the time I leave resulting in about a 2hr drive home. While I do come in twice a week on a very rare occasion, it is obvious by Friday that I have done so by the toll it takes on my body.

To add to the theme of long days, I tend to spend a lot of time working. This is partially because I just love what I do so much and partially because I really don’t ever leave work. I try to be good about getting up from the computer no later than 6:00 pm, but that doesn’t always happen and it isn’t out of the ordinary for me to sit down to dinner with the family and then be right back at it. My wife had been amazingly understanding and has done a great job in helping me to set those boundaries and not get sucked too far into the work I enjoy so much.

While my team does a great job at keeping me included, there are still plenty of things I miss out on because I am remote. I don’t get to participate in a lot of the things like the Monthly Birthday and Anniversary goodies most months, and I am almost never in the office for lunch on Friday.

Additionally, there have been a few occasions here and there where there has been a major incident or something going on and lots of brainstorming and discussion was going on in the office and I sat at my computer wondering what was happening because everyone had stopped responding. This has more or less gone away because of Vidyo, but it is a horrible feeling when you are sitting at your computer, the site is down or heavily impacted and you feel helpless, not knowing what to do next.

All and all, it has been an amazing experience. My team, the teams I interact with and everyone at Demand has always been supportive. While the drive is a bit crazy and the day is long, I love my one day a week in the office and am looking forward to the next few years at Demand.

Puppetbestpractices.com Hurdle Number One

| Comments

As I start to get things in order for puppetbestpractices.com, I am stuck on a relatively big decision, how do I setup the site? I basically see this two options:

  1. I setup Wordpress and grab a pretty theme and some plugins.
  2. I take a bit more time and setup the site using a static page generator like Jekyll.

The first aspect that makes the decision hard is collaboration. If I use Jekyll or some other static site generator, I can throw the page up on GitHub and anyone that wants to contribute can just create a pull request. This makes contributions super simple. Wordpress has the opposite problem. The only way I can accept contributions is to get emails and comments, or to hand out accounts. Neither of of those options are appealing.

From there the question of design comes into play. I almost titled this post “Design Eye for the SysAdmin Guy” because I suck so hard at it. IMHO, Wordpress is the clear winner in this battle. It has tons of free and easy to use themes etc. Jekyll or maybe even Octopress have a much more limited selection of themes and by and large require me to actually know what I am doing, which, as I mentioned before is not the case. If the themes from something like Themeforest made sense from a license standpoint, I would gladly pay the roughly $15 bucks for a starting point.

So yeah, this has been my frustration since PuppetConf, how to build the site. I guess this is just a reminder that anything worth doing is going to be hard.

That Sinking Feeling

| Comments

On Friday of PuppetConf I jumped on a video call with my coworker back at the office to get news that the president of engineering had asked for me to join the team supporting the new site that we had recently acquired. Everything about this was super exciting. It brought with it the chance to finally do something production in AWS and the potential to be a little more focused on a single property.

While I have generally kept up with how AWS works and have dabbled a bit here and there, I have never done anything that would constitute production. I had heard about the occasional shit show that is EBS and was able to grok how all the components came together. In my excitement, I spent the weekend using Cloud Formation to spin up simple HA setups and then tying in New Relic for monitoring.

Then Monday hit. Like a ton of bricks. So many unanswered questions and so much new infrastructure to learn. More technologies to contend with that I have never gotten super good at like MySQL and MongoDB. The overwhelmingness and uncertainty of things like team makeup, on-call and such added to the technical things I had in front of me.

The rest of the week was a blur. I made a few changes here and there but in general avoided doing so. Even basic changes like setting up nscd seemed to cause problems. I really started questioning if I was in over my head.

This all got me thinking about my own fears and apprehension about things. First of all, I realized the emotions I go through with these kinds of new technical changes are standard for me. I tend to be overly cautious, hesitant to make changes, and easily spooked when the metrics don’t look right.

But why? Why do these things have such an affect on me? In many ways experience is to blame. Not just a lack of experience but the bad experiences along the way. I know what the command or action is “supposed” to do but I have come to question, is that what is actually going to happen? While I have faith in reading the docs, you just don’t know until you have run the command a few times what to expect.

While I understand security groups, etc. there is always the question of environment separation. I know that as long as I do the right things, nothing bad is going to happen and the world is going to be separate. That said my API keys have just as much power to kill prod instances as they do dev instances. And, as I have already discussed, I have a hard time trusting the tools.

Finally, the known unknowns kill me. The more I learn about MySQL the more I understand how little I actually know. While I will learn and become more familiar with its inner workings, I continue to be a bit hesitant to make large changes because, almost without fail, the database is the thing that breaks everything.

So what now? How do I work through these inner demons? I guess the simplest answer is to experiment and move forward. To remind myself that this is no different than any of the other times I have wadded knee deep into new infrastructure. To setup mock production environments and get a feel for what is actually going on. To do some reading and reach out to my amazing friends for help and advice.

That said, taking the weekend to step away and reflect has helped tremendously. So here is to an amazing new week ahead of me.

PuppetConf 2013

| Comments

PuppetConf 2013 was an amazing experience. I could spend tons of time talking about all the things that went on but here are a few of the things that really stood out to me.

Developer Days and Open Spaces

The developer day stuff was great. While I didn’t get a chance to sit down and hack on much myself, it was great to be able to catch up with people in the community and talk about the things that are coming down the pipeline and need to be worked on.

The real win of Wednesday was the open space sessions that Michael Stanke put on. The discussions around testing and best practices/patterns were amazing. I feel like I got a ton out of just sitting and talking to others in the community. With that said, I am going to officially put it out there that I bought puppetbestpractices.com and my goal is to start collecting talks, slide decks and blog posts about patterns and module construction in one place. I haven’t decided how I am going to set it up yet, but I will make a decision soon.

Being a Speaker

This is probably the biggest speaking opportunity I have ever had. Puppet Labs not only did a great job putting on the conference but I felt super well taken care of as a speaker.

As I walked the halls I got a ton of positive response about my presentation. It was great to run into people and have them tell me, “great talk,” or ask questions about testing. I am looking forward to digging even deeper into testing and better understanding how rspec-system is going to become part of my workflow.

The Sessions

While I didn’t make it to as many sessions as I would have liked, the ones I did attend were great. I was impressed by how much knowledge each session was jam packed with. I can’t wait to go through and listen to all of the sessions as they are released.

In the meantime, the slides can be found at http://www.slideshare.net/PuppetLabs/puppetconf-at-a-glance.

The Hallway Track

Wow. I can’t even begin to list the tremendous number of amazing people I met while at PuppetConf. Whether it was getting a chance to catch up with @puppetmasterd himself or seeing old friends like @botchagalupe and @mitchellh the people are what really took the conference to the next level for me. Sitting down for, what must have been more than 2 hours, with @davemangot talking DevOps and systems architecture was one of the most enjoyable things I have done in a long time.

Lets just say, I am looking forward to PuppetConf 2014.

Taking Inventory

| Comments

Every 6-9 months I find myself taking inventory of life in general. To be honest, this is usually a result of work getting busy for a sustained period of time. I find that it is important to take the time to go through this mental exercise as it is usually an indicator something needs to change about the way I am currently approaching life.

It is amazing how much this thought process actually reveals. Even the frequency in which I find myself walking through it gives hints about changes in life. Over the last few years, I have learned that if I am mentally taking inventory every month, it is probably time to change jobs.

This morning, as I started to take inventory on my morning walk, I realized two things about the process:

  • I want to be open about where things stand as an attempt to put myself at ease.
  • This is probably useful for others to see and at least think about.

So here are a few of the things that I ponder when taking inventory.

Family

Family is at the core of everything. As a husband and father of two, it is important to make sure I am taking the time for them and fulfilling their needs. The following are some of the things I think about.

Are my wife and I connected?

My wife has always talked about this idea of being connected. When we are connected, we anticipate each others needs, we are helping to carry the emotional load for each other and well, connected.

Am I spending enough time with my girls?

I don’t know that I will ever be able to answer yes to this question. I always find myself thinking I should be doing more. That is especially true during the summer. For my almost 4 year old, being outside and dance class are our thing. Mommy goes along sometimes, but, by and large, those are Daddy things. So when the normal high is 90+ degrees outside, the park becomes less of an option. To add to it, dance closes during the summer because many of the families go on vacation, etc.

This means the Fall is always one of my favorite times of year. Lots of trips to the park, dance is back in full swing and it is time for high school football. Because my wife is a high school teacher, we make a point to get out to most home football games and a few away games each year. It has always been fun to take my oldest out and I can’t wait to introduce my youngest.

Am I making time for my parents (and in-laws)?

My in-laws have always been super active in my married life with my mother-in-law watching my daughters during the day. We do spend a ton of time with them so things are usually pretty good. My parents on the other hand are always super busy. And not only are they super busy, their schedule is offset of mine by about 2-3 hours. They live in the same area but I am usually heading to bed as they are finishing with dinner. I feel like we are seeing them fairly often and with the normal Fall stuff coming, I think we will be good.

Work/Career

So work and career stuff is the other big area I think about. These questions are generally a bit more pointed and tend to help change my thoughts about long term things. In general, this area of my life seems to impact family and my happiness the most. It is also important to point out that there will be days when the answer to these questions is not positive, the key is to look at the general case over the last few weeks.

Am I having fun or do I enjoy going into work?

So simple, yet so powerful. When I get up in the morning do I look forward to the day ahead of me? Interestingly enough, this is almost a meta-question for me as it is really dictated by the other things I think about.

Is my relationship with my boss good?

I have been in a place where this was not true for a sustained period of time. It is no fun. It was during that time that I learned that if my regular answer to this is no, it should be a resume generating event.

That said, I love my boss. It has been amazing over the last eight months to have someone that asks hard questions and actively works to help me get over my own insecurities. It is awesome to work for someone who is honest and sees the bigger picture.

Is my relationship with my teammates good?

The team is what makes us stronger. It is the people you can lean on in times of distress and people to lift you up during times of good. Just like the relationship with your boss, your team makes a huge impact on overall happiness.

So, I’m going to be a broken record here, but I love my team. Such a great group. As the one remote guy, they make sure I don’t feel left out and they make the 3-6 hr round trip drive to Santa Monica each week worth it.

Is my work interesting?

I hate being bored. If my work is boring it means a few things:

  • I am probably not having fun
  • I am probably not learning much
  • I am probably not developing as an engineer the way I need to be

So decidedly, yes, my work is interesting. Right now it seems to be more a study of process and history but there are some neat technical challenges on my plate too. It is always nice to have a good balance.

Is my workload going to kill me?

It is no secret to my friends and family that I am a workaholic. While a great trait in some regards, it is something that takes its toll on occasion. The idea of only working 8-5 and not thinking about work outside of that time is crazy talk as far as I am concerned. I love the problem sets and the challenges so working a bit more is no big deal.

It is when I try to keep up with an impossible workload that I start to step back from work and stop enjoying it. Even worse, I do it to myself, taking on extra work that I don’t need to. My boss and one of my teammates have been amazing at helping me to back off and not let work consume my life. I am not sure that a lot of these questions would be answered positively if it were not for them helping me to keep things in check.

Am I doing enough career development?

As far as I am concerned, the technical aspects of career development, by and large come from work. It is important to make sure I am doing things at work that are in-demand and make sense for my career but as long as I am working in the right position that is generally not an issue.

The real place I think career development happens is at conferences, on this blog and, in many mays, on twitter. Am I submitting enough talks? Am I going to enough conferences? Am I blogging enough? These things matter because people matter. The social interactions and the friends in industry are really where it is at.

Now that I have had a chance to decompress and am feeling a bit better about the state of things, it is time to get back to work.

Why I Suck at Open Source

| Comments

I love the idea behind open source and can’t imagine my career without the amazing contributions of so many people, but can’t help but feel that I don’t pull my own weight. I see amazing people in Ops like John Vincent (@lusis) and James Turnbull (@kartar) that are constantly shipping code and am reminded that I am not doing the same.

No matter how much I tell myself I need to push more code out into the world, I am regularly reminded that not only am I not great at writing good code, I don’t have the prerequisite knowledge about patterns and structure. To add insult to injury, the problems I find the most interesting are rarely general problems. The problems I find myself working on in my spare time are those internal issues that we can’t justify spending time on at the moment because there is so much else going on. So even if I am writing code, it either can’t be open sourced because it isn’t general enough, or I am too embarrassed to ask management to let me open source it.

When Packer was released it was a blast to have some time to go play with it. The bummer was that all I seemed able to do was report bugs and test to make sure they had been fixed. Every time I dug into the source code I found it difficult to make useful modifications to help solve problems. While I am usually able to read and help do basic code debug, I rarely find myself in a place where I have enough knowledge to make changes and contribute back.

While I know that evangelizing software and reporting bugs is useful and very much a part of open source, it is a bummer that I never seem to be able to get a Pull Request right.

Packer - First Thoughts

| Comments

As I spent the weekend eagerly awaiting the arrival of my 2nd daughter and avoiding the heat, I got a chance to play with Packer quite a bit. It has been a blast getting it bootstrapped and spinning up VMs.

While there are still a few bugs to be worked out, packer seems very stable and well thought out. Everything from Ctrl-C doing what you would want by cleaning up after itself, to insightful error messages it is top notch software. A huge shout out goes to @patrickdebois whose veewee project made it trivial to get going with Packer. The veewee-to-packer gem, written by @mitchellh made it a snap to take my veewee definitions and turn them into Packer templates.

Why Packer?

For anyone that has been a long time user of Vagrant the first question that probably comes to mind is, “Why should I switch to Packer from veewee?” The simple answer is that Packer already does 90% of what veewee does, but it can build in parallel. Plus, Packer will handle the automagical building of VMware Vagrant boxes. In addition to those things, For me, the biggest reasons to use Packer are the lack of a need for a ruby environment, and the longer term extensibility that the foundation has already been laid for.

On the lack of a need for a ruby environment… Of all the things I have struggled with as an Ops guy is getting other members of my team to jump through the hoops that is getting a modern ruby environment setup. Between the version crap, bundler, etc. removing the need for ruby will be a huge win for me as I evangelize the use of Vagrant.

My Experience So Far

For being software that was at version 0.1.0 when I installed it, Packer is amazingly well documented. The community is already developing quickly and fixes for the bugs are rolling in super fast. I have hit a few problems here and there but almost all of them have already been addressed in version 0.1.3. I just want to say a huge, thank you, to @smerrill for all the work he done as it has made my life tremendously easier.

Getting Started

So then, where to start? If you are using veewee, I would suggest starting with the veewee-to-packer gem. If you haven’t played much with veewee and just want to start with Packer, @smerril has put up a great starter template for CentOS 6.4 at https://github.com/smerrill/packer-templates. My goal is to get some templates up in the next week or so depending on how things go.

The Future

Personally, I will be paying a lot of attention to Packer as things move forward. It make it a lot easier to duplicate what our corporate kickstart servers are doing, and will make creation of Vagrant boxes much easier and more accessible than it has been in the past.

Puppet Design Patterns

| Comments

Programming and software engineering have never been areas where I would consider myself strong. Quite frankly, it is the exact opposite. Digging into code and troubleshooting is fine, but writing code from scratch is still something I struggle with. As I have jumped onto a team with a dev guy turned ops guy, discussions about how we write puppet code have come up.

Starting with a discussion about where ancillary services get instantiated (I’ll cover that in a separate blog post), I started looking for more details about design patterns in puppet. The more I looked and asked questions, the more I realized there is not much out there. At this point, the only real reference I have found was Simple Puppet Module Structure Redux by R.I. Pienaar. This article does a good job covering a self contained module that sets up a self contained service, but it really lacks some of the more advanced use cases.

My plan at this point is to start the conversation. Below is a list of issues that I think could be well served by design patterns and discussions of anti-patterns. As time goes by my goal is to start writing posts that help to better shed light on what I see as the issue and ways I understand that we can solve it. My hope is that this will start the discussion, because I am definitely not the one to suggest the right way of doing things.

Design Pattern Areas

  • Ancillary Service Configuration (Post is about 50% done)
  • Distinguishing between types of modules. (i.e. Type and Provider, Building Blocks, Roles, etc.)
  • Interaction with environments

Just incase it wasn’t clear, I am lost when it comes time to all of this stuff. My hope is that those that have been there and done that can help shed some light on the right ways to move forward.

Update

So the first response I got to this post in IRC was to take a look at a Designing Puppet – Roles and Profiles by Craig Dunn. This is a great starting point for a lot of the thoughts I have had.

Puppet on Windows: First Thoughts

| Comments

One of the things I didn’t expect when changing jobs was the amount of Windows Infrastructure I would be responsible for. While not expected, it will be an interesting challenge and is still very much a WebOps and not an enterprise Windows deployment. As we move forward with this infrastructure, there is a huge desire to puppetize it and manage it in a programatic way.

Setting Up a Dev Environment

As a long time Vagrant user and abuser, I started with attempting to setup a base box using Veewee and Vagrant. This was met with disaster. None of the templates I tried worked and, really, vagrant isn’t designed to handle Windows. But failure of this kind always seems to open new doors.

So, I cheated a bit. The veewee template left behind a floppy image with all the of the autounattend.xml stuff rolled inside. I created a new VM in VirtualBox, added a floppy controller, the virtual floppy disk and the Windows 2008 R2 iso. A few minutes later, I had a fully installed Windows 2008 R2 instance.

From there, I shared a folder with my Puppet manifests and modules, installed Puppet and the VirtualBox Guest Additions. At this point, I took a snapshot so I can easily revert back to where I started.

Out with the Package

So, what is the first thing you want to do when setting up a web server? Install the web server software, right? On a Linux box this looks something like:

1
2
3
4
5
6
7
8
package {"httpd":
  ensure => installed,
}

service {"httpd":
  ensure  => running,
  require => Package['httpd'],
}

Simple enough, right? So I set out to figure out how to install IIS. The first thing I looked for was a built-in type to handle setting up server roles and features. Nada. So off to the Forge I went. I found an amazing IIS module written by Simon Dean. The only problem with it, it doesn’t install IIS.

Enter DISM

DISM or Deployment Image Servicing and Management is where the heavy lifting gets done. is the command line tool that allows you to install roles and features, just like you would from the Server Manager GUI. I owe a huge thanks to Mark Bools for helping shine the light on DISM with a blog post. To get started with dism in puppet, Puppet Labs has published the dism module on the forge that has a type and provider to work with. To get a list of things that can be installed with dism you can run, . From there, you can use the dism type to install the needed packages. This looks something like:

1
2
3
4
5
6
7
8
dism { 'IIS-WebServerRole':
  ensure => present,
}

dism { 'IIS-WebServer':
  ensure  => present,
  require => Dism['IIS-WebServerRole'],
}

General Observations

In general, things worked as expected. Puppet does not seem to care whether the manifests have Windows or UNIX line endings which is very nice. The only thing I have found so far that didn’t, “just work,” was the module tool, otherwise things translate just fine. The big issues left to deal with are not ones that are easily solved by puppet. There are reboots that need to take place as part of basic installations that really don’t lend themselves well to configuration management as a whole.