This blog post is the post is an edited draft of something I wrote up for one of my work’s blogs.

This past month was the 11th annual Southern Califoria Linux Expo. It had been many years since I last went. For this visit, SCaLE has been extended to support other linux and FLOSS related events. On the Friday, Reliam helped sponsor Devops Day LA which ties directly to what we at Reliam strive for by communicating with our clients. On Saturday and Sunday, the classic SCaLE presentations occured. Also on the weekend, the exhibition hall was open with many organizantions and companies. With so many different things to do, I had to miss out on stuff that I wish I could have had time for.

I had to resort to SCaLE’s Wikipedia page to find that I last went to SCaLE 3x. My most vivid memory was attending the Open Source Telephony Using Asteristk and VOIP by Daryll Strauss presentation. I remember being fasinated with the concept of a free PBX, since at the time I worked for The School of Arts & Enterprise and we had gone with two Nortel PBXs. We would have to call the main line for the other campus and hit the room extension, just to reach a teacher at the other site. When I looked at photos from the event, I also found the Flightgear mock up which I was fasinated.

There were several special events that where held because so many like minded individuals where at the same place. The MySQL Community Day is one that I took advantage of. For many years, I’ve helped Reliam’s customers tune their databases so they get some of the best performance possible. Sheeri Cabral of Mozilla gave a talk on Are You Getting the Best Out of Your MySQL Indexes?. She covered the basics, including use of multi-column indexing which I have used many times already. New to me was the use of ENUM, because in the past used a SET where the ENUM will be a better description of what I mean. I also learned that mysql uses statistics on tables, and may not use indexes because it thinks that it will be faster to just read directly from the table instead of reading from an index and bouncing into the table for each match. Since Reliam has been using puppet to help manage the many hosts we run for our customers, I also attended a Puppet Camp LA talk on a method of integrating puppet and ldap. This talk covered how people can optimize puppet control of systems in a massive environment. LDAP allows admins to nest computers into organizational units, so you can have the California group, the New York group, and the Tokyo group. Within each group, you could have a developer group that needs certain compilers and other software installed to get their job done. There could also be a group for site operations, and they use certain software to track conference room schedules, and peoples vacation and actual sick time. As the objects representing peoples computers got moved into a group, puppet would be using ldap to make sure that person got the right software. This is a contrived example, but there are cases where it has been chosen to run a lean install, with just the required software for each person to do their job. Reliam alumni Brandon Burton also gave a talk on how Mozilla uses puppet. He covered their best practices of using revision control, along with verification on code before checkin. Mozilla has built puppetctl, a tool to be their way of disabling puppet for a certain time period.

I made it in time to attend the keynote presentation for DevOps Day covering Promise Theory. John Willis talked about while tools like puppet, chef and others go for determininistic declarations of how a server or system should be, such statements may become harder to enforce as systems scale. In a more dynamic environment, Promise Theory would allow new pieces could get added, and gain trust amongst its peers. Since this was an abstract talk, much of it went in my ears without true comprehention. But I will remember this talk as I encounter larger scale systems. Later in the afternoon, I made it back to the DevOps talks with Christopher Webber on Test Driven Development for Ops. Use of tests can allow you to make sure stupid mistakes don’t get put in by doing syntax checking. Linting allows the code to be reviewed for best practices, and catch things that are missing such as documentation. These were the easy steps in testing to do. Writing unit tests is a step up, since you have to write the test and know how to catch the corner/edge cases to have it work for the best benifet. For sysadmins, we are used to “testing” in production by constantly monitoring our systems, and verifying things are as we expect. Using configuration management tools to ensure that monitor checks get setup simplfies the steps needed to bring new systems up to speed, or changed settings are confirmed. Integration of monitoring is something I can’t wait to get implemented at Reliam. Brandon gave his second presentation with one of his co-workers Chris Turra on how Mozilla scales the web properties they run. It was great to see the details of how other companies run at scale, both in technology and people. They have the “Button” that is used for deploying site updates via a tool they call chief. The next day the website for the Firefox Phone got updated, and Chris commented that in the past his team would have be the ones deploying the update. Instead it was just a click away. And to defend Reliam, we have made these tools for our customers too. Just so you don’t think that this is something new to me.

Saturday’s keynote by Matthew Garrett on The Secure Boot Journey was delivered to a packed room. He covered how when Microsoft initially released the hardware requirements for Windows 8 laptops, Linux may have gotten locked out from those systems. Windows 8 required that the hardware verify that what it was booting had been verified to be authorized, and Microsoft said that their keys had to be trusted if Windows 8 was to be sold on it. Since there was no requirement to be able to add other keys, you can be sure the manufactures would have skipped that ability, since the majority of people would not care. Matthew recounts how he was able to create such an outcry that Microsoft added the requirement that there be an interface to manage the keys trusted for booting on this new hardware. He also covered the tools that where written to assist in users booting and installing Linux by having these tools signed by the Microsoft key. The next talk I attended was on how Facebook approached scalling the management of their servers. Phil Dibowitz covered the reasons why Facebook went with chef, such as the ability to make major changes to how the tool worked. Facebook reduced the reported data to the bare minimum, so it was a few kilobytes, instead of megabytes. He also covered that Facebook used both paid chef, and open source chef. With the paid version of chef, Facebook got early access to the 2012 port, which lead to drastic improvements of the resource usage of chef. In the afternoon, I geeked out with two talks on IPv6. The first one by Owen DeLong of Hurricane Electric (a major IPv6 tunnel broker) covered the growth of the internet, and how IPv4 has run out at the higher levels. Next, the vollunteer who acted as head geek of the network at SCaLE gave a step by step talk on how to get your own linux router provide IPv6 addresses at home, even if your ISP doesn’t give you any. I’ve had a tunnel setup in the past, but since taken it down when I changed my home ISP. These talks reminded me that I should get that up again. The final presentation I was able to stick around for was facinating to me, and at just my level. The Linux Containers presentation by Jérôme Petazzoni was about a group of technologies that help limit one group of programs from interfering with another group. Which is a great method to use when setting up a shared hosting environment. Configuring a new environtment for the next user takes mear minutes, and starting that environment up is typically seconds. This allows more environments to be running on one piece of hardware, and even more enviroments to be configured. Economies of scale at its finest.

Sunday was time for me to wind down. I went to Jordan Sissel’s talk on logstash which helped wake me up with humor. He covered how sysadmin’s don’t really like getting woken up in the middle of their sleep, just because a disk is getting full. A natural instinct is to delete all the logs. This is great for space, but when you need the information, causes you to curse your former self. Another problem described is level 1 tech support people needing access to logs, but security requiring restricted access. Tools are built to serve this purpose, but are often slow and difficult to use. Logstash to the rescue! With it, tech support personal can drill down to a required timeframe, search for a specific user. All in a single bound. And its pretty darn quick. I also went to a talk on automating installation of FreeBSD to expand my horizons, but I probably won’t be making use of it.

In my spare time, I strolled the exhibition hall. LinuxAstronomy was back from last year, showing even more devices that are going into the sensor platform they plan to launch via a balloon. Awesome. LinuxMCE would allow me to monitor and control my house. Its a slick looking thing compared to MisterHouse from a decade ago. Mozilla was there, showing of prototypes of FirefoxOS. I got to geek out, by working with Chris Turra in flashing one of the phones to a more recent build. ODROID Hard Kernel looked as a cool way to make your own systems, reminding me of the RaspberryPi. OpenBSD was there with swag, and classic copies of the OS. Ticketmaster deserves mention because they had Krispy Kreme doughnuts, and cat pictures. I personally don’t know what started the fascination with cat pictures, but them putting them up there did make me chuckle. I got to talk with one of the guys from AMD’s SeaMicro, which packs a lot of processor power into a “tiny” space. They seem to be a nice fit for virtualization, or Linux containers. Because of DevOps Day LA, many of the configuration management platforms also had booths.

Unfortunetly, I missed the talk on Linux Performance Analysis and Tools, and I’ll just have to watch a talk he gave in 2012. It might of been handy to go to Redhat’s talk on making packages. While the Jenkins talk was below me at the beginner level, it does deserve mention because its just darn cool, and I plan on setting one up again soon. If I ever find the time to play with hardware projects, I bet the talk on the Raspberry Pi would have been handy. There where many talks on filesystems that help people scale: OrangeFS, GlusterFS, Ceph, ZFS, FreeNAS Plugins. There where also BoF sessions that I wish I made it to: spam controls, functional programming, AWS, and coreboot. Now I remember why I loved college. I had so much time on my hands, I could get into and learn so many different things.