The security industry is slowly realizing what the developer community knew for years – collaboration is the key to and likely the future of innovation.
Even if you’re not a developer, you’ve certainly heard about GitHub, the world’s largest code hosting service. With over eight million people using it to build things by collaborating with others, GitHub has had a noticeable impact on the way software is developed. It even makes it easy for non-developers to contribute by opening issues for bugs and features.
As Daniel Maldonado, Lead Developer of WPHardening noted, “generating a GitHub project means to leave an open door to the entire community of programmers and enable an initial idea to grow. Essentially, GitHub provides the social side of the community.”
Like in many other modern organizations, GitHub employees are dispersed across the US and the rest of the world. Ohio-based Scott Roberts, an expert on network security monitoring, incident response, and threat intelligence, is a Bad Guy Catcher for GitHub and spends his days working with the other members of the security team, who live in Colorado, Illinois, and Louisiana.
He remembers a situation many have found themselves in: “I found myself looking for a better tool to do OS X triage for incident response. I found a project by a GitHub user @jipegit. His project OSXAuditor fit my needs perfectly and I started submitting Pull Requests. Months later we sat down for dinner when I was visiting Paris. That ability to work with someone you don’t even know from across the world, to the point where you develop a real sense of camaraderie, is something unique.”
Justin Collins, Lead Developer of Brakeman, said that one of the advantages of OSS in general is the diversity of people who use the software in all kinds of weird environments.
“GitHub proves a convenient way of reporting problems and contributing fixes for those unexpected use cases. One person can only cover so much, and oftentimes different people will have different needs. Opening up software to allow everyone to customize or fix it as needed is a really great way to grow software,” he says.
Robin Sommer, Senior Researcher at ICSI and lead of the Bro network analysis platform, says that GitHub provides a lot of visibility, and makes it really easy for external people to browse the content and contribute back patches and functionality. “GitHub’s APIs make it possible for us to integrate everything well even into our non-standard workflow,” he added.
GitHub has many advantages for developers working on their own.
“A perfect example came out of one of my projects. I’d built a number of scripts using CoffeeScript, a language I don’t know terribly well, that got the job done, but my development was routinely hampered when something didn’t do quite what I expected,” Roberts explained.
“So without any prompting, my friend Matt Johannsen of WhiteHat realized he could write tests for the project, and in a few minutes had built tests and submitted them as a Pull Request. I reviewed it, saw how useful it would be, and merged it. A few more minutes to enable continuous integration and now our project was being tested on every commit.”
“A person who wasn’t a member of a project was able to jump in, without setting up permissions or anything, and contribute useful code that improved the entire project,” he pointed out. “That’s something that just couldn’t be done 10 years ago and is leading to an explosion of new software that’s getting help from a wide variety of course.
The majority of GitHub’s server infrastructure runs Ubuntu, all managed with Puppet. Automation is key, since they need the ability to make changes quickly, especially when it comes to security updates and deploying new features.
“The real star of our network, aside from out ops team, is Hubot,” says Roberts, and explains: “Hubot is GitHub’s home grown chatbot. He does everything from finding silly pictures to deploying GitHub itself and to helping us search through our logs. Hubot has really been a game changer for us and how we run GitHub.Ã‚Â We also use Puppet on our OS X desktops too, a project we released called Boxen. It’s been an effective way for us to manage a geographically distributed fleet of laptops, as made it easy for our developers to collaborate on projects.”
Security measures and tools
Generally, when I see a software offered for download, I always wonder what the site in question has done to make sure there’s no malware bundled with it. Roberts says that is a scenario they think about often.
“One of the keys to protecting projects from these types of attacks is protecting the key accounts that manage the project. Without authenticated access to a project it’s impossible for bad guys to inject malicious code or trojaned downloads. We’ve made a lot of changes to make it easier for users to protect their accounts, such as providing a two factor authentication option last year and improving organization controls,” he shared.
In rare cases when something does make it through, they work quickly along side the project owners, the community, and the support team to get the offending code analyzed and removed.
In their daily work, the GitHub team uses a combination of commercial, open source and, unsurprisingly, custom internal tools.
“We’re big Splunk fans for logging while also looking into ELK (Elasticsearch, Logstash, and Kibana), but we’ve also put a lot of effort into tools like Graphite for data analytics,” says Roberts. “Our application security team gets a lot of use out of tools like Brakeman, Burp proxy, and a lot of custom code/tools. On the DFIR side we have a lot of instrumentation of GitHub itself, and are big fans of Maltego for investigations, Cuckoo Sandbox for malware analysis, and OSXCollector and Autopsy for forensics analysis. We’re also looking into Google’s GRR project for remote forensics.”
With a massive developer base that’s used to getting feature requests for their own projects, I can’t even imagine the amount of requests that GitHub gets for enhancements.
Roberts is currently focused on automating as many of their common security tasks as possible.
“My approach is always that the 2nd time you do a task is a coincidence, but by the third time you should be thinking about how to make something automated and repeatable. We do this using Hubot for many tasks, Boxen for Hubber systems, and Puppet for our servers,” he says.
“In my mind security is all about preparing for the emergencies, and one of the best ways to do that is to make sure that the simple things are out of the way.”