Linux – Who Rooted Kernel.org Servers Two Years Ago, How Did It Happen, And Why?

See also:

Linus Torvalds Talks Linux Development at LinuxCon (eWeek, Sep 18, 2013):

NSA Backdoor

Torvalds was also asked if he had ever been approached by the U.S. government to insert a backdoor into Linux. Torvalds responded “no” while nodding his head “yes,” as the audience broke into spontaneous laughter.


Who rooted kernel.org servers two years ago, how did it happen, and why? (Ars Technica, Sep 24, 2013):

Maintainers of Linux still haven’t delivered promised autopsy of serious breach.

More than two years after unknown hackers gained unfettered access over multiple computers used to maintain and distribute the Linux operating system kernel, officials still haven’t released a promised autopsy about what happened.

The compromise, which began no later than August 12, 2011, wasn’t detected for at least 16 days, a public e-mail and interviews immediately following the intrusion revealed. During that time, attackers were able to monitor the activities of anyone using the kernel.org servers known as Hera and Odin1, as well as personal computers belonging to senior Linux developer H. Peter Anvin. The self-injecting rootkit known as Phalanx had access to a wealth of sensitive data, possibly including private keys used to sign and decrypt e-mails and remotely log in to servers. A follow-up advisory a few weeks later opened the possibility that still other developers may have fallen prey to the attackers.

For three weeks in September and early October, officials kept kernel.org closed so the servers that run it could be rebuilt. When the site reopened on October 4, a message on the front page prominently warned of the breach and noted the steps taken to rebuild the site. “Thanks to all for your patience and understanding during our outage and please bear with us as we bring up the different kernel.org systems over the next few weeks,” the message concluded. “We will be writing up a report on the incident in the future.”

Almost two years later, the report has yet to be delivered. The promise to deliver an incident report remained on kernel.org as recently as March 1 of this year, before being quietly pulled the following day. To this day, officials have yet to provide key details, including exactly how many machines were compromised, how the attackers were able to gain root access to them, and what they did once they seized control. The delay contrasts sharply with autopsies that were delivered promptly following two similar compromises of Apache.org, the official distributor of the open-source Apache Web server.

“As a user, I think everyone should be a little bit disappointed they didn’t execute that transparency in a follow-up,” Dan Rosenberg, a senior security researcher at Azimuth Security, told Ars. Without a thorough autopsy, “it’s hard to really know what level of negligence was involved in the compromise.”

Linux developer and maintainer Greg Kroah-Hartman told Ars that the investigation has yet to be completed and gave no timetable for when a report might be released. He said officials remain confident of preliminary findings that the attackers were not able to tamper with the source code that millions of organizations use to compile their Linux systems.

“We went through many rounds of validation of the kernel releases on the site, regenerating them from the Git tree and old backups,” he wrote in an e-mail. “All of them were fine, nothing was found to be tampered with or touched at all.”

Git is the name of the system that tracks changes made to the source code for the Linux kernel. It uses a series of 160-bit cryptographic hashes to account for the revisions. Copies of the repository and all changes are then cached in thousands of locations around the world. A mismatched hash in one or more location would quickly indicate unofficial changes. Kroah-Hartman also told Ars kernel.org systems were rebuilt from scratch following the attack. Officials have developed new tools and procedures since then, but he declined to say what they are.

“There will be a report later this year about site [sic] has been engineered, but don’t quote me on when it will be released as I am not responsible for it,” he wrote.

While there’s no evidence that backdoors or other malicious code were surreptitiously inserted, the 2011 breach of one of the world’s most important software development organizations should nonetheless remain a concern. That’s especially true now that we’re living in an era where actors of powerful nation-states have been known to hijack Microsoft’s official Windows update mechanism and deliberately weaken cryptographic coding standards. Transparency is more important than ever. If Linux maintainers expect trust, they should make good on their promise to tell users precisely what happened and how.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.