Ubuntu Wiki: This last week, 5 of the 8 servers that are LoCo hosted but Canonical sponsored, had to be shut down due to reports that they were actively attacking other machines. These servers were found to have a variety of problems including, but not limited to, missing security patches, FTP (not sftp, without SSL) was being used to access the machines, and no upgrades past breezy due to problems with the network cards and later kernels.
More details from the Loco Contacts mailing list:
On Monday evening (UK time) it was reported that one of the hosted community servers that Canonical sponsors had been compromised. After investigation, it became apparent that 5 of the 8 machines had been compromised. Since it was reported that they were actively attacking other machines, the decision was taken to shut the machines down.
On Tuesday morning we started the procedure of bringing these machines up in a safe state so that we could recover data from them. Unfortunately, this took far longer than we would have hoped or liked due to a combination of having to use remote hands, arbitrary limits imposed by those remote hands and (relative) lack of bandwidth to copy data off site.
This process is still ongoing (though only one remain has yet to be fully recovered – tiber).
How did this happen
a) the servers, especially zambezi were running an incredible amount of web software (over 15 packages that we recognised) and of all the ones where it’s trivial to determine a version, they were without exception out-of-date and missing security patches. An attacker could have gotten a shell through almost any of these sites.
b) FTP (not sftp, without SSL) was being used to access the machines, so an attacker (in the right place) could also have gotten access by sniffing the clear-text passwords.
c) The servers have not been upgraded past breezy due to problems with the network card and later kernels. This probably allowed the attacker to gain root.