Red Hat, Fedora servers compromised

Red Hat warned on Friday that a network attack compromised some servers last week that are involved with both its commercially supported and free versions of Linux.

The breaches involved Red Hat Linux Enterprise servers and those from its community-supported Fedora project that it sponsors.

Red Hat said in a security advisory that it is confident the intrusion did not compromise the Red Hat Network, which is the chief mechanism used to distribute changes to its Red Hat Enterprise Linux product, or updates sent over the network. Therefore customers are not at risk, the company said.

The open-source vendor also released a script designed to detect potentially compromised OpenSSH (OpenBSD's Secure Shell protocol implementation) packages.

"We are issuing this alert primarily for those who may obtain Red Hat binary packages via channels other than those of official Red Hat subscribers," the advisory said.

The intruder was able to sign a "small number" of OpenSSH packages relating to Red Hat Enterprise Linux versions 4 and 5, so Red Hat is releasing an updated version of those packages. The company has published a list of the tampered packages and instructions for how to detect them.

A Fedora project leader issued an alert to a Fedora e-mail list that some Fedora servers were taken offline after they were found to have been illegally accessed last week.

"One of the compromised Fedora servers was a system used for signing Fedora packages. However, based on our efforts, we have high confidence that the intruder was not able to capture the passphrase used to secure the Fedora package signing key," the alert said.

Despite the fact that there is no evidence that the Fedora key has been compromised, Fedora is converting to new Fedora signing keys because Fedora packages are distributed via multiple third-party mirrors and repositories.

[Mr. Dee]

I thought that Linux was suppose to be secure? Guess not, oh well, back to Windows!

[JuggerNaut]

Yeah, back to something even (more) less secure :-P

[JCPayne]

Cool so the Linux community will have a patch completed sometime this weekend if not yet... Microsoft would be trying to hush the media before they could get a patch out in about another 2-3 weeks time.

[Penguinisto]

@Mr. Dee:

Compared to the swiss-cheese that's Windows, yes, it is secure. Or do I need to remind you of this:
http://news.zdnet.com/2100-9595_22-111513.html?legacy=zdnn
http://www.accessmylibrary.com/coms2/summary_0286-6438492_ITM

So, let's contrast:

* Windows 2000 (and Office 2000) source code files were stolen outright, and that was after the intruders had literally months on end to play around in MSFT's networks. Meanwhile, Microsoft vehemently denied any such thing had happened until Windows 2000's source code was splattered all over the Internet, and the denials were no longer plausible. It took Microsoft months to figure out what happened, and close the holes.

One public-facing server at RedHat managed to get compromised for a short period of time, and was caught before doing anything more than token damage to one file (which you can see the source code to anyway). RH's response was open and full, a detection script and numerous other proactive methods were immediately issued, and new package keys are being generated as a precaution.

So tell me - whom would you rather trust as a vendor of a secure product?

[n3td3v]

He was joking.

[Vegaman_Dan]

All of your FUD doesn't dimiss the fact that the servers were compromised and they were vulnerable. Mr. Dee's comments stand without contest.

You're doing an awful lot of apology and spinwork, Penguinisto. It would have been a better answer to simply say that no system is perfect, the servers were compromised and Red Hat took appropriate actions to address the situation. Instead you go on a FUD campaign which only derails any attempt at a legitimate and professional response. Your comments end up sounding entirely like a troll.

If it walks like a troll, talks alike a troll, and posts like a troll, then Penguinisto earns the title honestly.

[Penguinisto]

@n3td3v: yeah, but only as a chronic troll. ;)

@Dan: Everything I posted up there is true. Prove me wrong, that's all you need to do. Until then, you haven't a leg to stand on.

[walletless]

Penguinsto: Comparing a version that was almost 8 years old to the current version of Redhat is like comparing an apple to a rotten orange. Can you quote some significant breaches on Windows Server 2003 or newer of this scale?
Windows *is* fairly secure, and so is Redhat. You can never be 100% secured, and that is the reality. We will continue to find such attacks on most OSs.

[Penguinisto]

For Win2k3? Probably not ...yet. But then, when you consider that even Microsoft uses Akamai (a Linux-based caching host service) for the entire microsoft.com external-facing domain, what does that tell you?

/P

[Vegaman_Dan]

Penguinisto: It tells us that Walletless called your bluff and you caved having no evidence to back you up.

[Penguinisto]

Hiya Dan!

You still have yet to prove me wrong (as does "walletless". Try if you wish, but the point still stands: MSFT is more consistently vulnerable, doesn't disclose anything that they don't absolutely have to, and puts millions of users at risk with their behaviors. RedHat OTOH does not.

[imacommie]

@walletless: Penguinsto's point wasn't a compare-and-contrast of which OS is more secure. The point was that MS handles disclosure like the USSR. If MS was responsible for telling people about Chernobyl, we'd have gotten a report saying that there was a confirmed fire in the trash basket of the men's bathroom. It's not which OS is more secure; it's which vendor is going to tell you you have no clothes on.

[Vegaman_Dan]

You do have a point there. Red Hat is pretty good at keeping the public up to date. They release patches out to the community very quickly. Microsoft is a bit more quiet about the issue, and releases patches only after they have been tested to make sure they don't interfere with other products- something Linux rarely deals with. Apple is the king at denying reality and keeping quiet about everything unless it is so blatantly obvious to the world that they have no choice in the matter... even then calling it a 'bug fix.'.

Good points all. Your posting was perhaps the most sensible of them all, Walletless.

[imacommie]

@walletless, et al: Penguinsto's point wasn't a compare-and-contrast of which OS is more secure. The point was that MS handles disclosure like the USSR. If MS was responsible for telling people about Chernobyl, we'd have gotten a report saying that there was a confirmed fire in the trash basket of the men's bathroom. It's not which OS is more secure; it's which vendor is going to tell you you have no clothes on.

[imacommie]

@myself: sorry about the dupe post, must have been a refresh issue

[mbenedict]

@penguisto: you have no idea what you're talking about.

Among the security industry rumors that Fedora was hacked has been circulating for awhile now, with both Fedora and RedHat deciding to keep silent.

Then last week we hear from Fedora that some "infrastructure issues" were discovered with no explanation, only to expect intermittent failure of their servers. Nothing from RedHat.

THEN we suddenly hear rumors that there are actually TWO different intrusions: one at Fedora, and *a separate attack* on RedHat.

Only today after the rumor-mill was exploding that Fedora and RedHat made an announcement. However we are all still in the dark as to WHAT ACTUALLY HAPPENED.

What we know:

1. Several Fedora servers got hacked, including one used to sign Fedora packages. According to Fedora, their signing key was not compromised.

2. RedHat was also separately (and more seriously) hacked. The hacker was able to compromise RedHat's signing keys (!!!), then tamper & sign a number of security-related packages (!!!) including RedHat's OpenSSH distribution.

This is huge!!! It's an attack to the heart of RHN and bigger than any of those hacks at Microsoft you mentioned, because it means some RedHat's customers may have also been compromised because of this!!! Now everyone has to go back and check their packages against the blacklist.

It would be akin to a hacker getting into Microsoft's or Apple's Automatic Updates and being able to send out a tampered package. Doesn't get bigger than that!

But we still don't know HOW the compromise happened! Both Fedora and RedHat are keeping their customers in the dark. Is there an undisclosed gaping remotely-exploitable security hole in Fedora & RedHat?? If I were running either I won't be sleeping well tonight. RedHat should come clean instead of still keeping silent.

[wnstb]

@mbenedict August 22, 2008 3:26 PM PDT

Mr. Benedict,
You are just wrong. Red Hat's signing keys were not compromised. You can read details about the keys at
http://www.awe.com/mark/blog/200701300906.html

RHN was NOT compromised, no unauthorized packages were uploaded into the RHN systems.

Both the Fedora team and Red Hat have given details about the incident.

As far as not doing your homework or just being a full of it, it doesn't get bigger than you.

[mbenedict]

@wnstb:

The attacker *WAS ABLE TO SIGN REDHAT PACKAGES*. Which de facto means the signing keys are compromised, however you choose to bury your head in the sand or not. The link you posted from early 2007 doesn't add any value whatsoever to this discussion.

[ferretboy88]

The fact of the matter is that ANY system can be messed with. We are all screwed.

[ferretboy88]

Apple is next. If Fedora can be messed with then a mac with no security will sure to be next.

[vini156]

Where is Matt Asay?

[iBuzz]

Unix security is a myth. The big flaw with Unix is that it offers an all-powerful administrator account that has complete access to the system. Instead of segmenting access privileges, so say, if you ask for permission to change a password, you are only allowed to change a password and nothing else, Unix gives you the keys to entire store. For example, to change a password, the password changing program has to run under administrator and then it is allowed to do whatever it wants to the system. If you can somehow inject your own code into the password changing program, you can make it do whatever you want. You have complete access to everything.

Think of a computer as a house. A truly secure system would have motion detectors in each room. So, just because you were able to break through the front door or side window, you wouldn't be allowed to go anywhere else in the house. Instead, Unix security is like a house with a big lock on the front door but with nothing else. That lock often keeps people out, but once you get in, you have access to every room in the entire house.

That is not a secure system.

[Dalkorian]

What you failed to recognize is the fact that a "secure system" is a myth. "If you can somehow inject your own code into the password changing program ..." is a ridiculous comment, if you can inject your own code into any OS base you can make it do whatever you want. You also might want to read mbenedict's response below, you might find it enlightening.

Nothing is secure. Unix/Linux is better than most (all?) OS's out there at it, but it's not perfect either. Using your house analogy, there are some popular OS's out there which will remain unnamed that have nothing but a few strips of police tape across the unlocked door. At least Unix locks the door.

[mbenedict]

@iBuzz: you would be right 10 years ago. What you say though is generally no longer true.

1. The modern "Unix" kernel implement sophisticated compartments. A compromise in one part of the OS does not mean the entire OS is compromised. A password changing program in your example can be restricted to modify objects labeled as passwords, nothing else. We call this Trusted Computing Base (TCB). For Linux in particular, TCB has been available since 2000 (thanks to the NSA and their work with SE-Linux), and has been part of the mainline kernel since 2003 when SE-Linux patches were merged into kernel 2.6. It is standard on RedHat since 4.0, and the NSA patches also made it into FreeBSD and Darwin. Other Unices such as Solaris and AIX have had TCB options forever now.

2. Outside the kernel, in practice Unix administrators do not use an "all powerful root" anymore. Instead administration is done typically using a program called "sudo". Sudo limits what admins can do. Also some Unices (such as FreeBSD) have "secure levels" where even the "all powerful root" is actually restricted from doing many operations when running in multi-user mode. I.e., your compromised password-changing program still cannot overwrite critical system files, for example.

3. Also, a common Unix administration practice over the past few years is the use of virtualization (e.g., Solaris Zones, HP-UX VPARs, FreeBSD jail, etc.) Services are relegated to different zones, so a compromise in one zone does not affect other zones. This is becoming even more popular today with the advent of hypervisors.

4. In security conscious environments (e.g., certain financial institution systems or government systems) typically there is an access management system that's run on top of Unix. A popular example is the "eTrust Access Control" system made by Computer Associates. Like TCB, eTrust places strict controls on what users and processes (including those running as "root") can and cannot do. eTrust also provides alerting and audit logging.

[ferretboy88]

The guys who broke into the servers should be hanged until death. That might stop most of the nonsense.

[Tomofumi]

According to their openssh blacklist script, the affected versions are:

# The signed tampered packages were:
#
# openssh-3.9p1-8.RHEL4.24 for i386, x86_64 architecture
# openssh-3.9p1-9.el4 for i386, x86_64 architecture
# openssh-4.3p2-26 for x86_64 architecture
# openssh-4.3p2-26.el5 for x86_64 architecture

fortunately, my server's version is still openssh-4.3p2-24.el5....

[gmbidols]

Very Nice Blog. I Like Your Blog Please Visit My Website and Give Your Review.

http://www.gmb.in/ http://www.ancientpeaks.com