Critical hole found in encryption program

A popular open-source program for encrypted communications has a serious flaw that could let Internet attackers slip into servers running the software, said its creators and a security company this week.

The program, Open Secure Shell (OpenSSH), is included in many widely used operating system distributions, such as OpenBSD 3.0, OpenBSD 3.1 and FreeBSD-Current, all open-source variants of the Unix OS. Such operating systems appear on networking equipment and security appliances, among other things.

The flaw affects versions 3.0 to 3.2.3 of the software, said Grant Slender, principal consultant for Australasia at network protection company Internet Security Systems, which first discovered the vulnerability.

Slender said the flaw involves OpenSSH's inadequate handling of "buffer overflow" attacks, in which a message sent to a program is much longer than the program is designed to expect. Attackers exploit such holes by flooding programs with more characters than they can accommodate and running the excess characters as executable code.

Because of the flaw, "it is possible for a remote (off-site) attacker to send a specially crafted (message) that triggers an overflow," according to the ISS advisory. "This can result in a remote denial-of-service attack on the OpenSSH daemon." A denial-of-service attack overloads a server with requests for information, tying up the machine indefinitely.

The advisory also said that hackers exploiting the hole would enter a server at the highest level of access. "The OpenSSH daemon runs with superuser privilege, so remote attackers can gain superuser access by exploiting this vulnerability," it said.

ISS has been criticized recently for its handling of another security alert involving a flaw in the popular open-source Apache Web server. ISS alerted the public to the Apache hole the same day it warned the Apache developers, giving the programmers no head start on fixing the flaw. This time, the company gave notice.

Slender said ISS notified OpenSSH's senior developer, who had created a patch. "In this case, we did contact the senior developer and, with his coordination, we worked toward making sure the (programming) community was ready to have the vulnerability announced," he said.

ISS is advising system administrators to disable unused OpenSSH authentication mechanisms.

It's also possible for administrators to remove the vulnerability by disabling the challenge-response authentication parameter within the OpenSSH daemon configuration file, according to the advisory. Slender also said people should upgrade.

Information about the vulnerability has been posted on security mailing lists such as Bugtraq and Debian.

Staff writer Vivienne Fisher reported from Sydney. News.com's Robert Lemos contributed to this report.