OpenSSL Heartbleed bug, Apache still vulnerable?

[Jochen Spieker]

Hi,

as many others, I patched my machines today because of the horrible
OpenSSL bug:

$ apt-cache policy libssl1.0.0
libssl1.0.0:
Installed: 1.0.1e-2+deb7u6
Candidate: 1.0.1e-2+deb7u6
Version table:
1.0.1g-1 0
-10 http://http.debian.net/debian/ sid/main amd64 Packages
*** 1.0.1e-2+deb7u6 0
500 http://security.debian.org/ wheezy/updates/main amd64 Packages
100 /var/lib/dpkg/status
1.0.1e-2+deb7u4 0
500 http://http.debian.net/debian/ wheezy/main amd64 Packages

I made sure all relevant services were restarted after the upgrade. I
even rebooted the (virtual) machine just to be sure. But when using the
test tool from https://github.com/FiloSottile/Heartbleed I am notified
that Apache on my server is still vulnerable:

$ ./Heartbleed well-adjusted.de:443
2014/04/08 16:30:09 ([]uint8) {
00000000 02 00 79 68 65 61 72 74 62 6c 65 65 64 2e 66 69 |..yheartbleed.fi|
00000010 6c 69 70 70 6f 2e 69 6f 59 45 4c 4c 4f 57 20 53 |lippo.ioYELLOW S|
00000020 55 42 4d 41 52 49 4e 45 6e 10 a2 39 eb 0f 73 9e |UBMARINEn..9..s.|


}

Dovecot is apparently fine:

$ ./Heartbleed well-adjusted.de:993
2014/04/08 16:36:19 well-adjusted.de:993 - SAFE

Am I doing anything wrong? Is the testing tool broken? I also tried the
one at https://gist.github.com/takeshixx/10107280 which confirms there
is still a problem on port 443 (HTTPS served by Apache).

J.

[Reco]

Hi.

Post by Jochen Spieker
Am I doing anything wrong? Is the testing tool broken? I also tried the
one at https://gist.github.com/takeshixx/10107280 which confirms there
is still a problem on port 443 (HTTPS served by Apache).

No, chances are, you're using the tool correctly.
I'm using this [1] to test servers for this vulnerability (as I can't
force myself to install Go), and your server shows as vulnerable.

[1] http://pastebin.com/WmxzjkXJ

Reco

[Jochen Spieker]

Post by Reco
Hi.

Post by Jochen Spieker
Am I doing anything wrong? Is the testing tool broken? I also tried the
one at https://gist.github.com/takeshixx/10107280 which confirms there
is still a problem on port 443 (HTTPS served by Apache).

No, chances are, you're using the tool correctly.
I'm using this [1] to test servers for this vulnerability (as I can't
force myself to install Go), and your server shows as vulnerable.
[1] http://pastebin.com/WmxzjkXJ

Thanks for confirming this. But I don't get how this is possible. I
upgraded the library that Apache (supposedly) uses and rebooted the
machine afterwards. I even re-installed all Apache packages just to make
that they are ok.

J.

[Scott Ferguson]

Post by Jochen Spieker
Hi,
as many others, I patched my machines today because of the horrible
$ apt-cache policy libssl1.0.0
Installed: 1.0.1e-2+deb7u6
Candidate: 1.0.1e-2+deb7u6
1.0.1g-1 0
-10 http://http.debian.net/debian/ sid/main amd64 Packages
*** 1.0.1e-2+deb7u6 0
500 http://security.debian.org/ wheezy/updates/main amd64 Packages
100 /var/lib/dpkg/status
1.0.1e-2+deb7u4 0
500 http://http.debian.net/debian/ wheezy/main amd64 Packages
I made sure all relevant services were restarted after the upgrade.

# ps uwwp $(find /proc -maxdepth 2 -name maps -exec grep -HE
'/libssl\.so.* \(deleted\)' {} \; | cut -d/ -f3 | sort -u)

*Then regenerate keys*

Post by Jochen Spieker
I
even rebooted the (virtual) machine just to be sure. But when using the
test tool from https://github.com/FiloSottile/Heartbleed I am notified

Notice on http://filippo.io/Heartbleed/
"There are load (?) issues causing FALSE NEGATIVES."

Post by Jochen Spieker
$ ./Heartbleed well-adjusted.de:443
2014/04/08 16:30:09 ([]uint8) {
00000000 02 00 79 68 65 61 72 74 62 6c 65 65 64 2e 66 69 |..yheartbleed.fi|
00000010 6c 69 70 70 6f 2e 69 6f 59 45 4c 4c 4f 57 20 53 |lippo.ioYELLOW S|
00000020 55 42 4d 41 52 49 4e 45 6e 10 a2 39 eb 0f 73 9e |UBMARINEn..9..s.|
…
}
$ ./Heartbleed well-adjusted.de:993
2014/04/08 16:36:19 well-adjusted.de:993 - SAFE
Am I doing anything wrong? Is the testing tool broken? I also tried the
one at https://gist.github.com/takeshixx/10107280 which confirms there
is still a problem on port 443 (HTTPS served by Apache).

That test tool was updated a few hours ago to include checks for
patches. You may find you now get "Version number indicates vulnerable,
but your build is recent so may be patched."

Restart apache2, if you can, reboot.

Post by Jochen Spieker
J.

Use the code I quoted above to double check you restarted all relevant
services and daemons. All our servers test fine (as being patched) - but
we shut down completely for the key replacement process and some won't
be back on line until tomorrow. At this point we don't know whether to
consider everything potentially compromised - or only the last week or
so..... :(


Kind regards

[Jochen Spieker]

Post by Scott Ferguson

Post by Jochen Spieker
as many others, I patched my machines today because of the horrible
$ apt-cache policy libssl1.0.0
Installed: 1.0.1e-2+deb7u6
Candidate: 1.0.1e-2+deb7u6
1.0.1g-1 0
-10 http://http.debian.net/debian/ sid/main amd64 Packages
*** 1.0.1e-2+deb7u6 0
500 http://security.debian.org/ wheezy/updates/main amd64 Packages
100 /var/lib/dpkg/status
1.0.1e-2+deb7u4 0
500 http://http.debian.net/debian/ wheezy/main amd64 Packages
I made sure all relevant services were restarted after the upgrade.

# ps uwwp $(find /proc -maxdepth 2 -name maps -exec grep -HE
'/libssl\.so.* \(deleted\)' {} \; | cut -d/ -f3 | sort -u)

Doesn't work here because the find command produces no output. As I
said, I already rebooted the machine so it is impossible that Apache
uses a deleted file.

Post by Scott Ferguson
*Then regenerate keys*

Yes, I still need to do that.

Post by Scott Ferguson

Post by Jochen Spieker
I
even rebooted the (virtual) machine just to be sure. But when using the
test tool from https://github.com/FiloSottile/Heartbleed I am notified

Notice on http://filippo.io/Heartbleed/
"There are load (?) issues causing FALSE NEGATIVES."

This only applies to the web version. The command line program that I
used is not affected.

Post by Scott Ferguson

Post by Jochen Spieker
Am I doing anything wrong? Is the testing tool broken? I also tried the
one at https://gist.github.com/takeshixx/10107280 which confirms there
is still a problem on port 443 (HTTPS served by Apache).

That test tool was updated a few hours ago to include checks for
patches. You may find you now get "Version number indicates vulnerable,
but your build is recent so may be patched."

I have the most recent version and it still reports my system to be
vulnerable.

Post by Scott Ferguson
Use the code I quoted above to double check you restarted all relevant
services and daemons. All our servers test fine (as being patched) - but
we shut down completely for the key replacement process and some won't
be back on line until tomorrow. At this point we don't know whether to
consider everything potentially compromised - or only the last week or
so..... :(

I think the most important thing is to patch the systems and to replace
all certificates. But then I only run private systems with less than a
dozen users.

J.

[Sven Hartge]

Post by Jochen Spieker

Post by Scott Ferguson

Post by Jochen Spieker
Am I doing anything wrong? Is the testing tool broken? I also tried the
one at https://gist.github.com/takeshixx/10107280 which confirms there
is still a problem on port 443 (HTTPS served by Apache).

That test tool was updated a few hours ago to include checks for
patches. You may find you now get "Version number indicates vulnerable,
but your build is recent so may be patched."

I have the most recent version and it still reports my system to be
vulnerable.

Are you sure you restarted the right system? (Just asking, had the same
problem today, was looking at a totally different system than the one I
thought I was looking at.)

Maybe apache is using a different libssl than the one from the system.
What does "ldd /usr/lib/apache2/modules/mod_ssl.so" say?

Or maybe there is some kind of cache in front of your webserver.

Grüße,
Sven

[Jochen Spieker]

Post by Sven Hartge

Post by Jochen Spieker
I have the most recent version and it still reports my system to be
vulnerable.

Are you sure you restarted the right system? (Just asking, had the same
problem today, was looking at a totally different system than the one I
thought I was looking at.)

Yes, I am sure. :)

Post by Sven Hartge
Maybe apache is using a different libssl than the one from the system.
What does "ldd /usr/lib/apache2/modules/mod_ssl.so" say?

Ah, thanks. I tried ldd on the apache binary already but that is not
linked against libssl.

$ ldd /usr/lib/apache2/modules/mod_ssl.so
linux-vdso.so.1 => (0x00007ffffdd22000)
libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fe3c8139000)
libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007fe3c7d42000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe3c7b25000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe3c779a000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe3c7596000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fe3c737e000)
/lib64/ld-linux-x86-64.so.2 (0x00007fe3c85d3000)

Thinking about this 
 what I actually use is mod_spdy which is not
linked against libssl. It probably has the same bug 


Yes, here it is:
https://code.google.com/p/mod-spdy/issues/detail?id=85

| Note that just disabling the spdy module in Apache won't work, because
| the SSL library itself is replaced. Easiest fix on Debian is to remove
| the mod-spdy package from the system (for now).

Thanks for helping me to find this. After removing mod-spdy-beta
and stopping and starting Apache, the test tools deem my system safe.

J.

[Sven Hartge]

Thinking about this … what I actually use is mod_spdy which is not
linked against libssl. It probably has the same bug …
https://code.google.com/p/mod-spdy/issues/detail?id=85
| Note that just disabling the spdy module in Apache won't work, because
| the SSL library itself is replaced. Easiest fix on Debian is to remove
| the mod-spdy package from the system (for now).
Thanks for helping me to find this. After removing mod-spdy-beta
and stopping and starting Apache, the test tools deem my system safe.

Ürx, nasty one.

I presume mod_spdy is not from any offical package (cannot find any
package matching "spdy" in Debian anywhere) but a module compiled by
yourself?

Grüße,
Sven.

[Jochen Spieker]

Post by Sven Hartge

Post by Jochen Spieker
https://code.google.com/p/mod-spdy/issues/detail?id=85
| Note that just disabling the spdy module in Apache won't work, because
| the SSL library itself is replaced. Easiest fix on Debian is to remove
| the mod-spdy package from the system (for now).
Thanks for helping me to find this. After removing mod-spdy-beta
and stopping and starting Apache, the test tools deem my system safe.

Ürx, nasty one.
I presume mod_spdy is not from any offical package (cannot find any
package matching "spdy" in Debian anywhere) but a module compiled by
yourself?

I think I installed a .deb from Google which added the file
/etc/apt/sources.list.d/mod-spdy.list:

deb http://dl.google.com/linux/mod-spdy/deb/ stable main

As you wrote elsewhere, a patch is available and updated binaries should
be available soon:

https://code.google.com/p/mod-spdy/issues/detail?id=85#c2

J.

[Jochen Spieker]

Post by Jochen Spieker

Post by Sven Hartge
I presume mod_spdy is not from any offical package (cannot find any
package matching "spdy" in Debian anywhere) but a module compiled by
yourself?

I think I installed a .deb from Google which added the file
deb http://dl.google.com/linux/mod-spdy/deb/ stable main
As you wrote elsewhere, a patch is available and updated binaries should
https://code.google.com/p/mod-spdy/issues/detail?id=85#c2

The repository now contains a fixed version (0.9.4.2-r413). I tested it
and the new version looks fine.

J.

[Jochen Spieker]

Post by Jochen Spieker
The repository now contains a fixed version (0.9.4.2-r413). I tested it
and the new version looks fine.

Don't mean to hijack, but is this a useful tool?
http://filippo.io/Heartbleed/

Yes, it is. Qualys tests for the new attack as well now:

https://www.ssllabs.com/ssltest/

J.

[Curt]

Post by Jochen Spieker

Post by Scott Ferguson
http://filippo.io/Heartbleed/

https://www.ssllabs.com/ssltest/

Thank you. The ssllabs test seems quite thorough!

[Curt]

Post by Jochen Spieker
The repository now contains a fixed version (0.9.4.2-r413). I tested it
and the new version looks fine.

Don't mean to hijack, but is this a useful tool?

http://filippo.io/Heartbleed/

(I'm an ignorant end user who has just woken up to the issue of bleeding
hearts).

[Sven Hartge]

Post by Jochen Spieker
The repository now contains a fixed version (0.9.4.2-r413). I tested it
and the new version looks fine.

Don't mean to hijack, but is this a useful tool?
http://filippo.io/Heartbleed/

To scan your complete network in mere seconds:

https://github.com/robertdavidgraham/masscan
http://blog.erratasec.com/2014/04/using-masscan-to-scan-for-heartbleed.html

S°

[shawn wilson]

Post by Sven Hartge

Post by Jochen Spieker
The repository now contains a fixed version (0.9.4.2-r413). I tested it
and the new version looks fine.

Don't mean to hijack, but is this a useful tool?
http://filippo.io/Heartbleed/

https://github.com/robertdavidgraham/masscan

http://blog.erratasec.com/2014/04/using-masscan-to-scan-for-heartbleed.html
There's also ssl-heartbleed.nse which (even though its not threaded) is
probably a bit faster (not to mention has a familiar interface). It also
looks like they cleaned some stuff up in the port.

[Sven Hartge]

Thinking about this … what I actually use is mod_spdy which is not
linked against libssl. It probably has the same bug …
https://code.google.com/p/mod-spdy/issues/detail?id=85
| Note that just disabling the spdy module in Apache won't work, because
| the SSL library itself is replaced. Easiest fix on Debian is to remove
| the mod-spdy package from the system (for now).

And a fix has been comitted:

https://code.google.com/p/mod-spdy/issues/detail?id=85#c2
https://code.google.com/p/mod-spdy/source/detail?r=408

S°

[Gary Carter]

Hi guys,

Sorry if I end up doing this wrong (don't tend to post to lists often),
thread-wise, but I ran into the same issue where it seemed that despite
upgrading OpenSSL to the patched version, my Apache server was still
vulnerable to Heartbleed.

Just curious - are you running Google's mod_spdy? If so, that was the
culprit for me - check:

/etc/apache2/mods-enabled/ssl.load

They overrode mod_ssl like so:

LoadModule ssl_module /usr/lib/apache2/modules/mod_ssl_with_npn.so

I just decided to uninstall the package entirely to revert to
/usr/lib/apache2/modules/mod_ssl.so and that fixed it.

There may be other plugins that do this, so I'd recommend anyone running
into this double-check what modules Apache is *actually* set to load.

Hope this helps. :)

- Gary Carter

[Jochen Spieker]

Post by Gary Carter
Just curious - are you running Google's mod_spdy? If so, that was the

Yes, that was it. Thanks for the heads-up.

J.