Meltdown and Spectre

[Zhariak]

This quote is hidden because you are ignoring this member. Show Quote

My rep say this Friday. But I am skeptical because:

http://www.zdnet.com/article/meltdow...eds-to-do-now/
"In addition, devices that use pre-Haswell Intel CPUs (Ivy Bridge and earlier designs) are most likely to suffer serious performance issues as a result of software updates."

Gen8 is IvyBridge.

Thanks for passing along the info bud! I hope they release it as they say!

VMware released their microcode and ESXi updates yesterday. Deployed those. Haven't tested performance much, but had to do some emergency maintenance (due to a prolonged power outage last night). Definitely big performance change from what I'm used to seeing in my environment... Not sure if this was the ESXi and microcode updates, or if this was the MS updates inside of the VMs.

REALLY curious to see what the BIOS update will add to the mix, lmao...

[rage2]

BIOS update, Linux update and the VMWare update does the same thing. It pushes the latest 1/8/2018 microcode up to the CPU. In theory, if you do either or, you're good to go. The microcode update is a mitigation for Spectre. Meltdown is addressed completely in software (KPTI). Spectre is less of a risk, it's a very targeted and specific attack.

Windows is a bit of a pain in the ass because MS hasn't released the microcode update package as a windows update (which would remove the need for a BIOS update). With how many different MB manufacturers out there, the fragmentation is 100000x worse than Android, so yea, I can't see everyone having support still on their motherboards, or even attempting to update their damn BIOS. MS needs to get it out ASAP (last Microsoft Microcode update was back in 2015).

If you simply can't wait at all with Windows and no BIOS update is out, there's a hack way to update the microcode using the Linux update via windows: http://forum.notebookreview.com/thre...indows.787152/

Latest Intel Linux Microcode Data File available here: https://downloadcenter.intel.com/dow...-Data-File?v=t

Covers CPUs all the way back to 1995 Pentiums.

Hope that helps.

edit: If you run the powershell script, this is what you get to see. CVE-2017-5715 is Spectre Variant 2, CVE-2017-5754 is Meltdown.

Before microcode update:



After microcode update:



Spectre Variant 1 is always on as there is no performance impact. Spectre Variant 2 and Meltdown can access memory across processes/host so in a virtualized environment, it's pretty important. There is a performance hit on both of these, which is why it's suggested that you balance performance and security based on the uses of the systems. If you're certain that no untrusted code run on the system, you can get away with disabling 5715 and 5754 mitigations at the OS level. Spectre is very targeted so it's hard to reproduce without knowing a lot of environmental information, Meltdown is easy and needs to be patched if you run any untrusted code on an Intel CPU (AMD not affected).

[D'z Nutz]

This quote is hidden because you are ignoring this member. Show Quote

Those customer won't care. I tried it on them about a few Windows 2003 servers we still have.

Thank god at least they are on ESX so I don't have to worry about hardware end.

I got confirmation today we're pulling the plugs on those servers

[Swank]

This quote is hidden because you are ignoring this member. Show Quote

- 2018-01 update needed for Windows 7,8,10 (but make sure your AV scanner is also fixed before trying to update)

Unfortunately they pushed the patch out before this was common knowledge. If you're running Symantec AV (like I am) this will explain why your AV icon shows alerts, but it is purely cosmetic. When you open the interface itself there are no issues to report (at least in my case). A fix should eventually be pushed out.
https://support.symantec.com/en_US/a...ECH248552.html

[M.alex]

For large data operations this has to be > 30% performance hit.

I have a simplistic mysql database. just to pull out one year of data (20 million rows) with a select * from blah blah query went from 22 seconds to 28 seconds.

[Xtrema]

This quote is hidden because you are ignoring this member. Show Quote

For large data operations this has to be > 30% performance hit.

I have a simplistic mysql database. just to pull out one year of data (20 million rows) with a select * from blah blah query went from 22 seconds to 28 seconds.

Read someone reported 50% hit. Probably shit query or slow storage or both.

The more you have to go to storage or network to fetch stuff, the worst it get.

[Zhariak]

I can chime in and confirm I'm noticing massive performance decreases in a high performance spec'ed ESXi deployment (multiple servers, iSCSI 10G SAN). I'm also noticing performance hits on my customers (traditional, non-virtualized workloads).

This blows....

[firebane]

You know this is one of those times I think Apple has a bit of an edge.

Unless microcode and OS is updated in a way for the consumer to not know then you have a lot of vulnerable systems
Android is so segregated that there is simply no way for people to be updated

The server side will require a lot of work and I feel for you guys lol

[Xtrema]

This quote is hidden because you are ignoring this member. Show Quote

Thanks for passing along the info bud! I hope they release it as they say!

VMware released their microcode and ESXi updates yesterday. Deployed those. Haven't tested performance much, but had to do some emergency maintenance (due to a prolonged power outage last night). Definitely big performance change from what I'm used to seeing in my environment... Not sure if this was the ESXi and microcode updates, or if this was the MS updates inside of the VMs.

REALLY curious to see what the BIOS update will add to the mix, lmao...

Well HP pulled all updates for Gen9 servers too in addition of no release of Gen8.

There are talks about unexpected server reboots last week with the fixes.

[rage2]

This quote is hidden because you are ignoring this member. Show Quote

You know this is one of those times I think Apple has a bit of an edge.

Unless microcode and OS is updated in a way for the consumer to not know then you have a lot of vulnerable systems
Android is so segregated that there is simply no way for people to be updated

The server side will require a lot of work and I feel for you guys lol

Yes and no. Apple is probably the most susceptible to Spectre simply because there are less environmental differences to predict to make Spectre work. Fragmentation actually helps by lowering the chance of Spectre actually working on a target machine.

So Apple's systems, probably most vulnerable to Spectre, but patched the quickest.

[Zhariak]

This quote is hidden because you are ignoring this member. Show Quote

Well HP pulled all updates for Gen9 servers too in addition of no release of Gen8.

There are talks about unexpected server reboots last week with the fixes.

Oh snap! I didn't know that!

I know VMware pulled their microcode update too (causing PSOD on ESXi hosts). I have the VMware microcode deployed and no crashing yet, but I'm super pissed off at the performance loss.

[Xtrema]

Red Hat advisory released last night and it is undoing most if not all changes released in the last 2 weeks regarding this, including the Intel Microcode update.

[rage2]

This quote is hidden because you are ignoring this member. Show Quote

Red Hat advisory released last night and it is undoing most if not all changes released in the last 2 weeks regarding this, including the Intel Microcode update.

I only see the microcode rollback. KPTI is still there. Am I missing something?

[Xtrema]

This quote is hidden because you are ignoring this member. Show Quote

I only see the microcode rollback. KPTI is still there. Am I missing something?

Oops. You are right, only microcode.

[eblend]

We did our monthly reboot today, and my co-worker tells me everything feels much slower across both our Hyper-V hosts and ESXi 5.5. We only did the VM updates, so only the Windows patch went onto these VMs. Would the VM update make things feel slow on it's own? I was kind of under the assumption things would slow down when doing the hosts themselves, none of which have been done yet.

[Zhariak]

This quote is hidden because you are ignoring this member. Show Quote

We did our monthly reboot today, and my co-worker tells me everything feels much slower across both our Hyper-V hosts and ESXi 5.5. We only did the VM updates, so only the Windows patch went onto these VMs. Would the VM update make things feel slow on it's own? I was kind of under the assumption things would slow down when doing the hosts themselves, none of which have been done yet.

The OS patches will slow things down.
The hypervisor patch will slow things down further.
The microcode patch will probably slow things down even more (if someone could confirm this?)

I used to be running a powerhouse, and I'm really upset at the performance loss. I might even be pushing forward my infrastructure refresh (which technically I shouldn't have to do for another year).

[rage2]

The OS patches have 3 fixes, one for Meltdown (KPTI) and 2 for Spectre. KPTI slows down any kernel/user mode switching. One of the Spectre fixes has zero performance impact, and is enabled under all OS's. The second Spectre fixes requires microcode update, without it, it's not addressed. That has the potential to slow down as well. Both Meltdown and the second Spectre fix can be optionally disabled under Linux and Windows.

The official guidance from vendors is to evaluate your environment and see what mitigation steps you need based on your usage. Remember, to perform Meltdown and Spectre, you need to have the ability to run untrusted code on your OS as an attack vector. If you have users able to login to the box, they can run code to take advantage of this. This limits the exposure to just the OS, so someone would be able to dump memory across the OS. Basically a little worse than privilege escalation as it dumps memory contents that can contain unencrypted sensitive data. If you are locked down and have zero reasons for anyone to log in and run code, or you trust any user that has the ability to log in to not exploit it, you can get away with disabling the 2 mitigations.

The Hypervisor patch slows things down. It's KPTI to stop data leakage between VMs. There's no turning it off, this is a massively serious issue.

So depending on how much you trust your users and applications on your VM's, you can get away with just the Hypervisor patch, and walk away with a negligible performance hit that you probably won't even notice.

Hope that helps.

edit - as a side note, we have noticed zero performance impact on AWS. They were patching hosts to stop cross VM leakage starting in November, we were completely on patched hosts by mid-December based on our logs. Comparing metrics in the last 6 months, it's been flat.

[eblend]

This quote is hidden because you are ignoring this member. Show Quote

The OS patches have 3 fixes, one for Meltdown (KPTI) and 2 for Spectre. KPTI slows down any kernel/user mode switching. One of the Spectre fixes has zero performance impact, and is enabled under all OS's. The second Spectre fixes requires microcode update, without it, it's not addressed. That has the potential to slow down as well. Both Meltdown and the second Spectre fix can be optionally disabled under Linux and Windows.

The official guidance from vendors is to evaluate your environment and see what mitigation steps you need based on your usage. Remember, to perform Meltdown and Spectre, you need to have the ability to run untrusted code on your OS as an attack vector. If you have users able to login to the box, they can run code to take advantage of this. This limits the exposure to just the OS, so someone would be able to dump memory across the OS. Basically a little worse than privilege escalation as it dumps memory contents that can contain unencrypted sensitive data. If you are locked down and have zero reasons for anyone to log in and run code, or you trust any user that has the ability to log in to not exploit it, you can get away with disabling the 2 mitigations.

The Hypervisor patch slows things down. It's KPTI to stop data leakage between VMs. There's no turning it off, this is a massively serious issue.

So depending on how much you trust your users and applications on your VM's, you can get away with just the Hypervisor patch, and walk away with a negligible performance hit that you probably won't even notice.

Hope that helps.

edit - as a side note, we have noticed zero performance impact on AWS. They were patching hosts to stop cross VM leakage starting in November, we were completely on patched hosts by mid-December based on our logs. Comparing metrics in the last 6 months, it's been flat.

Thanks, it's been informative. I haven't had the chance to read much into it with work travel, so wasn't sure about all of this.

[Zhariak]

The HPe Security bulletin has been updated.

Got the update for my DL360p Gen8's yesterday. Also a couple weeks ago the updated microcode was available for some other newer Gen8's as well.

[Xtrema]

https://www.engadget.com/2018/05/21/...e-cpu-exploit/

Variant 4 just reared its ugly head. Another firmware, another performance hit.