Page 1 of 2

CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 7:31 am
by Insane_Homer
A Crowdstrike update is causing DSOB in Windows 10/Server.

Major fucking shit show.


Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 7:31 am
by Insane_Homer

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 7:32 am
by Insane_Homer
Potential fix


Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 7:53 am
by Guy Smiley
It's global...

but they've released a fix. Uh huh...

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 8:23 am
by Sandstorm
Oh no!! O365 might go down and I'll have to sit in the sunshine instead of working today. :thumbup:

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 8:34 am
by tabascoboy
Had to pay for shopping with cash earlier, and the in-store cash machine ran out just after 9am...

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 8:40 am
by Sandstorm
tabascoboy wrote: Fri Jul 19, 2024 8:34 am Had to pay for shopping with cash earlier, and the in-store cash machine ran out just after 9am...
Shit! I'm getting the train to London tomorrow, I have an online gift voucher for my lunch and no cash to buy cocktails afterwards. :cry:

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 9:15 am
by SaintK
Sandstorm wrote: Fri Jul 19, 2024 8:40 am
tabascoboy wrote: Fri Jul 19, 2024 8:34 am Had to pay for shopping with cash earlier, and the in-store cash machine ran out just after 9am...
Shit! I'm getting the train to London tomorrow, I have an online gift voucher for my lunch and no cash to buy cocktails afterwards. :cry:
:lol: It's a tough life!

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 9:24 am
by Dinsdale Piranha
Providing the card machine is working at the pub later.....

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 9:56 am
by fishfoodie
Lot of very smug Unix guys around today ! :grin:

OS Monoculture is a very bad idea.

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 10:09 am
by Dinsdale Piranha
Just had to use my emergency 20 quid note at Waitrose. Most cards not working.

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 10:58 am
by duke
My card was fine at Waitrose and the petrol station, perhaps Wiltshire is still a few years behind the times

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 11:00 am
by Sandstorm
Dinsdale Piranha wrote: Fri Jul 19, 2024 10:09 am Just had to use my emergency 20 quid note at Waitrose. Most cards not working.
Thank god I bought my rum, Fever Tree Mojito mix (fucking delicious BTW) and limes yesterday. :spin

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 11:24 am
by SaintK
duke wrote: Fri Jul 19, 2024 10:58 am My card was fine at Waitrose and the petrol station, perhaps Wiltshire is still a few years behind the times
All fine in M&S as well where I've just bought lunch.

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 11:27 am
by tabascoboy
Different card payment systems everywhere I guess, some affected and some not. What a good idea it was for every bank in our town to close and take their ATMs away...

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 11:28 am
by fishfoodie
I'd like to think some management in my company would pause for a moment, & wonder if their decision to can most of their remaining senior IT Staff, to outsource them to India, was really such a great idea ?

But I doubt it.

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 11:31 am
by tabascoboy


CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.

Image

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 11:40 am
by Ovals
Going to the T20 at the Bowl (Hampshire) tonight - it's a cashless venue - hope it's all fixed by then or I'll have to take my own food and drink !

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 11:41 am
by fishfoodie
I don't think this is survivable for CrowdStrike as a company; they'll be sued into Chapter-11 by the end of next week.

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 11:44 am
by sturginho
tabascoboy wrote: Fri Jul 19, 2024 11:31 am

CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.

Image
It works on my machine! 🤣

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 11:56 am
by tabascoboy
Ovals wrote: Fri Jul 19, 2024 11:40 am Going to the T20 at the Bowl (Hampshire) tonight - it's a cashless venue - hope it's all fixed by then or I'll have to take my own food and drink !
Seems all very hit and miss, my supermarket in town affected, another one not.

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 12:19 pm
by Ovals
fishfoodie wrote: Fri Jul 19, 2024 11:41 am I don't think this is survivable for CrowdStrike as a company; they'll be sued into Chapter-11 by the end of next week.
I assume they'd have insurance - but it might still be a killer blow as they'll have lost the trust of their customers.

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 12:20 pm
by Biffer
The Open Golf is entirely cashless but there don't seem to be any problems here.

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 12:26 pm
by fishfoodie
Biffer wrote: Fri Jul 19, 2024 12:20 pm The Open Golf is entirely cashless but there don't seem to be any problems here.
It just depends on if your admins configured systems to pull updates during the window the poison update was on servers. It's why I used to distribute update time across different days, & times of day, so you wouldn't end up with everything going tits up at the same time.

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 2:30 pm
by Sandstorm
Bunch of customers planned work in Singapore, Prague and Dublin cancelled over the weekend due to the outages. Happy, happy days!!! :clap: :clap: :clap:

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 2:32 pm
by Sandstorm
fishfoodie wrote: Fri Jul 19, 2024 12:26 pm
Biffer wrote: Fri Jul 19, 2024 12:20 pm The Open Golf is entirely cashless but there don't seem to be any problems here.
It just depends on if your admins configured systems to pull updates during the window the poison update was on servers. It's why I used to distribute update time across different days, & times of day, so you wouldn't end up with everything going tits up at the same time.
I'd do it 3 days after everyone else, so they experienced the problem and a fix was issued by the vendor 1-2 days later....

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 3:06 pm
by Slick
This is the start of the bitcoin takeover. We can’t say we weren’t warned

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 4:29 pm
by Tichtheid
Anyone know how Liz Truss’s first day at Microsoft is going?

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 8:45 pm
by JM2K6
Slick wrote: Fri Jul 19, 2024 3:06 pm This is the start of the bitcoin takeover. We can’t say we weren’t warned
Famously secure and reliable, the crypto exchanges

Re: CrowdStrike update causing massive global outages

Posted: Fri Jul 19, 2024 9:07 pm
by fishfoodie
JM2K6 wrote: Fri Jul 19, 2024 8:45 pm
Slick wrote: Fri Jul 19, 2024 3:06 pm This is the start of the bitcoin takeover. We can’t say we weren’t warned
Famously secure and reliable, the crypto exchanges
And all run by paragons of virtue .......

Re: CrowdStrike update causing massive global outages

Posted: Sat Jul 20, 2024 5:03 am
by ASMO
fishfoodie wrote: Fri Jul 19, 2024 12:26 pm
Biffer wrote: Fri Jul 19, 2024 12:20 pm The Open Golf is entirely cashless but there don't seem to be any problems here.
It just depends on if your admins configured systems to pull updates during the window the poison update was on servers. It's why I used to distribute update time across different days, & times of day, so you wouldn't end up with everything going tits up at the same time.
Good practice is to update test/dev servers first, then wait 1 ir 2 weeks before you do prodiction for precisely this reason. Thats what i get my team to do.

Re: CrowdStrike update causing massive global outages

Posted: Sat Jul 20, 2024 5:50 am
by Biffer
ASMO wrote: Sat Jul 20, 2024 5:03 am
fishfoodie wrote: Fri Jul 19, 2024 12:26 pm
Biffer wrote: Fri Jul 19, 2024 12:20 pm The Open Golf is entirely cashless but there don't seem to be any problems here.
It just depends on if your admins configured systems to pull updates during the window the poison update was on servers. It's why I used to distribute update time across different days, & times of day, so you wouldn't end up with everything going tits up at the same time.
Good practice is to update test/dev servers first, then wait 1 ir 2 weeks before you do prodiction for precisely this reason. Thats what i get my team to do.
Difficulty is if it’s a critical security fix. We always used to take a huge amount of care with updates / upgrades but if something was a critical vulnerability, it was now a known vulnerability that bad actors would be checking out immediately. So at a major bank, you needed to do it quickly.

Re: CrowdStrike update causing massive global outages

Posted: Sat Jul 20, 2024 8:58 am
by Insane_Homer
LOL. The Crowdstrike CEO, George Kurtz, was the Mcafee CTO in 2010 when they did a similar... oh dear.

Re: CrowdStrike update causing massive global outages

Posted: Sat Jul 20, 2024 9:59 am
by PCPhil
We really shouldn't have left the trees.

Re: CrowdStrike update causing massive global outages

Posted: Sat Jul 20, 2024 10:00 am
by fishfoodie
Insane_Homer wrote: Sat Jul 20, 2024 8:58 am LOL. The Crowdstrike CEO, George Kurtz, was the Mcafee CTO in 2010 when they did a similar... oh dear.
So hes the source of my randomization of updates !

That was another complete failure of QA, luckily back then I had a bunch of CoOps to mobilize for rebuilds & recoveries

Re: CrowdStrike update causing massive global outages

Posted: Sat Jul 20, 2024 11:40 am
by Rhubarb & Custard
PCPhil wrote: Sat Jul 20, 2024 9:59 am We really shouldn't have left the trees.
Or even the soup. A safer time when fires weren't possible

Re: CrowdStrike update causing massive global outages

Posted: Sat Jul 20, 2024 1:55 pm
by Torquemada 1420
Sandstorm wrote: Fri Jul 19, 2024 8:23 am Oh no!! O365 might go down and I'll have to sit in the sunshine instead of working today. :thumbup:
Once you try the new version of Outlook, you might want to make that a permanent state of affairs. :sick:

Re: CrowdStrike update causing massive global outages

Posted: Sat Jul 20, 2024 2:28 pm
by Biffer
The Mercedes F1 team trying to fix their IT while wearing Crowdstrike sponsored shorts is quite funny

Re: CrowdStrike update causing massive global outages

Posted: Sat Jul 20, 2024 3:05 pm
by Insane_Homer
Toto's still on hold with Crowdstrike support
PXL_20240720_104158838.jpg
PXL_20240720_104158838.jpg (833.42 KiB) Viewed 319 times

Re: CrowdStrike update causing massive global outages

Posted: Sat Jul 20, 2024 4:48 pm
by Slick
fishfoodie wrote: Fri Jul 19, 2024 9:07 pm
JM2K6 wrote: Fri Jul 19, 2024 8:45 pm
Slick wrote: Fri Jul 19, 2024 3:06 pm This is the start of the bitcoin takeover. We can’t say we weren’t warned
Famously secure and reliable, the crypto exchanges
And all run by paragons of virtue .......
For avoidance of doubt, this was in relation to DAC’s prediction