CrowdStrike update causing massive global outages

Where goats go to escape
User avatar
Insane_Homer
Posts: 5529
Joined: Tue Jun 30, 2020 3:14 pm
Location: Leafy Surrey

A Crowdstrike update is causing DSOB in Windows 10/Server.

Major fucking shit show.

“Facts are meaningless. You could use facts to prove anything that's even remotely true.”
User avatar
Insane_Homer
Posts: 5529
Joined: Tue Jun 30, 2020 3:14 pm
Location: Leafy Surrey

“Facts are meaningless. You could use facts to prove anything that's even remotely true.”
User avatar
Insane_Homer
Posts: 5529
Joined: Tue Jun 30, 2020 3:14 pm
Location: Leafy Surrey

Potential fix

“Facts are meaningless. You could use facts to prove anything that's even remotely true.”
User avatar
Guy Smiley
Posts: 6810
Joined: Mon Jun 29, 2020 7:52 pm

It's global...

but they've released a fix. Uh huh...
User avatar
Sandstorm
Posts: 11862
Joined: Mon Jun 29, 2020 7:05 pm
Location: England

Oh no!! O365 might go down and I'll have to sit in the sunshine instead of working today. :thumbup:
User avatar
tabascoboy
Posts: 6882
Joined: Tue Jun 30, 2020 8:22 am
Location: 曇りの街

Had to pay for shopping with cash earlier, and the in-store cash machine ran out just after 9am...
User avatar
Sandstorm
Posts: 11862
Joined: Mon Jun 29, 2020 7:05 pm
Location: England

tabascoboy wrote: Fri Jul 19, 2024 8:34 am Had to pay for shopping with cash earlier, and the in-store cash machine ran out just after 9am...
Shit! I'm getting the train to London tomorrow, I have an online gift voucher for my lunch and no cash to buy cocktails afterwards. :cry:
User avatar
SaintK
Posts: 7411
Joined: Tue Jun 30, 2020 7:49 am
Location: Over there somewhere

Sandstorm wrote: Fri Jul 19, 2024 8:40 am
tabascoboy wrote: Fri Jul 19, 2024 8:34 am Had to pay for shopping with cash earlier, and the in-store cash machine ran out just after 9am...
Shit! I'm getting the train to London tomorrow, I have an online gift voucher for my lunch and no cash to buy cocktails afterwards. :cry:
:lol: It's a tough life!
Dinsdale Piranha
Posts: 1031
Joined: Mon Jun 29, 2020 10:08 pm

Providing the card machine is working at the pub later.....
User avatar
fishfoodie
Posts: 8845
Joined: Mon Jun 29, 2020 8:25 pm

Lot of very smug Unix guys around today ! :grin:

OS Monoculture is a very bad idea.
Dinsdale Piranha
Posts: 1031
Joined: Mon Jun 29, 2020 10:08 pm

Just had to use my emergency 20 quid note at Waitrose. Most cards not working.
duke
Posts: 829
Joined: Wed Jul 01, 2020 6:54 am
Location: Smallsbury

My card was fine at Waitrose and the petrol station, perhaps Wiltshire is still a few years behind the times
User avatar
Sandstorm
Posts: 11862
Joined: Mon Jun 29, 2020 7:05 pm
Location: England

Dinsdale Piranha wrote: Fri Jul 19, 2024 10:09 am Just had to use my emergency 20 quid note at Waitrose. Most cards not working.
Thank god I bought my rum, Fever Tree Mojito mix (fucking delicious BTW) and limes yesterday. :spin
User avatar
SaintK
Posts: 7411
Joined: Tue Jun 30, 2020 7:49 am
Location: Over there somewhere

duke wrote: Fri Jul 19, 2024 10:58 am My card was fine at Waitrose and the petrol station, perhaps Wiltshire is still a few years behind the times
All fine in M&S as well where I've just bought lunch.
User avatar
tabascoboy
Posts: 6882
Joined: Tue Jun 30, 2020 8:22 am
Location: 曇りの街

Different card payment systems everywhere I guess, some affected and some not. What a good idea it was for every bank in our town to close and take their ATMs away...
User avatar
fishfoodie
Posts: 8845
Joined: Mon Jun 29, 2020 8:25 pm

I'd like to think some management in my company would pause for a moment, & wonder if their decision to can most of their remaining senior IT Staff, to outsource them to India, was really such a great idea ?

But I doubt it.
User avatar
tabascoboy
Posts: 6882
Joined: Tue Jun 30, 2020 8:22 am
Location: 曇りの街



CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.

Image
Ovals
Posts: 1573
Joined: Mon Jun 29, 2020 9:52 pm

Going to the T20 at the Bowl (Hampshire) tonight - it's a cashless venue - hope it's all fixed by then or I'll have to take my own food and drink !
User avatar
fishfoodie
Posts: 8845
Joined: Mon Jun 29, 2020 8:25 pm

I don't think this is survivable for CrowdStrike as a company; they'll be sued into Chapter-11 by the end of next week.
User avatar
sturginho
Posts: 2594
Joined: Tue Jun 30, 2020 12:51 pm

tabascoboy wrote: Fri Jul 19, 2024 11:31 am

CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.

Image
It works on my machine! 🤣
User avatar
tabascoboy
Posts: 6882
Joined: Tue Jun 30, 2020 8:22 am
Location: 曇りの街

Ovals wrote: Fri Jul 19, 2024 11:40 am Going to the T20 at the Bowl (Hampshire) tonight - it's a cashless venue - hope it's all fixed by then or I'll have to take my own food and drink !
Seems all very hit and miss, my supermarket in town affected, another one not.
Ovals
Posts: 1573
Joined: Mon Jun 29, 2020 9:52 pm

fishfoodie wrote: Fri Jul 19, 2024 11:41 am I don't think this is survivable for CrowdStrike as a company; they'll be sued into Chapter-11 by the end of next week.
I assume they'd have insurance - but it might still be a killer blow as they'll have lost the trust of their customers.
Biffer
Posts: 10202
Joined: Mon Jun 29, 2020 6:43 pm

The Open Golf is entirely cashless but there don't seem to be any problems here.
And are there two g’s in Bugger Off?
User avatar
fishfoodie
Posts: 8845
Joined: Mon Jun 29, 2020 8:25 pm

Biffer wrote: Fri Jul 19, 2024 12:20 pm The Open Golf is entirely cashless but there don't seem to be any problems here.
It just depends on if your admins configured systems to pull updates during the window the poison update was on servers. It's why I used to distribute update time across different days, & times of day, so you wouldn't end up with everything going tits up at the same time.
User avatar
Sandstorm
Posts: 11862
Joined: Mon Jun 29, 2020 7:05 pm
Location: England

Bunch of customers planned work in Singapore, Prague and Dublin cancelled over the weekend due to the outages. Happy, happy days!!! :clap: :clap: :clap:
User avatar
Sandstorm
Posts: 11862
Joined: Mon Jun 29, 2020 7:05 pm
Location: England

fishfoodie wrote: Fri Jul 19, 2024 12:26 pm
Biffer wrote: Fri Jul 19, 2024 12:20 pm The Open Golf is entirely cashless but there don't seem to be any problems here.
It just depends on if your admins configured systems to pull updates during the window the poison update was on servers. It's why I used to distribute update time across different days, & times of day, so you wouldn't end up with everything going tits up at the same time.
I'd do it 3 days after everyone else, so they experienced the problem and a fix was issued by the vendor 1-2 days later....
Slick
Posts: 13517
Joined: Tue Jun 30, 2020 2:58 pm

This is the start of the bitcoin takeover. We can’t say we weren’t warned
All the money you made will never buy back your soul
User avatar
Tichtheid
Posts: 10654
Joined: Wed Aug 26, 2020 11:18 am

Anyone know how Liz Truss’s first day at Microsoft is going?
User avatar
JM2K6
Posts: 10127
Joined: Wed Jul 01, 2020 10:43 am

Slick wrote: Fri Jul 19, 2024 3:06 pm This is the start of the bitcoin takeover. We can’t say we weren’t warned
Famously secure and reliable, the crypto exchanges
User avatar
fishfoodie
Posts: 8845
Joined: Mon Jun 29, 2020 8:25 pm

JM2K6 wrote: Fri Jul 19, 2024 8:45 pm
Slick wrote: Fri Jul 19, 2024 3:06 pm This is the start of the bitcoin takeover. We can’t say we weren’t warned
Famously secure and reliable, the crypto exchanges
And all run by paragons of virtue .......
User avatar
ASMO
Posts: 5609
Joined: Mon Jun 29, 2020 6:08 pm

fishfoodie wrote: Fri Jul 19, 2024 12:26 pm
Biffer wrote: Fri Jul 19, 2024 12:20 pm The Open Golf is entirely cashless but there don't seem to be any problems here.
It just depends on if your admins configured systems to pull updates during the window the poison update was on servers. It's why I used to distribute update time across different days, & times of day, so you wouldn't end up with everything going tits up at the same time.
Good practice is to update test/dev servers first, then wait 1 ir 2 weeks before you do prodiction for precisely this reason. Thats what i get my team to do.
Biffer
Posts: 10202
Joined: Mon Jun 29, 2020 6:43 pm

ASMO wrote: Sat Jul 20, 2024 5:03 am
fishfoodie wrote: Fri Jul 19, 2024 12:26 pm
Biffer wrote: Fri Jul 19, 2024 12:20 pm The Open Golf is entirely cashless but there don't seem to be any problems here.
It just depends on if your admins configured systems to pull updates during the window the poison update was on servers. It's why I used to distribute update time across different days, & times of day, so you wouldn't end up with everything going tits up at the same time.
Good practice is to update test/dev servers first, then wait 1 ir 2 weeks before you do prodiction for precisely this reason. Thats what i get my team to do.
Difficulty is if it’s a critical security fix. We always used to take a huge amount of care with updates / upgrades but if something was a critical vulnerability, it was now a known vulnerability that bad actors would be checking out immediately. So at a major bank, you needed to do it quickly.
And are there two g’s in Bugger Off?
User avatar
Insane_Homer
Posts: 5529
Joined: Tue Jun 30, 2020 3:14 pm
Location: Leafy Surrey

LOL. The Crowdstrike CEO, George Kurtz, was the Mcafee CTO in 2010 when they did a similar... oh dear.
“Facts are meaningless. You could use facts to prove anything that's even remotely true.”
User avatar
PCPhil
Posts: 2590
Joined: Tue Jun 30, 2020 10:06 am
Location: Where rivers meet

We really shouldn't have left the trees.
“It was a pet, not an animal. It had a name, you don't eat things with names, this is horrific!”
User avatar
fishfoodie
Posts: 8845
Joined: Mon Jun 29, 2020 8:25 pm

Insane_Homer wrote: Sat Jul 20, 2024 8:58 am LOL. The Crowdstrike CEO, George Kurtz, was the Mcafee CTO in 2010 when they did a similar... oh dear.
So hes the source of my randomization of updates !

That was another complete failure of QA, luckily back then I had a bunch of CoOps to mobilize for rebuilds & recoveries
Rhubarb & Custard
Posts: 2440
Joined: Tue Jun 30, 2020 4:04 pm

PCPhil wrote: Sat Jul 20, 2024 9:59 am We really shouldn't have left the trees.
Or even the soup. A safer time when fires weren't possible
User avatar
Torquemada 1420
Posts: 12015
Joined: Thu Jul 02, 2020 8:22 am
Location: Hut 8

Sandstorm wrote: Fri Jul 19, 2024 8:23 am Oh no!! O365 might go down and I'll have to sit in the sunshine instead of working today. :thumbup:
Once you try the new version of Outlook, you might want to make that a permanent state of affairs. :sick:
Biffer
Posts: 10202
Joined: Mon Jun 29, 2020 6:43 pm

The Mercedes F1 team trying to fix their IT while wearing Crowdstrike sponsored shorts is quite funny
And are there two g’s in Bugger Off?
User avatar
Insane_Homer
Posts: 5529
Joined: Tue Jun 30, 2020 3:14 pm
Location: Leafy Surrey

Toto's still on hold with Crowdstrike support
PXL_20240720_104158838.jpg
PXL_20240720_104158838.jpg (833.42 KiB) Viewed 318 times
“Facts are meaningless. You could use facts to prove anything that's even remotely true.”
Slick
Posts: 13517
Joined: Tue Jun 30, 2020 2:58 pm

fishfoodie wrote: Fri Jul 19, 2024 9:07 pm
JM2K6 wrote: Fri Jul 19, 2024 8:45 pm
Slick wrote: Fri Jul 19, 2024 3:06 pm This is the start of the bitcoin takeover. We can’t say we weren’t warned
Famously secure and reliable, the crypto exchanges
And all run by paragons of virtue .......
For avoidance of doubt, this was in relation to DAC’s prediction
All the money you made will never buy back your soul
Post Reply