A few days ago, a client’s data center (well, actually a server room) "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
@stavvers (round here RM are generally good, though they occasionally "accidentally open" the odd padded envelope of Lego bits and lose some. No idea what they think it might be but it's a pattern)
Evri, on the other hand... I've never wanted a shed roof so much, just so I'd know where my stuff went >:(
@stavvers why do you look to have stolen a bunch of post plus some post-office parcel-sizing/weighing kit?
also your home decor appears to be very open-plan sorting-officey and I'm not sure you need partitions in the home.
I can see the delivery folks' confusion, frankly
/s
New stickers came in! Well in time for @eth0 😈
At long last, I have created the Torment Nexus sticker from the obscure Don't Create the Torment Nexus meme
"Allow 'Google Chrome' to find devices on local networks?"
This prompt is singularly irritating. There is no indication of what causes it to pop. There is absolutely no way a normal user will be able to answer this intelligently.
Is it an indicator of an attack that was just blocked? If I say yes do I immediately get CSRF'ed?
UX is security, people. C'mon.
For more (I think): https://developer.chrome.com/blog/local-network-access
397) Birbbot. Entice birds to your bird table with this convincing facsimile of an avian pal. #SmallRobotsRemastered
Well, Holiday Inn is off the list when I’m on tour
https://www.manchestereveningnews.co.uk/news/greater-manchester-news/inhumane-homeless-people-turned-away-33179421.amp #hotel #homelessness
@robinince "I know you are from the street"
yeah, sorry about my lack of teleporter, hadn't realised it would be an issue for a hotel :/
@jschauma I thought we attached a wooden block to the end of a chain and swung it at them?
either way gets them to go quiet I guess
Steve:
frequent overthinker, compulsive fixer, digester-then-explainer, "why?" question relishing father, minor-irritant partner, excessive disassembler, original-form hacker, high-efficiency googler, borderline-competent car-fixer, expert-level car-breaker, faster-by-qwerty communicator, indiscriminate photo-taker, Leatherman owner.