It’s the late 90s and I was working in IT for a small (200 employees) company.
One of my tasks was performing daily data backups, which ran for about four hours and started after everyone else had left for the day.
We had an intercom system in the building, where if you picked up your phone and dialed a specific extension, you could page someone, or make announcements so everyone could hear.
One night I was in the process of running Ethernet cable from switches on the second floor down to individual terminals on the first floor, so there was a lot walking around the building.
I like to have music when I’m working, but I didn’t have an MP3 player at the time, so I thought… “I have music on my PC and no one is around. Why don’t I just…”, and I went around and cranked the volume knob on every ceiling intercom speaker in the whole building. Then I dialed into the intercom from the phone in my office, put the handset up as close as I could to my PC speakers, and boom, music throughout the building.
Cut to a few hours later, it’s late, I’m tired, the backup is finished, so I close down the system, hit the lights, lock the doors, and go home.
The next day, I come in, not thinking about anything from the night before, and get to work. About 11:00am, the owner’s secretary got on the intercom to page someone… and blew everyone out of their seats. There was an almost collective, “What the fuck!!!” from everyone, and an “Ohhh, shiiit” from me, because of course I immediately knew what happened.
I ran around like a madman, turning the volume down on all of the speakers that I had forgotten the night before, while saying, “No idea… Yeah it’s crazy… No no, it’s fixed now…”
No one ever found out exactly why the volume was so loud that morning.
Fortunately not as serious as other comments. For context, I have autism, and I frequently stim by making random noises.
One day I was at a global online meeting with very important people. I thought I was muted, so I said “MEOW”.
Suddenly everybody goes quiet and somebody says “did anybody say ‘meow’?”
I quickly turned off my microphone, and fortunately nobody mentioned that again.
Not mine, but another’s. She stating in her Foghorn Leghorn voice “I want to say “I can’t hear you over your hair. It’s too bright!!” to her”. As I round the corner in my short Spiked Blazing Orange hair. How embarrassing for someone.
Not anything destructive (to the Company) per se, but seeing a meeting with a senior and a member of HR suddenly pop up on my calendar after weeks of waiting in limbo to be placed on a new project was definitely an OH SHIT moment.
Yep, I was made redundant 🙃
I was working as a short order cook back in the 80s. One night after closing we were having a few beers as a farewell to a co-worker. We wanted some music so we taped our walkman earbuds to the intercom microphone. Came in the next day, our busiest day of the week and found out we had blown the amplifier.
I worked for the railways, in commuter trains. My job was to harass innocent passengers by walking back and forth and enabling them to buy a ticket should they need one. And at certain routes also check the tickets of every new passenger.
It was nice using a dialect from eastern Finland, the Savo dialect for that. It has a tendency to change plain vowels for diphtongs. It adds a kind of a “wee-wow” twang to the language. For example the word for “closet” is “kaappi” in standard Finnish, “kuappi” in the dialect spoken in Savo. The “aa” is just a long “a”, sound. Pretty much the same as the “a” in the English word “car”. And “ua” is pretty much like “oi” in “choir”. Imagine pronouncing the word “car” with the “a” replaced with the “oi” sound of “choir” and you might get an idea of how that crazy dialect likes to mangle every damn word. And the “ar” in “garlic” would be pronounced with the same change as well. Etc.
And maybe at this point it’s worth mentioning that I still had a fully male gender expression at that point.
So, we have just left the station of Haarajoki. In the Savo dialect that would be “Huaarajokj”.
I enter the part of the wagon through its end door. I start shouting “Haarajoki passengers, tickets please!”, but. But.As it happens, “huaara” is how they would pronounce also the word “huora” over in Savo. And that happens to mean “whore”. Luckily I noticed. So, I managed to stop myself from uttering anything further. Unfortunately, I only managed to stop after having already said the first half of that word, “huaarajokj”: Yes, “huaara”.
In other words, I entered the passenger wagon through the end door in the railway worker’s uniform, immediately bellowing “WHORE!”, and then shutting up.
Definitely an “oh shit” moment!
After that:
Me: “Hm. This station is not terribly compatible with the Savo dialect, apparently.”
A female passenger: “You should have continued to the end, would have gone much less badly”.
Me: “Yup!”Tooks out search on Wikipedia. I exposed a path that can perform an unconstrained Powerset Construction concurrently on 7 servers. It filled up memory and died. User retried. While cluster OOMing.
ELI-non-IT this “oh shit” moment?
Wikipedia has a search feature. Not many people use it, but Wikipedia is huge, so enough people do. Back then, a couple dozen full text searches a second. And maybe 100 quick prefix searches.
Anyway, one thing I made for it was a way to search the wiki text with regular expressions. It’s useful for maintainers.
Wikitext is the funny markup language Wikipedia uses. If you know html or md it’s like that, but it can kind of “call” other pages. Whatever. Doesn’t matter. It’s text. And people have strong opinions about it.
You can see it by clicking “edit” (sometimes looks like a pencil) and then clicking “edit without logging in” then “source editing”. Those lovely boxes at the top for a person are made with something like
{{Infobox person | name = Douglas Adams | birth_place = [[Cambridge]], England }}Imagine you have 1000 articles that use
birthplaceinstead ofbirth_place. Both work, but it’s driving you and all your editor friends crazy there are two ways to do it. This is believable. Trust me.You want to find every page that has an
{{Infobox personfollowed bybirthplace. Then fix them. The nerds who came before us invented a language to ask that question called “regular expressions”. The one for this looks like\{\{Infobox\wperson.+birthplace. Maybe. Nerds will know regular expressions are bad for this. But nerds will also know that regular expressions being bad has never stopped anyone from using them anyway.One way to run these regular expressions is to convert them into an https://en.wikipedia.org/wiki/Nondeterministic_finite_automaton . You build these in memory and they are not big. But you can’t really run them directly. They are pretty and fairly easy to read once you get used to them. But you can’t easily ask “does this match this text”. At least, not with the tools I had.
I could only run https://en.wikipedia.org/wiki/Deterministic_finite_automaton . It’s deterministic! Much nicer. And you can go from a nondeterministic one to a deterministic one. Easy. Our forenerds solved the problem. You use a https://en.wikipedia.org/wiki/Powerset_construction .
The trouble is, it can make very very very big deterministic finite automata. Like, if there are 3 states in the nfa you can get and 8 state dfa. 4 is 16. 5 is 32. In the worst case. Usually you get much better. But a fairly big but not super frightening regular expression can turn into a big nfa. And the dfa would need more states then there are grains of sand on earth. Too big for computer.
So, user asked for this regular expression. The search servers tried to convert it, and filled up their memory and died. They ran Out Of Memory. OOM. The usual thing is you copy the full memory so you can look later and start the server again.
But that can take time. And sometimes people are confused and haven’t set up the restart to be automatic. And they can’t find you. So search stays broken for a while.
Spot welded a screwdriver to the metal sidewall when I slipped trying to discharge a capacitor. At least I thought to wear gloves.
Tower cleared me to position and hold (back before it was called line up and wait), then cleared a 737 to land on the runway I was sitting on.
Now it was day VFR conditions and my little ship was painted bright red and white, I wasn’t exactly camouflaged, but I called the tower with my entire tail number and position. RALEIGH TOWER, NOVEMBER ONE TWO THREE ALPHA BRAVO HOLDING POSITION FIVE RIGHT.
Made a great lesson for my student, the tower WILL forget about you and clear a jet 500 times your mass to land on you.
I put the localization files in the wrong folder. Thankfully, we hadn’t pushed anything to prod yet, but we were trying to figure out why everything kept crashing even though our checking tool was giving the all-clear.
It was checking everything except the localization folders.
Reading through the comments, I remembered my first day on my second ever job, where we were instructed to turn off computers and power strips too, at the end of the day. When I turned mine off at the end of the day, I found out other power strips were using mine, so I turned off other people’s machines. Unfortunately, the guy that gave us the instructions was first to leave in the evening and hadn’t have his machine turned off that way.
Also remembered that I once broke production server (maybe it didn’t get to production, can’t remember) by adding a single line of log, I didn’t notice that what I was logging was computationally intense, so it halted everything eventually, but it was barely reviewed because it’s just one log line
Had a content filter server I had built. Dansguardian.
Installed some new packages, which I had wgot (joking) at / as root. Fuck, I was a dumbass kid.
After I installed the packages, I created the folder /installs and then: holy shit, mv is naturally recursive?!
mv * /installs
Well, let’s ls to see if the root directory is clean.
ls not found.
Um… ok, this is a call center, seems to still be working as everything is still in RAM, but as it is working as a proxy for content filtering, you could lose the company MILLIONS, so think fast!
I rebuilt another server, swapped it with ninja speed, fixed the other server, and swapped it back with ninja speed.
And everyone asked me if I was ok. “Nothing. I mean, yeah, good.”
edit: mv is recursive by default, in contrast to cp, which requires -R. Scary lesson to learn on the job.
edit 2: just in case someone doesn’t understand this, I moved the entire operating system into a subfolder /installs in a production environment…
If the computer had rebooted, no one would have internet at all, as DNS would fail. Fortune 500 company. Ah damn, that was some stress.
“Hey, is everything working alright?”
“Yeah.”
“Hey is everything still working? Everything good?”
“Yes, why do you keep asking that?”
No raisin.
The big annual release was coming soon and I was leading a small group of testers with after hours testing of a feature that was my responsibility to cover. It was already around 8pm and we’d been working for hours. Someone came and informed us that a specific device was incorrectly configured, thus invalidating all of the tests we’d run thus far. It was mine. What an asshole.
I told everyone to give me their devices and go home. I stayed until 2-3am running all of the tests myself. My manager video called me a couple of times and he had a good sense of humor about it, I think because I took responsibility for it and didn’t burden others with my mistake. I really felt terrible but after I got the report in before the deadline later that morning, no one ever gave me a hard time about it and it just was like it never happened.
Not trying to deflect or make excuses and owning a mistake is like a super power. An imprortant lesson that has basically never failed me when I’ve screwed up at work.
Seconded on owning a mistake. It might get you reprimanded, but it usually doesn’t get you fired.
If it gets you fired the company culture turns toxic very quickly. I’ve worked for a place like that once and quit when I saw that bs.
I was asked to provide technical on-site support at an event, because I knew the underlying server system under the software that project management had found to run this customer event.
They had also hired a bunch of laptops with wifi and a router for this, which was rather expensive back then (you could have bought a small car for that rent fee alone!), so a lot was hanging on this to be a success.
Of course, I knew about nothing what this job was about until I sat in the car with the project manager on our way to the first event. My reaction was: But that software cannot do what you just told me! Oh shit, indeed.
When we arrived in the evening before the first event, the server, wifi, and one of the laptops was set up before anything else, and I went to work writing a rudimetary software from scratch that would do what the customer wanted, and finally hit the bed around sunrise.
The event was a big success in all venues on the trip. The customer was extremely happy with the acceptence of the setup by the audience and the results they got.
I used to work at a big cable company whose name rhymes with bombast. One day I (M) found myself in a meeting with a bunch of female vice-presidents (Bombast has thousands of vice-presidents). The presenter couldn’t get her laptop to connect to the big TV so I pulled out one of my gadgets and got her set up. She said “wow, you’re pretty handy to have around.” My brain decided that it would be a clever thing indeed to say “well, I’m pretty well-endowed in the dongle department” but when I started to say it my brain realized how completely insane a comment it was. So my brain cut me off and I ended up just saying “well, I’m pretty well-endowed” which was not exactly any better.
To their credit everybody ignored this except for one woman who actually guffawed briefly. I wasn’t fired immediately so there’s that.
To be fair telling someone “wow, you’re pretty handy to have around” is opening the door for all kinds of inappropriate comments…
Decade or so ago, logging into a work computer and suddenly the screen is just tiled with the word “virus”, but a bit janky looking like it was a screenshot of the word that was enlarged and tiled with no thought to whether the right side of the image lined up with the left and such - it was a very 90s cheesy movie like depiction of what happens to your comp when you get a virus, like you’d expect the next scene to be some shady hacker on the other end exclaiming “I’m in!”
Then again, I don’t know jack fuck about viruses, and the machine is behaving super weird, so I unplugged the internet cable from the back, and called my manager over. He had a similar “…is this for fuckin real?” reaction that I did. Mind you, this is a hospital, so if one of our computers is fucked, this is a big deal for potentially leaking patient info, so even though it had that kind of slapstick appearance, no one was fucking around with it: so, manager pushed it up the chain, and after a few mins a fucking SWAT team of IT guys comes in and starts doing their own assessment.
After some deliberation, they start troubleshooting - keystrokes and whatnot don’t seem to be responsive. One of them eventually reaches back behind the monitor and ‘poof’ the screen pops back to the normal login screen.
So, that facility had a fancy custom login screen, which was some generic desktop background type scenery and a wall of text that none of us ever put any thought into. One of the snipits on that wall of text was instructions on what to do if you suspect your machine has virus (which was basically what I did - leave it running, unplug it from the internet, and call IT… hooray, I’m not completely stupid!). What ended up being the cause of the “virus” tiling was the fucking VGA cable slowing worming its way out of the back of the monitor - when I got to it, that thing was hanging on by a thread, and touching the desk it was on was just enough of a bump to make it cross that line between having enough of a connection to do its job vs not. That micrometer of pull of on that cable wasn’t enough to completely cut its signal, but it wasn’t enough to produce a normal display either: it just picked a random spot on the screen, and filled the monitor with that repeating image. The spot it just happened to do this to was the word “virus” in the custom login text explaining what to do if you think you have a virus, lol.
TLDR, got my management chain from my dept to the top dogs running the hospital, and the entire fucking IT department to swarm to my unit over a potential data breach that ended up being a wiggly VGA cable.
Whoops.
Better safe than cryptolockered.
