78 comments

  • A_D_E_P_T 1 day ago
    - Cards not working.

    - Physical branches closed or unresponsive.

    - Online banking down.

    This exact same thing happened in Europe with Revolut late last month. It was resolved in about 12 hours: https://www.revolut.com/blog/post/20octoutagetechnicalovervi...

    In that case, it was a Google Cloud outage. Something similar (database outage) could be the culprit here.

    • graemep 1 day ago
      Lots of banks have been down at various times, but its a lot more of a problem when multiple major banks are down at the same time.
      • busymom0 1 day ago
        Here in Canada, about 4 years ago, we had an outage where cards weren't working for many hours at any store due to a huge Rogers outage.
        • Scoundreller 17 hours ago
          When your redundant multi-homed links work over the same mpls circuits…
      • nektro 1 day ago
        article says it's mainly Zelle
        • Sebguer 1 day ago
          Zelle has explicitly said they're not experiencing an outage. And it being 'mostly' Zelle would not affect in-person branches.
    • alt227 1 day ago
      Its wild that even our banks are now reliant on cloud providers.
      • zenexer 1 day ago
        It doesn’t seem like a cloud provider was at fault:

        > “JPMorgan Chase has experienced a temporary issue related to a software bug in an on-premise hardware system — it is not cyber-related,” said Joe Evangelisti, spokesman for the bank.

        • JoBrad 1 day ago
          Nominative determinism ftw.
          • hydrogen7800 4 hours ago
            Cecil Fielder was the first time I noticed this as a kid. I love that there is a name for this.
          • anthonyrstevens 1 day ago
            I'm glad his name isn't Joe Leier
        • mkehrt 18 hours ago
          What the heck does "not cyber-related" mean? It's a bug.
          • Melatonic 16 hours ago
            They mean cyber security - as in it wasn't a hack
          • MASNeo 17 hours ago
            I wondered he same thing and smiled. The poor spokesperson was probably given some text to read. I feel for him.
      • socalgal2 22 hours ago
        Is it? I expect banks to have some of the worst IT infra out there. Partly because they are stuck with legacy systems. (Cobol based stuff for example). They are stuck because interoperating between banks is a hard problem. But also, because they likely see IT as an expense to be minimized instead of one of their core needs.
      • Buttons840 1 day ago
        It's wild that they pay like shit and attract 2nd rate programmers.

        Eh, actually, I don't know if that's true about 2nd rate programmers, in a lot of ways they're probably better, but they do get paid less.

        I realized this while working at a startup. The startup had a webpage that worked half the time--that's it, that's why we had funding. Well, that and the founder was going around telling investors we had fancy AI algorithms, which were actually mostly manual human work.

        Meanwhile my friend working at the bank made a lot less.

        • solraph 22 hours ago
          > It's wild that they pay like shit

          Interestingly, Australia banks so well compared to everyone else (with a few exceptions like Amazon, Atlassian, etc) that workers at them frequently refer to having "golden handcuffs".

        • nunez 23 hours ago
          JPM actually pays okay for engineers and tech folks, though not as well as hedge funds or startups.
        • reaperducer 1 day ago
          [flagged]
          • bbg2401 1 day ago
            I’ve always found it curious that FAANG/California-based employees receive comments of “hugs” from HN during periods of downtime, while those outside of the bubble receive only mild slander. Of course, no suggestion that you specifically have engaged in the former.
            • jeremyjh 1 day ago
              Also the FAANGwits write engineering blog articles about how fucking stupid they are how shit their incident management culture is and it gets hundreds of upvotes.
    • krunck 1 day ago
      I've got my cash stash.
      • deadnomore 1 day ago
        I’ve got hella weed
      • maerF0x0 1 day ago
        That will work for the first week to month of a disaster. Once people realize it's bigger then dollars will rapidly lose value compared to direct barter for useful things.
        • plorkyeran 1 day ago
          Having something that works for a week long disaster is really useful!
        • hgomersall 17 hours ago
          When this happened in the past, the banks were simple sidestepped in the credit issuance process, with pubs and shops taking over, and lots of cheques being issued: https://en.wikipedia.org/wiki/Irish_bank_strikes_(1966%E2%80...
        • reaperducer 1 day ago
          Guess how I know you've never been through a major hurricane.
          • dpoloncsak 1 day ago
            I don't agree with OP's point, but comparing the damage to the dollar that a localized hurricane and an extended national bank-outage would have is not an apples-to-apples comparison.

            In a hurricane, the dollar keeps it's value, as others outside of the affected region are still using it to trade. This may not be true in the event of a national/global outage....not to mention the whole 'trust in the USD' aspect

          • maerF0x0 1 day ago
            What is your point, ultimately? You made a statement about my experiences with a very specific kind of disaster, but didnt say anything useful to rebut my point.
            • pjc50 12 hours ago
              Hurricanes are a real risk that people have experienced. A collapse in the value of the dollar which the US is unable and unwilling to fix is in the realm of fantasy.
            • AlecSchueler 16 hours ago
              To be fair you gave no rationale for your original assertion.
      • sailfast 1 day ago
        I don’t think cash is gonna help if the entire monetary system stops working - gold maybe?
        • hbcdbff 1 day ago
          If the “entire monetary system” stops working, then things are fucked enough already that gold is essentially the same in terms of utility as cash would be, in that its only immediate value is its scarcity.

          Things like food, water, basic sanitary items, would be bartered for.

          • jsLavaGoat 1 day ago
            Diesel. And .223 and 9mm. And the addresses of the 10 nearest preppers.
            • darth_avocado 1 day ago
              It’s funny how people just automatically go down “I have a gun and I’ll be good scenario”, meanwhile every natural disaster in modern history suggests that in crisis, people come together to help each other instead of threatening each other with guns.
              • Buttons840 1 day ago
                I realized this watching Gaza. Here I am watching a people that more than a few voices would tell me are sub human, but every time I see video of them dealing with the destruction of literally every single building within sight, they seem to be acting like regular humans, not savages.

                No doubt there have been some crimes and things done in desperation, but humans don't just instantly devolve into roving gangs that kill on sight.

              • bigDinosaur 1 day ago
                People often come together initially, but it's a dramatic simplification to think that that implies that after the immediate issues are resolved (e.g. outright starvation) that competing groups don't often become extremely antagonistic.

                You'd still not want to be a lone wolf type with a gun, though, you really want to have a group in all cases.

              • trollbridge 1 day ago
                The people coming together helping each other can also wield guns far more effectively than some lone thief, who will be universally despised and viewed as a highly immoral agent.
              • addaon 1 day ago
                > people come together to help each other instead of threatening each other with guns

                People, collectively, come together. There's always outliers -- we call them assholes. It's nice that some folks are so clear on what side of line they intend to stand ahead of time -- pay attention to what your neighbors are saying, so you know which ones are the assholes.

              • DANmode 1 day ago
                It’s definitely both, what books did you read?!
            • delichon 1 day ago
              Careful, the preppers I know think your 9mm is adorable.
              • segfaultex 1 day ago
                9mm is the most common sidearm cartridge on the market. Works fine for the guys in Delta and DEVGRU.
              • dudefeliciano 1 day ago
                They’re talking about valuable goods to barter with in an apocalyptic scenario, they did not say they would go raid the preppers. At least that’s how I understood the comment.
                • ceejayoz 1 day ago
                  I don't read it that way.

                  Anyone known to be a prepper is a really great target in an apocalypse. One-stop shopping and typically doesn't work well with others!

                  • dudefeliciano 1 day ago
                    Exactly my point, which is why having the addresses of 10 preppers would be more valuable than gold. So you can give the addresses to some badass willing to risk his life in exchange for goods.
                    • kipchak 1 day ago
                      Why would they give you the goods once they took them by force from someone else?
                      • qlte 1 day ago
                        Access to the next prepper cache on the list.
                        • kipchak 10 hours ago
                          It seems like they'd be much better off treating the list the same way they treated the prepper's resources and taking it by force.
              • ndesaulniers 1 day ago
                That's all right. We'll let this .38 special do the talking.
              • ceejayoz 1 day ago
                Those preppers couldn't even handle two weeks staying home during COVID.

                All hat, no cattle.

                • DaSHacka 1 day ago
                  I think you may have an inherent selection bias there
                • chasd00 1 day ago
                  Man that’s the truth, people will just do what they’re told by authority. COVID proved it over and over.
              • cynicalkane 1 day ago
                All firearms are deadly. Your prepper friends are not serious if they think certain kinds of bullets are too adorable to kill them.

                .223, while we're on the subject, is used in the AR-15.

                • AngryData 1 day ago
                  They are all deadly, but out of any long gun besides .22, .223 is pretty much the weakest. The entire benefit to using .223 or 5.56 is that they are light so you can carry more ammo to enable suppression fire after traveling 20+ miles on foot.

                  If we were to judge people by the guns they carry, I would rank ar-15 carriers the least worrisome. They would be far less likely to be a decent shot on average, they are the least likely to know anything about actual firearm tactics and try some hollywood or COD:MW run and gun, and will likely be the first ones shot when they decide some 70 year old farmer or hunter is an easy target who suddenly blasts them from a dark window or tree stand.

                  • fredophile 15 hours ago
                    Or they're a veteran using a weapon system they're familiar with and trained on for years.

                    In a collapse of civilization scenario one of the biggest advantages of carrying an AR is how popular they are. It might not have the range of a hunting rifle in 308 lapua but you'll be able to find more bullets.

                    • AngryData 7 hours ago
                      I would hope a veteran would realize that group military tactics don't work without the group military and they don't need hundreds of rounds at a time.
                • doxymazus 1 day ago
                  556 is used in real ar15s. 223 is for people who couldn’t afford the m4 bolt carrier group and upper. Aka: poseurs.
                  • beAbU 18 hours ago
                    Not sure I understand, 5.56 and .223 are basically the same, and even though not advised you can fire a round from one in the chamber of another.

                    Why would there be a significant price difference between the bolt carriers and uppers for the two?

                    • kakacik 16 hours ago
                      1 is military aimed, 1 is civilian. Less power, less muzzle speed, less destruction on impact. One shouldn't fire 5.56 from purely .223 gun, its not specced for higher pressures. May still work or may blow up literally into your face.
                      • beAbU 6 hours ago
                        Everything I've read says pressures and energies are basically the same for similar sized projectiles and powder mass. In fact, according to wikipedia the external dimensions of the two cartridges are identical, only difference being the part where the rifling begins is sharper in a .223 compared to the 5.56, which can lead to overpressure situations. Albeit nothing that might cause the gun to blow up.

                        Still does not explain the claimed price difference you speak of, and why that makes .223 owners poseurs compared to 5.56 owners.

                  • jsLavaGoat 1 day ago
                    One kind of person who won't matter when SHTF are gear snobs.
                    • darksim905 20 hours ago
                      Oddly enough, they may actually matter more than people give them credit for. If they know enough about ordnance and weapons to know what works for a given scenario or someone's armor deployment, they could do very well with very little.
                    • doxymazus 18 hours ago
                      [dead]
              • islandfox100 1 day ago
                "it matters not whether a bullet is 9mm or .50, as long as it hits enemy soft tissue"

                -Deng Xiaoping (paraphrased)

                • jsLavaGoat 1 day ago
                  What matters is that its hugely common and works in easy to repair weapons. In an actual scenario where you would need such things, having a fancy ass rifle with only a few parts to fix it is fucking useless. AR-15s and 9s aren't the best weapons out there but they are ubiquitous.
              • lenerdenator 1 day ago
                I honestly do wonder, has there ever been a time where two parties have been in a confrontation and one party pulls out a 9mm, and the other party reacts to it like John Candy's character in National Lampoon's Family Vacation reacts to the pellet gun?
          • somenameforme 21 hours ago
            One huge difference with gold. In this sort of SHTF scenario it's completely conceivable that any given fiat currency remains worthless after things clear up. By contrast gold will, sooner or later, be exchangeable for whatever the new standard of exchange becomes.
        • francisofascii 1 day ago
          It is pretty common to for credit card systems to go offline temporarily at grocery stores and restaurants. Always a good idea to have cash.
          • SoftTalker 1 day ago
            I haven't carried enough cash to pay for groceries or dinner at a restaurant in over 10 years.
            • olyjohn 1 day ago
              You don't have to carry it with you to keep some available for emergencies.
            • ericpauley 1 day ago
              If the restaurant card system goes down I’m leaving and telling them to bill me.
              • unparagoned 8 hours ago
                It’s for like a week shutdown. No one is going to let you eat without cash upfront
              • ibejoeb 1 day ago
                Chase still offers embossed cards that can be imprinted. I think amex business cards, too. They can still mail in the drafts.
                • jamesfinlayson 18 hours ago
                  Oh gosh, I remember a big credit card outage in Australia in late 2015. I was completely oblivious until I bought some petrol and out came the imprint machine. I was amazed they still existed (hadn't seen them since the mid to late 1990s) but my boss on Monday told me that of course they'd have them for days when they couldn't accept cards - cost of doing business.
                • unparagoned 8 hours ago
                  I doubt any businesses would accept that
                • lxgr 23 hours ago
                  In terms of risk, that’s no different from just writing down the card number and expiry these days, which is why no merchant has embossing devices anymore.
                  • ibejoeb 11 hours ago
                    It's not really the same. The imprint exists to evidence the physical presence of the card at point of sale. If you dig through issuers' documentation about fraud and chargebacks, or PCI rules, you'll find the analysis. Of course someone could fabricate a forged embossed card, but it's a big step from just rattling off the numbers, so they give it some weight.
                • devmor 1 day ago
                  They just replaced mine with a fully printed card the last time it expired.
          • DaiPlusPlus 1 day ago
            Because credit-card systems ostensibly don't rely on debiting actual funds, they should be resilient and continue to work offline; when you use EMV (chip+PIN) it's the card that authenticates you and authorizes the transaction, not the bank down a phone-line (and who can forget credit-card imprinter machines?)
        • ifwinterco 1 day ago
          There are countries where the monetary system basically is cash (lots of people don’t have bank accounts).

          There are definitely issues with that, but it does work in the sense that people can buy and sell things and survive, provided the government is competent enough to provide good quality notes and coins and resists the temptation to print too many.

          After all it’s how western societies worked throughout human history until relatively recently

        • dontwannahearit 1 day ago
          Bullets and shotgun shells.
        • ImPostingOnHN 1 day ago
          lead maybe
        • _joel 1 day ago
          I only carry gold pressed latinum
        • reaperducer 1 day ago
          I don’t think cash is gonna help if the entire monetary system stops working - gold maybe?

          Yeah, I'll just get out my pocket knife and carve a little chunk off that big bar of gold when I'm trying to barter for gas at the truck stop. That'll work.

          • AngryData 1 day ago
            I would happily trade people for gold because there is no way at all they would ever get even close to the value of their gold in goods. You could probably get thousands of dollars worth for some beans by time they decide to start trading away their gold.

            Doesn't help if the world truly goes to shit and doesn't recover for decades, but if there is any kind of recovery then the gold would be worth something again.

          • cowlevel 14 hours ago
            Traditionally it was made into little circular discs.
          • eadmund 14 hours ago
            > Yeah, I'll just get out my pocket knife and carve a little chunk off that big bar of gold when I'm trying to barter for gas at the truck stop. That'll work.

            https://valcambi.com/product/combibar/

            It’s a roughly credit-card–sized gold bar containing detachable 1g bars each worth $125 at today’s gold prices. That will buy roughly an F-150 tank at today’s diesel prices.

            I would prefer something in even divisions of an ounce myself, but this looks like it would work.

            • reaperducer 8 hours ago
              It’s a roughly credit-card–sized gold bar containing detachable 1g bars each worth $125 at today’s gold prices.

              Congratulations. 500 years later they've reinvented Pieces of Eight.

              Its where we get the expression "two bits," meaning one quarter of a dollar.

        • triceratops 1 day ago
          I prefer ammo, canned goods, solar panels, and a well. And maybe a doctor or nurse practitioner friend.
          • unpleasurable 19 hours ago
            You can't live in a well dude just think what the local kids will say
        • _kb 1 day ago
          Gold is good. You need a fairly large chunk to bludgeon the marauding hordes though.
        • virgil_disgr4ce 1 day ago
          can't eat gold
      • rvz 1 day ago
        You can't send a single cent of your physical cash over the internet into someone else's bank account who is also experiencing this outage. FedNow is likely not working either.

        On the other hand stablecoins such as USDC can be sent over the internet on blockchains such as Tempo, Solana, XRP and Stellar worldwide, 24/7 in less than 5 mins and with extremely low fees.

        To Downvoters: Come on. You cannot send your physical paper dollar notes to someone else in another state or around the world in less than 5 seconds.

        I'd be annoyed if I was a BOA or Chase customer too. But the truth is that blockchains such as Tempo continue to work anyway worldwide 24/7 without fail.

        • trollbridge 1 day ago
          I knew eventually a cryptobro would show up.

          The U.S. government is very interested in keeping the banking system going and will throw all available resources at doing so.

          • roenxi 14 hours ago
            > I knew eventually a cryptobro would show up.

            A bit like how if you stand next to a water cooler someone thirsty will turn up? He makes a good point at an appropriate time, it seems quite possible that the blockchains will achieve better uptime than any given bank, resources of the US government or otherwise.

          • JKCalhoun 1 day ago
            Not a crypto-bro, bro. Has not yet used the word fiat.
          • iamnothere 21 hours ago
            That’s a big yikes from me, he did a heckin’ cryptobro, cringing so hard rn
            • rvz 20 hours ago
              So Stripe (who are building the Tempo blockchain) [0] are "cryptobros" because they understand the use-case of agentic payments for both humans and AI agents?

              Both crypto and your bank that just went offline, can co-exist.

              [0] https://tempo.xyz/

              • iamnothere 11 hours ago
                Sorry, it probably wasn’t clear, but I was making fun of the term “cryptobro” which to me is very childish.

                It seems to be part of the broader online Millennial-speak lexicon or “Redditese” along with words like “doggo” and “kiddo”. I do not like it.

          • rvz 1 day ago
            > I knew eventually a cryptobro would show up.

            ?

            But did you know that your bank would get knocked down offline?

            There's no need to get upset about that.

            > The U.S. government is very interested in keeping the banking system going and will throw all available resources at doing so.

            Good. They can coexist.

            The next time your bank goes back online, you can wait days to send your money around the world, while those using blockchains like Tempo (yes, Stripe built that) are sending money in milliseconds all around the world with very little fees at all.

            • trollbridge 11 hours ago
              My bank wasn’t knocked offline; it’s HQ is 5 miles from me and their computers are in house and they keep both data backups and power backup for their (small) data centre.

              They were also unaffected yesterday, other than the general Zelle problems.

              If I need to send money around the world, I can use a wire, which still often has lower overall transaction fees than a typical crypto transaction. I also won’t wake up to find out some hack has resulted in all my money disappearing.

            • sampullman 22 hours ago
              Only if your recipient thinks the way you do. It's not very common in practice.
              • rvz 21 hours ago
                Many people send stablecoins across the internet without a bank with low fees on fast blockchains.

                It's a common use-case, which is why so many people do it.

                • brewdad 20 hours ago
                  There are dozens of us! Dozens!
    • nowuh 11 hours ago
      Might be related to the Abrigo Sageworks syver breach. They notified their clients yesterday.
    • nnurmanov 16 hours ago
      We had a major bank outage for two days because of unsuccessful database upgrade.
    • trollbridge 1 day ago
      Strangely enough for me (with Chase):

      - Cards working

      - Branches (the few we have left) open, or at least it was when I drove by at lunch

      - Online banking has been working for me all day.

    • expedition32 12 hours ago
      The few bank branches my country still has don't have a vault.

      For emergencies it is wise to still have 20 euro bill lying somewhere or go hungry for a day I guess...

  • 999900000999 1 day ago
    Across all of tech QA is the first thing you cut. When you’re doing your job right no one cares.
    • jgilias 1 day ago
      Man. When I joined $DAYJOB quite a few moons ago, I was so happy to see that we had an actual QA department staffed with smart and active testers.

      All of them got laid off at the next layoff. That was before AI.

      • 999900000999 1 day ago
        A few threads ago people are arguing with me about this.

        Why have QA, every developer can simply QA and deploy, and talk to clients, and make sales calls.

        Then again I don’t understand Capitalism. At a certain point these companies are so firmly entrenched they can ship buggy half working software. The consumer is effectively a hostage, all competitors are also delivering barely working products.

        • nunez 23 hours ago
          All of those are individual skills that take entire careers to get good at.

          Knowing _what_ to test and how. Knowing how to steer customers towards the conversations you want to have. Knowing when your deal is a diamond instead of a dud. All skills that are very hard to master.

          • kakacik 16 hours ago
            Its a bit arrogance on dev side, sort of 'know it all' syndrome I've been also guilty in the past. Devs are terrible testers of their own code, period. Its not some big mystery, was repeated ad nausea 25 years ago when I started my professional career, still the same story.
            • expedition32 12 hours ago
              Objectively reviewing your own work is not something that most people are capable of.
              • kakacik 6 hours ago
                The mental shortcuts folks take to focus on higher-reward activities which testing is not... mental blind spot, its there every single time, I guess seniority is not about diminishing it but rather acknowledging and working with/around it.
        • rand846633 1 day ago
          Btw, not trying to say you are wrong with not understanding capitalism, I find it helps to view it not as an imperative but rather as a local optima in game theory. Capitalism just works really well for its own utility functions that it self dictates upon us.
        • dprkh 1 day ago
          Why are consumers hostage? Why can't they switch to a better product?
          • 999900000999 1 day ago
            I’ll give you an example.

            One of my employers picked a 401k provider with a mandatory 150$ withdrawal fee.

            I can’t switch to a different 401k and I’m stuck paying that fee.

            Capitalism falls apart once you realize it’s easier to just buy out your competitors or otherwise bully them out of the market.

            • dprkh 23 hours ago
              This is a poor example because it was your employer who chose a bad product and forced you to use it, not you who chose a bad product and is for some reason unable to switch to a different one. Your example does not show why your employer can't switch to a different 401k provider.
              • 999900000999 22 hours ago
                My employer has no real motivation to switch.

                Only so many 401k providers exist and most of them have issues with strange fees.

              • preg_match 22 hours ago
                They can, but said provider also probably sucks ass.

                For many (most?) software, customers are hostages because it's all bad. Usually bad in different ways, too. So maybe you improve X, but Y gets worse. Did you come out on top? Maybe. It takes a lot of analysis to know ahead of time. And that's a lot of risk. Ehh... better not.

            • dennis_jeeves2 9 hours ago
              >Capitalism falls apart once you realize it’s easier to just buy out your competitors or otherwise bully them out of the market.

              What is your definition of capitalism?

          • skywhopper 10 hours ago
            Because the monopolists/duopolists/cartels use their market power to buy/bully/regulate out of existence any “better product”.
          • zahlman 1 day ago
            Who offers this "better product", and how do consumers know it's "better"?
        • sota_pop 1 day ago
          What is “Minimally Viable”? And how many aspects can a business find to apply the concept across operations?

          This is (modern?) capitalism.

        • esikich 1 day ago
          Why stop there? Have them clean the bathrooms and mop the floors as well. A truly savvy business owner would have them do payroll and HR duties too. In fact, now that I'm thinking about it, just have one person do all the roles, imagine the money you'd save on staff.
          • addled 1 day ago
            I get your point, and largely agree with it.

            At the same time, I’ve also thought I’d be much better off if my desk job required I spent an hour or two a day mopping floors, loading pallets, etc.

      • winrid 1 day ago
        Of course, they were done testing!
    • lanthissa 1 day ago
      i worked a pretty similar place to JPM, if qa was ever cut delivery times went up and people were okay with that for mission critical stuff. If you said 'this isn't tested well we're delaying' everyone would be okay with it.

      on trading applications, and banking systems theres huge financial and legal consequences for cutting corners (look up knight capital). Pretty much the only way to get fired is to mess something like that up.

      In some jurisdictions if you pushed a change that broke and it was flagged internally as not tested enough someone in the org would quite literally go to jail.

    • cryptoegorophy 1 day ago
      Unfortunately highly advanced tech people lack “selling skills”. Hopefully this changes with ai - prompt: summarize the work I’ve done and setup I’ve implemented and how it will help the company and why my role is extremely important.
    • Dannymetconan 15 hours ago
      It's like internal IT support teams. When its working well it looks like you're doing nothing.
    • whateveracct 1 day ago
      nobody ever gets credit for fixing problems that never happened
    • rossdavidh 1 day ago
    • ernst_klim 11 hours ago
      > When you’re doing your job right no one cares

      Not just that but doing a bad job brings you more visibility. If you were asked to build the system and it works like a swiss clocks, the management will forget about it in a quarter. If you build the system and it's a mess, you'll get a lot of visibility as a firefighter. The incentives are terrible.

  • bobson381 1 day ago
    My nontechnical co-workers see the Microsoft 365 outage, the FAA ground stop, and things like this as one interconnected problem. They aren't entirely wrong, but the general fear is hard to dismiss
    • montagg 1 day ago
      They are at least interconnected in the sense that we are struggling societally to do big, hard things that we could do a decade ago.

      Roman citizen wondering why the aqueduct isn’t working kind of thing.

      • forshaper 1 day ago
        Interestingly enough it took about four generations, a century, for someone to go back in and investigate the aqueducts properly. I think a lot of this is due to previously easy, simple things being ignored, in part because they are considered menial. The basics of maintenance are easily forgotten.
        • noir_lord 1 day ago
          > The basics of maintenance are easily forgotten.

          Also undervalued by society as a whole and politicians especially.

          They'd rather allocate funding for a new bridge they can point at and say "I did that" than properly fund the unsexy work of making sure the existing bridges are maintained.

          Now apply that across most western democracies, repeat for decades and we are where we are.

          The countries doing their vast infrastructure build outs will likely repeat the pattern in a few decades.

        • camillomiller 17 hours ago
          The basics of maintenance are... easily cut by VP Chadson Chadenberg, former McKinsey, in his cocaine-fueled spree to increase profitability, you mean?
      • hn_throwaway_99 1 day ago
        The other issue though is that with 24/7 connectivity, any unplanned downtime can feel like a major outage. 50 years ago there were lots of outages, to, but people didn't care as much, they just went outside.
        • jzb 1 day ago
          50 years ago, an outage in Seattle didn't affect me in St. Louis as a rule. I'd go to the bank, hand the teller a check, they'd eyeball it, maybe hand me some cash, and process all the things on paper.

          When banks started moving to computerized systems, maybe a bank would be unable to process something... but not multiple banks. I can't recall any "outages" that affected banks nationwide for any period of time. Walk into your local bank and find out they can't do something because their big system at the mothership is down? Perhaps.

          Consolidation likely plays a part in this as well. I can't speak to the FAA/ATC systems 50 years ago, I have no recollection of what they were like then, but the only nationwide system I can think of around that era was Ma Bell. And I don't honestly recall any nationwide outages back then. Regional, sure. You might have a problem with part of the country getting calls if there was some kind of emergency... (e.g., everybody trying to call their grandmas in Kansas after reports of a tornado) but I don't remember there being any nation-wide problems like that.

          We had to count on actual snow for snowdays. :-)

          • mkjs 1 day ago
            The manual processes don't exist anymore. If the IT is down, nothing can happen.
          • toast0 1 day ago
            > And I don't honestly recall any nationwide outages back then

            There was this one in 1990 [1], but it's notable because the system was otherwise very reliable. Also, this report says only 50% of call attempts failed, presumably most people would try their call a few times and likely get through.

            [1] https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collap...

        • Taikonerd 1 day ago
          I would also say that there are more single points of failure on the internet than there were 20 years ago.

          In 2006, Amazon had just launched this new storage service called "S3." It would have struck people in 2006 as bizarre to think that AWS (and Cloudflare, and GCP, and friends) would one day be so crucial to the internet that a bug there could take down dozens or hundreds of companies.

          • annzabelle 1 day ago
            And for a lot of companies (I know this is true for Capital One), the sole redundancy plan for the vast majority of services when US-East-1 fails is to fail everything you can over to a different AWS region.
            • getpokedagain 21 hours ago
              I'm not surprised. The same type of plan can cover fedramp certification.
              • annzabelle 20 hours ago
                That was a frequent question asked at the annual CEO-on-a-stage-in-front-of-the-whole-company event, whether we could be cloud vendor agnostic and setup for an easy lift and shift to another cloud provider, and every year we'd get an answer about how great AWS vendor-lock was for us.

                At this point I'm pretty sure 99% of businesses have looked at whether to be ready for a lift and shift between vendors or whether to cut this month's compute budget by using more provider specific services, and pretty much all of them have gone with going with more provider specific services.

        • jamesjhare 1 day ago
          50 years ago was 1976. Basically nobody had a computer at home. "Outages" were not a thing.
          • maccard 1 day ago
            50 years ago my parents were getting power outages on the somewhat regular. I have had two in the last decade.
            • olyjohn 1 day ago
              I had one last week. And multiple each winter. Its highly environment specific. Unless you're saying you still live in their house?
              • maccard 16 hours ago
                Very much region dependent. I grew up in Ireland and currently live in the UK in a city. Power outages are extremely rare in both.
                • zenithirl 13 hours ago
                  Power outages rare in Ireland? Absolutely not. Unless by rare you mean a few per year.
              • Mezzie 1 day ago
                I had one this morning. I live in the middle of a metro area of half a million people.
              • duderific 1 day ago
                PG&E I presume? We get multiple per month at this point
          • hn_throwaway_99 1 day ago
            We still had a banking system, we still had phone lines, etc. But yes, the general lack of an expectation of always-on connectivity meant that service interruptions were much less of a big deal.
        • expedition32 1 day ago
          Fifty years ago the store owner just wrote down your name and you could pay the bill next week.
          • Mezzie 1 day ago
            One of the coolest things I've seen, system design wise, was working at libraries when the tech/power is out for a given period.

            Every service except digital lending/materials continued to function, and there were fallback processes for recording everything that happened so the information could be added back into the digital system once the tech was fixed/power was restored. Specifically designed forms that were easy to use and easy to scan, an exact understanding of what needed to be recorded and why and where it would go later, etc. And a staff of people who were meticulous enough to actually follow through.

            • nunez 23 hours ago
              This makes sense given that so much of how information theory and data processing works were influenced by libraries, which have existed for thousands of years
        • jojobas 1 day ago
          Anyone who hasn't embraced the 24/7 connectivity has been priced out.
        • jeremyjh 1 day ago
          That isn’t something that has changed in the last 10 years.
          • hn_throwaway_99 1 day ago
            I originally missed OP's comment about "a decade ago", but I don't think our "ability to get big things done" has changed in the last 10 years, nor the number of major outages.

            When I hear people talk about "we can't do big things anymore" in the US, they are usually referring to mid 20th century stuff a la https://patrickcollison.com/fast, not stuff that we did 10 years ago.

            • jeremyjh 1 day ago
              The topic of TFA and this thread is that these major outages appear to be happening more frequently this year - or in the last month - than we've seen in the past.

              I do not believe the FAA has ever grounded the entire country due to a network upgrade going bad. So that is possibly an example of a hard thing we could do a decade ago, that we can't do now.

              • hn_throwaway_99 1 day ago
                It may appear, but it's not.

                > I do not believe the FAA has ever grounded the entire country due to a network upgrade going bad.

                In 2023 the FAA grounded the entire country due to a faulty database update. So yeah, these things happen, people, like you, just forget about them.

                • brewdad 20 hours ago
                  Since 9/11, there have been two nationwide ground stops that lasted more than an hour. Both have happened within the past three years. If we get a third soon, we might well have a trend.
              • hunter2_ 1 day ago
                Deciding on the ground stop instead of some other type of quick rollback might've been more to do with furthering financial agendas by making a scene (under the guise of "an abundance of caution" which does seem like a more popular thing lately than decades ago) as opposed to it actually being the only reasonable action.
                • jeremyjh 1 day ago
                  No, that did not happen because every communication related to this is recorded, and will be audited in an investigation, and this would have been known to everyone ahead of time. Most likely the NSTB will investigate, though this has not been announced. There are also going to be congressional hearings and career civil-servants will be subpoenaed.
      • randomNumber7 1 day ago
        It doesn't seem hard to use two angle of attack sensors if you pitch the nose of a plane down based on its output.
      • 1270018080 1 day ago
        The thing done today weren't even computationally possible a decade ago. The bar for competitive software is rising exponentially.
        • pdonis 1 day ago
          > The thing done today weren't even computationally possible a decade ago.

          Not the things that are breaking, no. The FAA was doing ATC a decade ago. We had email and cloud servers a decade ago. We had bank websites a decade ago. The scale of those things today is perhaps larger than a decade ago, but not enormously larger; indeed, the increase in scale in those things is much smaller than the increase in computing power in the world from a decade ago. So these things should have gotten more reliable--but they haven't.

          • ranger_danger 1 day ago
            Perhaps we would be better off if businesses simply couldn't get this big.
          • nick486 1 day ago
            we skill-squeezed ourselves as an industry.

            we made these systems simpler to maintain, to reduce the amount of skill we need to pay for to maintain them. And that does reduce the maintenance costs, at the cost of more compute. but who cares about compute costs where compute per unit of money doubles every 18 months, amirite?

            unfortunately it also increases the competence required to change these systems without degrading their performance and usability.

            since no one wants to pay that extra cost either, we let the systems degrade a little bit at every change instead, hoping they degrade slower than hardware improves.

            that bet was bound to lose eventually, because of laws of physics, and because that degradation pattern affects hardware too.

            and the double whammy is - with little demand for expertise - the supply is effectively gone too. so now that the walls have closed in, there is no way out.

          • randomNumber7 1 day ago
            We also had this decentralized network which turned into a monopolized shithole that fries normies brains. Although that was probably more than a decade ago.
        • sdevonoes 1 day ago
          Important stuff is the same. Banking doesn’t need fancy stuff
          • alex0015 1 day ago
            Do you think someone could take a look at the things you do for work and accurately say whether it needs "fancy stuff?"
        • mixdup 1 day ago
          Both the FAA and the Microsoft services that failed have been around longer than a decade
        • nimih 1 day ago
          I'm pretty sure I was able to do online banking a decade ago. USAA even had working 2FA at that time.
        • jghn 1 day ago
          > The thing done today weren't even computationally possible a decade ago

          I was able to log into my bank's website in the late 90s, FWIW. Yet today I can't seem to do that.

        • Avshalom 1 day ago
          No we definitely had banking software in 2016.
          • ragequittah 1 day ago
            We definitely did. And stuff like this happened back then too [1]. And even worse stuff before then [2].

            [1]https://www.fintechfutures.com/baas/bt-fault-hits-uk-banks-d...

            [2]https://abcnews.com/US/hackers-block-us-banks-online-service...

          • fasterik 1 day ago
            "Nothing existed before I was an adult" type vibes.
          • whateveracct 1 day ago
            AI revisionism is out of control generally tbh lol
            • hunterpayne 1 day ago
              I've been working in software for 30 years. This is the largest collection of systems to be down at the same time in that period. I have been literally experiencing a degradation in online services over the last year or two. Its either poor quality AI generated code, or your generation is just incompetent. Your call...

              If I were you (or was working at somewhere with an outage), even if it wasn't AI, I would still be blaming AI. Because the alternative means you shouldn't have a job in tech.

      • Telaneo 10 hours ago
        Where's Agrippa when you need him as aedile?
      • convolvatron 1 day ago
        that's not really a good characterization. we have known this was a house of cards for decades. someone just turned on the fan.
      • bongofizzbuzz 1 day ago
        [flagged]
        • rogerrogerr 1 day ago
          What do people mean when they say this?
          • GenerocUsername 1 day ago
            They mean there is a large multi-decade conspiracy theory which ties many world events into one cohesive narrative which culminates in a large scale planned destruction of western power structures wherein that controlled demolition scares the proletariat into being accepting of a new world order in which the global currencies are consolidated to a singular digital currency. This shift basically marks the end of individual freedom since nothing short of a uncontrolled global collapse could ever unseat the hedgimony it would create.
            • NoidFonsense 1 day ago
              How could the CBDCs possibly be worse than EMV already is? The world of social credit scores and political debanking exists, unless you are buying coke off the darknet with monero.

              But for everything else you need, food water shelter healthcare, you need to submit to europay visa mastercard

              • GenerocUsername 1 day ago
                Yeah, it mostly exists now... And gets closer to the end state every year.
          • monegator 1 day ago
            ending of fight club?
          • mathfailure 1 day ago
            [flagged]
    • sigmoid10 1 day ago
      Well, digital centralisation and interdependency without fault hardening is a big problem. Just because it presents itself in slightly different ways across industries doesn't mean it is not a fundamental issue in our society.
      • philipallstar 1 day ago
        I don't see much of a problem with depending on AWS or Google Cloud. It's Microsoft who are particularly flaky. And always will be, because IT people choose them regardless.
        • pwg 1 day ago
          You are still "centralizing" on "someone else's computer". Just because those two have not, yet, had a MS365 event does not mean they won't have one at some point. And everyone "centralized" there will have a bad day when that happens.
          • ghusto 1 day ago
            AWS has had them, and is passed the growing pains. At this point when/if AWS goes down, it will still have a far higher uptime than others could manage, let alone those going it alone.

            Microsoft on the other hand, has proven that it will never be reliable.

            • gilfaethwy 1 day ago
              AWS quite literally lost an entire region in the Middle East and has said they will not be able to restore it.

              You can argue that this is a result of something outside of their control, and you would not be wrong, but if the point here is about centralization rather than simply a particular company's operational procedures, then direct strikes on datacenters should be part of that conversation.

              I was an Amazon red badge in the EC2 org, fwiw, and do think AWS has excellent operational culture, far beyond what I saw while working in the M365 org, and beyond what I've seen at other tech companies and hedge funds where I've worked. But that doesn't obviate issues with centralization.

              • mjr00 1 day ago
                Centralization is almost a good thing for extreme cases though.

                If Bank of America's central datacenter (cloud or on-prem) gets hit by a cluster bomb, taking out banking for ~half the US, you can be damn sure that very smart people are going to be working 24/7 until it's resolved.

                If you have all your savings stored with a local credit union, say Seattle Credit Union which has 50,000 customers, and they get hit with a ransomware attack... well, good luck. You (probably) won't lose any actual money thanks to FDIC rules, but it might be weeks or months before their services are back online.

                Similar to self-hosting vs cloud, you get a bit of a paradox where it's overall better for society if there's no interdependencies so a SPoF outage doesn't impact everyone, but things are much worse for an individual in the event a localized outage does occur due to economies of scale, or lack thereof in this case.

          • ikiris 1 day ago
            Anyone who says "someone else's computer" is just giving away that they don't understand the difference between some pet vm and a proper cloud infra.
        • corranh 1 day ago
          One team won’t win the cup every year, the issue here is that everyone has picked their favorite while ignoring the risks of the league.
          • novaleaf 1 day ago
            Have you tried using Azure? (I have, it is not good)
        • paavope 1 day ago
          It’s a tragedy that much of European industry and public sector are so deeply embedded with Microsoft that they barely even consider other options, even when everything Microsoft touches seems terrible
          • annzabelle 1 day ago
            This is also true in Oceania (especially NZ) in my experience. Companies here are much more embedded with Microsoft than companies I worked with back in the US and are less willing to consider other options. My company actually does most stuff with on prem windows servers and a bit with Azure. One aspect for NZ in particular is that there was no NZ region for Azure or AWS until 2025 and parts of the country are 70+ms from Sydney, so for anything where latency is significant, on prem stuck around here a lot later.
          • bijowo1676 1 day ago
            I feel like on-prem Microsoft stack is a very well known and controllable beast.

            Its pros/cons are known and gaps can be addressed with 3p software. You just keep on old outdated, but proven version of software and keep the perimeter locked tightly.

            its the azure cloud that is the wildcard.

            • murphyslaw 1 day ago
              GP isn't talking about on-prem I believe. Small/medium businesses in Europe are much more dependent on MS than their US counterparts. Alternatives are mostly out-of-scope from their point of view.

              Don't forget that MS has the advantage of non-English languages support that many alternatives lack.

            • throwaway7356 1 day ago
              > I feel like on-prem Microsoft stack is a very well known and controllable beast.

              I feel like it is very unknown, there is no documentation and everyone relies on comments on random forums that are now 21 years old which tells to set undocumented registry keys to undocumented values.

              • randomNumber7 1 day ago
                I don't understand why people don't prefer to edit text files at arbitrary locations.
              • retwakm4 1 day ago
                [dead]
          • esseph 1 day ago
            > It’s a tragedy that much of European industry and public sector are so deeply embedded with Microsoft

            It's not just Europe! :(

        • zdragnar 1 day ago
          How quickly we seem to have forgotten how often aws us east went down.
        • jimbob45 1 day ago
          They’re expensive and centralized. I always thought the general wisdom was to bootstrap your business with serverless infrastructure and then eventually work your way back to an on-prem-first philosophy. I guess a lot of companies skipped that last step to…save…something?
        • rsoto2 1 day ago
          Yeah the FAA didn't see a problem with a single point of failure either look how that went. It's not flaky until it is.
    • Legend2440 1 day ago
      >They aren't entirely wrong,

      They are entirely wrong.

      None of these outages, as far as we know, have a common cause. They just happened by coincidence inside the same news cycle.

      Nor is there any evidence to blame them on AI, or tech layoffs, or hackers from Iran. It's more likely something boring and technical that doesn't fit into anyone's narrative.

      • GoToRO 1 day ago
        My guess is a total disrespect for actually qualified people. You know, the ones that specialized in one thing, for their whole life. Every company tries to see how few people they can have and still run fine. And move fast. And no trainings. And no recognition for the people that could actually train other people. I think all these come together at some point and you get multiple catastrophic events in multiple industries.
        • randomNumber7 1 day ago
          If you have no skills it is hard to evaluate if someone you intended to hire has some.
        • mixdup 1 day ago
          Nailed it. Downward pressure on salaries, layoffs, outsourcing all drive technical excellence out of an organization. Then, when companies DO hire it's very hard to find competent people (and I mean find them at all, not find them at the cutrate you want to hire them at) and the people you do hire are just seat warmers
      • sanderjd 1 day ago
        I would ask a few questions:

        1. Does there seem to be a larger concentration of outages recently? (I would say yes.)

        2. Is there actually a larger concentration of outages recently, or does it only seem that way because of an increased focus on this? (I would say that there is really a larger concentration, but I'm less certain about this.)

        3. If there is actually a larger concentration of outages recently, is that because of AI? (This is the point at which I become pretty unsure either way.)

        4. If it is because of AI, how exactly is AI causing this? (I think more widespread vulnerability discovery / exploitation is more likely than "everyone is vibecoding everything now and that's screwing everything up", but I'm not entirely sure.)

        5. Is this just transitional growing pains, or is this a new normal, or is this the beginning phase of a pattern that is only going to accelerate? (I don't know the answer to this one at all! I hope it's the first thing, but I wish I were more confident.)

      • mickael-kerjean 1 day ago
        > They just happened by coincidence inside the same news cycle ... to blame them on AI, or tech layoffs, or hackers from Iran

        I will add myself as another datapoint, yesterday, I investigated my first ever reflection amplification attack that was bringing an entire network down, the ips originated from both china and iran. While running my oss work: https://github.com/mickael-kerjean/filestash, over the last 2 weeks, I have received about 20 security reports when the usual rate is once every few months, and on top of that literally every single form coming on my website is being abused by spam, the rate of spam message coming through there has 100x increase over the last few weeks. I have never seen that much security work pushed over me ever

      • 14u2c 1 day ago
        What evidence do you have that staff reduction is not at the root of some of this?
        • mh- 1 day ago
          Sorry, are you asking them to prove a negative? They said there was no evidence it was.
          • majorchord 1 day ago
            Or maybe the more evidence there is that's unrelated to staff reduction, the less likely it is a big factor?
      • omgwtfbyobbq 1 day ago
        It could be some people rushing to get code into prod before the holiday freeze, or just a coincidence.

        Either way, now that AI is more capable, you can really mess stuff up at scale.

        • bigstrat2003 1 day ago
          The messing stuff up at scale is more because LLMs aren't capable, and people treat them as though they are.
        • werdnapk 1 day ago
          Is this how the "AI will lead to the end of the world" type stuff I've been hearing about starts?
        • hunterpayne 1 day ago
          Code freezes for the holidays were several weeks ago. The code would have been deployed weeks ago too. And if AI writes better code than you, that says more about you than AI.
          • account42 14 hours ago
            Which company are you talking about?
      • dualvariable 1 day ago
        No, it all comes down to the decision-makers at the top being bad at their jobs.

        One way or another, they've tried to avoid addressing tech debt, avoid hiring and retaining good people, and have more or less been kicking the can down the road / running on the edge without knowing it.

        Or else they did embrace AI-driven "move fast and break things" approaches, and so they broke things.

        They flip-flop between maximizing profits and draining operational divisions of any resources, and hoping to sprinkle a bunch of IBM magic pixie dust all over their infrastructure to solve all their problems without needing to understand any of it. You rarely find one that strives to understand problems deeply and invests in long-term planning and maintenance before a crisis hits.

        The result is old systems that eventually break, system upgrades that go bad because nobody understands them, security holes that someone might have hacked into, etc. All the same basic root cause at the top, which is that most of our leadership class is full of incompetent short-sighted narcissists. And it "works" because most of the time it pumps the stock due to lower costs, and we only see the fraction of times that it causes headline-grabbing catastrophes. On a long enough timeline, though, they'll happily drive our entire society off a cliff--and we're getting closer and closer to that (at least politically).

        • sota_pop 1 day ago
          Even moreso, it might also be that responsibility (sic Liability) is applied asymmetrically AWAY from the decision-makers you reference.

          They reap all rewards and bear none of the responsibility beyond saying verbally “I accept responsibility” (usually in the form of more layoffs and firings for those more explicitly involved in implementation). How many C-Suite executives will be shown the door as a result of poor staffing and strategy that lead up to these various outages?

          Conversely, when outcomes are positive, compensation and notoriety are disproportionately attributed to said decision-makers.

        • retwakm4 1 day ago
          I sincerely doubt it is the IBM Pixie Dust and Mainframes these banks use that is the root cause. The mainframe systems programming teams at places like JP Morgan are supremely competent. I've worked with them many times.

          The app developers hooking that all up with the cloud? Sometimes less so.

          I suppose we will just have to see what the root cause is, but I'd put a bet on it not being down to "old IBM Pixie Dust nobody understands" or anything of the sort, and suspect it is cloud related due to this occurring across multiple banks.

          In outages I've gone through root cause analysis for that involved mainframes and cloud and just general hybrid cloud stuff (this is my specialization as a mainframe systems programmer and consultant on cloud integration), it's almost always the cloud that was flaky. Mainframes are actually understood quite well by the priestly class that is trained on them.

      • elAhmo 1 day ago
        The thing is, we don't know.

        You saying they *are* entirely wrong is as baseless as their claim. Sure, you can say they don't have a common cause, and I agree with that, but chances of these events with different causes occurring so close together is very low and the chances of a same root cause is probably more plausible.

      • peanut-walrus 1 day ago
        As history has shown again and again, blaming something only requires political will, not evidence.
      • mym1990 1 day ago
        I am assuming you don't have a direct line to each of these outages, and you have no proof that there is no common cause. A common cause could also be something like "poor production deployment practices" or "reliance on AI built software", in which case...maybe they aren't entirely not wrong.
        • Legend2440 1 day ago
          It's not my job to prove they aren't related. That's the default; most events are not related.

          It's your job to provide evidence they are related, and you have nothing except speculation.

          • vonneumannstan 1 day ago
            AI has been finding new critical 0 days at an increasing rate so it's actually not unreasonable to think these might be related.
          • mym1990 1 day ago
            Yes, but you have no proof while saying someone is for certain wrong. Seems like the burden is on you to back that up /shrug

            It’s actually hilarious that you made the irrefutable claim, and then you are telling me to find proof.

      • MrGilbert 1 day ago
        The boring news is that laying off 100th of thousands of IT workers because of a newly promised, never-seen-before technology, does indeed cause trouble. Who'd thought that these people did in fact create value?
        • randomNumber7 1 day ago
          The value is created by maximizing your screentime to show you ads that sell you stuff that you don't need.
      • dclowd9901 1 day ago
        Genuinely, what would it take for you to consider that it could, in fact, be the AI push?
        • Legend2440 1 day ago
          I'd want to read the post-mortem analysis from the team that diagnosed and resolved the fault.

          They'd really be the only people with information about the cause of the issue; if they blame an AI-generated code change or the actions of an AI agent, that would be good evidence.

      • tcdent 1 day ago
        Not trying to be that guy, but they do have a common cause: networked compute.

        There are many more systems in the world outside of it, even if it is relatively ubiquitous at this point.

      • beambot 1 day ago
        Where there's smoke...

        "As far as we know" just indicates a lack of data, not evidence that they're not related. This is especially relevant for a political administration whose response to COVID was to bury their head in the sand by hiding data. Also: A public comment of "AI is hacking all our shit" wouldn't exactly calm the public.

        It's fine to say, "I'll wait until we know more", but in the meantime AI progress continues and the sheer volume of cyber vulnerabilities it's uncovering is undeniable. It's not surprising that some state-sponsored bad actors would be use this as an opportunity to infiltrate too...

        • pianopatrick 1 day ago
          Seems to me that most state sponsored bad actors would want to infiltrate in a less noisy way.
        • Legend2440 1 day ago
          You also don't have any evidence that they are related.

          In the absence of data, you are assigning blame to things you already hate, like AI or the administration. Motivated reasoning is highly prone to error, and it's a form of stupid-brained thinking.

          • beambot 1 day ago
            No blame or hate. Just hypotheses with conditional probabilities that will be updated in response to more data...
      • aseipp 1 day ago
        It's extremely funny how many of your replies are proving your point exactly. One reply says "Yours is easy to hold since it can't have evidence" when it is simply stating that _there is no evidence of a claim_. Imagine that, "There's no proof of X" is somehow a magically unfair argument because it can never be proven! Wow! Nobody has ever thought of this before. Guess we have to assume there's inherently merit to X, then, and don't have to worry about actually constructing a proof of it (that requires patience and knowledge and first-party sources and does not get you internet points).
    • jessyco 1 day ago
      As a technical person, I tend to find it hard to dismiss as well. What's easier to explain. Someone said it in a comment, someone keeps pushing for the cheaper solution cloud infra was cheaper and now we see the cost of that.
    • mrtksn 1 day ago
      I swear nothing works fine anymore, the world feels broken. Maybe those particular incidents are not interconnected directly but they are connected in the ether in the sense that everything is incredibly complicated and precarious.
    • hardtke 1 day ago
      None of the responses to your comment get to the heart of the issue. People believe these are likely interconnected because they no longer trust the authorities telling us they are not connected. I certainly don't trust the government, and the mainstream media seems unable to accurately cover such issues.
      • user3939382 15 hours ago
        To me this is the key issue of every "conspiracy theory" regardless of the likelihood of its veracity, right up to the flat earthers.

        These theories are the product of a society in which every major institution is corrupt and untrustworthy as a norm.

    • adrithmetiqa 1 day ago
      It’s natural to turn to AI based development as one of the causes of the increase in large scale outages.

      I prefer to ask a different question. Will all this AI development delivering us amazing new software? Are we just going to get 1000 free clones of Adobe Photoshop or can we expect some really exciting groundbreaking new tools? Perhaps all the really clever use of AI development is in the pursuit of better AI and if that’s the case, why?

      • incompressible 1 day ago
        Yeah, I'd probably extend it to software too: which software actually delivers real value to society, and which is only created for software's own sake.

        As to why, well economy must grow, no matter what.

      • alfalfasprout 1 day ago
        It boils down to leadership. If all C-suite asks for is "more, faster and cheaper" then enshitiffication gets worse.

        Unfortunately it will take many more outages for folks to realize that serious engineering talent has great value in the AI age. It's their judgement that helps speeding headfirst into building unreliable unmaintainable and difficult to extend systems.

      • byzantinegene 17 hours ago
        turns out AI thinks like an engineer an engineers are mostly not that innovative.
        • solyin 16 hours ago
          innovative on which aspect? I want a database that works that is really fast but don't innovate in the area of losing transactions and making up transaction IDs and amounts
    • da_chicken 1 day ago
      It is an interconnected problem, but it's the problem we're all already aware of. All these businesses and the FAA have gutted their operations teams. There's too much to do, and AI doesn't take less time.
    • Flere-Imsaho 1 day ago
      I have a predication: In the next few years having any connection to the open Internet will come with massive risks. I.e. if you're not a government, military or $$$ company you won't be able to afford the steps it would take to harden said system against the onslaught from AI attacks.

      Some other thoughts:

      * Will the Internet become balkanised?

      * These smaller networks will require a lot of personal information to even gain access.

      * The return of the Sneakernet?

      • account42 12 hours ago
        It's entirely reasonable to write secure internet-connected software as long as you limit the scope. E.g., the web doesn't have to require each client to bring an insanely complex programming environment. 00s level of HTML and CSS was good enough for most use cases actually - certainly better to go back to that than to give up on the internet entirely.

        Similarly, if you don't insist on moving so fast you can actually fix bugs and create hardened systems.

        And finally, we can also give up on the notion that the Internet needs to be secured using entirely technological means. We don't have that requirement for anything in the physical world or even for other communication channels (e.g. phones, mail). We don't need to make murder literally impossible and we don't need to make computers literally unhackable. Doing so is a choice with a non-zero cost and if the cost increases dramatically we need to re-evaluate the choice and reconsider alternatives.

    • oliyoung 1 day ago
      I'm as technical as most of us ... and I'm lightly suspicious ...

      Not the same root cause of course, but there's something in the water

      • gib444 1 day ago
        The aura is definitely misaligned and our collective chakra is waning. My earth stone are vibrating in a concerning manner
    • rglover 11 hours ago
      I mean, from memory, we didn't use to have this many outages and especially not this severe, this close together.

      If everyone cuts the bullshit, we'd admit that save for edge cases, the industry is going off the rails thanks to AI and its promises of jewels. The tailpipe is out of exhaust and people are still huffing. Of course shit is falling apart right now. We're living in a state of utter delusion about a tool responsible for helping us build and maintain systems.

      The wise among us will take caution, but I think this was a tremor, not the actual quake.

    • zzzeek 1 day ago
      they're interconnected in that we're all getting really overloaded by the workload created by AI, and even though we can use AI to get through the workload as well, we humans still have to be aware of everything flying by, and I think things are falling through the cracks more often. These days I work on ten or twenty issues at a time. it's a little nuts
    • TZubiri 1 day ago
      software quality is going down in general

      not sure if vibecoding is a cause or consequence, but it’s related, it requires as an axiom to stop caring about internals. And it seems to require disregard for quality too.

      • LPisGood 1 day ago
        I think quality of most products is seemingly going down. To me it is a likely consequence consequence of increased financialization and money being expensive.
        • account42 11 hours ago
          Also of the increased monopilizations / duopolization as well as increased abuse of network effects in many areas that removes bargaining power from consumers.
      • kevin_thibedeau 1 day ago
        Basic competency is going down. Everything is outsourced such that no one knows how their critical systems work.
        • elxr 1 day ago
          Competency going down, timelines being rushed, project managers and CTOs forcing new tools at breakneck speeds.

          i'm all for experimentation and adding new tools when they're shown to actually help, but it seems like large software companies (especially banks and e-commerce) just don't care about quality anymore.

      • saghm 1 day ago
        I worry that this isn't the right metric for diagnosing whether there's a problem. I'd argue that we've had instances in the past where a massive increase in how accessible programming is ended up being a net gain in the long run for software even if the average quality went down (e.g., having computers move from giant room-sized mainframes that only companies and governments could afford to sitting in the living room on a table probably reduced the average quality because untrained amateurs could start write software even if they weren't skilled enough to be employed). I could potentially be convinced that this isn't the same and that it will end up being worse for software in the long run with a stronger argument, but "software quality is going down in general" isn't convincing enough evidence for me.

        All of this is also downstream of what I think is even more important, which is what's better for society in general. Obviously low quality software is not better for society than high quality in a vaccuum, but I also don't think there's a single obvious objective answer for whether the influence of software on society is a net positive or negative today compared to, for example, when computers were all mainframes and all software was produced by people paid to write it. There are a lot of differences in society today compared to back then, and software certainly has some role in how we ended up here, but trying to attribute exactly what and how much is not an exact science.

        This is a lot of words to basically say: if you're confident that this is a problem, I think it's important to state exactly why, since I think the interest parts of the discussion are in how you've concluded that.

      • derangedHorse 1 day ago
        Contrary to "vibecoding" (which is increasingly becoming how anyone programs) being a cause or consequence, it'll probably be part of the solution.
        • bigstrat2003 1 day ago
          > Contrary to "vibecoding" (which is increasingly becoming how anyone programs)

          Commonly asserted but not true. Most programmers are not just letting the model take the wheel.

          > it'll probably be part of the solution.

          LLMs can't solve the slop problem they themselves are creating.

          • derangedHorse 1 day ago
            > Commonly asserted but not true. Most programmers are not just letting the model take the wheel.

            In what way did you interpret vibecoding as taking the wheel away from programmers? It certainly does in writing code. Generally developers are still involved in architecture design and defining requirements to give their agents.

            > LLMs can't solve the slop problem they themselves are creating.

            Human slop is the real problem that LLMs can and are solving.

            • mrheosuper 22 hours ago
              >Generally developers are still involved in architecture design and defining requirements to give their agents.

              What do you call someone offload all of this to the agent too ?

              Yup, it's called vibecoding. You don't care how agent achieve it, you care only about the result.

      • fidotron 1 day ago
        Software quality peaked between XP SP2 and the victory in the market of node.
        • paganel 1 day ago
          Very well put, your comment alone should probably be a dedicated post in here with people commenting on that timeline (people who were actually programming back then).

          Maybe it was because I was a lot younger back then and a lot more idealistic, but especially for the ~2005 (i.e. just after the dot-com crash) - 2010 period (just before money would ruin everything in this industry) I can say that the programming bar was placed a lot, a lot higher compared to where I see it placed now. Software wasn't perfect back then, of course, but there was a real push (belief, maybe?) that with enough will and with enough new-knowledge-being-acquired then we (meaning all of us as an industry) will make things a lot better. In part the existence of this very forum is a simptom of that philosophy, what if it (originally, maybe it still is) being written in Arc, or Reddit originally being launched in Lisp.

        • TZubiri 1 day ago
          Maybe node software was good initially, but its model was unsustainable, it relied on outsourcing to volunteer programmers, relying on such programmers means increased vulnerabilities, supply chain incidents, and low quality of software.

          Vibecoding is an extension of the culture of overrelying on massive external dependency graphs. It's a preference for visual confirmation that something works and a disinterest in how it works. I think it's also related to the tendency of content feeds that are based on likes and swipes instead of searches, instead of specifying what we want, we are reduced to approving or dissaproving (until we are removed from the loop entirely)

      • Lammy 1 day ago
        Here's my “favorite” chart showing the steady decline in software quality over the last 25 years: https://trends.google.com/explore?q=%22something%20went%20wr...
        • sam1714 1 day ago
          That's more like the rise of Google (which may or may not be correlated to software quality), as "something went wrong" is an error message popularized by them.
      • freecodeio 1 day ago
        considering how the experience with most SaaS products right now is insufferable with broken states and UI, I imagine the same broken behavior is leaking in every other area of product development, so I agree
      • nswizzle31 1 day ago
        Source = trust me bro. What’s that bias where things are so rare that when they happen it makes news, so you think they’re very common (eg plane crashes?).

        Uptime and quality of software is about 10000x better than it was 10 years ago. Things just work all the time now.

        • sph 1 day ago
          > Uptime and quality of software is about 10000x better than it was 10 years ago

          Extraordinary claims require extraordinary evidence. You're being the 'trust me bro' in this situation.

          • saghm 1 day ago
            I'd argue that just illustrates the point that it's impossible to have an interesting discussion without a stronger claim from the parent comment. If I say something that's too vague to falsify, and you can just as plausibly claim the opposite, you're not at fault for failing to turn my claim into a more rigorous one.
        • hugeBirb 1 day ago
          Also, source = trust me bro? How can you make the opposite claim without backing it up either
    • philip1209 1 day ago
      Internetworking
      • Sardtok 1 day ago
        I think you mean internotworking.
    • arisAlexis 1 day ago
      Any reason why it isn't the case?
      • sosodev 1 day ago
        Because there’s not one, shared cause between the outages. The lay person claims it must be AI, but that’s a bit meaningless.
        • bobson381 1 day ago
          The reason I say not entirely, is because there is kind of a shared cause. One way of looking at is is that the root cause is complexity, optimized for efficiency and speed rather than robustness. It's a feature of the economic superorganism we are operating as part of. See Joseph Tainter on Collapse of Complex Societies, the work of Nate Hagens, Schmachtenberger & Liv Boree talking about Moloch, etc.
          • saltcured 1 day ago
            Or put a bit more cynically, it is a symptom of a society-wide drift towards lower accountability and duty of care. It's a type of tragedy of the commons. Selfishness and externalized cost/risk means nobody optimizes for systemic reliability.

            People "vibing" everything is just another component of this, not the root cause. Using AI tooling is just the new technique for what used to be called phoning it in, or going through the motions, or faking it until you make it.

            Unfortunately, it's nearly always been an effective tactic for a selfish few. It's problematic when we don't have an equilibrium that keeps it in the minority...

          • skybrian 1 day ago
            Yes, that's how you take separate causes and transmorgify them into one omnicause, by muddying the waters until you can't see the differences anymore. Don't do that! It just adds confusion.

            (The same thing happened with separate company's problems becoming "Big Tech." It didn't help.)

            • xboxnolifes 1 day ago
              Once you give separate things a shared name they become the same thing. We could call those in the claim to be "race to the bottom effect" or "incentives against resiliance". The water is no longer muddled, its now a slushie.
              • AnimalMuppet 1 day ago
                You could call it "category theory" or something.

                Snarky, but there's a point: Abstraction leads to viewing different things as the same thing. They are - at one level of abstraction. And they aren't at other levels of abstraction. It's not a problem to have both viewpoints.

                The trick is that, when someone blends things together, to not lose the fact that they are also separate (at other levels of abstraction).

                So I have no problem saying that "optimizing for efficiency and speed" is the common cause of all of these problems, and saying that they all have different (immediate) causes. Both can be true.

                • skybrian 1 day ago
                  Sometimes these abstractions are useful for design or analysis, but maybe not for explaining them to others? A library or programming language will probably be more usable if its users don't need to know any category theory.

                  I'm not sure "optimizing for speed and efficiency" can count as a common cause, though, because it's not a single event. It's more like a set of events that have something in common.

        • foo12bar 1 day ago
          Maybe we're not looking far enough up the chain. Everyone used to use forums and other websites run by other people (eg Substack) to read and discuss about things they don't understand.

          Now when people have a question, they ask AI, which is wrong a lot of the time.

        • vintermann 1 day ago
          What does the "pro" person claim? That it's Iran?
          • coldpie 1 day ago
            That there has never been an incentive to build quality software. As a result we spent 50 years building our society on top of poor quality software. We're paying for it now and will continue paying for it until the incentives change and quality becomes a focus.
          • UncleOxidant 1 day ago
            I suspect there are incentives for Iran not to be identified as the culprit right now whether they are or not. If it is indeed Iran that would show they have the capability to do this and maybe the administration doesn't want that to get widely circulated. (and if it is a state actor it doesn't have to only be Iran - could be a coordinated effort - NK just did the big Binance heist and Russia can do this kind of stuff too)
            • Jach 1 day ago
              NK's heist was against Binance, not Coinbase.

              The main incentive for any nation state to avoid attribution to direct and material harm to many Americans is that they risk awakening American bloodlust. Americans are unhappy about the current Iran situation and wish it didn't start and that it would end already, but sufficient Iranian escalation would turn that around real quick and suddenly there are boots on the ground among other things.

              • UncleOxidant 1 day ago
                From the other direction I could see that the administration wouldn't want to attribute incidents like this to Iran because it would just make Americans more unhappy that the administration decided to poke the hornet's nest when they really didn't need to. And like with not anticipating the closing of the SoH, they completely dropped the ball on cyberattacks as well.
          • tsol 1 day ago
            I've heard claims that it's the people trying to get the US to fight Iran, recently unsuccessfully. Mossad is known for being very skilled. Not saying it necessarily is, but it goes back into the distrust of government angle. Trump saying he's fighting this war in Part for Israel doesn't help.
          • sosodev 1 day ago
            That it’s a nuanced mess of factors. Like reality always is.
        • Waterluvian 1 day ago
          Imagine a tree of relationships branching up to the very top labelled, "Because the Universe Exists" as being the root issue that relates everything.

          Saying it's not related applies an assumption of what level of specificity you are thinking about. And that's how you quickly de-sync with other people, especially "lay people" who look at you funny when you claim there's just no way these things are related when it's hard to believe that based on experience alone.

          As other comments suggest, it probably is related to an extent. Related by complexity of our Internet-based systems, related by age, related maybe even by enshittification of expertise, etc.

          The further up the tree you find these relationships, the less specifically actionable they are (you cannot CVE enshittification), and the more they are about "what are the overall social/cultural/human issues we're grappling with in this era?"

          • mlyle 1 day ago
            Aviation got a whole lot safer when we started looking at the systems causes instead of the specific ways individual people, airlines, manufacturers screwed up. Ditto for anesthesiology, cars, etc.

            Sure, focusing on individual small causes is helpful too, and it's a lot harder to do the higher and higher level system stuff. But, it can be done.

            edit, P.S.: It's worth keeping a bare level of suspicions that there could be a common lower level cause, whether through inadvertent dependencies added or through hostile action.

        • PhunkyPhil 1 day ago
          To be fair, have there been post-mortems for any of them? Besides vague tweets or press statements?
        • Muromec 1 day ago
          > The lay person claims it must be AI, but that’s a bit meaningless.

          Must be the water then

        • cyberax 1 day ago
          Is there?

          Apparently the FAA outage was caused by the pressure to do things quickly, fueled by the AI urgency.

          It feels like all the IT industry is feeling this pressure.

        • aaroninsf 1 day ago
          a) this is not known, there is no reliable data on either the FAA or new bank issue; and the information forthcoming from Microsoft may obscure primary causation

          b) there is also a risk of sophistry and language games obscuring common sense truth

          In the world where a state actor is engaging in active sabotage as part of a cyberwarfare effort, there is one shared cause.

          Which exploits are responsible in each case may differ. Still one cause.

          Do I think it's Iran?

          No idea.

          But the point is, none of us here have any idea at present.

      • Fr0styMatt88 1 day ago
        I can see why people think it. I don’t think I remember a 7-day stretch like this…. ever?

        - Massive Microsoft outage - FAA ground stop - Now this outage

        Definitely a black swan at least? It’s not even the number of failures, it’s the scale.

        • TheAtomic 1 day ago
          Also last week, a hack of Nat'l Weather services web site (but not affecting services) and a minor outage at some Amtrak services on the East Coast.
      • doctor_radium 1 day ago
        It's true that we don't really know, and my first instincts lean toward hacking by a foreign government, but then I start thinking about media reports of shark attacks every summer. Shark attacks aren't unknown, but certain media channels like to pounce on them for ratings, and after a while to the average person it might seem like Sharknado. Stuff goes down all the time, but in this moment we're just extraordinary conscious of it.
    • ForHackernews 1 day ago
      One might argue that lazy sloppy work and the generally pathetic state of IT maintenance is the common cause.
    • cainxinth 1 day ago
      My dad is a doctor. He keeps asking me similar questions.

      I told him it's just like how he keeps telling his patients to watch what they eat, drink and smoke less, and get some exercise, but they don't, and then they have a heart attack and wonder why.

    • Alex3917 1 day ago
      If an unfriendly foreign power were attempting to impose on America the insecure and unstable vibe-coded digital infrastructure we’re building today, we’d probably view it as an act of war.
      • jsLavaGoat 1 day ago
        Everything you said has been true for a long time, vibe coded or not. How much of the Internet proved to be exposed to a supply chain attack on node? That wasn't AI, that was just how things worked out.
      • nsoam 1 day ago
        Or the H-1Bs
      • stasomatic 8 hours ago
        Interconnected or not, the events feel like they are. Like Mongolian skirmishes, hit a flank and retreat, Parthian shot, appear out of nowhere elsewhere and do the same and hit a different flank, a bite at a time, cause panic and distrust.

        This doesn't seem to happen in other countries (except the fridge hack) and they use the same tech. What are they doing right? Do they beg the Congress for $50B to improve their ATC infra? It's annoying to hear Duffy say "my network", "my routers" give me money. It's OUR network.

    • skeeter2020 1 day ago
      Same with my parents, until I told them if FAA comms ran on teams and outlook we would have been boned a long time ago!
    • porphyra 1 day ago
      Maybe Jensen was what was keeping all of the nation's infrastructure running
    • foo12bar 1 day ago
      There does seem to be a correlation with the fall in use of Substack and the frequency of these fuckups.
    • Dependance 1 day ago
      Not a bit occam razor's fan, eh ?
    • bparsons 1 day ago
      Three massive outages in three business days in a row does seem like a pattern.
      • _ette 1 day ago
        Or is this a period of every blip being watched more closely and becoming a headline after the recent two?
        • account42 11 hours ago
          Dude these are not minor issues that would have flown under the radar in the past either.
      • ignore 1 day ago
        [dead]
    • DIE_DIE_DIE 1 day ago
      This is the obvious consequence of (failing to) replace software developers with AI slop. Despite the protests of the Antrhopic shareholders who dominate this site's comments, LLMs are hideously poor at designing software.
    • madaxe_again 1 day ago
      I’ve had people asking me if it’s Jensen Huang’s doing.

      I guess for most “technology” is one big blurry mess.

    • jmyeet 1 day ago
      We (on HN) as generally technical people (despite the HNers who seem to think you can vibecode a NVidia competitor) that all of these incidents are unrelated with vastly different causes.

      But I think there's an increasing consensus among "normies" like your coworkers who see these as related. They don't know why but assuring people they are unrelated fails in messaging terms when it feels related.

      I think this will be an increasing problem where so much more code is not only written by AI but reviewed by AI. Things will slip through and companies won't be punished for these outages in terms of public perception (because it'll happen to everyone) or by the government. We just don't have serious consequences for corporate malfeasance or gross negligence. Like AFAICT the Sackler family should all die penniless in 6x8' grey prison cells.

      Unfortunately we live in a country where the Sacklers get to stay billionaires while 1 year olds are shot dead at the behest of Walmart over a pack of diapers that weren't even stolen... with zero consequences [1].

      It feels like this is just the next level of entshittification.

      [1]: https://www.nbcnews.com/news/us-news/mississippi-ag-says-pol...

    • reaperducer 1 day ago
      My nontechnical co-workers see the Microsoft 365 outage, the FAA ground stop, and things like this as one interconnected problem.

      Wait till they hear about United Airlines.

    • rsoto2 1 day ago
      It is interconnected. Everyone bought the "cloud is up 24hrs" bullshit. My company had physical servers and we had 99% uptime because we had competent people. Profits are driving unsafe transitions, such is capitalism. Look at our crumbling old infrastructure and ask yourself why the "digital highway" would be any safer.
      • neural_thing 1 day ago
        Banks, the FAA, Microsoft all have significantly higher uptime than 99%
    • joering2 1 day ago
      Because they are. Trump just said that deranged democrat/antifa was cutting cables everywhere.
    • saghm 1 day ago
      Honestly, I can't help but hope that ends up leading to enough popular support for more regulation for AI that something gets implemented. On the other hand, there's also ample historical precedent for hoping that a mass movement heading in the right direction ends up stopping at the point that feels reasonable and then being surprised when the momentum continues and people like me looking foolish for thinking that it wouldn't, so I might just be overly optimistic.
      • swatcoder 1 day ago
        AI is not the cause of this kind of problem and it's regulation won't make it go away. Maybe it needs regulation, maybe it doesn't, but you're missing what's happening if you think that's what these widespread failures and vulnerabilities are about.

        AI is acting as a sudden precipitator for problems that have been latent and growing for at least 15-20 years.

        In pursuit of maximally rapid innovation and reduced capital expenditure, the software industry adopted a "move fast and break things" mentality and homogenized code and service infrastructure over a few winning cloud vendors, SaaS providers, and code solutions/frameworks/packages/etc.

        The pitch was that you could rely on downstream experts to do better work than you could, that the biggest vendors and publishers had the best experts, that have them invest the capital would look better on your accounts, and that their scale would give you the best value for your investment.

        The caveat is that effectively all technology products have serious faults lurking in them, even with the finest expertise working on them (not that most actually had that), and that when everybody homogenizes over the same providers and solution the industry shifts from a bubbling froth of localized bugs to a heavily correlated house of cards where widely correlated failures manifest in widespread collapse and catastrophe.

        Whoops.

        AI is acting as a breeze against the house of cards, but lots of things could have and would have and did, especially as it continued to inflate year over year.

        • saghm 1 day ago
          If you're saying the parent comment I responded to wasn't talking about AI, what are you saying they were talking about? If you agree they were, why are you trying to correct me instead of them?
          • swatcoder 1 day ago
            From experience, since the parent commenter didn't specify, I would take the "coworkers" as having a vague sense of dread and distrust with no real confident insight to the cause. They're likely to nod along to any explanation they hear from anyone they think is more technical or news-following than themselves, and then maybe repeat whatever last such they heard when chatting. With the occasional exception of politics, that's how most people approach technical topics outside of their expertise. I didn't guess what they might have in mind today at all, and didn't feel the need to.

            But you did speak directly to AI, and specifically to advocate for regulation of it, which is why I replied to you. You were being specific and material in a way that the parent commenter was not, and that invited a different response.

            • saghm 1 day ago
              I didn't specifically claim that it was the direct cause of this though, so you still decided to confront something I didn't claim over something the parent comment didn't claim
            • randomNumber7 1 day ago
              > With the occasional exception of politics, that's how most people approach technical topics outside of their expertise

              You give other people way too much credit.

        • retwakm4 1 day ago
          [dead]
        • alala-lala-lon 1 day ago
          [dead]
      • SoftTalker 1 day ago
        What's actually more likley are power grabs executed while the chaos is attracting all the attention. That's how we ended up with DHS.
      • canibalrex 1 day ago
        I don't really see how regulation for AI helps. Companies were on this crash course of trusting Microsoft (or pick your other dependency that cant be eliminated as it is a monopoly or the minimum) to provide more and more of surrounding infrastructure. Until they had AI to run simulations of outside.competence there wasn't a snowball's chance in hell they were competent enough at IT to run someone else's infrastructure.
    • doctorpangloss 1 day ago
      Haha ask them how much they'd pay for fewer problems
      • account42 11 hours ago
        You should first explain to them how they can know that they are paying for fewer problems and not just higher executive bonuses and shareholder dividends.
        • doctorpangloss 3 hours ago
          Somebody pays you every month presumably knowing that. I understand the thrust of what you are saying but it's funny to me that your response to my simple idea, "people do not pay for shit," is, "lame hyperbole"
    • hparadiz 1 day ago
      Overall uptime is better than pre internet 9-5. It's so typical to become so used to technology that any disruption seems bad.
      • Mezzie 1 day ago
        This misses that expectation versus reality is key to how a human experiences something. If it's 20 degrees F in November where I live (MI), I'm going to grumble and get about my life. If it's 20 degrees F in July, I'm going to be freaked out. The same situation (no access to banking outside of business hours) can have radically different context and communicate different things to onlookers. See the former USSR's reaction to seeing Swan Lake on TV in 1991.

        The absence of something that once existed is not the same as that thing never having existed. A null entry and a non-existent entry aren't the same.

      • hackyhacky 1 day ago
        Yes, we should all stop complaining about unexplained technical interruptions in our infrastructure and go back to being grateful for every minute that my bank's website is open on the weekend.

        /s

  • nla 18 hours ago
    Don't believe a word they say. I ran technology in retail there over 20 years ago. When the SQL Slammer worm shutdown all the MSFT dBs, the bank put out PR after PR saying we weren't affected. We were hard down. They lie for a living.
    • nom 6 hours ago
      Yeah they will only admit to something when it no longer can be hidden.

      They can't hide the fact there were issues with their systems, many institutions must have observed the wrong behavior.

      So 'a temporary bug in an isolated system' is the absolute bare minimum they will say for now.

  • khriss 1 day ago
    Coming so soon after the Office 365 debacle and the meltdown in the FAA network for air traffic control which caused the ground stop, it does raise the possibility of a targeted cyber attack being the root cause.
    • rossdavidh 1 day ago
      Or, perhaps there was some change in the IT workflow a while back, which causes an accumulation of technical debt, that is starting to make itself felt?

      If it were an attack, it would be something that were more than annoying. The crypto theft by North Korea, for example, was obviously a bad actor, with a clear motive/benefit.

      FAA incident also came like 1-2 days after a new system was rolled out, which would be quite a coincidence if it were not related.

    • nashashmi 21 hours ago
      I rule out Russia or Iran behind this. These are long dormant bugs that surfaced. All of them surfaced at once. Someone knew of these bugs. And is now slowly preparing every industry to harden their systems against it.

      It is not an attack. It is cyber defense mechanism. Like constantly releasing viruses to build herd immunity.

      • mrbombastic 20 hours ago
        I don’t believe our government is competent enough to do something like this anymore.
        • nashashmi 20 hours ago
          I don’t believe there are competent enough people in govt to stop from doing this crazy stuff.
    • asdfman123 1 day ago
      It's possible, but if you were a state actor, why would you want to give the US advanced warning about your capabilities and give them a chance to fix their issues?
      • patcon 1 day ago
        Maybe you're just script kiddies? Or grey-hats trying to expose and create incentives to prioritize/fund repair before worse things (e.g. digital attacks as acts of war) start to happen

        But I don't know, I'm just someone with some minor experience larping in security who is spitballing :)

        • cheeze 21 hours ago
          This is very much a thing in the console hacking scene. Usually folks hoard 0days until they can get a hypervisor exploit or an untethered jailbreak, but with AI it's been easier for people to find (and publicize) 0days.
    • mastax 1 day ago
      Only if you ignore all the information released by the FAA and Microsoft about those incidents.
    • kokanee 1 day ago
      They said "it's nothing cyber-related" whatever that's supposed to mean
  • stmw 1 day ago
    Note than any of the core processes there likely run on very old but very reliable mainframes and associated software stacks (think COBOL, CICS, IMS, DB/2 on 390). More (non-technical): https://www.rocketsoftware.com/en-us/insights/why-financial-...
    • fock 12 hours ago
      how do you access those, when your network is down?
      • stmw 2 hours ago
        Which network? These are very on-prem and networking sometimes still means things like SNA, TokenRing, etc., no IP or Ethernet.
  • techmunky 1 day ago
    West Coast, 3 out of 5 of these load for me no problem (I only use 3 of these services, cant check the others). Might be an east coast AWS thing? Can't seem to open a WireGuard tunnel into my home network on the east coast though, which is unusual. o_O
    • marinmania 1 day ago
      BOA and Chase both work for me on East Coast.

      It's really hard for me to get a sense of how much outages are on the rise - versus every minor issue getting a lot of attention.

      • jjice 1 day ago
        Having success with Chase on the East Coast as well at the moment. I'd like to know the actual scope of this as well. DownDetector as a source is kind of low value IMO, but the banks have released statements. Curious if these are actually a large issue since I don't have the opportunity to run into it myself.
    • ecshafer 1 day ago
      You may be unable to actually perform changes. From my experience in companies, the main path homepage -> log in -> check balance was like literally 98% of all users, so those had some read only caches and backups that could be put into play for no downtime. But trading, moving money, etc. may still be down.
    • throwaway27448 1 day ago
      I can tunnel into the east coast from south africa at least. Doesn't touch AWS though.
    • CodingJeebus 1 day ago
      Are you actually authenticating or just landing on main sites? The more public/marketing type pages are often hosted separately from the account platforms.
      • techmunky 1 day ago
        Signed in via mobile app.
  • kpw94 1 day ago
    • Spalau 1 day ago
      Only in the sense that WWIII has already started, and this is a cyberattack.

      It's good that Iran, Russia, and North Korea are showing their teeth this early, when their attacks can be studied and counteracted.

      • tty456 1 day ago
        > when their attacks can be studied and counteracted

        You're assuming we have competent cabinet-level leaders along with a seasoned leadership structure that hasn't quit or been fired for no cause.

        • nukanse 1 day ago
          This war was planned long time ago. This is why there was so much migration to nonrenewable energy. Most oil comes from Iran and the Arab nations, which are cut and easy to paralyze in a global war. Some people also claim that the pandemic vax was a cover for immunization to a biological weapon and the covid was a common flu.
          • tty456 23 hours ago
            "and the covid was a common flu"

            You mean conspiracy theorists and Fox News hosts?

        • drstewart 12 hours ago
          You're assuming the cabinet is the one studying the attacks and doing red/blue-team analysis themselves
  • 1-6 1 day ago
    It's never a bad idea to carry a few dollars in case of a payment processing problem.
    • nunez 22 hours ago
      This is becoming ineffective as places increasingly go cashless.
      • speedylight 19 hours ago
        Cash will always be king, even if 1% of the population accepts it for goods and services.
      • account42 10 hours ago
        Those places will quickly start accepting cash again when a large number of their customers cannot pay with card because the system is down.
    • Spalau 1 day ago
      All governments are telling people to go prepper. Google it. This is WWIII. The nukes are coming.
  • silicon3383 1 day ago
    Scuttlebutt on Twitter is that it's a compromised Optimizely script
  • sega_sai 1 day ago
    Presumably this must be the "best" time for phishing/fraud ?
    • giwook 1 day ago
      You forgot to add "asking for a friend"
  • danso 1 day ago
    Why do people assume that this crash is the result of AI vibe coding (i.e. these companies rolling out buggy production code) versus AI-empowered vulnerability seekers? Seems more likely that a group is trying to swarm and exploit a possible vulnerability common to these banks' operations, than it is that Bank of America, Chase, and U.S. Bank all happened to roll out bad vibe code on Monday.
    • codingdave 1 day ago
      From the article: "...a technical issue at JPMorgan Chase caused ripple effects throughout the financial system."

      Nobody is claiming multiple parallel failures. They are claiming one failure that impacted multiple organizations.

    • bdcravens 1 day ago
      Because people want to believe that AI code is inferior, and that it's about to go poof with a return to the previous status quo.
      • JsonDemWitOster 1 day ago
        I know the leg I'm standing on is correlation but did we have such widespread outages pre-LLMs? The two that come to mind is that one time some intern pushed a config change that borked us-east-1 and Gitlab deleting their DB. And both had clear comms on the situation within acceptable time frames.

        Also, I don't think it's that "AI code is inferior" but rather devs got sloppy and no longer review code. Not to say that everyone's lazy but me---a lot of systemic/environmental factors could be the reason.

        • dghlsakjg 1 day ago
          Large outages are not a new thing.

          People were making memes about the GitHub unicorn a decade ago.

          • JohnMakin 1 day ago
            No one is saying they are, they’re asking whether the frequency and scale has changed.
            • tadfisher 1 day ago
              With GitHub specifically, yes. But I remember AWS/GCP/Azure outages were like a yearly occurrence before the 2020s, and we kept having the same conversations back then about infrastructure resilience, multi-cloud, etc. that were too hard to address then and still too expensive to address now.
              • jeremyjh 1 day ago
                Maybe one major platform had a major outage annually but AWS certainly didn't have a major outage every year. More like every 3-4 years. I remember exactly two in the last 10 years that impacted the production application I run.
                • dghlsakjg 1 day ago
                  Here’s a timeline of major AWS outages: https://awsmaniac.com/aws-outages/

                  Just because an outage doesn’t affect your specific app doesn’t mean it didn’t happen. By that logic Microsoft didn’t have an outage last week since my windows gaming pc was fine.

                  • jeremyjh 1 day ago
                    We're not really talking about the same things here. I'm talking about the outages where dozens or hundreds of known platforms are down and it makes the front page of HN and major media coverage. That is the magnitude of many of these recent incidents.

                    I know there are more incidents than that.

                • tyre 1 day ago
                  There have been many more than two in the past ten years. Maybe not for you, but depending on the region/AZ, especially us-east-1 as it’s aged, outages are impossible to avoid.

                  Much better than Azure, but that’s not a high bar.

                  • jeremyjh 1 day ago
                    How many made the front page of hacker news and featured on major broadcast networks? Because that is the level of incident we're talking about here.
            • bdcravens 1 day ago
              It absolutely has. The "why" is a matter of first-party versus third-party (bad AI code making bad services, or good services unable to handle the load from everyone else pushing AI code, or a combination of the two)
              • esseph 1 day ago
                > The "why" is a matter of first-party versus third-party (bad AI code making bad services, or good services unable to handle the load from everyone else pushing AI code, or a combination of the two)

                Decades of cycles of layoffs and outsourcing, where the core business or organization no longer operates business-critical or organization-critical functions.

                • jeremyjh 1 day ago
                  Or, the thing that has happened on a huge scale in much less time than decades.
                  • esseph 1 day ago
                    It's the combination of moving faster with less domain knowledge, but AI confidence.
            • joesb 1 day ago
              But how do we know it is because of AI vibe coding though?

              It could be because some critical infrastructure are due to upgrade after a long delay and this is inevitable at some point.

              It could be that AI agentic hacking has pushed for some rearchitecture in faster deployment time, which may result in big outage for industry that has so much legacy and used to big bulk deployment only.

              Or it could just be because we rely more on cloud infrastructure than previously.

        • bdcravens 1 day ago
          Failures aren't just a product of AI-written code in the infrastructure, but the infrastructure trying to bear the load of everyone else's AI-written code at massive scale.
        • theseamusjames 1 day ago
          The crowdstrike outage was much much bigger, both in reach and in impact. It took days to weeks to recover from.
        • CptKriechstrom 1 day ago
          Remember when AWS messed up their BGP Routes?

          edit: it was Facebook, not AWS

        • fourside 1 day ago
          But that’s the point around correlation =/= causation. What the parent comment is saying is that the cause could be that attackers have much more sophisticated tools to bring down services. The timing with AI adaption doesn’t prove on its own that lower code quality is the reason.
      • paimapi 1 day ago
        It's not that AI code is inherently inferior, it's that the bulk of the work engineering does now is code review and the scale of reviews has become exponential which means a lot of stuff just gets punted to agents and not carefully examined by anybody with actual subject matter expertise. Plus automation and platform work (like managing test suites) has also similarly become deprioritized and QA in general is being obsoleted as a field (for worse, really).

        This is a systemic issue that's driven by AI but the responsibility lies with leadership who are still operating on the move fast and break things mentality. With everybody swinging for the fences and the median blast radius increasing and the checks/roadbumps eliminated you end up with a lot of incidents because the total number of merges and the size of the diff has increased by quite a bit per sprint.

        • jeremyjh 1 day ago
          Also most engineers have always been terrible at reviewing code. They could find most bugs when working through it line by line manually as they wrote the code and the tests, but most could never find bugs in their peer's code through inspection. Now that this - along with business and product understanding - are the primary skills they need, a lot of developers are under-skilled for their job requirements as they suddenly exist today.

          Staff-level engineers can do this work because they always have, but if they are suddenly the only ones who can in an org then they are simply overwhelmed.

      • EricDeb 1 day ago
        No the issue with AI code is not that it's inferior but that it solves the wrong problem or doesnt believe an edge case is necessary or other contextual issues unrelated to whether the code is actually correct as written.
      • uludag 1 day ago
        Or maybe that cyber attacks are generally known after the fact that they are cyber attacks via things like extortion.

        Like take the Binance hack for example. A naughty hacker, instead of stealing bitcoin, could have just caused mass outage and disruption without stealing anything, but didn't because bad actors usually want some financial benefit.

        Like if we learn that PII was stolen, then it's be safe to say that this was a cyber attack.

      • lbrito 1 day ago
        That's just a strawman argument.

        While there may be people arguing for that, the more relevant point is that regardless of any possible positives of AI code, there are clear negatives that are being largely ignored. Specifically in such outage cases, the volume of code being put into production with little or no review or understanding.

      • shoobiedoo 1 day ago
        I don't know if it's as much as people just hoping that it fails, or the fact that in my opinion a programmer is far more likely to verify code that their reputation might depend on instead of just shoveling in whatever an agent spits out.
      • freecodeio 1 day ago
        blaming this problem to vibe coding != ai bad at code

        shit programmers have always been a thing, these same shit programmers now putting output from claude straight to production

        on top of this there are literally PMs getting involved and doing their own PRs, see vscode recently

      • rs999gti 1 day ago
        AI code is inferior, but business, the media, and regular people don't care about that. They just want fast and cheap, so you get the slop code you pay for.
        • Loughla 1 day ago
          I find it fascinating that anyone thinks computer programs will be different than anything else.

          Clothes. Food. Appliances. Tools. Electronics.

          All of these things got worse, but got WAY cheaper. All of these industries had to shed excess labor like crazy in pursuit of optimization at the cost of quality.

          Why would coding be different?

          • randomNumber7 1 day ago
            Because you write a program once and then you can sell copies without extra costs per unit you sell.
      • pessimizer 1 day ago
        > Because people want to believe that AI code is inferior

        No, people want to believe that it is good, despite all evidence. However the status quo will not be returning, because it will get better. The catch is that it will somehow end up in your pacemaker or the brakes on your car while it still sucks.

      • Barrin92 1 day ago
        >Because people want to believe that AI code is inferior,

        This is armchair psychology and extremely bad one because the people who have been concerned with engineering quality have been vastly outnumbered by people who just want to be able to turn their brain off and "ship fast" for probably 15+ years, and that's been reflected in reliability long before LLMs came around.

        The code these systems churn out at scale is absolute trash but the financial incentives are to produce out more of it because nobody is ever actually held liable for software breaking in our age

        • fock 1 day ago
          yes, I read some java today where I wished for the clankerd to have done it...

          Think system(curl) and random-if-cascades which are never reached - they removed the system(python) in this iteration though...

          At this point I think AI-code is just average and probably just fine for corporate environments/SaaS (especially given what thkse produce today), but I still think infrastructure should be a) understood by humans b) produced without installing a backdoor on the dev machine and c) a human should decide what to build and be able to reign in the machine.

      • whateveracct 1 day ago
        if only...
      • Grimblewald 1 day ago
        it is though. at minimum qa/QC of ai code is way worse. Things are hitting prod right now in many sectors that should never have hit prod. Amazon on mobile (at least android + ff) is unusable ATM as the most recent example.
        • 1970-01-01 1 day ago
          Just like humans, it varies wildly. Stating anything other than that is speculation at best.
          • bdcravens 1 day ago
            This gets to the implied argument: that humans are inherently better to machine-generated output. The truth is there are a ton of programmers getting paid well to generate crap. Anyone's who been a part of any team that isn't some highly curated A-squad knows this is true. Hell, even teams of 1 know it to be true, when you were rushed, didn't understand the domain or stack, under personal pressure etc.
            • Grimblewald 1 day ago
              I never implied human was better, im sure AI can and does generate better code. The problem is mostly better doesn't mean always better, and the issue here is Qa/Qc is gone, because of this "ai code is better" assumption. With it came a deluge of bugs that no longer get caught. We all see it everywhere, all the time, im bot syre why this take of mine is in anyway controversial?
          • Grimblewald 1 day ago
            All I know is I used to assume my device is malfunctioning when things didnt work, but given how common place broken UI's have become, my devices are now rarely the suspected point of failure.
          • Creamsicle47 1 day ago
            Except human code is reviewable at a reasonable speed
        • swozey 1 day ago
          Before laying off half of my company the last place I worked fired QA first
    • Aperocky 1 day ago
      The initial postmortums are out already for the 2 previous event, and they appear to have nothing to do with vulnerability.
      • greenchair 1 day ago
        my sweet summer child
        • Aperocky 1 day ago
          The reliability of reports today are generally poor, however, a technical person can tell relatively easily as to whether the report is likely to happen vs. unlikely. And if it involves the simplest thing that could go wrong gone wrong, then that's usually the right explanation. Otherwise known as Occam's Razor.

          A deployment on Friday screwed things up vs. an adversarial attack which involved multiple agency needing to cover up together with unclear motive and huge risk down the line? It's an easy pick.

          • eliago 7 hours ago
            I would not underestimate the US government coverup abilities, especially when the department of homeland security is involved. I do agree it is most likely incompetence though.
        • DEF14A 1 day ago
          What a snarky, condescending thing to say.
          • eliago 1 day ago
            I personally thought it was humorous but I can see the perception.
      • werdnapk 1 day ago
        Do you really believe a guy doing some wiring caused an air traffic shutdown last week?
        • motionlessveloc 1 day ago
          Yes, at least in a "straw that broke the camel's back" sense.

          Chernobyl was caused by a guy flipping a switch, Challenger blew up because of a faulty O-ring, etc.

          • DaiPlusPlus 1 day ago
            That is all true, but those shouldn't be the stated reason that's reported to the public in a press-conference headed up by a US departmental head; that's how misconceptions poison the information-space for everyone.

            I also wonder if Trump's administrations' spokespeople even realize they have a massive credibility problem, and that if they want people like me to believe anything then I have to hear it from a mid-level functionary, not a political appointee.

            • pantulis 1 day ago
              Proper root cause analysis takes its time so the last obvious fail in the chain of events is what gets published quickly.
            • augment_me 1 day ago
              You are in a minority, and no-one cares anymore.

              You say whatever makes your stocks go up and now down, and if you don't, you are stupid because it's normalized now, there are no repercussions, and if you don't someone else will.

            • ignore 1 day ago
              If the mid-level functionary tacks on, "This is why we demand that Congress allocate $50 billion in new funding for modernization!" do you still believe them?
              • DaiPlusPlus 1 day ago
                > If the mid-level functionary tacks on, "This is why we demand that Congress allocate $50 billion in new funding for modernization!" do you still believe them?

                Of course not, but that's the point of wanting to hear something from someone lower in the org chart: people lower-down on the totem-pole have fewer things within their remit that they can credibly talk about publicly; so if they ever speak of things above-their-pay-grade (like a technical team manager giving an official statement concerning departmental-level funding) then it's a signal they aren't speaking honestly.

          • hunterpayne 1 day ago
            "Chernobyl was caused by a guy flipping a switch"

            No, just no...Chernobyl was caused by the staff doing an experiment that to put it mildly was wildly risky. They disabled safety systems with wrenches (meaning they couldn't easily be reenabled). Challenger blew up because staff at NASA ignored warnings from the people who made the O-rings. Saying these were "straws" is completely irresponsible. Attitudes like this are what cause outrages. Also, unless you are being dishonest, this is the worst string of outrages in the history of tech. There is a cause. It can be found. And unless you think in these terms, you have no business working with a production system in any way.

            • Liuser 1 day ago
              You’re not in disagreement with op. That’s the point motionlessveloc is making, it’s a series of mistakes (straws that make up the hay on the camel’s back) that points to a broader systemic problem.
            • fwip 1 day ago
              I think you missed their point a bit. I believe they are saying that the cause was the faulty wiring in the same way that Chernobyl was caused by a switch flip.

              In other words, the wiring problem might be the wrong thing to focus on - but it probably happened, and the systems likely wouldn't have crashed that day without it.

        • DaSHacka 1 day ago
          Do you have any reason to believe it didn't?
        • forshaper 1 day ago
          It's the sort of thing that is common in all systems I've been inside- which is to say, something little, to do with configuration, hardware fiddling, etc.
        • tygon 1 day ago
          While AI rollouts have had issues, it is far from safe to assume just because there is an issue, it is automatically AI-related or vibe code specifically. Things always are breaking. Lines go down, servers have power failures, etc.
          • dclowd9901 1 day ago
            With this level of frequency and breadth? The _only_ common factor is AI implementation.
            • tygon 1 day ago
              The trend is your friend until it bites your rear end. To give a topical answer, the Outlook issues were blamed on a software bug from months ago. While AI is being embraced en masse and causing all manner of issues, we have AI issues as _well_ as traditional "it broke" moments. The main point is we should not jump to a conclusion without stated evidence. Just because AI is to blame sometimes, does not mean it always is.
          • cgio 1 day ago
            I think it is the co-occurrence of many incidents across different industries that points to potentially common causality.
        • tiahura 1 day ago
          LAX outage is blamed on a single computer

          https://www.latimes.com/travel/la-trw-laxdelays15aug15-story...

          By Tami Abdollah Aug. 15, 2007 12 AM PT

          U.S. Customs officials said Tuesday that they had traced the source of last weekend’s system outage that left 17,000 international passengers stranded in airplanes to a malfunctioning network interface card on a single desktop computer in the Tom Bradley International Terminal at LAX.

          The card, which allows computers to connect to a local area network, experienced a partial failure that started about 12:50 p.m. Saturday, slowing down the system, said Jennifer Connors, a chief in the office of field operations for the Customs and Border Protection agency.

          As data overloaded the system, a domino effect occurred with other computer network cards, eventually causing a total system failure a little after 2 p.m., Connors said.

          • BeratnasGas 1 day ago
            I automatically trusted the accuracy/honesty of this explanation from 2007. Zero trust today. We have much to fear from fear itself.
          • solyin 1 day ago
            Running Window 98 SE.
        • brendoelfrendo 1 day ago
          I can really believe that they plugged in a misconfigured router and that a new router on a faster link was able to flood downstream legacy hardware with junk requests to the point where it started to fail.
          • tiahura 1 day ago
            Are RIP and BGP still a thing?
            • cestith 1 day ago
              BGP4 is only still a thing on The Internet.
            • brendoelfrendo 1 day ago
              BGP is very much a thing, and increasingly used inside larger networks instead of just on the edge. RIP should very much be dead, but probably isn't. Afaik OSPF is the interior routing protocol of choice for situations where iBGP isn't used.
        • mh- 1 day ago
          Do I believe a change in networking caused a networking issue? Yes?
        • icepush 1 day ago
          I don't know, do computers use wires ???
        • schiffern 1 day ago

            >Do you really believe
          
          Thanks! Conspiratorial thinking hadn't annihilated my last brain cell yet.
        • shermantanktop 1 day ago
          I bet he feels bad about it. He should have known that flipping over to the fiber link would take out the whole FAA. /s
        • takeda 1 day ago
          [flagged]
          • ragequittah 1 day ago
            This right here is what needs to stop. AI has its problems but everyone blaming everything on AI is starting to wear thin. By all reports the new SMART AI system wasn't even in any way connected to the network that went down[1]. I guess we can put our tinfoil hat on and say every problem we have from now on is AI's fault and nobody will report it, but I'm not sure that helps anything. Probably better to wait and try to see what actually happened [2] before our knees jerk and credibility gets lost for when actual AI problems occur.

            [1] https://apnews.com/article/faa-smart-ai-outage-08af345749c81...

            [2] https://www.faa.gov/newsroom/nationwide-ground-stop-question...

            • fwip 1 day ago
              Hang on - that first link says: "there’s no indication the communications issues that caused the ground stop are linked to the new computer system,"

              As near as I can tell, they are saying that the communication issues are not related to the new computer system (not causative) - not that the new system is physically disconnected from the communications network.

              (The article is a bit confusing, using "connected" and "linked to", which could either be describing physical or causative relationships).

              • ragequittah 1 day ago
                FAA's site says:

                Q: Was this caused by SMART system? A: No. The two systems are not related. SMART is a data aggregator, and it does not feed information into the air traffic control system.

                If you dig a bit further the aggregator gets its data from SWIM, which receives (but does not send) data from the system that went down. I believe it's a 1 way VPN connection that they use to aggregate data because SWIM is consumer facing as well.[1]

                So unless something went terribly wrong the AI couldn't have had anything to do with it. But that won't stop it being the top narrative for all time because nobody will look into it this far and we no longer believe any official stories about anything.

                [1] https://www.faa.gov/air_traffic/technology/swim/sfdps

                • fwip 1 day ago
                  Thanks - I appreciate the clarification. :)
            • Forgeties79 1 day ago
              Can you blame people for not trusting government officials when for the last decade Trump and co have been doing everything in their power to make sure people don’t trust government officials?
              • ragequittah 1 day ago
                Not at all. But the AI would've had to be doing something completely different than its function. It doesn't send or receive data to the system that was overloaded, it's built to receive data from a completely different system. I also think this would be analyzed by enough people that one of network engineers or someone who just generally hates the idea of AI in their workload would've come out and said something by now.

                Don't counter lack of credibility by spreading information you can't verify. It just muddies the water even more. There's people out there right now who think it's the AI apocalypse because a banking system is a bit slower.

                • takeda 5 hours ago
                  It needs data to function and since it is LLM more likely they they implemented MCP that starting DDoSing the rest of the system when LLM was being asked for every single flight.

                  > It doesn't send or receive data to the system that was overloaded, it's built to receive data from a completely different system.

                  You're adding your own assumptions. This is what they said:

                  > Q: Was this caused by SMART system?

                  > A: No. The two systems are not related. SMART is a data aggregator, and it does not feed information into the air traffic control system.

                  It doesn't feed data, they didn't say it is completely isolated, which fits what I wrote.

                  They absolutely have incentive to avoid investigation that the company that donated to Duffy's son in law received nearly a billion no bid contract and failed to deliver a working product.

                  • Forgeties79 4 hours ago
                    I looked into your claim about the son-in-law, and it seems you’re kind of right but mostly wrong. It looks like there were about a dozen bids for the contract and he does not own the company. But he is definitely involved in the company that received the contract.

                    Now don’t get me wrong, Duffy could totally be running cover for him and that is certainly the kind of corruption I expect from this administration. But the details are not quite as severe and direct as you are making them out to be it seems.

                • Forgeties79 1 day ago
                  I didn’t spread anything. I just think people are inclined to think it’s whatever this current administration says it isn’t. I don’t blame them. There is no shared reality anymore, everyone is basically a little conspiracy theorist
                  • randomNumber7 1 day ago
                    It will get better at some point. When Trump is away I promise you to blindly trust the government again.
            • takeda 5 hours ago
              Except on Friday Mullin accidentally said that (when there was a question about attack) it happened after the system was patched and you can on Duffy's face that he was not supposed to say that.

              Then next day they had conference where they came up with the maintenance guy.

              Are you truly believe that a guy replacing a switch somewhere in Arkansas can just shut down the whole air travel in the entire country? The biggest disruption since 9/11 25 years ago? And just happened a day after SMART was deployed.

              We are on tech forum, I would assume I'm talking to tech people who I hope understand how networking equipment operates.

    • fedpost 1 day ago
      I don't think the average person cares tbh.

      It's caused my AI, it doesn't really matter who's fault it is

    • dclowd9901 1 day ago
      The companies in question have confirmed it wasn't due to cyber attacks. That's why.

      What's it going to take for the AI pushers to see that maybe this isn't the road to go down.

    • rs999gti 1 day ago
      > AI vibe coding

      What's new? What has been operating fine until the AI, cloud, and centralized computing came onto the scene?

    • itishappy 1 day ago
      I'd assume that whatever happened affected a single centralized service that these banks all relied upon.
    • tedd4u 1 day ago
      Like everything else it’s probably a combination of both (as well as other causes).
    • bottlepalm 1 day ago
      Everyone is vibe coding now. It's not just you. It's not an assumption.

      Vibe coding, vibe reviewing, vibe fixing. No one has a clue how software works anymore.

      • snorkel 1 day ago
        … and no one wants to work anymore! Back in my day we’d hand assemble code on paper and then whistle it over the phone to a 300 baud teletype in the server room … and that shit never broke! Not once!
        • bottlepalm 22 hours ago
          The difference is back then there was usually someone that understood the system and could fix it.

          Now, no one does. We ask AI to figure it out and fix it. What happens if the AI goes down?

    • surgical_fire 1 day ago
      "AI cannot fail, you can only fail at AI".

      Somehow I knew I would find something like this on this comment section.

    • Ancalagon 1 day ago
      Why can't it be both?
    • rewgs 1 day ago
      IMO the problem is not AI-generated code per se, but rather the utterly absurd expectations and resulting destruction of best-practices and discipline that developers all over are now dealing with.

      My first thought for this week's outages has been that people are moving too fast and are, as the saying goes, breaking things.

    • superfrank 1 day ago
      Because there’s a lot of anti-ai sentiment going around right now. Blaming it on AI slop code supports pre-existing beliefs that AI is useless. If it’s AI hackers then that not only means AI is actually good at something, it’s potentially an argument for why we need increased spending and that there really is a race dynamic at play that we need to win.

      Basically, blaming AI slop supports a lot of pre-existing beliefs, while blaming AI hackers does not.

      • jenniferhooley 1 day ago
        I don't really think it's that. From the people I work with, it's more the push beyond the "capabilities" of AI to the point that we have this amazing tool (AI), which theoretically should allow us to ship much better and more resilient systems. But instead we are pushed to ship really cruddy systems without the proper amount of diligence, review, and testing due to the C-suite's desperation to ship as fast as possible - consequences be damned.
        • preg_match 22 hours ago
          Yeah this is how I view it too.

          AI really does have the potential to make software more robust and higher quality than ever before. We can write more tests, new kinds of tests, do pen testing, do deeper analysis of code. But we're not. This, like software before, is a slop machine. It's about getting things to kind of work just enough that customers are, very temporarily, dazed, and sales are able to sell it. It's a house of cards situation and all we've done is starting stacking more cards on faster.

      • retwakm4 1 day ago
        [dead]
    • sunbum 1 day ago
      Hanlons razor. “Never attribute to malice that which is adequately explained by stupidity.”
      • SAI_Peregrinus 8 hours ago
        SAI's razor: Never attribute to stupidity that which is adequately explained by greed.
      • sanderjd 1 day ago
        I think we need a new razor: "Never attribute to AI incompetence that which is adequately explained by human incompetence."
        • djeastm 1 day ago
          "There's three ways to do things. The right way, the wrong way, and the Max Power way!"

          "Isn't that the wrong way?"

          "Yeah, but faster!"

        • dclowd9901 1 day ago
          If companies are forcing their employees to use AI, which human's incompetence is at play?
          • sanderjd 1 day ago
            Probably multiple humans, I would say!
  • samk3nny 1 day ago
    From Microsoft to the FAA to Chase.
    • marginalia_nu 1 day ago
      The MTBF singularity is the one I'm concerned about.
      • nyc_data_geek 1 day ago
        Yeah, this is getting way past the point where coincidence seems a viable explaination.
        • SoftTalker 1 day ago
          Randomness is not evenly distributed.
          • nick486 1 day ago
            neither is incompetence, unfortunately.
        • esseph 1 day ago
          I thought you were a data geek?
    • CrzyLngPwd 1 day ago
      What did MS say caused the Office 365 downtime?
      • vinyl7 1 day ago
        There was a thread contention bug in one if their backend services in an update that was pushed out in July and didn't happen until the system was under normal loads across all their services.
        • dclowd9901 1 day ago
          How was the bug introduced?
        • cube00 1 day ago
          And only on newer hardware with more cores that only exists in production
    • SubiculumCode 1 day ago
      To see what's next, let's turn to prediction markets!
  • throwaway_digit 18 hours ago
    This bank hires people based on where they were born, the primary qualification seems to be Greek origin like Jamie dimon. The man is a buffoon, has no understanding of technology, ai, or competence of any sort, just so long as you stay in the office 12 hours a day, that’s what counts. Oh and don’t get me started on the stupid McKenzie cabal, another incestous self feeding bunch of idiots whose entire qualification is going to an Ivy League and working at one of these idiot factories (big 4). In the absence of any competent competition, one bank is sure to rise to the top, it does not reflect the excellence of Jpmc, only how much worse the rest are. How do I know all of this? I am a managing director here and weep daily at the loyalty and familiarity driven culture that drives out competent people every day. By the way, I am only talking about md level hiring but I am sure the rot goes deep.
  • cloudc0de 1 day ago
    Logged into my account at all four of those banks just fine, even submitted payments.
    • ph4rsikal 1 day ago
      Thank you for this information. Then it must be nothing.
  • enahs-sf 1 day ago
    takes me back to the 90's...

    "on a long enough timeline the survival rate for everyone drops to zero"

  • nasalgoat 4 hours ago
    One of my current tasks is automating my company's various bank info and I have to say, the JP Morgan system is the worst one by a wide margin. It's taken me three months to even get to a point where I can use their API, and I'm still blocked because their docs are garbage and nothing is easy to use even with AI help.

    I'm not surprised their systems are having trouble, they're dogshit.

  • feliksgorisek23 17 hours ago
    This hits hard on reputation and credibility of the company. In Slovenia this happens quite often, new production versions are not tested correctly and rollbacks are common.
  • mercurywells 1 day ago
    This appears it could to be related to the compromise of Optimizely, I see tags related to it on Wells Fargo's home page
  • rz2k 1 day ago
    Is Zelle the commonality for this outage? All of those banks work for me, but I have been careful to never enable Zelle on any accounts.
    • john01dav 1 day ago
      Why do you avoid Zelle? I often find it to be useful, so if it has some non obvious issues I prefer to know.
      • kccqzy 1 day ago
        I avoid Zelle because Zelle shows my legal name rather than a nickname to any random stranger I transact with. At least with Venmo you can provide a pseudonym to the other person. The only use case I have involved buying / selling secondhand items on Craigslist and Facebook marketplace.
        • JoshGlazebrook 1 day ago
          I view this as a positive. If someone sells me something that ends up being broken/not as described I would much rather have their full legal name vs. a nickname.

          I also don't see how you would ever end up in this situation if transacting via fb marketplace as you're not even allowed to have accounts that are not you.

          • kccqzy 1 day ago
            Facebook allows nicknames. Zelle does not.

            What are you going to do if some rando sells you something that doesn’t work? Sue them in court? I’d rather not and just eat the loss. It’s not eBay.

      • fc417fc802 1 day ago
        Not OP but isn't zelle yet another commercially motivated third party that will aggregate and mine user data for profit? If nothing else they're an additional point at which data could leak.

        My bank used to let me send checks via online banking (either electronic or mailed out physically by the bank depending on the recipient) until they adopted zelle and discontinued that service. My credit union still offers the check thing for free. That's an easy choice at least for me.

        • piperswe 1 day ago
          It's only sort-of a third party; Zelle is owned by the banks

          > Zelle (/zɛl/) is an American digital payments network run by a private financial services company owned by Bank of America, Truist, Capital One, JPMorgan Chase, PNC Bank, U.S. Bank, and Wells Fargo.

          https://en.wikipedia.org/wiki/Zelle

      • ceejayoz 1 day ago
        It's a bit like Bitcoin, in that money can disappear from your account permanently in seconds with little recourse.
        • sam1714 1 day ago
          Not exactly. Zelle allows chargebacks against unauthorized origination, i.e. your account gets hacked.

          Banks only allow a couple $k of transfers per day anyway. The company behind Zelle, Early Warning, is a bank-owned centralized fraud detection network as the name suggests.

          The ability to send wires from your account is riskier... I wired the down payment on my house in one go.

          • ceejayoz 1 day ago
            > Not exactly. Zelle allows chargebacks against unauthorized origination, i.e. your account gets hacked.

            And your bank will always say it was authorized.

            https://www.nytimes.com/2022/10/03/business/zelle-fraud-warr...

            "Banks draw a distinction between what they consider scams — when a customer is tricked into sending money to a thief — and fraud, when an unauthorized person gains access to a consumer’s account and siphons off cash."

            If you're drunk at a bar and someone unlocks your phone with your face and sends themselves your money, that's gonna be authorized from their standpoint.

            • sam1714 1 day ago
              There's no consumer protection for any payment method that's not a credit/debit card or PayPal, Cash App, etc. which chain off credit/debit You pay for that in transaction fees; up to couple of percent.

              Cash, check, wires, ACH, money orders are no different from Zelle in that respect.

              • eli 1 day ago
                Paypal Purchase Protection covers eligible purchases. Otherwise, yeah, it's a great reason to use a credit card and then pay off your balance.
              • monkpit 1 day ago
                Every bank has a big warning before you use Zelle that says basically “you’re not getting this money back, are you sure?”
              • ceejayoz 1 day ago
                Yes. And that's why I carry a tiny bit of cash, a few credit cards, and none of the rest in my day-to-day life.
          • chneu 1 day ago
            You will, quite literally, never get a US bank to refund a Zelle transaction.

            They pretty explicitly say this.

        • lurking_swe 1 day ago
          not quite? it's more like a bank wire. except bank wires can _sometimes_ be reversed.
        • stogot 1 day ago
          So can cash if you pay someone and they walk away?
          • ceejayoz 1 day ago
            Yes, and most people don't carry their entire savings account worth of cash on them for that very reason.
            • Liuser 1 day ago
              There are strict limits on Zelle compared to wire transfers, or in this case your insistence on relating this to bitcoin.

              You can’t drain your entire savings through Zelle in a single day (unless your entire savings is $3k).

              • ceejayoz 1 day ago
                That's a life-ruining amount to a whole bunch of people living paycheck to paycheck.
                • Liuser 1 day ago
                  You’re goal post moving. If you don’t want to use Zelle, fine. But don’t spread false info that it’s prone to drain your entire account relating the risks to crypto transfers.
            • lurking_swe 1 day ago
              And most people don't store their life savings in a single checking account?

              not sure what you're trying to say. I can be stupid with cash AND in my digital life, the choice is mine.

  • enjoyyourlife 1 day ago
    Have been getting some core.finicity.shared.noAccountAvailable.instruction errors with Capital One when trying to connect back accounts
  • vegnus 1 day ago
    Im having GCP problems. Anyone else?
  • vondur 1 day ago
    Just checked with my Chase account. It's fine. Was able to login via the mobile and web app.
  • bijowo1676 1 day ago
    companies relying on a single cloud (Azure) for mission critical workloads is a big problem
  • mococa 1 day ago
    America is melting
    • viklove 1 day ago
      The businessmen and entrepreneurs have full control of the country. We're cooked.
      • vnchr 1 day ago
        Better empower the bartenders and career politicians. They'll surely save us.
        • wildrhythms 1 day ago
          I'd rather have a bartender in charge than a businessman.
        • djeastm 1 day ago
          When you're in a hole first stop digging.
        • dudefeliciano 1 day ago
          Society was unironically functioning better before the “draining of the swamp”
    • DaSHacka 1 day ago
      To be expected from the 'melting pot', I suppose
    • testtestmetest 1 day ago
      I kinda expected things like this to happen the week of the Anthropic IP0. EA sees everything as a trolley problem anf tries to create them wherever they can. Thwy will mess with critical infra to get themselves the regulation they want.
      • reasonableklout 1 day ago
        How do you think this is being caused by EA?
        • testtestmetest 1 day ago
          Its a pet conspiracy, that I know has no evidence, but EA would love to get the public in agreement with them on AI so that the public consents to them being the arbiters of AI regulation.
          • reasonableklout 3 hours ago
            I feel you are suffering from the same logical fallacy where when you dislike someone anything they do makes you dislike them more.

            What if mass cyber attacks that EAs are worried about actually start to happen (not saying that this is one)? Should we decline to regulate simply because we do not trust the motivations of EAs?

  • epsilonic 1 day ago
    “Life can change in a blink.”

    - Nathaniel Rothschild Nov 5, 2026

    • david927 1 day ago
      Your daily reminder that

      - Your bank balance is not how much you have, but how much the bank owes you.

      - Your portfolio is not how much stock you have, but how much the broker owes you.

      • qwerpy 1 day ago
        People owing you is a good thing, if they pay you interest/dividends for it. Or if you like pithy quotes:

        - Your lack of bank balance and portfolio is not how secure your money is, but how much work you owe the world to pay for food and shelter.

      • asdfman123 1 day ago
        Your portfolio value is how much the current market is willing to pay for your portfolio.
      • IAmGraydon 1 day ago
        Here’s your daily reminder that low effort, reductionist half-truths are better received on Reddit.
  • fragmede 1 day ago
    Went into a branch and they couldn't help me!
  • computer23 1 day ago
    Did anyone have Spamhaus put them on their spam list within the last week for not putting the proper HELO?

    Ours was misconfigured as ourdomain.com instead of mailserver.ourdomain.com

    Here's their announcement: https://www.spamhaus.org/resource-hub/deliverability/spamhau...

    • Sebguer 1 day ago
      what does this have to do with a banking outage?
  • e40 1 day ago
    Hmm, Apple pay didn't work for me this morning, but Paypal (with a CC) did.
  • swozey 1 day ago
    There's going to be a big swing to get off the cloud methinks, hopefully it benefits my career
    • phatfish 1 day ago
      I reckon, it would take 3 days of full outage for an average company to even consider the work needed to migrate from cloud providers or services. Obviously with some exceptions around risk to life systems, which should have an independent disaster recovery plan already.

      M365 would have needed to be down for a full work week for job adverts to go out for an internal IT infrastructure team.

    • cube00 1 day ago
      I doubt it, c-suite will never approve the spike capital expenditure for buying hardware now they're under a nice smooth subscription curve which doesn't harm their profits as dramatically.
    • chrismustcode 1 day ago
      Hows the cloud responsible for the outages in a way that is mitigated by self hosting in recent outages and this one?
      • cube00 1 day ago
        Not needing to run at big tech scale makes self hosting easier.

        You also have full control of downtime and recovery rather the refreshing the MSFT status page.

      • ultrarunner 1 day ago
        Presumably a central provider takes down your dependent services with no local ability to navigate around the outage. Removing the dependency obviates the outage risk. It introduces its own downsides as well, but that's a different discussion.
  • anti_fed313 1 day ago
    I have feelings foreign entities gave there hands on this
    • juliusceasar 1 day ago
      I wouldn't be suprised if it's Israel. They are known for their false flags.
  • AstroNutt 1 day ago
    Just used my Chase card at Walmart. Worked fine.
  • irfanhasnani 1 day ago
    In past week i have heard about lot's of outages.
  • jsisto 1 day ago
    I was able to login and check my account balance
  • MostlyStable 1 day ago
    USAA also appears to be down, at least for me.
  • k0morebi 1 day ago
    Aw man just begrudgingly switched to them
    • bob1029 1 day ago
      I'm curious the reason.
      • k0morebi 1 day ago
        Their website isn't bad compared to some others I'm familiar with, they have locations everywhere, WF is banned from my current area (my previous bank), local and regional banks had terrible websites and security options, and they gave me $1k to open the account. Ultimately it's about convenience.
    • fcarraldo 1 day ago
      To who, banks?
      • k0morebi 19 hours ago
        This made me chuckle. Was talking about chase
      • sph 1 day ago
        They probably got tired of carrying coffers full of gold coins everywhere. Bad timing.
  • starkeeper 1 day ago
    We are under attack and the enemy is incompetence or actual terrorists.

    They would/will never admit it is AI breaking everything either.

  • nullbio 21 hours ago
    It's a well coordinated psy-op.
  • Kapura 1 day ago
    Yes, it's sad these systems are going down all the time, but if we wanted them to stay up we'd have to pay the people who maintain them, which means less money for the stockholders. So you see, our hands are tied.
  • matt123456789 1 day ago
    I asked Claude to log in to everyone's accounts and check on them. Don't worry, it had no trouble getting in and it says everything is fine.
  • p1necone 1 day ago
    Whole lotta people in this thread blaming these outages on whatever their personal software industry bugbear is with zero evidence.
  • Chance-Device 1 day ago
    I love how trusting everyone is after three major outages on consecutive business days, hot on the heels of months of warnings and demonstrations of AI enabled attacks, that these aren’t attacks, solely because people incentivised to say that everything is normal are saying everything is normal.
  • shantnutiwari 8 hours ago
    Im sure its a complete coincidence, and nothing to do with hundreds of jobs cut and people replaced with "AI".

    Already people in the comments are blaming "Russians/Iranians".

  • wnevets 1 day ago
    I'm a little surprised I didn't learn about this outage from Flavor Flav [1]

    [1] https://www.mediaite.com/media/weird/flavor-flav-more-than-o...

  • o_-_o 18 hours ago
    At this point, it sounds a lot like "fire sale"[1]

    Planes stop, bank issues, power issues, widespread ai cyberattacks.

    "...who hacks into government and commercial computers across the United States with the goal of starting a "fire sale" cyber attack that would disable key elements of the nation's infrastructure."

    [1] https://en.wikipedia.org/wiki/Live_Free_or_Die_Hard

  • cdrnsf 1 day ago
    All of the project managers running large-scale production systems with Claude Code are having a rough go of it lately.
  • thewillowcat 1 day ago
    So glad that coding has been solved.
  • rdtsc 1 day ago
    They, too, didn't need programmers any longer and let AI take the wheel?

    The thing is, even if true, they'll never admit it. People who make these kinds of mistakes (if this is what it is) are also not very prone to honest bouts of introspection either.

    • drstewart 12 hours ago
      >The thing is, even if true, they'll never admit it.

      Something in me doubts you'll be back to apologize if these issues aren't related to AI

      • rdtsc 6 hours ago
        > Something in me doubts you'll be back to apologize if these issues aren't related to AI

        Well, don't you want to be worried, I can issue an apology! Who should I be apologizing to? I figure once I have clear evidence this is not AI related, I'll shoot Satya an email, I promise.

    • lenerdenator 1 day ago
      Honest bouts of introspection aren't necessary to keep the line going up.

      So, your company shit the bed due to your plans. So what? You went to the right school and have some experience in the c-suite. You probably have enough money to go a decade without cleaning out your savings.

      Break down the company for scrap, file an IPO for the good parts, whatever. You'll make the money back.

      • rdtsc 1 day ago
        Sounds about right
  • foobarthrow22 1 day ago
    Thursday: MS 365 Friday: FAA Monday: banks Wednesday: Anthropic IPO
  • jaybrendansmith 23 hours ago
    Hmm. Sure. All of these outages are not because of AI hacks. Do they think we're stupid??
  • m4rtink 1 day ago
    Outage at SEELE you say ? Does NERV know ?
  • ckrapu 1 day ago
    Wow, I wonder if there is an underlying cause.
    • Spalau 1 day ago
      Its WWIII

      Reuters, March 26, 2025 EU tells the public to hold 72 hours of emergency supplies.

      _

      The Guardian, March 26, 2025 Stockpile 72 hours of supplies in case of disaster or attack, EU tells citizens.

      _

      The Guardian, October 23, 2026 Tinned mackerel, hand-crank radios and bags for poo: crisis-prepping guides around the world.

      It list government preparedness guidance from the United Kingdom, France, Germany, the United States, Finland, Sweden, Norway, Denmark, Australia and India.

  • boringg 1 day ago
    Chase web services aren't down...
  • SilentM68 1 day ago
    If I had any money in the bank, I'd be worried. Fortunately, I'm tapped out now :|
  • deadbabe 23 hours ago
    What pisses me off is people insisting all these recent outages and events are not related. Just “jimmy unplugged the wrong wire!” type IT shit.
  • gib444 1 day ago
    Is there a 'wikipedia' of outages anywhere? Something more like a database.

    Cloud outages, infrastructure outages, transport disruption, significant websites going offline etc.

    • IAmGraydon 1 day ago
      You can pull historical data from DownDetector’s API, but it requires a paid enterprise account. It would be interesting to see it visualized.
  • Madmallard 1 day ago
    AI fearmongeeing post to boost stocks along with comments to corroborate.
  • aleqs 1 day ago
    The degradation in software quality across the board (Spotify, Google Maps, GitHub, YouTube - to name a few) has been very noticeable for me, and I suspect AI has a lot to do with this.

    The thing is - I also think AI can be used to build very high quality software. The problem, I think, is that the typical big tech/VC tech/MBA tech type company (who control most proprietary software at this point) have very little incentive to actually care about quality, and that was quite obvious even before AI.

    • pkulak 1 day ago
      I remember before I started using AI (I was pretty late to the game), I assumed all the bugs and instability were because people were writing software with AI. Now that I've seen how good the code that AI writes can be (CAN be), I'm moving more to the thinking that it's from AI being used to find vulnerabilities.
      • SoftTalker 1 day ago
        I've never had AI one-shot even a 10 line script that I didn't manually revise. To be fair, it usually produces something that works, but after seeing it I'll realize that my prompt was too vague or I left out some edge case. So it's the old problem with software "it does what was asked for but not what was needed."

        So I speculate that a lot of problems with AI code are just that it is accepted uncritically because it builds and doesn't have any syntax errors.

        • skydhash 1 day ago
          There’s a pressure to deliver more and faster from management. So instead of using LLM to reduce the coding time and allocate more time for review, everything is scaled down and there’s even less time for review. And you’re responsible for more as teams are also scaled down.

          Not really a great recipe for good code.

      • antupis 1 day ago
        It’s more that management pushes more output and with AI it’s very easy to achieve but that will create slop.
    • sakopov 1 day ago
      I think majority of companies don't know how to build safely with AI. It's just kind of handed off to engineering and mostly used as one-off chat interaction to get stuff done. Very few are building any kind of harnesses around SDLC that focus on architecture, testing, security and etc. I think we'll get there at some point, but in my experience the smaller tech players are just now on-boarding to use AI in general in any kind of capacity. We're not talking about best practices yet.
    • ozozozd 1 day ago
      Wondering if this has anything to do with the QA function being folded into the dev team in the last 15 years or so.

      I always defended this, but it appears maybe we don’t have enough developers that take pride in the work they ship. LLM coding made this pretty obvious.

      • s__s 1 day ago
        It’s not lack of pride. It’s because they can’t say: “ok development is done, now I need a few days to thoroughly test this”.
      • aleqs 1 day ago
        Engineers don't really decide what gets shipped at a typical tech company, you either ship trash when the marketing/MBA/'strategy' NPC says or they'll find someone else who will.
    • DIE_DIE_DIE 1 day ago
      > I also think AI can be used to build very high quality software.

      I don't, and I think your comment is rhetorically useless because you've been bullied into making this ridiculous concession.

      • aleqs 1 day ago
        bro... what model did you use to write that? or is that how you actually talk? Lol
    • gtowey 1 day ago
      IMHO the only way to build good things is to pay attention to the details. And coding agents, while greatly increasing the appearance of productivity, they are causing people to pay less attention to what they're actually producing. This is the inevitable and obvious result.

      Agentic coding could be a world-changing benefit, but the current way it's being sold and used as as a cost cutting measure -- produce software faster than ever before with fewer software engineers! Get rid of those pesky experts!

    • pwillia7 1 day ago
      We're in the v1 machines screws stage where everything is worse but with a clear path to get better .... I hope
    • lotharcable 1 day ago
      It isn't a software or tool problem.

      It is a C-level executive problem. With the FAA it is the agency directorship problem. Same issue.

      These people, very literally, are trying to run the world with power point presentations and executive summaries drafted by sycophants. 70 years ago these people had actual experience doing actual work and trying to grow businesses and doing stuff at the ground level. Now they are given their positions through a combination of credentials, connections, and political maneuvering. None of which actually relates to how well they can or cannot run a organization.

      When reality disagrees with what they get in their emails and presentations then it reality that is wrong. It is the messengers that are the problem, not their competency.

      This is nothing new. But the failures in leadership compound over time, like interest to a unpaid debt. They start off with healthy growing companies and thanks to government protections and bailouts it takes decades to for these organizations to decay into total dysfunction. In the past these sorts of people would be removed and their assets reallocated to competent management through bankruptcy and competition. However that natural process is not allowed anymore.

      All AI is doing is handing them more foot-guns.

      They think they can do better/faster/stronger/cheaper with AI and if AI causes problems then it can just be fixed by paying for better AI. Which they assume is just right around the corner, always.

      Just wait until they start to realize that it is

      • aleqs 1 day ago
        Very well put, and I think (open) AI will help competent people outcompete these clowns, which is what makes me optimistic. AI is a multiplier, but multiplying bullshit just creates more bullshit. These people are far more replaceable than competent engineers, even more so with AI.
    • matt3210 1 day ago
      broken windows
    • dmitrygr 1 day ago
      Part of being sane is accepting that some tools, while being very powerful, are too easily misused to be widely distributed to those lacking proper training. This is why you cannot just go and buy 5 kilos of azithromycin
      • miningape 21 hours ago
        You're getting downvoted but honestly I think you've got a point - the average person that gets a hold of plutonium could cause serious issues through nothing but sheer ignorance.

        Maybe the problems with AI are not as destructive, but this is one of the few fair comparisons/analogies I've seen applied to AI.

      • aleqs 1 day ago
        Ehh I don't really buy that. Who decides what tools are distributed to who and how?

        Don't want to get into a political debate here but I'll say that I think that open AI models and access to them is a benefit to society, while closed, proprietary models, controlled by tech billionaires are probably net bad for society.

        • gilrain 1 day ago
          > Who decides what tools are distributed to who and how?

          The same people we already delegate such responsibilities to: the government; ideally, a representative one.

          • aleqs 1 day ago
            > the government

            Uhhhh... you mean the corrupt, baby-eating pedophiles who are owned by tech billionaires right?

            Cool plan bro

            • gilrain 9 hours ago
              If you have a better idea than civilisation, we'd all love to hear it. Giving up isn't a plan.
      • Ylpertnodi 1 day ago
        > some tools, while being very powerful, are too easily misused to be widely distributed to those lacking proper training.

        Could you define 'proper training'? I've endured plenty of fucked-up shit from so-called 'professionals/ experts/ trained people' over the last 30 years or so.

        I'm currently assisting a couple of vibe-coders improve their 'too easily misused...[and] widely distributed' (available on their sites, or github) tools - I mean slop - that benefit me in ways traditional companies have no interest in. And the updates are fast (daily, mostly) and accurate because, unlike jeremy with a pretty CS degree and a nice job, these people actually give a crap' and don't have a manager to say 'you can't do that'. Also part of being sane is accepting that the cat is out of the bag and big $oftware/ tool manufacturer better get a move on. Disclosure: my interests are not critical to anything; quite the opposite, but people are learning faster and faster with Ai learning faster and faster. Who needs azithromycin if a series 1's and 0's can accomplish so much more? For good or bad.

        • dmitrygr 1 day ago
          No, i cannot. If i could, i would be proposing concrete ideas. But I also understand why I cannot buy 3 kilos of plutonium or 5 kilos of antibiotics, and understand the modern ML models can do just as much damage while being more available.
  • c23vv232 7 hours ago
    It's coming! Fuck you all :)
  • ButlerianJihad 16 hours ago
    «“…it is not cyber-related,” said Joe Evangelisti…»

    Language is crazy and should not be using jargon to communicate with the public through the credulous mainstream media.

    I can recall at least 4 very popular terms beginning with “cyber-” of course we know that an on-prem bug in software is neither cyberpunk nor cybersex[citation needed], but it definitely happened in “Cyberspace”.

    “Crypto-” being another crazy prefix that became a term in its own right, which is probably the most annoying thing about cryptocurrencies, that the bros appropriated, well, both components.

    Anyway enough fulminating about trifles, but I am wondering if my app’s announced “login issues from Android mobile app” were Chase’s fault too? Ripple effects like us-east-1 going down, folks: it’s the new normal.

  • syngrog66 1 day ago
    The US has no natsec currently due to its leadership. There has been an intentional purging of those with real expertise at the topmost layers. Mass layoffs of experienced folks. AI use rise and all that entails. Several billionaires with motive and resources to do sabotage or cause chaos. Now add the US attacks on Iran giving huge motives for the latter to push whatever buttons they can. One could argue we should be surprised 2026 hasn't been much worse than it has been. And there are reasons to expect it to worsen much more before it can ever get better.
    • tedd4u 1 day ago
      [flagged]
    • mitchweaver 1 day ago
      > There has been an intentional purging of those with real expertise at the topmost layers.

      People love to say this but can never point to who these experts are or what their expertise was.

      • syngrog66 1 day ago
        what I refer to has been documented repeatedly and redundantly across US news orgs for over a year and a half into the current admin. Its no secret and not up for debate.
        • drstewart 12 hours ago
          Saying the US has no "natsec" is clearly and objectively false and not up for debate
        • mitchweaver 1 day ago
          Thanks for confirming you can't point to any specifics beyond vague partisan accusations, but that it's "not up for debate".

          Hello, Congress? Send up 8 more experts!

          • syngrog66 1 day ago
            your 1st sentence there is rude and false. rude alone is the showstopper
  • 50208 1 day ago
    Queue the "It's Biden's fault" excuse in 3,2,1 ...
  • ceejayoz 1 day ago
    "according to the website downdetector.com"

    Ugh. DownDetector's methodology is… not great.

    Steam isn't down: https://downdetector.com/status/steam/

    Amex isn't down: https://downdetector.com/status/americanexpress/

    Visa isn't down: https://downdetector.com/status/visa/

    AWS isn't down: https://downdetector.com/status/aws-amazon-web-services/

    But each has a huge corresponding spike there.

    • Miraste 1 day ago
      DownDetector's methodology is fine, it's just noisy. The way to use it is to look at the top-reported failing sites in the ranked list on their homepage, which has been the correct root cause for every major outage I've seen. Ignore the lower ranked ones.
      • ceejayoz 1 day ago
        Google's in the top 20 (#11) right now. Down?

        "User reports show problems with Google"

        https://downdetector.com/status/google/

        • Miraste 1 day ago
          By top-reported I mean the top two, maybe up to four. It's not high-fidelity. I still think it's useful, especially since in 2026 the art of building a status page not dependent on the infrastructure it monitors has been lost.
          • hinkley 1 day ago
            So if there ever is a widespread outage, how will you know? I mean besides "AWS us-east-1 is down" -> "everybody is down".
          • alala-lala-lon 1 day ago
            [dead]
          • ceejayoz 1 day ago
            > By top-reported I mean the top two, maybe up to four.

            As the Rick & Morty quote goes, "that just sounds like horrible methodology with extra steps".

            • Miraste 1 day ago
              There's an outage > you go to downdetector > you see the cause of the outage.

              Seems straightforward. They could add some qualifiers about the lower ranks somewhere, but that's the base value proposition and it works fine.

              • ceejayoz 1 day ago
                There's not an outage > you go to downdetector > it shows an outage > you write a news article, apparently.

                > you see the cause of the outage

                What does DownDetector say the cause of today's apparent Amazon and Google outages are?

                • Miraste 1 day ago
                  This kind of rigid thinking is such an HN thing. The "cause" is obviously problems with payments from the bank outages, which, again, is clear from looking at the ranked list.

                  You can't source details that depend on technical knowledge from a non-technical audience, regardless of methodology. Downdetector is useful nonetheless.

                  • ceejayoz 1 day ago
                    > The "cause" is obviously problems with payments from the bank outages, which, again, is clear from looking at the ranked list.

                    You told me only "the top two, maybe up to four" slots count. Five are named in the headline. Only Chase has confirmed any kind of issue, hours later.

                    (In fact, the article has now been updated to say that Bank of America doesn't have an issue, and suggests it's a Chase-specific outage.)

                  • flkiwi 1 day ago
                    But downdetector doesn’t track that financial transactions are failing. It tracks whether people are checking to see if a service is failing. It is only a charitable reading that people are only checking if they are experiencing a failure. A signal, user queries, is presented as another signal, service status, even though there is no correlation.
                    • transcriptase 1 day ago
                      In much the same way, every time there’s a large conservative or republican gathering like a national convention in the USA, someone starts spreading fake headlines that “grindr” is down for that specific city. People go to down detector en masse which makes it display that grindr is down (it must be since so many people are checking), lending credit to the hoax. It was never down, and no app has city specific infra that would go down with a spike of a few hundred or even thousand people assuming the premise of the hoax was true.
                      • jacobolus 1 day ago
                        From what I can gather from a quick search, Grindr hasn't gone down in response to GOP conventions, but there is commonly lots of traffic during places/times where GOP conventions happen, and it's plausible that if many people are trying to access a service from the same public wifi they could experience connectivity issues without anything being wrong on the service end.

                        Grindr CEO: “Yes, it was actually true that there was a significant spike in usage in Milwaukee during the [2024 GOP] convention.”

                  • coldtea 1 day ago
                    >This kind of rigid thinking is such an HN thing. The "cause" is obviously problems with payments from the bank outages, which, again, is clear from looking at the ranked list.

                    Not to offend you with all this rigid accuracy, but what you describe is not a cause but an effect.

                    • Miraste 1 day ago
                      Right, which is the reason I put "cause" in scare quotes...
            • monkpit 1 day ago
              Make a better one and we can use it instead!
              • coldtea 1 day ago
                We can start by not using a bad one.

                False positives and false negatives can be worse than no reporting at all.

                • monkpit 1 day ago
                  Sure, while we are at it I’ll stop using my insurance because I’d prefer single payer, and stop using my car because I’d prefer a functional public transit system.

                  I’ll also stop eating meals cooked for me if I’d rather have something else.

                  • coldtea 5 hours ago
                    Or you can stop over-extending an argument about a specific thing to everything, strawman style...
      • KennyBlanken 1 day ago
        [flagged]
    • creatonez 1 day ago
      How do you know some of these aren't experiencing a partial outage? Responding instantly to partial outages that never get reported on status pages seems to be one of DownDetector's biggest strength.
      • ceejayoz 1 day ago
        > How do you know some of these aren't experiencing a partial outage?

        Is there any other sign they are? That Google, AWS, Steam, Amex, Visa, etc. are all simultaneously having significant outages right now?

        > Responding instantly to partial outages that never get reported on status pages seems to be one of DownDetector's biggest strength.

        I've predicted all fifty of the last one recessions.

        • hinkley 1 day ago
          I live close (as the fiber flies) to one of the more popular AWS datacenters so I have no doubt that there could be situations where I can access sites that others can't, or in the case of a proper AWS cockup, where I can't but many of you can.

          And when it's not DNS (it's always DNS), it kind of depends on how load redistribution works. At some point I have to switch to load shedding, and so Idaho and Montana might just be shit out of luck.

          If the NYC office is still able to do work, I don't want the Denver and Chicago office to overload their servers and now nobody can get any work done.

        • shiandow 1 day ago
          Payment not working could be considered a partial outage.
        • creatonez 1 day ago
          > I've predicted all fifty of the last one recessions.

          Meh. I've come to the realization that the answer to the question "is X service up or down right now" is as fuzzy as forecasting the weather or predicting recessions. With giant distributed systems it's always half-down in some unique way. Especially so for anything with a dependency on Microsoft services.

    • paxy 1 day ago
      Also happened last week during the Outlook outage. It showed the entire internet down for some reason.
      • hinkley 1 day ago
        I always think of the time the company that acquired us put our Wiki and (some of) production storage on the same drive array and when it failed we couldn't access our runbooks. Lots of anger lines coming off of people who are permanently grumpy the rest of the time. Like people too angry to trust themselves to talk, so one word answers.

        After that our operations team got most of its autonomy back.

        How do you make an app that watches everyone else's infrastructure but can't be taken out by a common cause?

    • doawoo 1 day ago
      i see this comment on every single "its down" post, it's not a useful comment- it is down, Chase is partially working.
      • ceejayoz 1 day ago
        The article cites a Chase rep saying they are impacted by a bug of some kind.

        The article writer then went to DownDetector and wrote a headline "Outages reported at Bank of America, Chase, U.S. Bank, Wells Fargo and Zelle" based on just that. As far as I can tell, there's zero genuine evidence for that.

        • mylesp 1 day ago
          Is it not true that Outages have been reported - via DownDetector?

          DownDetector is a way for customers to report outages so the headline is not lying at all. It is not claiming that the root cause of these outages are these banks, or that the banks themselves have reported the outages.

          • ceejayoz 1 day ago
            > Is it not true that Outages have been reported - via DownDetector?

            Yes. That is basically always the case. Bring it up any day, at any time; DownDetector is always reporting spurious outages.

            This story morphed into "AI took down a bunch of banks" already because of an irresponsible headline. "Bank outages" is the #5 trending on my Bluesky (https://bsky.app/profile/did:plc:qrz3lhbyuxbeilrc6nekdqme/fe...) right now, still.

    • ianm218 1 day ago
      I’m experiencing degraded AWS ECS service today so it’s atleast a bit helpful. AWS is probably never going to be down accross all regions and all services 100%.
    • nimrodyunana562 1 day ago
    • Mogzol 1 day ago
      Isn't it just based on user reports? What explains the spike in user reported issues in those services recently?
      • ceejayoz 1 day ago
        > Isn't it just based on user reports?

        DownDetector considers someone looking up "is Visa down?" a report, even if it isn't.

        Click my links and it contributes to those "reports".

        One provider has a major issue, and people go there and click into non-down companies, which is seen as a spike.

        • dubcanada 1 day ago
          Is there a problem with that? Do normal people google "is Visa down?" if they don't think Visa is down?

          On the other side, does a user google "is Visa down?" going to a website and clicking "yes I think visa is down" any better?

          • ceejayoz 1 day ago
            > Do normal people google "is Visa down?"

            Normal people Google "is Chase down?", find DownDetector, and click it. (Note that the article has a Chase rep quoted saying they do indeed have an outage, and none from other companies.)

            Some of them then click into the DownDetector home page and browse around. Suddenly everything is spiking.

            Every time I've seen any one major company with issues, DownDetector has spiked across the board. As it is now. Google and Amazon are apparently down.

        • akersten 1 day ago
          > DownDetector considers someone looking up "is Visa down?" a report, even if it isn't. Click my links and it contributes to those "reports".

          Really? It seems to me you have to click a specific aspect of the service to report as down once you land on that page to report an outage.

          • danso 1 day ago
            I assume their methodology weights active reports higher than the implicit ones.
          • ceejayoz 1 day ago
            https://downdetector.com/methodology/

            > We process tens of millions of problem reports each month and cover over 36,000 services in 72 countries.

            Ask a few normies if they know what it is, let alone have filed a report.

            https://downdetector.com/about-us/

            > Downdetector collects reports from users and monitors other online sources for service issues.

            Or look at the comments on any of the pages. "My card declined"; "18yr femboy down for DM".

    • baxtr 1 day ago
      Has downdetector ever been down?!
    • stackskipton 1 day ago
      I mean, if Visa/Amex transactions are not working because their bank is down, end users don't know the difference. Card says "Visa" on it, card is not working, end user blames Visa.

      Steam/AWS, I got nothing.

    • gamblor956 1 day ago
      I can't comment on the others, but Steam and Amex were both partially down earlier this morning.
  • elvisgodevelop 13 hours ago
    [flagged]
  • SouraVamseekarM 11 hours ago
    [flagged]
  • anigbrowl 1 day ago
    [dead]
  • ahmedketata 1 day ago
    [dead]
  • remusrm 1 day ago
    [dead]
  • rumburak 1 day ago
    [dead]
  • gibsonf1 1 day ago
    Its so much easier to generate code with genAI though.
    • dccoolgai 1 day ago
      It's so fast. You can also put "make no mistakes" in the prompt.
    • sedan_baklazhan 1 day ago
      And we can fire so many programmers now
  • epsilonic 1 day ago
    “Life can change in a blink”

    - Nathaniel, Rothschild Nov 5, 2026

  • vinyl7 1 day ago
    Software is solved!
    • enjoyyourlife 1 day ago
      Software is solved but not software engineering
  • rvz 1 day ago
    Crypto blockchains such as Solana, Tempo and XRP are still up and running while the banks are down and you have no choice but to wait for them.

    What a disaster.

    • Aboutplants 1 day ago
      Great, now where can I buy groceries with these garbage coins!
      • rvz 1 day ago
        Doordash accepts stablecoins such as USDC so you can buy your groceries from there. [0] [1]

        No excuses.

        [0] https://cointelegraph.com/news/doordash-stablecoin-payments-...

        [1] https://tempo.xyz/customer-stories/doordash/

        • ceejayoz 1 day ago
          "Plans to".

          By this metric, I can hop on a supersonic flight on Boom.

          • rvz 1 day ago
            That is just one large example, and Shopify supports stablecoin payments right now and grocery stores built on them can accept stablecoins at in their point of sale systems.

            But with that attitude, no progress will ever be made and you will continue to professionally complain on HN.

            But thankfully, companies using the Tempo blockchain such as Stripe (who built it), and others do not care and already have business interest with the use-case of payments.

  • DIE_DIE_DIE 1 day ago
    [flagged]
    • toraora 1 day ago
      incredibly misinformed take - llms are undoubtedly performing real work. it's true that the rate of production of bad quality code has skyrocketed, but there are certainly teams that are increasingly productive and systems that are increasingly more secure because of them
      • mistercheph 1 day ago
        For example...
      • DIE_DIE_DIE 1 day ago
        > but there are certainly teams that are increasingly productive and systems that are increasingly more secure because of them

        I don't take your assertion of certainty seriously. In my opinion, you are not qualified to evaluate the quality or security of code. You are looking at a series of related disasters and shouting that nothing is wrong. Please forgive me for not taking your perspective seriously.

    • jjice 1 day ago
      It's a tired discussion. LLM generated code can be awful, but it can also be very good. If you think all LLM generated code is bad, you haven't used any in the last year.

      Company process and how LMS are treated in the development process is actually of interest. Are they YOLO shipping code or are they reviewing it and using LLMs as a tool, instead of a replacement for an engineer? That is where LLMs bring danger.

      • bigstrat2003 1 day ago
        > If you think all LLM generated code is bad, you haven't used any in the last year.

        I use Opus 5.5 and it can't generate code anywhere near as good as mine. I'm not an exceptional programmer either, I'm fairly middle of the road. LLMs remain bad at programming, despite the claims of AI bros that only within the last 6 months have models become good (which they curiously have claimed constantly for years).

    • LPisGood 1 day ago
      I think they’re pretty clearly not designed by idiots. A lot of smart people work there.