I see multiple posts here speculating on cyberattack—as opposed to "we pushed a bad configuration update which messed everything up irreparably"—you know, like it has been every other time before this.
E.g., Cloudflare, Meta (who in doing so also locked themselves out of the building), and didn't some bumbling major Canadian telco knock themselves offline for like a week not too long ago?
It works both ways, a lot of people also take the "nothing ever happens" position and it is true that most of the time "nothing ever happens", so by taking that position, they're right 99% of the time and sound smart
Yes, but the majority opinion is "nothing ever happens" by default for everything, not just tech outages. It's not about sounding smart, but getting ahead of grifters and clout chasers.
Your post reflects another online observation. With the rise of online sports books, this sort of predictive doomerism has flooded almost every team's online comment section. It no longer feel like fandom or community in the same way. Just lots of voices that will be glad to say, "I told you so," in the loss and crickets with the W. Wish there was some accountability mechanism for all the negative noise broadcasted into the channel.
The great thing is that with all of the encrypted/signed DNS thingys, you now have not one, but multiple options to combine “invalid certificate” and DNS in one outage. You no longer need to choose!
Yeah, the Canadian telco was Rogers. Total recovery took multiple days. From the Wikipedia writeup:
> In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network.
I took my kids swimming that day and the pool couldn't take our money since the payment terminal was on the Rogers cellular network, so it was a free family swim.
Cyberattacks are a good scapegoat for any large incompetent non-tech company that is unable to admit a mistake. (tech companies are more open to admitting actual mistakes - and reluctant to disclose cyberattacks even if there actually was one - where as non-tech ones would rather allude to an attack than admit a mistake)
Cyberattack scenarios pretty much never make sense in case of complete outages; if you have the access required to cause such an outage it’s always more profitable to keep this access and use it for covert spying/targeted attacks or save it for later than to burn it by causing a massive, visible problem.
Verizon had issues routing calls to a provider I'm aware of yesterday, and had to make some sort of change today to fix it. I'm definitely thinking bad configuration update.
20 years ago much less of the infrastructure of everyday life depended on an always-on network connection. Smartphones in particular were a relatively niche product. I didn’t even have a cell phone (and not because I was too young), much less expect it to work all the time.
No, it's affecting Verizon and people on ATT & TMO trying to get in touch with the Verizon customers who are affected.
Note the tiny fraction of people reporting for ATT & TMO compared to Verizon on DownDetector: https://downdetector.com/
(You have to click the links to see actual magnitude because the graphs are scaled to show relative outage within a given service -- order of 150k for Verizon vs 1.5k for the others.)
I got sniped away from Visible (Verizon MVNO) by US Mobile (multi-carrier MVNO) during a Black Friday sale. USM has an interesting thing where you can actually get a separate eSIM for each of the major carriers, and switch between them. I was curious so I signed up for all 3. It's been interesting to see how the signals vary from location to location, and at least a couple times I've been able to get significantly better signal by switching.
The main downside is that you have different numbers for each eSIM, but that doesn't really affect me because I use Google Voice for SMS.
I switched to US Mobile a long time ago just due to pricing. I was WFH most days of the week (before Covid) so I could do minimal data and pay around $100/yr for unlimited talk/text. Now, I do unlimited everything as I commute 5 days each week, but it's only $200/yr. Still significant savings.
Unfortunately GV has been having a ~week outage of outgoing MMS group messages. Well, let's say a brownout – many messages make it through to some recipients.
I really love US Mobile. I was a flagship postpaid $90/mo VZW customer for a decade and was so hesitant to switch. It's been 4 years now, and all I can say is that I can't believe I waited so long.
For those who weren't aware: Verizon got a new CEO late last year and laid off 15% of the workforce (15,000 people). This included people working in network, IT and cyber security.
15% seems like the magic number. It seems like many corporations layoff approximately that fraction of their workforce; it's hard to believe it's coincidence.
Like all the doom and gloom after the Twitter layoffs predicting the site would implode and go permanently offline "within a month" which...never happened.
It's also ironic in the sense it implies the indignant people were so bad at their jobs they designed and built a system so fragile it would collapse without constant intervention from thousands of individuals.
You do realize it's possible for an organization to be overstaffed?
Verizon is a traditional for-profit telco. Not some VC funded startup trying to hit a burn rate. Very unlikely they were overstaffed by 15k, sounds more like overzealous cost-cutting to hit a quarterly target.
This is unrealistic and seems to be biased by some kind of broad un-focused hostility. Yes, maybe they were overstaffed. But it's reasonable to suspect that leadership overcut, given the current climate and the number being 15,000. Your characterization of Twitter predictions relies on cherry-picking and ignores the actual impacts, and there's no evidence that the system goes down without "constant" intervention from "thousands". Your tone also implies that large, complex systems, even if designed well, don't normally require a lot of maintenance from many people.
>Your tone also implies that large, complex systems, even if designed well, don't normally require a lot of maintenance from many people.
That's correct.
In the case of Twitter, it was disclosed that many of their systems were running out of date EOL software, to the point of being a security liability, which raises the question: if the systems weren't being maintained, wtf were all those people doing? Taste-testing the free food and cappuccinos?
> many of their systems were running out of date EOL software, to the point of being a security liability
This is more likely a management problem rather than a staffing problem. Lower level management knows about these kind of things but often they are not incentivized to make them a priority due to a culture focused on growth and “winning”.
If you’ve got 100k people to run something that should run essentially on autopilot, you’ve got much deeper problems where merely laying off selected chunks of people will no longer help. The whole company is rotten and the only way is to start from scratch and not make the mistakes that led you to accumulate 100k people.
I can't even begin to imagine what those 100k people actually do. For starters, none of the telcos actually develop their own equipment - they buy pre-made from vendors like Ericsson. Often that includes ongoing maintenance too. The only "engineering" is building the back-office and customer-facing UIs, and even that is often outsourced (as a rule of thumb, if something can be outsourced, telcos will do it: https://berthub.eu/articles/posts/5g-elephant-in-the-room/).
Customer service might be part of that number (assuming that too isn't outsourced), but even then 100k feels extreme.
10k is ok although leaning on the more bloated side. But 100k?
It's well known that big tech companies are overstaffed. You probably can't build Dropbox in a weekend starting from scratch, but a smaller scale cloud-based storage solution can be deployed very quickly if you start from existing open source components. And a small team of experienced web devs can certainly build a cloud storage thing in a matter of weeks from scratch too.
Almost; you need maintenance and monitoring but that doesn't take anywhere near 100k people - assuming they even use their own headcount for this instead of just outsourcing maintenance to their equipment vendor.
Big Tech companies operate much more complex systems (for starters, they actually build greenfield stuff instead of buying ready-made equipment from a vendor and plugging it in) and have way less headcount.
You're building new cell towers, managing countless failed backhual links (thanks to fiber's natural enemy, the backhoe), working with whatever obscure bugs your MVNOs have managed to uncover, certifying new cell phone designs, and still working on upgrading everything to 5G while simultaneously planning for 6G (keeping in mind that the 5G network architecture looks radically different than the LTE architecture). Much of that work is necessarily physically distributed across the entire country.
Not to mention dealing with end-user sales and support, which unfortunately often needs physical stores.
I'm not going to say whether 100k is too many, but there's a lot more involved here than just maintenance and monitoring - especially if you want your network capacity to keep up with growing demand.
ChatGPT can very well be an upgrade compared to the "engineering" capability of a lot of telcos (they have very little, are hell-bent on outsourcing as much as possible and are even proud of that). But don't take it from me, here's a more reliable source: https://berthub.eu/articles/posts/5g-elephant-in-the-room/
Obviously, you know exactly how to run a major telecomm operation ten times as efficiently as the dominant operator in the most prosperous nation on earth — you are wasting your skills and should absolutely be given funding to disrupt them and make Billions!! What is holding you back from joining the Oligarchs?
Have you considered that the inefficiency is a feature and provides cushy jobs for a lot of people and subcontractors? But yes, a modern telco can absolutely be run more efficiently if you operate it like a tech company and don't have to deal with decades of legacy sludge (whether bloated headcounts or heterogenous legacy infrastructure you have to support).
The problem is that this is a culture problem and once a company is ossified it is really hard to enact such change from the inside even if you wanted to because everyone enjoys the status quo (and who doesn't wouldn't be there to begin with).
Another example: have you seen the UK & EU banking scene and the boom of fintech and "neobanks" around 2017 like Revolut, Monzo, Starling, N26, etc? They managed to build from scratch on relatively shoestring budgets their own implementation of a consumer bank, something that their legacy competitors still can't replicate despite having way more budget and resources.
Unfortunately, the telco world is an oligopoly and they don't like new entrants (banking in the UK was actually a much more level playing field in comparison), so we can never actually see an experiment that proves or disproves my theory.
The problem with being a nationwide ISP - and Verizon runs mobile phones, fiber and all kinds of other stuff - is that you need lots of hands across the country. A lot of stuff can be done remotely and with automation, but often enough you still need actual physical hands on site, and you can't just say "eh, we'll come around tomorrow, we can't make it there faster".
Also 'Verizon' is not really one company. It is 2. Telco and wireless. Each of those is a mashup of dozens of other phone companies VZ gobbled up over the years. With tech stacks going back decades. At one point while I worked there about 10 years ago they were running the 56k dialup for AOL. They also run a decent amount of stores. They are not going to automate a retail store in the same way you would amazon. You have to have people standing there. Then there is the "i need to talk to someone about why my phone keeps doing weird things" helpdesks/servicedesks. Then the line workers like you point out. Plus the backend people who might be able to work from home. But only if they are not in a secure area working (they have lots of that). That 15k of people was probably the result of several big projects that were scaling up but didnt work out. They have all sorts of projects to try to 'monetize the last mile they own'. Almost all fail.
Are amazon stores automated? I have Amazon fresh store next to me, they have the smart grocery carts that no one uses because they are overly finicky. They have same number of employees as other grocery stores.
Recently downloaded bitchat so I can contact my family members if outages like this happen when we’re out and about. I got a few T1000 cards for meshtastic but there’s just too much friction to teach my spouse and others how and when to use it. I wish haloW was built into phones which would make long range local communication much better.
Does bitchat work during a phone outage? Range on BLE is pretty low, right, so I'd expect it to not do much unless you happen to live in an area where there are bitchat users every 100 ft or so.
I'm in a similar boat to you in wanting a LoRa mesh. I tried out MeshCore on the LilyGo T-Deck+ hoping it would be a device I could hand out to family members, but I found the hardware and software disappointing.[0] But I'm weirdly tempted to try the LilyGo Pager.
Unfortunately as you said, the range is abysmal since it uses BLE. It’s barely enough for a box store but it’s better than nothing. I like the idea of the lilygo pager too. I first thought it was e-ink from some of the pictures but sadly it was not.
Lol it's network core. Postpaid users on Verizon branded accounts affected. Most MVNO's just fine (checked US Mobile and Visible, different cores... and they are working fine on the Verizon towers)
T-Mobile had this issue 5 years or so ago where the IMS core went down due to fiber issues at one of their major backbone providers (pretty sure it was zayo).
As I've said elsewhere, never underestimate Verizon's incompetence. A couple years ago they shut down vtext service which also happened to host their UAprof's for most Android phones. Without a working link to these descriptors, those devices would download any MMS (pre-RCS picture messages) in compatibility mode at the lowest resolution.
It's also weird that some people here in the office on Verizon work fine, and others are on SOS. No correlation between phone versions or hardware that I could deduce. I also see the same on X: https://x.com/CPTholen/status/2011520566159982758
It came up for around 5-10 minutes at 15:00 EST, but is currently still down.
If they're on different phones, they could be on different bands, different towers, and different paths, one or more of which could be impacted by whatever the underlying problem is. iPhone vs Android would be the most blatant tell that something like this is at fault, but it could also be different configurations from different stores causing them to interact with the cellular network in different ways.
Something like a routing configuration, BGP failure, or underlying network misconfiguration would cause seemingly bizarre results with some phones working and some not with no obvious correlation. Compare Access Point Names under Mobile Settings on android, and whatever the equivalent is on iPhones, and check things like whether 5G allowed and data roaming is enabled.
If it's a cyber attack of some sort, then there's all sorts of different attack vectors that would cause these outcomes.
My wife and I are on the same Verizon family plan. One of us can be down while the other is fine, then 30 minutes later it's the opposite. It's been like that all day.
I find this kind of odd. Yesterday, 2026-01-13, I - who lives in the greater Washington D.C. area - experienced A LOT of Verizon disruption. However, today, my service has been excellent. Maybe I'm from the future and don't know it. Did anybody else in my general area experience outages yesterday?
"Verizon engineering teams are continuing to address today's service interruptions. Our teams remain fully deployed and are focused on the issue. We understand the impact this has on your day and remain committed to resolving this as quickly as possible."
As someone responsible my whole career for uptime and network response, I really feel for the engineers, at the same time hoping my service comes back up soon. SOS
That's very often the case. I bet there are frantic calls being made right now (probably using a competing carrier!) to Ericsson/etc to come to their rescue, which they happily will in exchange for a hefty fee.
i know alot are joking / sarcastic about its a cyber attack- that said, Wouldn't it make more sense that whenever there is a "cyber attack" its more likely it would only affect one provider? ie, each has to have different systems / security postures ect, such that a non-public vuln useful to attack Verizon would likely not be exploitable/exposed at AT&T (or vise versa)?
It's a population map lacking Dallas for sure ... which might mean something real ... but that doesn't match the per-city charts they offer. Who knows.
https://downdetector.com/ shows verizon, tmobile, and att. BUT if you look at the magnitude of the outages for tmobile and att vs verizon it's fractions of a percent. Likely those people with tmobile and att reporting when they have trouble communicating with verizon customers.
(Note, you have to click on the providers to see absolute magnitude -- the graphs are scaled to show relative outage over time within a given provider; order of 150k verizon vs 1.5k others)
Most users don't notice any real differences vs using their ISP DNS, and seeing it up and configuring it is yet another thing to take time or go wrong.
We had edge delivery issues when I didn't use my ISP's DNS, especially from Apple. Not exactly sure of the mechanism, but downloading Xcode would take 2 hours instead of 10 minutes.
That’s really weird that’s the case. DNS simply resolves “google.com” to an IP address (8.8.8.8 or something). Shouldn’t impact anything download related. I’m pretty sure DNS isn’t used for geolocating either
I wanted to correct you but than I stopped myself because I'm not sure if you meant that sarcastically. Because with a /s at the end your post makes sense.
Many reasons not do use the provided DNS. First, you don't want to give the ISP more information on your browsing habits than it can already gather otherwise. Second, in some countries, ISPs censor websites at the request of of the movie and music industries. Those are enough reasons to rely on a neutral DNS provider like Quad9 or your own DNS server.
Do you use your car’s built in navigation function — that you paid for — or do you plug your phone in and use its free Google Maps or Apple Maps to navigate?
I didn't pay for any navigation for my car, so I'm not sure what your point is? ISPs provide DNS. People shouldn't have to fuck with the internet's phone book when they plug their modem in (and they haven't for a very long time). Maybe we can expect more from the people who provide the services society relies on, instead of just saying "why don't you just..." every time someone has a legitimate complaint about something that ought to Just Work.
Yeah, this is notable. If true, every mobile carrier getting hit at the same time says "coordinated incident", from whatever source and for whatever purpose.
Edit to say: my Verizon FioS and cell service are both working fine, no noticeable interruption at any point today.
Second edit to say: never mind, downdetector's home page normalizes report spikes so 1k and 100k both look identical.
I've got a Verizon Network Extender and it appears to be online - the tunnel is up back to VZW's security gateway, but all of my phones are refusing to register to it.
I did manage to roam onto an international network on the boarder near me in New York/Canada, so some bits of the core seem functional for authentication.
When I roam internationally I appear to be on Telus's 3G network (no LTE) for data and voice is falling even further back it looks like.
Parts of the HLR might be unavailable or under a thundering herd as all phones are trying to register at once. The HLR would also be involved if using EAP-SIM/EAP-AKA so that would explain failures on the Wi-Fi access points.
I have two Verizon phones on very different networks and both have not been working well since Tuesday - anyone also having this? I kept restarting my phones, airplane mode on/off etc
The US would never purposefully cut of it's own communications, right? What would be the tells if that would have happened? And what could the average citizen do to be able to communicate regardless? Might be useful to always have that knowledge around, even if it isn't a personal threat to yourself today.
Get into Meshtastic or Meshcore! (So far I prefer meshtastic)
It's a citizen-run mesh network that allows text communication. It's not perfect and not 100% reliable (signal and hops limit delivery guarantees) but it's better than 0 ways to communicate in this kind of event!
E.g., Cloudflare, Meta (who in doing so also locked themselves out of the building), and didn't some bumbling major Canadian telco knock themselves offline for like a week not too long ago?
https://www.wsj.com/politics/national-security/china-cyberat...
The truth is things do occasionally happen and we should be prepared, even if most of the time they don't
Keep in mind who the President is.
If so, successful test.
These network topologies are incredibly complex and edges you think wouldn't exist have ways of suddenly appearing when things go awry.
> In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network.
https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...
I took my kids swimming that day and the pool couldn't take our money since the payment terminal was on the Rogers cellular network, so it was a free family swim.
Cyberattack scenarios pretty much never make sense in case of complete outages; if you have the access required to cause such an outage it’s always more profitable to keep this access and use it for covert spying/targeted attacks or save it for later than to burn it by causing a massive, visible problem.
Note the tiny fraction of people reporting for ATT & TMO compared to Verizon on DownDetector: https://downdetector.com/
(You have to click the links to see actual magnitude because the graphs are scaled to show relative outage within a given service -- order of 150k for Verizon vs 1.5k for the others.)
The main downside is that you have different numbers for each eSIM, but that doesn't really affect me because I use Google Voice for SMS.
https://www.google.com/appsstatus/dashboard/incidents/THtvkD...
I was previously on Google Fi, and MVNO roaming is bloody fantastic - I always had reception.
Like all the doom and gloom after the Twitter layoffs predicting the site would implode and go permanently offline "within a month" which...never happened.
It's also ironic in the sense it implies the indignant people were so bad at their jobs they designed and built a system so fragile it would collapse without constant intervention from thousands of individuals.
You do realize it's possible for an organization to be overstaffed?
That's correct.
In the case of Twitter, it was disclosed that many of their systems were running out of date EOL software, to the point of being a security liability, which raises the question: if the systems weren't being maintained, wtf were all those people doing? Taste-testing the free food and cappuccinos?
This is more likely a management problem rather than a staffing problem. Lower level management knows about these kind of things but often they are not incentivized to make them a priority due to a culture focused on growth and “winning”.
Many think Twitter has imploded, though it's online.
> You do realize it's possible for an organization to be overstaffed?
It's possible to be understaffed or appropriately staffed. Anything is possible!
I can't even begin to imagine what those 100k people actually do. For starters, none of the telcos actually develop their own equipment - they buy pre-made from vendors like Ericsson. Often that includes ongoing maintenance too. The only "engineering" is building the back-office and customer-facing UIs, and even that is often outsourced (as a rule of thumb, if something can be outsourced, telcos will do it: https://berthub.eu/articles/posts/5g-elephant-in-the-room/).
Customer service might be part of that number (assuming that too isn't outsourced), but even then 100k feels extreme.
10k is ok although leaning on the more bloated side. But 100k?
To who? What evidence is there either way?
Is it an often-repeated story online? That sort of information is both well known and unreliable - it's well-known misinformation.
Big Tech companies operate much more complex systems (for starters, they actually build greenfield stuff instead of buying ready-made equipment from a vendor and plugging it in) and have way less headcount.
You're building new cell towers, managing countless failed backhual links (thanks to fiber's natural enemy, the backhoe), working with whatever obscure bugs your MVNOs have managed to uncover, certifying new cell phone designs, and still working on upgrading everything to 5G while simultaneously planning for 6G (keeping in mind that the 5G network architecture looks radically different than the LTE architecture). Much of that work is necessarily physically distributed across the entire country.
Not to mention dealing with end-user sales and support, which unfortunately often needs physical stores.
I'm not going to say whether 100k is too many, but there's a lot more involved here than just maintenance and monitoring - especially if you want your network capacity to keep up with growing demand.
The problem is that this is a culture problem and once a company is ossified it is really hard to enact such change from the inside even if you wanted to because everyone enjoys the status quo (and who doesn't wouldn't be there to begin with).
Another example: have you seen the UK & EU banking scene and the boom of fintech and "neobanks" around 2017 like Revolut, Monzo, Starling, N26, etc? They managed to build from scratch on relatively shoestring budgets their own implementation of a consumer bank, something that their legacy competitors still can't replicate despite having way more budget and resources.
Unfortunately, the telco world is an oligopoly and they don't like new entrants (banking in the UK was actually a much more level playing field in comparison), so we can never actually see an experiment that proves or disproves my theory.
Yes, this explains most of our jobs.
I'm in a similar boat to you in wanting a LoRa mesh. I tried out MeshCore on the LilyGo T-Deck+ hoping it would be a device I could hand out to family members, but I found the hardware and software disappointing.[0] But I'm weirdly tempted to try the LilyGo Pager.
[0] https://mtlynch.io/first-impressions-of-meshcore/#testing-th...
As I've said elsewhere, never underestimate Verizon's incompetence. A couple years ago they shut down vtext service which also happened to host their UAprof's for most Android phones. Without a working link to these descriptors, those devices would download any MMS (pre-RCS picture messages) in compatibility mode at the lowest resolution.
It came up for around 5-10 minutes at 15:00 EST, but is currently still down.
Something like a routing configuration, BGP failure, or underlying network misconfiguration would cause seemingly bizarre results with some phones working and some not with no obvious correlation. Compare Access Point Names under Mobile Settings on android, and whatever the equivalent is on iPhones, and check things like whether 5G allowed and data roaming is enabled.
If it's a cyber attack of some sort, then there's all sorts of different attack vectors that would cause these outcomes.
"Verizon engineering teams are continuing to address today's service interruptions. Our teams remain fully deployed and are focused on the issue. We understand the impact this has on your day and remain committed to resolving this as quickly as possible."
As someone responsible my whole career for uptime and network response, I really feel for the engineers, at the same time hoping my service comes back up soon. SOS
I assume state on state cyber attacks are commonplace but get minimized to avoid public fear.. perhaps this will be the first notable one.
Made worse by the fact Estonia is a more networked society than, for example, the US.
https://downdetector.com/status/t-mobile/ ~ 1,600
https://downdetector.com/status/att/ ~ 1,500
https://downdetector.com/status/verizon/ ~ peaked at ~169k, dropped to 67k
https://xkcd.com/1138/
https://downdetector.com/ shows verizon, tmobile, and att. BUT if you look at the magnitude of the outages for tmobile and att vs verizon it's fractions of a percent. Likely those people with tmobile and att reporting when they have trouble communicating with verizon customers.
(Note, you have to click on the providers to see absolute magnitude -- the graphs are scaled to show relative outage over time within a given provider; order of 150k verizon vs 1.5k others)
I noticed that a little while ago too. Very bad UI. Tempted to post in r/dataisugly.
Looking at the replies some of the people who are more worried about the outage seem to be the OF models
Not judging or trying to make a point. Just find it interesting the ways things are interconnected
Do you use your car’s built in navigation function — that you paid for — or do you plug your phone in and use its free Google Maps or Apple Maps to navigate?
Edit to say: my Verizon FioS and cell service are both working fine, no noticeable interruption at any point today.
Second edit to say: never mind, downdetector's home page normalizes report spikes so 1k and 100k both look identical.
I did manage to roam onto an international network on the boarder near me in New York/Canada, so some bits of the core seem functional for authentication.
When I roam internationally I appear to be on Telus's 3G network (no LTE) for data and voice is falling even further back it looks like.
It's a citizen-run mesh network that allows text communication. It's not perfect and not 100% reliable (signal and hops limit delivery guarantees) but it's better than 0 ways to communicate in this kind of event!
BART Defends Cutting Off Cell Service In Subway
https://www.npr.org/2011/08/16/139664637/bart-defends-cuttin...
And will happen again.