Update: Systems and services have now been restored, but expect intermittent issues.
If you’ve tried to use Wells Fargo online or mobile banking or a Wells Fargo ATM today then chances are it won’t have worked. This is as a result of a smoke at a server farm in Shoreview, Minnesota. Due to this farm servers have been shut down leading to this large outage.
We’re experiencing a systems issue that is causing intermittent outages, and we’re working to restore services as soon as possible. We apologize for the inconvenience.
— Wells Fargo (@WellsFargo) February 7, 2019
Some media outlets are reporting that the shut down is due to a fire rather than just smoke, but the official Wells Fargo twitter account only mentions smoke (although there is the old saying where there is smoke there is fire).
We’re experiencing system issues due to a power shutdown at one of our facilities, initiated after smoke was detected following routine maintenance. We’re working to restore services as soon as possible. We apologize for the inconvenience.
— Wells Fargo (@WellsFargo) February 7, 2019
What’s worrying is that because this is an isolated incident it appears that Wells Fargo doesn’t have any in built redundancy if one server farm goes down. It seems ridiculous that all of the Wells Fargo server assets for online, mobile and ATM banking is located in a single server farm. Normally for essential services such as this you’d have the load spread out over multiple different physical locations with additional resources available if one or more of those experiences downtime. Interestingly enough there are a number of people on reddit claiming to have first hand knowledge of what is currently occurring.
Let ’em burn to the ground 🙂
Last week my wells fargo business card number was used in a fraudulent airbnb transaction. hmmm. Good news is my state tax refund is appeared in my checking account.
interesting. I had the same thing happen about 3 weeks ago (wells fargo business CC used in a fraudulent airbnb transaction). I had only used the card a few times at giftcardmall last summer in order to earn the sign-up bonus. Card was in my possession at all times and no one else had access to it. So it seems either Wells Fargo or giftcardmall has been hacked.
wells fargos been hacked. I have never used giftcardmall. I used mine april and may of 2018 for the bonus and its been in my possession ever since.
Whats the worst out of the two:
1. Citi’s IT?
2. wells fargos scandals?
People think im a bad person for churning, but I would argue im just punching back for everything the bank has raped me with.
When I logged onto my Wells Fargo account this morning it asked me to verify a 2nd email address linked to my checking and savings accounts. It was almost exactly the same as my only email account but a few letters were different. I of course said the email was not correct and deleted it. Makes me wonder if some accounts were hacked/compromised during their technical issues. These are not my main checking and savings accounts but it’s still very concerning.
Still can’t activate a new debit card. Almost 24 hours since I began having problems. I feel bad for those who use as their main account.
My direct deposit didn’t go through last night, which it does every other Friday. I’m totally screwed as I was waiting on that to pay off my credit cards. It’s my only checking account. I think I just learned a lesson.
They’re saying that deposits actually posted, they’re just not displaying on their website yet. (And that problem might be fixed now.)
I logged in around 2:45pm pacific time and the app was working for me, it was sluggish and took a bit to load but I was able to check my balance and activity.
Crazy long reddit thread. Here’s the actual ‘first-hand knowledge’ bit: https://www.reddit.com/r/sysadmin/comments/ao4g2y/wells_fargo_is_down_declining_transactions_and_no/efy6wch/
” Throwaway since I have first hand knowledge. Fire suppression went off in one of their main Data Centres from some utility work this morning. No power to any of the network or compute equipment and some failovers did not work as expected.
​Edit: I’ve gotten a number of DMs asking for updates over the last few hours – so for those interested the last I heard (15-20 min ago) was that everything minus core network gear was manually being unplugged from any PDUs to help the control the initial power-on. No ETA on when that will be finished, let alone when things will be back up. “
That’s ridiculous lol. Manual unplugging…??
Checking account applications are affected too. This morning I couldn’t get past entering a zipcode without it timing out. This afternoon/evening I’ve been able to fill out the app, but it times out when submitting.
C-Suite executives had to look for new area’s to earn bonuses after losing bonuses based on new accounts(fake) an new product elections.
So a bonuses based on infrastructure & IT spending and savings materialized in Quarterly and Annual bonuse checks of C-Suite employees
hahahah
FWIW I Can access biz DDA accounts via mobile but not CC and a little banner on the acct summary page indicates as much, go thing that this is not my main business acct that I depend on for running my business.