Post Mortem: 986themix.com email issues
Hello folks,
This is a post mortem report with an issue we experienced on Thursday evening and a good majority of Friday. This only affected email, and all email was lost with the advice of moving forward as restoring could have caused issues. I don’t have any proof of that, but the person that works for the company I pay is an IT person by trade, and so we’re following what they are saying.
On Friday morning with no sign of email and the fact that when I tried to resend what was in my outbox and getting errors which I didn’t hear through my client, I opened a ticket.
The first email asked what client i was using, and of course I am using IOS and Thunderbird.
What was interesting about this, was none of the other email across other properties did not have issues until I find out otherwise.
Item 1 in a later response did indicate we moved servers which I already knew about. I had mentioned that I noticed that we changed IP addresses and i knew we were on a different one. That’s where that came from.
Item 2 was the most interesting item. It says:
2. The issue was due to mismatched permissions that apparently happened during the automatic cPanel upgrades that occur or someone changed the permissions on the “mail” folder within the home folder for themix cpanel account. Thus the Exim and Dovecot mail servers couldn’t access the mailbox which is why it defaulted to a no such user message.
I know I didn’t change anything, and I had access to email all day thursday. It was around 9 pacific time when I noticed email was completely gone, and many boxes I spotted were at 0 KB out of their alotted limit(s) which may or may not have meant something.
I am still unclear what permissions may have gotten changed, as i wasn’t even logged in, and usually updates don’t do things like this.
I honestly don’t know how an update would affect permissions like it did, and so, changing them back seemed to fix everything as mail was coming in after the change.
When I mentioned it wiped mail out of two folders including mine, they said it was definitely interesting.
As it goes in this industry, the path or least resistance is the way to go, so trying to restore mail could potentially cause problems with whatever they put in place. So since its working, new mail was arriving and restoring those boxes could lose that mail, we are not going to have that happening. Its a double edge problem that many may face, but I understand why this isn’t going to happen at this point.
My mailing for the first ticket opening was short.
What’s going on with 986themix.com email? I had email in my account and we had email in management, and I also tried to send to TSB and the phone app says it is placed in the outbox. When I looked, all of the mailboxes looked like they had no mail. I had email in my box yesterday.
When sending a test message, the mix’s server returned the customized message I personally set up for bouncing. So I knew something was working so it was definitely strange.
When I talk about the path of least resistance, this is what I wrote which caused that response.
Things seem to now be working, but I know I lost a ton of mail, some sent o my TSB list that I had intended to blog. That’s what got me confused. Is all that mail gone, or do we just push forward? It honestly doesn’t matter to me, but I appreciate it. I did go and close and reopen the app to see if it would repopulate the folder(s) but it did not.
As far as we can tell, everything is still working and we apologize for the issues. This type of thing can happen to anyone, and we’re pushing forward.
Please use jaredrimer.net and contact me with any questions. I’ll try to answer them the best I can. If I don’t have an answer, I’ll see if I can get one.
With everyone working together we can learn and unfortunately understandwhat to look for and fix it if we can later on.
Who knows.
I have had issues witth things where windows had well mangled itself due to a bug.
Support are as usefull though as a wet paper bag at least on the ms end.
In these cases its easier to remove and hide the effected update ratther than getting x, testing and rebuilding and not bothering.
It certainly is possible to lose bits of data, permitions and such during an update especially if x update is converting to a newer format or something.
I mean they are suupposed to tell you but the vender or maker may miss it in their manuals or notes I have had that happen a few times.
Google microsoft, etc just about everyone has done it.
No excuse butt still.
Its possible that during the update something screwed causing corrupted data so things were set wrongly and data was lost.
You will never know what that was.
I have had this happen on my end while doing a diagnostics run on a server with a wireless card who’s manufacturer was bustt and the card no longer worked.
A new card did fix it but it took ages to figure that out.
The issue looked like windows was corrupted.
The user did try to fix it himself but just made it worse.
Windows acted like it was really screwed and I got the unit and tested it.
Knowing it had an internal network card I pulled the wireless card, then plugged it into the network finding that at least half the issues were gone.
I had to reset some permitions, and recreate some accounts and reinstall corrupted drivers, updates, and clean the system but the systtem did then work.
I plugged the wireless card in and the issues returned.
Even with the card out windows was still unsttable.
I did a full rebuild and updated every app.
At the same time I ordered a new card and got it shipped.
It took a week for the rebuild and testing.
It took about the same for the card to show except I realised it was not a miniture card so it is a bit bulbous.
On the other hand the user’s performance went up and the system worked.
So not only did I get payed but I got a case of beer and a box of chocolate so yeah I was satisfied.
I have had nettwork systtems fail and filters, routers, etc.
And when I ask technical they are like well they are plastic, we don’t know why they do it but thats what happens.
This week main power was shut down for 4 hours.
It wasn’t a big loss but could have been.
It turns out that a main connecter had decided to expire.
It justt expired.
The techs appeared and it got replaced.
There was nothing super critical at the time though an invoice for some medical expenses was delayed by a day.
It would have been worse early morning or at night when it gets colder.
Its wintter now so that.