What happened during our downtime?

Hi folks,
I think it is time for an explanation of what happened during the time we were offline. Shaun was partially correct in that the big issue was the heartbleed bug. That was patched on the server the week the news was out about it and only lasted a few hours. This was a lot different.

Let me make this clear that we still do not know what really happened. I can tell you that there was a report of a trojan somewhere. We don’t know where it came from, nor how it was found and removed beforehand. I was tipped off on Sunday, the 13th, of a bounce claiming that the IP was blocked. I thought nothing of this because IPS get blocked all the time. On Monday, the 14th., I was talking to someone, and found that this was a bigger issue than we thought.

On the 14th, I evidently submitted two tickets, but only intended to send one. This lead to an investigation. The provider acted quickly, and did the best he could. It was confirmed that something was on the server, and that server is still offline today. On my live journal page, I was trying to keep people updated with updates. I really think our provider did the best they could in getting us back and operating in a matter of days instead of weeks.

I was told that to provision a server, it takes weeks. The contact that was contacted happened to have one not being used, and we were moved to it. I was fielding text messages and one call during this time. I really commend the support staff at the provider level and their support staff for getting us up quickly. We’re truly sorry that something as bad as a trojan could knock a server which operated well offline. Sadly, this could happen to any provider whether it is a small provider or big one. I did reach out to a security guy to ask some questions for my own knowledge. I’m not going to sit here and publish everything I find out, as this is not who I am as a person who does podcasting and blogging. Should you have any questions, please write me at tech at menvi dot org, or call me. We look to be stable once again, and for that we’re greatful. Have a great day.

• final network update (April 18, 2014
• network status update April 16, 2014
• server down April 15, 2014
• a mail issue effecting the network April 14, 2014

As you can see from these posts, we really don’t go in to any type of detail, but I try to give updates. Its always good to provide updates where people can read what is up, and I am doing that here. Again, I want to thank the people at the hosting provider and the provider above him for helping get services up and running as quick as possible. Keep up the great work.