Hello Gang,
It has been a long two weeks or so with a lot of stupidity that happened which caused this post mortem report.
No … the entire network did not have a security incident. All data has been safely kept and it was never in danger.
I’m going to start this report with a question. How many people have subscribed to mailing lists through providers? I’m not talking about us specificly, say your favorite radio station. They provide mailing lists and its easy to subscribe.
Most people do not realize how things are done and this is supposed to be the case. That is, until shit isn’t even working.
Let’s now go back two weeks ago, almost 3. TSB, one of the two podcasts along the tech blog and podcast series, was completely finished and we were starting the process of getting things up on the RSS feed and EMHS’s TSB directory. We’ll have more on the RSS in a future blog post as we’re making changes.
I got an email indicating that an email couldn’t be delivered to a mailing list and that it wasn’t clear why. When I responded, I asked what the error message was. The short version, “the email address was not valid” was the response.
While investigating, I heard from one of my biggest customers and the main network where TSB is housed, 98.6 the mix and its owner, Herbie Allen.
He basicly said that the mailing lists were gone. The Mix has several lists one for internal use and one for external use.
My other biggest client, Menvi has has three external lists and one for internal use. Two of those externals are preferences for application processing.
The JRN itself has one internal/external list, which I set up for a specific purpose.
All of these lists that I am talking about were all completely gone. How? That seems to be a mystery, although the backbone we use for maintaining our web site and services did do an upgrade. These backbones are called control pannels and one of the biggest ones out there is called cpanel.
According to my provider, it seems like this control panel company does not really want to keep the mailing list software around in its current form. Instead, they broke things by only letting us have Python version 3 running when mailman needs pythoon version 2. This upgrade was done in one of the 126.x updates that is automaticly pushed to customers who pay for the software either through their provider such as our setup at the JRN or directly through them.
During this time, the provider that provides us services was able to contact their support team and get our lists back up and running as it seemed like Mailman, the program that is used, was not starting. I thought that was the end of it, until I did a little mre digging.
When I decided to find out what version of Mailman was running, it was telling me it was 2.3.x alpha. You read right, alpha.
Alpha software is used in testing what is new, and what is new is that it was using python version 3. Also, DKIM, a required protocol for mail these days, didn’t seem supported here. Turning this off had no affect.
What were we going to do now that we’ve got this problem and Cpanel doesn’t really communicate with anyone to tell them what to expect? We rolled back our control panel version which works with the current configuration with Python version 2.
During this time, we put up messages on the affected web sites we knew were affected, as well as posting on social media where appropriate.
We asked those who wanted updates through MENVI to contact us and we would provide updates either by telephone or by electronic communication.
During that time, accounts needed to be moved to a different server. We started communicating once again, and then we were moved back.
During the relocation period, the servers were talking to each other which gave us very interesting mail issues which were resolved.
Overall, I can’t complain about my provider, they definitely tried under the circumstances to provide the support in which we were asking.
Unfortunately, with business, you need to give refunds for things that are not working that customers are expecting. If I know a customer uses the thing that is down, it is my responsibility to refund them for the time that is down.
I’m happy to report that except for one thing, we’re back up and running. We fixed a major issue yesterday but found another one today dealing with form processing. That should be fixed very soon.
Cpanel, you aught to be asjhamed of yourselves. You should be communicating with your providers who pay whether directly or through resellers what your intent is. We should not be auto updating and finding our critical systems like mailing lists not working. It seems like you don’t know who uses it, and you should probably ask who uses it.
From what I’m told, you’ve been baught out recently, and now we’re given cookie consent messages by your new provider that is not necessary. We’re using things in a panel, we’re not filling out forms and giving you data you do not need.
I hope that Cpanel decides to keep Mailman around as there are not many options to have for discussion lists these days.
The biggest provider is Groups.io but they will not meet MENVI’s needs. Just looking at that alone, its $.04 per member, and I have three lists for specific things. The 100 member limit is pretty low, and MENVI has been established for 20 years online.
Sending email through your domain to 100 plus members is just not practical. Mail Chimp is good for announcement lists and can be used on domains, but I’ve not studied whether it works for discussion lists. My initial thought is no.
Heliohost, a provider I checked out, was forced to move away from Cpanel. The panel they use now offers mailman, but it is very very slow.
If Cpanel is expected to be revelent in today’s space, it should be advisable to know your customer base and what features those customers may be utalizing. I am not saying to know the small fries like me, but know who is supplying your product to their customers and ask questions.
You guys got the moron of the program this past Saturday and I’m sure you’ll get the moron of the program come TSB time as well.
I’d personally like to thank my provider whose customers besides me were quite affected by this, and my provider runs a full day job too.
Questions can be directed to:
Jared Rimer
The Jared Rimer Network
(888) 405-7524 or (818) 527-4754
Want to text or WhatsApp? (804-442-6975
Imessage: tech at menvi.org