Status Report April 20

[ultra]

Originally posted by sigurdur
At the moment there is 5 of these servers (also with dog-related names:-) that are the actual MX's. ...

Many thanks, considering you are both off the clock and with girlfriend wanting some time I had a feeling it would be much more than just one server, and understand about the forum getting less frequent responses.... I will take your word for it about dogs' names Thanks for your time.

[sigurdur]

Originally posted by marcus0263

[...]
But if you prefer to check further you can check dnsreports.com like ultra did to check Runbox. Or go ahead trace the routes, check out the mail headers, etc. etc. But again, I'm pretty easy and actually and took them at their word.
/B]

Marcus,

First off, I have not done any thorough check of Imap4all, and can't voucher for any redundancy or not to any big extent. I checked their incoming MX's very quickly, and found that they have two listed MX's:

mx0.imap4all.com. 213.201.213.169
mx1.imap4all.com. 213.201.213.170

and if you connect to them they bothpresent them selves as

220 mx0.imap4all.com ESMTP Postfix

since they have IP's right after each other it's not probeble that their in the same location (think of them as next-door neighbours). Whether they run one server with two IP's or have two servers, or run a system like Runbox's (see my description in another post) I can't say. But I doubt they have a setup that provides much more redundancy than Runbox'.

If you could forward their answer to your questioins I'd love that. Send it to sigurdur@runbox.com (my test account at Runbox:-)

kind regards,
-sig

[marcus0263]

Thanks Sig;

I'll fire off an e-mail to them and see what kind of response I get. Like I posted in an earlier post I think most of us are pretty much in the same boat. We're gun shy because of the issues of recent, this is why I for one am pursing the question of redundancy.

Your post more than explained that you do have in place fail over inside your facility. It has provided me with allot more confidence, and thanks for taking the time to respond

I would still like to focus on the traffic to and from your facility. Yes different geographic locations are additional costs and may not be cost effective for runbox at this time. Which I for one can live with and not be demanding about it. The basic concern for me though is do you have a secondary pipeline into the facility? Using the duel pipeline approach has the benefit of load balancing and fail over. There are software and hardware solutions like F5 Big IP that provide appliances to handle this, or do you have your own home grown solution? This is a sensible approach, and could actually be a benefit. I've been in environments where two telecommunication companies provided their pipeline into the facility. Our organization used them for both load balancing and fail over. The benefit was not only in speed and uptime but also the two vendors would compete against each other.

I'm not familiar with what is available over across the pond but I do think multiple pipelines into the facility can be both cost effective and a necessity. Especially for providing 24x7 global services, this should be considered. I'm assuming you do have somthing at least in comparison, but we just need a confirmation to help us restore our confidence. Also I need to state that no I'm not a network engineer but I have a enough base knowledge and experiance to be dangerous.

Thanks

[marcus0263]

Something to think of while you're reworking your infrastructure. Open up a couple more ports to IMAP and SMTP. To use my mail client I've been using port 443 for IMAP and SMTP with SSL with my other mail service. This has gotten me through numerous Firewalls with my mail client.

[carverrn]

Sig,

Thank you so much for taking the time to post such detailed information. I'm sure it will make a number of people feel a little better.

Regards,
Rich

[marc_otten]

Hi Sig,

Thanks for posting this very, very detailed feedback. It not only makes me feel more confident in Runbox's service, it also to a great extend increases my understanding of how (mail)providers design their infrastructure...most welcome

Regards,
Marc

[tore]

Originally posted by marcus0263

I would still like to focus on the traffic to and from your facility. Yes different geographic locations are additional costs and may not be cost effective for runbox at this time. Which I for one can live with and not be demanding about it. The basic concern for me though is do you have a secondary pipeline into the facility? Using the duel pipeline approach has the benefit of load balancing and fail over. There are software and hardware solutions like F5 Big IP that provide appliances to handle this, or do you have your own home grown solution? This is a sensible approach, and could actually be a benefit. I've been in environments where two telecommunication companies provided their pipeline into the facility. Our organization used them for both load balancing and fail over. The benefit was not only in speed and uptime but also the two vendors would compete against each other.

Yes, there are multiple lines going into the colo. It's one of Comnet's POPs in Oslo, and it's connected to the other POPs with dedicated fibres, probably in a partial mesh topology (I don't have a map of the trenches here, I'm afraid). Several of the POPs are connected to Nordic and global transit suppliers, and also to NIX 1 and 2.

Their border routers all speak BGP4 (of course) with their peers and transit suppliers, ensuring redudancy and automatic re-routing of traffic should one transit supplier or fibre fail. I'm not exactly sure what IGP they use, but probably OSPF or possibly IS-IS, and this also ensures automatic re-routing within Comnet's own AS.

I know that the router that are used in the core of the network (border routers and core routers at the various POPs) are Junipers - kick-arse gear, if I may say so. For providing interfaces to customers such as Runbox I believe they use Ciscos.

All in all, its everything you'd expect from a serious ISP. Comnet aka EUNet aka kpn/Qwest has been very reliable the last years, so I wouldn't worry much about them - very few, if any, of the problems Runbox has had lately can be blamed on the ISP.

I hope that makes you sleep better at night, and that I didn't get too technical for my audience (I tend to..).

PS: I'm with Linpro, just like Sigurd, and work quite a bit on Runbox, and I can only concur with what's been said by Hans and others - with the new backends they've hit bull's eye. Even though there's still some tweaks we can do to increase performance and reliability, the BIG problem was the old SAN, which is now totally unused and cannot affect the rig anymore. Good riddance, too.. Perhaps Runbox will arrange a storage array throwing contest some day, it seems it's the only thing it can be used for.

Tore

Edit: typo - IGP, not IRP

[marc_otten]

Hehehe

Perhaps Runbox will arrange a storage array throwing contest some day, it seems it's the only thing it can be used for

Kewl. . .what's going to be the location of this event?

But seriously, thanks for the info. Very detailed and a great addition to Sig's earlier explanation. I'm not a techie myself, but I'm highly interested in this kind of information!

-Marc

[marcus0263]

Thanks Tore;

Okay, I'll stop being a nag and pain where you sit. I just needed to get some kind of assurance, and it sounds like Runbox and Linpro have identified the problematic areas and have either fixed them or are well on the way to being fixed.

I myself have not asked for or expected any refunds, but know others have. I'll continue to monitor and play with my account and see how things go. If Runbox remains stable (let's all give them a few days to break in the new hardware though) I'll more than likely renew my account when it comes up for renewal. Also this is also a relief since I have my 78 year old father on a runbox account and was not looking forward to migrating him to another account. It took me 2 years to get him off his hotmail account to runbox.

I set him up with Linux (got tired of fixing Windows every other week) and I'm looking forward to runbox's return to stability so I won't have to hassle with his computer any longer

Again thank you Hans, Sig and Tor for taking the time to reassure us. I believe you're on your way back and we can all chalk this up as a learning experiance.

Cheers

[carverrn]

Originally posted by tore
... the BIG problem was the old SAN, which is now totally unused and cannot affect the rig anymore. Good riddance, too.. Perhaps Runbox will arrange a storage array throwing contest some day, it seems it's the only thing it can be used for.

And may they RIP!

Thanks for all the information Tore! I think? I'm going to have to go lookup what all those acronyms mean so I know what you just said.

Rich

[carverrn]

Originally posted by tore
Perhaps Runbox will arrange a storage array throwing contest some day, it seems it's the only thing it can be used for

Originally posted by marc_otten
Hehehe Kewl. . .what's going to be the location of this event?

Well it probably won't be a "multiple geographical locations" event. As we have already discussed that's a bit expensive when it comes to "hosting" servers ... or would that be "hoisting" servers in this case?

Rich

[imap4all]

Originally posted by sigurdur
Marcus,
<knip>
if you connect to them they bothpresent them selves as

220 mx0.imap4all.com ESMTP Postfix

But I doubt they have a setup that provides much more redundancy than Runbox'.
[/b]

Hi,

Just to clarify, yes our incoming server is mx0.imap4all.com and mx1.imap4all are the same. The DNS setup we currently have is mainly aimed at the future. Although I doubt if there still is a need for fallback MX services as current sending mail servers are pretty tolerant when they cannot connect to the primary or secondary MX server, they usualy start complaining after 4 hours. In case of a critical datacenter failure (long periods of loss of power, fire, flooding etc.) we are able to bring our core services up within a very short time in our second datacenter.

With regards to our redundancy. The changes that occur to the IMAP mailboxes are very frequent during the day. Daily backup to tape did not give us the level of comfort that we liked. If we would backup to tape our customers could loose 24 hour of email in a major catastrophe. We have opted for "warm" backups to a second datacenter. All our critical data is synced every hour to a secondary datacenter. Bandwidth is so cheap these day’s. In addition we are looking to backup the mirrors to tape as a kind of last resort insurance. We are not sure if the added benefit wais up against the substantial cost of a large tape library with multiple tape drive’s needed for fast tape backups.

In case of a major failure we can make the secondary site with the mirrors operational. With today's prices of disks, CPU’s and memory chips this setup has the same tco as a large backup system. With the added benefit that we have very good data reliability compared to tape backup. Last time we tested this it took 1 hour to become up and running again. Realistically including diagnostics it will take 3 hours to recover from a major disaster. Which is within the limits of our 99.95% service uptime target.

With regards,

- Brendan

[marcus0263]

Originally posted by imap4all
<snip>
In case of a major failure we can make the secondary site with the mirrors operational. With today's prices of disks, CPU’s and memory chips this setup has the same tco as a large backup system. With the added benefit that we have very good data reliability compared to tape backup. Last time we tested this it took 1 hour to become up and running again. Realistically including diagnostics it will take 3 hours to recover from a major disaster. Which is within the limits of our 99.95% service uptime target.

With regards,

- Brendan

Thanks Brendan for clarification.
I'd like to say I've been impressed with the speed and relaibility of IMAP4all

Cheers

[barbroyce]

This sounds like the possible cause of all the emails in my inbox vanishing! I discovered yesterday about 6pm EST. Is there a way to get them back? Thanks.