| Posted by quad3datwork, 10-30-2010, 03:06 AM | | I'm just throwing ideas around and trying to get some opinions.
Right now I got two back-end servers running its own Apache & MySQL server. People hit a front-end server running nginx reverse proxy back to the two in active-passive setup. The two back-end servers are setup with rsync for flat files and master-to-master replication for the DBs. Surprisingly, this setup worked damn well for many months.
Couple a days ago I was bored and thought I want to take things further. So i researched on Varnish and HAProxy. At the moment I guess I don't need to think too much about Varnish. I'd like to put up two HAProxy servers in a round robin DNS fashion with four back-end servers.
I already scripted the flat files rsync process across the four servers and works well.
I'm just a little puzzled how MySQL would work. Never dealt with MySQL cluster before... so I'm wondering what's the ideal way to keep the DBs synced across four servers? Since all six servers (2 HAProxy & 4 back-end) would all be in different geographical locations. I think that split out additional servers as dedicated MySQL would probably not be ideal performance wise.
I think doing multiple master-to-master MySQL replications is risky. When thing breaks, it will look like a lot of work to sync up again.
So... all been said, I'm welcome to any and all suggestions and ideas. Let's keep a nice, healthy conversation going! Cheers and thanks in advance!
|
| Posted by matt2kjones, 10-30-2010, 08:12 AM | | Hello,
We have a simular setup. We use HAProxy on two front end servers running Heartbeat. Only one of the HAProxy servers have a public ip, but both have private addresses. Heartbeat monitors the frontend servers and if one goes down, the other HAProxy server takes the public IP.
We have lots of backend servers running apache and a mysql cluster. MySQL clusters scale well and are a much better solution that just plain old master slave replication.
We use unison for bi-directional updates between the backend machines for flat files.
Here is the page I used to setup my mysql cluster: http://www.howtoforge.com/loadbalanc...cluster_debian
It is pretty easy, works well and reliably.
|
| Posted by quad3datwork, 10-30-2010, 09:56 AM | | matt2kjones
Thanks for your suggestion. Yes, that was one of my primary concern - scalability. I just want an easy-to-maintain system and able to scale up and down without much fuss.
I'm actually using Unison as well, not necessary rsync. When you said 'bi-directional' you meant like daisy chain style? If so, how do you manage a node failure in Unison? My script will detect timeout and try the next node. This style works well in four nodes setup... but I see some scale issues later down the road scaling up. Well, this is just a pet project. If I was all serious about it, I probably will have some type of shared storage.
Thanks again!
|
| Posted by matt2kjones, 10-30-2010, 10:03 AM | | When I say Bi-Directional I simply meant transfering files in both directions.
We have it in a kind of "hub leaf" setup where we generally put new files on one server and replicate them all outwards from there (because the only files going onto our servers are uploaded by us, not users). If I wanted files replicating from any server to any other server then I would look for another solution, however, I can't really help you with this as I haven't done any research on the matter.
|
| Posted by mugo, 10-31-2010, 02:46 AM | | With a little more background, I've done similar setups...how often does your data change, and is it possible for you to split write / reads? There are complex ways to do it, but I've also done something like Master-master with added slave servers, the slaves get mostly reads, all writes go to the master(s).
Not knowing exactly how you run, if you have, say, CMSs that are updated occasionally, I've even done it by pointing "admin.domain.com" to master(s), and just reads (*.domain.com) hit with the LBs.
Doing the two HAproxy's with round-robin is great, too, if you set it up right..just finished one where the client had 4 backend nodes, each LB preferred (by weight) two different nodes, so it really spreads and balances traffic well. With DNS failover, it self-heals even if an LB dies. There are so many cooler methods than heartbeat I have found, it's been a while since I've even done one.
Small db's I've even host-to-host mysqldumped (straight from db to db using --opt -h) on a cron schedule...only works if the db's are updated infrequently.
Lordy, I've done a lot of Haproxy / file/db sync setups lately...I guess everyone is seeing the light? "downtime sucks".
|
| Posted by quad3datwork, 10-31-2010, 03:36 AM | | Thanks for both of your inputs!
@mugo, I'm not doing anything serious. I'm just experimenting and learning. I prefer have every and all nodes, regardless of purpose, in active node - for no really specific reason.
The only thing with the round robin DNS I'm worrying about is the short TTL and how the session cookie would be handled. Looks like you have some experience in this field... just having multiple A records points to the dual HAproxies with very short TTL cause any session issues?
|
| Posted by mugo, 10-31-2010, 01:16 PM | | The end user will be fine as long as they don't let the HTTP session time out, usually about 15 min. The haproxy cookies and session persistence will assure that in most instances a user stays at the same LB unless the http session times out. You can also adjust that TTL to what you can live with to get more assurance.
Are you doing this with co-lo, dedi, vps?
Since you are experimenting, you could seven have a 3rd Haproxy with the other two HAs as it's backends, assuring you have ultimate control over sessions.
|
| Posted by UnixCabin, 11-01-2010, 11:44 PM | | Make sure your app would be fully supported under ndbcluster store. Depend on your mysql version, ndbcluster don't support all the features myisam would.
Multi-master may also be look at it since you only have 4 nodes. Make sure latency is short.
|
| Posted by quad3datwork, 11-02-2010, 04:50 PM | | Anyone can comment on this type of setup?
|
| Posted by DataCentric, 11-02-2010, 05:12 PM | | Hi,
Is your application read or write heavy? Do you have general numbers? It might make sense, depending on your application to have reads sent to read only DB's, and save updates inserts and other changes for the masters.
You could also look into database sharding.
|
| Posted by quad3datwork, 11-03-2010, 10:50 AM | | RackPoint-Morgan
It's just standard CMSs like WP and Drupal. The end goal is not achieving to solve a problem, but for my own personal learning experience.
I've been reading up on sharding. Looks like it's the big word in DB world right now. One & two interesting articles I read yesterday.
Thanks for your input. Appreciated!
|
| Posted by viGeek, 11-03-2010, 11:44 AM | | Is the mySQL cluster going to be active-active?
You can take a look at DRBD which can synchronize the data, I've only been working with it a little for about a year or so but have done several deployments, works nicely, albeit in active-passive configuration.
|
| Posted by quad3datwork, 11-03-2010, 12:52 PM | | vigor
DRBD would be viable if I'm not dealing with VPSs this time. I base learnings on VPSs for proof-of-concept. Yes, regardless of how it is setup, I want all nodes in active-active mode.
Think I'm backing out on the MySQL cluster due to store types (ndbcluster). Having dealt with managing source+package hybrid servers before, this time I prefer stick with the distro packages. To ease the level of maintenance. I'm using Debian with Dotdeb repos.
I stuck with the plain good old multi-master replication schema across four nodes utilizing federated tables & event scheduler. I'm still working on it. I'll update how it goes!
Thanks!
|
|
 Add to Favourites
 Print this Article
|
