Building clusterized proxy farms using LVS

2010-10-15T07:13:42Z

Wokfel: /* Introduction */

== Introduction ==
Sometimes, you just need to build a proxy farm with the following needs:
*you need high-availability, because of your SLA.
*you need load balancing, because you're serving numerous Kusers.
*you need caching with Squid.
*you need extra-services over Squid (URL filtering with DansGuardian/SquidGuard, Antivirus softwares, ...)
*you need scalability because you want to be able to grow.

Acai Berry is the best supplement available in the market. [http://www.articlesbase.com/supplements-and-vitamins-articles/my-weight-loss-experience-with-the-acai-berry-diet-1726693.html Acai Berry] has been a revolution and is preferred by most of the people. There are various benefits of [http://www.buzzle.com/articles/the-facts-and-half-truths-about-the-acai-berry-diet.html] but they are questioned at times. It is quite helpful for the weight losers and this fruit also provides you with enormous amount of energy.

In these cas, you've got plenty of answers to proceed. [[LVS]] is one of them (probably the best suited anyway).

But as you're doing extra-services over Squid, the base [[LVS]] [[ldirectord]] tests won't be sufficient.

This page explains how to make some more advanced tests to be sure some extra-services will be checked too, and the corresponding realserver service will be dropped down if one of the extra service (or Squid itself) is not available.

It present a proposed patch to [[ldirectord]] to do so.

== Architecture ==

The system architecture of web cache cluster using [[LVS]] is illustrated in the following figure.

[[Image:sl-ha-lb-overview-ultramonkey3.png|center]]

* it uses [[LVS]] with Ultramonkey 3 settings for a [http://www.ultramonkey.org/3/topologies/sl-ha-lb-overview.html Streamline High Availability and Load Balancing] setting.
* it's based on the usage for Squid + DansGuardian (but could feet for any Squid + *).

== Configuration Example ==
To set up this configuration with [[LVS]], just proceed to [http://www.ultramonkey.org/3/installation.html installation] and [http://www.ultramonkey.org/3/topologies/sl-ha-lb-eg.html configuration] as indicated on the Ultramonkey documentations.

Then :
* set up your Squid server, listening on port 3128 (example) and using DansGuardian on port 8000
* set up your DansGuardian server, listening to port 8000
* set up DansGuardian to reject (filter) the "http://proxy.testing.net/" URL (adding this domain/url to the blacklists. Be sure the "URL Filtering" will be returned by the proxy in case of filtering this URL.
* patch /usr/sbin/ldirectord with the FIXME patch
* configure your /etc/ha.d/ldirectord.cf with the following tests (example IP addresses):

# /etc/ha.d/ldirectord.cf
# Virtual Server for Proxy Service and Filtering Service
virtual=192.168.0.100:3128
real=192.168.0.10:3128 gate
real=192.168.0.11:3128 gate
real=192.168.0.12:3128 gate
real=192.168.0.12:3128 gate
service=proxy_http
request="http://proxy.testing.net"
receive="URL Filtering"
protocol=tcp
scheduler=wlc
protocol=tcp
checktype=negotiate

== Conclusion ==
If you were using http test, you could only be able to check Squid is running. In case of DansGuardian issue, it could be a bad thing. This way, if the filtering engine is down/not responding, the whole Squid realserver will be dropped down for [[LVS]].

Of course, you must still monitor all process to be warned anything is going wrong. But you will have time to resolve as your users won't notice anything :) Thank's a lot to [[LVS]] again!

== The ldirectord patch ==
This patch is to be applied to release +ldirectord,v 1.77.2.32+, but is basic enought for you to report it to any newer release of [[ldirectord]].

--- ldirectord_sav 2005-10-03 08:18:19.000000000 +0200
+++ ldirectord 2005-11-13 14:39:51.000000000 +0100
@@ -236,7 +236,7 @@ checking will take place and no real or
On means no checking will take place and real servers will always be
activated. Default is I<negotiate>.

-B<service = ftp>|B<smtp>|B<http>|B<pop>|B<nntp>|B<imap>|B<ldap>|B<https>|B<dns>|B<mysql>|B<pgsql>|B<sip>|B<none>
+B<service = ftp>|B<smtp>|B<http>|B<proxy_http>|B<pop>|B<nntp>|B<imap>|B<ldap>|B<https>|B<dns>|B<mysql>|B<pgsql>|B<sip>|B<none>

The type of service to monitor when using checktype=negotiate. None denotes
a service that will not be monitored. If the port specfied for the virtual
@@ -866,8 +866,8 @@ sub read_config
}
} elsif ($rcmd =~ /^service\s*=\s*(.*)/) {
lc($1);
- $1 =~ /(\w+)/ && ($1 eq "http" || $1 eq "https" || $1 eq "ldap" || $1 eq "ftp" || $1 eq "none" || $1 eq "smtp" || $1 eq "pop" || $1 eq "imap" || $1 eq "nntp" || $1 eq "dns" || $1 eq "mysql" || $1 eq "pgsql" || $1 eq "sip")
- or &config_error($line, "service must be http, https, ftp, smtp, pop, imap, ldap, nntp, dns, mysql, pgsql, sip, or none");
+ $1 =~ /(\w+)/ && ($1 eq "http" || $1 eq "proxy_http" || $1 eq "https" || $1 eq "ldap" || $1 eq "ftp" || $1 eq "none" || $1 eq "smtp" || $1 eq "pop" || $1 eq "imap" || $1 eq "nntp" || $1 eq "dns" || $1 eq "mysql" || $1 eq "pgsql" || $1 eq "sip")
+ or &config_error($line, "service must be http, https, proxy_http, ftp, smtp, pop, imap, ldap, nntp, dns, mysql, pgsql, sip, or none");
$vsrv{service} = $1;
if($vsrv{service} eq "ftp" and
$vsrv{login} eq "") {
@@ -1635,6 +1635,8 @@ sub ld_main
$$r{num_connects} = 0 if (check_http($v, $r));
# my $req = new HTTP::Request(GET=>"$$r{url}");
# $ua->register($req, \&http_received);
+ } elsif ($$v{service} eq "proxy_http") {
+ $$r{num_connects} = 0 if (check_proxy_http($v, $r));
} elsif ($$v{service} eq "pop") {
$$r{num_connects} = 0 if (check_pop($v, $r));
} elsif ($$v{service} eq "imap") {
@@ -1693,6 +1695,51 @@ sub ld_main
}
}

+sub check_proxy_http
+{
+ use LWP::UserAgent;
+ use LWP::Debug;
+
+ my ($v, $r) = @_;
+
+ my $proxy_request = substr($$r{request},1);
+
+ my $ua = new LWP::UserAgent();
+
+ $ua->timeout($$v{negotiatetimeout});
+ $ua->proxy(['http', 'ftp'], "http://$$r{server}:$$r{port}/");
+
+ my $h = new HTTP::Headers("Host" => $$v{virtualhost});
+ my $req = new HTTP::Request("$$v{httpmethod}", "$proxy_request", $h);
+
+ my $res;
+ {
+ # LWP makes ungaurded calls to eval
+ # which throw a fatal exception if they fail
+ # Needless to say, this is completely stupid.
+ local $SIG{'__DIE__'} = "DEFAULT";
+ $res = $ua->request($req);
+ }
+
+ my $recstr = $$v{receive};
+
+ if ($res->is_success && (!($recstr =~ /.+/) || $res->content =~ /$recstr/)) {
+ service_set($v, $r, "up");
+ &ld_debug(2, "check_proxy_http: $proxy_request is up\n");
+ return 1;
+ }
+
+ service_set($v, $r, "down");
+ &ld_debug(3, "Headers " . $res->headers->as_string);
+ &ld_debug(2, "check_proxy_http: $proxy_request is down\n");
+ return 0;
+
+}

More infos? mailto:christian.avramakis@siemens.com
[[Category:LVS Examples|Cache]]

LVSKB - User contributions [en]

Building clusterized proxy farms using LVS