Building clusterized proxy farms using LVS - Revision history

Wensong: Reverted edits by Wokfel (Talk); changed back to last version by Wensong

2010-10-24T04:40:02Z

Reverted edits by Wokfel (Talk); changed back to last version by Wensong

Wokfel: /* Introduction */

2010-10-15T07:13:42Z

‎Introduction

Wensong: add more internal links

2006-08-27T15:53:36Z

add more internal links

Olem at 15:58, 16 November 2005

2005-11-16T15:58:58Z

Olem at 15:51, 16 November 2005

2005-11-16T15:51:48Z

Olem at 15:49, 16 November 2005

2005-11-16T15:49:10Z

Olem at 15:39, 16 November 2005

2005-11-16T15:39:23Z

Olem at 15:38, 16 November 2005

2005-11-16T15:38:11Z

New page

== Introduction ==
Sometimes, you just need to build a proxy farm with the following needs:
*you need high-availability, because of your SLA.
*you need load balancing, because you're serving numerous Kusers.
*you need caching with Squid.
*you need extra-services over Squid (URL filtering with DansGuardian/SquidGuard, Antivirus softwares, ...)
*you need scalability because you want to be able to grow.

In these cas, you've got plenty of answers to proceed. LVS is one of them (probably the best suited anyway).

But as you're doing extra-services over Squid, the base LVS ldirectord tests won't be sufficient.

This page explains how to make some more advanced tests to be sure some extra-services will be checked too, and the corresponding realserver service will be dropped down if one of the extra service (or Squid itself) is not available.

It present a proposed patch to ldirectord to do so.

== Architecture ==

The system architecture of web cache cluster using LVS is illustrated in the following figure.

FIXME
[[Image:Cache-cluster.jpg|center]]

* it use LVS with Ultramonkey 3 settings for a [http://www.ultramonkey.org/3/topologies/sl-ha-lb-overview.html Streamline High Availability and Load Balancing] setting.
* it is base on the usage for Squid + DansGuardian (but could feet for any Squid + *).

== Configuration Example ==

== Conclusion ==

[[Category:LVS Examples|Cache]]

@@ Line 7: / Line 7: @@
 *you need scalability because you want to be able to grow.
-In these cas, you've got plenty of answers to proceed. LVS is one of them (probably the best suited anyway).
+In these cas, you've got plenty of answers to proceed. [[LVS]] is one of them (probably the best suited anyway).
-But as you're doing extra-services over Squid, the base LVS ldirectord tests won't be sufficient.
+But as you're doing extra-services over Squid, the base [[LVS]] [[ldirectord]] tests won't be sufficient.
 This page explains how to make some more advanced tests to be sure some extra-services will be checked too, and the corresponding realserver service will be dropped down if one of the extra service (or Squid itself) is not available.
-It present a proposed patch to ldirectord to do so.
+It present a proposed patch to [[ldirectord]] to do so.
 == Architecture ==
-The system architecture of web cache cluster using LVS is illustrated in the following figure.
+The system architecture of web cache cluster using [[LVS]] is illustrated in the following figure.
   [[Image:sl-ha-lb-overview-ultramonkey3.png|center]]
-* it uses LVS with Ultramonkey 3 settings for a [http://www.ultramonkey.org/3/topologies/sl-ha-lb-overview.html Streamline High Availability and Load Balancing] setting.
+* it uses [[LVS]] with Ultramonkey 3 settings for a [http://www.ultramonkey.org/3/topologies/sl-ha-lb-overview.html Streamline High Availability and Load Balancing] setting.
 * it's based on the usage for Squid + DansGuardian (but could feet for any Squid + *).
 == Configuration Example ==
-To set up this configuration with LVS, just proceed to [http://www.ultramonkey.org/3/installation.html installation] and [http://www.ultramonkey.org/3/topologies/sl-ha-lb-eg.html configuration] as indicated on the Ultramonkey documentations.
+To set up this configuration with [[LVS]], just proceed to [http://www.ultramonkey.org/3/installation.html installation] and [http://www.ultramonkey.org/3/topologies/sl-ha-lb-eg.html configuration] as indicated on the Ultramonkey documentations.
 Then :
@@ Line 52: / Line 52: @@
 == Conclusion ==
-If you were using http test, you could only be able to check Squid is running. In case of DansGuardian issue, it could be a bad thing. This way, if the filtering engine is down/not responding, the whole Squid realserver will be dropped down for LVS.
+If you were using http test, you could only be able to check Squid is running. In case of DansGuardian issue, it could be a bad thing. This way, if the filtering engine is down/not responding, the whole Squid realserver will be dropped down for [[LVS]].
-Of course, you must still monitor all process to be warned anything is going wrong. But you will have time to resolve as your users won't notice anything :) Thank's a lot to LVS again!
+Of course, you must still monitor all process to be warned anything is going wrong. But you will have time to resolve as your users won't notice anything :) Thank's a lot to [[LVS]] again!
 == The ldirectord patch ==
-This patch is to be applied to release +ldirectord,v 1.77.2.32+, but is basic enought for you to report it to any newer release of ldirectord.
+This patch is to be applied to release +ldirectord,v 1.77.2.32+, but is basic enought for you to report it to any newer release of [[ldirectord]].
   --- ldirectord_sav	2005-10-03 08:18:19.000000000 +0200

← Older revision		Revision as of 15:58, 16 November 2005
Line 56:		Line 56:
	Of course, you must still monitor all process to be warned anything is going wrong. But you will have time to resolve as your users won't notice anything :) Thank's a lot to LVS again!		Of course, you must still monitor all process to be warned anything is going wrong. But you will have time to resolve as your users won't notice anything :) Thank's a lot to LVS again!

		+	== The ldirectord patch ==
		+	This patch is to be applied to release +ldirectord,v 1.77.2.32+, but is basic enought for you to report it to any newer release of ldirectord.
		+
		+	--- ldirectord_sav 2005-10-03 08:18:19.000000000 +0200
		+	+++ ldirectord 2005-11-13 14:39:51.000000000 +0100
		+	@@ -236,7 +236,7 @@ checking will take place and no real or
		+	On means no checking will take place and real servers will always be
		+	activated. Default is I<negotiate>.
		+
		+	-B<service = ftp>\|B<smtp>\|B<http>\|B<pop>\|B<nntp>\|B<imap>\|B<ldap>\|B<https>\|B<dns>\|B<mysql>\|B<pgsql>\|B<sip>\|B<none>
		+	+B<service = ftp>\|B<smtp>\|B<http>\|B<proxy_http>\|B<pop>\|B<nntp>\|B<imap>\|B<ldap>\|B<https>\|B<dns>\|B<mysql>\|B<pgsql>\|B<sip>\|B<none>
		+
		+	The type of service to monitor when using checktype=negotiate. None denotes
		+	a service that will not be monitored. If the port specfied for the virtual
		+	@@ -866,8 +866,8 @@ sub read_config
		+	}
		+	} elsif ($rcmd =~ /^service\s=\s(.*)/) {
		+	lc($1);
		+	- $1 =~ /(\w+)/ && ($1 eq "http" \|\| $1 eq "https" \|\| $1 eq "ldap" \|\| $1 eq "ftp" \|\| $1 eq "none" \|\| $1 eq "smtp" \|\| $1 eq "pop" \|\| $1 eq "imap" \|\| $1 eq "nntp" \|\| $1 eq "dns" \|\| $1 eq "mysql" \|\| $1 eq "pgsql" \|\| $1 eq "sip")
		+	- or &config_error($line, "service must be http, https, ftp, smtp, pop, imap, ldap, nntp, dns, mysql, pgsql, sip, or none");
		+	+ $1 =~ /(\w+)/ && ($1 eq "http" \|\| $1 eq "proxy_http" \|\| $1 eq "https" \|\| $1 eq "ldap" \|\| $1 eq "ftp" \|\| $1 eq "none" \|\| $1 eq "smtp" \|\| $1 eq "pop" \|\| $1 eq "imap" \|\| $1 eq "nntp" \|\| $1 eq "dns" \|\| $1 eq "mysql" \|\| $1 eq "pgsql" \|\| $1 eq "sip")
		+	+ or &config_error($line, "service must be http, https, proxy_http, ftp, smtp, pop, imap, ldap, nntp, dns, mysql, pgsql, sip, or none");
		+	$vsrv{service} = $1;
		+	if($vsrv{service} eq "ftp" and
		+	$vsrv{login} eq "") {
		+	@@ -1635,6 +1635,8 @@ sub ld_main
		+	$$r{num_connects} = 0 if (check_http($v, $r));
		+	# my $req = new HTTP::Request(GET=>"$$r{url}");
		+	# $ua->register($req, \&http_received);
		+	+ } elsif ($$v{service} eq "proxy_http") {
		+	+ $$r{num_connects} = 0 if (check_proxy_http($v, $r));
		+	} elsif ($$v{service} eq "pop") {
		+	$$r{num_connects} = 0 if (check_pop($v, $r));
		+	} elsif ($$v{service} eq "imap") {
		+	@@ -1693,6 +1695,51 @@ sub ld_main
		+	}
		+	}
		+
		+	+sub check_proxy_http
		+	+{
		+	+ use LWP::UserAgent;
		+	+ use LWP::Debug;
		+	+
		+	+ my ($v, $r) = @_;
		+	+
		+	+ my $proxy_request = substr($$r{request},1);
		+	+
		+	+ my $ua = new LWP::UserAgent();
		+	+
		+	+ $ua->timeout($$v{negotiatetimeout});
		+	+ $ua->proxy(['http', 'ftp'], "http://$$r{server}:$$r{port}/");
		+	+
		+	+ my $h = new HTTP::Headers("Host" => $$v{virtualhost});
		+	+ my $req = new HTTP::Request("$$v{httpmethod}", "$proxy_request", $h);
		+	+
		+	+ my $res;
		+	+ {
		+	+ # LWP makes ungaurded calls to eval
		+	+ # which throw a fatal exception if they fail
		+	+ # Needless to say, this is completely stupid.
		+	+ local $SIG{'__DIE__'} = "DEFAULT";
		+	+ $res = $ua->request($req);
		+	+ }
		+	+
		+	+ my $recstr = $$v{receive};
		+	+
		+	+ if ($res->is_success && (!($recstr =~ /.+/) \|\| $res->content =~ /$recstr/)) {
		+	+ service_set($v, $r, "up");
		+	+ &ld_debug(2, "check_proxy_http: $proxy_request is up\n");
		+	+ return 1;
		+	+ }
		+	+
		+	+ service_set($v, $r, "down");
		+	+ &ld_debug(3, "Headers " . $res->headers->as_string);
		+	+ &ld_debug(2, "check_proxy_http: $proxy_request is down\n");
		+	+ return 0;
		+	+
		+	+}
		+
		+	More infos? mailto:christian.avramakis@siemens.com
	[[Category:LVS Examples\|Cache]]		[[Category:LVS Examples\|Cache]]

← Older revision		Revision as of 15:51, 16 November 2005
Line 52:		Line 52:

	== Conclusion ==		== Conclusion ==
		+	If you were using http test, you could only be able to check Squid is running. In case of DansGuardian issue, it could be a bad thing. This way, if the filtering engine is down/not responding, the whole Squid realserver will be dropped down for LVS.

		+	Of course, you must still monitor all process to be warned anything is going wrong. But you will have time to resolve as your users won't notice anything :) Thank's a lot to LVS again!

	[[Category:LVS Examples\|Cache]]		[[Category:LVS Examples\|Cache]]

@@ Line 20: / Line 20: @@
 FIXME
-  [[Image:Cache-cluster.jpg|center]]
+  [[Image:sl-ha-lb-overview-ultramonkey3.png|center]]
 * it use LVS with Ultramonkey 3 settings for a [http://www.ultramonkey.org/3/topologies/sl-ha-lb-overview.html Streamline High Availability and Load Balancing] setting.