In the process of setting up my new server environment I have learned quite a few new little tricks to make this work. Firstly lets walk through the environment as I planned it before implementation
The key to note here is that I only have one externally facing IP address. I then have a decent router capable of port forwarding (any home model router will do here) at the front of my network. It routes traffic to the QNAP NAS and the Server running the virtual machines. You can also see that I setup a SAN between the QNAP and Server on the 10GBe network connections. The IP Addresses are not actually correct anymore but in terms of planning this worked out to help me wrap my head around what I wanted to do.
The DomO (server) is running CentOS 5.X since it had best hardware support – reasonable end of life and is generally secure and stable at this point. All of my DomU instances are running CentOS 6.X because they do the real work and I want the newest packages to install on them. Most of my instances are used for serving some sort of web content or application server content. All of them are proxied behind a dedicated Apache instance running ProxyPass (mod_proxy) rules to connect either via AJP or HTTP to the other instances. This allows my router to just forward all traffic coming in on port 80 to the dedicated Apache instance. I may later evaluate things like HAProxy or just some load balanced Apache solution, but for now this works as I am not dealing with high throughput traffic – maybe one day that’ll be a great problem to have and I will just move this setup to some fatter pipes.
It is worth noting that CentOS has some SE Linux rules enabled by default and they will break ProxyPass without making these changes:
http://sysadminsjourney.com/content/2010/02/01/apache-modproxy-error-13permission-denied-error-rhel
Immediately allow apache to connect out to other ports
/usr/sbin/setsebool httpd_can_network_connect 1
Permanently allow apache to connect out to other ports
/usr/sbin/setsebool -P httpd_can_network_connect 1
My DomO instance is synchronizing the entire CentOS distribution down via RSync so that installing new packages is super fast and occurs over the 1.x.x.x network (10GBe) rather than fetching from the outside world.
This is how I accomplished that
0 */6 * * * /usr/bin/rsync -avvzH –delete rsync://mirror.anl.gov/centos/ /opt/ISOS/CentOS/ > /tmp/centos.rsync.log
I noticed that I needed to be able to connect up to my machine remotely to adminster it so I installed X11VNC on the Dom0 instance and invoke it by calling these commands via SSH from anywhere I want to connect. I will likely lock this down to only accept ssh connections from certain IPs:
Connect to server with the following command and launch x11vnc
ssh -L 5900:localhost:5900 root@host.com ‘x11vnc’
or if no one is logged in yet
ssh -L 5900:localhost:5900 root@host.com ‘x11vnc -auth /var/gdm/\:0.Xauth’
After this I can run the command:
vncviewer localhost:0
(Reference above http://wiki.zenwalk.org/index.php?title=HOWTO_Use_x11vnc)
to connect to my machine and see the desktop. This is useful if I need to use graphical installers to configure my other DomU instances. This was a blocker to install Glassfish actually for my NETS instances so now it is good that I have solved that issue.
With respect to actually spinning up new servers, I installed a virtual machine by following documentation I found here:
http://pbraun.nethence.com/doc/sysutils_xen/dom0.html
However after creating a vm-skeleton and leaving it turned off I was able to clone it via a script to get me a few machines installed quickly and allocated.
Example of creating a new server:
virt-install -p –name=vm-1 –location=/opt/ISOS/CentOS/6/isos/x86_64/CentOS-6.2-x86_64-bin-DVD1.iso –bridge=xenbr0 –ram=4096 –disk path=/var/lib/xen/images/vm-skeleton,size=40,sparse=false
Duplication of servers (creates 10 servers)
for i in 1 2 3 4 5 6 7 8 9 10 ; do virt-clone -o vm-skeleton -n vm-$i -f /var/lib/xen/images/vm-$i ; done
The last issue I had to resolve was related to using AJP with Glassfish. I was creating proxypass rules like this:
<VirtualHost *:80>
ProxyPass / ajp://192.168.1.61:8080/ keepalive=on timeout=600
ProxyPassReverse / ajp://192.168.1.61:8080/
ServerName nets.neosavvy.com
</VirtualHost>
However the default number of threads that Apache allocates is 100 but the default that Glassfish will listen for is 5. So quickly Apache will overload Glassfish and start queuing requests and will cause them to timeout. Errors look like this:
[Wed Jan 18 14:08:59 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can’t receive header
[Wed Jan 18 14:08:59 2012] [error] ajp_read_header: ajp_ilink_receive failed
[Wed Jan 18 14:08:59 2012] [error] (120006)APR does not understand this error code: proxy: read response failed from (null) (192.168.1.61)
[Wed Jan 18 14:09:33 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can’t receive header
[Wed Jan 18 14:09:33 2012] [error] ajp_read_header: ajp_ilink_receive failed
[Wed Jan 18 14:09:33 2012] [error] (120006)APR does not understand this error code: proxy: read response failed from (null) (192.168.1.61)
All I had to do to fix this was to change the http-thread-listener’s thread-pool to be 100 instead of 5.
Reference (http://www.java.net/forum/topic/glassfish/glassfish/jkenabled-thread-leak)
I am sure I will have more notes to post and hopefully these will help someone out when they are setting up their own small network.
