Java Obsession: apache

Terracotta and Tomcat Clustering - Page 3

Ok, our tutorial will end by adding Apache web server to load balance the traffic between both Tomcats.

install Apache Web Server

just run the msi installer or install it from an RPM or from the OS repositories, i will refer to the installation directory as ${APACHE_HOME}.

point your browser to localhost and make sure you are welcomed with the most famous message

It Works!

install mod_jk

extract the file tomcat-connectors-x.x.xx-windows-arch-httpd-2.2.x.zip, copy mod_jk.so to ${APACHE_HOME}\modules

Configure Apache Web Server

httpd.conf

edit the file ${APACHE_HOME}\conf\httpd.conf to add the following settings:

LoadModule jk_module modules/mod_jk.so
JkWorkersFile conf/workers.properties
JkShmFile logs/mod_jk.shm
JkLogFile logs/mod_jk.log
JkLogLevel info
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkMount /examples loadbalancer
JkMount /examples/* loadbalancer
JkMount /status status
JkMount /status/* status

here we are loading the mod_jk module, the mod_jk will load cluster members (Tomcats) from a file named workers.properties under ${APACHE_HOME}\conf directory.
also, any requests coming to apache at /examples will be forwarded to one of our Tomcats.
while /status will direct us to the mod_jk control panel.

workers.properties

create a new file under the directory ${APACHE_HOME}\conf named workers.properties with the following content:

# Define list of workers that will be used
worker.list=loadbalancer,status
# Define tomcat1
worker.tomcat1.port=8009
worker.tomcat1.host=127.0.0.1
worker.tomcat1.type=ajp13
worker.tomcat1.lbfactor=1
# Define tomcat2
worker.tomcat2.port=8010
worker.tomcat2.host=127.0.0.1
worker.tomcat2.type=ajp13
worker.tomcat2.lbfactor=1
# Load-balancing behaviour
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=tomcat1,tomcat2
worker.loadbalancer.sticky_session=0
# Status worker for managing load balancer
worker.status.type=status

note that the worker names must match the jvmRoute configurations on the Tomcats. also, you can enable sticky sessions or disable to fit your need, however, i have turned it off here.

restart apache web server, if everything went as it should, navigating to http://localhost/examples would result in this page.

if you have debugging installed on your browser, like firebug, you can check your JSESSIONID value, you will find it post fixed with the jvmRoute value of the tomcat server who served the response.

if you hit the refresh button you can see that you are moving for every hit from one Tomcat to the other.

Testing Sessions Failover

point your browser to http://localhost/examples/servlets/servlet/SessionExample add a couple of values in the session.

now it is up to you, bring any Tomcat down (not both, of course) or any Terracotta (not both, of course), play around, your minimal setup is one Tomcat and one Terracotta server.

your values in the session should never disappear!!!

Have Fun.

Terracotta and Tomcat Clustering - Page 2

Now, download apache-tomcat-6.0.35 and extract it twice to two directories of your choice, i will refer to them as ${CATALINA_HOME_NODE1} and ${CATALINA_HOME_NODE2}.

Copy Terracotta Libraries

we need two jar files to be copied to both Tomcats lib directory.

${TC_HOME}\common\terracotta-toolkit-1.6-runtime-5.0.0.jar

${TC_HOME}\sessions\terracotta-session-1.3.0.jar

Copy Terracotta Config File

we need to copy the same Terracotta configuration file to the conf directory of both Tomcats

${TC_HOME}\config\tc-config.xml

one small change to do, the client log location, both clients (Tomcat servers) can't be configured with the same client log location.

edit ${CATALINA_HOME_NODE2}\conf\tc-config.xml

change log location under the clients tag

<logs>%(user.home)/terracotta/client-logs</logs>

to be

<logs>%(user.home)/terracotta/client-logs2</logs>

Terracotta Valve

Tomcat connects to Terracotta by intercepting requests via a value, now let's hook that valve.

under the conf directory of both Tomcats, edit the file context.xml to look like this:

<Context>

 <!-- Default set of monitored resources -->
 <WatchedResource>WEB-INF/web.xml</WatchedResource>

 <!-- Uncomment this to disable session persistence across Tomcat restarts -->
    <Manager pathname="" />
 
 <!-- Hook the Terracotta Valve and configure it using the tc-config.xml file -->
 <Valve className="org.terracotta.session.TerracottaTomcat60xSessionValve" tcConfigUrl="../conf/tc-config.xml"/>

</Context>

note that we have disabled session persistence across Tomcat restarts because that is a task for Terracotta now.

Tomcat HA Configurations

jvmRoute

we need to define a unique jvmRoute for each Tomcat server.

edit ${CATALINA_HOME_NODE1}\conf\server.xml, change the Engine tag to look like this:

<Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat1">

edit ${CATALINA_HOME_NODE2}\conf\server.xml, change the Engine tag to look like this:

<Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat2">

Make sure you don't have port conflicts

as i am running both Tomcats on the same box, i had to change the ports for the second Tomcat.
in ${CATALINA_HOME_NODE2}\conf\server.xml, i have changed
shutdown port: 8005 to 8006
http port: 8080 to 8081
ajp port: 8009 to 8010

Run both Tomcats

open two more terminals and navigate to ${CATALINA_HOME_NODE1}\bin on one and ${CATALINA_HOME_NODE2}\bin on the other, on both terminals execute:

startup.bat

if everything went as it should, you should see something like that on both Tomcat terminals

Terracotta 3.7.0, as of 20120717-133013 (Revision unknown-20453 by cruise@rh5vmo113.terracotta.lan from 3.7.0)
Successfully loaded base configuration from file at 'H:\terracotta\apache-tomcat-6.0.35\bin\.\..\conf\tc-config.xml'.
Successfully loaded base configuration from file at 'H:\terracotta\apache-tomcat-6.0.35\temp\tc-config8509448930662041153.xml'.
Log file: 'C:\terracotta\client-logs\terracotta-client.log'.
Connection successfully established to server at 127.0.0.1:9510

Test it

Tomcat comes by default with a web application called examples, we will use it till the end of this tutorial to test our setup

point your browser to http://localhost:8080/examples/ and http://localhost:8081/examples/ and make sure both servers are responding.

open your Terracotta Development Console and make sure that both clients (the tomcat server) are present as below.

also note that the sessions button is now active.

Next - Load balancing with Apache Web Server

Terracotta and Tomcat Clustering - Page 1

In this tutorial we will learn a lot of stuff :)

How to install Terracotta in an Active/Passive HA (High Availability) model.
How to configure Tomcat 6.0 to store/retrieve sessions on Terracotta.
How to install and configure Apache Web Server configure it to relay requests to Tomcat.

Why?

i can do session replication using Tomcat built-in clustering module, so why do i use Terracotta?

well, the Tomcat clustering works just perfect with major four concerns:

Clients' sessions data are part of the Tomcat JVM instance.
There is extra work for Tomcat to make sure that the sessions and its data are replicated among the cluster
not suitable for larger cluster because of using multicasting, imagine the network traffic generated by eight tomcat nodes replicating their sessions.
You can't store an object which is not serializable.

So, moving this responsibility to Terracotta would help eliminating those four concerns.

Our Final architecture

it is really simple:

Both Tomcats store/retrieve sessions on the active Terracotta server.
Apache web server forwards requests to both Tomcats.
If the active Terracotta fails the passive Terracotta will be the active.

Let's Digg in

First things first! we need:

My five servers built on a single windows 7 box as a POC, you should have no troubles with running the same example on Linux, only minor chages are needed.

Install and Configure Terracotta

open a terminal and change the directory to the place where you downloaded terracotta-3.7.0-installer.jar

java -jar terracotta-3.7.0-installer.jar

you will see a setup wizard, it is straight forward, install Terracotta to the directory of your choice, from now on i will refer to that directory as ${TC_HOME}
now, go to ${TC_HOME} and create a new directory called 'config', this directory will contain our Terracotta configuration file that will set up the Terracotta cluster.
create a new file inside the config directory called tc-config.xml with the following content:

<tc-config xmlns="http://www.terracotta.org/config">
 <servers secure="false" xmlns:tc="http://www.terracotta.org/config" xmlns:con="http://www.terracotta.org/config" xmlns="">
  <server bind="0.0.0.0" host="127.0.0.1" name="node1">
   <data>C:\terracotta\server-data</data>
   <logs>C:\terracotta\server-logs</logs>
   <statistics>C:\terracotta\server-statistics</statistics>
   <dso-port bind="0.0.0.0">9510</dso-port>
   <jmx-port bind="0.0.0.0">9520</jmx-port>
   <l2-group-port bind="0.0.0.0">9530</l2-group-port>
   <data-backup>C:\terracotta\data-backup</data-backup>
   <index>C:\terracotta\server-data\index</index>
   <dso>
    <client-reconnect-window>120</client-reconnect-window>
    <persistence>
     <mode>temporary-swap-only</mode>
    </persistence>
    <garbage-collection>
     <enabled>true</enabled>
     <verbose>false</verbose>
     <interval>3600</interval>
    </garbage-collection>
   </dso>
  </server>
  <server bind="0.0.0.0" host="127.0.0.1" name="node2">
   <data>C:\terracotta\server-data2</data>
   <logs>C:\terracotta\server-logs2</logs>
   <statistics>C:\terracotta\server-statistics2</statistics>
   <dso-port bind="0.0.0.0">9511</dso-port>
   <jmx-port bind="0.0.0.0">9521</jmx-port>
   <l2-group-port bind="0.0.0.0">9531</l2-group-port>
   <data-backup>C:\terracotta\data-backup2</data-backup>
   <index>C:\terracotta\server-data\index2</index>
   <dso>
    <client-reconnect-window>120</client-reconnect-window>
    <persistence>
     <mode>temporary-swap-only</mode>
    </persistence>
    <garbage-collection>
     <enabled>true</enabled>
     <verbose>false</verbose>
     <interval>3600</interval>
    </garbage-collection>
   </dso>
  </server>
  <mirror-groups>
   <mirror-group>
    <members>
     <member>node1</member>
     <member>node2</member>
    </members>
   </mirror-group>
  </mirror-groups>
  <ha>
   <mode>networked-active-passive</mode>
   <networked-active-passive>
    <election-time>5</election-time>
   </networked-active-passive>
  </ha>
  <update-check>
   <enabled>true</enabled>
   <period-days>7</period-days>
  </update-check>
 </servers>
 <system xmlns:tc="http://www.terracotta.org/config" xmlns:con="http://www.terracotta.org/config" xmlns="">
  <configuration-model>production</configuration-model>
 </system>

 <clients xmlns:tc="http://www.terracotta.org/config" xmlns:con="http://www.terracotta.org/config" xmlns="">
  <logs>%(user.home)/terracotta/client-logs</logs>
  <modules>
   <module name="terracotta-toolkit-1.6" group-id="org.terracotta.toolkit"/>
  </modules>
 </clients>
</tc-config>

the important thing to note here is the servers node

for each Terracotta server that will run in the cluster you need to define a server node.
the server tag has three attributes:

bind: the default bind address which Terracotta listen to.
host: the IP address that will be used to connect to the Terracotta server.
name: the name of this node

data: is where this server stores its data (make it unique for each server if you are running the cluster on one box)
logs: is where the server store its logs (make it unique for each server if you are running the cluster on one box)
statistics: is where the server store its statistics (make it unique for each server if you are running the cluster on one box)
port configurations: JMX, DSO and group ports (make it unique for each server if you are running the cluster on one box)
data-backup: is where the server store its data backups (make it unique for each server if you are running the cluster on one box)
index: is where the server store its index files (make it unique for each server if you are running the cluster on one box)
dso: is the Distributed Shared Objects specific options.

we have now defined two servers, we need to define how they would work.

mirror-group: a tag to define Terracotta groups

members: a tag to add a server to a group using its name

ha: is a tag to define the High Availability options of the group

mode: networked-active-passive it means that the communications between the servers will relay on networking.
election-time: the Terracotta server would wait for that time to decide if it should start as an Active or passive.

let's start out cluster an make sure everything is OK.

open three consoles and navigate to ${TC_HOME}\bin on the three of them

Start node1

start-tc-server.bat -f ..\config\tc-config.xml -n node1

you should see

Becoming State[ ACTIVE-COORDINATOR ]
Terracotta Server instance has started up as ACTIVE node on 0.0.0.0:9510 successfully, and is now ready for work.

Start node2

start-tc-server.bat -f ..\config\tc-config.xml -n node2