I have used several tools to load test websites and webapps: ab, siege, httperf. They are very good tools but their are designed to test static URLs like:
http://mysite.com/somepath
http://mysite.com/someotherpath?parameter=value
and not the kind of URLs generated by Seaside.
Some of them can be given a list of URLs and will hammer your server with requests to each of those URLs. Others can also accept and resend cookies with each request. Some have a concept of a session (mostly with a cookie holding the session key value) that can be used to send a series of requests as part of the same session. Few of them can follow a redirect automatically. Well, isn’t a fault on the tools, but that they are created to test static web pages.
Of course there are commercial tools that (I think) can be scripted to simulate a real session by logging in to the webapp, navigating on it and putting values on forms and fields and running them thousand of times. I have never used them.
Other more recent tool is
Selenium, that build test cases by using a web browser. You start Selenium and it records all the actions you perform on a website/webapp and then can repeat them with just a click. Everything runs on the web browser. Also it is very slow when you want to repeat the test 1000 times. But it is great to test complex scenarios having several link and button clicks and form submissions.
We can use ab or httperf on the SeasideProxyTester applications. This can be useful even if they can’t navigate the application or “click” the links on the application. If you run a command like:
ab -c 10 -n 100 http://seaside.example.com/
it will create 10 concurrent connections, each of them requesting 100 times exactly the same URL. The problem with this kind of request is that you aren’t testing correctly your application. Here you are testing how many sessions can be created by Seaside under load. Why? As this URL hasn’t a _s parameter on it (the session key), each time that a request like this arrives to Seaside, a new session will be created for it. And then it is forgotten and never used again. Also, this command will hit a new Seaside image each time because as isn’t accepting the cookie, the webserver is doing a round-robin balancing for each request that doesn’t include the cookie.
With the following httperf command, things are a little better:
httperf –hog –session-cookie –server=seaside.example.com –wsess=10,100,1 –rate 2
This creates 10 threads at a rate of 2 per second, each of them will send 100 requests spaced out by 1 second. It also accepts and holds a cookie containing the session key for the session. This command is better in that the requests are more uniformly created over a period of time and not all of them at the start of the command, as the ab command does. The problem is that only can hold one cookie and the SeasideProxyTester is already using a cookie for storing the server that is handling the request. So the httperf command shown isn’t really sending requests as part of the same Seaside session but it guarantees that the same httperf thread always send its requests to the same Seaside image by sending the cookie value received on the first request. This is, as I said, a little better than ab, but not what we want.
The tool I will be using is
JMeter from the Apache Foundation. This tool can be scripted and made to follow links on the application being tested. Also, it can graph the results or save them to a file. It can handle cookies and check for strings on the response for testing purposes. Lets begin testing the SeasideProxyTester apps.
Download JMeter and unzip it. Change to the unzipped directory and run:
miguel@laptop:~$ cd jakarta-jmeter-2.3.4
miguel@laptop:~/jakarta-jmeter-2.3.4$ ./bin/jmeter
JMeter opens with two panels. To the left the tree of elements that you use to test your site. On the right, the panel where you configure the component selected on the left. Initially you have two components, a Test Plan and a Workbench. We will not use the Workbench.
Select the Test Plan and in the right panel configure:
Name: SeasideProxyTester
In JMeter the changes are saved when you select a different element. So if you select the Workbench, the Test Plan name will change.
Now, select the SeasideProxyTester test plan and right-click on it to open a pop up menu. Select Add -> Thread Group. Configure the following options:
Name: Users
Number of threads (users): 1
Ramp-Up Period (in seconds): 1
Loop counter: (leave unchecked the Forever checkbox): 1
The Thread Group simulates the users that will use the webapp. This means simulate 1 user that will be created on a timespan of 1 second. That is, after 1 second, you’ll have 1 user ready to do what the Test Plan specifies. For the moment we’ll work with one user. Also, the plan will be ran just one time (Loop counter). When the Test Plan is ready we will increment the number of users to load on the application.
Next, select the Users thread group and right-click Add -> Config element -> HTTP Cookie Manager. Nothing to configure for this element.
Select the Users thread group and right-click Add -> Config element -> HTTP Request Defaults. Configure:
Server Name or IP: seaside.example.com
Path: /
This will establish the default parameters for HTTP requests so that you don’t have to set them everywhere on your Test Plan (useful if you are doing a lot of different HTTP requests).
Again, right-click on the Users thread group and Add -> Listener -> View Results Tree. Nothing to configure.
This element will show the individual requests done by JMeter for you to review. DON’T include this element on test plans that make a lot of requests or your client machine will be using a lot of RAM just to update this element. We’ll use it only for testing that the plan is correctly configured. When doing the real load testing, we’ll remove it.
On the Users thread group do Add -> Listener -> Summary Report. Nothing to configure.
This element will show the number of requests done, the time to receive responses, errors and other useful data obtained when executing the Test Plan. This is the element that will tell us how many requests our application is capable of process.
Lets add the elements that the Test Plan will execute to test our web application. Select the Users thread group and Add -> Sampler -> HTTP Request. Configure:
Name: Index page
Uncheck: Redirect automatically.
Check: Follow redirects.
This establish that the Test Plan must make one request to the “/” of the seaside.example.com server (specified by the HTTP request defaults).
At this step you can already run the Test Plan. Either press Ctrl + R or in the menu Run -> Start. Now select the Result Tree.
You’ll see that JMeter has made two requests although the Test Plan only specifies one request. The first one is to:
GET http://seaside.example.com/
[no cookies]
Request Headers:
Connection: keep-alive
The full dialog for the first request is:
Thread Name: Users 1-1
Sample Start: 2009-09-30 11:15:41 CDT
Load time: 6
Latency: 6
Size in bytes: 0
Sample Count: 1
Error Count: 0
Response code: 302
Response message: Found
Response headers:
HTTP/1.1 302 Found
Server: KomHttpServer/7.1.2 (unix)
Location: http://seaside.example.com/?_s=A1oTuGM8OQmGMbZ5&_k=aI9HNGnu&_c
Date: Wed, 30 Sep 2009 11:15:41 GMT
Session: A1oTuGM8OQmGMbZ5
Set-Cookie: server=app9003; path=/
Content-type: text/html;charset=utf-8
Content-length: 0
HTTPSampleResult fields:
ContentType: text/html;charset=utf-8
DataEncoding: utf-8
Here we can see the redirect that the Seaside image that processed the request (at port 9003 as we can see from the Set-Cookie header) is responding. So JMeter does a second request:
GET http://seaside.example.com/?_s=A1oTuGM8OQmGMbZ5&_k=aI9HNGnu&_c
Cookie Data:
$Version=0; server=app9003; $Path=/
Request Headers:
Connection: keep-alive
Check that the cookie is being added to all the subsequent requests by JMeter (just as the web browser does). This will result in the request being forwarded to the same image that received the first request.
The full dialog for the second request is:
Thread Name: Users 1-1
Sample Start: 2009-09-30 11:15:41 CDT
Load time: 7
Latency: 7
Size in bytes: 1516
Sample Count: 1
Error Count: 0
Response code: 200
Response message: OK
Response headers:
HTTP/1.1 200 OK
Expires: Wed, 11 Jun 1980 12:00:00 GMT
Server: KomHttpServer/7.1.2 (unix)
Pragma: no-cache
Date: Wed, 30 Sep 2009 11:15:41 GMT
Session: A1oTuGM8OQmGMbZ5
Cache-Control: no-cache, must-revalidate
Content-type: text/html;charset=utf-8
Content-length: 1516
HTTPSampleResult fields:
ContentType: text/html;charset=utf-8
DataEncoding: utf-8
Until now everything is ok. Now lets add elements that will follow the links on the SeasideProxyTester summary page.
We have received a page with the HTML of the response. Inside this HTML there is a link that makes a new request in the same Seaside session (it sends the _s and _k parameters on it). We must find this link and make a request for it. After that, Seaside will respond a new HTML page with a new link on it, although with other values for _s and _k. We must again find the link and make a request for it. We must do this every time we need to follow the link. For this to work we need to add a post-processor (an element that triggers after each sampler execution, in this case, our HTTP requests). This post-processor will find the link and will save it to a variable in order to make the next request to the application. Then we will setup a loop controller to make the session counter increase in the SeasideProxyTester.
Select the Users thread Group and Add -> Post processor -> Regular Expression Extractor. Configure:
Uncheck: Body
Check: Body (unescaped)
Reference Name: newRequestURL
Regular Expression: \?(_s=[^&]+?)&(_k=[^&]+)&2
Template: ?$1$&$2$&2
Match No. (0 for Random): 1
Default Value: REGEX_FAILED
Then add a loop element to queue several requests one after other. Select the Users thread group and Add -> Logic Controller -> Loop Controller. Configure:
Loop count (leave Forever unchecked): 10
Select the Loop Controller and Add -> Sampler -> HTTP Request. Configure:
Name: Follow link
Path: /${newRequestURL}
Uncheck: Redirect automatically
Check: Follow redirects
That is it. Run it again and you’ll see that now there are eleven requests. One for the Index page, that is responsible of following the redirect and accepting the server cookie and ten for simulating clicking ten times the link “Make a new request inside this session”. Review the View Results Tree to see the individual requests.
What have we done. We have simulated a user accessing the seaside.example.com, receiving a redirect and a cookie as response. After following the redirect and using the cookie in any subsecuent request, it follows a link to simulate clicking it ten times. The View Results Tree page shows the responses and that the last HTML has a counter of 11 in the number of requests of the session.
So we are ready for the last step. Load testing the application.
First, remove the View Results Tree. We don’t want the JMeter client lost time trying to allocate memory just for storing and updating this element. Those CPU cycles and memory should be used to stress the application. Now update the Users thread group:
Number of threads: 100
Ramp-Up Period: 20
And update the Loop Controller:
Loop Count: 500
And run it again. This time, after 20 seconds you’ll have 100 users (threads) created, each of them running the Test Plan, that is, requesting the Index Page and then following 500 times the link.
This will put a real load on your server by doing 100 * 501 = 50100 requests.
Try changing the parameters involved:
- Number of images started (scripts/start_app.sh)
- Number of concurrent users created by JMeter
- Total time to create the users (so that the load is uniformly distributed over a bigger timespan)
- Total number of requests made by each user.
- Changing the number of processes running on the production server.
- Distributing the images (magma and seaside) on different machines.
- Putting the webserver on a different machine.
- Starting several instances of JMeter on several distinct machines.
Well, you get an idea. Only after varying the environment you’ll get the correct and optimal setup.
This test can be reproduced by everyone so that you can check your setup and compare it to others.
Finally read the
posts from Dale Henrichs
blog about Seaside scaling but using Gemstone/S 64.
The next post I will show the results on my machine both for the magma.example.com and for the seaside.example.com.
Now, select the SeasideProxyTester test plan and right-click on it to open a pop up menu. Select Add -> Thread Group. Configure the following options:
Name: Users
Number of threads (users): 1
Ramp-Up Period (in seconds): 1
Loop counter: (leave unchecked the Forever checkbox): 1
The Thread Group simulates the users that will use the webapp. This means simulate 1 user that will be created on a timespan of 1 second. That is, after 1 second, you’ll have 1 user ready to do what the Test Plan specifies. For the moment we’ll work with one user. Also, the plan will be ran just one time (Loop counter). When the Test Plan is ready we will increment the number of users to load on the application.
Next, select the Users thread group and right-click Add -> Config element -> HTTP Cookie Manager. Nothing to configure for this element.
Select the Users thread group and right-click Add -> Config element -> HTTP Request Defaults. Configure:
Server Name or IP: seaside.example.com
Path: /
This will establish the default parameters for HTTP requests so that you don’t have to set them everywhere on your Test Plan (useful if you are doing a lot of different HTTP requests).
Again, right-click on the Users thread group and Add -> Listener -> View Results Tree. Nothing to configure.
This element will show the individual requests done by JMeter for you to review. DON’T include this element on test plans that make a lot of requests or your client machine will be using a lot of RAM just to update this element. We’ll use it only for testing that the plan is correctly configured. When doing the real load testing, we’ll remove it.
On the Users thread group do Add -> Listener -> Summary Report. Nothing to configure.
This element will show the number of requests done, the time to receive responses, errors and other useful data obtained when executing the Test Plan. This is the element that will tell us how many requests our application is capable of process.
Lets add the elements that the Test Plan will execute to test our web application. Select the Users thread group and Add -> Sampler -> HTTP Request. Configure:
Name: Index page
Uncheck: Redirect automatically.
Check: Follow redirects.
This establish that the Test Plan must make one request to the “/” of the seaside.example.com server (specified by the HTTP request defaults).
At this step you can already run the Test Plan. Either press Ctrl + R or in the menu Run -> Start. Now select the Result Tree.
You’ll see that JMeter has made two requests although the Test Plan only specifies one request. The first one is to:
GET http://seaside.example.com/
[no cookies]
Request Headers:
Connection: keep-alive
The full dialog for the first request is:
Thread Name: Users 1-1
Sample Start: 2009-09-30 11:15:41 CDT
Load time: 6
Latency: 6
Size in bytes: 0
Sample Count: 1
Error Count: 0
Response code: 302
Response message: Found
Response headers:
HTTP/1.1 302 Found
Server: KomHttpServer/7.1.2 (unix)
Location: http://seaside.example.com/?_s=A1oTuGM8OQmGMbZ5&_k=aI9HNGnu&_c
Date: Wed, 30 Sep 2009 11:15:41 GMT
Session: A1oTuGM8OQmGMbZ5
Set-Cookie: server=app9003; path=/
Content-type: text/html;charset=utf-8
Content-length: 0
HTTPSampleResult fields:
ContentType: text/html;charset=utf-8
DataEncoding: utf-8
Here we can see the redirect that the Seaside image that processed the request (at port 9003 as we can see from the Set-Cookie header) is responding. So JMeter does a second request:
GET http://seaside.example.com/?_s=A1oTuGM8OQmGMbZ5&_k=aI9HNGnu&_c
Cookie Data:
$Version=0; server=app9003; $Path=/
Request Headers:
Connection: keep-alive
Check that the cookie is being added to all the subsequent requests by JMeter (just as the web browser does). This will result in the request being forwarded to the same image that received the first request.
The full dialog for the second request is:
Thread Name: Users 1-1
Sample Start: 2009-09-30 11:15:41 CDT
Load time: 7
Latency: 7
Size in bytes: 1516
Sample Count: 1
Error Count: 0
Response code: 200
Response message: OK
Response headers:
HTTP/1.1 200 OK
Expires: Wed, 11 Jun 1980 12:00:00 GMT
Server: KomHttpServer/7.1.2 (unix)
Pragma: no-cache
Date: Wed, 30 Sep 2009 11:15:41 GMT
Session: A1oTuGM8OQmGMbZ5
Cache-Control: no-cache, must-revalidate
Content-type: text/html;charset=utf-8
Content-length: 1516
HTTPSampleResult fields:
ContentType: text/html;charset=utf-8
DataEncoding: utf-8
Until now everything is ok. Now lets add elements that will follow the links on the SeasideProxyTester summary page.
We have received a page with the HTML of the response. Inside this HTML there is a link that makes a new request in the same Seaside session (it sends the _s and _k parameters on it). We must find this link and make a request for it. After that, Seaside will respond a new HTML page with a new link on it, although with other values for _s and _k. We must again find the link and make a request for it. We must do this every time we need to follow the link. For this to work we need to add a post-processor (an element that triggers after each sampler execution, in this case, our HTTP requests). This post-processor will find the link and will save it to a variable in order to make the next request to the application. Then we will setup a loop controller to make the session counter increase in the SeasideProxyTester.
Select the Users thread Group and Add -> Post processor -> Regular Expression Extractor. Configure:
Uncheck: Body
Check: Body (unescaped)
Reference Name: newRequestURL
Regular Expression: \?(_s=[^&]+?)&(_k=[^&]+)&2
Template: ?$1$&$2$&2
Match No. (0 for Random): 1
Default Value: REGEX_FAILED
Then add a loop element to queue several requests one after other. Select the Users thread group and Add -> Logic Controller -> Loop Controller. Configure:
Loop count (leave Forever unchecked): 10
Select the Loop Controller and Add -> Sampler -> HTTP Request. Configure:
Name: Follow link
Path: /${newRequestURL}
Uncheck: Redirect automatically
Check: Follow redirects
That is it. Run it again and you’ll see that now there are eleven requests. One for the Index page, that is responsible of following the redirect and accepting the server cookie and ten for simulating clicking ten times the link “Make a new request inside this session”. Review the View Results Tree to see the individual requests.
What have we done. We have simulated a user accessing the seaside.example.com, receiving a redirect and a cookie as response. After following the redirect and using the cookie in any subsecuent request, it follows a link to simulate clicking it ten times. The View Results Tree page shows the responses and that the last HTML has a counter of 11 in the number of requests of the session.
So we are ready for the last step. Load testing the application.
First, remove the View Results Tree. We don’t want the JMeter client lost time trying to allocate memory just for storing and updating this element. Those CPU cycles and memory should be used to stress the application. Now update the Users thread group:
Number of threads: 100
Ramp-Up Period: 20
And update the Loop Controller:
Loop Count: 500
And run it again. This time, after 20 seconds you’ll have 100 users (threads) created, each of them running the Test Plan, that is, requesting the Index Page and then following 500 times the link.
This will put a real load on your server by doing 100 * 501 = 50100 requests.
Try changing the parameters involved: