• IS organizations trying to understand Performance and Capacity Planning issues related to their mail configuration, mail applications (or mail-enabled applications like workflow apps), mail client, etc. MailSizr's values also makes a compelling argument to ISPs who want to understand how many mail users they can support successfully. MailSizr can be used to conduct performance and stress tests against various configurations. In addition, it can also document results for such tests automatically and thus helps in the analysis to arrive at the most optimal configuration. By providing the ability to document the configuration files, MailSizr provides an ability to tune these system deployments. MailSizr can also be used to measure the reliability, availability and security of applications under load.
• Application developers who develop/debug mail clients, mail-enabled applications or server applications where performance and scalability is important.

MailSizr solves many such problems to give you a more realistic result set by mimicing the activity of users and applications more closely than any other. MailSizr, combined with Technovations NetCorder(tm) can make scripting and benchmarking much easier than ever before. Here are some unique features of MailSizr

• Point and Click interfaces make MailSizr easy to use. Script files have simple and intuitive verbs and definitions. Technovations NetCorder(tm) makes it easy to develop custom scripts by recording transactions from your mail client to generate MailSizr scripts.
• Allows the user to define Test Acceptance Criteria before starting User Simulation.
• Gives logging facilities which allow developers to debug application programs.
• Gives user the ability to define Error Handling which can control the course of the experiment.
• Allows user to automatically desynchronize themselves for more realistic simulations.
• Allows users to define timeouts for user operations.
• Allows user to collect abstract timing info for calls or can scope into socket level timing information
• Allows simulation of any Mail Client using NetCorder recording technology.
• Graphically and Numerically Presents Run-Time performance and status views during test.
• Allows pages (from applications) during test to be logged to HTML files based on user's preference. These files can be used to debug your applications.
• Presents performance results in a easy to understand concise form.
• Uses Username and Passwords and other credentials based authentication mechanisms
• Logs status codes received from the Messaging server for each transaction within a mail operation so you can scope into problems easily.
• Has an automatic Report Writer and Configuration Manager which produces a HTML report for each test.
• Allows simulation of multiple scripts in the context of a single test.
• Allows the test user to define custom collections to analyze test activity.
• Has constructs to create normal or bursty tests.
• Has an automatic Report Writer and Configuration Manager that produces an HTML report for each test.
• MailSizr can be customized for your individual testing scenario using many features mentioned above and more.

Here are the set of command verbs and the syntax for the commands.

• POP : The POP command is used to mimic a typical mail retrieval operation using the POP3 protocol. User Credentials supplied in the setup screen will be used to authenticate each user's retrieval operation.

Syntax: POP Port# POPServer
where POPServer is the hostname of the POP server (which can also be an IP address) and Port# is the Port Number on which the POP server is listening (this is almost always port 110).
Example: POP 110 tech10.technovations.com
 
• PND : The PND (Pop and Delete) command is used to not only retrive messages from the server, but also to delete messages that have been retrieved.

Syntax: PND Port# POPServer
where POPServer is the hostname of the POP server (which can also be an IP address) and Port# is the Port Number on which the POP server is listening (this is almost always port 110).
Example: PND 110 tech10.technovations.com
 
• SMTP: This command is used to exercise SMTP functionality to send mail to the server.

Syntax: SMTP Port# SMTPServer <file>BODYTEXT <file>SUBJECTTEXT toAddress
where SMTPServer is hostname of the SMTP server (which can also be an IP address),
Port# is the Port Number on which the SMTP server is listening for requests (almost always port 25),
the <file> directive specifies the file whose contents will be sent as part of the the body or the subject of the mail message (the first directive sends body text and the second specifies the text of the subject field for the mail message),
and the toAddress is the address of the person or group to whom the mail message is intended
Example: SMTP 25 tech10.technovations.com <file>memo.txt JShmoe@test.com
In the above example, the file memo.txt will be sent to the user JShmoe@test.com through the SMTP server tech10.technovations.com listening on Port 25.
 
• PAUSE : This command is used to create event lists using the script file. It takes at least one argument which specifies the number of seconds to pause. It could optionally be overloaded with a second argument which is used to create randomness in the PAUSE times for different threads and different instances of the PAUSE command within the same script.

 

 
 
 

Syntax : PAUSE WaitLowerLimit(seconds) [WaitUpperLimit(seconds)]

Examples :

PAUSE 3 //will make the execution thread pause for 3 seconds

PAUSE 5 10 // will cause a random wait anywhere between 5 and 10 seconds

Tip: The PAUSE command could optionally be used to customize desynchronization of script activity.

• REWIND : This command allows a pattern in the scriptfile to be iterated a number of times.

Syntax : REWIND #Iterations
Example : REWIND 300
This example specifies that all lines in the script file which occur after the REWIND command will be iterated 300 times.
 
• SEARCH : The SEARCH command is used to find a string from the content screen returned after a successful POP3 request. If the string is found in the screen buffer, it results in a FOUND event for that specific thread. Else, the NOT_FOUND event is flagged.

Syntax : Search <search string>
Example : Search This is a test
In this example, the screen buffer will be tested for the presence of the string "This is a test". Remember that the search is case-sensitive. If the string is present in the buffer, a FOUND event will result for the user. If not, a NOTFOUND event will result. These events can be used to force execution or ignore execution of Run-time management commands. EXIT and TERMINATE are examples of such commands.
• EXIT : This command is an instruction for the test kernel to terminate activity for the User. This termination is usually instantiated using the FOUND/NOTFOUND events. Note that the EXIT command can also be invoked unconditionally. Also note that you should not have an EXIT command at the end of the script file if you also have a REWIND command in the script. If you do, the EXIT command will prevent the script from executing multiple iterations.

Syntax: EXIT <IFFOUND|IFNOTFOUND>
Examples:
• EXIT IFFOUND
• EXIT IFNOTFOUND
• EXIT
Refer to the SEARCH command to understand more about how this command works.
• TERMINATE: This command is used to terminate the experiment, usually based on an event. The FOUND/NOTFOUND events are used to invoke this command.

Syntax: TERMINATE <IFFOUND|IFNOTFOUND>
Examples:
• TERMINATE IFFOUND
• TERMINATE IFNOTFOUND
• TERMINATE
Note that the TERMINATE command can also be invoked unconditionally to stop and experiment when one of the simulated users reaches a specific simulation point in the script. Also note that you should not have an TERMINATE command at the end of the script file if you also have a REWIND command in the script. If you do, the TERMINATE command will prevent the script from executing multiple iterations.

Refer to the SEARCH command to understand more about how this command works.

• RUNFOR: This command is used to specify the length of time for each run.

Syntax: RUNFOR #seconds
Example: RUNFOR 3600
The above example command would halt a test after an hour f testing. The RUNFOR command overrides the REWIND command.

Besides the basic commands described above, there are a set of advanced command verbs that MailSizr can also aceept. The advanced commands are used to mimic the conversation between the Messaging Client and the Messaging Server _EXACTLY_. These commands are typically created using Technovations NetCorder(tm), which is an advanced recording product to capture Socket Transactions. Using the advanced verbs, a user can simulate IMAP and NNTP requests.

• RPP3: Recorded POP.

Syntax: RPP3 Port# POPHost <file>COMMANDFILE
Example: RPP3 110 tech10@technovations.com <file>helloworld0
The above example makes a connection to the POP3 server on tech10@technovations.com listening on port 110 and translates all lines in the file helloworld0 into transactional commands to the server to retrieve mail messages.

For those not familiar with the POP3 protocol RFC#1725, you don't need to learn the RFC! These transactions are recorded by Technovations NetCorder and the scriptfile is automatically generated. For those who may be developing applications conforming to the RFC, we have defined a very simple transactional language which can be used to prototype your application _BEFORE_ you complete implementing it. Contact Technovations for more details.

• RSMTP: Recorded SMTP

Syntax: RSMTP Port# SMTPHost <file>COMMANDFILE
Example: RSMTP 25 tech10@technovations.com <file>helloworld1
The above example makes a connection to the SMTP server on tech10@technovations.com listening on port 25 and translates all lines in the file helloworld1 into transactional commands to the server to send the mail message encapsulated in the script itself.

For those not familiar with the SMTP protocol RFC#821, you don't need to understand the RFC! These transactions are recorded by Technovations NetCorder and the scriptfile is automatically generated. For those who may be developing applications conforming to the RFC, we have defined a very simple transactional language which can be used to prototype your application _BEFORE_ you complete implementing it. Contact Technovations for more details.

• RIMAP: Recorded IMAP

Syntax: RIMAP Port# IMAPHost <file>COMMANDFILE
Example: RIMAP 143 tech10@technovations.com <file>helloworld2
The above example makes a connection to the IMAP server on tech10@technovations.com listening on port 143 and translates all lines in the file helloworld2 into transactional commands to the server to retrieve mail messages.

For those not familiar with the IMAP4 protocol RFC#1730, you don't need to understand the RFC! These transactions are recorded by Technovations NetCorder and the scriptfile is automatically generated. For those who may be developing applications conforming to the RFC, we have defined a very simple transactional language which can be used to prototype your application _BEFORE_ you complete implementing it. Contact Technovations for more details.

• RNNTP: Recorded NNTP

Syntax: RNNTP Port# NNTPHost <file>COMMANDFILE
Example: RNNTP 119 tech10@technovations.com <file>helloworld3
The above example makes a connection to the NNTP server on tech10@technovations.com listening on port 119 and translates all lines in the file helloworld3 into transactional commands to the server to retrieve news messages. Note that this is just an example and you could alo POST messages, retrieve group information, etc using the same technique.

For those not familiar with the NNTP protocol RFC#977, you don't need to understand the RFC! These transactions are recorded by Technovations NetCorder and the scriptfile is automatically generated. For those who may be developing applications conforming to the RFC, we have defined a very simple transactional language which can be used to prototype your application _BEFORE_ you complete implementing it. Contact Technovations for more details.

• RPAUSE(msecs): Recorded Think time Pause

Syntax: RPAUSE(msecs) #msecs
Example: RPAUSE(msecs) 1232
The above example pauses the user for 1.232 seconds (1232 milliseconds). This command is typically generated automatically using Technovations NetCorder, which monitors think time between user requests.

Manipulating and using the SETUP Applet

The SETUP applet allows you to set up and customize the test per your specifications. The SETUP applet has several tab panels that allow such customization and this subsection will explain these in more detail.

This screen allows the user to specify the script file for this test. Remember that this script file can be either constructed/edited manually or can be generated automatically using Technovations NetCorder.

Choosing how many users to simulate during test

The total number of users (Simultaneous Mail Clients) that need to be simulated for the test that we are about to conduct can be specified by either entering this in the "Concurrent Users to Simulate" field or by using the dropdown button to pick from preset choices.

Choosing how to handle errors

The SETUP screen allows the user to specify actions on errors generated during test.
By default, all errors are ignored (but reported) and tests continue in spite of them. This setting can be used if you can allow a margin for errors to hapen during test.
The test can be configured where test users are dropped as they encounter errors. This setting may be useful to figure out the optimal number of users that can be fielded by the server or application.
The test can be configured to terminate the test when any user encounters errors. This setting is frequently used during reliability and regression testing of applications.
There is also a mode where you could choose to retry operations whenever errors occur. This setting mimics the way some real applications work.

Assigning credentials to simulated users

For each user to execute the assigned script, the user may need to log in to the server. These User names and passwords are input through the credentals file associated with MailSizr. This can be initiated using the "Edit Credentials" button. Picture below shows the format of the credentials file.

The second field is the name of the user and the third field is the password associated with the username. In this case, we have simplified setup by simulating the same user multiple times. Note that many applications may not allow the same user to be logged in twice. If this is the case, then the test user needs to register unique user credentials and supply them in this file.

User credentials can also be entered/edited/imported using the SIZR.CRD file as opposed to using the above Graphical interface. The SIZR.CRD is located in the MailSizr program directory.

The credentials file is also used to store cookies associated with a certain user. In cases where cookies are encountered by MailSizr. the value of the cookie is written back into this file at the end of the test. If you would like to conduct tests by using pre generated cookies, you can input these values into the credentials file.

Now back to the details of the SETUP screen. The SETUP screen allows the test user to start a new test. Before we start a new test, let us look at some of the other settings that can be manipulated as part of test SETUP.

The "Screen/Result Logging" screen is used to

• Specify the log file for the test.
• Specify the level of instrumentation needed from MailSizr.
• Specify if data should be read back from the System Under Test and how it should be presented

Result Logs: Specifying the Log file for test
 

Result Logs contains data for each operation performed by MailSizr during test. A section of the log file which contains result data (that we decided to log in Abstract form) is shown below.

Time:125770, USER#:5, RT:4175ms, OP:SMTP
Time:126539, USER#:2, RT:15188ms, OP:PAU
Time:126821, USER#:2, RT:282ms, OP:SMTP
Time:129547, USER#:10, RT:25379ms, OP:PAU
Time:132720, USER#:1, RT:426ms, OP:POP3-read
Time:133997, USER#:3, RT:838ms, OP:POP3-read
Time:135919, USER#:6, RT:919ms, OP:POP3-read
Time:129794, USER#:10, RT:247ms, OP:SMTP

The Time: field shows the relative time in Milliseconds from test start time. The User#: field is the identity of the simulated user for the experiment. The RT: field shows the Round-Trip Response-Time for this operation, and lastly, the OP: field shows the operation performed.

Let us see what would happen if we rerun the experiment with Result Logging to get "Detailed(Socket Level Timing)" info.

Time:82288, USER#:31, OP:POP3-read, ToCreateSocket:9ms, ToConnect:4ms, ToSendCmd:7ms, ToReadFirstByte:457ms, ToReadRestOfBytes:104ms
Time:82288, USER#:31, RT:581ms, OP:POP3-read
Time:82572, USER#:16, RT:13180ms, OP:PAU
Time:82623, USER#:40, OP:SMTP, ToCreateSocket:8ms, ToConnect:3ms, ToSendCmd:21388ms, ToReadFirstByte:8ms, ToReadRestOfBytes:283ms
Time:82623, USER#:40, RT:21690ms, OP:SMTP
Time:82739, USER#:7, OP:SMTP, ToCreateSocket:44ms, ToConnect:17ms, ToSendCmd:21289ms, ToReadFirstByte:72ms, ToReadRestOfBytes:292ms
Time:82739, USER#:7, RT:21714ms, OP:SMTP

Instrumenting additional detail from MailSizr Runs.

In addition to the abstract log information, the detailed timing information about when certain events happened for these requests is presented.
ToCreateSocket: is the time required to resolve the Server name and create a socket to communicate with it.
ToConnect: is the time to connect to the bound port.
ToSendCommand: is the time to send the Messaging Command to the server
ToReadFirstByte: is the time taken until the first byte of data is read from the socket.
ToReadRestOfBytes: is the time to read all the remaining bytes on the socket (for this request).

You will notice that adding the 5 quantities above is what results in the overall ResponseTime for the Mail operation. All times are in units of Milliseconds.

Screen Logging - Specifying recording of data read back from the System Under Test.

Screen logging is the log that keeps track of the data read by the "Test Mail Client". You can optionally decide not to log this data to avoid additional IO on the load generator machine, but this could come in handy for debugging applications under load.
If you conducted a certain test where there were failures only with a load of 200 simultaneous users. How would you debug this problem ? You would enable screen logging to understand what happened during test execution and analyze the screen logs to understand the nature of the problem. what would make it more convenient is to have the ability to log screens (mail data) for each simulated user into a seperate file. This would make it easier to read the screen logs. Also, logging into seperate files provides the ability to compare screen output between different users.

The Screen page size is the size of the buffer that is used to collect data that is read from the server. This should be larger than the largest page that will be read through this experiment, but also should not be too big because this consume valuable run-time memory resources.

The "Miscellaneous Settings" panel allows you to set specific characteristics, most of which define how this experiment is run.

The home Domain Name is one of the settings that you can define. This is used to fill the from field of the user on a SMTP command. Note that while it is not essential to have this field populated in most cases with the correct domain name, there may be cases when this would be required when email filtering by domain is enabled as a Mail Router Agent.
 
 

Specifying Time-outs for Operations.

The Operation Time-out parameter defines the length of time that a request will wait to be serviced when inactivity occurs. If the request takes longer than this specified time, the request will terminate and an error will result. This error can be caught and thrown if desired or can terminate the user or experiment, as defined in the "Error Handling" panel. Note that if you anticipate the operation to take a long period of time, you would need to set this parameter to be high enough value. Note that you can set the timeout value to "0" and override this function.
 

Customizing the creation of users for test

It may be important in certain test scenarios to replicate the pattern in which users get created during test. MailSizr gives you the ability to ramp up users by allowing you to specify the Minimum frequency and Maximum frequency of creation.

For example, if you wanted users to be created between 5 seconds and 20 seconds, you would specify 5 seconds in the "Minimum Frequency Text box" and 20 Seconds in the "Maximum Frequency Text box". Note that the users will still be created at random.

If you want to create users at a constant rate (rather than being random as in the above example), you can do this by keeping the values in the  "Minimum Frequency Text box" and " "Maximum Frequency Text box" the same.
 

User De-synchronization:

This panel also allows user to be automatically desynchronized so that not all users are generating requests at the same time. This is recommended to mimic real world behavior. A user can also defeat automatic de-synchronization and put de-synchronization commands manually in the script. file. This is done by having PAUSE commands at the top of the script file. As an example a "PAUSE 0 300" command at the top of the script file starts user activity randomly between 0 and 5 minutes.

Automatic de-synchronization can be disabled to enable very quick user RampUp. This can be useful to try quick tests with large number of users without necessarily waiting for MailSizr to RampUp over an extended period of time.
 
 

Ignoring Pause directives in script:

This is a shortcut to quickly run tests without pauses. This could be desirable in many scenarios, especially during debug of scripts or application. In such cases the Sizr is usually in "single user mode" - number of users simulated = 1.
There are other cases where this is used to simulate very large loads by ignoring all the "Pause" events in the script.
 

Defining when to start simulating test activity:

By default, each user after becoming active will start simulating test activity as defined by the basic test script or the test script assigned to that user. This is the natural and the best way of conducting tests because this is exactly the pattern that happens in real life.

However, there are sometimes specific needs to understand the impact of a large number of users hitting a server at the same time. Such tests can be used to determine the health of the server under extremely bursty workloads. MailSizr can enable these tests by proving the test user the ability to "start simulating user activity only after all users are ramped up". In this case, MailSizr will wait for all users to become active and then will hit the server with an active workload.

This "Automatic De-synchronization" feature can also be used creatively in cases where such bursty tests are being designed. Since the bursty tests by definition defeat user de-synchronization, the "Automatic de-synchronization" option can be turned off for such tests to enable users to become active as fast as the workload generator can ramp up the users.
 

Defining when to start simulating test activity:

By default, each user after becoming active will start simulating test activity as defined by the basic test script or the test script assigned to that user. This is the natural and the best way of conducting tests because this is exactly the pattern that happens in real life.

However, there are sometimes specific needs to understand the impact of a large number of users hitting a server at the same time. Such tests can be used to determine the health of the server under extremely bursty workloads. MailSizr can enable these tests by proving the test user the ability to "start simulating user activity only after all users are ramped up". In this case, MailSizr will wait for all users to become active and then will hit the server with an active workload.

This "Automatic Desynchronization" feature can also be used creatively in cases where such bursty tests are being designed. Since the bursty tests by definition defeat user desynchronization, the "Automatic desynchronization" option can be turned off for such tests to enable users to become active as fast as the workload generator can ramp up the users.

By default, the collections are started when the test starts - user hits the "start test" button - and ends when test activity ends.

Collections can also be started "after user Rampup". This method analyzes the data that is collected only after all users have been activated. The time between the first user becoming active to the last user becoming active is the total RampUp time.

In some cases, you may want to enable collections after a specific period of time. You can do this by using the third option and by specifying the time. Note that the time is specified in Milliseconds.

As in the case where the test user can define the start of collections, the time when the collections can be ended can also be customized using this panel. Collections can be started "before user RampDown". This method analyzes the data that is collected until the time when the first user exits from the experiment. The time between when the first user completes test activity to the time when the last user completes such activity is defined as RampDown time.

In some cases, you may want to enable collections until a specific period of time. You can do this by using the third option and by specifying the time when you want to end collections. Note that the time is specified in Milliseconds.

Irrespective of how you define the collection/marshalling of test statistics, the test harness will still collect the data for all operations and record them in the log. So, the customization of test statistics can be used as a way to use MailSizr to "pre-analyze test data" without sacrificing the ability to do post-test analysis.

Once you have completed the required setups in this screen, you can proceed to execute the test by depressing the Start Test button (in the Vital Settings/Controls panel). This will start the experiment and pop the display to monitor run-time events.
 
 
 

Advanced Test Settings

 

 
 

Assigning individual scripts to users.

During some tests, there comes a need to simulate users who do very different things. For example, when you are trying to mimic some users whose jobs are technical in nature and others whose jobs are administrative in nature, etc., it can be quite painful to come up with a single script which models the collective behavior of these users. One solution is to run different test instances for each user type and then merge the result sets. But even this approach can be quite tedious, error prone and requires excessive manual coordination. Technovations has invented a new and simple paradigm so you do not have to endure such pains any more.

MailSizr gives you the ability to simulate such users by eliminating the need for modeling/result set merging. This is enabled by simulating multiple scripts in the same run. There are 3 approaches to accomplishing this:

Assigning by Pattern:

Here is how this is done:

• First, the test user checks the box that indicates that the user intends to "Assign individual scripts to test users".
• Second, defining a pattern (in the "By Pattern Text box") for the files that contain the directives that need to be simulated for the users. For example, if you have a pattern like "mult_" as shown above, this pattern will be used by MailSizr to read individual files. If there is a file called mult_1.scr, this will be assigned to User#1. If there is a file called mult_37.scr, this script file will be assigned for User#37, and so on. So, in a 1000 user simulation scenario, it is possible to simulate the 1000 users each with its own script.

Let us assume that we need to simulate 25 users of a certain type and 975 users of a certain different type. In this case, we create 25 files as described above (say mult_1.scr through mult_25.scr with directives for User#1 through User#25 respectively). The basic script file that will be used for the test will consist of directive for the rest of the 1000 users (975).

MailSizr will specifically search for the mult_*.scr pattern while assigning the individual scripts to users. When it cannot find a file that matches the specific pattern, it will assign the basic script pointed to by the "script file" field to that user. For example, if mult_26.scr does not exist (according to our example, it will not be), then the basic script is assigned to User#26.

Assigning by File

It is also possible to assign explicity using a text file. This is accomplished by specifying the bonding between a script and a simulated user. Picture below shows a file which defines such bonding.

Note that if a bond has not explicitly been defined, the test will automatically allocate the "basic test script" (defined in the Control Panel Applet of Setup) for this particular user.

Assigning Scripts Randomly

In some cases, it may be desirable to allocate script users randomly as the user is being created. This can be accomplished by defining a text file that contains a set of scripts that will be picked up randomly by each user on creation. Here is a illustration of such a file:

Note that the probability of choosing a specific script is defined implicitly by the number of occurrences of such a script in the container file.
 
 

Test Stop Modes

There may be several reasons why you may want to stop a test, ranging from "Satisfactory collection of data" to "Test needs more refinements". In any of these cases, there are 2 ways to stop the tests. The "Soft Termination" mode issues a signal to all users to terminate tests after completing the current task at hand. The "Hard Termination" option specifies that the test should be terminated immediately and all activity should be truncated to instantly "obey" the termination directive.
 
 

Sending Mail to a List

When using the SMTP command to send mail, recipients can be specified as part of the command (as described in the script syntax section of this document). While this can be very useful, a even more useful paradigm would be to distribute the messages across a larger set of users, and equally importantly, to control the distribution statistically, but with a very simple interface. This can be accomplished using the MAILLIST file. The maillist file has a set of user entries which the SMTP command can pick from when it is instructed to pick users at RANDOM.
 
 

Creating very large Collection of Users (Tens of Thousands of Users)

Would you like to create more than 2000 users ?
Would you like to create users from multiple distributed test locations ?
Would you like to manage tests easily (from a setup and go perspective) ?
Would you like result sets merged automatically into one comprehensive report ?

If you answered yes to one or more of these questions, you need the power of Technovations Load Director and Load Module logic to do this. Use of Load Director and Load Modules are part of MailSizr enterprise edition. The use of these are also part of the MailSizr training sessions.
 
 

The next screen-shot shows the MailSizr run-time monitor. On the left-hand side of the screen, you see a graphical picture of the throughput through the experiment (shown in solid blue) and an average of the throughput (in dotted blue). Throughput is expressed in Operations/Second and is graphed over time (abcissa or x-axis). At the bottom of the monitor screen is a test state indicator and a counter to show the number of Socket and Connection errors. These are errors that happen over the socket and can happen due to the server's inability to field requests or because of the inability of the workload generator machine to generate enough socket connections. In a Microsoft Windows 95 system, there are restrictions on the number of concurrent socket connections that can be active on the system. Which is why we recommend use of a Windows NT workstation or server as the workload generator.

The run-time monitor also gives the user controls which can be used to PAUSE TEST or STOP TESTs. If a test is PAUSEd while executing, it sends a signal to all simulated users to stop after completing their current request. Just remember that if you have a lot of simulated users and you have heavy load on the server, the PAUSE may not happen right away.

The STOP TEST command button stops the test in the middle of test execution. Even if you decide to stop tests by using the STOP TEST button, you will still be able to get detailed statistics up until the time that you stopped the test. There are 2 STOP TEST modes.

• Soft Termination : This is a signal which instructs all simulated users to stop after the current command. Again, Just remember that if you have a lot of simulated users and you have heavy load on the server, the STOP may not happen right away.
• Hard Termination : This is a hard stop at the time the user decides to terminate the test. The simulated users are all terminated immediately.

The run-time screen has updates the display every 5 seconds. This display interval can be changed for your convenience.

Using the Run-Time Monitor Applet

This section will demonstrate the use of the Run-Time Monitor and the facilities that it provides for analysis and control.

Test Warmup Time is the time to setup the experiment. This starts from the time the user hit the "Start Test" button to the time when the first user is generated.
Test Rampup Time is the time to start up all the users for the test. This is measured from the time when the first user starts up to the time when the last user is started up.
Test Time is the time from when the last user is started up to the time when the first user completes or drops out of the test.
Rampdown Time is the exact opposite to Rampup Time. This is the time from when the first user drops off or terminates to the time when the last user drops off or terminates.

In the viewing panel, there is a indicator which show the total number of Active users Vs. Specified Users. This shows you the progress through test Rampup, but there is a more important use for this. Consider te following example: You want to find out how many users you can put on your system without generating errors, etc. Let us for the moment think that this magic number is 632. Rather than conducting many tests to figure out this optimal number of users for your system, MailSizr allows you to come up with the answer with a single experiment. How ? Just specify the number of user in the SETUP screen to a high number. In our case, let us set it to 800. Set the "Error Handling" in the SETUP screen to "Terminiate User on Errors". Start Test. As the test progresses, users who encounter errors are dropped off automatically. 632 users will be running on the system without errors and will be visible through the Run-time screen. And this is how you will know that your system can support 632 users.

Whenever errors occur during test, the "Output from Test" button pops up to remind the user to acknowledge these errors before proceeding further. Notice that as we were running the experiment, we encountered several errors - shown at the bottom of the run-time screen in the "Socket/Conn Errors" category. Depressing the "Output from Test" button will show these errors.

When the test completes and any errors returned from the test are acknowledged, the user can then proceed to "View Test Results".

The stats show the total number of each type of operation played against the system under test. The "Average Rate" is the Throughput metric which presents the total number of operations completed per unit time on the system under test. "Average Latency" is the metric which reveals Response Time details for different operations conducted against the System Under Test. This is an average number (in milliseconds). Other Numbers presented on this screen are the Standard deviation of response time about the average Response Time, which shows the variability of individual response times from the average. There are 3 discrete points on the Response Time graph that are captured and presented on this screen - The MIN, MAX and LAST values. These, along with the Standard Deviation can be used to get a quick synopsis of the test before going into more details.

The "ALL" category folds the metrics for the individual operations into an overall test average. It is possible that you may need more detailed statistical information than what is presented in this screen. For this exact reason, a complete audit trail of Response times and timestamps are maintained for each individual operation conducted through the test. This log is in Comma Seperated Variable (spreadsheet-friendly CSV) format which can be manipulated further if necessary.

From the Statistics screen, you will be given the option of documenting this test or to start a new one. If you choose to Start Another Test, you will be redirected to the Basic Setup screen. If you decide to document results, you will be directed to the screen which helps you define the elements in your test report and generate your test report.

Note that you also have the flexibility to also log the content generated by the server (during the test). You need to be aware that this can log a lot of data into the test report that may not exactly be relevant to analyze the report.

MailSizr provides a mechanism to automatically document all test parameters and values of importance. This is useful for analysis at a later point of time.

Conclusions:

MailSizr presents a powerful paradigm to setup, run, view and develop reports for experiments which simulate hundreds of users. It very easily adapts to your test plans to deliver data and results from your experiments which can be used for Capacity Planning, Performance Analysis and Tuning, Regression Testing and Patch Management. Also, you can debug applications against realistic user Workloads and configurations for Reliability and Availability

If you are developing a mission critical application, you can performance engineer it using MailSizr. Without it, you can only guess how good your application could be under load.

If you are deploying a mission critical application and tuning it to meet ever changing user workload needs, MailSizr can not only make your deployment more robust, but also optimize your deployment cycle by providing you with the right testing technology as you incorporate new patches and versions of Application software.L