|
|
MailSizr solves many such problems to give you a more realistic result set by mimicing the activity of users and applications more closely than any other. MailSizr, combined with Technovations NetCorder(tm) can make scripting and benchmarking much easier than ever before. Here are some unique features of MailSizr
Here are the set of command verbs and the syntax for the commands.
Syntax : PAUSE WaitLowerLimit(seconds) [WaitUpperLimit(seconds)]
Examples :
PAUSE 3 //will make the execution thread pause for 3 seconds
PAUSE 5 10 // will cause a random wait anywhere between 5 and 10 seconds
Tip: The PAUSE command could optionally be used to customize desynchronization of script activity.
Refer to the SEARCH command to understand more about how this command works.
Besides the basic commands described above, there are a set of advanced command verbs that MailSizr can also aceept. The advanced commands are used to mimic the conversation between the Messaging Client and the Messaging Server _EXACTLY_. These commands are typically created using Technovations NetCorder(tm), which is an advanced recording product to capture Socket Transactions. Using the advanced verbs, a user can simulate IMAP and NNTP requests.
For those not familiar with the POP3 protocol RFC#1725, you don't need to learn the RFC! These transactions are recorded by Technovations NetCorder and the scriptfile is automatically generated. For those who may be developing applications conforming to the RFC, we have defined a very simple transactional language which can be used to prototype your application _BEFORE_ you complete implementing it. Contact Technovations for more details.
For those not familiar with the SMTP protocol RFC#821, you don't need to understand the RFC! These transactions are recorded by Technovations NetCorder and the scriptfile is automatically generated. For those who may be developing applications conforming to the RFC, we have defined a very simple transactional language which can be used to prototype your application _BEFORE_ you complete implementing it. Contact Technovations for more details.
For those not familiar with the IMAP4 protocol RFC#1730, you don't need to understand the RFC! These transactions are recorded by Technovations NetCorder and the scriptfile is automatically generated. For those who may be developing applications conforming to the RFC, we have defined a very simple transactional language which can be used to prototype your application _BEFORE_ you complete implementing it. Contact Technovations for more details.
For those not familiar with the NNTP protocol RFC#977, you don't need to understand the RFC! These transactions are recorded by Technovations NetCorder and the scriptfile is automatically generated. For those who may be developing applications conforming to the RFC, we have defined a very simple transactional language which can be used to prototype your application _BEFORE_ you complete implementing it. Contact Technovations for more details.
Manipulating
and using the SETUP Applet
The SETUP applet allows you to set up and customize the test per your specifications. The SETUP applet has several tab panels that allow such customization and this subsection will explain these in more detail.

This screen allows the user to specify the script file for this test. Remember that this script file can be either constructed/edited manually or can be generated automatically using Technovations NetCorder.
Choosing
how many users to simulate during test
The total number of users (Simultaneous Mail Clients) that need to be simulated for the test that we are about to conduct can be specified by either entering this in the "Concurrent Users to Simulate" field or by using the dropdown button to pick from preset choices.
The SETUP screen allows the user to specify actions on errors generated
during test.
By default, all errors are ignored (but reported) and tests continue
in spite of them. This setting can be used if you can allow a margin for
errors to hapen during test.
The test can be configured where test users are dropped as they encounter
errors. This setting may be useful to figure out the optimal number of
users that can be fielded by the server or application.
The test can be configured to terminate the test when any user encounters
errors. This setting is frequently used during reliability and regression
testing of applications.
There is also a mode where you could choose to retry operations whenever
errors occur. This setting mimics the way some real applications work.
Assigning
credentials to simulated users
For each user to execute the assigned script, the user may need to log in to the server. These User names and passwords are input through the credentals file associated with MailSizr. This can be initiated using the "Edit Credentials" button. Picture below shows the format of the credentials file.

The second field is the name of the user and the third field is the password associated with the username. In this case, we have simplified setup by simulating the same user multiple times. Note that many applications may not allow the same user to be logged in twice. If this is the case, then the test user needs to register unique user credentials and supply them in this file.
User credentials can also be entered/edited/imported using the SIZR.CRD file as opposed to using the above Graphical interface. The SIZR.CRD is located in the MailSizr program directory.
The credentials file is also used to store cookies associated with a certain user. In cases where cookies are encountered by MailSizr. the value of the cookie is written back into this file at the end of the test. If you would like to conduct tests by using pre generated cookies, you can input these values into the credentials file.
Now back to the details of the SETUP screen. The SETUP screen allows the test user to start a new test. Before we start a new test, let us look at some of the other settings that can be manipulated as part of test SETUP.

The "Screen/Result Logging" screen is used to
Result
Logs: Specifying the Log file for test
Result Logs contains data for each operation performed by MailSizr during test. A section of the log file which contains result data (that we decided to log in Abstract form) is shown below.
Time:125770, USER#:5, RT:4175ms, OP:SMTP
Time:126539, USER#:2, RT:15188ms, OP:PAU
Time:126821, USER#:2, RT:282ms, OP:SMTP
Time:129547, USER#:10, RT:25379ms, OP:PAU
Time:132720, USER#:1, RT:426ms, OP:POP3-read
Time:133997, USER#:3, RT:838ms, OP:POP3-read
Time:135919, USER#:6, RT:919ms, OP:POP3-read
Time:129794, USER#:10, RT:247ms, OP:SMTP
The Time: field shows the relative time in Milliseconds from test start time. The User#: field is the identity of the simulated user for the experiment. The RT: field shows the Round-Trip Response-Time for this operation, and lastly, the OP: field shows the operation performed.
Let us see what would happen if we rerun the experiment with Result Logging to get "Detailed(Socket Level Timing)" info.
Time:82288, USER#:31, OP:POP3-read, ToCreateSocket:9ms, ToConnect:4ms,
ToSendCmd:7ms, ToReadFirstByte:457ms, ToReadRestOfBytes:104ms
Time:82288, USER#:31, RT:581ms, OP:POP3-read
Time:82572, USER#:16, RT:13180ms, OP:PAU
Time:82623, USER#:40, OP:SMTP, ToCreateSocket:8ms, ToConnect:3ms, ToSendCmd:21388ms,
ToReadFirstByte:8ms, ToReadRestOfBytes:283ms
Time:82623, USER#:40, RT:21690ms, OP:SMTP
Time:82739, USER#:7, OP:SMTP, ToCreateSocket:44ms, ToConnect:17ms,
ToSendCmd:21289ms, ToReadFirstByte:72ms, ToReadRestOfBytes:292ms
Time:82739, USER#:7, RT:21714ms, OP:SMTP
Instrumenting
additional detail from MailSizr Runs.
In addition to the abstract log information, the detailed timing information
about when certain events happened for these requests is presented.
ToCreateSocket: is the time required to resolve the Server name and
create a socket to communicate with it.
ToConnect: is the time to connect to the bound port.
ToSendCommand: is the time to send the Messaging Command to the server
ToReadFirstByte: is the time taken until the first byte of data is
read from the socket.
ToReadRestOfBytes: is the time to read all the remaining bytes on the
socket (for this request).
You will notice that adding the 5 quantities above is what results in the overall ResponseTime for the Mail operation. All times are in units of Milliseconds.
Screen
Logging - Specifying recording of data read back from the System Under
Test.
Screen logging is the log that keeps track of the data read by the "Test
Mail Client". You can optionally decide not to log this data to avoid additional
IO on the load generator machine, but this could come in handy for debugging
applications under load.
If you conducted a certain test where there were failures only with
a load of 200 simultaneous users. How would you debug this problem ? You
would enable screen logging to understand what happened during test execution
and analyze the screen logs to understand the nature of the problem. what
would make it more convenient is to have the ability to log screens (mail
data) for each simulated user into a seperate file. This would make it
easier to read the screen logs. Also, logging into seperate files provides
the ability to compare screen output between different users.
The Screen page size is the size of the buffer that is used to collect data that is read from the server. This should be larger than the largest page that will be read through this experiment, but also should not be too big because this consume valuable run-time memory resources.
The "Miscellaneous Settings" panel allows you to set specific characteristics, most of which define how this experiment is run.

The home Domain Name is one of the settings that you can define. This
is used to fill the from field of the user on a SMTP command. Note that
while it is not essential to have this field populated in most cases with
the correct domain name, there may be cases when this would be required
when email filtering by domain is enabled as a Mail Router Agent.
The Operation Time-out parameter defines the length of time that a request
will wait to be serviced when inactivity occurs. If the request takes longer
than this specified time, the request will terminate and an error will
result. This error can be caught and thrown if desired or can terminate
the user or experiment, as defined in the "Error Handling" panel. Note
that if you anticipate the operation to take a long period of time, you
would need to set this parameter to be high enough value. Note that you
can set the timeout value to "0" and override this function.
It may be important in certain test scenarios to replicate the pattern in which users get created during test. MailSizr gives you the ability to ramp up users by allowing you to specify the Minimum frequency and Maximum frequency of creation.
For example, if you wanted users to be created between 5 seconds and 20 seconds, you would specify 5 seconds in the "Minimum Frequency Text box" and 20 Seconds in the "Maximum Frequency Text box". Note that the users will still be created at random.
If you want to create users at a constant rate (rather than being random
as in the above example), you can do this by keeping the values in the
"Minimum Frequency Text box" and " "Maximum Frequency Text box" the same.
This panel also allows user to be automatically desynchronized so that not all users are generating requests at the same time. This is recommended to mimic real world behavior. A user can also defeat automatic de-synchronization and put de-synchronization commands manually in the script. file. This is done by having PAUSE commands at the top of the script file. As an example a "PAUSE 0 300" command at the top of the script file starts user activity randomly between 0 and 5 minutes.
Automatic de-synchronization can be disabled to enable very quick user
RampUp. This can be useful to try quick tests with large number of users
without necessarily waiting for MailSizr to RampUp over an extended period
of time.
This is a shortcut to quickly run tests without pauses. This could be
desirable in many scenarios, especially during debug of scripts or application.
In such cases the Sizr is usually in "single user mode" - number of users
simulated = 1.
There are other cases where this is used to simulate very large loads
by ignoring all the "Pause" events in the script.
By default, each user after becoming active will start simulating test activity as defined by the basic test script or the test script assigned to that user. This is the natural and the best way of conducting tests because this is exactly the pattern that happens in real life.
However, there are sometimes specific needs to understand the impact of a large number of users hitting a server at the same time. Such tests can be used to determine the health of the server under extremely bursty workloads. MailSizr can enable these tests by proving the test user the ability to "start simulating user activity only after all users are ramped up". In this case, MailSizr will wait for all users to become active and then will hit the server with an active workload.
This "Automatic De-synchronization" feature can also be used creatively
in cases where such bursty tests are being designed. Since the bursty tests
by definition defeat user de-synchronization, the "Automatic de-synchronization"
option can be turned off for such tests to enable users to become active
as fast as the workload generator can ramp up the users.
Defining
when to start simulating test activity:
By default, each user after becoming active will start simulating test activity as defined by the basic test script or the test script assigned to that user. This is the natural and the best way of conducting tests because this is exactly the pattern that happens in real life.
However, there are sometimes specific needs to understand the impact of a large number of users hitting a server at the same time. Such tests can be used to determine the health of the server under extremely bursty workloads. MailSizr can enable these tests by proving the test user the ability to "start simulating user activity only after all users are ramped up". In this case, MailSizr will wait for all users to become active and then will hit the server with an active workload.
This "Automatic Desynchronization" feature can also be used creatively in cases where such bursty tests are being designed. Since the bursty tests by definition defeat user desynchronization, the "Automatic desynchronization" option can be turned off for such tests to enable users to become active as fast as the workload generator can ramp up the users.


By default, the collections are started when the test starts - user hits the "start test" button - and ends when test activity ends.
Collections can also be started "after user Rampup". This method analyzes the data that is collected only after all users have been activated. The time between the first user becoming active to the last user becoming active is the total RampUp time.
In some cases, you may want to enable collections after a specific period of time. You can do this by using the third option and by specifying the time. Note that the time is specified in Milliseconds.
As in the case where the test user can define the start of collections, the time when the collections can be ended can also be customized using this panel. Collections can be started "before user RampDown". This method analyzes the data that is collected until the time when the first user exits from the experiment. The time between when the first user completes test activity to the time when the last user completes such activity is defined as RampDown time.
In some cases, you may want to enable collections until a specific period of time. You can do this by using the third option and by specifying the time when you want to end collections. Note that the time is specified in Milliseconds.
Irrespective of how you define the collection/marshalling of test statistics, the test harness will still collect the data for all operations and record them in the log. So, the customization of test statistics can be used as a way to use MailSizr to "pre-analyze test data" without sacrificing the ability to do post-test analysis.
Once you have completed the required setups in this screen, you can
proceed to execute the test by depressing the Start Test button (in the
Vital Settings/Controls panel). This will start the experiment and pop
the display to monitor run-time events.

During some tests, there comes a need to simulate users who do very different things. For example, when you are trying to mimic some users whose jobs are technical in nature and others whose jobs are administrative in nature, etc., it can be quite painful to come up with a single script which models the collective behavior of these users. One solution is to run different test instances for each user type and then merge the result sets. But even this approach can be quite tedious, error prone and requires excessive manual coordination. Technovations has invented a new and simple paradigm so you do not have to endure such pains any more.
MailSizr gives you the ability to simulate such users by eliminating the need for modeling/result set merging. This is enabled by simulating multiple scripts in the same run. There are 3 approaches to accomplishing this:
Assigning by Pattern:
Here is how this is done:
Let us assume that we need to simulate 25 users of a certain type and 975 users of a certain different type. In this case, we create 25 files as described above (say mult_1.scr through mult_25.scr with directives for User#1 through User#25 respectively). The basic script file that will be used for the test will consist of directive for the rest of the 1000 users (975).
MailSizr will specifically search for the mult_*.scr pattern while assigning the individual scripts to users. When it cannot find a file that matches the specific pattern, it will assign the basic script pointed to by the "script file" field to that user. For example, if mult_26.scr does not exist (according to our example, it will not be), then the basic script is assigned to User#26.
Assigning by File
It is also possible to assign explicity using a text file. This is accomplished by specifying the bonding between a script and a simulated user. Picture below shows a file which defines such bonding.

Note that if a bond has not explicitly been defined, the test will automatically allocate the "basic test script" (defined in the Control Panel Applet of Setup) for this particular user.
Assigning Scripts Randomly
In some cases, it may be desirable to allocate script users randomly as the user is being created. This can be accomplished by defining a text file that contains a set of scripts that will be picked up randomly by each user on creation. Here is a illustration of such a file:

Note that the probability of choosing a specific script is defined implicitly
by the number of occurrences of such a script in the container file.
There may be several reasons why you may want to stop a test, ranging
from "Satisfactory collection of data" to "Test needs more refinements".
In any of these cases, there are 2 ways to stop the tests. The "Soft Termination"
mode issues a signal to all users to terminate tests after completing the
current task at hand. The "Hard Termination" option specifies that the
test should be terminated immediately and all activity should be truncated
to instantly "obey" the termination directive.
When using the SMTP command to send mail, recipients can be specified
as part of the command (as described in the script syntax section of this
document). While this can be very useful, a even more useful paradigm would
be to distribute the messages across a larger set of users, and equally
importantly, to control the distribution statistically, but with a very
simple interface. This can be accomplished using the MAILLIST file.
The maillist file has a set of user entries which the SMTP command can
pick from when it is instructed to pick users at RANDOM.

Would you like to create more than 2000 users ?
Would you like to create users from multiple distributed test locations
?
Would you like to manage tests easily (from a setup and go perspective)
?
Would you like result sets merged automatically into one comprehensive
report ?
If you answered yes to one or more of these questions, you need the
power of Technovations Load Director and Load Module logic to do this. Use of Load Director and
Load Modules are part of MailSizr enterprise edition. The use of these are
also part of the MailSizr training sessions.
The next screen-shot shows the MailSizr run-time monitor. On the left-hand side of the screen, you see a graphical picture of the throughput through the experiment (shown in solid blue) and an average of the throughput (in dotted blue). Throughput is expressed in Operations/Second and is graphed over time (abcissa or x-axis). At the bottom of the monitor screen is a test state indicator and a counter to show the number of Socket and Connection errors. These are errors that happen over the socket and can happen due to the server's inability to field requests or because of the inability of the workload generator machine to generate enough socket connections. In a Microsoft Windows 95 system, there are restrictions on the number of concurrent socket connections that can be active on the system. Which is why we recommend use of a Windows NT workstation or server as the workload generator.
The run-time monitor also gives the user controls which can be used to PAUSE TEST or STOP TESTs. If a test is PAUSEd while executing, it sends a signal to all simulated users to stop after completing their current request. Just remember that if you have a lot of simulated users and you have heavy load on the server, the PAUSE may not happen right away.
The STOP TEST command button stops the test in the middle of test execution. Even if you decide to stop tests by using the STOP TEST button, you will still be able to get detailed statistics up until the time that you stopped the test. There are 2 STOP TEST modes.
The run-time screen has updates the display every 5 seconds. This display interval can be changed for your convenience.
Using
the Run-Time Monitor Applet
This section will demonstrate the use of the Run-Time Monitor and the facilities that it provides for analysis and control.

Test Warmup Time is the time to setup the experiment. This starts from
the time the user hit the "Start Test" button to the time when the first
user is generated.
Test Rampup Time is the time to start up all the users for the test.
This is measured from the time when the first user starts up to the time
when the last user is started up.
Test Time is the time from when the last user is started up to the
time when the first user completes or drops out of the test.
Rampdown Time is the exact opposite to Rampup Time. This is the time
from when the first user drops off or terminates to the time when the last
user drops off or terminates.
In the viewing panel, there is a indicator which show the total number of Active users Vs. Specified Users. This shows you the progress through test Rampup, but there is a more important use for this. Consider te following example: You want to find out how many users you can put on your system without generating errors, etc. Let us for the moment think that this magic number is 632. Rather than conducting many tests to figure out this optimal number of users for your system, MailSizr allows you to come up with the answer with a single experiment. How ? Just specify the number of user in the SETUP screen to a high number. In our case, let us set it to 800. Set the "Error Handling" in the SETUP screen to "Terminiate User on Errors". Start Test. As the test progresses, users who encounter errors are dropped off automatically. 632 users will be running on the system without errors and will be visible through the Run-time screen. And this is how you will know that your system can support 632 users.




Whenever errors occur during test, the "Output from Test" button pops up to remind the user to acknowledge these errors before proceeding further. Notice that as we were running the experiment, we encountered several errors - shown at the bottom of the run-time screen in the "Socket/Conn Errors" category. Depressing the "Output from Test" button will show these errors.
When the test completes and any errors returned from the test are acknowledged, the user can then proceed to "View Test Results".
Reading
Test results and statistics - The Test Results Applet

The stats show the total number of each type of operation played against the system under test. The "Average Rate" is the Throughput metric which presents the total number of operations completed per unit time on the system under test. "Average Latency" is the metric which reveals Response Time details for different operations conducted against the System Under Test. This is an average number (in milliseconds). Other Numbers presented on this screen are the Standard deviation of response time about the average Response Time, which shows the variability of individual response times from the average. There are 3 discrete points on the Response Time graph that are captured and presented on this screen - The MIN, MAX and LAST values. These, along with the Standard Deviation can be used to get a quick synopsis of the test before going into more details.
The "ALL" category folds the metrics for the individual operations into an overall test average. It is possible that you may need more detailed statistical information than what is presented in this screen. For this exact reason, a complete audit trail of Response times and timestamps are maintained for each individual operation conducted through the test. This log is in Comma Seperated Variable (spreadsheet-friendly CSV) format which can be manipulated further if necessary.
From the Statistics screen, you will be given the option of documenting this test or to start a new one. If you choose to Start Another Test, you will be redirected to the Basic Setup screen. If you decide to document results, you will be directed to the screen which helps you define the elements in your test report and generate your test report.

Note that you also have the flexibility to also log the content generated by the server (during the test). You need to be aware that this can log a lot of data into the test report that may not exactly be relevant to analyze the report.

MailSizr provides a mechanism to automatically document all test parameters and values of importance. This is useful for analysis at a later point of time.
MailSizr presents a powerful paradigm to setup, run, view and develop reports for experiments which simulate hundreds of users. It very easily adapts to your test plans to deliver data and results from your experiments which can be used for Capacity Planning, Performance Analysis and Tuning, Regression Testing and Patch Management. Also, you can debug applications against realistic user Workloads and configurations for Reliability and Availability
If you are developing a mission critical application, you can performance engineer it using MailSizr. Without it, you can only guess how good your application could be under load.
If you are deploying a mission critical application and tuning it to meet ever changing user workload needs, MailSizr can not only make your deployment more robust, but also optimize your deployment cycle by providing you with the right testing technology as you incorporate new patches and versions of Application software.L