• System Architects/Engineers and Database Administrators who are charged with design and implementation of a Database System to meet specific user growth and workload needs. DbSizr can be used to conduct performance and stress tests against various server configurations. In addition, it can also document results for such tests automatically and thus helps in the analysis to arrive at the most optimal configuration. By providing the ability to document the configuration files, DbSizr provides an ability to tune these system deployments. DbSizr can be used to measure the reliability, availability and security of applications under load.
• Application developers who develop/debug complex applications where performance and scalability is important. DbSizr can be used to understand the impact of SQL code as well as stored procedures under load.

• Allows the user to define Test Acceptance Criteria and enforces them before starting User Simulation.
• Gives logging facilities that allow developers to debug application programs.
• Gives user the ability to define Error Handling that can control the course of the experiment.
• Allows user to automatically desynchronize themselves for more realistic simulations.
• Allows users to define timeouts for user operations.
• Allows user to collect abstract timing info for calls or can scope into socket level timing information
• Graphically and numerically Presents Run-Time performance and status views during test.
• Allows pages (Query Data) during test to be logged to HTML files based on user's preference.
• Presents performance results in an easy to understand concise form.
• Allows simulation of multiple scripts in the context of a single test.
• Allows the test user to define custom collections to analyze test activity.
• Has constructs to create normal or bursty tests.
• Has an automatic Report Writer and Configuration Manager that produces an HTML report for each test.

Here are the currently supported set of command verbs and the syntax for the commands.

• OPEN : This command opens a specific database connection.

Syntax #1: OPEN

• CONNECT (OPEN with retry capabilities) : This command's syntax is exactly like the OPEN command, except that the functionality is slightly different. In the case of the CONNECT command, the DbSizr client retries any failed connections for a predetermined number of times (currently 10 by default) with built in PAUSEs (default of 60 Seconds) between retries. This command can be used to stress the connection limits of Database applications.
• CLOSE (connection to database) : Closes the connection to the opened Data Source. The CLOSE command has no arguments.
• RUNFOR (to define length of test Run) : This command is used to specify the length of the test run. This command, when used is typically the first command in the script file.

• PAUSE : This command is used to create event lists using the script file. It takes at least one argument that specifies the number of seconds to pause. It could optionally be overloaded with a second argument that is used to create randomness in the PAUSE times for different threads and different instances of the PAUSE command within the same script.

Syntax : PAUSE WaitLowerLimit(seconds) [WaitUpperLimit(seconds)]
Examples :
• PAUSE 3 //will make the execution thread pause for 3 seconds
• PAUSE 5 10 // will cause a random wait anywhere between 5 and 10 seconds

• REWIND : This command allows a pattern in the scriptfile to be iterated a number of times.

Syntax : REWIND #Iterations
Example : REWIND 300
This example specifies that all lines in the script file which occur after the REWIND command will be iterated 300 times.
• EXEC: This is a DbSizr command that can be used to OPEN and CLOSE the connection for the execution of a specific SQL statement on the default Data Source. The argument to the EXEC command is a SQL or other database command.

Syntax: EXEC <SQL COMMAND>

• SEARCH : The SEARCH command is used to find a string from the content screen (Query data) returned after a successful query execution or stored procedure. If the string is found, it results in a FOUND event for that specific thread. Else, the NOT_FOUND event is flagged.

Syntax : Search <search string>
Example : Search This is a test

In this example, the screen buffer will be tested for the presence of the string "This is a test". Remember that the search is case-sensitive. If the string is present in the buffer, a FOUND event will result for the user. If not, a NOTFOUND event will result. These events can be used to force execution or ignore execution of some test management commands.

• EXIT : This command is an instruction for the test kernel to terminate activity for the User. This termination is usually instantiated using the FOUND/NOTFOUND or on IFCLOSED/IFOPEN events. Note that the EXIT command can also be invoked unconditionally.

Syntax: EXIT <IFFOUND|IFNOTFOUND>
Examples:
EXIT IFFOUND
EXIT IFNOTFOUND
EXIT IFCLOSED
EXIT IFOPEN

Refer to the SEARCH command to understand more about how this command works in the context of the IFFOUND and IFNOTFOUND events.

The IFCLOSED clause causes termination of activity for this user when it encounters a closed database (or an invalid database handle). The IFOPEN clause causes termination of activity for this user when it encounters an open database (or a valid database handle).

• TERMINATE: This command is used to terminate the experiment, usually based on an event. The FOUND/NOTFOUND events are used to invoke this command.

Syntax: TERMINATE <IFFOUND|IFNOTFOUND>
Examples:
TERMINATE IFFOUND
TERMINATE IFNOTFOUND
TERMINATE

Manipulating and using the SETUP Applet

The SETUP applet allows you to set up and customize the test per your specifications. The SETUP applet has several tab panels that allow such customization and this subsection will explain these in more detail.

Let us start with the "Vital Settings/Controls" view of the SETUP screen. Note that there is a set of tasks that you need to do to make sure that your Data Source is visible from your client system that is running DbSizr. Please go to this link for a step-by-step setup tutorial.
This applet, among other things, allows you to define the duration of your test. If you do not choose a test duration, the test will run to reproduce all the events defined in  the script file(s) used for the test. The duration of the test can be limited by using the "Run Test for" checkbox and the associated numeric value for the number of seconds.

The "Data Source" field defines the default Data Source that will be used for this test. Note that this Data Source needs to be registered on the client system before it can be specified in the DbSizr setup screen. Note that DbSizr allows you to operate on multiple number of Data Sources simultaneously during an experiment.

Choosing a script file to use for test

The Script File is used to play scripts which contain commands which instruct the DbSizr run-time kernel about operations that we want to play through each simulated user. You can also edit and modify the script file from this setup screen.

Choosing how many users to simulate during test

The "Concurrent Users to Simulate" field defines the number of concurrent user sessions that will be generated during the experiment. This could also be the number of Application Threads that are being simulated, if you have a multi-tiered Client/Server system. You can pick this number by entering it in this field or by picking a number from preset choices.

Choosing how to handle errors

The SETUP screen allows the user to specify actions on errors generated during test.
By default, all errors are ignored (but reported) and tests continue in spite of them. This setting can be used if you can allow a margin for errors to hapen during test.
The test can be configured where test users are dropped as they encounter errors. This setting may be useful to figure out the optimal number of users that can be fielded by the server or application.
The test can be configured to terminate the test when any user encounters errors. This setting is frequently used during reliability and regression testing of applications.
There is also a mode where you could choose to retry operations (SQL calls) whenever errors occur. This setting mimics the way some real applications work.

Assigning credentials to simulated users

For each user to execute the assigned script, the user may need to log in to the server. These User names and passwords are input through the credentals file associated with DbSizr. This can be initiated using the "Edit Credentials" button. Picture below shows the format of the credentials file.

The second field is the name of the user and the third field is the password associated with the username. In this case, we have simplified setup by simulating the same user multiple times. Note that many applications may not allow the same user to be logged in twice. If this is the case, then the test user needs to register unique user credentials and supply them in this file.

User credentials can also be entered/edited/imported using the SIZR.CRD file as opposed to using the above Graphical interface. The SIZR.CRD is located in the DbSizr program directory.

The credentials file is also used to store cookies associated with a certain user. In cases where cookies are encountered by DbSizr. the value of the cookie is written back into this file at the end of the test. If you would like to conduct tests by using pre generated cookies, you can input these values into the credentials file.

Now back to the details of the SETUP screen. The SETUP screen allows the test user to start a new test. Before we start a new test, let us look at some of the other settings that can be manipulated as part of test SETUP.

• Specify the log file for the test.
• Specify the level of instrumentation needed from DbSizr.
• Specify if data should be read back from the System Under Test and how it should be presented

Result Logs: Specifying the Log file for test

Result Logs contains data for each operation performed by DbSizr during test. We crafted out a simple script file to perform a bunch of SQL queries. A section of the result data (that we decided to log in Abstract form) is shown below.

Time:10614, USER#:49, RT:39ms, OP:Insert, dbRName:testdb, Query:INSERT INTO credentialsTable VALUES ('STEVE DONAHUE', 'THEDON', '1003', 'false', '')
Time:10633, USER#:50, RT:42ms, OP:Insert, dbRName:testdb, Query:INSERT INTO credentialsTable VALUES ('STEVE DONAHUE', 'THEDON', '1003', 'false', '')
Time:10643, USER#:43, RT:1706ms, OP:PAU
Time:10666, USER#:49, RT:52ms, OP:Insert, dbRName:testdb, Query:INSERT INTO credentialsTable VALUES ('Larry+Jones', 'RK', '1004', 'false', '')
Time:10667, USER#:50, RT:34ms, OP:Insert, dbRName:testdb, Query:INSERT INTO credentialsTable VALUES ('Larry+Jones', 'RK', '1004', 'false', '')
Time:10754, USER#:48, RT:146ms, OP:Open, dbRName:testdb, Command:open

The Time: field shows the relative time in Milliseconds from test start time. The USER#: field is the identity of the simulated user for the experiment. The RT: field shows the Round-Trip Response-Time for this operation, and the final few fields show the "Operation Type", the "Data Source" that was used and the specifics of the operation that was performed.

Instrumenting additional detail from DbSizr Runs.

Let us see what would happen if we rerun the experiment with Result Logging to get "Detailed (Db Internals Timing)" information. Again, we use a section of an example run to illustrate.

Time:25572, USER#:46, OP:Open, ToConnect:373ms, ToExecute:0ms, ToStartFetch:0ms, ToEndFetch:0ms, ToDisconnect:0ms
Time:25572, USER#:46, RT:373ms, OP:Open, dbRName:testdb, Command:open
Time:25628, USER#:37, OP:Open, ToConnect:422ms, ToExecute:0ms, ToStartFetch:0ms, ToEndFetch:0ms, ToDisconnect:0ms
Time:25628, USER#:37, RT:422ms, OP:Open, dbRName:testdb, Command:open
Time:25723, USER#:38, RT:2380ms, OP:PAU
Time:25737, USER#:50, RT:1769ms, OP:PAU
Time:25740, USER#:38, OP:Insert, ToConnect:0ms, ToExecute:17ms, ToStartFetch:0ms, ToEndFetch:0ms, ToDisconnect:0ms
Time:25740, USER#:38, RT:17ms, OP:Insert, dbRName:testdb, Query:INSERT INTO credentialsTable VALUES ('Joe Schmoe', 'secret', '12345678', 'false', 'ApplePicker')

In addition to the abstract log information, the detailed timing information about when certain events happened for these requests is presented.

• ToConnect: is the time to connect to the Data Source. Note that if the Data source is already open and does not need to be reopened, this number will be 0 MilliSecs.
• ToExecute: is the time to execute the SQL query or stored procedure.
• ToStartFetch: is the elapsed time taken until the first byte of data is received by DbSizr.
• ToEndFetch: is the time to read all the remaining data bytes (for this query or command).
• ToDisconnect is the time to disconnect from the database. Note again that this number will be equal to 0 Milliseconds if the connection is implied to be persistent.

Screen Logging - Specifying recording of data read back from the System Under Test.

Screen logging is the log that keeps track of the data read by DbSizr from the Database Application or server. You can optionally decide not to log this data to avoid additional IO on the load generator machine, but this could come in handy for debugging applications under load.
If you conducted a certain test where there were failures only with a load of 200 simultaneous users. How would you debug this problem ? You would enable screen logging to understand what happened during test execution and analyze the screen logs to understand the nature of the problem. What would make it more convenient is to have the ability to log screens for each simulated user into a separate file. This would make it easier to read the screen logs. Also, logging into separate files provides the ability to compare screen output between different users.

The Screen page size is the size of the buffer that is used to collect data read from the server. This should be larger than the largest page that will be read through this experiment, but also should not be too big because this consumes valuable run-time memory resources.

The "Miscellaneous Settings" panel allows you to set specific characteristics, most of which define how this experiment is run.

Specifying Time-outs for Operations.

The Operation Time-out parameter defines the length of time that a request will wait to be serviced when inactivity occurs. If the request takes longer than this specified time, the request will terminate and an error will result. This error can be caught and thrown if desired or can terminate the user or experiment, as defined in the "Error Handling" panel. Note that if you anticipate the operation to take a long period of time, you would need to set this parameter to be high enough value. Note that you can set the timeout value to "0" and override this function.
 

Customizing the creation of users for test

It may be important in certain test scenarios to replicate the pattern in which users get created during test. DbSizr gives you the ability to ramp up users by allowing you to specify the Minimum frequency and Maximum frequency of creation.

For example, if you wanted users to be created between 5 seconds and 20 seconds, you would specify 5 seconds in the "Minimum Frequency Text box" and 20 Seconds in the "Maximum Frequency Text box". Note that the users will still be created at random.

If you want to create users at a constant rate (rather than being random as in the above example), you can do this by keeping the values in the  "Minimum Frequency Text box" and " "Maximum Frequency Text box" the same.
 

User De-synchronization:

This panel also allows user to be automatically desynchronized so that not all users are generating requests at the same time. This is recommended to mimic real world behavior. A user can also defeat automatic de-synchronization and put de-synchronization commands manually in the script. file. This is done by having PAUSE commands at the top of the script file. As an example a "PAUSE 0 300" command at the top of the script file starts user activity randomly between 0 and 5 minutes.

Automatic de-synchronization can be disabled to enable very quick user RampUp. This can be useful to try quick tests with large number of users without necessarily waiting for DbSizr to RampUp over an extended period of time.
 
 

Ignoring Pause directives in script:

This is a shortcut to quickly run tests without pauses. This could be desirable in many scenarios, especially during debug of scripts or application. In such cases the Sizr is usually in "single user mode" - number of users simulated = 1.
There are other cases where this is used to simulate very large loads by ignoring all the "Pause" events in the script.
 

Defining when to start simulating test activity:

By default, each user after becoming active will start simulating test activity as defined by the basic test script or the test script assigned to that user. This is the natural and the best way of conducting tests because this is exactly the pattern that happens in real life.

However, there are sometimes specific needs to understand the impact of a large number of users hitting a server at the same time. Such tests can be used to determine the health of the server under extremely bursty workloads. DbSizr can enable these tests by proving the test user the ability to "start simulating user activity only after all users are ramped up". In this case, DbSizr will wait for all users to become active and then will hit the server with an active workload.

This "Automatic De-synchronization" feature can also be used creatively in cases where such bursty tests are being designed. Since the bursty tests by definition defeat user de-synchronization, the "Automatic de-synchronization" option can be turned off for such tests to enable users to become active as fast as the workload generator can ramp up the users.
 

Defining when to start simulating test activity:

By default, each user after becoming active will start simulating test activity as defined by the basic test script or the test script assigned to that user. This is the natural and the best way of conducting tests because this is exactly the pattern that happens in real life.

However, there are sometimes specific needs to understand the impact of a large number of users hitting a server at the same time. Such tests can be used to determine the health of the server under extremely bursty workloads. DbSizr can enable these tests by proving the test user the ability to "start simulating user activity only after all users are ramped up". In this case, DbSizr will wait for all users to become active and then will hit the server with an active workload.

This "Automatic Desynchronization" feature can also be used creatively in cases where such bursty tests are being designed. Since the bursty tests by definition defeat user desynchronization, the "Automatic desynchronization" option can be turned off for such tests to enable users to become active as fast as the workload generator can ramp up the users.

By default, the collections are started when the test starts - user hits the "start test" button - and ends when test activity ends.

Collections can also be started "after user Rampup". This method analyzes the data that is collected only after all users have been activated. The time between the first user becoming active to the last user becoming active is the total RampUp time.

In some cases, you may want to enable collections after a specific period of time. You can do this by using the third option and by specifying the time. Note that the time is specified in Milliseconds.

As in the case where the test user can define the start of collections, the time when the collections can be ended can also be customized using this panel. Collections can be started "before user RampDown". This method analyzes the data that is collected until the time when the first user exits from the experiment. The time between when the first user completes test activity to the time when the last user completes such activity is defined as RampDown time.

In some cases, you may want to enable collections until a specific period of time. You can do this by using the third option and by specifying the time when you want to end collections. Note that the time is specified in Milliseconds.

Irrespective of how you define the collection/marshalling of test statistics, the test harness will still collect the data for all operations and record them in the log. So, the customization of test statistics can be used as a way to use DbSizr to "pre-analyze test data" without sacrificing the ability to do post-test analysis.

Once you have completed the required setups in this screen, you can proceed to execute the test by depressing the Start Test button (in the Vital Settings/Controls panel). This will start the experiment and pop the display to monitor run-time events.

Advanced Test Settings

 
 

Assigning individual scripts to users.

During some tests, there comes a need to simulate users who do very different things. For example, when you are trying to mimic some users whose jobs are technical in nature and others whose jobs are administrative in nature, etc., it can be quite painful to come up with a single script which models the collective behavior of these users. One solution is to run different test instances for each user type and then merge the result sets. But even this approach can be quite tedious, error prone and requires excessive manual coordination. Technovations has invented a new and simple paradigm so you do not have to endure such pains any more.

DbSizr gives you the ability to simulate such users by eliminating the need for modeling/result set merging. This is enabled by simulating multiple scripts in the same run. There are 3 approaches to accomplishing this:

Assigning by Pattern:

Here is how this is done:

• First, the test user checks the box that indicates that the user intends to "Assign individual scripts to test users".
• Second, defining a pattern (in the "By Pattern Text box") for the files that contain the directives that need to be simulated for the users. For example, if you have a pattern like "mult_" as shown above, this pattern will be used by DbSizr to read individual files. If there is a file called mult_1.scr, this will be assigned to User#1. If there is a file called mult_37.scr, this script file will be assigned for User#37, and so on. So, in a 1000 user simulation scenario, it is possible to simulate the 1000 users each with its own script.

Let us assume that we need to simulate 25 users of a certain type and 975 users of a certain different type. In this case, we create 25 files as described above (say mult_1.scr through mult_25.scr with directives for User#1 through User#25 respectively). The basic script file that will be used for the test will consist of directive for the rest of the 1000 users (975).

DbSizr will specifically search for the mult_*.scr pattern while assigning the individual scripts to users. When it cannot find a file that matches the specific pattern, it will assign the basic script pointed to by the "script file" field to that user. For example, if mult_26.scr does not exist (according to our example, it will not be), then the basic script is assigned to User#26.

Assigning by File

It is also possible to assign explicity using a text file. This is accomplished by specifying the bonding between a script and a simulated user. Picture below shows a file which defines such bonding.

Note that if a bond has not explicitly been defined, the test will automatically allocate the "basic test script" (defined in the Control Panel Applet of Setup) for this particular user.

Assigning Scripts Randomly

In some cases, it may be desirable to allocate script users randomly as the user is being created. This can be accomplished by defining a text file that contains a set of scripts that will be picked up randomly by each user on creation. Here is a illustration of such a file:

Note that the probability of choosing a specific script is defined implicitly by the number of occurrences of such a script in the container file.
 
 

Test Stop Modes

There may be several reasons why you may want to stop a test, ranging from "Satisfactory collection of data" to "Test needs more refinements". In any of these cases, there are 2 ways to stop the tests. The "Soft Termination" mode issues a signal to all users to terminate tests after completing the current task at hand. The "Hard Termination" option specifies that the test should be terminated immediately and all activity should be truncated to instantly "obey" the termination directive.
 

Creating very large Collection of Users (Tens of Thousands of Users)

Would you like to create more than 2000 users ?
Would you like to create users from multiple distributed test locations ?
Would you like to manage tests easily (from a setup and go perspective) ?
Would you like result sets merged automatically into one comprehensive report ?

If you answered yes to one or more of these questions, you need the power of Technovations Load Director and Load Module logic to do this. Use of Load Director and Load Modules are part of DbSizr enterprise edition. The use of these are also part of the DbSizr training sessions.
 
 

The next screen-shot shows the DbSizr run-time monitor. On the left-hand side of the screen, you see a graphical picture of the throughput through the experiment (shown in solid blue) and an average of the throughput (in dotted blue). Throughput is expressed in Operations/Second and is graphed over time (abcissa or x-axis). At the bottom of the monitor screen is a test state indicator and a counter to show the number of Socket and Connection errors. These are errors that happen over the socket and can happen due to the server's inability to field requests or because of the inability of the workload generator machine to generate enough socket connections. In a Microsoft Windows 95 system, there are restrictions on the number of concurrent socket connections that can be active on the system. Which is why we recommend use of a Windows NT workstation or server as the workload generator.

The run-time monitor also gives the user controls which can be used to PAUSE TEST or STOP TESTs. If a test is PAUSEd while executing, it sends a signal to all simulated users to stop after completing their current request. Just remember that if you have a lot of simulated users and you have heavy load on the server, the PAUSE may not happen right away.

The STOP TEST command button stops the test in the middle of test execution. Even if you decide to stop tests by using the STOP TEST button, you will still be able to get detailed statistics up until the time that you stopped the test. There are 2 STOP TEST modes.

• Soft Termination : This is a signal that instructs all simulated users to stop after the current command. Again, Just remember that if you have a lot of simulated users and you have heavy load on the server, the STOP may not happen right away.
• Hard Termination : This is a hard stop at the time the user decides to terminate the test. The simulated users are all terminated immediately.

The run-time screen has updates the display every 5 seconds. This display interval can be changed for your convenience.

Using the Run-Time Monitor Applet

This section will demonstrate the use of the Run-Time Monitor and the facilities that it provides for analysis and control.

Test Warm-up Time is the time to setup the experiment. This starts from the time the user hit the "Start Test" button to the time when the first user is generated.
Test Rampup Time is the time to start up all the users for the test. This is measured from the time when the first user starts up to the time when the last user is started up.

Test Time is the time from when the last user is started up to the time when the first user completes or drops out of the test.
Rampdown Time is the exact opposite to Rampup Time. This is the time from when the first user drops off or terminates to the time when the last user drops off or terminates.

In the viewing panel, there is an indicator, which show the total number of Active users Vs. Specified Users. This shows you the progress through test Rampup, but there is a more important use for this. Consider the following example: You want to find out how many users you can put on your system without generating errors, etc. Let us for the moment think that this magic number is 632. Rather than conducting many tests to figure out this optimal number of users for your system, DbSizr allows you to come up with the answer with a single experiment. How ? Just specify the number of user in the SETUP screen to a high number. In our case, let us set it to 800. Set the "Error Handling" in the SETUP screen to "Terminate User on Errors". Start Test. As the test progresses, users who encounter errors are dropped off automatically. 632 users will be running on the system without errors and will be visible through the Run-time screen. And this is how you will know that your system can support 632 users.

The statistics show the total number of each type of operation played against the system under test. The "Average Rate" is the Throughput metric that presents the total number of operations completed per unit time on the system under test. "Average Latency" is the metric which reveals Response Time details for different operations conducted against the System Under Test. This is an average number (in milliseconds). Other Numbers presented on this screen are the Standard deviation of response time about the average Response Time, which shows the variability of individual response times from the average. There are 3 discrete points on the Response Time graph that are captured and presented on this screen - The MIN, MAX and LAST values. These, along with the Standard Deviation can be used to get a quick synopsis of the test before going into more details.

The "ALL" category folds the metrics for the individual operations into an overall test average. It is possible that you may need more detailed statistical information than what is presented in this screen. For this exact reason, a complete audit trail of Response times and timestamps are maintained for each individual operation conducted through the test. This log is in Comma Separated Variable (spreadsheet-friendly CSV) format that can be manipulated further if necessary.

From the Statistics screen, you will be given the option of documenting this test or to start a new one. If you choose to Start Another Test, you will be redirected to the Basic Setup screen. If you decide to document results, you will be directed to the screen that helps you define the elements in your test report and generate your test report.

Note that you also have the flexibility to also log the content generated by the server (during the test). You need to be aware that this can log a lot of data into the test report that may not exactly be relevant to analyze the report.

If the test involves simulation of separate scripts, all such scripts will be logged separately into the Script section of the report.

DbSizr provides a mechanism to automatically document all test parameters and values of importance. This is useful for analysis at a later point of time.

Conclusions

DbSizr presents a powerful paradigm to setup, run, view and develop reports for experiments that simulate hundreds of users. It very easily adapts to your test plans to deliver data and results from your experiments that can be used for Capacity Planning, Performance Analysis and Tuning, Regression Testing and Patch Management. Also, you can debug applications against realistic user Workloads and configurations for Reliability and Availability

If you are developing a mission critical application, you can performance engineer it using DbSizr. Without it, you can only guess how good your application could be under load.
If you are deploying a mission critical application and tuning it to meet ever changing user workload needs, DbSizr can not only make your deployment more robust, but also optimize your deployment cycle by providing you with the right testing technology as you incorporate new patches and versions of Application software.