Using HP-UX Workload Manager with BEA WebLogic Server™

1 Abstract

HP-UX Workload Manager (WLM) can help you manage and prioritize BEA WebLogic Server™ workloads. This paper demonstrates various management techniques. In particular, it:

• Demonstrates example uses of WLM with WebLogic workloads, first with individual instances, then for clustered configurations
• Provides a framework from which you can extend these concepts to other commercial J2EE application server workloads
• Illustrates general WLM workload separation and management techniques outside the classic enterprise data center

2 Goal

After completing the integration, you will be able to use WLM to actively monitor a WebLogic instance, moving CPUs to or from the instance as needed to maintain acceptable performance, while also using less net CPU resources over time than an unmanaged instance would. This is a net win for you as the unused CPU resources can then be used for other computing tasks when the WebLogic instances do not require them.

3 Target audience

The discussion of use cases and management techniques should be accessible to most technical professionals with some UNIX background. The detailed information on applying changes to various configuration files may require previous knowledge of WebLogic Server and its associated terminology and administrative tools.

4 Abbreviations used

WLS

BEA's WebLogic Server

WLM

HP-UX Workload Manager

PRM

HP-UX Process Resource Manager

PSETs

HP-UX Processor Sets

FSS

HP-UX Fair Share Scheduler

VPars

HP-UX Virtual Partitions

JVM

Java Virtual Machine

MBean

JMX Management Bean

JMX

Java Management Extensions

wlmwlsdc

The Workload Manager WebLogic Server data collector utility

5 Workload requirements

The workload separation described here is largely aimed at CPU-bound workloads, or workloads that are contending for CPU resources. Although not covered in this paper, WLM can also control disk bandwidth and memory. Examples of those controls are described in other documentation. Also not covered here are workloads that are contending for, or limited by, network resources.

6 Security of the workloads

The workload-separation techniques described here are not substitutes for good security measures and isolation such as firewalls. If the workloads do not share the same level of trust, or have differing needs for user or system security, do not use WLM to combine them. WLM workload groups provide resource consumption separation; however, because the workloads share an HP-UX operating system instance, WLM does not provide the type of isolation offered by hardware partitions or systems separated by a firewall.

7 Platforms used for this paper

The WebLogic/WLM techniques outlined here were tested with the software revisions indicated in the table below.

8 Separating processes into WLM workload groups

To understand what tools we have at our disposal to separate and control workloads, this section provides a quick overview of WLM workload groups and separation. Much more information on the operation of WLM (and PRM, which provides some of WLM's functionality) can be found in the WLM User's Guide and the wlm(5) man page. These documents are available on systems with WLM installed. In addition, they are available through http://www.hp.com/go/wlm.

8.1 WLM overview

WLM controls workloads on HP-UX servers based on WLM workload groups. You define the workload groups and assign applications and users to the groups. WLM then adjusts the CPU allocations for the groups based on your defined service-level objectives (SLOs). WLM can also control memory and disk bandwidth allocations for the workload groups. (In the WLM context, a 'workload' is a group of HP-UX processes that jointly perform some work.)

WLM has two types of workload groups:

• FSS workload groups
  With these groups, WLM uses the Fair Share Scheduler (FSS) to allocate CPU based on CPU time slices. A given FSS group is provided with a particular share of the CPU allocation across all CPUs in the system. This provides good resource sharing, as the CPU allocation is very fine, but provides very little isolation and limited CPU locality.

• PSET workload groups
  WLM can use PSETs (processor sets) as the foundation for workload groups. For these groups, WLM manages the number of CPUs assigned to the underlying PSETs. A PSET offers its processes dedicated CPUs—no other processes on the system can access those CPUs.

WLM can also manage CPUs across either:

• virtual partitions (vPars)

  or

• nPartitions (nPars) that use Instant Capacity cores

vPars offer dedicated resources to their processes. In addition, a vPar has its own instance of the HP-UX operating system, providing OS isolation. nPars also offer dedicated resources and their own instances of HP-UX. In addition, nPars offer hardware isolation. With Instant Capacity, CPU resources are installed on a system but are not available for use until purchased. This feature allows you to buy a system that meets current needs but also provides a quick and efficient upgrade path.

With virtual partitions (vPars), WLM migrates CPUs between the vPars as their respective SLOs demand. Using nPars that have CPU resources managed by Instant Capacity software, WLM can deactivate CPU resources on one nPar and activate a purchased but inactive Instant Capacity core on another nPar that needs the CPU resource more. This pseudo-movement of CPUs ensures resources are available where they are needed without incurring the charge of purchasing more CPU resources.

To summarize, WLM can manage the items in the following table.

While FSS workload groups provide very flexible CPU allocation, they provide no OS or hardware isolation. Conversely, vPars and nPars offer more isolation, but CPU allocation is less flexible and administration overhead is increased.

8.2 WLM's separation mechanisms

WLM provides several methods to place processes in workload groups. It is important to understand these methods as they form the basis of our WebLogic workload separation methods. First, let's see how we define these workloads. The following snippet from a WLM configuration file creates three workload groups: servers_grp, wls_grp, and OTHERS.

	prm {
	    groups = OTHERS : 1,
		 servers_grp : 2,
		 wls_grp : PSET;
	}
    

The following sections explore how processes are placed in defined workload groups.

8.2.1 Application records: workload separation by binary name

8.2.1.1 Separation by binary names

	prm {
	    groups = OTHERS : 1,
		 servers_grp : 2,
		 wls_grp : PSET;

	    apps = wls_grp : /opt/weblogic/bin/httpd ;
	}
    

The app record above causes the PRM application manager to place any newly running /opt/weblogic/bin/httpd executables into the group wls_grp, which is a dynamic PSET.

8.2.1.2 Separation by alternate names

	prm {
	    groups = OTHERS : 1,
		 servers_grp : 2,
		 wls_grp : PSET;

	    # note that /bin/sh must appear in either
	    # /etc/shells or /opt/prm/shells for this to work
	    apps = wls_grp : /bin/sh startInstA.sh ;
	}
    

The app record above causes the PRM application manager to place any newly running /bin/sh script named startInstA.sh into the group wls_grp, which is a dynamic PSET. Note that for this to work, /bin/sh must appear in either /etc/shells or /opt/prm/shells. See the wlmconf(1M) man page for more information on alternate names.

NOTE on polling: It is important to realize that for either application name or alternate names, the process is not placed in the workload group at fork() time. Rather, the PRM application manager periodically polls for newly started processes that match app records. If it detects one or more matches, the processes are moved to their respective workload groups. For more information, see the application manager discussion in the WLM or PRM documentation.

8.2.2 User records: workload separation by process owner UID

	prm {
	    groups = OTHERS : 1,
		     testers : 2,
		     coders : 3,
		     surfers : 4;

	    users = moe : coders surfers,
		    curly : coders surfers,
		    larry : testers surfers;
	}
       

Besides the default OTHERS group, this example has three groups of users: testers, coders, and surfers. The user records cause processes started by users moe and curly to be run in group coders by default, and user larry's processes to be run in group testers by default. Each user is also given permission to run jobs in group surfers if they wish, using the prmrun or prmmove commands mentioned below. Users not belonging to either group are placed in OTHERS by default.

For information on group permissions and precedence of user and app records, see the PRM and WLM documentation.

8.2.3 prmrun: starting a process in a workload group

	    % my_really_big_job &
	

	    % prmrun -g surfers my_really_big_job &
	

This has particular application to WebLogic, as it can be used to run httpd and its children directly in a group:

	    % prmrun -g wls_grp /opt/weblogic/bin/weblogicctl start
	

	    % prmrun -g wls_grp /opt/tomcat/bin/tomcat start
	

prmmove: moving an existing process to a workload group

Besides using prmrun to start new processes in particular groups, existing processes can be moved to a new group with the prmmove command. If larry from the above example has a job running with process ID (PID) 4065 running in group testers, he could move that process to group surfers by running the command:

	% prmmove surfers -p 4065
       

For WebLogic, which runs as a daemon and becomes the leader of its own process group, sweeping all members of httpd's process group into a workload group is quick and easy. Here we use the contents of the stored PID file to locate httpd's PID. Because httpd is a daemon, its PID is also its process group ID. We can sweep the process group into the workload group wls_grp as follows:

	% prmmove wls_grp -g `cat /opt/weblogic/logs/httpd.pid`
       

8.2.4 Default: inheriting workload group from parent process

If a process is not named by an explicit user or app record, or has not been started with prmrun or moved with prmmove, it simply starts and runs in the same group as its parent process. So for a setup like this:

	prm {
	    groups = OTHERS : 1,
		    mygrp : 2;
	}
       

10.4 Monitoring and managing the WebLogic instance

10.4.1 WebLogic Server command-line tools

There are a set of command-line uses of the weblogic.Admin class that may be useful for the administrator to do a quick check of an instance's availability, confirm usernames and passwords, or even load some queue-related metrics.

For instance, this command

	$JAVA_HOME/bin/java \
		-classpath $WL_HOME/server/lib/weblogic.jar \
		weblogic.Admin \
		-url t3://myhost.mydomain.com:7001 \
		-username system \
		-password weblogic \
		get -pretty \
		-mbean 'instAdomain:Location=admin,Name=default,ServerRuntime=admin,Type=ExecuteQueueRuntime'
	

11 WebLogic Server metrics to use with WLM

11.1 WLS queues When Enterprise Java Bean (EJB) requests are submitted to a given WebLogic instance, it is actually placed in a queue of work, referred to as an execute queue. This queue is serviced by a pool of work threads. The default queue for a given instance has the unsurprising name of 'default', and generally is configured with roughly 15 threads, a few of which are consumed by system functions. To quote the WebLogic Tuning Guide:

By default, a new WebLogic Server instance is configured with a default execute queue (named "default") that contains 15 threads, which are used by all applications running on the server instance. A WebLogic Server instance also contains two built-in execute queues named __weblogic_admin_html_queue and __weblogic_admin_rmi_queue, but these queues are reserved for communicating with the Administration Console. If you configure no additional execute queues, all Web applications and RMI objects use the default queue.

See the "Execute Queue —> Configuration" help text for more information on configuring queues and thread pools. Also, see the "Using Execute Queues to Control Thread Usage" section of the WebLogic Server Performance and Tuning Guide for more information on queues, thread pools, and why and how to configure new queues in an instance.

Note that each queue (and most other instance features) have Management Beans (MBeans) associated with them that are used by the management tools to monitor and control instance components.

11.2 WLS queue metrics We will use two metrics to estimate an instance's need for more or fewer resources. These metrics are:

queue_depth

MBean parameter based on: PendingRequestCurrentCount

idle_thread_count

MBean parameter based on: ExecuteThreadCurrentIdleCount

In general, if a given queue has idle threads waiting for work, then it has enough CPU. If it has several idle threads, then it may have more CPU than it needs. Conversely, if there are no idle threads and there is a nonzero queue_depth, there is work waiting to be done and more CPU resources may be useful to the instance.

For a given work queue, there can be either idle threads or a nonzero queue depth, but not both at once. As mentioned later in the wlmwlsdc discussion, we will be using a third metric, queue_busy, which is a combination of the idle_thread_count and queue_depth metrics. This combination lets us see idle threads as queue_busy < 0, and items on the queue as queue_busy > 0.

12 How to use the WLS metrics with WLM
With the exception of a few built-in metrics like CPU usage, WLM must be provided application-specific performance data to evaluate how the application is doing. The performance data must be fresh, accurate, efficient to collect, and indicate resource needs. WLM uses that data stream to evaluate how successful previous allocation changes were and decide what changes to make in the future. The metrics are provided by data collectors, which have application-specific knowledge, both what metrics to fetch and how to fetch them.

12.1 Metric collection: wlmwlsdc

WLM comes with a number of data collectors that are dedicated to collecting a certain type of data. The data collector for WLS metrics is wlmwlsdc.

12.1.1 Metrics available through wlmwlsdc

The data collector wlmwlsdc returns the metrics listed below.

The queue_busy metric is a synthetic metric calculated from two WebLogic internal metrics. For many SLOs, this combination is more useful than either metric alone—with a single metric it provides information about whether the instance has extra threads available (idle_thread_count) or more work than the threads can currently process simultaneously (queue_depth). This combination metric allows similar SLOs to be written with fewer input metrics, using fewer external data collector processes than collecting queue_depth and idle_thread_count separately.

12.1.2 wlmwlsdc usage

The data collector wlmwlsdc monitors selected WLS instance execution queue metrics. wlmwlsdc outputs a stream of metrics to stdout, and any errors to stderr. The user can make trial runs of the data collector from the command line, and then use the same command line inside a WLM configuration file.

Here is how the command looks if run from the command line:

	% /opt/wlm/toolkits/weblogic/bin/wlmwlsdc \
	    my_instance_props.properties
	

If the location of WLS or the JVM is nonstandard, the command can be run with a second argument like this:

	% /opt/wlm/toolkits/weblogic/bin/wlmwlsdc \
	    /home/work/my_instance_props.properties \
	    /home/work/bea/weblogic700/server/bin/setWLSEnv.sh
	

Within a WLM configuration file, the same data collector invocation would be done like this:

	#
	# Send stdout to wlmrcvdc (and on to wlmd) and stderr to syslog.
	#
	tune my_metric {
	    coll_argv =
		wlmrcvdc
		    wlmwlsdc \
			/home/work/my_instance_props.properties \
			/home/work/bea/weblogic700/server/bin/setWLSEnv.sh
	    ;
	    coll_stderr = syslog;
	}
	

The coll_stderr statement sends stderr for wlmwlsdc to syslog.

12.1.3 wlmwlsdc properties

You can set numerous properties to be used by wlmwlsdc as it collects metrics. Set these properties, which are described in the table below, in a file given on your wlmwlsdc invocation line, as shown in the previous section.

12.1.4 wlmwlsdc POSIX environment variable scripts

However, these same requirements exist for a given WebLogic instance, and are conventionally solved by the instance executing a simple POSIX sh script before its JVM is invoked. This is usually named setEnv.sh and is executed by the instance startWebLogic.sh or startManagedWebLogic.sh script.

If any of the settings for the instance do not match the defaults in the table above, then a file containing the correct environment variable settings must be provided as a second argument:

	% /opt/wlm/toolkits/weblogic/bin/wlmwlsdc \
	    my.props \
	    my_env_vars.sh
	

12.1.4.1.1 Benefit: simplicity

Also, if the instance is updated or other environment variable settings are changed, the wlmwlsdc monitoring the instance will use the new settings.

12.1.4.1.2 Drawback: executing file as root

12.1.4.2 Run instance setEnv.sh script directly as nonroot

Here's how you'd modify an existing example WLM configuration file to run setEnv.sh as a user other than root. As an example we'll use wls_2inst_q_goal and assume the WLS instance is normally run by a UNIX user 'wladmin'. This means we'd like to also run wlmwlsdc as the user 'wladmin' (instead of root):

1. Modify the wlmrcvdc lines, adding -u wladmin

   A coll_argv line like the following would accomplish that:

   
   	    tune wls.default.queue_busy {
   	      coll_argv =
   		  wlmrcvdc -u wladmin
   		      /opt/wlm/toolkits/weblogic/bin/wlmwlsdc
   			/opt/wlm/toolkits/weblogic/config/instA_qbusy.props
   			/opt/bea/domains/testdomain/setEnv.sh
   		;
   
   	      # smooth out the peaks and valleys in this metric stream
   	      cntl_smooth = 0.5 ;
   
   	      # send data collector errors to syslog
   	      coll_stderr = syslog ;
   	    }
   	    

   This will cause wlmwlsdc to be run as user 'wladmin'.

2. Add app records for wlmwlsdc.
   By default, nonroot owner processes will be migrated by WLM to the group OTHERS or to the user's default group. However, we need to keep the wlmwlsdc data collector processes in the PRM_SYS group, along with wlmd, to isolate them from any resource shortages. We do that by writing a specific app record to place them in the special group PRM_SYS. See below for the configuration file text.

3. Add PRM_SYS to the PRM groups list
   WLM requires that any groups mentioned in app or user records be explicitly listed in the groups statement. After adding that, the whole prm structure looks like this:

   
   	    prm {
   		groups =
   			PRM_SYS : 0,
   			OTHERS : 1,
   			wls1_grp : PSET,
   			wls2_grp : PSET
   		;
   
   		apps =
   		    PRM_SYS  : /bin/sh wlmwlsdc
   		;
   
   		users = wladmin: OTHERS wls1_grp wls2_grp ;
   	    }
   	    

12.1.4.2.1 Benefit: secure, reuses existing setEnv.sh file

Also, if the instance is updated with custom environment settings in its setEnv.sh file, the wlmwlsdc command monitoring the instance will automatically use the new settings.

12.1.4.2.2 Drawback: more complicated WLM configuration file

12.1.4.3 Run root-owned copy of instance setEnv.sh script as root

12.1.4.3.1 Bene