$Revision: 1.17 $
Last Modified: $Date: 2006/05/08 20:06:48 $
Available at
1 Table of Contents
|
2 Goal
To effectively configure, tune, and manage the HP-UX Workload Manager and its associated data collectors, some method of visualizing both the performance metrics seen by HP-UX WLM and WLM's resulting changes to CPU entitlements is invaluable.
In this short paper, we will set up a configuration that monitors the WLM performance metrics (wlmd statistics), filters them, and relays them to HP PerfView via Data Source Integration (DSI).
An excellent description of DSI integration is given in "Using Glance in Advisor Only Mode to Collect and Log Diagnostic Metrics" by Denise Morris. See the References section below for information on obtaining this white paper. The general methods and steps outlined there will be followed for the specific WLM statistics case.
All the example and generated files in this white paper are located in their default installation directory. It is strongly recommended that during your own implementation you store all required files in some local directory that is preserved through reinstallation or upgrade of software specified in the next section. Otherwise, files may/will get lost or overwritten after a reinstall or upgrade, resulting in loss of work/data.
|
3 Requirements
- HP-UX Workload Manager (WLM) A.02.00 B8843CA
Contains wlmd and the perl script /opt/wlm/examples/dsi/wlmdstats/wlmdstats2dsi.pl
- HP GlancePlus/UX Pak B3701AA
Contains Data Source Integration (DSI) tools dsilog, sdlcomp, export, extract
- HP PerfView Software/UX
Contains all PerfView utilities needed to draw and analyze PerfView graphs
- perl 5
Used to power the wlmdstats2dsi.pl filter script. The script assumes that the perl interpreter is located under /opt/perl/bin. If that is not the case on your system, you can always create a symbolic link /opt/perl that will point to your actual perl directory. Available from a multitude of sources, including http://www.hp.com/go/softwaredepot, http://www.perl.org, or http://www.cpan.org.
- root access
All steps outlined below are done as superuser. All the commands are run on a machine named host running HP-UX 11.0.
|
| Figure 1 shows how the data from the WLM file wlmdstats is processed so it can be displayed graphically in PerfView. |
| Figure 1: normal wlmdstats to DSI data flow |
4 Example configuration file
This paper uses a WLM configuration file named config.wlm, that can be found under the /opt/wlm/examples/dsi/wlmdstats directory. The configuration file has four service-level objectives (SLOs) defined. Each SLO implements a different feature of WLM so that different types of logging information are written into the wlmdstats log file. The example configuration file does not try to implement any real business scenario. It's used only as a simple example.
|
A quick inspection of the file shows that there are four SLOs. The SLOs are listed below in priority order (value of the pri keyword in the slo structures):
- slo_usage has a usage goal. It will try to keep the CPU usage between 60% to 90% of the CPU shares that have been allocated to it. The goal is satisfied as long as usage is not more than 90%.
- slo_goal is a goal-based SLO that uses a metric. The goal of this SLO is to keep the value of metric mymetric less than 45.0. The metric value can be sent by using the wlmsend command (wlmsend ). For example:
% /opt/wlm/bin/wlmsend mymetric 10.5
sends 10.5 as the value of the metric to WLM. As long as the metric value is less than 45.0, the goal is satisfied.
- slo_cshar is a cpushares type SLO. This SLO will be given the default minimum CPU shares first, and then based on the value of the metric--additional shares will be given.
- slo_fixed requests a fixed CPU entitlement. It has no goal.
NOTE: These different types of SLOs are included to demonstrate the types of WLM goals and metrics and their interaction with the Measureware tools. The combination of the four SLO types all in a single file is more complicated than most real-world WLM configuration files.
5 WLM statistics logging
5.1 Enabling wlmd logging
WLM logs its input metrics, SLO names, and output PRM entitlements when wlmd is invoked with the -l all flag. See the manpage wlmd(1M) for more information on the -l flag. For our example, the command
% /opt/wlm/bin/wlmd -l all -a config.wlm
starts wlmd and logs the statistics to the file /var/opt/wlm/wlmdstats file at every wlm_interval. The wlm_interval is specified inside the WLM configuration file. To log statistics less frequently, such as every third wlm_interval, you can do the following:
% /opt/wlm/bin/wlmd -l all=3 -a config.wlm
| 5.2 Viewing the wlmd log Statistics about wlmd metrics and entitlements are logged in /var/opt/wlm/wlmdstats. To watch these changing statistics, run the following command: |
% tail -f /var/opt/wlm/wlmdstats
| A sample section of a /var/opt/wlm/wlmdstats file is in the Files section at the end of this document. Statistics for a single wlm_interval look like this: |
1085523324 05/25/04 17:15:24 (p21961) wlm_interval statistics start
1085523324 host=localhost numcpu=6 interval=30
1085523324 metric=_CPU_g4 value=0.000000 fresh=1 pid=-1
1085523324 metric=mymetric value=nan fresh=0 pid=21963
1085523324 grp=OTHERS id=1 cpuent=42.000 cpuuse=0.083 mement=0.000 grpactive=1
1085523324 grp=g2 id=2 cpuent=15.000 cpuuse=0.000 mement=0.000 grpactive=1
1085523324 grp=g3 id=3 cpuent=33.000 cpuuse=0.000 mement=0.000 grpactive=1
1085523324 grp=g4 id=4 cpuent=5.000 cpuuse=0.000 mement=0.000 grpactive=1
1085523324 grp=mygrp id=5 cpuent=5.000 cpuuse=0.000 mement=0.000 grpactive=1
1085523324 SLO=slo_usage sloactive=1 goaltype=usage lowgoal=60.000000 highgoal=
90.000000 goalsatis=1 met=0.000000 mementitl=0 cpuentitl=5 clipped=0 controlling=0
pri=1 ask=5.000000 got=5.000000 grp=4
1085523324 SLO=slo_goal sloactive=1 goaltype=metric goal=45.000000 goalsatis=1
met=nan metfresh=0 mementitl=0 cpuentitl=33 clipped=0 controlling=1 pri=2
ask=32.500000 got=33.000000 grp=3 coll=21963
1085523324 SLO=slo_cshr sloactive=1 goaltype=shares additive=1 goalsatis=1
satisfied=1 value=0.250000 met=nan metfresh=0 mementitl=0 cpuentitl=5 clipped=0
controlling=0 pri=3 ask=5.000000 got=5.000000 grp=5 coll=21963
1085523324 SLO=slo_fixed sloactive=1 goaltype=nogoal goal=nan goalsatis=1 met=nan
metfresh=0 mementitl=0 cpuentitl=15 clipped=0 controlling=1 pri=3 ask=15.000000
got=15.000000 grp=2 coll=0
1085523324 05/25/04 17:15:24 (p21961) wlm_interval statistics end
The format is a start line with timestamp, a series of lines, each with information about one SLO, grp, metric, or host, and then an end line with timestamp. The group of SLO, grp, metric, or host data are referred to in the wlmdstats2dsi documentation as entities.
| 6 The filter: wlmdstats2dsi.pl Now that a stats log file is being produced, the data must be modified to match the dsilog(1) input format. The simple perl script wlmdstats2dsi.pl runs constantly, tracking updates to the wlmdstats file and modifying the one-entity-per-line format into all-entity-on-one-line dsilog input. In order to do that, the script must know what keys of the wlmdstats file you are interested in. Indicate this information by specifying the entity names and key names (separated by a space) in a file. A sample key file wlmdstats2dsi.keys is in the Files section at the end of this document. It looks like the following:
|
slo_usage SLO lowgoal highgoal met cpuentitl
slo_goal SLO goaltype goal met cpuentitl goalsatis
slo_cshr SLO goaltype additive value met cpuentitl
slo_fixed SLO goaltype cpuentitl
mygrp grp id cpuent cpuuse mement
mymetric metric value fresh pid
You can see a demonstration of this transformation by running wlmdstats2dsi.pl without piping its output into any other command:
% /opt/wlm/examples/dsi/wlmdstats/wlmdstats2dsi.pl wlmdstats2dsi.keys /var/opt/wlm/wlmdstats
slo_usage 60.000000 90.000000 0.000000 5 slo_goal metric 45.000000 10.500000 15 1
slo_cshr shares 1 0.250000 10.500000 8 slo_fixed nogoal 15 mygrp 5 8.000 0.000 0.000 mymetric 10.500000 1 21963
slo_usage 60.000000 90.000000 0.000000 5 slo_goal metric 45.000000 10.500000 15 1
slo_cshr shares 1 0.250000 10.500000 8 slo_fixed nogoal 15 mygrp 5 8.000 0.000 0.000 mymetric 10.500000 0 21963
slo_usage 60.000000 90.000000 0.000000 5 slo_goal metric 45.000000 10.500000 15 1
slo_cshr shares 1 0.250000 10.500000 8 slo_fixed nogoal 15 mygrp 5 8.000 0.000 0.000 mymetric 10.500000 0 21963
Note that start and end lines were removed, only values specified in the key file were reported, and all SLO lines were combined in a single line. Careful examination also shows that the order of the entities in the output matches the order in the .keys file.
If you encounter a message like
Nonnumeric value 'nan' encountered(met=nan) Please confirm keys file wlmdstats2dsi.keys
and that your data collectors are working properly.
then run
% wlminfo metrics
Tue May 25 17:27:23 2004
Metric Name PID State Value
_CPU_g4 21988 NEW 0.000000
mymetric 21990 INIT nan
You probably have one or more metrics that have no value yet. In the example case, this means the mymetric has not been given any value yet. Use wlmsend to supply one:
% /opt/wlm/bin/wlmsend mymetric 10.5
and then check again:
% wlminfo metrics
Tue May 25 17:29:53 2004
Metric Name PID State Value
_CPU_g4 21988 NEW 0.000000
mymetric 21990 NEW 10.500000
7 DSI and dsilog
We recommend copying all the files from /opt/wlm/examples/dsi/wlmdstats into your own directory so that none of the installed files get overwritten, and you don't lose your modified files due to a reinstall or upgrade of WLM.
7.1 Creating, compiling the specification file Now that our perl filter has removed unwanted information and combined all SLOs into one line, we can pipe that data stream into a dsilog process. However, DSI needs to have some metadata to define the number, type, and labels for the columns it will be receiving, over what period the data will be summarized, and how long it should be saved on the system before old data is discarded.
These various bits of metadata are defined in a class specification file, which is consumed by the sdlcomp tool to produce a DSI datafile. You can generate the class specification file (or spec file) using the wlmdstats2dsi.pl script. However, because the script writes the output to stdout, you will need to redirect it to a file. The spec file for our example is wlmdstats.spec, which is in the Files section at the end of this document. This file was generated using the following command:
|
% /opt/wlm/examples/dsi/wlmdstats/wlmdstats2dsi.pl wlmdstats2dsi.keys > wlmdstats.spec
The script uses SLO names to create metric names and labels in the spec file. The resulting metric name or label must not be more than 20 characters long; otherwise sdlcomp will fail to compile the generated spec file. If your SLO name is X characters long and the name of the key you want to monitor is Y characters long, then make sure that X+ Y < 20. If you cannot keep the SLO names short then, as an alternative, you can modify strings in the generated spec file that cause sdlcomp to fail. Also remember that if you make changes in your WLM configuration file in the future, you may need to create a new spec file and recompile it with sdlcomp.
The generated spec file can be modified to change values like MAX INDEXES, RECORDS PER HOUR, Label, etc depending on your specific business needs. Consult the MeasureWare document Data Source Integration Guide (B4967-90048 / E0901) for more information on building a spec file. This title can be accessed on the web by browsing the Network and Systems Management library at HP Docs. Also, you can find the PDF version of this document under /opt/perf/paperdocs/mwa/C/.
Now that we have a class specification file, we need to compile it with sdlcomp. The sdlcomp command compiles the spec file to produce the wlmdstats.gpdata file. The following example shows this compilation:
|
% mkdir /var/opt/perf/dsisources
% /opt/perf/bin/sdlcomp wlmdstats.spec /var/opt/perf/dsisources/wlmdstats.gpdata
% ls -l /var/opt/perf/dsisources/wlmdstats.gpdata*
-rw-rw-r-- 1 root sys 3268 May 25 17:35 /var/opt/perf/dsisources/wlmdstats.gpdata
-rw-rw-r-- 1 root sys 60384 May 25 17:35 /var/opt/perf/dsisources/wlmdstats.gpdata.WLMDSTATS
-rw-rw-r-- 1 root sys 30512 May 25 17:35 /var/opt/perf/dsisources/wlmdstats.gpdata.desc
Note that although three files are actually produced, they are collectively referred to as the wlmdstats.gpdata 'DSI datafile' or 'MWA logfile set'. Figure 2 shows this compilation step. 
Figure 2: creation of DSI log file
|
| 7.2 Confirming DSI datafile attributes To confirm that the newly created log file set has the correct labels, fields, and types, use the sdlutil command as shown below: |
% /opt/perf/bin/sdlutil /var/opt/perf/dsisources/wlmdstats.gpdata -stats
# Self Describing Logfile Set: /var/opt/perf/dsisources/wlmdstats.gpdata
**** METRICS ****
X-axis item : (0) DATE_TIME
Metric Name ID Metric Label Type Min Max Scale Summarization
DATE_TIME 0* DATE & TIME TIME STAMP
METRIC_NAME -1* NAME TEXT 41 bytes
METRIC_LABEL -2* LABEL TEXT 41 bytes
METRIC_HELPTAG -3* HELPTAG TEXT 41 bytes
METRIC_ANNOTATION -4* ANNOTATION TEXT 65 bytes
METRIC_ID -5* ID NUMERIC 0 0 1 AVERAGE
METRIC_TYPE -6* TYPE NUMERIC 0 0 1 AVERAGE
METRIC_MINIMUM -7* MINIMUM NUMERIC 0 0 1 AVERAGE
METRIC_MAXIMUM -8* MAXIMUM NUMERIC 0 0 1 AVERAGE
METRIC_SUMMARIZATION -9* SUMMARIZATION NUMERIC 0 0 1 AVERAGE
METRIC_SCALE_FACTOR -10* SCALE FACTOR NUMERIC 0 0 1 AVERAGE
METRIC_SIZE -11* SIZE NUMERIC 0 0 1 AVERAGE
OBJECT_ID_LEN -12* OBJ ID LEN NUMERIC 0 0 1 AVERAGE
OBJECT_ID -13* OBJ ID NUM LIST 0 0 1 AVERAGE
* predefined metric id
Data class control information offset: 4644
Creation time: 05/25/04, 17:35:24
Modify time: 05/25/04, 17:35:26
I N D E X E S
D A T A R O W S
Data Offset First@ Last@ Next@
788 788 25412 25868
Capacity: UNLIMITED; Written: 55; Current: 55
Last X metric value: 01/14/04, 11:14:00
Roll interval: NONE; Roll action:
A large amount of information about the format and metadata for the MWA logfile is displayed. Use the listing to confirm that the metric-related fields match your specification file, both in number and type.
7.3 Running dsilog The MWA logfile set is now initialized and ready to receive real data. Run the following command to start collecting wlmd statistics in the MWA logfile set:
% wlmdstats2dsi.pl wlmdstats2dsi.keys /var/opt/wlm/wlmdstats | \
/opt/perf/bin/dsilog \
/var/opt/perf/dsisources/wlmdstats.gpdata wlmdstats &
To debug this step, you may wish to explore the -vi and -vo flags to dsilog. Let this run for several minutes.
7.4 Confirming output To confirm that real data is being collected, run extract to see, in plain text, the collected values in /var/opt/perf/dsisources/wlmdstats.gpdata. The output is saved to a file. The contents of the file should look similar to the example below, varying some depending on command-line flags and what value is sent to the mymetric discussed in the "Example configuration file" section):
% /opt/perf/bin/extract \
-C wlmdstats DETAIL \
-xp \
-l /var/opt/perf/dsisources/wlmdstats.gpdata \
-f ./extract.out
% cat ./extract.out
DATE &
TIME slo_usage_SLO
05/19/04 18:15:00 slo_usage
slo_usage_lowgoal slo_usage_highgoal slo_usage_met slo_usage_cpuentitl slo_goal_SLO
60.00 90.00 0.00 5 slo_goal
slo_goal_goaltype slo_goal_goal slo_goal_met slo_goal_cpuentitl slo_goal_goalsatis slo_cshr_SLO
metric 45.00 12.50 15 1 slo_cshr
slo_cshr_goaltype slo_cshr_additive slo_cshr_value slo_cshr_met slo_cshr_cpuentitl slo_fixed_SLO
shares 1 0.25 12.50 9 slo_fixed
slo_fixed_goaltype slo_fixed_cpuentitl mygrp_grp
nogoal 15 mygrp
mygrp_id mygrp_cpuent mygrp_cpuuse mygrp_mement mymetric_metric
5 9.00 0.00 0.00 mymetric
mymetric_value mymetric_fresh mymetric_pid
12.50 0 11630
7.5 Exporting DSI to other applications
The extract(1) tool is very useful to pull flat text or data formatted for a spreadsheet from the MeasureWare Agent log files. Figure 3 shows the data flow for this process. 
Figure 3: optional extraction of text or spreadsheet data from DSI datafile
|
7.6 Setting up the start/kill scripts
If you'd like the dsilog setup to be run each time the machine is booted, copy the example start/stop file and set up symbolic links in the /sbin/rc?.c directories. The example start/stop files /sbin/init.d/wlmdstats and /etc/rc.config.d/wlmdstats are included in the Files section at the end of this document.
|
% cp /opt/wlm/examples/dsi/wlmdstats/sbin_init.d_wlmdstats /sbin/init.d/wlmdstats
% cp /opt/wlm/examples/dsi/wlmdstats/etc_rc.config.d_wlmdstats /etc/rc.config.d/wlmdstats
Review the settings in /etc/rc.config.d/wlmdstats to confirm that the parameters match your file locations, edit if necessary and then create symbolic links for the HP-UX boot start/kill facility:
% ln -s /sbin/init.d/wlmdstats /sbin/rc2.d/K001wlmdstats
% ln -s /sbin/init.d/wlmdstats /sbin/rc2.d/S999wlmdstats
You can then test the script setup with the following commands:
% /sbin/init.d/wlmdstats stop
wlmdstats2dsi.pl stopped
% /sbin/init.d/wlmdstats start
wlmdstats2dsi.pl started
For more information on start/kill scripts, see rc(1) or /usr/share/doc/start_up.txt on an HP-UX 11.11 system.
Note that for the logging to work, wlmd must be run with the -l all flag, so if you are starting wlmd with the /sbin/init.d/wlm script, you'll need to WLM_STATS_LOGGING set to "-l all" in /etc/rc.config.d/wlm.
8 Viewing the data with PerfView
8.1 MWA repository server registry
To have the MeasureWare agent automatically start a repository server daemon on your new WLMDSTATS DSI collection:
Add this line to the /var/opt/perf/perflbd.rc file:
DATASOURCE=WLMDSTATS LOGFILE=/var/opt/perf/dsisources/wlmdstats.gpdata
Restart the MeasureWare system:
% mwa restart
Wait a few minutes, then confirm that the repository server for wlmdstats has started:
% perfstat | grep -i wlmdstats
Running rep_server 1908 WLMDSTATS
Running dsilog 1640 /var/opt/perf/dsisources/wlmdstats.gpda
Confirm that the new metrics are available for graphing by using the the /opt/perf/bin/repinfo command:
% /opt/perf/bin/repinfo -s host | grep -i wlmdstats
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:slo_usage_lowgoal
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:slo_usage_highgoal
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:slo_usage_met
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:slo_usage_cpuentitl
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:slo_goal_goal
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:slo_goal_met
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:slo_goal_cpuentitl
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:slo_goal_goalsatis
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:slo_cshr_additive
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:slo_cshr_value
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:slo_cshr_met
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:slo_cshr_cpuentitl
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:slo_fixed_cpuentitl
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:mygrp_id
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:mygrp_cpuent
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:mygrp_cpuuse
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:mygrp_mement
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:mymetric_value
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:mymetric_fresh
host:WLMDSTATS:WLMDSTATS:WLMDSTATS:mymetric_pid
8.2 Creating a graph
There are several ways to build graphs. To start, run the PerfView command pv specifying the normal SCOPE metric set and the new DSI supplied metric set, WLMDSTATS:
% pv -ds host:SCOPE -ds host:WLMDSTATS
The SCOPE metric set supplies various system metric values available from HP-UX. If you do not need to use any of them, you can run PerfView only with the WLMDSTATS metric set:
% pv -ds host:WLMDSTATS
Using the Graphs->Create New Graph... menu, you can interactively build a new graph. You can also specify metric names directly on the command line or in a command file. For more information on the latter, see /opt/perf/paperdocs/pv/C/pvcmdline.html The graph we produced, using a subset of the available wlmdstats metric, is shown below:
| 8.3 Customizing a graph You can then use the Settings->Change Time Settings... menu to zoom in to particular times, or the Settings->Change Visual Settings... to change the layout, colors, etc. of the graph presentation. Here is a graph similar to the first that was resized, zoomed to specific times, had its background color and one line color changed for better visibility, and had grids added: |
9 Configuring PerfView alarms on WLMDSTATS data
9.1 The alarmdef file
You can add alarms to the /var/opt/perf/alarmdef file now, based on combinations of SCOPE and WLMDSTATS metrics.
In this example, we will create alarms in the new file /var/opt/perf/dsisources/alarmdef.wlmdstats and add a include /var/opt/perf/dsisources/alarmdef.wlmdstats to the end of the standard /var/opt/perf/alarmdef file to enable our new alarms.
9.2 Viewing available metric names
To see the names of all metrics available for alarms, use the /opt/perf/bin/repinfo command as above, but specify the -mwa flag to dump in MeasureWare alarmdef format. The following command looks for the wlmdstats metrics we've added via DSI:
% /opt/perf/bin/repinfo -mwa -s host | grep -i wlmdstats
WLMDSTATS:WLMDSTATS:DATE_TIME
WLMDSTATS:WLMDSTATS:SLO_USAGE_SLO
WLMDSTATS:WLMDSTATS:SLO_USAGE_LOWGOAL
WLMDSTATS:WLMDSTATS:SLO_USAGE_HIGHGOAL
WLMDSTATS:WLMDSTATS:SLO_USAGE_MET
WLMDSTATS:WLMDSTATS:SLO_USAGE_CPUENTITL
WLMDSTATS:WLMDSTATS:SLO_GOAL_SLO
WLMDSTATS:WLMDSTATS:SLO_GOAL_GOALTYPE
WLMDSTATS:WLMDSTATS:SLO_GOAL_GOAL
WLMDSTATS:WLMDSTATS:SLO_GOAL_MET
WLMDSTATS:WLMDSTATS:SLO_GOAL_CPUENTITL
WLMDSTATS:WLMDSTATS:SLO_GOAL_GOALSATIS
WLMDSTATS:WLMDSTATS:SLO_CSHR_SLO
WLMDSTATS:WLMDSTATS:SLO_CSHR_GOALTYPE
WLMDSTATS:WLMDSTATS:SLO_CSHR_ADDITIVE
WLMDSTATS:WLMDSTATS:SLO_CSHR_VALUE
WLMDSTATS:WLMDSTATS:SLO_CSHR_MET
WLMDSTATS:WLMDSTATS:SLO_CSHR_CPUENTITL
WLMDSTATS:WLMDSTATS:SLO_FIXED_SLO
WLMDSTATS:WLMDSTATS:SLO_FIXED_GOALTYPE
WLMDSTATS:WLMDSTATS:SLO_FIXED_CPUENTITL
WLMDSTATS:WLMDSTATS:MYGRP_GRP
WLMDSTATS:WLMDSTATS:MYGRP_ID
WLMDSTATS:WLMDSTATS:MYGRP_CPUENT
WLMDSTATS:WLMDSTATS:MYGRP_CPUUSE
WLMDSTATS:WLMDSTATS:MYGRP_MEMENT
WLMDSTATS:WLMDSTATS:MYMETRIC_METRIC
WLMDSTATS:WLMDSTATS:MYMETRIC_VALUE
WLMDSTATS:WLMDSTATS:MYMETRIC_FRESH
WLMDSTATS:WLMDSTATS:MYMETRIC_PID
9.3 Creating the alarmdef file
To demonstrate alarms, we've created an alarm definition that watches the reported value of the mymetric used inside the config file. If the value exceeds the goal for more than 5 minutes, an alarm is issued. The alarmdef.wlmdstats file contents are in the Files section at the end of this document.
|
9.4 Enabling an alarm
Append a line like this one:
include "/var/opt/perf/dsisources/alarmdef.wlmdstats"
to end of the /var/opt/perf/alarmdef file to include the new definition we've created.
Then restart the alarm system to enable the new alarm:
% mwa restart alarms
NOTE: Sometimes it may be necessary to exit all running copies of PerfView and perform the more 'forceful' mwa restart:
% mwa restart
You can check that the alarm subsystems are running with the command:
% perfstat
Confirm from the alarmgen and agdbserver utilities that the alarm subsystem is working. If not, use perfstat -t to see mwa error messages.
9.5 Alarm results In this simple case, alarms are delivered by e-mail. Because the 'alarm' keyword is used in the /var/opt/perf/dsisources/alarmdef.wlmdstats file, OpenView has access to the alarm as well.
9.6 Historical data against alarms
It is possible to process old system data against current alarms to look for past problems. The following command scans old system data and checks for past alarm violations:
% /opt/perf/bin/utility -D -xa
10 References
The following web sites are referenced in this paper:
|
11 Hardware and Software configuration
HP-UX Workload Manager Version A.02.00 was used for this paper. Hardware and software information about the platform used is listed below. This is useful for comparison if your performance tools' output differs from the listed output.
11.1 /opt/perf/bin/perfstat output
% /opt/perf/bin/perfstat -v -c
**********************************************************
** perfstat for host on Wed Jun 2 16:53:12 CDT 2004
**********************************************************
listing version strings for performance tool files:
--------------------------------------------------
NOTE: The following software version information can be compared
with the version information shown in the /opt/perf/ReleaseNotes file(s).
pv C.03.10.00 06/12/01 HP-UX 10.20+
pvalarmd C.03.10.00 06/12/01 HP-UX 10.20+
pvmapd C.03.10.00 06/12/01 HP-UX 10.20+
pvtemputil C.03.10.00 06/12/01 HP-UX 10.20+
pvsysdb C.03.10.00 06/12/01 HP-UX 10.20+
pvcomm C.03.10.00 06/12/01 HP-UX 10.20+
pvalarmxp C.03.10.00 06/12/01 HP-UX 10.20+
repinfo C.03.10.00 06/12/01 HP-UX 10.20+
MeasureWare executables in the directory /opt/perf/bin
scopeux C.03.71.00 03/20/03 HP-UX 11
ttd C.03.71.00 03/20/03 HP-UX 11
perflbd C.03.71.00 03/20/03 HP-UX 11
alarmgen C.03.71.00 03/20/03 HP-UX 11
agdbserver C.03.71.00 03/20/03 HP-UX 11
agsysdb C.03.71.00 03/20/03 HP-UX 11
rep_server C.03.71.00 03/20/03 HP-UX 11
extract C.03.71.00 03/20/03 HP-UX 11
utility C.03.71.00 03/20/03 HP-UX 11
mwa C.03.71.00 03/11/03
dsilog C.03.71.00 03/20/03 HP-UX 11
sdlcomp C.03.71.00 03/20/03 HP-UX 11
sdlexpt C.03.71.00 03/20/03 HP-UX 11
sdlgendata C.03.71.00 03/20/03 HP-UX 11
sdlutil C.03.71.00 03/20/03 HP-UX 11
midaemon C.03.71.00 03/20/03 HP-UX 11
Measureware libraries in the directory /opt/perf/lib
libmwa.sl C.03.71.00 03/20/03 HP-UX 11
libarm.sl C.03.71.00 03/20/03 HP-UX 11
libnums.sl C.03.71.00 03/20/03 HP-UX 11
Measureware metric description file in the directory /var/opt/perf
metdesc C.03.71.00 03/20/03
All critical MeasureWare files are accessible
GlancePlus executables in directory /opt/perf/bin
glance C.03.71.00 03/20/03 HP-UX 11
gpm C.03.71.00 03/20/03 HP-UX 11
Support executables in the directory /opt/perf/bin
perfstat S.00.00.05 03/20/03
breakout C.03.71.00 03/20/03
************* (end of perfstat -v output) ****************
system configuration information:
uname -a: HP-UX host B.11.11 U 9000/800 1279364181 unlimited-user license
model: 9000/800/N4000-75
<--many lines deleted-->
************* (end of perfstat -c output) ****************
12 Files
This section contains various files and sections of files referenced in this paper. The files/section are:
|
12.1 config.wlm
#################################################################
# (C) Copyright 2002-2005 Hewlett-Packard Development Company, L.P.
#
# $RCSfile: config.wlm,v $
# $Date: 2005/03/08 19:50:51 $
# $Revision: 1.9 $
#
# Contents:
# This is an example HP-UX WLM configuration file that
# contains different types of SLOs.
#
# Using this configuration:
# You can activate this configuration with the following command:
# wlmd -a config.wlm
# Use the following command to activate the configuration
# with logging enabled:
# wlmd -a config.wlm -l slo=1,metric=1
################################################################
# PRM structure is used to declare workload groups.
prm {
groups = g2 : 2,
g3 : 3,
g4 : 4,
mygrp : 5;
apps = g4 : /opt/perl/bin/perl wlmdstats2dsi.pl;
gmincpu = g2 : 5, g3 : 5, g4 : 5, mygrp : 5;
gmaxcpu = g2 : 30, g3 : 60, g4 : 50, mygrp : 20;
}
# Example SLO to illustrate fixed allocation
slo slo_fixed {
pri = 3;
mincpu = 15;
maxcpu = 15;
entity = PRM group g2;
}
# Example SLO to illustrate metric-based goal
slo slo_goal {
pri = 2;
mincpu = 15;
maxcpu = 50;
entity = PRM group g3;
goal = metric mymetric < 45.0;
}
# Example SLO to illustrate usage goal
slo slo_usage {
pri = 1;
mincpu = 5;
maxcpu = 50;
entity = PRM group g4;
goal = usage _CPU 60 90;
}
# Example SLO to illustrate cpushares
slo slo_cshr {
pri = 3;
entity = PRM group mygrp;
cpushares = 0.25 more per metric mymetric;
}
# Tune structure for mymetric
tune mymetric {
# type 'wlmsend ' from the command prompt to
# send values to this metric. e.g.
# wlmsend mymetric 12.5
# the metric will have a value of 0.0 if no value is sent
coll_argv = wlmrcvdc;
}
# Global tune structure
tune {
# calculate entitlement every 5 sec
wlm_interval = 5;
# trim /var/opt/wlm/wlmdstats file when it exceeds 20M
wlmdstats_size_limit = 20;
# send any stderr output to syslog
coll_stderr = syslog;
}
12.2 wlmdstats2dsi.keys
#################################################################
# (C) Copyright 2002-2005 Hewlett-Packard Development Company, L.P.
#
# $RCSfile: wlmdstats2dsi.keys,v $
# $Date: 2005/02/09 20:10:18 $
# $Revision: 1.6 $
#
# Contents:
# This is a sample file that provides the SLO names, grp names, and
# keys to gather data from the wlmdstats file. The format is:
# slo_name key1 key2 key3 key4 ...
#
################################################################
slo_usage SLO lowgoal highgoal met cpuentitl
slo_goal SLO goaltype goal met cpuentitl goalsatis
slo_cshr SLO goaltype additive value met cpuentitl
slo_fixed SLO goaltype cpuentitl
mygrp grp id cpuent cpuuse mement
mymetric metric value fresh pid
12.3 Sample section of a /var/opt/wlm/wlmdstats file
1014940183 02/28/02 17:49:43 (p18260) wlm_interval statistics start
1014940183 SLO=slo_usage sloactive=1 goaltype=usage lowgoal=60.000000 highgoal=90.000000 goa
lsatis=0 met=100.000000 mementitl=0 cpuentitl=27 clipped=0 controlling=1
1014940183 SLO=slo_goal sloactive=1 goaltype=metric goal=45.000000 goalsatis=1 met=0.000000
metfresh=0 mementitl=0 cpuentitl=33 clipped=0 controlling=1
1014940183 SLO=slo_cshr sloactive=1 goaltype=shares additive=1 goalsatis=1 satisfied=1 value
=0.250000 met=0.000000 metfresh=0 mementitl=0 cpuentitl=5 clipped=0 controlling=0
1014940183 SLO=slo_fixed sloactive=1 goaltype=nogoal goal=nan goalsatis=1 met=nan metfresh=0
mementitl=0 cpuentitl=15 clipped=0 controlling=1
1014940183 02/28/02 17:49:43 (p18260) wlm_interval statistics end
1014940213 02/28/02 17:50:13 (p18260) wlm_interval statistics start
1014940213 SLO=slo_usage sloactive=1 goaltype=usage lowgoal=60.000000 highgoal=90.000000 goa
lsatis=1 met=0.000000 mementitl=0 cpuentitl=5 clipped=0 controlling=0
1014940213 SLO=slo_goal sloactive=1 goaltype=metric goal=45.000000 goalsatis=1 met=10.500000
metfresh=1 mementitl=0 cpuentitl=15 clipped=0 controlling=1
1014940213 SLO=slo_cshr sloactive=1 goaltype=shares additive=1 goalsatis=1 satisfied=1 value
=0.250000 met=10.500000 metfresh=1 mementitl=0 cpuentitl=8 clipped=0 controlling=1
1014940213 SLO=slo_fixed sloactive=1 goaltype=nogoal goal=nan goalsatis=1 met=nan metfresh=0
mementitl=0 cpuentitl=15 clipped=0 controlling=1
1014940213 02/28/02 17:50:13 (p18260) wlm_interval statistics end
1014940244 02/28/02 17:50:44 (p18260) wlm_interval statistics start
1014940244 SLO=slo_usage sloactive=1 goaltype=usage lowgoal=60.000000 highgoal=90.000000 goa
lsatis=1 met=0.000000 mementitl=0 cpuentitl=5 clipped=0 controlling=0
1014940244 SLO=slo_goal sloactive=1 goaltype=metric goal=45.000000 goalsatis=1 met=10.500000
metfresh=0 mementitl=0 cpuentitl=15 clipped=0 controlling=1
1014940244 SLO=slo_cshr sloactive=1 goaltype=shares additive=1 goalsatis=1 satisfied=1 value
=0.250000 met=10.500000 metfresh=0 mementitl=0 cpuentitl=8 clipped=0 controlling=1
1014940244 SLO=slo_fixed sloactive=1 goaltype=nogoal goal=nan goalsatis=1 met=nan metfresh=0
mementitl=0 cpuentitl=15 clipped=0 controlling=1
1014940244 02/28/02 17:50:44 (p18260) wlm_interval statistics end
1014940273 02/28/02 17:51:13 (p18260) wlm_interval statistics start
1014940273 SLO=slo_usage sloactive=1 goaltype=usage lowgoal=60.000000 highgoal=90.000000 goa
lsatis=1 met=0.000000 mementitl=0 cpuentitl=5 clipped=0 controlling=0
1014940273 SLO=slo_goal sloactive=1 goaltype=metric goal=45.000000 goalsatis=0 met=60.000000
metfresh=1 mementitl=0 cpuentitl=34 clipped=0 controlling=1
1014940273 SLO=slo_cshr sloactive=1 goaltype=shares additive=1 goalsatis=1 satisfied=1 value
=0.250000 met=60.000000 metfresh=1 mementitl=0 cpuentitl=20 clipped=0 controlling=1
1014940273 SLO=slo_fixed sloactive=1 goaltype=nogoal goal=nan goalsatis=1 met=nan metfresh=0
mementitl=0 cpuentitl=15 clipped=0 controlling=1
1014940273 02/28/02 17:51:13 (p18260) wlm_interval statistics end
12.4 wlmdstats.spec
#
# (C) Copyright 2001-2005 Hewlett-Packard Development Company, L.P.
#
# This software is not supported by HP and is provided for educational
# purposes only. Please see /opt/wlm/examples/DISCLAIMER for
# more information.
#
#-------------------------------------------------
# DSI spec:
#
# definitions for metrics that will be produced from a wlmd,
# perl script combo and piped into dsilog ala
# wlmd -> wlmdstats -> wlmdstats2dsi.pl -> dsilog
#
# This header information is a sample one. You should change it
# according to your need. The 'RECORDS PER HOUR' clause assumes
# that your are generating one log record per minute in the
# wlmdstats file. If your wlm_interval is X sec and you used
# '-l slo=Y' with the wlmd command, your 'RECORDS PER HOUR'
# value should be:
# (3600/(X*Y))
#
CLASS wlmdstats = 2002
LABEL "WLMDSTATS"
INDEX BY HOUR
MAX INDEXES 7440
RECORDS PER HOUR 60
ROLL BY MONTH;
METRICS
############ slo_usage
slo_usage_SLO = 401
Label "slo_usage_SLO"
TYPE TEXT LENGTH 35
;
slo_usage_lowgoal = 402
Label "slo_usage_lowgoal"
PRECISION 2
;
slo_usage_highgoal = 403
Label "slo_usage_highgoal"
PRECISION 2
;
slo_usage_met = 404
Label "slo_usage_met"
PRECISION 2
;
slo_usage_cpuentitl = 405
Label "slo_usage_cpuentitl"
PRECISION 0
;
############ slo_goal
slo_goal_SLO = 406
Label "slo_goal_SLO"
TYPE TEXT LENGTH 35
;
slo_goal_goaltype = 407
Label "slo_goal_goaltype"
TYPE TEXT LENGTH 10
;
slo_goal_goal = 408
Label "slo_goal_goal"
PRECISION 2
;
slo_goal_met = 409
Label "slo_goal_met"
PRECISION 2
;
slo_goal_cpuentitl = 410
Label "slo_goal_cpuentitl"
PRECISION 0
;
slo_goal_goalsatis = 411
Label "slo_goal_goalsatis"
PRECISION 0
;
############ slo_cshr
slo_cshr_SLO = 412
Label "slo_cshr_SLO"
TYPE TEXT LENGTH 35
;
slo_cshr_goaltype = 413
Label "slo_cshr_goaltype"
TYPE TEXT LENGTH 10
;
slo_cshr_additive = 414
Label "slo_cshr_additive"
PRECISION 0
;
slo_cshr_value = 415
Label "slo_cshr_value"
PRECISION 2
;
slo_cshr_met = 416
Label "slo_cshr_met"
PRECISION 2
;
slo_cshr_cpuentitl = 417
Label "slo_cshr_cpuentitl"
PRECISION 0
;
############ slo_fixed
slo_fixed_SLO = 418
Label "slo_fixed_SLO"
TYPE TEXT LENGTH 35
;
slo_fixed_goaltype = 419
Label "slo_fixed_goaltype"
TYPE TEXT LENGTH 10
;
slo_fixed_cpuentitl = 420
Label "slo_fixed_cpuentitl"
PRECISION 0
;
############ mygrp
mygrp_grp = 421
Label "mygrp_grp"
PRECISION 0
;
mygrp_id = 422
Label "mygrp_id"
PRECISION 0
;
mygrp_cpuent = 423
Label "mygrp_cpuent"
PRECISION 2
;
mygrp_cpuuse = 424
Label "mygrp_cpuuse"
PRECISION 2
;
mygrp_mement = 425
Label "mygrp_mement"
PRECISION 2
;
############ mymetric
mymetric_metric = 426
Label "mymetric_metric"
TYPE TEXT LENGTH 35
;
mymetric_value = 427
Label "mymetric_value"
PRECISION 2
;
mymetric_fresh = 428
Label "mymetric_fresh"
PRECISION 0
;
mymetric_pid = 429
Label "mymetric_pid"
PRECISION 0
;
12.5 /sbin/init.d/wlmdstats
#!/sbin/sh
#
# (C) Copyright 2000-2005 Hewlett-Packard Development Company, L.P.
#
# example start/stop script: $File: $
# included as part of HP-UX Workload Manager setup,
# designed to be manually moved to /sbin/init.d/wlmdstats
#
# NOTE: This script is not configurable! Any changes made to this
# scipt will be overwritten when you upgrade to the next
# release of HP-UX.
#
# WARNING: Changing this script in any way may lead to a system that
# is unbootable. Do not modify this script.
#
#
# Start wlmd stats -> dsilog processes
#
PATH=/sbin:/usr/sbin:/usr/bin
export PATH
# defaults--can be overridden in /etc/rc.config.d/wlmdstats
# It is recommended not to use the default paths for your
# actual implementation, since default files will get overwritten
# during reinstall or upgrade of WLM
W2D=/opt/wlm/examples/dsi/wlmdstats/wlmdstats2dsi.pl
W2D_KEYS=/opt/wlm/examples/dsi/wlmdstats/wlmdstats2dsi.keys
WLMD_STATSLOG=/var/opt/wlm/wlmdstats
DSILOG=/opt/perf/bin/dsilog
DSIFILE=/var/opt/perf/dsisources/wlmdstats.gpdata
CLASS=wlmdstats
rval=0
set_return() {
x=$?
if [ $x -ne 0 ]; then
echo "ERROR CODE $x"
rval=1
fi
}
case $1 in
start_msg)
echo "Start wlmdstats to DSI facility"
;;
stop_msg)
echo "Stop wlmdstats to DSI facility"
;;
'start')
if [ -f /etc/rc.config.d/wlmdstats ] ; then
. /etc/rc.config.d/wlmdstats
else
echo "ERROR: /etc/rc.config.d/wlmdstats defaults file MISSING"
fi
# is there a compiled DSI file?
if [ ! -e $DSIFILE ]; then
echo "ERROR: cannot start wlmdstats: DSI target file $DSIFILE does not exist"
exit 2;
fi
# and is there a valid wlmd statistics log file?
if [ ! -e $WLMD_STATSLOG ]; then
echo "ERROR: cannot start wlmdstats: wlmd stats file $WLMD_STATSLOG does not exist"
exit 2;
fi
# and is there a valid key file?
if [ ! -e $W2D_KEYS ]; then
echo "ERROR: cannot start wlmdstats: key file $W2D_KEYS does not exist"
exit 2;
fi
# if all pieces exist, attempt starting them in background
if [ "$WLMDSTATS" -eq 1 -a -x $W2D -a -x $DSILOG ]; then
$W2D $W2D_KEYS $WLMD_STATSLOG 2>&1 | $DSILOG $DSIFILE $CLASS 2>&1 >/dev/null &
# return code from background job not very meaningful, but
# it's the best we can do
set_return
echo "wlmdstats2dsi.pl started"
else
rval=2
fi
;;
'stop')
#
# Determine PID of process(es) to stop
#
pid=`ps -el | awk '( ($NF ~ /wlmdstats2dsi./) && ($4 != mypid) && ($5 != mypid) ){ print $4 }'
mypid=$$ | tr '\012' ' '`
if [ "X$pid" != "X" ]; then
if kill $pid; then
echo "wlmdstats2dsi.pl stopped"
else
set_return
echo "Unable to stop wlmdstats2dsi.pl"
fi
fi
;;
*)
echo "usage: $0 {start|stop}"
;;
esac
exit $rval
12.6 /etc/rc.config.d/wlmdstats
#!/sbin/sh
#
# (C) Copyright 2000-2005 Hewlett-Packard Development Company, L.P.
#
# example start/stop script configuration file: $File: $
# included as part of HP-UX Workload Manager setup,
# designed to be manually moved to /etc/rc.config.d/wlmdstats
#
# wlmdstats to DSI facility setup.
# See wlmd(1m) and /sbin/init.d/wlmdstats
#
# WLMDSTATS: Set to 1 to start wlmdstats to dsilog processes
#
WLMDSTATS=1
DSIFILE=/var/opt/perf/dsisources/wlmdstats.gpdata
CLASS=wlmdstats
12.7 /var/opt/perf/dsisources/alarmdef.wlmdstats
#####################
# (C) Copyright 2000-2005 Hewlett-Packard Development Company, L.P.
# $RCSfile: alarmdef.wlmdstats,v $
# $Date: 2005/02/09 20:10:18 $
# $Revision: 1.5 $
#####################
#################################################
# if reported value for the metric used in slo_goal is greater
# than the goal for more than 5 minutes, give an alarm
alarm WLMDSTATS:WLMDSTATS:SLO_GOAL_MET > WLMDSTATS:WLMDSTATS:SLO_GOAL_GOAL
for 5 minutes
TYPE="WLMDSTATS"
start
IF WLMDSTATS:WLMDSTATS:SLO_GOAL_MET > WLMDSTATS:WLMDSTATS:SLO_GOAL_GOAL then {
RED alert "SLO_GOAL_MET is at",
WLMDSTATS:WLMDSTATS:SLO_GOAL_MET, "%"
exec "echo 'SLO_GOAL_MET> SLO_GOAL_GOAL' | mail root"
}
ELSE
YELLOW ALERT "SLO_GOAL_MET is at",
WLMDSTATS:WLMDSTATS:SLO_GOAL_MET, "%"
REPEAT EVERY 120 MINUTES
IF WLMDSTATS:WLMDSTATS:SLO_GOAL_MET > WLMDSTATS:WLMDSTATS:SLO_GOAL_GOAL then {
RED alert "SLO_GOAL_MET continues at",
WLMDSTATS:WLMDSTATS:SLO_GOAL_MET, "%"
exec "echo 'SLO_GOAL_MET continues at> SLO_GOAL_GOAL' | mail root"
}
ELSE
YELLOW ALERT "SLO_GOAL_MET continues at",
WLMDSTATS:WLMDSTATS:SLO_GOAL_MET, "%"
END
reset alert "SLO_GOAL_MET has returned to acceptable."
HP-UX WLM A.03.02.01 (2006_08_28_16_37_32) hpux_pa
$RCSfile: perfview_howto.html,v $
$Date: 2006/05/08 20:06:48 $
$Revision: 1.17 $ |
|
Printable version
