Sitworld: ITM Protocol Usage and Protocol Modifiers

T

3condors

John Alvord, IBM Corporation

jalvord@us.ibm.com

Follow on twitter

Introduction

Recently I have encountered customers doing a lot of simultaneous SOAP activity. The ITM default communications default setup works very well indeed. I estimate a 98% success rate out of the box. However the default only permits 6 simultaneous SOAP connections and each SOAP connection will time out after 300 seconds. Exceeding these limits caused processing errors such as tacmd login failures.

In March 2010, I documented how and why to update the ITM communications string using Protocol Modifiers. I am re-publishing that here with new information to change the SOAP related configurations. I have also made the document more self contained. Here are two newly documented protocol modifiers.

SOAP Concurrency

HTTP_CONCURRENCY defaults to 8 connections. 2 connections are dedicated to the internal web server listening ports and that leaves 6 that are available for SOAP request. Many tacmd functions uses SOAP. However anyone can write a SOAP process and start processing ITM Data. In addition there are commercial products using SOAP. When the number of SOAP ports needed exceeds the available ports, everything slows down and even tacmd login can time out.

A warning note: I have worked with several cases where customers or some product attempted to get too much data for the hub TEMS to handle. One in particular would have required processing 20 gigabytes an hour of data through the hub TEMS and that just killed normal ITM processing. Please avoid that path by estimating and measuring usage and making sure the TEMS can handle the load.

To increase the number of SOAP to 30 ports use the following modifier: HTTP_CONCURRENCY:32

See below as how to implement this change.

SOAP Maximum Age

HTTP_MAXAGE defaults to 300 seconds. That means a SOAP request will time out after that amount of idle time.  If you are doing anything that will take longer, you will lose control of the process.

To increase the time a SOAP request can remain inactive to 15 minutes use the following modifier: HTTP_MAXAGE:900

ITM Protocol Usage and Protocol Modifiers

Important

Much of the following material was incorporated into the ITM 623 Installation and Setup Guide here. There is additional material here which can be useful in specific circumstances.

Tivoli Monitoring protocol usage and protocol modifiers

http://publib.boulder.ibm.com/infocenter/tivihelp/v15r1/topic/com.ibm.itm.doc_6.2.3/itm623_install472.htm?path=3_0_2_0_7_0_10#protocol_mod

Overview

ITM basic services communications are defined by the KDE_TRANSPORT environmental variable, and after this paragraph that will be the only variable mentioned. At an many stages of ITM development, KDC_FAMILIES was used, but KDE_TRANSPORT is intended to be the successor. At this moment, the two variables are processed identically. However many agent installers are only aware of the earlier variable. In every case examine the agent diagnostics and adopt whatever is being used.

ITM uses other communication protocols such as 1) Portal client to Portal server uses CORBA IIOP after startup, 2) TEPS/WPA/SPA can use ODBC or JDBC to communicate with the warehouse database, and 3) several other cases. Most of ITM to ITM communications uses TCP/IP or SNA [z/OS only] protocols. This document ignores the SNA case.

Modifiers to the protocols are of the form ‘attribute:value’. If they occur first, then they are global in effect if meaningful. If they occur after a protocol and before the next protocol, then the effect is only on that protocol. The protocol names and modifiers are case insensitive although they are presented in upper-case here.

In the Appendix there are references on how to implement the needed changes depending on platform.

KDE_TRANSPORT Structure

KDE_TRANSPORT is a string which lists protocols and modifiers. All protocols are assumed present. A protocol will be activated only if an interface is available for use.

The USE modifier activates a specific protocol if Y is specified, otherwise it deactivates it. The scanning starts with an implicit global USE:Y, meaning that all protocols are assumed to be activated by default. For example:

IP.PIPE PORT:1918 USE:Y IP.SPIPE USE:N

means that IP.PIPE will be available but IP.SPIPE will not. In addition all the unnamed protocols listed later are activated such as IPV6.

If a modifier is specified before any protocol, it generally applies globally. For example, a pool setting first will apply to all protocols.

KDE_TRANSPORT Transmission Control Protocol

TCP is a connection oriented protocol. Connection is made via a port number, which can be 0-65535. The connection continues until the application(s) tear it down.

Here are the protocol names:

IP.PIPE – tcp

IP.SPIPE – secure tcp

IP6.PIPE – ipv6 tcp

IP6.SPIPE – ipv6 secure tcp

The secure protocols are implemented with the Global Secure Toolkit (GSKIT) component.

Here are the tcp protocol modifiers:

PORT

PORT defaults to 1918 for tcp and 3660 for secure tcp. These numbers are registered with the Internet Assigned Numbers Authority, www.iana.org/assignments/port-numbers.

PORT defines the base port number. For a TEMS, the base number is used as the listening port, and is the port that Agents connect to. For agents, ITM default processing opens a listening port at number base+N*4096, where N is 1 to 15. If one is already in use, it tries the next higher iteration. The base port is reserved in case a TEMS gets started later.

For an Agent, the listening port is used for two main purposes: 1) TEMS requesting real-time data from the agent and 2) Agent receiving notifications from the TEMS, such as awareness of a WPA [Warehouse Proxy Agent] re-registration at a new IP address or port number.

Example: IP.PIPE PORT:1918 USE:Y

Note that in this mode of definition, there are a maximum of 15 agents on a server. If this is an issue see the EPHEMERAL modifier section further below.

SKIP and COUNT

SKIP and COUNT modifiers are used to control the port search algorithm. Default search is for baseport+N*4096 with N from 1 to 15. The SKIP modifier forces it to start with N equal to the SKIP value in the above calculation. The COUNT modifier means to only try that number of times, not all the way to 15.

The following is often used with Warehouse Proxy Agent:

IP.PIPE PORT:1918 SKIP:15 COUNT:1 USE:Y

It means that the only port checked will be 1918+15*4096 or 63358. This is useful because it means that Warehouse Proxy Agent (WPA) will have a fixed address even though a TEMS and other agents may be starting up. Having a fixed port number is vital if firewall rules are used. At some future ITM maintenance level, the WPA port will likely be fixed to a high port number and configuration will not be required,

EPHEMERAL

This modifier has three different values.

A value of Y means that the connection to the TEMS listening port will be used for all communications. The agent will not need a listening port. The advantage can be important in cases where firewall rules are used, since fewer ports are used. It is also a way to avoid the 15 agent limitation. The negative side is that if historical data collection is being done, then historical data must either be stored at the TEMS, or there must be a WPA process running on the same server as the TEMS the agent reports to if the historical data is being stored at the Agent. Having a WPA on the TEMS  the agent connects to is considered a best practice in large environments so that is often already the case.

A value of OUTBOUND means the same thing as Y.

A value of INBOUND can be used at a TEMS and it means that every agent connecting to it is configured to ephemeral mode.

POOL

ITM processes use ports for communication within the server, never seen on the outside network. The temporary ports that are used can be controlled with this option.

Examples:

IP.PIPE POOL:50900-51923

IP.UDP POOL:01000-01023 POOL:01024-02048

Note that each pool modifier can specify a maximum of 1024, but you can have multiple such specifications.

Any pool range must fit within the local system defined ranges. On AIX the “no -a” will display current ranges. See this URL for a general overview:

http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html

The POOL control does not limit localhost port usage. See the next topic an environment variable to handle that,

When an ITM process loses contact with the target and then reconnects, a new listening port is created. That follows internet standards. The old port is retained by TCP itself to handle any late arriving, duplicate or fragmented packets. Default timing is to hold the older port for 120 seconds. The new listening port is from the ephemeral pool and is also controlled by the POOL protocol modifier.

KDE_LOOPBACK_POOL

This is not a protocol modifier exactly but it serves a parallel purpose. ITM makes use of a number of loopback [127.0.0.1]  socket connections for intra-ITM process communication and awareness. By default the ports are supplied by TCP with no constraints. When this environment variable is present, the loopback ports are constrained just like the POOL protocol modifier. Here is an example usage:

KDE_LOOPBACK_POOL=50900-51923

The number of ports needed depends on a lot of aspects such has how many ITM processes are running, whether internal web server is running and other things. All ITM processes on that system should be defined identically. The function was introduced in ITM 630 FP6 and here is fuller documentation:

IV78573: TEBS: PORT POOLS FOR LOOPBACK DEVICE SOCKET ALLOCATIONS

KDE_TRANSPORT User Datagram protocol

For ITM, this protocol is not usually best overall. The advantage is a somewhat lower storage requirement. The negatives are 1) less reliability since applications are responsible for error recovery, 2) higher CPU resources are required, and 3) it cannot be used where firewall rules are in place.

Here are the protocol names:

IP.UDP – User Datagram Protocol

IP – Synonym for IP.UDP

IP6.UDP – IP V6 version of IP.UDP

The POOL and PORT modifiers can be used for this case The other modifiers are connection oriented.

KDE_TRANSPORT Hypertext Transfer Protocol

Each ITM process has an internal web server. These define what protocols are used to access the internal web server. The web server gives access to the ITM service console, the portal client, the SOAP server tryout page, and [starting with ITM 6.2.2] the Agent Service Index pages.

Here are the protocol names:

ip.tcp.http – http communications

ip.ssl.https – secure http communications

ip6.tcp.http – ipv6 http communications

ip6.ssl.https – ipv6 secure http communications

Here are the protocol modifiers

HTTP_CONCURRENCY

Defaults to 8 listen pipes. When SOAP is in heavy use this should be increased to a large number such as 16 or 32.

HTTP_MAXAGE

Defaults to 300 seconds. If a SOAP request is expected to be idle for a considerable period, increase this value accordingly.

HTTP_SERVER

Defaults to Y. If set to N then the internal web server is not started.

HTTP_CONSOLE

Defaults to Y. If set to N then the ITM service console is not started.

HTTP

Defaults to 1920. If set to 0, port number is set to a temporary or ephemeral number. That number will be will be controlled by the ip.tcp POOL number if present.

HTTPS

Defaults to 3661 . If set to 0, port number is set to a temporary or ephemeral number. That number will be will be controlled by the ip.ssl POOL number if present.

POOL

The TCP socket protocol also uses temporary ports and the usage is controlled by separate pool control settings. These are not protocols, but are required for the POOL setting:

ip.tcp – pool control for the ip.tcp.http protocol

ip.ssl – pool control for the ip.ssl.https protocol

ip6.tcp – pool control for the ip6.tcp.http protocol

ip6.ssl – pool control for the ip6.ssl.https protocol

It would be easier to have leading pool setting(s) which would apply to all protocols.

Interactions with other environment variables

KDEB_INTERFACELIST and KDEB_INTERFACELIST_IPV6

The dash “-“ option is used alone, these environment variables do not scan any of the related interfaces. In that case a protocol might go unused even though it is specified in KDE_TRANSPORT. You could also eliminate all the interfaces by name to achieve the same result.

Summary

This document explains how to understand and change the KDE_TRANSPORT environment variable to change ITM communication defaults.

Sitworld: Table of Contents

Photo Note: Three California Condors Looking For Trouble

Appendix: A Guide to Changing the KDE_TRANSPORT settings

Most of this document is a reference guide to constructing a KDE_TRANSPORT [or KDC_FAMILIES]. In practice what you do is highly platform dependent and KDC_FAMILIES is often what is needed.

If you decide to change KDE_TRANSPORT, make sure to change the settings on all ITM tasks when appropriate. For example, if you decide to turn off the ITM internal web server using the HTTP_SERVER:N modifier, you must make the same change for all ITM tasks. The task actually running the internal web server can switch depending on which one is up first and on ITM task recycles, so all ITM processes must have the same setting.

That is not true for agent specific operations such as using SKIP/COUNT to force a specific port number to be used by an ITM process.

The following change methods are focused on ITM 623 and earlier. If you need to work with ITM 622 and earlier, here are two references which can be followed.

Updating Linux/Unix agent KDC_FAMILIES configuration

Mass Configuration Change for Windows/Linux/Unix agents

Linux/Unix ITM process

Determine what the communications string is now. Review the diagnostic log [example: ibtems_lz_klzagent_53d40bd3-01.log] and search for KDE_TRANSPORT

…   KDE_TRANSPORT=KDC_FAMILIES=”ip.spipe port:3660 ip use:n ip.pipe use:n sna use:n HTTP:1920

The communications string is

ip.spipe port:3660 ip use:n ip.pipe use:n sna use:n HTTP:1920

In the logs directory you will also find lz.env and reviewing it you should see something like this

KDC_FAMILIES=ip.spipe port:3660 ip use:n ip.pipe use:n sna use:n HTTP:1920

Now you can see that the KDC_FAMILIES is being used in this case,

Next construct an lz.environment file in <installdir>/config. This file will have one line [to start with] and must have the same attributes/owner/group as the lz.ini file. Use the chmod/chown/chgrp commands to make that true. The contents of that line will be the same as the lz.env line with your added protocol modifiers. For example

KDC_FAMILIES=EPHEMERAL:Y ip.spipe port:3660 ip use:n ip.pipe use:n sna use:n HTTP:1920

When the ITM process is recycled, the new control will override what is there.

Note 1: This override will persist even if you do a manual reconfiguration of the agent… for example switching to ip.pipe. Therefore you must remember this permanent override and maintain it into the future.

Note 2: If a global change is needed, like turning off the internal web server

KDC_FAMILIES=HTTP_SERVER:N ip.spipe port:3660 ip use:n ip.pipe use:n sna use:n HTTP:1920

This must be repeated for every ITM process running on the system with separate environment files.

Note 3: The change for a TEMS is almost the same. There will be a leading export because that is required for a TEMS. It will be seen in the ms.env file. TEMS [and TEPS] require the internal web server and so cannot be disabled without crippling the product.

Windows Agents

The supported way to update KDC_FAMILIES uses the MTEMS graphical

interface. From that interface:

1) right-click on task line

2) Select Advanced

3) Select Edit Variables…

4) click on listbox arrow and find KDC_FAMILIES

5) click on Edit

6) You should see a value of @Protocol@. This value is kept in the kin.ini file. In a simple case, you would just insert a modifier before that text and hit OK. In a more complex case you would determine what the full value is and replace it all.

The Windows registry involved is actually

32-bit environment

HKEY_LOCAL_MACHINE\SOFTWARE\Candle\KNT\Ver610\Primary\Environment

64-bit environment

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Candle\KNT\Ver610\Primary\Environment

I worked with one customer who reported that altering the registry was easiest for him. Changing it for one agent changed usages for all agents on that system.

There is an technote example of doing the registry change via a .reg file here.

i/5

The environment variable is manually changed in

QAUTOTMP/KMSPARM(KBBENV)

z/OS

In z/OS these values are kept in the RKANPARU(KDSENV) member. for TEMS and in RKANPARU(KppENV) for Agents. Configuration changes here are manual.

 

2 thoughts on “Sitworld: ITM Protocol Usage and Protocol Modifiers

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: