Sitworld: Eliminating Duplicate Agents

orchid

John Alvord, IBM Corporation

jalvord@us.ibm.com

Draft #1 – 29 May 2020 – Level 0.50000

Follow on twitter

Inspiration

Duplicate agent names in a ITM environment present a multi-dimensional horror. That is when two different agents with the same name are connecting to a remote or hub TEMS. Here are the bad things that can happen:

1) Reduced monitoring because only one agent at a time can report results. As each agent connects, the earlier one is ignored and the new one has to be populated with the right situations. Effectively no agent gets to warn of issues reliably and thus monitoring is at best diminished and and at worst almost eliminated. It can even lead to some historical data always being all zeroes.

2) Each time an agent is seen to change in some way [different IP address, different remote TEMS, different agent type, different agent version number, different listening port etc] the TEPS needs to recalculate the Topology… a representation of what ITM processes are connected to the remote and hub TEMSes. That can be very expensive and duplicate agents cause the recalculation to happen frequently. TEPS performance gets very very bad – visible as Navigator Updates Pending keep showing up.

3) The process of dropping one agent instance and starting up another is a workload drain on the remote TEMS [mostly] and the hub TEMS somewhat. This sometimes causes TEMS instability including crashes.

4) Duplicate agents cause human confusion and wasted work. An alert is seen at an agent [one of a duplicate pair]. An investigation is launched at its duplicate pair and – no trouble found! The issue continues and human work is wasted.

5) Historical data becomes suspect. Historical data involves the agent name and in cases of duplicate agents you get double data that really apply to the pair. More human confusion.

There are lots of reasons to avoid duplicate agents. Until now repairing those cases have required intensive manual work. This blog post automates some of the work and thus reduces manual labor in correcting the condition.

Parallel: If you watch the original Thomas Crown Affair (1968) – the story of an art robbery. The thief is notably dressed in a sharp suit, thin tie, a attache case and a bowler hat etc. In the climatic scene hundreds of hired actors dressed identically are walking at random in the art museum halls and rooms. The thief walks away with the prize. I often thought of this an an excellent parallel to having duplicate agents.

Background – How Duplicate Agent Arise

Here are some of the ways duplicate agents arise. This list is certainly incomplete because new methods are found from time to time. Agents names by default are formed from a 1) a prefix [like the Primary] seen in Windows OS Agents; 2) the hostname; and 3) a Suffix which indicates the agent type. Those are concatenated with colons [:] between the values, the result is truncated to 32 characters at most and that is the agent name. The system hostname can be modified by setting the CTIRA_HOSTNAME and CTIRA_SYSTEM_NAME environment variables to control the hostname and thus the agent name. That is a common theme in recovery actions and won’t be specifically called out in the following examples.

1) The hostname may be so long that the composed agent name is more than 32 characters long. If there are multiple agents on the same system, the truncation may create duplicate agent names even though the agents are of different types.

2) The same type of agent may be installed in different install directories on the same system. In one memorable case, a Unix OS Agent was installed at /opt/IBM/ITM and a few years later a revised Unix OS Agent was installed on 345 systems at  /opt/IBM/ITM/bin. In another case, the customer really wanted two OS Agents on a Linux system and just installed them in two installation directories. In a third case, a separate team installed an agent and was unaware of the existing agent.

3) Two different systems may be created with the same hostname. Thus two agents will be on different systems with the same name.

4) During a remote deploy operation, the old agent is stopped. Sometimes that shutdown fails and the old agent lingers with the same name. This can be identified because the two agents are from the same system and with the same listening port. That is “impossible” for TCP communications so it means the old agent is mostly stopped but is still sending node status updates even though it isn’t doing anything else. The recovery here is to stop the agent manually. Then kill -9 the remaining agent. Then start the agent again.

5) Two agents could have the same CTIRA_HOSTNAME and CTIRA_SYSTEM_NAME settings. Often that happens when a system cloning process fails to adjust the above settings. The worst case seen [so far] was 500+ Windows OS Agents with the same name.

6) The two agents could be in a cluster environment. That means two or more systems which run a single service like a database server… of course the database server only is action on one system at a time. In such a case a monitoring agent is sometimes set to run on all sides of the cluster. The hostname is often the same and that creates the duplicate agents condition. The correct recovery is to 1) start and stop the agent as the system being monitored is started/stopped, 2) use CTIRA_HOSTNAME and CTIRA_SYSTEM_NAME to make sure the agent has the same name and 3) configure KDEB_INTERFACELIST=xx.xx.xx.xx  where xx.xx.xx.xx is the Virtual IP Address for the cluster – a single ip address to access the system [like a database] regardless of which system in the cluster is active. In this way the TEMS sees a single system that occasionally stops and starts and NEVER changes name or ip address.

7) Some agents take on the role of managing agents. For example Tivoli Log Agent can run singly or it can run with multiple subnode agents. Those subnode agents are specified in a configuration list. It is all to possible for two subnode agents, connected to different managing agents, can have the same name. In that case you need to change the managing agent configuration list to ensure unique agent names.

8) The MQ agent has a different way to specify a name. If that is not configured, the agent name will look like XXXXXX::MQ. That can create duplicate agents. The recovery here is to update the agent cfg and add “SET AGENT NAME <hostname>”  or some other unique specifier. That fills in the blank name between the colons.

9) In ITM environments with z/OS agents, you may see some duplicate agents. There is no known way at present of resolving most of those issues.

 

A Largely Automated Recovery Plan

Many of the above cases can only be recovered using manual configuration. The most labor intensive cases arise when there are many of them. It is not uncommon to see hundreds. Some are the simple case of a OS Agent and a Tivoli Log agent, same name on two different systems. Imagine if there were 120 of those pairs. You also see cases of 50 to 500 identical agents. Tackling the 500+ identical Windows OS Agents could take weeks and require access to each system to reconfigure the agents.

The goal of this project is to automate the most painful tasks. This requirement was a paradox for a long time until I recognized this fact:  if you have two duplicate agents, they are almost worthless for all the above reasons. Agent configured hostnames can be changed using a command like this

./tacmd setagentconnection -n <os_agent_name> -t <pc>
-e CTIRA_HOSTNAME=xxxxxxxxx CTIRA_SYSTEM_NAME=xxxxxxxx

Where

os_agent: the agent name of the OS agent on the system
pc: the product code [like NT for Windows OS Agent]
xxxxxxxx: the new hostname.
*note we always change CTIRA_HOSTNAME [ for TEMS awareness by agent name] and CTIRA_SYSTEM_NAME [for TEPS awareness] and make them the same to avoid confusion.

If there were just two such duplicate agents, we could use the revised name as hostname-DUP1. When this completes – one of the two agents will have the old name and one will have a hostname with the characters -DUP1 appended. Thus there will be two different agent names and the duplicate agent name issue is resolved. More work is needed, of course.

Any agent should be integrated into the ITM environment. Every agent is added to a system generated Managed System List like *NT_SYSTEM. The agent name may also have been added to custom Managed System Lists. Finally the agent name may have been added to the direct distribution of some situations. There are also uses of Situation Group direct distribution which is awaiting development [a test case is needed!]

To complete the work the user must

1) Rename the hostname-DUPn agent appropriately.
2) Add the new agent to any custom Managed System Lists where the old agent was used.
3) Add the new agent to the distribution of any situations where the old agent was used.
4) Determine out why it happened and change processes so it doesn’t happen again.

The integration gets a little more lengthy if, say you had 151 duplicate OS Agents. However the benefit is immediate: You get some monitoring going on – which is better than none. You can make all the subsequent changes remotely – for these cases anyway. As earlier there are still cases where manual configuration is needed.

You may also have some cleanup work to do. One customer with two sets of 100+ agents with same name found that the automation partially failed. Some of the attempted changes were rejected because the agent was *OFFLINE. However many changes were successful and the manual work was seriously reduced. You could also repeat the automation process after the first wave of cleanup.

Package Installation

The package is dup2do.0.50000 and contains the dup2do.pl Perl program and a dup2do.cmd for Windows usage.

If this will be run on a Linux/Unix system, Perl is almost always installed in the system. If this will be run on a Windows system, you will have to install a Perl. My current choice is Strawberry Perl.

http://strawberryperl.com/

It is good quality without serious license restrictions. Of course if it will be used on a company system, you will likely want to contact your manager and check with legal.

DUP2DO Process

Unlike many projects, this one needs files created by three other Sitworld projects.

Sitworld: TEMS Audit Process and Tool

The first step is to review the TEMS Audit report file  to see if duplicate agents have been identified. Here is a small extract of a good example:

TEMSREPORT082: Agent Flipping Report - Multiple Systems
Count,Agent,Hostaddrs, 
2,ibm_au_winatca5820:NT,ip.spipe:#9.13.192.26[REMOTE_AUULDPLITM020] ip.spipe:#10.114.95.68[REMOTE_AUBHDPLITM030],
2,ibm_id_cgkdcplesb01a:LZ,ip.spipe:#9.132.101.100[REMOTE_cgkibplitm010] ip.spipe:#10.132.187.63[REMOTE_cgkibplitm010],
2,ibm_id_cgkdcqwcom02:NT,ip.spipe:#9.132.101.30[REMOTE_cgkibplitm020] ip.spipe:#10.132.101.103[REMOTE_cgkibplitm010],
This involves three agent names that are reporting from two different systems each. You see a count, the agent name, the protocol/system and remote TEMS involved.

Use a new recent copy of TEMS Audit 2.25000. Rerun it on your hub TEMS logs directory

perl temsaud.pl -v -logpath /opt/IBM/ITM/logs  -dup

This will produce a file dedup.csv for dup2do. Here is an extract from the same source.

ibm_au_winatca5820:NT,ip.spipe:#9.114.95.68,
ibm_au_winatca5820:NT,ip.spipe:#9.13.192.26,
ibm_id_cgkdcplesb01a:LZ,ip.spipe:#9.132.101.100,
ibm_id_cgkdcplesb01a:LZ,ip.spipe:#9.132.187.63,
ibm_id_cgkdcqwcom02:NT,ip.spipe:#9.132.101.103,
ibm_id_cgkdcqwcom02:NT,ip.spipe:#9.132.101.30,

Copy that file to the directory where dup2do will be run. You can leave it in place and copy in the second and third needed file.

The second file comes from this project

Sitworld: Situation Distribution Report

There is an initial data capture program

Windows – sitinfo.cmd
Linux/Unix – sitinfo.sh [in sitinfo.tar]

Run that as instructed which will create a set of .lst files.

Next run this command

perl sitinfo.pl -lst -onerow

One result will be in sitinfo.csv report file. Here is a small excerpt.

ibm_cpuutil_gntf_gsmabase,Fatal,9.13.192.26,ibm_au_winatca5820:NT,M|ibm_nt_infinity_prod;,*IF *VALUE NT_Processor.%_Processor_Time *GE 95 *AND *VALUE NT_Processor_Summary.High_Process_Name *NE mcshield *AND *VALUE NT_Processor.Processor *EQ ‘_Total’,

ibm_dsp_gntc_win,Critical,9.13.192.26,ibm_au_winatca5820:NT,M|ibm_nt_infinity_prod;,*IF *VALUE NT_Logical_Disk.%_Used *GE 90 *AND *VALUE NT_Logical_Disk.Disk_Name *NE ‘_Total’,

ibm_dsp_gntf_win,Fatal,9.13.192.26,ibm_au_winatca5820:NT,M|ibm_nt_infinity_prod;,*IF *VALUE NT_Logical_Disk.%_Used *GE 95 *AND *VALUE NT_Logical_Disk.Disk_Name *NE ‘_Total’,

As you can see, it names the situation involved, the severity, the system IP address, the agent name, the distribution and the situation formula. The distribution field

M|ibm_nt_infinity_prod;

means it is distributed via a Managed System List [M] and gives the name.

Copy sitinfo.csv to the directory where dup2do will run. That might be in the logs directory.

The third file comes from the

Sitworld: Database Health Checker

That package comes with a shell file datasql.sh [in a datasql.tar container] or datasql.cmd. Follow the post instructions to run that using TEPS batch commands and it will create a file QA1DNSAV.DB.TXT. That is the third file for DUP2DO. It is option and if not present the DUP2DOP.CSV file will be missing.

With those the files present, run the dup2do command

perl dup2do.pl

Options

-dupsleep nnnn     add nnnn seconds after each tacmd setagentconnection. This should usually be 600 seconds if there are cases of 3 or more systems where the duplicate agents are running. After a rename the original agent may not be online for a while. It does mean the process may run for a long time [even overnight], but the results are beneficial.

-dupall  hostname   for this hostname, rename all ITM agents on that system. Default is just the OS Agent. Reviewing the dedup.csv manually. See iff there are cases where the OS Agent is duplicate and also  other agents. In that case you can set the dupall option which means all agents on that system will change hostname with a single command. You can set a dupall option for each such cases. Otherwise you will have to do manual setagentconnection commands for the non-OS Agents on each such agent and system.

The dup2do.pl command creates six output files.

dedup.sh – Linux/Unix shell command to run the needed tacmd commands
following is an example from that command [a single line].

./tacmd setagentconnection -n ibm_au_winatca5820:NT -t NT -e CTIRA_HOSTNAME=ibm_au_winatca5820-DUP1 CTIRA_SYSTEM_NAME=ibm_au_winatca5820-DUP1

dedup.cmd – Windows shell command to run the needed tacmd commands
same as dedup.sh but for windows

dup2do.csv – Report on how to re-integrate the -DUPn agents into the ITM environment.
example lines

MSL,ibm_nt_infinity_prod
add,ibm_au_winatca5820-DUP1:NT,

The new agent name has to be added to the above managed system list. The full report also shows the situations that need the agent name added to the distribution.

dup2do.sh – Linux/Unix shell command to run the needed  commands

Example tacmd which will add in the new agents to the named managed system list. In this case there were actually many new agents needed adding.

./tacmd editsystemlist -e ibm_nt_infinity_prod -a ibm_au_aubhdpwcva001-DUP1:NT ibm_au_aubhdpwcva001-DUP2:NT …

dup2do.cmd – Windows shell command to run the needed commands
same as dup2do.sh but for windows

The last file DUP2DOP.CSV knits together the DEDUP.CSV data and the QA1DNSAV.DB.TXT data. It supplies valuable supporting data when validating what will be happening. The second column “dup” means it came from the DEDUP.CSV. The “msn” means it came from the TEMS database snapshot.

Case 1:  simplest case

9.22.71.58,dup,ibm_s4450024:06,ip.spipe:#9.22.71.58,
9.22.71.58,dup,ibm_s4450024:NT,ip.spipe:#9.22.71.58,

9.22.71.62,dup,ibm_s4450024:06,ip.spipe:#9.22.71.62,
9.22.71.62,dup,ibm_s4450024:NT,ip.spipe:#9.22.71.62,
9.22.71.62,msn,ibm_s4450024:06,Y,06,03.43.00,ip.spipe:#9.22.71.62[11853]<NM>ibm_S4450024</NM>,
9.22.71.62,msn,ibm_s4450024:NT,Y,NT,06.30.07,ip.spipe:#9.22.71.62[7757]<NM>ibm_s4450024</NM>,

The hostname ibm_s4450024 was used in two different systems on a NT agent and a 06 agent. Both systems were called out as duplicate agents. The TEMS database shows that only two are actually online at the moment, as system stole the other’s identity. This is a case where you can add -dupall ibm_s4450024 to the DUP2DO invocation and both NT and 06 agents will be get the new names.

Case 2 – more complicated

9.30.34.38,dup,blinsts:ibm_smlsxls032:UD,ip.spipe:#9.30.34.38,
9.30.34.38,msn,ibm_smlsxlm032:KUL,Y,UL,06.22.02,ip.spipe:#9.30.34.38[15949]<NM>ibm_smlsxlm032</NM>,
9.30.34.38,msn,ibm_smlsxlm032:08,Y,08,03.20.00,ip.spipe:#9.30.34.38[7757]<NM>ibm_smlsxlm032</NM>,
9.30.34.38,msn,ibm_smlsxlm032:LZ,Y,LZ,06.30.07,ip.spipe:#9.30.34.38[11853]<NM>ibm_smlsxlm032</NM>,

9.30.34.39,dup,blinsts:ibm_smlsxls032:UD,ip.spipe:#9.30.34.39,
9.30.34.39,msn,blinsts:ibm_smlsxls032:UD,Y,UD,07.10.00,ip.spipe:#9.30.34.39[15949]<NM>ibm_smlsxls032</NM>,
9.30.34.39,msn,ibm_smlsxlm033:08,Y,08,03.20.00,ip.spipe:#9.30.34.39[7757]<NM>ibm_smlsxlm033</NM>,
9.30.34.39,msn,ibm_smlsxlm033:LZ,Y,LZ,06.30.07,ip.spipe:#9.30.34.39[11853]<NM>ibm_smlsxlm033</NM>,

Here you see a case where the UL/08/LZ agents have proper names and different. But the UD agent has the same hostname on both systems. At this moment, the UD 9.30.34.38 on is not seen on TEMS because the UD on 9.30.34.39 with the same agent name has stolen the identity. In this case someone needs to manually fix up the UD agent on 9.30.34.38 so they two UD Agents will cooperate.

This may actually be a clustering case, where the UD agent(s) should be started and stopped with the DB2 systems, and the ip address forced to the Virtual IBM address that DB2 uses whatever system it is running on.

 

Manual Finishing Work

After the initial dedup.sh [or dedup.cmd] creates non-duplicated agents but with strange names:

Decide on what your site agent names should be. Do another tacmd setagetconnection command to set the agent name as desired. Carefully track the old and new -DUPn names and the eventual site agent names. Run a tacmd cleanms command [or the TEP equivalent] to remove the -DUPn names which show as offline after that process.

Update needed MSLs and Situation Distributions using the dup2do.csv report file or the dup2do.sh [or dup2do.cmd] commands. Track the original name, the -DUPn name and the selected site agent name to make the right choices.

After a hub TEMS recycle and running for while, redo the TEMS Audit and look at the TEMSREPORT082 report section to see what issues if any remain. Usually at this point a manual effort is required.  Here is an example:

 

3,MSSQLSERVER:ibm_my_kuldcpwsql06:,ip.spipe:#9.136.174.49[REMOTE_IBBFDPLITM020] ip.spipe:#9.136.174.30[REMOTE_IBBFDPLITM020] ip.spipe:#9.136.174.48[REMOTE_IBBFDPLITM020],
2,MSSQLSERVER:ibm_sg_sgdcwpwsql01,ip.spipe:#9.105.171.34[REMOTE_sinibplitm030] ip.spipe:#9.105.171.33[REMOTE_sinibplitm030],

The names are so long truncations have occurred. In cases like these you need to examine each system and see what is happening. There could be clustering conflicts happening, but it could be subnode agents. Or maybe you should set the CTIRA_SUBSYSTEM_ID to a shorter value. That sets the first section of the agent name… which defaults to blank except for Windows OS Agent.

 

Summary

This tool and process will ease the effort of detecting and resolving duplicate agent name issues. This action will improve monitoring, reduce TEMS impact, reduce human confusion and help TEPS performance. The benefit is well worth the effort.

Sitworld: Table of Contents

History and Earlier versions

dup2do.0.50000 initial version published

Photo Note: Orchids Galore on the Kitchen Counter

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: