Sitworld: Discovering Historical Data Export Problems at Agent

cruise1

John Alvord, IBM Corporation

jalvord@us.ibm.com

Follow on twitter

Introduction

ITM 6 has a marvelous ability to collect historical data. Best practice is to collect the historical data at the TEMA or Agent and then export the data to the Warehouse Proxy Agent which then forwards the data to the data warehouse. With a large number of agents almost anything can go wrong and require fixing. Identifying the problem cases has been challenging. A few years ago a new TEMA attribute group was added to expose the last export status. This can be used in a situation formula and generate situation alerts to problem cases. This post shows exactly how to do that. An appendix at the end lists all the current status codes. In some cases you can resolve it yourself, in other cases IBM Support will be involved.

Step by Step Situation Development

Right click on a TEP navigation node such as Linux OS under a test Linux System. Select Situations… Click on the new Situation action. Enter a situation name in the dialog box. If not matching what you want, also enter Monitored Application.

export1

Next click OK and define the attribute group [ITM Historical Exports] and attribute item [Last Export Status].

export2

Click OK. For the first experiment set the test to be == 0, meaning alert when things are working as expected.

export3

Click on Advanced,Display Item and select Collection Identifier.

export4

Now make sure situation is distributed to your test system and OK out. The situation should start immediately and in the Situation Event Console you will see

imageexport5

For the next steps you will likely want to test for Last Export Status not equal to zero. Next you will expand the distribution to more agents, like all Linux agents.

Last Export Status – what it means

The appendix has a list of all currently known export status values. 0 means success and can be ignored for this purpose.

One common one is 26

   CTX_MetafileNotfound,            26

In general this means that historical data was configured, but no data was every collected. The example studied closely was some Linux LPAR attribute group, but the Linux system being looked at did not have any LPAR capability. For this site, we recommended that the attribute group should not be collected. Another way to avoid alerting would be to extend the formula to exclude the 26 status.

Other errors may lead to obvious conditions – like an inability for the agent to connect to the WPA or maybe a nearly full mount point. It any case you need to investigate and resolve… with IBM Support if need.

Other possibilites

Another common issue involves unhealthy agents – online but not responding and not running situations. Here is a blog post and program to help track them down:

Sitworld: ITM Agent Health Survey

These cases will not be running the historical data UADVISOR situations, which have the same effect. However they will not alert because no situations are running.

Summary

This shows how to create a situation to alert on some Historical Data collection and export problems.

Sitworld: Table of Contents

Appendix 1: Historical Data Export Error codes

KRAHIST – ITM History Exort – Last Export Status

CTX_Success = 0,                     0

CTX_InvalidParameter,             1

CTX_InvalidOutputFormat,          2

CTX_NoMemory,                     3

CTX_InternalError,                4

CTX_LogonFailed,                  5

CTX_InvalidUserid,                6

CTX_InvalidPassword,              7

CTX_ConnectionFailed,             8

CTX_TargetTypeUndetectable,       9

   CTX_EmailSendFailed = 10,        10

CTX_InvalidRecipType,            11

CTX_SMTPError,                   12

CTX_OLEInitializationFailed,     13

CTX_InvalidInitCall,             14

CTX_SessionStartupFailed,        15

CTX_CMSConnectFailed,            16

CTX_DSError,                     17

CTX_EndOfData,                   18

CTX_InvalidDataType,             19

   CTX_ODBCError = 20,              20

CTX_TableNotFound,               21

CTX_ParmDataLengthError,         22

CTX_InvalidParameterFormat,      23

CTX_InvalidExportType,           24

CTX_MetafileFormatError,         25

CTX_MetafileNotfound,            26

CTX_MetafileIOError,             27

CTX_MetafileCloseError,          28

CTX_HistoryIOError,              29

   CTX_HistoryFileNotfound = 30,    30

CTX_HistoryCloseError,           31

CTX_SocketIOError,               32

CTX_GetHostnameError,            33

CTX_SMTPConnectError,            34

CTX_GetHostByNameError,          35

CTX_GetServByNameError,          36

CTX_SocketError,                 37

CTX_SocketServerResponseError,   38

CTX_ColumnsNotBound,             39

   CTX_ColumnsAlreadyBound = 40,    40

CTX_NotExporterMessage,          41

CTX_InvalidSocketBufferLength,   42

CTX_SocketBufferOverflow,        43

CTX_MessageParseError,           44

CTX_SchemaFormatError,           45

CTX_OLEError,                    46

CTX_RequestRouted,               47

CTX_NoListenTask,                48

CTX_RPCError,                    49

   CTX_SpreadsheetNotFound = 50,    50

CTX_IncompatibleSpreadsheet,     51

CTX_NoObjectResolution,          52

CTX_ServerDied,                  53

CTX_PDSLoadError,                54

CTX_PDSNotAvailable,             55

CTX_RPCRequestHandleError,       56

CTX_PropertyNotFound,            57

CTX_NoProperties,                58

CTX_EnvNo,                       59

   CTX_InitJVMError = 60,           60

CTX_JavaError,                   61

CTX_JDBCError,                   62

CTX_TempFileError,               63

CTX_ColumnLengthError,           64

CTX_PThreadError,                65

CTX_CfgFileError,                66

CTX_ConfigRecNotFound,           67

CTX_SkipRecord,                  68

CTX_InvalidRPCFunction,          69

   CTX_ConfigOpenError = 70,        70

CTX_ConfigCloseError,            71

CTX_RedriveExport,               72

CTX_WarehouseProxyNotRegistered, 73

CTX_GLBUnavailable,              74

CTX_InitializationFailed,        75

CTX_ParseError,                  76

CTX_SQLFileNotFound,             77

CTX_RTNCustIDNotSet,             78

CTX_NotConnected,                79

   CTX_GetJavaVMInitArgsFailed = 80,80

CTX_CreateJavaVMFailed,          81

CTX_GetJavaMethodFailed,         82

CTX_FindJavaClassFailed,         83

CTX_CompressionError,            84

CTX_InvalidOracleODBCDriver,     85

CTX_DataTypenameUnavailable,     86

CTX_RPCInterfaceRegisterError,   87

CTX_RouteNotifyMismatch,         88

CTX_PDSOpenInputNoRecords,       89

   CTX_NotFound = 90,               90

CTX_NameConversionFailed,        91

CTX_NameCompatibility,           92

CTX_HistorySaveError,            93

CTX_HistoryDeleteError,          94

CTX_MetaSaveError,               95

CTX_MetaDeleteError,             96

CTX_PDSSetupError,               97

CTX_DecodeError,                 98

CTX_BeginExport,                 99

   CTX_ExportInProgress = 100,     100

CTX_NoDataFound,                101

CTX_ShutdownRequested,          102

   CTX_UCS2TranslationError = 200, 200

CTX_InvalidDatabaseEncoding,    201

CTX_ColumnNotFound,             202

CTX_ExistAlready,               203

CTX_InvalidList,                204

CTX_Reserved,                   205

CTX_Not_Reserved,               206

   CTX_No_More_Connection = 210,   210

CTX_Not_Initialized ,           211

CTX_DB_Not_Connected,           212

CTX_AggregationTable,           213

CTX_OracleInternalTable,        214

CTX_Cnx_Null,                   215

CTX_ServerTimeout,              216

CTX_ServerTimeout_BeforeCommit, 217

CTX_Table_Altered,              218

CTX_DBError,                    219

CTX_SampleCommitError,          220

CTX_SampleLogStatusError,       221

CTX_QueueNotInitialized,        222

CTX_QueueFull,                  223

CTX_QueueStopped,               224

CTX_JavaEnv_Null,               225

   CTX_NoNeedToRenameFile,         226

CTX_SkipWrite,                  227

CTX_InitializationWarning = 300,300

CTX_EncodingError,              301

CTX_AddressError,               302

CTX_InitFailedRecently,         303

CTX_Table_Deleted,              304

CTX_Wrong_Schema,               305

CTX_RenameConfigFileError,      306

CTX_RemoveConfigFileError,      307

CTX_ConfigFileEmptyError,       308

   CTX_SelectSiteError = 320,      320

CTX_GetCurrentCMSAddressError,  321

CTX_RemoveMetaFileError = 325,  325

CTX_MetaFileSizeError,          326

   CTX_Failed_Batch_Row_Not_Found=350, 350

CTX_RowExistAlready,            351

CTX_ODBC_DSN_Exceed_Max_Len,    352

CTX_Compress_Failed,            353

CTX_UnCompress_Failed,          354

CTX_Compression_Warning,        355

   CTX_TableSpaceNotFound=400,     400

CTX_TableSpaceTooSmall,         401

CTX_TableSpaceNotOnline,        402

CTX_IndexTableSpaceInvalid,     403

   CTX_SequenceNotFound=450,       450

   CTX_HistoryRenameError=500,     500

   CTX_Restore_Primary_WPA=550     550

Photo Note: New Cruise Ship Buffet 2016

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: