Hi All,
I upgraded CRM system from 6.0 to 7.02. System is very slow and dispatcher is often getting stopped in the night time. I have posted disp trace. Please help ASAP. Kernel 7.21_EXT patch 400. SQL 2008/Windows 2008. Solman MOPZ calculated the kernel target. I also checked in PAM, that kernel release is fine for the target. "stoc check failed kernel not compatible with system (rc=-100)". I have also attached dev_ms.
---------------------------------------------------
trc file: "dev_disp", trc level: 1, release: "721"
---------------------------------------------------
sysno 00
sid CRD
systemid 562 (PC with Windows NT)
relno 7210
patchlevel 0
patchno 400
intno 20020600
make multithreaded, Unicode, 64 bit, optimized
profile F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
pid 3664
kernel runs with dp version 139000(ext=121000) (@(#) DPLIB-INT-VERSION-139000-UC)
length of sys_adm_ext is 592 bytes
*** SWITCH TRC-HIDE on ***
***LOG Q00=> DpSapEnvInit, DPStart (00 3664) [dpxxdisp.c 1323]
shared lib "dw_xml.dll" version 400 successfully loaded
shared lib "dw_xtc.dll" version 400 successfully loaded
shared lib "dw_stl.dll" version 400 successfully loaded
shared lib "dw_gui.dll" version 400 successfully loaded
shared lib "dw_mdm.dll" version 400 successfully loaded
shared lib "dw_rndrt.dll" version 400 successfully loaded
shared lib "dw_abp.dll" version 400 successfully loaded
shared lib "dw_sym.dll" version 400 successfully loaded
shared lib "dw_aci.dll" version 400 successfully loaded
rdisp/softcancel_sequence : -> 0,5,-1
use internal message server connection to port 3900
rdisp/dynamic_wp_check : 1
rdisp/calculateLoadAverage : 1
Tue Jan 13 08:39:57 2015
*** WARNING => DpNetCheck: NiAddrToHost(1.0.0.0) took 5 seconds
***LOG GZZ=> 1 possible network problems detected - check tracefile and adjust the DNS settings [dpxxtool2.c 6525]
MtxInit: 30000 0 0
DpSysAdmExtInit: ABAP is active
DpSysAdmExtInit: VMC (JAVA VM in WP) is active
display_tcp_ip: 0
Set Check Logoff Flags to 0x0
DpIPCInit2: write dp-profile-values into sys_adm_ext
DpIPCInit2: start server >SAPCRMDEV_CRD_00 <
DpShMCreate: sizeof(wp_adm) 49632 (2256)
DpShMCreate: sizeof(tm_adm) 5517056 (27448)
DpShMCreate: sizeof(wp_ca_adm) 64000 (64)
DpShMCreate: sizeof(appc_ca_adm) 128000 (64)
DpCommTableSize: max/headSize/ftSize/tableSize=500/16/1392064/1392080
DpShMCreate: sizeof(comm_adm) 1392080 (2768)
DpSlockTableSize: max/headSize/ftSize/fiSize/tableSize=512/48/163904/96560/260512
DpShMCreate: sizeof(slock_adm) 260512 (296)
DpFileTableSize: max/headSize/ftSize/tableSize=4400/16/422464/422480
DpShMCreate: sizeof(file_adm) 422480 (80)
DpShMCreate: sizeof(vmc_adm) 47520 (2160)
DpShMCreate: sizeof(wall_adm) (41664/42896/64/192)
DpShMCreate: sizeof(gw_adm) 48
DpShMCreate: sizeof(j2ee_adm) 3952
DpShMCreate: SHM_DP_ADM_KEY (addr: 00000000076C0050, size: 7984112)
DpShMCreate: allocated sys_adm at 00000000076C0060
DpShMCreate: allocated wp_adm_list at 00000000076C30B0
DpShMCreate: allocated wp_adm at 00000000076C32A0
DpShMCreate: allocated tm_adm_list at 00000000076CF490
DpShMCreate: allocated tm_adm at 00000000076CF4E0
DpShMCreate: allocated wp_ca_adm at 0000000007C123F0
DpShMCreate: allocated appc_ca_adm at 0000000007C21E00
DpShMCreate: allocated comm_adm at 0000000007C41210
DpShMCreate: allocated slock_adm at 0000000007D94FF0
DpShMCreate: allocated file_adm at 0000000007DD49A0
DpShMCreate: allocated vmc_adm_list at 0000000007E3BC00
DpShMCreate: allocated vmc_adm at 0000000007E3BCB0
DpShMCreate: allocated gw_adm at 0000000007E47660
DpShMCreate: allocated j2ee_adm at 0000000007E476A0
DpShMCreate: allocated ca_info at 0000000007E48620
DpShMCreate: allocated wall_adm at 0000000007E486B0
DpCommAttachTable: attached comm table (header=0000000007C41210/ft=0000000007C41220)
DpSysAdmIntInit: initialize sys_adm
rdisp/test_roll : roll strategy is DP_NORMAL_ROLL
dia token check not active (9 token)
MBUF state OFF
DpCommInitTable: init table for 500 entries
DpFileInitTable: init table for 4400 entries
DpSesCreateTable: created session table at 00000000034B0050 (len=167472)
DpRqQInit: keep protect_queue / slots_per_queue 0 / 2001 in sys_adm
rdisp/queue_size_check_value : -> on,50,30,40,500,50,500,80
EmInit: MmSetImplementation( 2 ).
MM global diagnostic options set: 0
<ES> client 0 initializing ....
<ES> EsILock: use Mutex for locking
<ES> InitFreeList
ES> block size is 4096 kByte.
Using implementation view
<EsNT> Using memory model view.
<EsNT> Memory Reset disabled as NT default
<ES> 2865 blocks reserved for free list.
ES initialized.
mm.dump: set maximum dump mem to 192 MB
mm.dump: set global maximum dump mem to 192 MB
EsRegisterEmCheck: Register EmGetEsHandles at 00000001402B4270
Es2ResCreate: Changed resource parameters to
filenamePrefix =
blockSizeKB = 2048 KB
initialSizeMB = 480 MB
extendSizeMB = 240 MB
maxSizeMB = 15840 MB
DpVmcSetActive: set vmc state DP_VMC_ENABLED
DpVmcSetActive: set vmc state DP_VMC_ACTIVE
DpVmcInit2: o.k.
MPI: dynamic quotas disabled.
MPI init: pipes=4000 buffers=1279 reserved=383 quota=10%
*** WARNING => DpJ2eeStart: profile parameter 'system/type' not correctly set to start J2EE; use 'J2EE' for Java only and 'DS' for double stack instance (info=0x301) [dpxxj2ee.c 343]
J2EE server info
start = TRUE
state = STARTED
pid = 4888
argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[3] = -DSAPSTART=1
argv[4] = -DCONNECT_PORT=65000
argv[5] = -DSAPSYSTEM=00
argv[6] = -DSAPSYSTEMNAME=CRD
argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00
argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[9] = -DFRFC_FALLBACK=ON
argv[10] = -DFRFC_FALLBACK_HOST=localhost
start_lazy = 0
start_control = SAP J2EE startup framework
DpJ2eeStart: j2ee state = STARTED
rdisp/http_min_wait_dia_wp : 1 -> 1
**LOG CPS=> DpLoopInit, ICU ( 4.0.1 4.0.1 5.1) [dpxxdisp.c 1709]
***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c 12944]
MBUF state LOADING
DpStartStopMsg: send start message (myname is >SAPCRMDEV_CRD_00 <)
DpStartStopMsg: start msg sent
Tue Jan 13 08:39:58 2015
CCMS: Initialized monitoring segment of size 60000000.
CCMS: Initialized CCMS Headers in the shared monitoring segment.
CCMS: Checking Downtime Configuration of Monitoring Segment.
CCMS: AlMsUpload called by wp 1024.
Tue Jan 13 08:39:59 2015
CCMS: AlMsUpload successful for F:\usr\sap\CRD\DVEBMGS00\log\ALMTTREE.DAT (2844 MTEs).
Tue Jan 13 08:40:05 2015
CCMS: start to initalize 3.X shared alert area (first segment).
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1296]
MBUF state ACTIVE
DpWpBlksLow: max wp blocks in queue is 800 (80 %)
MBUF component UP
DpMsgProcess: 1 server in MBUF
DpAppcBlksLow: max appc blocks in queue is 1000 (50 %)
Tue Jan 13 08:40:37 2015
*** ERROR => [DpProcDied] Process died (PID:4888 HANDLE:952 rc:0x0) [dpnttool2.c 147]
DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN
Tue Jan 13 08:40:57 2015
J2EE server info
start = TRUE
state = STARTED
pid = 3752
argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[3] = -DSAPSTART=1
argv[4] = -DCONNECT_PORT=64998
argv[5] = -DSAPSYSTEM=00
argv[6] = -DSAPSYSTEMNAME=CRD
argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00
argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[9] = -DFRFC_FALLBACK=ON
argv[10] = -DFRFC_FALLBACK_HOST=localhost
start_lazy = 0
start_control = SAP J2EE startup framework
DpJ2eeStart: j2ee state = STARTED
Tue Jan 13 08:40:59 2015
DpModState: change server state from STARTING to ACTIVE
Tue Jan 13 08:41:17 2015
*** ERROR => [DpProcDied] Process died (PID:3752 HANDLE:1308 rc:0x0) [dpnttool2.c 147]
DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN
Tue Jan 13 08:41:37 2015
J2EE server info
start = TRUE
state = STARTED
pid = 2496
argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[3] = -DSAPSTART=1
argv[4] = -DCONNECT_PORT=64997
argv[5] = -DSAPSYSTEM=00
argv[6] = -DSAPSYSTEMNAME=CRD
argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00
argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[9] = -DFRFC_FALLBACK=ON
argv[10] = -DFRFC_FALLBACK_HOST=localhost
start_lazy = 0
start_control = SAP J2EE startup framework
DpJ2eeStart: j2ee state = STARTED
Tue Jan 13 08:41:57 2015
*** ERROR => [DpProcDied] Process died (PID:2496 HANDLE:1320 rc:0x0) [dpnttool2.c 147]
DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN
Tue Jan 13 08:42:17 2015
J2EE server info
start = TRUE
state = STARTED
pid = 2120
argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[3] = -DSAPSTART=1
argv[4] = -DCONNECT_PORT=64996
argv[5] = -DSAPSYSTEM=00
argv[6] = -DSAPSYSTEMNAME=CRD
argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00
argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[9] = -DFRFC_FALLBACK=ON
argv[10] = -DFRFC_FALLBACK_HOST=localhost
start_lazy = 0
start_control = SAP J2EE startup framework
DpJ2eeStart: j2ee state = STARTED
Tue Jan 13 08:42:37 2015
*** ERROR => [DpProcDied] Process died (PID:2120 HANDLE:1328 rc:0x0) [dpnttool2.c 147]
DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN
Tue Jan 13 08:42:58 2015
J2EE server info
start = TRUE
state = STARTED
pid = 6024
argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[3] = -DSAPSTART=1
argv[4] = -DCONNECT_PORT=64995
argv[5] = -DSAPSYSTEM=00
argv[6] = -DSAPSYSTEMNAME=CRD
argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00
argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[9] = -DFRFC_FALLBACK=ON
argv[10] = -DFRFC_FALLBACK_HOST=localhost
start_lazy = 0
start_control = SAP J2EE startup framework
DpJ2eeStart: j2ee state = STARTED
Tue Jan 13 08:43:18 2015
*** ERROR => [DpProcDied] Process died (PID:6024 HANDLE:656 rc:0x0) [dpnttool2.c 147]
DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN
Tue Jan 13 08:43:38 2015
J2EE server info
start = TRUE
state = STARTED
pid = 5728
argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[3] = -DSAPSTART=1
argv[4] = -DCONNECT_PORT=64994
argv[5] = -DSAPSYSTEM=00
argv[6] = -DSAPSYSTEMNAME=CRD
argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00
argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[9] = -DFRFC_FALLBACK=ON
argv[10] = -DFRFC_FALLBACK_HOST=localhost
start_lazy = 0
start_control = SAP J2EE startup framework
DpJ2eeStart: j2ee state = STARTED
Tue Jan 13 08:43:58 2015
*** ERROR => [DpProcDied] Process died (PID:5728 HANDLE:484 rc:0x0) [dpnttool2.c 147]
DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN
Tue Jan 13 08:44:07 2015
*** ERROR => DpMsgProcess: MsReceive () -> MSEPARTNERUNKNOWN, partner: - [dpxxdisp.c 2601]
Tue Jan 13 08:44:18 2015
J2EE server info
start = TRUE
state = STARTED
pid = 6524
argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[3] = -DSAPSTART=1
argv[4] = -DCONNECT_PORT=64992
argv[5] = -DSAPSYSTEM=00
argv[6] = -DSAPSYSTEMNAME=CRD
argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00
argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[9] = -DFRFC_FALLBACK=ON
argv[10] = -DFRFC_FALLBACK_HOST=localhost
start_lazy = 0
start_control = SAP J2EE startup framework
DpJ2eeStart: j2ee state = STARTED
Tue Jan 13 08:44:38 2015
*** ERROR => [DpProcDied] Process died (PID:6524 HANDLE:944 rc:0x0) [dpnttool2.c 147]
DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN
Tue Jan 13 08:44:58 2015
J2EE server info
start = TRUE
state = STARTED
pid = 6692
argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[3] = -DSAPSTART=1
argv[4] = -DCONNECT_PORT=64991
argv[5] = -DSAPSYSTEM=00
argv[6] = -DSAPSYSTEMNAME=CRD
argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00
argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[9] = -DFRFC_FALLBACK=ON
argv[10] = -DFRFC_FALLBACK_HOST=localhost
start_lazy = 0
start_control = SAP J2EE startup framework
DpJ2eeStart: j2ee state = STARTED
Tue Jan 13 08:45:18 2015
*** ERROR => [DpProcDied] Process died (PID:6692 HANDLE:484 rc:0x0) [dpnttool2.c 147]
DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN
Tue Jan 13 08:45:21 2015
*** ERROR => DpMsgProcess: MsReceive () -> MSEPARTNERUNKNOWN, partner: - [dpxxdisp.c 2601]
Tue Jan 13 08:45:38 2015
J2EE server info
start = TRUE
state = STARTED
pid = 6856
argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[3] = -DSAPSTART=1
argv[4] = -DCONNECT_PORT=64990
argv[5] = -DSAPSYSTEM=00
argv[6] = -DSAPSYSTEMNAME=CRD
argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00
argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[9] = -DFRFC_FALLBACK=ON
argv[10] = -DFRFC_FALLBACK_HOST=localhost
start_lazy = 0
start_control = SAP J2EE startup framework
DpJ2eeStart: j2ee state = STARTED
Tue Jan 13 08:45:58 2015
*** ERROR => [DpProcDied] Process died (PID:6856 HANDLE:656 rc:0x0) [dpnttool2.c 147]
DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN
Tue Jan 13 08:46:18 2015
J2EE server info
start = TRUE
state = STARTED
pid = 7152
argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE
argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[3] = -DSAPSTART=1
argv[4] = -DCONNECT_PORT=64989
argv[5] = -DSAPSYSTEM=00
argv[6] = -DSAPSYSTEMNAME=CRD
argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00
argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
argv[9] = -DFRFC_FALLBACK=ON
argv[10] = -DFRFC_FALLBACK_HOST=localhost
start_lazy = 0
start_control = SAP J2EE startup framework
DpJ2eeStart: j2ee state = STARTED
Tue Jan 13 08:46:38 2015
*** ERROR => [DpProcDied] Process died (PID:7152 HANDLE:1308 rc:0x0) [dpnttool2.c 147]
DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN
Tue Jan 13 08:46:58 2015
DpEnvCheckJ2ee: switch off j2ee start flag
Tue Jan 13 09:35:56 2015
*** ERROR => 13.01.2015 14:35:56.034 PID=3664, TID=4196 SapTimer error. SapTime differs from Windows time. ERRORS: 0, SapTime-WindowsUtcTime=-85 seconds. TotalTimeCorrection=+0 sec, QpcTime-WindowsUtcTime=-85 sec (*). [dpxxdisp.c 21913]
13.01.2015 14:35:56.034 PID=3664, TID=4196 SapTimer info. QPC/Windows time mismatch. System suspended, busy, missing ticks. Mismatch QpcTime-WindowsTime: -86300 millisec (max +0). Slept 1000 millisec, Qpc elapsed: +999 millisec, WindowsUtc elapsed +87299 millisec. SapTime-WindowsUtcTime=-85 seconds. TotalTimeCorrection=+0 millisec, QpcTime-WindowsUtcTime=-85736 millisec (*).
Tue Jan 13 09:40:24 2015
13.01.2015 14:40:24.480 PID=3664, TID=4196 SapTimer info. QPC/Windows time mismatch. System suspended, busy, missing ticks. Mismatch QpcTime-WindowsTime: -87997 millisec (max -86300). Slept 1000 millisec, Qpc elapsed: +1000 millisec, WindowsUtc elapsed +88997 millisec. SapTime-WindowsUtcTime=-88 seconds. TotalTimeCorrection=+85786 millisec, QpcTime-WindowsUtcTime=-173903 millisec (*).
Tue Jan 13 09:51:17 2015
SoftCancel request for T30 U812 M0 received from IC_MAN
Tue Jan 13 09:56:48 2015
SoftCancel request for T37 U978 M0 received from IC_MAN
Tue Jan 13 09:57:59 2015
SoftCancel request for T37 U994 M0 received from IC_MAN
Tue Jan 13 10:19:25 2015
DpHdlDeadWp: W0 (pid=3736) terminated automatically
Tue Jan 13 10:45:23 2015
SoftCancel request for T33 U1231 M0 received from IC_MAN
Tue Jan 13 10:45:47 2015
DpHdlDeadWp: W0 (pid=6728) terminated automatically
Tue Jan 13 10:47:29 2015
*** WARNING => DpEnvCheck: no answer from msg server since 33 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c 8538]
Tue Jan 13 10:51:18 2015
*** ERROR => 13.01.2015 15:51:18.702 PID=3664, TID=4196 SapTimer warning. Took time keeping ownership. PID=3664, old owner PID:2268 [dpxxdisp.c 21913]
Tue Jan 13 10:52:29 2015
*** WARNING => DpEnvCheck: no answer from msg server since 33 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c 8538]
Tue Jan 13 10:52:50 2015
*** WARNING => DpEnvCheck: no answer from msg server since 54 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c 8538]
Tue Jan 13 10:53:11 2015
*** WARNING => DpEnvCheck: no answer from msg server since 75 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c 8538]
Tue Jan 13 10:53:32 2015
*** WARNING => DpEnvCheck: no answer from msg server since 96 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c 8538]
Tue Jan 13 11:10:33 2015
DpHdlDeadWp: W6 (pid=4084) terminated automatically
DpHdlDeadWp: W7 (pid=3220) terminated automatically
Tue Jan 13 11:49:36 2015
SoftCancel request for T33 U1879 M0 received from IC_MAN
Tue Jan 13 11:50:34 2015
DpHdlDeadWp: W2 (pid=4288) terminated automatically
Tue Jan 13 11:52:32 2015
*** ERROR => V3 already deleted [dpxxdisp.c 8106]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c 6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c 6266]
-IN-- sender_id DISPATCHER tid 33 wp_ca_blk 90 wp_id 0
-IN-- action SEND_TO_WP uid 2141 appc_ca_blk -1 type DIA
-IN-- new_stat NO_CHANGE mode 0 len 1108 rq_id 20595
-IN-- req_info LOGIN,MSG_WITH_REQ_BUF,MSG_WITH_OH
-IN-- vmcHdl 3 sesTab -1 vmcControlsCheckedCount 0
-IN-- msg_name SAPCRMDEV_CRD_00
Tue Jan 13 12:03:55 2015
DpHdlDeadWp: W3 (pid=4356) terminated automatically
Tue Jan 13 15:50:00 2015
SoftCancel request for T31 U2842 M0 received from IC_MAN
Tue Jan 13 15:58:03 2015
SoftCancel request for T27 U4211 M0 received from IC_MAN
SoftCancel request for T29 U4212 M0 received from IC_MAN
Tue Jan 13 17:39:04 2015
SoftCancel request for T37 U5251 M0 received from IC_MAN
SoftCancel request for T23 U5252 M0 received from IC_MAN
Tue Jan 13 17:41:53 2015
SoftCancel request for T31 U5306 M0 received from IC_MAN
SoftCancel request for T22 U5307 M0 received from IC_MAN
Tue Jan 13 20:47:22 2015
DpHdlDeadWp: W7 (pid=6344) terminated automatically
Tue Jan 13 20:58:21 2015
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:65029: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 25/sock 968
(SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:65029) [nixxi.cpp 5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c 2833]
*****************************************************************************
*
* LOCATION SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
* ERROR Reading data from the message server (SAPCRMDEV / 3900)
* failed.
* Please check the trace file of the message server.
*
* TIME Tue Jan 13 20:58:21 2015
* RELEASE 721
* COMPONENT MS (message handling interface, multithreaded)
* VERSION 4
* RC -36
* MODULE msxxi.c
* LINE 2852
* COUNTER 8744
*
*****************************************************************************
*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c 1867]
MBUF state OFF
MBUF component DOWN
*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner: [dpxxdisp.c 2601]
*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c 2672]
Tue Jan 13 20:58:26 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Tue Jan 13 20:58:38 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Tue Jan 13 20:59:03 2015
***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c 12944]
Tue Jan 13 20:59:11 2015
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1296]
MBUF state ACTIVE
MBUF component UP
DpMsgProcess: 1 server in MBUF
DpMsgProcess: 1 server in MBUF
Tue Jan 13 21:00:43 2015
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:49645: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 20/sock 984
(SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:49645) [nixxi.cpp 5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c 2833]
*****************************************************************************
*
* LOCATION SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
* ERROR Reading data from the message server (SAPCRMDEV / 3900)
* failed.
* Please check the trace file of the message server.
*
* TIME Tue Jan 13 21:00:43 2015
* RELEASE 721
* COMPONENT MS (message handling interface, multithreaded)
* VERSION 4
* RC -36
* MODULE msxxi.c
* LINE 2852
* COUNTER 8761
*
*****************************************************************************
*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c 1867]
MBUF state OFF
MBUF component DOWN
*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner: [dpxxdisp.c 2601]
*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c 2672]
Tue Jan 13 21:00:48 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Tue Jan 13 21:00:54 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Tue Jan 13 21:01:09 2015
***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c 12944]
Tue Jan 13 21:01:12 2015
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1296]
Tue Jan 13 21:01:26 2015
MBUF state ACTIVE
MBUF component UP
DpMsgProcess: 1 server in MBUF
***LOG Q0I=> NiIRead: P=127.0.0.1:49717; L=127.0.0.1:3200: recv (10053: WSAECONNABORTED: Software caused connection abort) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 27/sock 968
(SI_ECONN_BROKEN/10053; I4; ST; P=127.0.0.1:49717; L=127.0.0.1:3200) [nixxi.cpp 5082]
DpMsgProcess: 1 server in MBUF
Tue Jan 13 21:32:09 2015
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:49716: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 23/sock 1340
(SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:49716) [nixxi.cpp 5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c 2833]
*****************************************************************************
*
* LOCATION SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
* ERROR Reading data from the message server (SAPCRMDEV / 3900)
* failed.
* Please check the trace file of the message server.
*
* TIME Tue Jan 13 21:32:09 2015
* RELEASE 721
* COMPONENT MS (message handling interface, multithreaded)
* VERSION 4
* RC -36
* MODULE msxxi.c
* LINE 2852
* COUNTER 9088
*
*****************************************************************************
*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c 1867]
MBUF state OFF
MBUF component DOWN
*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner: [dpxxdisp.c 2601]
*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c 2672]
Tue Jan 13 21:32:14 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
*** ERROR => MsISnd2: not_attached [msxxi.c 1510]
***LOG Q0N=> DpRqNoWpHandle, MsSndAdmin () [dpxxdisp.c 5590]
Tue Jan 13 21:32:17 2015
***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c 12944]
MBUF state LOADING
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1296]
MBUF state ACTIVE
MBUF component UP
DpMsgProcess: 1 server in MBUF
Tue Jan 13 22:41:28 2015
***LOG Q0I=> NiIRead: P=127.0.0.1:52332; L=127.0.0.1:3200: recv (10053: WSAECONNABORTED: Software caused connection abort) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 67/sock 944
(SI_ECONN_BROKEN/10053; I4; ST; P=127.0.0.1:52332; L=127.0.0.1:3200) [nixxi.cpp 5082]
Tue Jan 13 22:41:42 2015
SoftCancel request for T42 U8108 M0 received from IC_MAN
Tue Jan 13 22:41:59 2015
SoftCancel request for T32 U8113 M0 received from IC_MAN
SoftCancel request for T22 U8114 M0 received from IC_MAN
Tue Jan 13 22:42:09 2015
SoftCancel request for T26 U8115 M0 received from IC_MAN
Tue Jan 13 22:43:05 2015
DpHdlDeadWp: W0 (pid=5292) terminated automatically
DpHdlDeadWp: W3 (pid=4252) terminated automatically
DpHdlDeadWp: W4 (pid=1528) terminated automatically
Tue Jan 13 22:54:19 2015
SoftCancel request for T22 U8171 M0 received from IC_MAN
Tue Jan 13 22:55:05 2015
DpHdlDeadWp: W6 (pid=6644) terminated automatically
Tue Jan 13 23:10:45 2015
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:50690: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 17/sock 1000
(SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:50690) [nixxi.cpp 5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c 2833]
*****************************************************************************
*
* LOCATION SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
* ERROR Reading data from the message server (SAPCRMDEV / 3900)
* failed.
* Please check the trace file of the message server.
*
* TIME Tue Jan 13 23:10:45 2015
* RELEASE 721
* COMPONENT MS (message handling interface, multithreaded)
* VERSION 4
* RC -36
* MODULE msxxi.c
* LINE 2852
* COUNTER 10145
*
*****************************************************************************
*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c 1867]
MBUF state OFF
MBUF component DOWN
*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner: [dpxxdisp.c 2601]
*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c 2672]
Tue Jan 13 23:10:50 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Tue Jan 13 23:10:56 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Tue Jan 13 23:11:20 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Tue Jan 13 23:11:35 2015
***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c 12944]
Tue Jan 13 23:11:42 2015
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1296]
MBUF state ACTIVE
MBUF component UP
DpMsgProcess: 1 server in MBUF
DpMsgProcess: 1 server in MBUF
Tue Jan 13 23:59:59 2015
SoftCancel request for T31 U8648 M0 received from IC_MAN
Wed Jan 14 00:00:00 2015
SoftCancel request for T28 U8649 M0 received from IC_MAN
Wed Jan 14 00:00:04 2015
DpRqCheck: mode 0 in status CANCEL
SoftCancel request for T31 U8648 M0 received from IC_MAN
*** ERROR => DpRqCheck: T31 in state TM_SLOT_FREE [dpxxdisp.c 8135]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c 6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c 6266]
-IN-- sender_id DISPATCHER tid 31 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 8648 appc_ca_blk -1 type DIA
-IN-- new_stat NO_CHANGE mode 0 len 0 rq_id 12995
-IN-- prio HIGH
-IN-- req_info DP_SOFT_CANCEL
-IN-- msg_name -
Wed Jan 14 00:00:31 2015
***LOG Q0I=> NiIRead: P=192.168.101.90:2900; L=192.168.101.63:3200: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 32/sock 988
(SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.90:2900; L=192.168.101.63:3200) [nixxi.cpp 5082]
DpRTmPrepareReq: network error of client T25, NiBufReceive (-6: NIECONN_BROKEN), dp_tm_status=3
Wed Jan 14 00:00:40 2015
*** WARNING => DpAddrToHost: NiAddrToHost took 8422 ms [dpxxdisp.c 20855]
*** WARNING => DpAddrToHost: Disabling reverse name lookup [dpxxdisp.c 20856]
DpRTmPrepareReq: client address of T25 is 192.168.101.90(TERMSRVCRESTRON)
***LOG Q04=> DpRTmPrep, NiBufReceive (5728 SRASTOGI 25 TermSrvCrest) [dpxxdisp.c 12573]
RM-T25, U5728, 100 SRASTOGI, TermSrvCrestron, 23:59:55, M0, W5, SE16, 2/2
***LOG Q0I=> NiIRead: P=192.168.101.90:4082; L=192.168.101.63:3200: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 90/sock 944
(SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.90:4082; L=192.168.101.63:3200) [nixxi.cpp 5082]
***LOG Q0I=> NiIRead: P=192.168.101.90:4068; L=192.168.101.63:3200: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 66/sock 1308
(SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.90:4068; L=192.168.101.63:3200) [nixxi.cpp 5082]
***LOG Q0I=> NiIRead: P=192.168.101.90:4061; L=192.168.101.63:3200: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 30/sock 984
(SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.90:4061; L=192.168.101.63:3200) [nixxi.cpp 5082]
DpRTmPrepareReq: network error of client T34, NiBufReceive (-6: NIECONN_BROKEN), dp_tm_status=3
DpRTmPrepareReq: client address of T34 is 192.168.101.90(192.168.101.90)
***LOG Q04=> DpRTmPrep, NiBufReceive (8644 34 TermSrvCrest) [dpxxdisp.c 12573]
RM-T34, U8644, 000 , TermSrvCrestron, 23:59:53, M0, W8, SESS, 2/0
DpRTmPrepareReq: network error of client T23, NiBufReceive (-6: NIECONN_BROKEN), dp_tm_status=3
DpRTmPrepareReq: client address of T23 is 192.168.101.90(192.168.101.90)
***LOG Q04=> DpRTmPrep, NiBufReceive (8646 23 TermSrvCrest) [dpxxdisp.c 12573]
RM-T23, U8646, 000 , TermSrvCrestron, 23:59:54, M0, W4, SESS, 2/0
DpRTmPrepareReq: network error of client T42, NiBufReceive (-6: NIECONN_BROKEN), dp_tm_status=3
DpRTmPrepareReq: client address of T42 is 192.168.101.90(192.168.101.90)
***LOG Q04=> DpRTmPrep, NiBufReceive (8647 42 TermSrvCrest) [dpxxdisp.c 12573]
RM-T42, U8647, 000 , TermSrvCrestron, 23:59:56, M0, W0, SESS, 2/0
Wed Jan 14 00:00:41 2015
DpHdlDeadWp: W1 (pid=4608) terminated automatically
Wed Jan 14 00:49:19 2015
*** WARNING => DpITimeCheck: suspicious forward jump of current time 1421214559, last time 1421211655 [dpxxdisp.c 9205]
Wed Jan 14 00:49:20 2015
DpITimeCheck: new time seems to be OK now, I accept 1421214560
*** WARNING => DpTimeCheck: huge diff between current 1421211676 and next schedule (act_time 1421214560 delay 120 [dpxxdisp.c 9379]
DpRqCheck: mode 0 in status CANCEL
*** WARNING => DpHdlSoftCancel: T28/M0 still active in W1 with state WP_NEW, can't deligate SOFT_CANCEL [dpxxdisp.c 18360]
Wed Jan 14 00:49:22 2015
SoftCancel request for T26 U8655 M0 received from REMOTE_TERMINAL
Wed Jan 14 01:08:41 2015
DpHdlDeadWp: W7 (pid=5988) terminated automatically
Wed Jan 14 01:15:12 2015
SoftCancel request for T42 U8862 M0 received from IC_MAN
SoftCancel request for T25 U8859 M0 received from IC_MAN
Wed Jan 14 01:15:30 2015
DpRqCheck: mode 0 in status CANCEL
SoftCancel request for T42 U8862 M0 received from IC_MAN
*** ERROR => DpRqCheck: T42 in state TM_SLOT_FREE [dpxxdisp.c 8135]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c 6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c 6266]
-IN-- sender_id DISPATCHER tid 42 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 8862 appc_ca_blk -1 type DIA
-IN-- new_stat NO_CHANGE mode 0 len 0 rq_id 21027
-IN-- prio HIGH
-IN-- req_info DP_SOFT_CANCEL
-IN-- msg_name -
***LOG Q0I=> NiIRead: P=127.0.0.1:55047; L=127.0.0.1:3200: recv (10053: WSAECONNABORTED: Software caused connection abort) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 86/sock 1304
(SI_ECONN_BROKEN/10053; I4; ST; P=127.0.0.1:55047; L=127.0.0.1:3200) [nixxi.cpp 5082]
Wed Jan 14 01:15:35 2015
DpHdlSoftCancelTimeout: previous kill in wp 0 not yet handled, reschedule softcancel (level 2)
Wed Jan 14 01:15:40 2015
DpHdlSoftCancelTimeout: previous kill in wp 0 not yet handled, reschedule softcancel (level 2)
Wed Jan 14 01:15:45 2015
DpHdlSoftCancelTimeout: previous kill in wp 0 not yet handled, reschedule softcancel (level 2)
Wed Jan 14 01:15:50 2015
DpHdlSoftCancelTimeout: previous kill in wp 0 not yet handled, reschedule softcancel (level 2)
Wed Jan 14 01:16:00 2015
DpRqCheck: mode 0 in status CANCEL
SoftCancel request for T25 U8859 M0 received from IC_MAN
*** ERROR => DpRqCheck: T25 in state TM_SLOT_FREE [dpxxdisp.c 8135]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c 6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c 6266]
-IN-- sender_id DISPATCHER tid 25 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 8859 appc_ca_blk -1 type DIA
-IN-- new_stat NO_CHANGE mode 0 len 0 rq_id 21060
-IN-- prio HIGH
-IN-- req_info DP_SOFT_CANCEL
-IN-- msg_name -
Wed Jan 14 01:20:56 2015
SoftCancel request for T22 U8887 M0 received from IC_MAN
SoftCancel request for T35 U8885 M0 received from IC_MAN
Wed Jan 14 01:21:10 2015
DpHdlDeadWp: W1 (pid=5524) terminated automatically
Wed Jan 14 01:21:24 2015
DpHdlDeadWp: W8 (pid=564) terminated automatically
***LOG Q0I=> NiIRead: P=127.0.0.1:55171; L=127.0.0.1:3200: recv (10053: WSAECONNABORTED: Software caused connection abort) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 102/sock 1016
(SI_ECONN_BROKEN/10053; I4; ST; P=127.0.0.1:55171; L=127.0.0.1:3200) [nixxi.cpp 5082]
Wed Jan 14 01:22:16 2015
DpRqCheck: mode 0 in status CANCEL
*** WARNING => DpHdlSoftCancel: T35/M0 still active in W1 with state WP_NEW, can't deligate SOFT_CANCEL [dpxxdisp.c 18360]
Wed Jan 14 01:22:44 2015
DpRqCheck: mode 0 in status CANCEL
Wed Jan 14 01:22:49 2015
SoftCancel request for T22 U8887 M0 received from IC_MAN
*** ERROR => DpRqCheck: T22 in state TM_SLOT_FREE [dpxxdisp.c 8135]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c 6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c 6266]
-IN-- sender_id DISPATCHER tid 22 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 8887 appc_ca_blk -1 type DIA
-IN-- new_stat NO_CHANGE mode 0 len 0 rq_id 21394
-IN-- prio HIGH
-IN-- req_info DP_SOFT_CANCEL
-IN-- msg_name -
Wed Jan 14 01:25:19 2015
SoftCancel request for T35 U8951 M0 received from IC_MAN
Wed Jan 14 01:25:20 2015
SoftCancel request for T22 U8954 M0 received from IC_MAN
Wed Jan 14 01:25:35 2015
DpRqCheck: mode 0 in status CANCEL
***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c 6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4231): [dpxxdisp.c 6266]
-IN-- sender_id IC_MAN tid 24 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 8823 appc_ca_blk -1 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 0 rq_id 21226
-IN-- req_info DP_ICM_EVENT
-IN-- mpi read <f9d> mpi write <f9e> prot 1 obj fd -1 rq_id 0
-IN-- icm_opcode: ICM_PLUGIN_REQUEST virt_host_idx: 0
-IN-- dtrace {root-id=00505682314E1EE4A6F674B291BD340B}_{conn-id=00505682314E1EE4A6F69543410C740B}_1
-IN-- msg_name -
Wed Jan 14 01:52:12 2015
DpHdlDeadWp: W7 (pid=6772) terminated automatically
Wed Jan 14 01:54:51 2015
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:53037: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 22/sock 1324
(SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:53037) [nixxi.cpp 5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c 2833]
*****************************************************************************
*
* LOCATION SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
* ERROR Reading data from the message server (SAPCRMDEV / 3900)
* failed.
* Please check the trace file of the message server.
*
* TIME Wed Jan 14 01:54:51 2015
* RELEASE 721
* COMPONENT MS (message handling interface, multithreaded)
* VERSION 4
* RC -36
* MODULE msxxi.c
* LINE 2852
* COUNTER 11354
*
*****************************************************************************
*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c 1867]
MBUF state OFF
MBUF component DOWN
*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner: [dpxxdisp.c 2601]
*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c 2672]
Wed Jan 14 01:54:57 2015
***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c 12944]
Wed Jan 14 01:55:08 2015
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1296]
MBUF state ACTIVE
MBUF component UP
DpMsgProcess: 1 server in MBUF
DpMsgProcess: 1 server in MBUF
Wed Jan 14 02:07:58 2015
*** WARNING => DpTimeCheck: skipped 1 buffer refresh requests because one is in the queue [dpxxdisp.c 9397]
Wed Jan 14 02:12:07 2015
SoftCancel request for T26 U9125 M0 received from IC_MAN
Wed Jan 14 02:41:40 2015
DpHdlDeadWp: W7 (pid=7164) terminated automatically
Wed Jan 14 03:33:27 2015
DpHdlDeadWp: W7 (pid=3892) terminated automatically
Wed Jan 14 03:44:27 2015
DpHdlDeadWp: W4 (pid=6928) terminated automatically
Wed Jan 14 03:47:47 2015
*** WARNING => DpEnvCheck: no answer from msg server since 20 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c 8538]
Wed Jan 14 03:48:07 2015
*** WARNING => DpEnvCheck: no answer from msg server since 40 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c 8538]
Wed Jan 14 03:48:25 2015
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:55941: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 23/sock 1348
(SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:55941) [nixxi.cpp 5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c 2833]
*****************************************************************************
*
* LOCATION SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
* ERROR Reading data from the message server (SAPCRMDEV / 3900)
* failed.
* Please check the trace file of the message server.
*
* TIME Wed Jan 14 03:48:25 2015
* RELEASE 721
* COMPONENT MS (message handling interface, multithreaded)
* VERSION 4
* RC -36
* MODULE msxxi.c
* LINE 2852
* COUNTER 12537
*
*****************************************************************************
*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c 1867]
MBUF state OFF
MBUF component DOWN
*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner: [dpxxdisp.c 2601]
*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c 2672]
Wed Jan 14 03:48:30 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Wed Jan 14 03:48:40 2015
*** ERROR => MsISnd2: not_attached [msxxi.c 1510]
***LOG Q0N=> DpRqNoWpHandle, MsSndAdmin () [dpxxdisp.c 5590]
Wed Jan 14 03:48:46 2015
***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c 12944]
Wed Jan 14 03:48:53 2015
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1296]
MBUF state ACTIVE
MBUF component UP
DpMsgProcess: 1 server in MBUF
DpMsgProcess: 1 server in MBUF
Wed Jan 14 03:49:15 2015
*** WARNING => DpHdlSoftCancel: T27/M0 still active in W4 with state WP_NEW, can't deligate SOFT_CANCEL [dpxxdisp.c 18360]
Wed Jan 14 03:51:40 2015
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:58283: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 17/sock 984
(SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:58283) [nixxi.cpp 5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c 2833]
*****************************************************************************
*
* LOCATION SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
* ERROR Reading data from the message server (SAPCRMDEV / 3900)
* failed.
* Please check the trace file of the message server.
*
* TIME Wed Jan 14 03:51:40 2015
* RELEASE 721
* COMPONENT MS (message handling interface, multithreaded)
* VERSION 4
* RC -36
* MODULE msxxi.c
* LINE 2852
* COUNTER 12564
*
*****************************************************************************
*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c 1867]
MBUF state OFF
MBUF component DOWN
*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner: [dpxxdisp.c 2601]
*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c 2672]
SoftCancel request for T27 U10155 M0 received from IC_MAN
Wed Jan 14 03:51:45 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Wed Jan 14 03:52:20 2015
***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c 12944]
***LOG Q0I=> NiIWrite: P=127.0.0.1:58350; L=127.0.0.1:3200: WSASend (10053: WSAECONNABORTED: Software caused connection abort) [nixxi.cpp 4599]
*** ERROR => NiIWrite: SiSendV failed for hdl 33/sock 1348
(SI_ECONN_BROKEN/10053; I4; ST; P=127.0.0.1:58350; L=127.0.0.1:3200) [nixxi.cpp 4599]
*** ERROR => NiICheck: NiBufIWriteUserMsg failed (rc=-6) [nixxi.cpp 6096]
*** ERROR => NiBufIProcMsg: NiICheck failed (rc=-6) [nibuf.cpp 2027]
*** ERROR => NiBufIIn: NiBufIProcMsg failed (rc=-6) [nibuf.cpp 2951]
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1296]
MBUF state ACTIVE
MBUF component UP
DpMsgProcess: 1 server in MBUF
DpMsgProcess: 1 server in MBUF
Wed Jan 14 04:18:03 2015
DpHdlDeadWp: W8 (pid=6688) terminated automatically
Wed Jan 14 04:32:45 2015
SoftCancel request for T34 U10386 M0 received from IC_MAN
Wed Jan 14 04:39:05 2015
DpHdlDeadWp: W0 (pid=6904) terminated automatically
Wed Jan 14 04:56:46 2015
DpHdlDeadWp: W1 (pid=7028) terminated automatically
Wed Jan 14 05:01:47 2015
DpHdlDeadWp: W8 (pid=5276) terminated automatically
Wed Jan 14 05:18:06 2015
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:58369: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 19/sock 984
(SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:58369) [nixxi.cpp 5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c 2833]
*****************************************************************************
*
* LOCATION SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
* ERROR Reading data from the message server (SAPCRMDEV / 3900)
* failed.
* Please check the trace file of the message server.
*
* TIME Wed Jan 14 05:18:06 2015
* RELEASE 721
* COMPONENT MS (message handling interface, multithreaded)
* VERSION 4
* RC -36
* MODULE msxxi.c
* LINE 2852
* COUNTER 13511
*
*****************************************************************************
*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c 1867]
MBUF state OFF
MBUF component DOWN
*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner: [dpxxdisp.c 2601]
*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c 2672]
Wed Jan 14 05:18:11 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Wed Jan 14 05:18:17 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Wed Jan 14 05:18:37 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Wed Jan 14 05:18:57 2015
***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c 12944]
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1296]
MBUF state ACTIVE
MBUF component UP
DpMsgProcess: 1 server in MBUF
DpMsgProcess: 1 server in MBUF
Wed Jan 14 05:53:07 2015
DpHdlDeadWp: W8 (pid=6128) terminated automatically
Wed Jan 14 06:08:11 2015
DpRqCheck: mode 0 in status CANCEL
***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c 6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4231): [dpxxdisp.c 6266]
-IN-- sender_id IC_MAN tid 29 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 11175 appc_ca_blk -1 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 0 rq_id 45714
-IN-- req_info DP_ICM_EVENT
-IN-- mpi read <19e1> mpi write <19e2> prot 1 obj fd -1 rq_id 0
-IN-- icm_opcode: ICM_PLUGIN_REQUEST virt_host_idx: 0
-IN-- dtrace {root-id=00505682314E1EE4A6FB44610803940B}_{conn-id=00505682314E1EE4A6FB8DA63301940B}_1
-IN-- msg_name -
*** ERROR => DpRqCheck: T29 in state TM_SLOT_FREE [dpxxdisp.c 8135]
***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c 6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4231): [dpxxdisp.c 6266]
-IN-- sender_id DISPATCHER tid 29 wp_ca_blk 226 wp_id 3
-IN-- action SEND_TO_WP uid 11175 appc_ca_blk -1 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 8 rq_id 45832
-IN-- req_info CANCELMODE
-IN-- msg_name -
Wed Jan 14 06:12:28 2015
DpHdlDeadWp: W1 (pid=6508) terminated automatically
Wed Jan 14 06:32:38 2015
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:60578: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
Wed Jan 14 06:32:39 2015
*** ERROR => NiIRead: SiRecv failed for hdl 23/sock 1344
(SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:60578) [nixxi.cpp 5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c 2833]
*****************************************************************************
*
* LOCATION SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
* ERROR Reading data from the message server (SAPCRMDEV / 3900)
* failed.
* Please check the trace file of the message server.
*
* TIME Wed Jan 14 06:32:39 2015
* RELEASE 721
* COMPONENT MS (message handling interface, multithreaded)
* VERSION 4
* RC -36
* MODULE msxxi.c
* LINE 2852
* COUNTER 14236
*
*****************************************************************************
*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c 1867]
MBUF state OFF
MBUF component DOWN
*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner: [dpxxdisp.c 2601]
*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c 2672]
Wed Jan 14 06:32:44 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Wed Jan 14 06:32:49 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Wed Jan 14 06:32:59 2015
*** ERROR => DpRqCheck: T38 in state TM_SLOT_FREE [dpxxdisp.c 8135]
***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c 6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4231): [dpxxdisp.c 6266]
-IN-- sender_id DISPATCHER tid 38 wp_ca_blk 272 wp_id 5
-IN-- action SEND_TO_WP uid 11318 appc_ca_blk -1 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 8 rq_id 48480
-IN-- req_info CANCELMODE
-IN-- msg_name -
*** ERROR => DpRqCheck: T38 in state TM_SLOT_FREE [dpxxdisp.c 8135]
***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c 6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4231): [dpxxdisp.c 6266]
-IN-- sender_id DISPATCHER tid 38 wp_ca_blk 451 wp_id 5
-IN-- action SEND_TO_WP uid 11318 appc_ca_blk -1 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 8 rq_id 48482
-IN-- req_info CANCELMODE
-IN-- msg_name -
Wed Jan 14 06:33:04 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Wed Jan 14 06:33:25 2015
*** ERROR => MsISnd2: not_attached [msxxi.c 1510]
***LOG Q0N=> DpRqNoWpHandle, MsSnd2Type () [dpxxdisp.c 5557]
Wed Jan 14 06:33:30 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c 718]
Wed Jan 14 06:34:22 2015
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:62123: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 20/sock 1356
(SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:62123) [nixxi.cpp 5082]
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:62123: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 20/sock 1356
(SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:62123) [nixxi.cpp 5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c 2833]
*****************************************************************************
*
* LOCATION SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
* ERROR The connection to the specified message server
* (SAPCRMDEV / 3900) failed. Please check the trace file
* of the message server. If a network filter has
* been activated (see message server parameters
* ms/acl_file_...) please check the configuration
*
* TIME Wed Jan 14 06:34:22 2015
* RELEASE 721
* COMPONENT MS (message handling interface, multithreaded)
* VERSION 4
* RC -35
* MODULE msxxi.c
* LINE 2861
* COUNTER 14239
*
*****************************************************************************
*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c 1867]
***LOG Q1K=> MsIAttachEx: StoC check failed, Kernel not compatible with system (rc=-100) [msxxi.c 820]
*** ERROR => Kernel incompatible to already connected instances (see dev_ms for details) [dpxxdisp.c 12889]
DpHalt: shutdown server >SAPCRMDEV_CRD_00 < (normal)
DpHalt: stop work processes
Wed Jan 14 06:34:28 2015
SoftCancel request for T0 U1 M0 received from DISPATCHER
softCancel request for T22 U10574 M3 received from DISPATCHER
SoftCancel request for T39 U11246 M0 received from DISPATCHER
*** ERROR => DpHdlSoftCancel: terminal has token [dpxxdisp.c 18317]
RM-T22, U10574, 100 DKASHYAP, TermSrvCrestron, 06:32:50, M3, W1, SE24, 4/1
SoftCancel request for T35 U11340 M0 received from DISPATCHER
SoftCancel request for T15 U16 M0 received from DISPATCHER
Wed Jan 14 06:35:29 2015
*** ERROR => DpWpKill(5956, SIGUSR2): kill failed [dpxxtool.c 2792]
DpHalt: stop gateway
DpHalt: stop icman
DpHalt: terminate gui connections
DpHalt: wait for end of work processes
DpHalt: waiting for termination of work processes ...
Wed Jan 14 06:35:30 2015
DpHalt: waiting for termination of work processes ...
Wed Jan 14 06:35:31 2015
DpHalt: waiting for termination of work processes ...
Wed Jan 14 06:35:32 2015
DpHalt: waiting for termination of work processes ...
Wed Jan 14 06:35:33 2015
DpHalt: waiting for termination of work processes ...
Wed Jan 14 06:35:34 2015
pHalt: wait for end of gateway
*** ERROR => [DpProcDied] Process died (PID:880 HANDLE:948 rc:0x0) [dpnttool2.c 147]
DpHalt: wait for end of icman
*** ERROR => [DpProcDied] Process died (PID:4132 HANDLE:956 rc:0x0) [dpnttool2.c 147]
DpHalt: not attached to the message server
DpHalt: cleanup EM
Wed Jan 14 06:35:35 2015
***LOG Q05=> DpHalt, DPStop ( 3664) [dpxxdisp.c 11819]
Thanks,
Kavitha Rajan.