Dear Experts,
there is a strange problem, that I have been fighting with during the last week or two.
Each morning I can see e-mails from the Solution Manager Technical Monitoring, that the Solution Manager system (7.1 SPS 12)
itself has been reported as unavailable around 22h. It is probably a short outage, as on the next day I always find the system running,
it is NOT necessary to start it again. When reviewing the SM21 system log, I could see a couple of messages:
Request (type DIA) cannot be processed.
I guess that this has been triggering the alerts. Then with time, I have increased the number of DIA work processes, I have also
set the following parameters:
| rdisp/tm_max_no | 1000 |
| rdisp/element_per_queue | 5000 |
This did not resolve the problem, it just changed the type of the error to:
Request (type NOWP) cannot be processed
I have been googling, but didn't find any reasonable solution. The only thing that was worthy to check was:
Dispatcher fails (Request (type NOWP) cannot be processed) - Toolbox for IT Groups
The parameters rdisp/appc_ca_blk_no and rdisp/appc_ca_blk_no have their default values of 1000. If I would calculate them according
to Note 3223, as advised in the thread above, I would come to much smaller values, like about 250, so this can't be my problem.
Also the kernel is much newer than buggy versions, reported in various notes - it is 7.21 EXT patch 300.
I have checked the dev_disp file - there are some strange messages for which I couldn't find an explanation, even after extensive googling.
I am posting here a large part of the log from the time of the error, while trying not to overflow the thread:
Thu Feb 26 22:01:15 2015
*** WARNING => DpTimeCheck: skipped 30 buffer refresh requests because one is in the queue [dpxxdisp.c 9266]
Thu Feb 26 22:05:16 2015
SoftCancel request for T47 U12945 M0 received from IC_MAN
Thu Feb 26 22:05:26 2015
SoftCancel request for T47 U12947 M0 received from IC_MAN
Thu Feb 26 22:05:36 2015
SoftCancel request for T47 U12949 M0 received from IC_MAN
Thu Feb 26 22:05:46 2015
SoftCancel request for T47 U12951 M0 received from IC_MAN
Thu Feb 26 22:05:56 2015
SoftCancel request for T47 U12953 M0 received from IC_MAN
Thu Feb 26 22:06:06 2015
SoftCancel request for T47 U12955 M0 received from IC_MAN
Thu Feb 26 22:06:16 2015
SoftCancel request for T47 U12957 M0 received from IC_MAN
Thu Feb 26 22:06:26 2015
SoftCancel request for T47 U12959 M0 received from IC_MAN
Thu Feb 26 22:06:36 2015
SoftCancel request for T47 U12961 M0 received from IC_MAN
Thu Feb 26 22:06:46 2015
SoftCancel request for T47 U12963 M0 received from IC_MAN
Thu Feb 26 22:06:56 2015
SoftCancel request for T47 U12965 M0 received from IC_MAN
Thu Feb 26 22:07:06 2015
SoftCancel request for T47 U12967 M0 received from IC_MAN
Thu Feb 26 22:07:16 2015
SoftCancel request for T47 U12970 M0 received from IC_MAN
Thu Feb 26 22:07:26 2015
SoftCancel request for T47 U12972 M0 received from IC_MAN
Thu Feb 26 22:07:36 2015
SoftCancel request for T47 U12974 M0 received from IC_MAN
Thu Feb 26 22:07:46 2015
SoftCancel request for T47 U12976 M0 received from IC_MAN
Thu Feb 26 22:08:03 2015
SoftCancel request for T118 U13005 M0 received from IC_MAN
SoftCancel request for T47 U13006 M0 received from IC_MAN
SoftCancel request for T89 U13007 M0 received from IC_MAN
SoftCancel request for T94 U13009 M0 received from IC_MAN
SoftCancel request for T64 U13011 M0 received from IC_MAN
SoftCancel request for T54 U13012 M0 received from IC_MAN
SoftCancel request for T32 U13013 M0 received from IC_MAN
SoftCancel request for T107 U13014 M0 received from IC_MAN
SoftCancel request for T73 U13015 M0 received from IC_MAN
SoftCancel request for T110 U13016 M0 received from IC_MAN
SoftCancel request for T36 U13017 M0 received from IC_MAN
SoftCancel request for T105 U13018 M0 received from IC_MAN
Thu Feb 26 22:08:09 2015
DpHdlDeadWp: W0 (pid=25994) terminated automatically
DpHdlDeadWp: W1 (pid=28898) terminated automatically
DpHdlDeadWp: W2 (pid=27212) terminated automatically
DpHdlDeadWp: W3 (pid=25995) terminated automatically
DpHdlDeadWp: W4 (pid=23595) terminated automatically
DpHdlDeadWp: W5 (pid=23379) terminated automatically
DpHdlDeadWp: W6 (pid=23596) terminated automatically
DpHdlDeadWp: W7 (pid=27214) terminated automatically
DpHdlDeadWp: W8 (pid=27219) terminated automatically
DpHdlDeadWp: W9 (pid=28531) terminated automatically
DpHdlDeadWp: W10 (pid=27216) terminated automatically
DpHdlDeadWp: W11 (pid=5981) terminated automatically
Thu Feb 26 22:08:16 2015
SoftCancel request for T88 U13021 M0 received from IC_MAN
SoftCancel request for T98 U13019 M0 received from IC_MAN
Thu Feb 26 22:08:21 2015
SoftCancel request for T73 U13022 M0 received from IC_MAN
SoftCancel request for T54 U13023 M0 received from IC_MAN
SoftCancel request for T89 U13024 M0 received from IC_MAN
SoftCancel request for T94 U13027 M0 received from IC_MAN
SoftCancel request for T47 U13028 M0 received from IC_MAN
SoftCancel request for T118 U13029 M0 received from IC_MAN
SoftCancel request for T36 U13035 M0 received from IC_MAN
SoftCancel request for T105 U13037 M0 received from IC_MAN
DpHdlDeadWp: W21 (pid=14687) terminated automatically
DpHdlDeadWp: W22 (pid=14625) terminated automatically
Thu Feb 26 22:08:27 2015
DpHdlDeadWp: W0 (pid=14757) terminated automatically
DpHdlDeadWp: W1 (pid=14758) terminated automatically
DpHdlDeadWp: W7 (pid=14766) terminated automatically
DpHdlDeadWp: W9 (pid=14768) terminated automatically
DpHdlDeadWp: W10 (pid=14769) terminated automatically
DpHdlDeadWp: W11 (pid=14770) terminated automatically
Thu Feb 26 22:08:30 2015
SoftCancel request for T32 U13039 M0 received from IC_MAN
SoftCancel request for T110 U13043 M0 received from IC_MAN
SoftCancel request for T51 U13052 M0 received from IC_MAN
SoftCancel request for T36 U13085 M0 received from IC_MAN
SoftCancel request for T98 U13087 M0 received from IC_MAN
SoftCancel request for T25 U13088 M0 received from IC_MAN
SoftCancel request for T35 U13089 M0 received from IC_MAN
SoftCancel request for T64 U13090 M0 received from IC_MAN
SoftCancel request for T105 U13092 M0 received from IC_MAN
SoftCancel request for T88 U13093 M0 received from IC_MAN
Thu Feb 26 22:08:36 2015
DpHdlDeadWp: W2 (pid=14759) terminated automatically
DpHdlDeadWp: W3 (pid=14760) terminated automatically
DpHdlDeadWp: W4 (pid=14763) terminated automatically
DpHdlDeadWp: W5 (pid=14764) terminated automatically
DpHdlDeadWp: W6 (pid=14765) terminated automatically
DpHdlDeadWp: W8 (pid=14767) terminated automatically
DpHdlDeadWp: W21 (pid=14870) terminated automatically
DpHdlDeadWp: W22 (pid=14871) terminated automatically
SoftCancel request for T54 U13095 M0 received from IC_MAN
SoftCancel request for T94 U13096 M0 received from IC_MAN
SoftCancel request for T73 U13098 M0 received from IC_MAN
SoftCancel request for T47 U13100 M0 received from IC_MAN
SoftCancel request for T89 U13101 M0 received from IC_MAN
Thu Feb 26 22:08:41 2015
DpHdlDeadWp: W0 (pid=15233) terminated automatically
DpHdlDeadWp: W1 (pid=15234) terminated automatically
DpHdlDeadWp: W7 (pid=15235) terminated automatically
DpHdlDeadWp: W9 (pid=15236) terminated automatically
DpHdlDeadWp: W10 (pid=15237) terminated automatically
Thu Feb 26 22:08:42 2015
SoftCancel request for T32 U13102 M0 received from IC_MAN
Thu Feb 26 22:08:48 2015
DpHdlDeadWp: W11 (pid=15238) terminated automatically
Thu Feb 26 22:08:52 2015
SoftCancel request for T35 U13106 M0 received from IC_MAN
SoftCancel request for T36 U13107 M0 received from IC_MAN
SoftCancel request for T110 U13111 M0 received from IC_MAN
SoftCancel request for T98 U13112 M0 received from IC_MAN
SoftCancel request for T105 U13113 M0 received from IC_MAN
SoftCancel request for T64 U13119 M0 received from IC_MAN
SoftCancel request for T51 U13122 M0 received from IC_MAN
SoftCancel request for T26 U13121 M0 received from IC_MAN
SoftCancel request for T118 U13126 M0 received from IC_MAN
SoftCancel request for T50 U13135 M0 received from IC_MAN
SoftCancel request for T57 U13183 M0 received from IC_MAN
SoftCancel request for T49 U13185 M0 received from IC_MAN
SoftCancel request for T109 U13186 M0 received from IC_MAN
SoftCancel request for T25 U13187 M0 received from IC_MAN
SoftCancel request for T47 U13191 M0 received from IC_MAN
SoftCancel request for T73 U13193 M0 received from IC_MAN
SoftCancel request for T94 U13195 M0 received from IC_MAN
SoftCancel request for T43 U13194 M0 received from IC_MAN
SoftCancel request for T89 U13197 M0 received from IC_MAN
Thu Feb 26 22:08:57 2015
DpHdlDeadWp: W0 (pid=17063) terminated automatically
DpHdlDeadWp: W1 (pid=17065) terminated automatically
DpHdlDeadWp: W2 (pid=15886) terminated automatically
DpHdlDeadWp: W3 (pid=15887) terminated automatically
DpHdlDeadWp: W4 (pid=15888) terminated automatically
DpHdlDeadWp: W5 (pid=15889) terminated automatically
DpHdlDeadWp: W6 (pid=15890) terminated automatically
DpHdlDeadWp: W7 (pid=17069) terminated automatically
DpHdlDeadWp: W8 (pid=15891) terminated automatically
DpHdlDeadWp: W9 (pid=17071) terminated automatically
DpHdlDeadWp: W21 (pid=15892) terminated automatically
DpHdlDeadWp: W22 (pid=15893) terminated automatically
Thu Feb 26 22:09:02 2015
SoftCancel request for T54 U13198 M0 received from IC_MAN
SoftCancel request for T110 U13221 M0 received from IC_MAN
Thu Feb 26 22:09:07 2015
DpHdlDeadWp: W10 (pid=17073) terminated automatically
DpHdlDeadWp: W11 (pid=18503) terminated automatically
Thu Feb 26 22:09:08 2015
SoftCancel request for T57 U13234 M0 received from IC_MAN
SoftCancel request for T35 U13235 M0 received from IC_MAN
SoftCancel request for T50 U13236 M0 received from IC_MAN
SoftCancel request for T73 U13240 M0 received from IC_MAN
SoftCancel request for T109 U13242 M0 received from IC_MAN
SoftCancel request for T32 U13253 M0 received from IC_MAN
Thu Feb 26 22:09:18 2015
SoftCancel request for T118 U13297 M0 received from IC_MAN
SoftCancel request for T105 U13306 M0 received from IC_MAN
SoftCancel request for T117 U13307 M0 received from IC_MAN
SoftCancel request for T68 U13311 M0 received from IC_MAN
SoftCancel request for T236 U13478 M0 received from IC_MAN
SoftCancel request for T246 U13495 M0 received from IC_MAN
SoftCancel request for T242 U13712 M0 received from IC_MAN
SoftCancel request for T451 U13713 M0 received from IC_MAN
SoftCancel request for T440 U13716 M0 received from IC_MAN
SoftCancel request for T247 U13728 M0 received from IC_MAN
SoftCancel request for T224 U13719 M0 received from IC_MAN
SoftCancel request for T240 U13768 M0 received from IC_MAN
Thu Feb 26 22:09:28 2015
SoftCancel request for T230 U13471 M0 received from IC_MAN
SoftCancel request for T231 U13473 M0 received from IC_MAN
SoftCancel request for T227 U13468 M0 received from IC_MAN
SoftCancel request for T109 U13458 M0 received from IC_MAN
Thu Feb 26 22:09:35 2015
SoftCancel request for T109 U13458 M0 received from IC_MAN
DpHdlSoftCancel: wp 6 inside critical section, do not interrupt now
*** ERROR => DpRqCheck: T109 in state TM_SLOT_FREE [dpxxdisp.c 8009]
***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c 6137]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4229): [dpxxdisp.c 6139]
-IN-- sender_id IC_MAN tid 109 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13458 appc_ca_blk -1 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 0 rq_id 52599
-IN-- req_info DP_ICM_EVENT
-IN-- mpi read <1e98b> mpi write <1e98c> prot 1 obj fd -1 rq_id 0
-IN-- icm_opcode: ICM_PLUGIN_REQUEST virt_host_idx: 0
-IN-- dtrace {root-id=8010E03E596C1ED4AFBE29E311A414C7}_{conn-id=8010E03E596C1EE4AFBE29E311FF525C}_1
-IN-- msg_name -
DpRqCheck: mode 0 in status CANCEL***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c 6137]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4229): [dpxxdisp.c 6139]
-IN-- sender_id IC_MAN tid 227 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13468 appc_ca_blk -1 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 0 rq_id 52605
-IN-- req_info DP_ICM_EVENT
-IN-- mpi read <1e99f> mpi write <1e9a0> prot 1 obj fd -1 rq_id 0
-IN-- icm_opcode: ICM_PLUGIN_REQUEST virt_host_idx: 0
-IN-- dtrace {root-id=8010E03E596C1ED4AFBE2CE0A7875FCC}_{conn-id=8010E03E596C1EE4AFBE2CE0A7CC0FD9}_1
-IN-- msg_name -
SoftCancel request for T227 U13468 M0 received from IC_MAN
DpHdlSoftCancel: wp 6 inside critical section, do not interrupt now
DpRqCheck: mode 0 in status CANCEL***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c 6137]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4229): [dpxxdisp.c 6139]
-IN-- sender_id IC_MAN tid 231 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13473 appc_ca_blk -1 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 0 rq_id 52609
-IN-- req_info DP_ICM_EVENT
-IN-- mpi read <1e975> mpi write <1e976> prot 1 obj fd -1 rq_id 0
-IN-- icm_opcode: ICM_PLUGIN_REQUEST virt_host_idx: 0
-IN-- dtrace {root-id=8010E03E596C1ED4AFBE2FC94B8F9DD9}_{conn-id=8010E03E596C1EE4AFBE2FC94BD316A8}_1
-IN-- msg_name -
SoftCancel request for T231 U13473 M0 received from IC_MAN
DpHdlSoftCancel: wp 6 inside critical section, do not interrupt now
DpRqCheck: mode 0 in status CANCEL***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c 6137]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4229): [dpxxdisp.c 6139]
-IN-- sender_id IC_MAN tid 230 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13471 appc_ca_blk -1 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 0 rq_id 52613
-IN-- req_info DP_ICM_EVENT
-IN-- mpi read <1e99b> mpi write <1e99c> prot 1 obj fd -1 rq_id 0
-IN-- icm_opcode: ICM_PLUGIN_REQUEST virt_host_idx: 0
-IN-- dtrace {root-id=8010E03E596C1ED4AFBE2E4B6EB70A5B}_{conn-id=8010E03E596C1EE4AFBE2E4B6EFF9635}_1
-IN-- msg_name -
SoftCancel request for T230 U13471 M0 received from IC_MAN
DpHdlSoftCancel: wp 6 inside critical section, do not interrupt now
-IN-- sender_id APPC_SERVER tid 240 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13768 appc_ca_blk 73 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 1038 rq_id 52617
-IN-- req_info DP_ASYNC_RFC
-IN-- msg_name -
SoftCancel request for T445 U13973 M0 received from APPC_SERVER
-IN-- sender_id APPC_SERVER tid 224 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13719 appc_ca_blk 49 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 1038 rq_id 52629
-IN-- req_info DP_ASYNC_RFC
-IN-- msg_name -
SoftCancel request for T116 U13966 M0 received from APPC_SERVER
-IN-- sender_id APPC_SERVER tid 247 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13728 appc_ca_blk 56 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 1038 rq_id 52642
-IN-- req_info DP_ASYNC_RFC
-IN-- msg_name -
SoftCancel request for T30 U13968 M0 received from APPC_SERVER
-IN-- sender_id APPC_SERVER tid 440 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13716 appc_ca_blk 50 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 1038 rq_id 52653
-IN-- req_info DP_ASYNC_RFC
-IN-- msg_name -
SoftCancel request for T423 U13967 M0 received from APPC_SERVER
-IN-- sender_id APPC_SERVER tid 451 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13713 appc_ca_blk 46 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 1038 rq_id 52666
-IN-- req_info DP_ASYNC_RFC
-IN-- msg_name -
SoftCancel request for T23 U13965 M0 received from APPC_SERVER
-IN-- sender_id APPC_SERVER tid 242 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13712 appc_ca_blk 70 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 1038 rq_id 52679
-IN-- req_info DP_ASYNC_RFC
-IN-- msg_name -
SoftCancel request for T449 U13969 M0 received from APPC_SERVER
-IN-- sender_id APPC_SERVER tid 246 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13495 appc_ca_blk 71 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 1038 rq_id 52692
-IN-- req_info DP_ASYNC_RFC
-IN-- msg_name -
SoftCancel request for T446 U13963 M0 received from APPC_SERVER
-IN-- sender_id APPC_SERVER tid 236 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13478 appc_ca_blk 62 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 1038 rq_id 52705
-IN-- req_info DP_ASYNC_RFC
-IN-- msg_name -
SoftCancel request for T447 U13971 M0 received from APPC_SERVER
-IN-- sender_id APPC_SERVER tid 68 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13311 appc_ca_blk 69 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 1038 rq_id 52719
-IN-- req_info DP_ASYNC_RFC
-IN-- msg_name -
SoftCancel request for T330 U13962 M0 received from APPC_SERVER
-IN-- sender_id APPC_SERVER tid 117 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13307 appc_ca_blk 67 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 1038 rq_id 52733
-IN-- req_info DP_ASYNC_RFC
-IN-- msg_name -
SoftCancel request for T239 U13961 M0 received from APPC_SERVER
-IN-- sender_id APPC_SERVER tid 105 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13306 appc_ca_blk 68 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 1038 rq_id 52740
-IN-- req_info DP_ASYNC_RFC
-IN-- msg_name -
SoftCancel request for T84 U13960 M0 received from APPC_SERVER
-IN-- sender_id APPC_SERVER tid 118 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13297 appc_ca_blk 65 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 1038 rq_id 52763
-IN-- req_info DP_ASYNC_RFC
-IN-- msg_name -
SoftCancel request for T25 U13956 M0 received from APPC_SERVER
Thu Feb 26 22:09:46 2015
SoftCancel request for T220 U13975 M0 received from IC_MAN
Thu Feb 26 22:09:51 2015
DpHdlDeadWp: W7 (pid=20251) terminated automatically
Thu Feb 26 22:09:52 2015
SoftCancel request for T105 U13978 M0 received from IC_MAN
SoftCancel request for T117 U13980 M0 received from IC_MAN
SoftCancel request for T118 U13982 M0 received from IC_MAN
Thu Feb 26 22:09:58 2015
DpHdlDeadWp: W2 (pid=20246) terminated automatically
DpHdlDeadWp: W3 (pid=20247) terminated automatically
DpHdlDeadWp: W6 (pid=20250) terminated automatically
Thu Feb 26 22:10:06 2015
SoftCancel request for T117 U13987 M0 received from IC_MAN
SoftCancel request for T118 U13988 M0 received from IC_MAN
SoftCancel request for T220 U13984 M0 received from IC_MAN
SoftCancel request for T105 U13986 M0 received from IC_MAN
Thu Feb 26 22:10:12 2015
DpHdlDeadWp: W2 (pid=23682) terminated automatically
DpHdlDeadWp: W3 (pid=23683) terminated automatically
DpHdlDeadWp: W6 (pid=23684) terminated automatically
DpHdlDeadWp: W7 (pid=22489) terminated automatically
Thu Feb 26 22:10:26 2015
SoftCancel request for T220 U13996 M0 received from IC_MAN
SoftCancel request for T105 U13994 M0 received from IC_MAN
SoftCancel request for T118 U13990 M0 received from IC_MAN
SoftCancel request for T102 U13995 M0 received from IC_MAN
Thu Feb 26 22:10:31 2015
DpHdlDeadWp: W2 (pid=24083) terminated automatically
DpHdlDeadWp: W3 (pid=24085) terminated automatically
DpHdlDeadWp: W6 (pid=24086) terminated automatically
DpHdlDeadWp: W7 (pid=24087) terminated automatically
Thu Feb 26 22:10:33 2015
-IN-- sender_id APPC_SERVER tid 116 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13966 appc_ca_blk 73 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 80 rq_id 53415
-IN-- msg_name -
-IN-- sender_id APPC_SERVER tid 445 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13973 appc_ca_blk 61 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 80 rq_id 53418
-IN-- msg_name -
-IN-- sender_id APPC_SERVER tid 30 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13968 appc_ca_blk 78 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 80 rq_id 53421
-IN-- msg_name -
-IN-- sender_id APPC_SERVER tid 423 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13967 appc_ca_blk 56 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 80 rq_id 53424
-IN-- msg_name -
-IN-- sender_id APPC_SERVER tid 23 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13965 appc_ca_blk 79 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 80 rq_id 53429
-IN-- msg_name -
-IN-- sender_id APPC_SERVER tid 449 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13969 appc_ca_blk 80 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 80 rq_id 53432
-IN-- msg_name -
-IN-- sender_id APPC_SERVER tid 446 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13963 appc_ca_blk 81 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 80 rq_id 53435
-IN-- msg_name -
-IN-- sender_id APPC_SERVER tid 447 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13971 appc_ca_blk 82 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 80 rq_id 53438
-IN-- msg_name -
-IN-- sender_id APPC_SERVER tid 330 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13962 appc_ca_blk 83 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 80 rq_id 53441
-IN-- msg_name -
-IN-- sender_id APPC_SERVER tid 84 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13960 appc_ca_blk 86 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 80 rq_id 53446
-IN-- msg_name -
-IN-- sender_id APPC_SERVER tid 239 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13961 appc_ca_blk 85 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 80 rq_id 53449
-IN-- msg_name -
-IN-- sender_id APPC_SERVER tid 25 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 13956 appc_ca_blk 65 type NOWP
-IN-- new_stat NO_CHANGE mode 0 len 80 rq_id 53452
-IN-- msg_name -
DIA water mark underrun (<=30)
Thu Feb 26 22:10:46 2015
SoftCancel request for T118 U14020 M0 received from IC_MAN
SoftCancel request for T25 U14024 M0 received from IC_MAN
SoftCancel request for T40 U14036 M0 received from APPC_SERVER
SoftCancel request for T441 U14037 M0 received from APPC_SERVER
Thu Feb 26 22:10:49 2015
SoftCancel request for T239 U14013 M0 received from IC_MAN
SoftCancel request for T117 U14012 M0 received from IC_MAN
SoftCancel request for T107 U14038 M0 received from APPC_SERVER
SoftCancel request for T452 U14034 M0 received from APPC_SERVER
Thu Feb 26 22:10:59 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ
Thu Feb 26 22:11:02 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ
Thu Feb 26 22:11:37 2015
DpWPCheck: dyn W22, pid 20255 no longer needed, terminate now
Thu Feb 26 22:11:39 2015
DpHdlDeadWp: W22 (pid=20255) terminated automatically
Thu Feb 26 22:11:53 2015
DpHdlDeadWp: W15 (pid=28202) terminated automatically
DpWPCheck: dyn W21, pid 20254 no longer needed, terminate now
DpHdlDeadWp: W21 (pid=20254) terminated automatically
Thu Feb 26 22:12:27 2015
DpHdlDeadWp: W14 (pid=20344) terminated automatically
Thu Feb 26 22:12:41 2015
DpHdlDeadWp: W16 (pid=22481) terminated automatically
Thu Feb 26 22:14:22 2015
DIA water mark reached (>30)
Thu Feb 26 22:14:27 2015
DIA water mark underrun (<=30)
DIA water mark reached (>30)
DIA water mark underrun (<=30)
Thu Feb 26 22:14:29 2015
DIA water mark reached (>30)
Thu Feb 26 22:14:30 2015
DIA water mark underrun (<=30)
Many many thanks for your analysis and help in advance!