intermittent SCS to LCA connection failures

I am running v8.5 on VMWARE - windows 2012 R2 servers. SCS is running on framework server and time to time losts connection to any LCA running on any other server... Connection lost recovers by itself in a second or so but alerts are generated,,, There is absolutely no pattern of failures, it can happen any time of the day on any LCA/server .... Seems like a hiccup on the newtwork but all servers are on VMWare, there is no real physical network...    Does anyone had similar issue?   any help is greatly appreciated.

-Ap[65617]-<-864 @18:16:46.2390
ADDP: closing 588 - timed out (10000 msec), remote socket [364]
18:16:47.707 Notify Host: HOST_STATUS_UNAVAILABLE {117, W*********}
18:16:47.707   Trace   w****fr**1   SCS   GCTI-43-20511   Disconnected from LCA [W*********] on port [4999], socket [588]
18:16:47.707 ### LOCAL MESSAGE#=20511, 'Disconnected from LCA [W*********] on port [4999], socket [588]'
18:16:47.707   Standard   w****fr**1   SCS   GCTI-00-08002   Host 'W*********' unavailable
18:16:47.707 ### LOCAL MESSAGE#=8002, 'Host 'W*********' unavailable'
18:16:47.707 CREATE ActiveAlarm <107, Host Unavailable> for App=<124, SCS>, createTime=1499822207 under control=1
18:16:47.707   Trace   w****fr**1   SCS   GCTI-43-10612   Active Alarm 'Host Unavailable' created for Application 'SCS'
18:16:47.707 ### LOCAL MESSAGE#=10612, 'Active Alarm 'Host Unavailable' created for Application 'SCS' '
18:16:47.707   Alarm   w****fr**1   SCS   GCTI-00-04115   Host Unavailable, alarm condition(107)
18:16:47.707 ### LOCAL MESSAGE#=4115, 'Host Unavailable, alarm condition(107)'
18:16:47.707   Alarm   w****fr**1   SCS   GCTI-00-04200   Active Alarm "Host Unavailable" Created (ID=4D17FA21-1EB7-44E1-BE4E-FD5927583D1B)
18:16:47.707 ### LOCAL MESSAGE#=4200, 'Active Alarm "Host Unavailable" Created (ID=4D17FA21-1EB7-44E1-BE4E-FD5927583D1B)'
18:16:47.707 ActiveAlarm::SetDeleteTimer <107, Host Unavailable> for App=<124, SCS>, LastSetTime=1499822207, timeout=172800
18:16:47.707 Found Condition <107, Host Unavailable>, Detect Message = 8002
18:16:47.707 !Execute all Reaction for AlarmCondition <107, Host Unavailable>
18:16:47.707   Trace   w****fr**1   SCS   GCTI-43-10500   Reaction 'Send Email Alert' activated
18:16:47.707 ### LOCAL MESSAGE#=10500, 'Reaction 'Send Email Alert' activated'
18:16:47.707 Reaction text: An alarm has been detected by Genesys Solution Management Layer in the n******** environment.

Message: Host 'W*********' unavailable

18:17:15.207 ****************************************************
18:17:15.207   Trace   w****fr**1   SCS   GCTI-43-20510   Connected to LCA [W*********] on port [4999], socket [664]
18:17:15.207 ### LOCAL MESSAGE#=20510, 'Connected to LCA [W*********] on port [4999], socket [664]'
18:17:15.207   Standard   w****fr**1   SCS   GCTI-00-08001   Host 'W*********' operates in normal condition
18:17:15.207 ### LOCAL MESSAGE#=8001, 'Host 'W*********' operates in normal condition'
18:17:15.207 Found Condition <108, Host Unreachable>, Removal Message = 8001
18:17:15.207 Found Condition <107, Host Unavailable>, Removal Message = 8001
18:17:15.207 !Execute all Reaction for AlarmCondition <107, Host Unavailable>
18:17:15.207   Alarm   w****fr**1   SCS   GCTI-00-04210   Active Alarm "Host Unavailable" Cleared (ID=4D17FA21-1EB7-44E1-BE4E-FD5927583D1B)
18:17:15.207 ### LOCAL MESSAGE#=4210, 'Active Alarm "Host Unavailable" Cleared (ID=4D17FA21-1EB7-44E1-BE4E-FD5927583D1B)'
18:17:15.207 ~ActiveAlarm <107, Host Unavailable> for App=<124, SCS>, FistCreateTime=1499822207, SetTime=1499822207, deleteTime=1499822235
18:17:15.207   Trace   w****fr**1   SCS   GCTI-43-10611   Active Alarm 'Host Unavailable' deleted for Application 'SCS'
18:17:15.207 ### LOCAL MESSAGE#=10611, 'Active Alarm 'Host Unavailable' deleted for Application 'SCS''
18:17:15.207 Found Condition <105, Host Inaccessible>, Removal Message = 8001
18:17:15.207 Registered on LCAServer 'W*********' port=4999 (fd=664)
18:17:15.207 Notify Host: HOST_STATUS_RUNNING {117, W*********}
18:17:15.207   Standard   w****fr**1   SCS   GCTI-00-08009   NTP service is available on the host 'W*********'.
18:17:15.207 ### LOCAL MESSAGE#=8009, 'NTP service is available on the host 'W*********'.'
18:17:15.207 REventHostInfo {117, W*********}
18:17:15.207
-AP[148760]->-880 @18:17:16.2700

July 12, 2017
·
Answer
·
Like
1
·
Follow
0

Jakub Nemec
This really seems like problem on VMWare platform, so find out the root-cause there. On other hand, you can configure Genesys to prevent execute switchover in case short outage on SCS<->LCA connection, but it will not fix the issue at all.

July 13, 2017
·
Like
0
·
Dislike
0

Keven Boudreau
If these VM's are using vMotion it may cause some issue. Remove them from the vMotion process and see if it continues to disconnect.

July 18, 2017
·
Like
0
·
Dislike
0

Walter Kirychuk
I faced a similar issue recently and resolved it by configuring ADDP between the host object where the LCA lives, and the SCS. The default with no ADDP configuration is 9 seconds - try increasing to 15 seconds local/20 or 30 seconds remote and enable the trace so you can monitor it. Remember to build the ADDP in the options on the host object rather than one of the apps on the host - and this should keep the SCS in communication with the LCA.
good luck!

July 18, 2017
·
Like
1
·
Dislike
0

Jakub Nemec
ADDP might help keep the connection opened as it sending "keep-alive" packets and prevent resetting the connection by the firewalls. So, it should solve your problem, but on otherhand the problem will be still there (unsolved) and can occurs anytime in the future.

July 19, 2017
·
Like
0
·
Dislike
0

Senih Demren
Thank you all for responding.. Current ADDP (at host level) is set to 10/20. I don't want to increase it too high to avoid missing real failures... Currently issue is under investigation at vmware and network level, I will share details once root cause is identified.

July 20, 2017
·
Like
1
·
Dislike
0

Sabrina Collins
Any updates? We just started getting these lost connections. We are on Windows 2008.

September 6, 2017
·
Like
0
·
Dislike
0

Senih Demren
Hi Sabrina,
Our VM team is still working on the problem so far they identified some kind of SAN connectivity issue with EMC. In our case, its not a Genesys issue definitely related to VM platform. And a reminder, do NOT use VMotion at all, I have official response from Genesys tech, its NOT supported.

September 7, 2017
·
Like
0
·
Dislike
0

Rutger Bockholts
Correct that it's not related to the Genesys platform. We had for a long time the same messages, every day: < host unavalable>, <port lost>, for every server, no mater it was Linux or Windows. Our VMWare tools was very outdated, so update these first the the newest versions. Besides this the biggest improvement was related to changing backup software client on every server. The old backup client (V..... (will not name it)) of every server caused in most of the cases a short block/pause of the vm when the backup job started, which result is a not available server and therefor these messages were generated.
After changing the backup client to a different tool, and updating the vmware tools, we didn't had any port lost, host unavailable, etc, messages anymore.