クラスタ固有の Docker ログ調査

INS-CD1: ネットワークの問題または特定ノードのインターフェイスのダウン

特定のノードにネットワークの問題またはインターフェイスのダウン場合、さまざまなエラーが発生する可能性があります。特に、特定のノードで Docker のログを確認すると、「ログのサンプルエントリ」のようなログが表示される可能性があります。

主なエラーメッセージ

• msg="error receiving response" error="rpc error: code = Canceled desc = context canceled"
• msg="memberlist: Failed to send UDP ping: write udp [::]:7946->10.20.47.12:7946: sendto: network is unreachable"
• level=warning msg="bulk sync to node 26b516451957 failed: failed to send a TCP message during bulk sync: dial tcp 10.20.47.12:7946: connect: network is unreachable"
• level=error msg="error receiving response" error="rpc error: code = Canceled desc = context canceled"

ログのサンプルエントリ

Node: ke2-rhel89-swarm-3

Sep  9 13:24:53 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:53.660104866+09:00" level=warning msg="memberlist: Failed to send UDP ping: write udp [::]:7946->10.20.47.12:7946: sendto: network is unreachable"
Sep  9 13:24:53 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:53.660393049+09:00" level=warning msg="memberlist: Failed to send indirect UDP ping: write udp [::]:7946->10.20.47.11:7946: sendto: network is unreachable"
Sep  9 13:24:53 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:53.660411484+09:00" level=warning msg="bulk sync to node d7dbb1513873 failed: failed to send a TCP message during bulk sync: dial tcp 10.20.47.11:7946: connect: network is unreachable"
Sep  9 13:24:53 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:53.660646948+09:00" level=warning msg="bulk sync to node 26b516451957 failed: failed to send a TCP message during bulk sync: dial tcp 10.20.47.12:7946: connect: network is unreachable"
Sep  9 13:24:53 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:53.660658881+09:00" level=error msg="periodic bulk sync failure for network wxpj9trr7nqzfxyf5jo20o04d: bulk sync to node 26b516451957 failed: failed to send a TCP message during bulk sync: dial tcp 10.20.47.12:7946: connect: network is unreachable"
Sep  9 13:24:53 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:53.660757727+09:00" level=warning msg="bulk sync to node 26b516451957 failed: failed to send a TCP message during bulk sync: dial tcp 10.20.47.12:7946: connect: network is unreachable"
Sep  9 13:24:53 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:53.660866121+09:00" level=warning msg="bulk sync to node d7dbb1513873 failed: failed to send a TCP message during bulk sync: dial tcp 10.20.47.11:7946: connect: network is unreachable"
Sep  9 13:24:53 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:53.660881811+09:00" level=error msg="periodic bulk sync failure for network wgjapriy9ytdcbrd2z42isy9w: bulk sync to node d7dbb1513873 failed: failed to send a TCP message during bulk sync: dial tcp 10.20.47.11:7946: connect: network is unreachable"
Sep  9 13:24:54 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:54.660523142+09:00" level=info msg="memberlist: Suspect 26b516451957 has failed, no acks received"
Sep  9 13:24:54 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:54.661032423+09:00" level=warning msg="memberlist: Failed to send UDP compound ping and suspect message to 10.20.47.12:7946: write udp [::]:7946->10.20.47.12:7946: sendto: network is unreachable"
Sep  9 13:24:54 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:54.661122262+09:00" level=warning msg="memberlist: Failed to send indirect UDP ping: write udp [::]:7946->10.20.47.11:7946: sendto: network is unreachable"
Sep  9 13:24:54 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:54.859948875+09:00" level=warning msg="memberlist: Failed to send gossip to 10.20.47.11:7946: write udp [::]:7946->10.20.47.11:7946: sendto: network is unreachable"
Sep  9 13:24:54 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:54.860013167+09:00" level=warning msg="memberlist: Failed to send gossip to 10.20.47.12:7946: write udp [::]:7946->10.20.47.12:7946: sendto: network is unreachable"
Sep  9 13:24:55 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:55.060338976+09:00" level=warning msg="memberlist: Failed to send gossip to 10.20.47.12:7946: write udp [::]:7946->10.20.47.12:7946: sendto: network is unreachable"
Sep  9 13:24:56 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:56.661696183+09:00" level=info msg="memberlist: Suspect 26b516451957 has failed, no acks received"
Sep  9 13:24:56 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:56.661843631+09:00" level=warning msg="memberlist: Failed to send UDP ping: write udp [::]:7946->10.20.47.11:7946: sendto: network is unreachable"
Sep  9 13:24:58 ke2-rhel89-swarm-3 dockerd[1441]: time="2024-09-09T13:24:58.661028298+09:00" level=info msg="memberlist: Marking 26b516451957 as failed, suspect timeout reached (0 peer confirmations)"

解決策

ネットワークの問題が解決されると、Docker Swarmノードは通常、自動的に回復します。ただし、回復になってない場合は、ノード間通信をご確認ください。ノード間通信が正常に戻っても、特定のノードでKE2 APPが正常に動作していない場合は、Dockerサービスを再起動してください。

$ systemctl restart docker.service

INS-CD2: 外部データベースまたはSwarmノードとの高いネットワーク遅延

外部データベースやSwarmノードとの間で高いネットワーク遅延や高いパケットロスが発生する場合、さまざまなエラーが発生する可能性があります。特に、特定ノードの Docker のログを確認すると、「ログのサンプルエントリ」のようなログが表示される可能性があります。

主なエラーメッセージ

• level=error msg="fatal task error" error="task: non-zero exit (1)"
• level=warning msg="failed to deactivate service binding for container ke2_kengine.2.s9jv96yf2uw0k88omd8y11erw" error="No such container: ke2_kengine.2.s9jv96yf2uw0k88omd8y11erw"
• level=warning msg="NetworkDB stats ke2-rhel89-swarm-2(a6a2e4038ef8) - healthscore:1 (connectivity issues)"

ログのサンプルエントリ

Node: ke2-rhel89-swarm-2

Sep 11 18:32:17 ke2-rhel89-swarm-2 dockerd[785695]: time="2024-09-02T08:47:39.509568544+09:00" level=warning msg="NetworkDB stats ke2-rhel89-swarm-2(a6a2e4038ef8) - healthscore:1 (connectivity issues)"
Sep 11 18:32:17 ke2-rhel89-swarm-2 dockerd[785695]: time="2024-09-11T18:32:17.192989709+09:00" level=info msg="NetworkDB stats ke2-rhel89-swarm-2(05360ec70c80) - netID:rk6harno4s00l98f6ho4rdb7p leaving:false netPeers:3 entries:37 Queue qLen:0 netMsg/s:0"
Sep 11 18:32:17 ke2-rhel89-swarm-2 dockerd[785695]: time="2024-09-11T18:32:17.193462600+09:00" level=info msg="NetworkDB stats ke2-rhel89-swarm-2(05360ec70c80) - netID:wxpj9trr7nqzfxyf5jo20o04d leaving:false netPeers:3 entries:15 Queue qLen:0 netMsg/s:0"
Sep 11 18:37:17 ke2-rhel89-swarm-2 dockerd[785695]: time="2024-09-11T18:37:17.392411952+09:00" level=info msg="NetworkDB stats ke2-rhel89-swarm-2(05360ec70c80) - netID:rk6harno4s00l98f6ho4rdb7p leaving:false netPeers:3 entries:35 Queue qLen:0 netMsg/s:0"
Sep 11 18:37:17 ke2-rhel89-swarm-2 dockerd[785695]: time="2024-09-11T18:37:17.393110199+09:00" level=info msg="NetworkDB stats ke2-rhel89-swarm-2(05360ec70c80) - netID:wxpj9trr7nqzfxyf5jo20o04d leaving:false netPeers:3 entries:15 Queue qLen:0 netMsg/s:0"
Sep 11 18:39:58 ke2-rhel89-swarm-2 dockerd[785695]: time="2024-09-11T18:39:58.069328404+09:00" level=info msg="ignoring event" container=e0a3465370bbeb20eb84952344933a158fe9665b295653591774b5df8e5a489a module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Sep 11 18:39:58 ke2-rhel89-swarm-2 dockerd[785695]: [2024-09-11T18:39:58+09:00] Discarding queued events...
Sep 11 18:39:58 ke2-rhel89-swarm-2 dockerd[785695]: time="2024-09-11T18:39:58.185132146+09:00" level=error msg="fatal task error" error="task: non-zero exit (1)" module=node/agent/taskmanager node.id=ly4ik4v3pw9rgom0htt281aus service.id=2gjb1rv9xn4i9hlydsmt15nio task.id=cnstkeybneberx2tcloc0r3t1
Sep 11 18:39:58 ke2-rhel89-swarm-2 dockerd[785695]: time="2024-09-11T18:39:58.607441058+09:00" level=warning msg="failed to deactivate service binding for container ke2_kengine.2.s9jv96yf2uw0k88omd8y11erw" error="No such container: ke2_kengine.2.s9jv96yf2uw0k88omd8y11erw" module=node/agent node.id=ly4ik4v3pw9rgom0htt281aus

解決策

外部データベースやSwarmノードとの間でネットワークの問題が解決されると、KE2 APP が自動的に回復します。ただし、問題が回復にならない場合は、以下の状況を見てください。

• ノード間通信をご確認ください。
• KE2 APP ホストサーバと外部データベース間のネットワーク導通をご確認ください。
• 外部データベース間のネットワーク導通に問題があれば kengine が不安定の可能性があります。

  • kengine の状況をご確認ください。

    docker ps -a -f name=kengine
    

    kengine が再起動を繰り返す場合、kengine コンテナを再起動してください。

    # kengine コンテナ ID: $ docker ps -q -f name=kengine を実行してコンテナ ID を取得できます。
    $ docker restart < kengine コンテナ ID>
    

  • 特定ノードの KE2 APP が正常に戻らな場合 docker サービスを再起度してください。

    $ systemctl restart docker.service
    

INS-CD3: Docker 不安定性

特定のノードにネットワークの問題、またはノードにリソース使用率が高い場合、さまざまなエラーが発生する可能性があります。特に、特定のノードで以下のコマンドで Docker のログを確認すると、「ログのサンプルエントリ」のようなログが表示される可能性があります。

主なエラーメッセージ

• underweighting node <node-id> for service <service-id> because it experienced 5 failures or rejections within 5m0s
• Could not parse VIP address while releasing
• error deallocating vip error="invalid CIDR address: " vip.addr= vip.network=<network-id>
• Failed to allocate network resources for node <node-id> error="could not find network allocator state for network <network-id>"

ログのサンプルエントリ

Node: ke2-rhel89-swarm-1

Sep 19 14:37:21 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:21.482572833+09:00" level=warning msg="underweighting node sw16fqkz94al7qwwqeksgtzeh for service m23oykzvchn3nc6n6cvsyebso because it experienced 5 failures or rejections within 5m0s" module=scheduler node.id=apy0tpjdo8njulc4kxkm6u53i
Sep 19 14:37:21 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:21.482965448+09:00" level=error msg="Could not parse VIP address  while releasing"
Sep 19 14:37:21 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:21.482987921+09:00" level=error msg="error deallocating vip" error="invalid CIDR address: " vip.addr= vip.network=ut2671phg9ixr96r49j1uambb
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.286255147+09:00" level=error msg="Could not parse VIP address  while releasing"
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.286490844+09:00" level=error msg="error deallocating vip" error="invalid CIDR address: " vip.addr= vip.network=ut2671phg9ixr96r49j1uambb
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.300954287+09:00" level=error msg="Could not parse VIP address  while releasing"
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.300982541+09:00" level=error msg="error deallocating vip" error="invalid CIDR address: " vip.addr= vip.network=ut2671phg9ixr96r49j1uambb
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.308515598+09:00" level=error msg="Could not parse VIP address  while releasing"
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.308531979+09:00" level=error msg="error deallocating vip" error="invalid CIDR address: " vip.addr= vip.network=ut2671phg9ixr96r49j1uambb
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.317007374+09:00" level=error msg="Event api.EventUpdateTask: Failed to get service z0hc7bmssm60q3i3th0bg3tpy for task qfg8afb4gmdo35qnr1aeiaxnw state NEW: could not find service z0hc7bmssm60q3i3th0bg3tpy" module=node node.id=apy0tpjdo8njulc4kxkm6u53i
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.317044034+09:00" level=error msg="Event api.EventUpdateTask: Failed to get service z0hc7bmssm60q3i3th0bg3tpy for task uoqxq0l8yoy8sybeyakshu6ou state NEW: could not find service z0hc7bmssm60q3i3th0bg3tpy" module=node node.id=apy0tpjdo8njulc4kxkm6u53i
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.317071836+09:00" level=error msg="Event api.EventUpdateTask: Failed to get service z0hc7bmssm60q3i3th0bg3tpy for task z5o1144xliwegvminmu6l017u state NEW: could not find service z0hc7bmssm60q3i3th0bg3tpy" module=node node.id=apy0tpjdo8njulc4kxkm6u53i
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.323309366+09:00" level=error msg="Event api.EventUpdateTask: Failed to get service kiziizne7llebo4klwhnt2v1d for task il2yrfjr9xj60q48upezgu43g state NEW: could not find service kiziizne7llebo4klwhnt2v1d" module=node node.id=apy0tpjdo8njulc4kxkm6u53i
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.323340395+09:00" level=error msg="Event api.EventUpdateTask: Failed to get service kiziizne7llebo4klwhnt2v1d for task qds3ohkgzl8xmbbuxi88jh7s9 state NEW: could not find service kiziizne7llebo4klwhnt2v1d" module=node node.id=apy0tpjdo8njulc4kxkm6u53i
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.323360243+09:00" level=error msg="Event api.EventUpdateTask: Failed to get service kiziizne7llebo4klwhnt2v1d for task wh098t4mzlm8kl20rdrz5lpe6 state NEW: could not find service kiziizne7llebo4klwhnt2v1d" module=node node.id=apy0tpjdo8njulc4kxkm6u53i
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.361772687+09:00" level=warning msg="network ofrqt0bvca0uddwrc4t23vkcg should be removed, but still has active attachments" module=node/agent node.id=apy0tpjdo8njulc4kxkm6u53i
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.361858459+09:00" level=info msg="initialized VXLAN UDP port to 4790 " module=node node.id=apy0tpjdo8njulc4kxkm6u53i
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.367768599+09:00" level=error msg="Failed to allocate network resources for node apy0tpjdo8njulc4kxkm6u53i" error="could not find network allocator state for network ofrqt0bvca0uddwrc4t23vkcg" module=node node.id=apy0tpjdo8njulc4kxkm6u53i
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.367864079+09:00" level=error msg="Failed to allocate network resources for node lgwvkt31hp7iikk425s1hjbi6" error="could not find network allocator state for network ofrqt0bvca0uddwrc4t23vkcg" module=node node.id=apy0tpjdo8njulc4kxkm6u53i
Sep 19 14:37:47 ke2-rhel89-swarm-1 dockerd[1064]: time="2024-09-19T14:37:47.367947097+09:00" level=error msg="Failed to allocate network resources for node sw16fqkz94al7qwwqeksgtzeh" error="could not find network allocator state for network ofrqt0bvca0uddwrc4t23vkcg" module=node node.id=apy0tpjdo8njulc4kxkm6u53i

解決策

Node Underweighting: あるノードが短期間に複数回の失敗やタスク拒否を経験したため、Docker Swarmのスケジューリングアルゴリズムがこのノードの優先度を下げて、他のノードにタスクを振り分けようとしています。原因としては、ノードの過負荷やネットワークの問題が考えられます。

• 特定ホストサーバのリソース状況を確認確認してください。
• ネットワーク導通を確認してください。
• 特定ノードのコンテナで利用のリソース状況をご確認ください。
• 特定ノードの KE2 APP が正常に戻らな場合 docker サービスを再起度してください。

  $ systemctl restart docker.service
  

INS-CD4: Swarm ノードダウン・VMダウン・ネットワーク障害

VMダウン・ネットワーク障害・Dockerダウン場合、他のノードの docker でさまざまなエラーが発生する可能性があります。他のノード docker のログを確認すると、「ログのサンプルエントリ」のようなログが表示される可能性があります。

主なエラーメッセージ

• error sending message to peer
• rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.20.47.12:2377: connect: no route to host"

ログのサンプルエントリ

Node: ke2-rhel89-swarm-1

Sep 12 11:32:20 ke2-rhel89-swarm-1 dockerd[1428]: time="2024-09-12T11:32:20.523563797+09:00" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.20.47.12:2377: connect: no route to host\""
Sep 12 11:32:21 ke2-rhel89-swarm-1 dockerd[1428]: time="2024-09-12T11:32:21.523164173+09:00" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.20.47.12:2377: connect: no route to host\""
Sep 12 11:32:22 ke2-rhel89-swarm-1 dockerd[1428]: time="2024-09-12T11:32:22.523003474+09:00" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.20.47.12:2377: connect: no route to host\""
Sep 12 11:32:23 ke2-rhel89-swarm-1 dockerd[1428]: time="2024-09-12T11:32:23.522837316+09:00" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.20.47.12:2377: connect: no route to host\""
Sep 12 11:32:24 ke2-rhel89-swarm-1 dockerd[1428]: time="2024-09-12T11:32:24.522786775+09:00" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.20.47.12:2377: connect: no route to host\""
Sep 12 11:32:25 ke2-rhel89-swarm-1 dockerd[1428]: time="2024-09-12T11:32:25.522822736+09:00" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.20.47.12:2377: connect: no route to host\""
Sep 12 11:32:26 ke2-rhel89-swarm-1 dockerd[1428]: time="2024-09-12T11:32:26.523336108+09:00" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.20.47.12:2377: connect: no route to host\""
Sep 12 11:32:27 ke2-rhel89-swarm-1 dockerd[1428]: time="2024-09-12T11:32:27.523037596+09:00" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.20.47.12:2377: connect: no route to host\""
Sep 12 11:32:28 ke2-rhel89-swarm-1 dockerd[1428]: time="2024-09-12T11:32:28.522947326+09:00" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.20.47.12:2377: connect: no route to host\""
Sep 12 11:32:29 ke2-rhel89-swarm-1 dockerd[1428]: time="2024-09-12T11:32:29.522647103+09:00" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.20.47.12:2377: connect: no route to host\""
Sep 12 11:32:30 ke2-rhel89-swarm-1 dockerd[1428]: time="2024-09-12T11:32:30.523393817+09:00" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.20.47.12:2377: connect: no route to host\""
Sep 12 11:32:31 ke2-rhel89-swarm-1 dockerd[1428]: time="2024-09-12T11:32:31.523423053+09:00" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.20.47.12:2377: connect: no route to host\""
Sep 12 11:32:32 ke2-rhel89-swarm-1 dockerd[1428]: time="2024-09-12T11:32:32.523047422+09:00" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.20.47.12:2377: connect: no route to host\""
Sep 12 11:32:33 ke2-rhel89-swarm-1 dockerd[1428]: time="2024-09-12T11:32:33.523162797+09:00" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.20.47.12:2377: connect: no route to host\""

解決策

• ホストサーバの正常性をご確認ください。
• ネットワーク導通を確認してください。
• docker の正常動作をご確認ください

INS-CD5: Swarmノードとの高いネットワーク遅延

Swarmノードとの高いネットワーク遅延・高いパケットロス場合、さまざまなエラーが発生する可能性があります。dockerのログを確認すると、「ログのサンプルエントリ」のようなログが表示される可能性があります。

主なエラーメッセージ

• i/o timeout
• Bulk sync to node 464a64613c02 timed out

ログのサンプルエントリ

Node: ke2-rhel89-swarm-1

Sep 24 09:23:31 ke2-rhel89-swarm-1 dockerd[1491]: time="2024-09-24T09:54:53.779568607+09:00" level=error msg="Bulk sync to node 464a64613c02 timed out"
Sep 24 09:23:31 ke2-rhel89-swarm-1 dockerd[1491]: time="2024-09-24T09:23:31.507549348+09:00" level=warning msg="memberlist: Failed fallback TCP ping: timeout 1s: read tcp 10.20.47.11:32850->10.20.47.13:7946: i/o timeout"
Sep 24 09:23:31 ke2-rhel89-swarm-1 dockerd[1491]: time="2024-09-24T09:23:31.507592099+09:00" level=info msg="memberlist: Suspect 464a64613c02 has failed, no acks received"
Sep 24 09:23:38 ke2-rhel89-swarm-1 dockerd[1491]: time="2024-09-24T09:23:38.508187215+09:00" level=warning msg="memberlist: Failed fallback TCP ping: timeout 1s: read tcp 10.20.47.11:52826->10.20.47.13:7946: i/o timeout"
Sep 24 09:23:38 ke2-rhel89-swarm-1 dockerd[1491]: time="2024-09-24T09:23:38.508220989+09:00" level=info msg="memberlist: Suspect 464a64613c02 has failed, no acks received"
Sep 24 09:23:53 ke2-rhel89-swarm-1 dockerd[1491]: time="2024-09-24T09:23:53.507787431+09:00" level=warning msg="memberlist: Failed fallback TCP ping: timeout 1s: read tcp 10.20.47.11:40330->10.20.47.13:7946: i/o timeout"
Sep 24 09:23:53 ke2-rhel89-swarm-1 dockerd[1491]: time="2024-09-24T09:23:53.507818329+09:00" level=info msg="memberlist: Suspect 464a64613c02 has failed, no acks received"

解決策

ネットワーク導通を確認してください。

DIAG-CD6: Gluster ボリュームが正しくマウントされてない

FAQ-21: Gluster ボリュームが正しくマウントされてない

Gluster ボリュームが正しくマウントされてない場合、docker のログを確認すると、「ログのサンプルエントリ」のようなログが表示される可能性があります。

主なエラーメッセージ

• invalid mount config for type \"bind\": bind source path does not exist: /mnt/gluster/var"

ログのサンプルエントリ

Node: ke2-rhel89-swarm-1

Sep 25 13:23:45 ke2-rhel89-swarm-1 dockerd[1444]: time="2024-09-25T13:23:45.270116728+09:00" level=error msg="fatal task error" error="invalid mount config for type \"bind\": bind source path does not exist: /mnt/gluster/var" module=node/agent/taskmanager node.id=fki8ndj78y2khgyt7j6xn2duf service.id=sluwu58co2z03uda6c152n1zs task.id=7ju5umkdsfu4fmeftvx1fnnds
Sep 25 13:23:45 ke2-rhel89-swarm-1 dockerd[1444]: time="2024-09-25T13:23:45.270198833+09:00" level=error msg="fatal task error" error="invalid mount config for type \"bind\": bind source path does not exist: /mnt/gluster/var" module=node/agent/taskmanager node.id=fki8ndj78y2khgyt7j6xn2duf service.id=rm6u3rvx5ky7d338yzxweu3ld task.id=k42sbhn8l1jgsxk3tqbemluu6
Sep 25 13:23:45 ke2-rhel89-swarm-1 dockerd[1444]: time="2024-09-25T13:23:45.270116788+09:00" level=error msg="fatal task error" error="invalid mount config for type \"bind\": bind source path does not exist: /mnt/gluster/var" module=node/agent/taskmanager node.id=fki8ndj78y2khgyt7j6xn2duf service.id=sxiy4vyddfl0dj48ksnwkl3m3 task.id=08t2nbt92lty5q58mnpf03o61
Sep 25 13:23:45 ke2-rhel89-swarm-1 dockerd[1444]: time="2024-09-25T13:23:45.270134992+09:00" level=error msg="fatal task error" error="invalid mount config for type \"bind\": bind source path does not exist: /mnt/gluster/var" module=node/agent/taskmanager node.id=fki8ndj78y2khgyt7j6xn2duf service.id=ei4r6335y09ifqa32b1nx44pu task.id=qg2ybb4x35dvxsl5f6trd3r8q
Sep 25 13:23:45 ke2-rhel89-swarm-1 dockerd[1444]: time="2024-09-25T13:23:45.270342094+09:00" level=error msg="fatal task error" error="invalid mount config for type \"bind\": bind source path does not exist: /mnt/gluster/ssl" module=node/agent/taskmanager node.id=fki8ndj78y2khgyt7j6xn2duf service.id=g6e2w2pqhyq2czgk5dspmbzzg task.id=yl0eh4618ynufvqihtdm7yl0k
Sep 25 13:23:45 ke2-rhel89-swarm-1 dockerd[1444]: time="2024-09-25T13:23:45.382825963+09:00" level=warning msg="failed to deactivate service binding for container ke2_kompira.3.vmyc0f2y5pcafg7990hzdonkp" error="No such container: ke2_kompira.3.vmyc0f2y5pcafg7990hzdonkp" module=node/agent node.id=fki8ndj78y2khgyt7j6xn2duf
Sep 25 13:23:45 ke2-rhel89-swarm-1 dockerd[1444]: time="2024-09-25T13:23:45.382829870+09:00" level=warning msg="failed to deactivate service binding for container ke2_kengine.1.tgyt99r4jsnjucv6ap407l7l5" error="No such container: ke2_kengine.1.tgyt99r4jsnjucv6ap407l7l5" module=node/agent node.id=fki8ndj78y2khgyt7j6xn2duf
Sep 25 13:23:45 ke2-rhel89-swarm-1 dockerd[1444]: time="2024-09-25T13:23:45.382845971+09:00" level=warning msg="failed to deactivate service binding for container ke2_rabbitmq.3.vwaw3y3hfzvtzzgi1zvt4z3cr" error="No such container: ke2_rabbitmq.3.vwaw3y3hfzvtzzgi1zvt4z3cr" module=node/agent node.id=fki8ndj78y2khgyt7j6xn2duf
Sep 25 13:23:45 ke2-rhel89-swarm-1 dockerd[1444]: time="2024-09-25T13:23:45.382825853+09:00" level=warning msg="failed to deactivate service binding for container ke2_jobmngrd.2.m2r74rpixjnmbizghj3afry86" error="No such container: ke2_jobmngrd.2.m2r74rpixjnmbizghj3afry86" module=node/agent node.id=fki8ndj78y2khgyt7j6xn2duf
Sep 25 13:23:45 ke2-rhel89-swarm-1 dockerd[1444]: time="2024-09-25T13:23:45.382858785+09:00" level=warning msg="failed to deactivate service binding for container ke2_nginx.1.v59tx2b2tgvwg1hal4w7s58ar" error="No such container: ke2_nginx.1.v59tx2b2tgvwg1hal4w7s58ar" module=node/agent node.id=fki8ndj78y2khgyt7j6xn2duf

解決策

• 共有ファイルシステム (glusterfs)の正常性をご確認ください。

DIAG-DO7: docker ログに「error while reading from stream」というエラーが記録されている

特定ノードにリソース使用率が高い場合、さまざまなエラーが発生する可能性があります。docker のログを確認すると、「ログのサンプルエントリ」のようなログが表示される可能性があります。

ログのサンプルエントリ

Node: ke2-rhel89-swarm-1

Sep  2 08:47:39 ke2-rhel89-swarm-1 dockerd[752072]: time="2024-09-02T08:47:39.284653297+09:00" level=warning msg="memberlist: Refuting a suspect message (from: a6a2e4038ef8)"
Sep  2 08:47:39 ke2-rhel89-swarm-1 dockerd[752072]: time="2024-09-02T08:47:39.509297932+09:00" level=error msg="error while reading from stream" error="rpc error: code = Canceled desc = context canceled"
Sep  2 08:51:09 ke2-rhel89-swarm-1 dockerd[752072]: time="2024-09-02T08:51:09.699574772+09:00" level=error msg="error while reading from stream" error="rpc error: code = Canceled desc = context canceled"
Sep  2 08:51:09 ke2-rhel89-swarm-1 dockerd[752072]: time="2024-09-02T08:51:09.700112278+09:00" level=warning msg="memberlist: Refuting a dead message (from: 0d7709ba92eb)"

解決策

リソース使用率が正常化すると、Docker で実行中コンテナは通常、自動的に回復します。ただし、回復になってない場合は、ホストサーバのリソース状況を確認確認してください。