Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

It takes about 5mins to re-establish TCP streaming connection #19143

Open
4 tasks done
XiuhuaRuan opened this issue Jan 8, 2025 · 0 comments
Open
4 tasks done

It takes about 5mins to re-establish TCP streaming connection #19143

XiuhuaRuan opened this issue Jan 8, 2025 · 0 comments
Labels
type/bug

Comments

@XiuhuaRuan
Copy link

XiuhuaRuan commented Jan 8, 2025 •
edited by ahrtr
Loading

Bug report criteria

What happened?

I set up an etcd cluster with 3 nodes, 2 network cards per node and configured two URLs with two network segments for each node. When I ifdown one network card, TCP streaming connections was re-established after about 5mins on another network card.

What did you expect to happen?

TCP streaming connections could be re-established on another network segment within several seconds.

How can we reproduce it (as minimally and precisely as possible)?

  1. Set up an etcd cluster with 3 nodes, 2 network cards per node and configured two URLs with two network segments for each node.
    initial-advertise-peer-urls: http://192.168.31.181:2380,http://192.168.199.181:2380 advertise-client-urls: http://192.168.31.181:2379,http://192.168.199.181:2379 initial-cluster: etcd_01=http://192.168.31.181:2380,etcd_01=http://192.168.199.181:2380,etcd_02=http://192.168.31.182:2380,etcd_02=http://192.168.199.182:2380,etcd_03=http://192.168.31.183:2380,etcd_03=http://192.168.199.183:2380
  2. There are actually 6 endpoints when show cluster endpoint status.
 ./etcdctl endpoint status --cluster -w table
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://192.168.199.182:2379 | 5d73ea50db7c3509 |   3.5.9 |   53 kB |     false |      false |         7 |        104 |                104 |        |
|  http://192.168.31.182:2379 | 5d73ea50db7c3509 |   3.5.9 |   53 kB |     false |      false |         7 |        104 |                104 |        |
| http://192.168.199.181:2379 | 6e6dd596fdb455e0 |   3.5.9 |   53 kB |      true |      false |         7 |        104 |                104 |        |
|  http://192.168.31.181:2379 | 6e6dd596fdb455e0 |   3.5.9 |   53 kB |      true |      false |         7 |        104 |                104 |        |
| http://192.168.199.183:2379 | a53ee61596a804e6 |   3.5.9 |   53 kB |     false |      false |         7 |        104 |                104 |        |
|  http://192.168.31.183:2379 | a53ee61596a804e6 |   3.5.9 |   53 kB |     false |      false |         7 |        104 |                104 |        |
  1. Execute 'ifdown enp0s8' with network segment 192.168.199.x, check cluster status. Cluster service becomes available after about 5mins.
./etcdctl endpoint status --cluster -w table
{"level":"warn","ts":"2025-01-08T15:07:29.615317+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000310c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Error: failed to fetch endpoints from etcd cluster member list: context deadline exceeded
[root@sophia-pghost2 etcd]# ./etcdctl endpoint status --cluster -w table
{"level":"warn","ts":"2025-01-08T15:09:46.170956+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000334a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Failed to get the status of endpoint http://192.168.199.182:2379 (context deadline exceeded)
{"level":"warn","ts":"2025-01-08T15:09:51.177806+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000334a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Failed to get the status of endpoint http://192.168.199.181:2379 (context deadline exceeded)
{"level":"warn","ts":"2025-01-08T15:09:56.182872+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000334a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Failed to get the status of endpoint http://192.168.199.183:2379 (context deadline exceeded)
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://192.168.31.182:2379 | 5d73ea50db7c3509 |   3.5.9 |   53 kB |      true |      false |         8 |        111 |                111 |        |
| http://192.168.31.181:2379 | 6e6dd596fdb455e0 |   3.5.9 |   53 kB |     false |      false |         8 |        111 |                111 |        |
| http://192.168.31.183:2379 | a53ee61596a804e6 |   3.5.9 |   53 kB |     false |      false |         8 |        111 |                111 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Anything else we need to know?

I checked etcd log and found that TCP streaming connection was re-established about 5mins after ifdown one network card. If TCP streaming connection could be re-established quicked, raft could elect new leader and recover cluster service quickly. Please help check could "lost TCP streaming connection with remote peer" be detected quickly in such scenario.

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.5.9
Git SHA: bdbbde998
Go Version: go1.19.9
Go OS/Arch: linux/amd64


$ etcdctl version
etcdctl version: 3.5.9
API version: 3.5

Etcd configuration (command line flags or environment variables)

name: 'etcd_01' listen-peer-urls: http://0.0.0.0:2380 listen-client-urls: http://0.0.0.0:2379 initial-advertise-peer-urls: http://192.168.31.181:2380,http://192.168.199.181:2380 advertise-client-urls: http://192.168.31.181:2379,http://192.168.199.181:2379 initial-cluster: etcd_01=http://192.168.31.181:2380,etcd_01=http://192.168.199.181:2380,etcd_02=http://192.168.31.182:2380,etcd_02=http://192.168.199.182:2380,etcd_03=http://192.168.31.183:2380,etcd_03=http://192.168.199.183:2380 initial-cluster-token: 'etcd-cluster' initial-cluster-state: 'new' strict-reconfig-check: false enable-v2: true proxy: 'off' force-new-cluster: false

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
+------------------+---------+---------+--------------------------------------------------------+--------------------------------------------------------+------------+
|        ID        | STATUS  |  NAME   |                       PEER ADDRS                       |                      CLIENT ADDRS                      | IS LEARNER |
+------------------+---------+---------+--------------------------------------------------------+--------------------------------------------------------+------------+
| 5d73ea50db7c3509 | started | etcd_02 | http://192.168.199.182:2380,http://192.168.31.182:2380 | http://192.168.199.182:2379,http://192.168.31.182:2379 |      false |
| 6e6dd596fdb455e0 | started | etcd_01 | http://192.168.199.181:2380,http://192.168.31.181:2380 | http://192.168.199.181:2379,http://192.168.31.181:2379 |      false |
| a53ee61596a804e6 | started | etcd_03 | http://192.168.199.183:2380,http://192.168.31.183:2380 | http://192.168.199.183:2379,http://192.168.31.183:2379 |      false |
+------------------+---------+---------+--------------------------------------------------------+--------------------------------------------------------+------------+



$ etcdctl --endpoints=<member list> endpoint status -w table
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+-------------
|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+-------------
| http://192.168.199.182:2379 | 5d73ea50db7c3509 |   3.5.9 |   53 kB |      true |      false |         8 |        111 |
|  http://192.168.31.182:2379 | 5d73ea50db7c3509 |   3.5.9 |   53 kB |      true |      false |         8 |        111 |
| http://192.168.199.181:2379 | 6e6dd596fdb455e0 |   3.5.9 |   53 kB |     false |      false |         8 |        111 |
|  http://192.168.31.181:2379 | 6e6dd596fdb455e0 |   3.5.9 |   53 kB |     false |      false |         8 |        111 |
| http://192.168.199.183:2379 | a53ee61596a804e6 |   3.5.9 |   53 kB |     false |      false |         8 |        111 |
|  http://192.168.31.183:2379 | a53ee61596a804e6 |   3.5.9 |   53 kB |     false |      false |         8 |        111 |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+-------------

Relevant log output

{"level":"warn","ts":"2025-01-08T15:04:22.75727+0800","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"a53ee61596a804e6","rtt":"953.387µs","error":"dial tcp 192.168.199.183:2380: i/o timeout"}
{"level":"info","ts":"2025-01-08T15:04:38.960332+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104, vote: 6e6dd596fdb455e0] ignored MsgPreVote from 5d73ea50db7c3509 [logterm: 7, index: 104] at term 7: lease is not expired (remaining ticks: 1)"}
{"level":"warn","ts":"2025-01-08T15:04:39.085046+0800","logger":"raft","caller":"etcdserver/zap_raft.go:85","msg":"6e6dd596fdb455e0 stepped down to follower since quorum is not active"}
{"level":"info","ts":"2025-01-08T15:04:39.085088+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 became follower at term 7"}
{"level":"info","ts":"2025-01-08T15:04:39.085095+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 6e6dd596fdb455e0 lost leader 6e6dd596fdb455e0 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:41.486208+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 is starting a new election at term 7"}
{"level":"info","ts":"2025-01-08T15:04:41.486275+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 became pre-candidate at term 7"}
{"level":"info","ts":"2025-01-08T15:04:41.486285+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 received MsgPreVoteResp from 6e6dd596fdb455e0 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:41.486297+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to 5d73ea50db7c3509 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:41.486337+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to a53ee61596a804e6 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:41.761164+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104, vote: 6e6dd596fdb455e0] cast MsgPreVote for 5d73ea50db7c3509 [logterm: 7, index: 104] at term 7"}
{"level":"info","ts":"2025-01-08T15:04:43.885568+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 is starting a new election at term 7"}
{"level":"info","ts":"2025-01-08T15:04:43.88561+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 became pre-candidate at term 7"}
{"level":"info","ts":"2025-01-08T15:04:43.885624+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 received MsgPreVoteResp from 6e6dd596fdb455e0 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:43.885634+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to 5d73ea50db7c3509 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:43.885639+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to a53ee61596a804e6 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:44.576215+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104, vote: 6e6dd596fdb455e0] cast MsgPreVote for 5d73ea50db7c3509 [logterm: 7, index: 104] at term 7"}
{"level":"info","ts":"2025-01-08T15:04:46.285325+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 is starting a new election at term 7"}
{"level":"info","ts":"2025-01-08T15:04:46.285364+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 became pre-candidate at term 7"}
{"level":"info","ts":"2025-01-08T15:04:46.285372+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 received MsgPreVoteResp from 6e6dd596fdb455e0 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:46.285382+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to 5d73ea50db7c3509 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:46.285388+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to a53ee61596a804e6 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:47.359919+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104, vote: 6e6dd596fdb455e0] cast MsgPreVote for 5d73ea50db7c3509 [logterm: 7, index: 104] at term 7"}
{"level":"info","ts":"2025-01-08T15:04:48.686678+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 is starting a new election at term 7"}
{"level":"info","ts":"2025-01-08T15:04:48.686723+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 became pre-candidate at term 7"}
{"level":"info","ts":"2025-01-08T15:04:48.686732+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 received MsgPreVoteResp from 6e6dd596fdb455e0 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:48.686761+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to 5d73ea50db7c3509 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:48.686767+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to a53ee61596a804e6 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:50.160106+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104, vote: 6e6dd596fdb455e0] cast MsgPreVote for 5d73ea50db7c3509 [logterm: 7, index: 104] at term 7"}
{"level":"info","ts":"2025-01-08T15:04:51.098935+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 is starting a new election at term 7"}
{"level":"info","ts":"2025-01-08T15:04:51.098978+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 became pre-candidate at term 7"}
{"level":"info","ts":"2025-01-08T15:04:51.098987+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 received MsgPreVoteResp from 6e6dd596fdb455e0 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:51.099007+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to 5d73ea50db7c3509 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:51.099016+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to a53ee61596a804e6 at term 7"}
{"level":"info","ts":"2025-01-08T15:04:52.977122+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104, vote: 6e6dd596fdb455e0] cast MsgPreVote for 5d73ea50db7c3509 [logterm: 7, index: 104] at term 7"}
{"level":"info","ts":"2025-01-08T15:04:53.485349+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 is starting a new election at term 7"}
{"level":"info","ts":"2025-01-08T15:04:53.485421+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 became pre-candidate at term 7"}
...................................
{"level":"info","ts":"2025-01-08T15:09:07.886032+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to a53ee61596a804e6 at term 7"}
{"level":"info","ts":"2025-01-08T15:09:10.285049+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 is starting a new election at term 7"}
{"level":"info","ts":"2025-01-08T15:09:10.285092+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 became pre-candidate at term 7"}
{"level":"info","ts":"2025-01-08T15:09:10.2851+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 received MsgPreVoteResp from 6e6dd596fdb455e0 at term 7"}
{"level":"info","ts":"2025-01-08T15:09:10.285116+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to 5d73ea50db7c3509 at term 7"}
{"level":"info","ts":"2025-01-08T15:09:10.285121+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to a53ee61596a804e6 at term 7"}
{"level":"info","ts":"2025-01-08T15:09:10.560095+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104, vote: 6e6dd596fdb455e0] cast MsgPreVote for 5d73ea50db7c3509 [logterm: 7, index: 104] at term 7"}
{"level":"info","ts":"2025-01-08T15:09:12.686338+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 is starting a new election at term 7"}
{"level":"info","ts":"2025-01-08T15:09:12.686386+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 became pre-candidate at term 7"}
{"level":"info","ts":"2025-01-08T15:09:12.686399+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 received MsgPreVoteResp from 6e6dd596fdb455e0 at term 7"}
{"level":"info","ts":"2025-01-08T15:09:12.686409+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to 5d73ea50db7c3509 at term 7"}
{"level":"info","ts":"2025-01-08T15:09:12.686414+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to a53ee61596a804e6 at term 7"}
{"level":"info","ts":"2025-01-08T15:09:13.361661+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104, vote: 6e6dd596fdb455e0] cast MsgPreVote for 5d73ea50db7c3509 [logterm: 7, index: 104] at term 7"}
{"level":"info","ts":"2025-01-08T15:09:15.085558+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 is starting a new election at term 7"}
{"level":"info","ts":"2025-01-08T15:09:15.0856+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 became pre-candidate at term 7"}
{"level":"info","ts":"2025-01-08T15:09:15.085608+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 received MsgPreVoteResp from 6e6dd596fdb455e0 at term 7"}
{"level":"info","ts":"2025-01-08T15:09:15.08562+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to 5d73ea50db7c3509 at term 7"}
{"level":"info","ts":"2025-01-08T15:09:15.085624+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to a53ee61596a804e6 at term 7"}
{"level":"info","ts":"2025-01-08T15:09:16.162403+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104, vote: 6e6dd596fdb455e0] cast MsgPreVote for 5d73ea50db7c3509 [logterm: 7, index: 104] at term 7"}
{"level":"warn","ts":"2025-01-08T15:09:17.126961+0800","caller":"rafthttp/stream.go:421","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"6e6dd596fdb455e0","remote-peer-id":"a53ee61596a804e6","error":"read tcp 192.168.199.184:51808->192.168.199.183:2380: read: connection timed out"}
{"level":"warn","ts":"2025-01-08T15:09:17.127028+0800","caller":"rafthttp/peer_status.go:66","msg":"peer became inactive (message send to peer failed)","peer-id":"a53ee61596a804e6","error":"failed to read a53ee61596a804e6 on stream MsgApp v2 (read tcp 192.168.199.184:51808->192.168.199.183:2380: read: connection timed out)"}
{"level":"info","ts":"2025-01-08T15:09:17.229004+0800","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"a53ee61596a804e6"}
{"level":"info","ts":"2025-01-08T15:09:17.229062+0800","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"6e6dd596fdb455e0","remote-peer-id":"a53ee61596a804e6"}
{"level":"info","ts":"2025-01-08T15:09:17.550616+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 is starting a new election at term 7"}
{"level":"info","ts":"2025-01-08T15:09:17.550665+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 became pre-candidate at term 7"}
{"level":"info","ts":"2025-01-08T15:09:17.550675+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 received MsgPreVoteResp from 6e6dd596fdb455e0 at term 7"}
{"level":"info","ts":"2025-01-08T15:09:17.550687+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to 5d73ea50db7c3509 at term 7"}
{"level":"info","ts":"2025-01-08T15:09:17.550694+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to a53ee61596a804e6 at term 7"}
{"level":"warn","ts":"2025-01-08T15:09:18.275652+0800","caller":"rafthttp/stream.go:421","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"6e6dd596fdb455e0","remote-peer-id":"a53ee61596a804e6","error":"read tcp 192.168.199.184:51806->192.168.199.183:2380: read: connection timed out"}
{"level":"warn","ts":"2025-01-08T15:09:18.275698+0800","caller":"rafthttp/peer_status.go:66","msg":"peer became inactive (message send to peer failed)","peer-id":"a53ee61596a804e6","error":"failed to read a53ee61596a804e6 on stream Message (read tcp 192.168.199.184:51806->192.168.199.183:2380: read: connection timed out)"}
{"level":"info","ts":"2025-01-08T15:09:18.276614+0800","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"a53ee61596a804e6"}
{"level":"info","ts":"2025-01-08T15:09:18.276707+0800","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"6e6dd596fdb455e0","remote-peer-id":"a53ee61596a804e6"}
{"level":"info","ts":"2025-01-08T15:09:18.961885+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104, vote: 6e6dd596fdb455e0] cast MsgPreVote for 5d73ea50db7c3509 [logterm: 7, index: 104] at term 7"}
{"level":"info","ts":"2025-01-08T15:09:19.885805+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 is starting a new election at term 7"}
{"level":"info","ts":"2025-01-08T15:09:19.885845+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 became pre-candidate at term 7"}
{"level":"info","ts":"2025-01-08T15:09:19.885853+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 received MsgPreVoteResp from 6e6dd596fdb455e0 at term 7"}
{"level":"info","ts":"2025-01-08T15:09:19.885863+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to 5d73ea50db7c3509 at term 7"}
{"level":"info","ts":"2025-01-08T15:09:19.885868+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104] sent MsgPreVote request to a53ee61596a804e6 at term 7"}
{"level":"info","ts":"2025-01-08T15:09:19.949416+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104, vote: 6e6dd596fdb455e0] cast MsgPreVote for a53ee61596a804e6 [logterm: 7, index: 104] at term 7"}
{"level":"info","ts":"2025-01-08T15:09:21.235698+0800","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"6e6dd596fdb455e0","to":"a53ee61596a804e6","stream-type":"stream Message"}
{"level":"warn","ts":"2025-01-08T15:09:21.235729+0800","caller":"rafthttp/stream.go:265","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"6e6dd596fdb455e0","remote-peer-id":"a53ee61596a804e6"}
{"level":"info","ts":"2025-01-08T15:09:21.235739+0800","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"6e6dd596fdb455e0","remote-peer-id":"a53ee61596a804e6"}
{"level":"info","ts":"2025-01-08T15:09:21.23577+0800","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"6e6dd596fdb455e0","to":"a53ee61596a804e6","stream-type":"stream MsgApp v2"}
{"level":"warn","ts":"2025-01-08T15:09:21.235786+0800","caller":"rafthttp/stream.go:265","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"6e6dd596fdb455e0","remote-peer-id":"a53ee61596a804e6"}
{"level":"info","ts":"2025-01-08T15:09:21.235792+0800","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"6e6dd596fdb455e0","remote-peer-id":"a53ee61596a804e6"}
{"level":"info","ts":"2025-01-08T15:09:21.768687+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104, vote: 6e6dd596fdb455e0] cast MsgPreVote for 5d73ea50db7c3509 [logterm: 7, index: 104] at term 7"}
{"level":"info","ts":"2025-01-08T15:09:21.770346+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [term: 7] received a MsgVote message with higher term from 5d73ea50db7c3509 [term: 8]"}
{"level":"info","ts":"2025-01-08T15:09:21.770373+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 became follower at term 8"}
{"level":"info","ts":"2025-01-08T15:09:21.770379+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6e6dd596fdb455e0 [logterm: 7, index: 104, vote: 0] cast MsgVote for 5d73ea50db7c3509 [logterm: 7, index: 104] at term 8"}
{"level":"info","ts":"2025-01-08T15:09:21.771451+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 6e6dd596fdb455e0 elected leader 5d73ea50db7c3509 at term 8"}
{"level":"warn","ts":"2025-01-08T15:09:26.307192+0800","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-01-08T15:09:24.638654+0800","time spent":"1.668455659s","remote":"127.0.0.1:58012","response type":"/etcdserverpb.KV/Put","request count":1,"request size":279,"response count":0,"response size":0,"request content":"key:\"/service/highgo-see-cluster/members/see_03\" value_size:222 lease:3821748775777705746 "}
{"level":"warn","ts":"2025-01-08T15:09:26.307599+0800","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-01-08T15:09:24.638123+0800","time spent":"1.669446785s","remote":"127.0.0.1:58012","response type":"/etcdserverpb.KV/Put","request count":1,"request size":57,"response count":0,"response size":0,"request content":"key:\"/service/highgo-see-cluster/status\" value_size:19 "}
{"level":"warn","ts":"2025-01-08T15:09:26.30802+0800","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-01-08T15:09:24.638628+0800","time spent":"1.669382034s","remote":"127.0.0.1:58012","response type":"/etcdserverpb.KV/Put","request count":1,"request size":279,"response count":0,"response size":0,"request content":"key:\"/service/highgo-see-cluster/members/see_02\" value_size:222 lease:3821748775777705742 "}
{"level":"warn","ts":"2025-01-08T15:09:34.469095+0800","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-01-08T15:09:32.787407+0800","time spent":"1.681640984s","remote":"127.0.0.1:58012","response type":"/etcdserverpb.KV/Put","request count":1,"request size":279,"response count":0,"response size":0,"request content":"key:\"/service/highgo-see-cluster/members/see_03\" value_size:222 lease:3821748775777705746 "}
{"level":"info","ts":"2025-01-08T15:09:39.64315+0800","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"6e6dd596fdb455e0","to":"5d73ea50db7c3509","stream-type":"stream MsgApp v2"}
{"level":"warn","ts":"2025-01-08T15:09:39.64318+0800","caller":"rafthttp/stream.go:265","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"6e6dd596fdb455e0","remote-peer-id":"5d73ea50db7c3509"}
{"level":"info","ts":"2025-01-08T15:09:39.643195+0800","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"6e6dd596fdb455e0","remote-peer-id":"5d73ea50db7c3509"}
{"level":"info","ts":"2025-01-08T15:09:40.055163+0800","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"6e6dd596fdb455e0","to":"5d73ea50db7c3509","stream-type":"stream Message"}
{"level":"warn","ts":"2025-01-08T15:09:40.055209+0800","caller":"rafthttp/stream.go:265","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"6e6dd596fdb455e0","remote-peer-id":"5d73ea50db7c3509"}
{"level":"info","ts":"2025-01-08T15:09:40.055216+0800","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"6e6dd596fdb455e0","remote-peer-id":"5d73ea50db7c3509"}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug
Milestone
No milestone
Development

No branches or pull requests
1 participant
@XiuhuaRuan