[pkg-go] Bug#976924: golang-github-hashicorp-raft: FTBFS on ppc64el (arch:all-only src pkg): dh_auto_test: error: cd obj-powerpc64le-linux-gnu && go test -vet=off -v -p 160 github.com/hashicorp/raft github.com/hashicorp/raft/bench returned exit code 1
Lucas Nussbaum
lucas at debian.org
Wed Dec 9 09:03:52 GMT 2020
Source: golang-github-hashicorp-raft
Version: 1.1.2-1
Severity: serious
Justification: FTBFS on ppc64el
Tags: bullseye sid ftbfs
Usertags: ftbfs-20201209 ftbfs-bullseye ftbfs-ppc64el
Hi,
During a rebuild of all packages in sid, your package failed to build
on ppc64el. At the same time, it did not fail on amd64.
I'm marking this bug as severity:serious since your package has only
Architecture:all binary packages, and should thus, in theory, build
everywhere. Failure to build on ppc64el might indicate a serious issue
in this package or in another package.
But feel free to downgrade or close if you believe that this is only a
build-time issue. (I would personnally prefer a severity:minor bug just to
track that the package can only be built on specific architectures.)
Relevant part (hopefully):
> debian/rules build
> dh build --buildsystem=golang --with=golang
> dh_update_autotools_config -O--buildsystem=golang
> dh_autoreconf -O--buildsystem=golang
> dh_auto_configure -O--buildsystem=golang
> dh_auto_build -O--buildsystem=golang
> cd obj-powerpc64le-linux-gnu && go install -trimpath -v -p 160 github.com/hashicorp/raft github.com/hashicorp/raft/bench
> vendor/golang.org/x/crypto/cryptobyte/asn1
> encoding
> crypto/subtle
> container/list
> internal/nettrace
> golang.org/x/sys/internal/unsafeheader
> unicode/utf16
> math/bits
> internal/unsafeheader
> unicode/utf8
> vendor/golang.org/x/crypto/internal/subtle
> internal/cpu
> internal/race
> sync/atomic
> unicode
> crypto/internal/subtle
> runtime/internal/sys
> runtime/internal/atomic
> runtime/cgo
> math
> internal/testlog
> runtime/internal/math
> internal/bytealg
> runtime
> internal/reflectlite
> sync
> internal/singleflight
> math/rand
> errors
> sort
> internal/oserror
> github.com/hashicorp/golang-lru/simplelru
> io
> vendor/golang.org/x/net/dns/dnsmessage
> strconv
> syscall
> hash
> crypto/internal/randutil
> bytes
> strings
> crypto/hmac
> hash/crc64
> hash/crc32
> vendor/golang.org/x/crypto/hkdf
> crypto/rc4
> crypto
> reflect
> vendor/golang.org/x/text/transform
> path
> bufio
> github.com/hashicorp/go-immutable-radix
> regexp/syntax
> html
> internal/syscall/execenv
> internal/syscall/unix
> time
> regexp
> context
> internal/poll
> encoding/binary
> internal/fmtsort
> os
> crypto/sha256
> crypto/sha512
> encoding/base64
> crypto/md5
> crypto/sha1
> crypto/cipher
> vendor/golang.org/x/crypto/poly1305
> crypto/ed25519/internal/edwards25519
> golang.org/x/sys/unix
> encoding/pem
> crypto/des
> vendor/golang.org/x/crypto/chacha20
> crypto/aes
> os/signal
> runtime/debug
> path/filepath
> fmt
> net
> vendor/golang.org/x/crypto/chacha20poly1305
> io/ioutil
> encoding/hex
> net/url
> log
> vendor/golang.org/x/crypto/curve25519
> compress/flate
> encoding/json
> vendor/golang.org/x/net/http2/hpack
> mime
> mime/quotedprintable
> net/http/internal
> vendor/golang.org/x/text/unicode/norm
> runtime/trace
> go/token
> flag
> text/template/parse
> math/big
> encoding/gob
> vendor/golang.org/x/text/unicode/bidi
> testing
> github.com/mattn/go-isatty
> compress/gzip
> github.com/mattn/go-colorable
> vendor/golang.org/x/text/secure/bidirule
> github.com/fatih/color
> text/template
> vendor/golang.org/x/net/idna
> github.com/hashicorp/go-hclog
> crypto/dsa
> encoding/asn1
> crypto/rand
> crypto/elliptic
> crypto/ed25519
> crypto/rsa
> html/template
> vendor/golang.org/x/crypto/cryptobyte
> crypto/x509/pkix
> crypto/ecdsa
> vendor/golang.org/x/net/http/httpproxy
> net/textproto
> crypto/x509
> vendor/golang.org/x/net/http/httpguts
> mime/multipart
> crypto/tls
> net/http/httptrace
> net/http
> github.com/armon/go-metrics
> net/rpc
> github.com/hashicorp/go-msgpack/codec
> github.com/hashicorp/raft
> github.com/hashicorp/raft/bench
> dh_auto_test -O--buildsystem=golang
> cd obj-powerpc64le-linux-gnu && go test -vet=off -v -p 160 github.com/hashicorp/raft github.com/hashicorp/raft/bench
> === RUN TestCommitment_setVoters
> --- PASS: TestCommitment_setVoters (0.00s)
> === RUN TestCommitment_match_max
> --- PASS: TestCommitment_match_max (0.00s)
> === RUN TestCommitment_match_nonVoting
> --- PASS: TestCommitment_match_nonVoting (0.00s)
> === RUN TestCommitment_recalculate
> --- PASS: TestCommitment_recalculate (0.00s)
> === RUN TestCommitment_recalculate_startIndex
> --- PASS: TestCommitment_recalculate_startIndex (0.00s)
> === RUN TestCommitment_noVoterSanity
> --- PASS: TestCommitment_noVoterSanity (0.00s)
> === RUN TestCommitment_singleVoter
> --- PASS: TestCommitment_singleVoter (0.00s)
> === RUN TestConfiguration_Configuration_Clone
> --- PASS: TestConfiguration_Configuration_Clone (0.00s)
> === RUN TestConfiguration_configurations_Clone
> --- PASS: TestConfiguration_configurations_Clone (0.00s)
> === RUN TestConfiguration_hasVote
> --- PASS: TestConfiguration_hasVote (0.00s)
> === RUN TestConfiguration_checkConfiguration
> --- PASS: TestConfiguration_checkConfiguration (0.00s)
> === RUN TestConfiguration_nextConfiguration_table
> --- PASS: TestConfiguration_nextConfiguration_table (0.00s)
> === RUN TestConfiguration_nextConfiguration_prevIndex
> --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
> === RUN TestConfiguration_nextConfiguration_checkConfiguration
> --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
> === RUN TestConfiguration_encodeDecodePeers
> --- PASS: TestConfiguration_encodeDecodePeers (0.00s)
> === RUN TestConfiguration_encodeDecodeConfiguration
> --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
> === RUN TestDiscardSnapshotStoreImpl
> --- PASS: TestDiscardSnapshotStoreImpl (0.00s)
> === RUN TestDiscardSnapshotSinkImpl
> --- PASS: TestDiscardSnapshotSinkImpl (0.00s)
> === RUN TestFileSnapshotStoreImpl
> --- PASS: TestFileSnapshotStoreImpl (0.00s)
> === RUN TestFileSnapshotSinkImpl
> --- PASS: TestFileSnapshotSinkImpl (0.00s)
> === RUN TestFileSS_CreateSnapshotMissingParentDir
> testing.go:153: 2020-12-08T22:13:08.445Z [INFO] creating new snapshot: path=/tmp/raft395049780/raft705539843/snapshots/3-10-1607465588445.tmp
> --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.10s)
> === RUN TestFileSS_CreateSnapshot
> testing.go:153: 2020-12-08T22:13:08.542Z [INFO] creating new snapshot: path=/tmp/raft928426630/snapshots/3-10-1607465588542.tmp
> testing.go:153: 2020-12-08T22:13:08.582Z [WARN] found temporary snapshot: name=3-10-1607465588542.tmp
> --- PASS: TestFileSS_CreateSnapshot (0.12s)
> === RUN TestFileSS_CancelSnapshot
> testing.go:153: 2020-12-08T22:13:08.667Z [INFO] creating new snapshot: path=/tmp/raft196096557/snapshots/3-10-1607465588667.tmp
> --- PASS: TestFileSS_CancelSnapshot (0.05s)
> === RUN TestFileSS_Retention
> testing.go:153: 2020-12-08T22:13:08.717Z [INFO] creating new snapshot: path=/tmp/raft379884456/snapshots/3-10-1607465588717.tmp
> testing.go:153: 2020-12-08T22:13:08.836Z [INFO] creating new snapshot: path=/tmp/raft379884456/snapshots/3-11-1607465588836.tmp
> testing.go:153: 2020-12-08T22:13:08.937Z [INFO] creating new snapshot: path=/tmp/raft379884456/snapshots/3-12-1607465588937.tmp
> testing.go:153: 2020-12-08T22:13:09.037Z [INFO] reaping snapshot: path=/tmp/raft379884456/snapshots/3-10-1607465588717
> testing.go:153: 2020-12-08T22:13:09.038Z [INFO] creating new snapshot: path=/tmp/raft379884456/snapshots/3-13-1607465589038.tmp
> testing.go:153: 2020-12-08T22:13:09.138Z [INFO] reaping snapshot: path=/tmp/raft379884456/snapshots/3-11-1607465588836
> testing.go:153: 2020-12-08T22:13:09.139Z [INFO] creating new snapshot: path=/tmp/raft379884456/snapshots/3-14-1607465589139.tmp
> testing.go:153: 2020-12-08T22:13:09.239Z [INFO] reaping snapshot: path=/tmp/raft379884456/snapshots/3-12-1607465588937
> --- PASS: TestFileSS_Retention (0.52s)
> === RUN TestFileSS_BadPerm
> --- PASS: TestFileSS_BadPerm (0.00s)
> === RUN TestFileSS_MissingParentDir
> --- PASS: TestFileSS_MissingParentDir (0.00s)
> === RUN TestFileSS_Ordering
> testing.go:153: 2020-12-08T22:13:09.242Z [INFO] creating new snapshot: path=/tmp/raft983377163/snapshots/5-130350-1607465589242.tmp
> testing.go:153: 2020-12-08T22:13:09.341Z [INFO] creating new snapshot: path=/tmp/raft983377163/snapshots/36-204917-1607465589341.tmp
> --- PASS: TestFileSS_Ordering (0.20s)
> === RUN TestDeferFutureSuccess
> --- PASS: TestDeferFutureSuccess (0.00s)
> === RUN TestDeferFutureError
> --- PASS: TestDeferFutureError (0.00s)
> === RUN TestDeferFutureConcurrent
> --- PASS: TestDeferFutureConcurrent (0.00s)
> === RUN TestInmemSnapshotStoreImpl
> --- PASS: TestInmemSnapshotStoreImpl (0.00s)
> === RUN TestInmemSnapshotSinkImpl
> --- PASS: TestInmemSnapshotSinkImpl (0.00s)
> === RUN TestInmemSS_CreateSnapshot
> --- PASS: TestInmemSS_CreateSnapshot (0.00s)
> === RUN TestInmemSS_OpenSnapshotTwice
> --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
> === RUN TestInmemTransportImpl
> --- PASS: TestInmemTransportImpl (0.00s)
> === RUN TestInmemTransportWriteTimeout
> --- PASS: TestInmemTransportWriteTimeout (0.01s)
> === RUN TestRaft_Integ
> --- SKIP: TestRaft_Integ (0.00s)
> === RUN TestLogCache
> --- PASS: TestLogCache (0.00s)
> === RUN TestNetworkTransport_CloseStreams
> --- PASS: TestNetworkTransport_CloseStreams (0.00s)
> === RUN TestNetworkTransport_StartStop
> --- PASS: TestNetworkTransport_StartStop (0.00s)
> === RUN TestNetworkTransport_Heartbeat_FastPath
> --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
> === RUN TestNetworkTransport_AppendEntries
> --- PASS: TestNetworkTransport_AppendEntries (0.00s)
> === RUN TestNetworkTransport_AppendEntriesPipeline
> --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s)
> === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams
> net_transport_test.go:339: Disabled unreliable test
> --- SKIP: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.00s)
> === RUN TestNetworkTransport_RequestVote
> --- PASS: TestNetworkTransport_RequestVote (0.00s)
> === RUN TestNetworkTransport_InstallSnapshot
> --- PASS: TestNetworkTransport_InstallSnapshot (0.00s)
> === RUN TestNetworkTransport_EncodeDecode
> --- PASS: TestNetworkTransport_EncodeDecode (0.00s)
> === RUN TestNetworkTransport_EncodeDecode_AddressProvider
> --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
> === RUN TestNetworkTransport_PooledConn
> --- PASS: TestNetworkTransport_PooledConn (0.00s)
> === RUN TestNetworkTransport_ListenBackoff
> net_transport_test.go:718: countingWriter: 2020-12-08T22:13:09.463Z [ERROR] test: failed to accept connection: error="intentional error in test"
>
> net_transport_test.go:718: countingWriter: 2020-12-08T22:13:09.468Z [ERROR] test: failed to accept connection: error="intentional error in test"
>
> net_transport_test.go:718: countingWriter: 2020-12-08T22:13:09.478Z [ERROR] test: failed to accept connection: error="intentional error in test"
>
> net_transport_test.go:718: countingWriter: 2020-12-08T22:13:09.498Z [ERROR] test: failed to accept connection: error="intentional error in test"
>
> net_transport_test.go:718: countingWriter: 2020-12-08T22:13:09.538Z [ERROR] test: failed to accept connection: error="intentional error in test"
>
> net_transport_test.go:718: countingWriter: 2020-12-08T22:13:09.618Z [ERROR] test: failed to accept connection: error="intentional error in test"
>
> net_transport_test.go:718: countingWriter: 2020-12-08T22:13:09.779Z [ERROR] test: failed to accept connection: error="intentional error in test"
>
> net_transport_test.go:718: countingWriter: 2020-12-08T22:13:10.099Z [ERROR] test: failed to accept connection: error="intentional error in test"
>
> net_transport_test.go:718: countingWriter: 2020-12-08T22:13:10.739Z [ERROR] test: failed to accept connection: error="intentional error in test"
>
> net_transport_test.go:718: countingWriter: 2020-12-08T22:13:11.739Z [ERROR] test: failed to accept connection: error="intentional error in test"
>
> net_transport_test.go:718: countingWriter: 2020-12-08T22:13:12.739Z [ERROR] test: failed to accept connection: error="intentional error in test"
>
> --- PASS: TestNetworkTransport_ListenBackoff (4.00s)
> === RUN TestPeersJSON_BadConfiguration
> --- PASS: TestPeersJSON_BadConfiguration (0.00s)
> === RUN TestPeersJSON_ReadPeersJSON
> --- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
> === RUN TestPeersJSON_ReadConfigJSON
> --- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
> === RUN TestRaft_StartStop
> testLoggerAdapter verbose: server-61ebd578-3eb3-8ea0-372e-e2ad5a0ac0e1: 2020-12-08T22:13:13.465Z [INFO] server-61ebd578-3eb3-8ea0-372e-e2ad5a0ac0e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61ebd578-3eb3-8ea0-372e-e2ad5a0ac0e1 Address:61ebd578-3eb3-8ea0-372e-e2ad5a0ac0e1}]"
> testing.go:149: server-61ebd578-3eb3-8ea0-372e-e2ad5a0ac0e1: 2020-12-08T22:13:13.465Z [INFO] server-61ebd578-3eb3-8ea0-372e-e2ad5a0ac0e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61ebd578-3eb3-8ea0-372e-e2ad5a0ac0e1 Address:61ebd578-3eb3-8ea0-372e-e2ad5a0ac0e1}]"
> --- PASS: TestRaft_StartStop (0.00s)
> === RUN TestRaft_AfterShutdown
> testLoggerAdapter verbose: server-c9357880-e9da-acf1-841d-d89b86232abd: 2020-12-08T22:13:13.467Z [INFO] server-c9357880-e9da-acf1-841d-d89b86232abd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9357880-e9da-acf1-841d-d89b86232abd Address:c9357880-e9da-acf1-841d-d89b86232abd}]"
> testing.go:149: server-c9357880-e9da-acf1-841d-d89b86232abd: 2020-12-08T22:13:13.467Z [INFO] server-c9357880-e9da-acf1-841d-d89b86232abd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9357880-e9da-acf1-841d-d89b86232abd Address:c9357880-e9da-acf1-841d-d89b86232abd}]"
> --- PASS: TestRaft_AfterShutdown (0.00s)
> === RUN TestRaft_LiveBootstrap
> testLoggerAdapter verbose: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.468Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: initial configuration: index=0 servers=[]
> testing.go:149: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.468Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: initial configuration: index=0 servers=[]
> testLoggerAdapter verbose: server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: 2020-12-08T22:13:13.468Z [INFO] server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: initial configuration: index=0 servers=[]
> testing.go:149: server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: 2020-12-08T22:13:13.468Z [INFO] server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: initial configuration: index=0 servers=[]
> testLoggerAdapter verbose: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.468Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: entering follower state: follower="Node at f38f036d-2aa5-0883-ddd0-d740e49738ec [Follower]" leader=
> testLoggerAdapter verbose: server-a96139d6-3e92-6962-a4df-23dd15dee1a8: 2020-12-08T22:13:13.468Z [INFO] server-a96139d6-3e92-6962-a4df-23dd15dee1a8: initial configuration: index=0 servers=[]
> testing.go:149: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.468Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: entering follower state: follower="Node at f38f036d-2aa5-0883-ddd0-d740e49738ec [Follower]" leader=
> testLoggerAdapter verbose: server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: 2020-12-08T22:13:13.468Z [INFO] server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: entering follower state: follower="Node at 5191d735-40ca-7e89-2b4e-ecce064ddcb7 [Follower]" leader=
> testing.go:149: server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: 2020-12-08T22:13:13.468Z [INFO] server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: entering follower state: follower="Node at 5191d735-40ca-7e89-2b4e-ecce064ddcb7 [Follower]" leader=
> testing.go:149: server-a96139d6-3e92-6962-a4df-23dd15dee1a8: 2020-12-08T22:13:13.468Z [INFO] server-a96139d6-3e92-6962-a4df-23dd15dee1a8: initial configuration: index=0 servers=[]
> testLoggerAdapter verbose: server-a96139d6-3e92-6962-a4df-23dd15dee1a8: 2020-12-08T22:13:13.468Z [INFO] server-a96139d6-3e92-6962-a4df-23dd15dee1a8: entering follower state: follower="Node at a96139d6-3e92-6962-a4df-23dd15dee1a8 [Follower]" leader=
> testing.go:149: server-a96139d6-3e92-6962-a4df-23dd15dee1a8: 2020-12-08T22:13:13.468Z [INFO] server-a96139d6-3e92-6962-a4df-23dd15dee1a8: entering follower state: follower="Node at a96139d6-3e92-6962-a4df-23dd15dee1a8 [Follower]" leader=
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:13.468Z [INFO] starting stability test: raft-state=Follower
> testing.go:149: cluster: 2020-12-08T22:13:13.468Z [INFO] starting stability test: raft-state=Follower
> testLoggerAdapter verbose: server-a96139d6-3e92-6962-a4df-23dd15dee1a8: 2020-12-08T22:13:13.522Z [WARN] server-a96139d6-3e92-6962-a4df-23dd15dee1a8: no known peers, aborting election
> testing.go:149: server-a96139d6-3e92-6962-a4df-23dd15dee1a8: 2020-12-08T22:13:13.522Z [WARN] server-a96139d6-3e92-6962-a4df-23dd15dee1a8: no known peers, aborting election
> testLoggerAdapter verbose: server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: 2020-12-08T22:13:13.542Z [WARN] server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: no known peers, aborting election
> testing.go:149: server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: 2020-12-08T22:13:13.542Z [WARN] server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: no known peers, aborting election
> testLoggerAdapter verbose: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [WARN] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [WARN] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: entering candidate state: node="Node at f38f036d-2aa5-0883-ddd0-d740e49738ec [Candidate]" term=2
> testing.go:149: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: entering candidate state: node="Node at f38f036d-2aa5-0883-ddd0-d740e49738ec [Candidate]" term=2
> testLoggerAdapter verbose: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: election won: tally=2
> testing.go:149: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: election won: tally=2
> testLoggerAdapter verbose: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: entering leader state: leader="Node at f38f036d-2aa5-0883-ddd0-d740e49738ec [Leader]"
> testing.go:149: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: entering leader state: leader="Node at f38f036d-2aa5-0883-ddd0-d740e49738ec [Leader]"
> testLoggerAdapter verbose: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: added peer, starting replication: peer=server-5191d735-40ca-7e89-2b4e-ecce064ddcb7
> testing.go:149: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: added peer, starting replication: peer=server-5191d735-40ca-7e89-2b4e-ecce064ddcb7
> testLoggerAdapter verbose: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: added peer, starting replication: peer=server-a96139d6-3e92-6962-a4df-23dd15dee1a8
> testing.go:149: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: added peer, starting replication: peer=server-a96139d6-3e92-6962-a4df-23dd15dee1a8
> testLoggerAdapter verbose: server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: 2020-12-08T22:13:13.560Z [WARN] server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: failed to get previous log: previous-index=1 last-index=0 error="log not found"
> testLoggerAdapter verbose: server-a96139d6-3e92-6962-a4df-23dd15dee1a8: 2020-12-08T22:13:13.560Z [WARN] server-a96139d6-3e92-6962-a4df-23dd15dee1a8: failed to get previous log: previous-index=1 last-index=0 error="log not found"
> testing.go:149: server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: 2020-12-08T22:13:13.560Z [WARN] server-5191d735-40ca-7e89-2b4e-ecce064ddcb7: failed to get previous log: previous-index=1 last-index=0 error="log not found"
> testing.go:149: server-a96139d6-3e92-6962-a4df-23dd15dee1a8: 2020-12-08T22:13:13.560Z [WARN] server-a96139d6-3e92-6962-a4df-23dd15dee1a8: failed to get previous log: previous-index=1 last-index=0 error="log not found"
> testLoggerAdapter verbose: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [WARN] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: appendEntries rejected, sending older logs: peer="{Voter server-5191d735-40ca-7e89-2b4e-ecce064ddcb7 5191d735-40ca-7e89-2b4e-ecce064ddcb7}" next=1
> testing.go:149: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [WARN] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: appendEntries rejected, sending older logs: peer="{Voter server-5191d735-40ca-7e89-2b4e-ecce064ddcb7 5191d735-40ca-7e89-2b4e-ecce064ddcb7}" next=1
> testLoggerAdapter verbose: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [WARN] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: appendEntries rejected, sending older logs: peer="{Voter server-a96139d6-3e92-6962-a4df-23dd15dee1a8 a96139d6-3e92-6962-a4df-23dd15dee1a8}" next=1
> testing.go:149: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [WARN] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: appendEntries rejected, sending older logs: peer="{Voter server-a96139d6-3e92-6962-a4df-23dd15dee1a8 a96139d6-3e92-6962-a4df-23dd15dee1a8}" next=1
> testLoggerAdapter verbose: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: pipelining replication: peer="{Voter server-5191d735-40ca-7e89-2b4e-ecce064ddcb7 5191d735-40ca-7e89-2b4e-ecce064ddcb7}"
> testing.go:149: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: pipelining replication: peer="{Voter server-5191d735-40ca-7e89-2b4e-ecce064ddcb7 5191d735-40ca-7e89-2b4e-ecce064ddcb7}"
> testLoggerAdapter verbose: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: pipelining replication: peer="{Voter server-a96139d6-3e92-6962-a4df-23dd15dee1a8 a96139d6-3e92-6962-a4df-23dd15dee1a8}"
> testing.go:149: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.560Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: pipelining replication: peer="{Voter server-a96139d6-3e92-6962-a4df-23dd15dee1a8 a96139d6-3e92-6962-a4df-23dd15dee1a8}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:13.665Z [INFO] stable state for Follower reached at 2020-12-08 22:13:13.560339356 +0000 UTC m=+5.117499922 (2 nodes), 91.611438ms from start of poll, 92.03477ms from cluster start. Timeout at 2020-12-08 22:13:13.665425166 +0000 UTC m=+5.222585738, 105.085816ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:13.665Z [INFO] stable state for Follower reached at 2020-12-08 22:13:13.560339356 +0000 UTC m=+5.117499922 (2 nodes), 91.611438ms from start of poll, 92.03477ms from cluster start. Timeout at 2020-12-08 22:13:13.665425166 +0000 UTC m=+5.222585738, 105.085816ms after stability
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:13.665Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:13.665Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:13.770Z [INFO] stable state for Leader reached at 2020-12-08 22:13:13.665510645 +0000 UTC m=+5.222671209 (1 nodes), 484ns from start of poll, 197.206057ms from cluster start. Timeout at 2020-12-08 22:13:13.770621589 +0000 UTC m=+5.327782156, 105.110947ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:13.770Z [INFO] stable state for Leader reached at 2020-12-08 22:13:13.665510645 +0000 UTC m=+5.222671209 (1 nodes), 484ns from start of poll, 197.206057ms from cluster start. Timeout at 2020-12-08 22:13:13.770621589 +0000 UTC m=+5.327782156, 105.110947ms after stability
> testLoggerAdapter verbose: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.780Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: aborting pipeline replication: peer="{Voter server-a96139d6-3e92-6962-a4df-23dd15dee1a8 a96139d6-3e92-6962-a4df-23dd15dee1a8}"
> testing.go:149: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.780Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: aborting pipeline replication: peer="{Voter server-a96139d6-3e92-6962-a4df-23dd15dee1a8 a96139d6-3e92-6962-a4df-23dd15dee1a8}"
> testLoggerAdapter verbose: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.780Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: aborting pipeline replication: peer="{Voter server-5191d735-40ca-7e89-2b4e-ecce064ddcb7 5191d735-40ca-7e89-2b4e-ecce064ddcb7}"
> testing.go:149: server-f38f036d-2aa5-0883-ddd0-d740e49738ec: 2020-12-08T22:13:13.780Z [INFO] server-f38f036d-2aa5-0883-ddd0-d740e49738ec: aborting pipeline replication: peer="{Voter server-5191d735-40ca-7e89-2b4e-ecce064ddcb7 5191d735-40ca-7e89-2b4e-ecce064ddcb7}"
> --- PASS: TestRaft_LiveBootstrap (0.31s)
> === RUN TestRaft_RecoverCluster_NoState
> testLoggerAdapter verbose: server-46e38166-c713-aca4-ede2-8dc35e4b9113: 2020-12-08T22:13:13.782Z [INFO] server-46e38166-c713-aca4-ede2-8dc35e4b9113: initial configuration: index=0 servers=[]
> testing.go:149: server-46e38166-c713-aca4-ede2-8dc35e4b9113: 2020-12-08T22:13:13.782Z [INFO] server-46e38166-c713-aca4-ede2-8dc35e4b9113: initial configuration: index=0 servers=[]
> testLoggerAdapter verbose: server-46e38166-c713-aca4-ede2-8dc35e4b9113: 2020-12-08T22:13:13.782Z [INFO] server-46e38166-c713-aca4-ede2-8dc35e4b9113: entering follower state: follower="Node at 46e38166-c713-aca4-ede2-8dc35e4b9113 [Follower]" leader=
> testing.go:149: server-46e38166-c713-aca4-ede2-8dc35e4b9113: 2020-12-08T22:13:13.782Z [INFO] server-46e38166-c713-aca4-ede2-8dc35e4b9113: entering follower state: follower="Node at 46e38166-c713-aca4-ede2-8dc35e4b9113 [Follower]" leader=
> --- PASS: TestRaft_RecoverCluster_NoState (0.00s)
> === RUN TestRaft_RecoverCluster
> raft_test.go:119: Disabled unreliable test
> --- SKIP: TestRaft_RecoverCluster (0.00s)
> === RUN TestRaft_HasExistingState
> testLoggerAdapter verbose: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.783Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d476dfa-560a-0e4a-4c40-3a0862d18a53 Address:3d476dfa-560a-0e4a-4c40-3a0862d18a53} {Suffrage:Voter ID:server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9 Address:2ff7f1c4-7d37-8e6e-650d-16507c396cd9}]"
> testing.go:149: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.783Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d476dfa-560a-0e4a-4c40-3a0862d18a53 Address:3d476dfa-560a-0e4a-4c40-3a0862d18a53} {Suffrage:Voter ID:server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9 Address:2ff7f1c4-7d37-8e6e-650d-16507c396cd9}]"
> testLoggerAdapter verbose: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.783Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: entering follower state: follower="Node at 3d476dfa-560a-0e4a-4c40-3a0862d18a53 [Follower]" leader=
> testing.go:149: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.783Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: entering follower state: follower="Node at 3d476dfa-560a-0e4a-4c40-3a0862d18a53 [Follower]" leader=
> testLoggerAdapter verbose: server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9: 2020-12-08T22:13:13.783Z [INFO] server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d476dfa-560a-0e4a-4c40-3a0862d18a53 Address:3d476dfa-560a-0e4a-4c40-3a0862d18a53} {Suffrage:Voter ID:server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9 Address:2ff7f1c4-7d37-8e6e-650d-16507c396cd9}]"
> testing.go:149: server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9: 2020-12-08T22:13:13.783Z [INFO] server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d476dfa-560a-0e4a-4c40-3a0862d18a53 Address:3d476dfa-560a-0e4a-4c40-3a0862d18a53} {Suffrage:Voter ID:server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9 Address:2ff7f1c4-7d37-8e6e-650d-16507c396cd9}]"
> testLoggerAdapter verbose: server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9: 2020-12-08T22:13:13.783Z [INFO] server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9: entering follower state: follower="Node at 2ff7f1c4-7d37-8e6e-650d-16507c396cd9 [Follower]" leader=
> testing.go:149: server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9: 2020-12-08T22:13:13.783Z [INFO] server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9: entering follower state: follower="Node at 2ff7f1c4-7d37-8e6e-650d-16507c396cd9 [Follower]" leader=
> testLoggerAdapter verbose: server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: 2020-12-08T22:13:13.783Z [INFO] server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: initial configuration: index=0 servers=[]
> testing.go:149: server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: 2020-12-08T22:13:13.783Z [INFO] server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: initial configuration: index=0 servers=[]
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:13.783Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: 2020-12-08T22:13:13.783Z [INFO] server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: entering follower state: follower="Node at 5dda7d91-07a9-8e21-c15d-cc72380b90e9 [Follower]" leader=
> testing.go:149: cluster: 2020-12-08T22:13:13.783Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: 2020-12-08T22:13:13.783Z [INFO] server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: entering follower state: follower="Node at 5dda7d91-07a9-8e21-c15d-cc72380b90e9 [Follower]" leader=
> testLoggerAdapter verbose: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.841Z [WARN] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.841Z [WARN] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.841Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: entering candidate state: node="Node at 3d476dfa-560a-0e4a-4c40-3a0862d18a53 [Candidate]" term=2
> testing.go:149: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.841Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: entering candidate state: node="Node at 3d476dfa-560a-0e4a-4c40-3a0862d18a53 [Candidate]" term=2
> testLoggerAdapter verbose: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.841Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: election won: tally=2
> testing.go:149: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.841Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: election won: tally=2
> testLoggerAdapter verbose: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.841Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: entering leader state: leader="Node at 3d476dfa-560a-0e4a-4c40-3a0862d18a53 [Leader]"
> testing.go:149: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.841Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: entering leader state: leader="Node at 3d476dfa-560a-0e4a-4c40-3a0862d18a53 [Leader]"
> testLoggerAdapter verbose: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.841Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: added peer, starting replication: peer=server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9
> testing.go:149: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.841Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: added peer, starting replication: peer=server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9
> testLoggerAdapter verbose: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.841Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: pipelining replication: peer="{Voter server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9 2ff7f1c4-7d37-8e6e-650d-16507c396cd9}"
> testing.go:149: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.841Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: pipelining replication: peer="{Voter server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9 2ff7f1c4-7d37-8e6e-650d-16507c396cd9}"
> testLoggerAdapter verbose: server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: 2020-12-08T22:13:13.841Z [WARN] server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: no known peers, aborting election
> testing.go:149: server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: 2020-12-08T22:13:13.841Z [WARN] server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: no known peers, aborting election
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:13.946Z [INFO] stable state for Leader reached at 2020-12-08 22:13:13.841451253 +0000 UTC m=+5.398611821 (1 nodes), 57.851054ms from start of poll, 58.485171ms from cluster start. Timeout at 2020-12-08 22:13:13.946547613 +0000 UTC m=+5.503708181, 105.09636ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:13.946Z [INFO] stable state for Leader reached at 2020-12-08 22:13:13.841451253 +0000 UTC m=+5.398611821 (1 nodes), 57.851054ms from start of poll, 58.485171ms from cluster start. Timeout at 2020-12-08 22:13:13.946547613 +0000 UTC m=+5.503708181, 105.09636ms after stability
> testLoggerAdapter verbose: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.946Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: updating configuration: command=AddStaging server-id=server-5dda7d91-07a9-8e21-c15d-cc72380b90e9 server-addr=5dda7d91-07a9-8e21-c15d-cc72380b90e9 servers="[{Suffrage:Voter ID:server-3d476dfa-560a-0e4a-4c40-3a0862d18a53 Address:3d476dfa-560a-0e4a-4c40-3a0862d18a53} {Suffrage:Voter ID:server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9 Address:2ff7f1c4-7d37-8e6e-650d-16507c396cd9} {Suffrage:Voter ID:server-5dda7d91-07a9-8e21-c15d-cc72380b90e9 Address:5dda7d91-07a9-8e21-c15d-cc72380b90e9}]"
> testing.go:149: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.946Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: updating configuration: command=AddStaging server-id=server-5dda7d91-07a9-8e21-c15d-cc72380b90e9 server-addr=5dda7d91-07a9-8e21-c15d-cc72380b90e9 servers="[{Suffrage:Voter ID:server-3d476dfa-560a-0e4a-4c40-3a0862d18a53 Address:3d476dfa-560a-0e4a-4c40-3a0862d18a53} {Suffrage:Voter ID:server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9 Address:2ff7f1c4-7d37-8e6e-650d-16507c396cd9} {Suffrage:Voter ID:server-5dda7d91-07a9-8e21-c15d-cc72380b90e9 Address:5dda7d91-07a9-8e21-c15d-cc72380b90e9}]"
> testLoggerAdapter verbose: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.946Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: added peer, starting replication: peer=server-5dda7d91-07a9-8e21-c15d-cc72380b90e9
> testing.go:149: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.946Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: added peer, starting replication: peer=server-5dda7d91-07a9-8e21-c15d-cc72380b90e9
> testLoggerAdapter verbose: server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: 2020-12-08T22:13:13.946Z [WARN] server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: failed to get previous log: previous-index=3 last-index=0 error="log not found"
> testing.go:149: server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: 2020-12-08T22:13:13.946Z [WARN] server-5dda7d91-07a9-8e21-c15d-cc72380b90e9: failed to get previous log: previous-index=3 last-index=0 error="log not found"
> testLoggerAdapter verbose: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.946Z [WARN] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: appendEntries rejected, sending older logs: peer="{Voter server-5dda7d91-07a9-8e21-c15d-cc72380b90e9 5dda7d91-07a9-8e21-c15d-cc72380b90e9}" next=1
> testing.go:149: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.946Z [WARN] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: appendEntries rejected, sending older logs: peer="{Voter server-5dda7d91-07a9-8e21-c15d-cc72380b90e9 5dda7d91-07a9-8e21-c15d-cc72380b90e9}" next=1
> testLoggerAdapter verbose: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.946Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: pipelining replication: peer="{Voter server-5dda7d91-07a9-8e21-c15d-cc72380b90e9 5dda7d91-07a9-8e21-c15d-cc72380b90e9}"
> testing.go:149: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:13.946Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: pipelining replication: peer="{Voter server-5dda7d91-07a9-8e21-c15d-cc72380b90e9 5dda7d91-07a9-8e21-c15d-cc72380b90e9}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:13.951Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:13.951Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:14.057Z [INFO] stable state for Leader reached at 2020-12-08 22:13:13.951972661 +0000 UTC m=+5.509133229 (1 nodes), 829ns from start of poll, 169.006579ms from cluster start. Timeout at 2020-12-08 22:13:14.057051872 +0000 UTC m=+5.614212436, 105.079207ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:14.057Z [INFO] stable state for Leader reached at 2020-12-08 22:13:13.951972661 +0000 UTC m=+5.509133229 (1 nodes), 829ns from start of poll, 169.006579ms from cluster start. Timeout at 2020-12-08 22:13:14.057051872 +0000 UTC m=+5.614212436, 105.079207ms after stability
> testLoggerAdapter verbose: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:14.057Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: aborting pipeline replication: peer="{Voter server-5dda7d91-07a9-8e21-c15d-cc72380b90e9 5dda7d91-07a9-8e21-c15d-cc72380b90e9}"
> testing.go:149: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:14.057Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: aborting pipeline replication: peer="{Voter server-5dda7d91-07a9-8e21-c15d-cc72380b90e9 5dda7d91-07a9-8e21-c15d-cc72380b90e9}"
> testLoggerAdapter verbose: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:14.057Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: aborting pipeline replication: peer="{Voter server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9 2ff7f1c4-7d37-8e6e-650d-16507c396cd9}"
> testing.go:149: server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: 2020-12-08T22:13:14.057Z [INFO] server-3d476dfa-560a-0e4a-4c40-3a0862d18a53: aborting pipeline replication: peer="{Voter server-2ff7f1c4-7d37-8e6e-650d-16507c396cd9 2ff7f1c4-7d37-8e6e-650d-16507c396cd9}"
> --- PASS: TestRaft_HasExistingState (0.28s)
> === RUN TestRaft_SingleNode
> testLoggerAdapter verbose: server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: 2020-12-08T22:13:14.059Z [INFO] server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acdb5a62-54a7-44f7-cb73-33f1836c9e06 Address:acdb5a62-54a7-44f7-cb73-33f1836c9e06}]"
> testing.go:149: server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: 2020-12-08T22:13:14.059Z [INFO] server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acdb5a62-54a7-44f7-cb73-33f1836c9e06 Address:acdb5a62-54a7-44f7-cb73-33f1836c9e06}]"
> testLoggerAdapter verbose: server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: 2020-12-08T22:13:14.059Z [INFO] server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: entering follower state: follower="Node at acdb5a62-54a7-44f7-cb73-33f1836c9e06 [Follower]" leader=
> testing.go:149: server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: 2020-12-08T22:13:14.059Z [INFO] server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: entering follower state: follower="Node at acdb5a62-54a7-44f7-cb73-33f1836c9e06 [Follower]" leader=
> testLoggerAdapter verbose: server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: 2020-12-08T22:13:14.141Z [WARN] server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: 2020-12-08T22:13:14.141Z [WARN] server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: 2020-12-08T22:13:14.142Z [INFO] server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: entering candidate state: node="Node at acdb5a62-54a7-44f7-cb73-33f1836c9e06 [Candidate]" term=2
> testing.go:149: server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: 2020-12-08T22:13:14.142Z [INFO] server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: entering candidate state: node="Node at acdb5a62-54a7-44f7-cb73-33f1836c9e06 [Candidate]" term=2
> testLoggerAdapter verbose: server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: 2020-12-08T22:13:14.142Z [INFO] server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: election won: tally=1
> testing.go:149: server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: 2020-12-08T22:13:14.142Z [INFO] server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: election won: tally=1
> testLoggerAdapter verbose: server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: 2020-12-08T22:13:14.142Z [INFO] server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: entering leader state: leader="Node at acdb5a62-54a7-44f7-cb73-33f1836c9e06 [Leader]"
> testing.go:149: server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: 2020-12-08T22:13:14.142Z [INFO] server-acdb5a62-54a7-44f7-cb73-33f1836c9e06: entering leader state: leader="Node at acdb5a62-54a7-44f7-cb73-33f1836c9e06 [Leader]"
> --- PASS: TestRaft_SingleNode (0.08s)
> === RUN TestRaft_TripleNode
> testLoggerAdapter verbose: server-58bb052b-e5e8-daa3-820b-cc43034b9d62: 2020-12-08T22:13:14.143Z [INFO] server-58bb052b-e5e8-daa3-820b-cc43034b9d62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58bb052b-e5e8-daa3-820b-cc43034b9d62 Address:58bb052b-e5e8-daa3-820b-cc43034b9d62} {Suffrage:Voter ID:server-0c4402ca-0359-82e2-343c-93a7106dc0b8 Address:0c4402ca-0359-82e2-343c-93a7106dc0b8} {Suffrage:Voter ID:server-d179aa01-8043-5924-7869-51d5c6c3d45e Address:d179aa01-8043-5924-7869-51d5c6c3d45e}]"
> testing.go:149: server-58bb052b-e5e8-daa3-820b-cc43034b9d62: 2020-12-08T22:13:14.143Z [INFO] server-58bb052b-e5e8-daa3-820b-cc43034b9d62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58bb052b-e5e8-daa3-820b-cc43034b9d62 Address:58bb052b-e5e8-daa3-820b-cc43034b9d62} {Suffrage:Voter ID:server-0c4402ca-0359-82e2-343c-93a7106dc0b8 Address:0c4402ca-0359-82e2-343c-93a7106dc0b8} {Suffrage:Voter ID:server-d179aa01-8043-5924-7869-51d5c6c3d45e Address:d179aa01-8043-5924-7869-51d5c6c3d45e}]"
> testLoggerAdapter verbose: server-58bb052b-e5e8-daa3-820b-cc43034b9d62: 2020-12-08T22:13:14.143Z [INFO] server-58bb052b-e5e8-daa3-820b-cc43034b9d62: entering follower state: follower="Node at 58bb052b-e5e8-daa3-820b-cc43034b9d62 [Follower]" leader=
> testing.go:149: server-58bb052b-e5e8-daa3-820b-cc43034b9d62: 2020-12-08T22:13:14.143Z [INFO] server-58bb052b-e5e8-daa3-820b-cc43034b9d62: entering follower state: follower="Node at 58bb052b-e5e8-daa3-820b-cc43034b9d62 [Follower]" leader=
> testLoggerAdapter verbose: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.143Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58bb052b-e5e8-daa3-820b-cc43034b9d62 Address:58bb052b-e5e8-daa3-820b-cc43034b9d62} {Suffrage:Voter ID:server-0c4402ca-0359-82e2-343c-93a7106dc0b8 Address:0c4402ca-0359-82e2-343c-93a7106dc0b8} {Suffrage:Voter ID:server-d179aa01-8043-5924-7869-51d5c6c3d45e Address:d179aa01-8043-5924-7869-51d5c6c3d45e}]"
> testing.go:149: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.143Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58bb052b-e5e8-daa3-820b-cc43034b9d62 Address:58bb052b-e5e8-daa3-820b-cc43034b9d62} {Suffrage:Voter ID:server-0c4402ca-0359-82e2-343c-93a7106dc0b8 Address:0c4402ca-0359-82e2-343c-93a7106dc0b8} {Suffrage:Voter ID:server-d179aa01-8043-5924-7869-51d5c6c3d45e Address:d179aa01-8043-5924-7869-51d5c6c3d45e}]"
> testLoggerAdapter verbose: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.143Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: entering follower state: follower="Node at 0c4402ca-0359-82e2-343c-93a7106dc0b8 [Follower]" leader=
> testing.go:149: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.143Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: entering follower state: follower="Node at 0c4402ca-0359-82e2-343c-93a7106dc0b8 [Follower]" leader=
> testLoggerAdapter verbose: server-d179aa01-8043-5924-7869-51d5c6c3d45e: 2020-12-08T22:13:14.143Z [INFO] server-d179aa01-8043-5924-7869-51d5c6c3d45e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58bb052b-e5e8-daa3-820b-cc43034b9d62 Address:58bb052b-e5e8-daa3-820b-cc43034b9d62} {Suffrage:Voter ID:server-0c4402ca-0359-82e2-343c-93a7106dc0b8 Address:0c4402ca-0359-82e2-343c-93a7106dc0b8} {Suffrage:Voter ID:server-d179aa01-8043-5924-7869-51d5c6c3d45e Address:d179aa01-8043-5924-7869-51d5c6c3d45e}]"
> testing.go:149: server-d179aa01-8043-5924-7869-51d5c6c3d45e: 2020-12-08T22:13:14.143Z [INFO] server-d179aa01-8043-5924-7869-51d5c6c3d45e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58bb052b-e5e8-daa3-820b-cc43034b9d62 Address:58bb052b-e5e8-daa3-820b-cc43034b9d62} {Suffrage:Voter ID:server-0c4402ca-0359-82e2-343c-93a7106dc0b8 Address:0c4402ca-0359-82e2-343c-93a7106dc0b8} {Suffrage:Voter ID:server-d179aa01-8043-5924-7869-51d5c6c3d45e Address:d179aa01-8043-5924-7869-51d5c6c3d45e}]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:14.143Z [INFO] starting stability test: raft-state=Follower
> testing.go:149: cluster: 2020-12-08T22:13:14.143Z [INFO] starting stability test: raft-state=Follower
> testLoggerAdapter verbose: server-d179aa01-8043-5924-7869-51d5c6c3d45e: 2020-12-08T22:13:14.144Z [INFO] server-d179aa01-8043-5924-7869-51d5c6c3d45e: entering follower state: follower="Node at d179aa01-8043-5924-7869-51d5c6c3d45e [Follower]" leader=
> testing.go:149: server-d179aa01-8043-5924-7869-51d5c6c3d45e: 2020-12-08T22:13:14.144Z [INFO] server-d179aa01-8043-5924-7869-51d5c6c3d45e: entering follower state: follower="Node at d179aa01-8043-5924-7869-51d5c6c3d45e [Follower]" leader=
> testLoggerAdapter verbose: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [WARN] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [WARN] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: entering candidate state: node="Node at 0c4402ca-0359-82e2-343c-93a7106dc0b8 [Candidate]" term=2
> testing.go:149: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: entering candidate state: node="Node at 0c4402ca-0359-82e2-343c-93a7106dc0b8 [Candidate]" term=2
> testLoggerAdapter verbose: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: election won: tally=2
> testing.go:149: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: election won: tally=2
> testLoggerAdapter verbose: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: entering leader state: leader="Node at 0c4402ca-0359-82e2-343c-93a7106dc0b8 [Leader]"
> testing.go:149: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: entering leader state: leader="Node at 0c4402ca-0359-82e2-343c-93a7106dc0b8 [Leader]"
> testLoggerAdapter verbose: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: added peer, starting replication: peer=server-58bb052b-e5e8-daa3-820b-cc43034b9d62
> testing.go:149: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: added peer, starting replication: peer=server-58bb052b-e5e8-daa3-820b-cc43034b9d62
> testLoggerAdapter verbose: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: added peer, starting replication: peer=server-d179aa01-8043-5924-7869-51d5c6c3d45e
> testing.go:149: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: added peer, starting replication: peer=server-d179aa01-8043-5924-7869-51d5c6c3d45e
> testLoggerAdapter verbose: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: pipelining replication: peer="{Voter server-58bb052b-e5e8-daa3-820b-cc43034b9d62 58bb052b-e5e8-daa3-820b-cc43034b9d62}"
> testing.go:149: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: pipelining replication: peer="{Voter server-58bb052b-e5e8-daa3-820b-cc43034b9d62 58bb052b-e5e8-daa3-820b-cc43034b9d62}"
> testLoggerAdapter verbose: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: pipelining replication: peer="{Voter server-d179aa01-8043-5924-7869-51d5c6c3d45e d179aa01-8043-5924-7869-51d5c6c3d45e}"
> testing.go:149: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.223Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: pipelining replication: peer="{Voter server-d179aa01-8043-5924-7869-51d5c6c3d45e d179aa01-8043-5924-7869-51d5c6c3d45e}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:14.328Z [INFO] stable state for Follower reached at 2020-12-08 22:13:14.223767477 +0000 UTC m=+5.780928049 (2 nodes), 79.744455ms from start of poll, 80.292875ms from cluster start. Timeout at 2020-12-08 22:13:14.328861699 +0000 UTC m=+5.886022251, 105.094202ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:14.328Z [INFO] stable state for Follower reached at 2020-12-08 22:13:14.223767477 +0000 UTC m=+5.780928049 (2 nodes), 79.744455ms from start of poll, 80.292875ms from cluster start. Timeout at 2020-12-08 22:13:14.328861699 +0000 UTC m=+5.886022251, 105.094202ms after stability
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:14.328Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:14.328Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:14.434Z [INFO] stable state for Leader reached at 2020-12-08 22:13:14.328940668 +0000 UTC m=+5.886101226 (1 nodes), 662ns from start of poll, 185.466052ms from cluster start. Timeout at 2020-12-08 22:13:14.434017103 +0000 UTC m=+5.991177658, 105.076432ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:14.434Z [INFO] stable state for Leader reached at 2020-12-08 22:13:14.328940668 +0000 UTC m=+5.886101226 (1 nodes), 662ns from start of poll, 185.466052ms from cluster start. Timeout at 2020-12-08 22:13:14.434017103 +0000 UTC m=+5.991177658, 105.076432ms after stability
> testLoggerAdapter verbose: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.444Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: aborting pipeline replication: peer="{Voter server-58bb052b-e5e8-daa3-820b-cc43034b9d62 58bb052b-e5e8-daa3-820b-cc43034b9d62}"
> testing.go:149: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.444Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: aborting pipeline replication: peer="{Voter server-58bb052b-e5e8-daa3-820b-cc43034b9d62 58bb052b-e5e8-daa3-820b-cc43034b9d62}"
> testLoggerAdapter verbose: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.444Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: aborting pipeline replication: peer="{Voter server-d179aa01-8043-5924-7869-51d5c6c3d45e d179aa01-8043-5924-7869-51d5c6c3d45e}"
> testing.go:149: server-0c4402ca-0359-82e2-343c-93a7106dc0b8: 2020-12-08T22:13:14.444Z [INFO] server-0c4402ca-0359-82e2-343c-93a7106dc0b8: aborting pipeline replication: peer="{Voter server-d179aa01-8043-5924-7869-51d5c6c3d45e d179aa01-8043-5924-7869-51d5c6c3d45e}"
> --- PASS: TestRaft_TripleNode (0.30s)
> === RUN TestRaft_LeaderFail
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.446Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b885cfcd-019e-62b7-937a-1fa28385d172 Address:b885cfcd-019e-62b7-937a-1fa28385d172} {Suffrage:Voter ID:server-99f94e4e-1ab1-64de-b525-ddfe174636b4 Address:99f94e4e-1ab1-64de-b525-ddfe174636b4} {Suffrage:Voter ID:server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 Address:2bf9493e-0309-447d-e2ee-b856d2a9ddd8}]"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.446Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b885cfcd-019e-62b7-937a-1fa28385d172 Address:b885cfcd-019e-62b7-937a-1fa28385d172} {Suffrage:Voter ID:server-99f94e4e-1ab1-64de-b525-ddfe174636b4 Address:99f94e4e-1ab1-64de-b525-ddfe174636b4} {Suffrage:Voter ID:server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 Address:2bf9493e-0309-447d-e2ee-b856d2a9ddd8}]"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.446Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: entering follower state: follower="Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Follower]" leader=
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.446Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: entering follower state: follower="Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Follower]" leader=
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.446Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b885cfcd-019e-62b7-937a-1fa28385d172 Address:b885cfcd-019e-62b7-937a-1fa28385d172} {Suffrage:Voter ID:server-99f94e4e-1ab1-64de-b525-ddfe174636b4 Address:99f94e4e-1ab1-64de-b525-ddfe174636b4} {Suffrage:Voter ID:server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 Address:2bf9493e-0309-447d-e2ee-b856d2a9ddd8}]"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.446Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b885cfcd-019e-62b7-937a-1fa28385d172 Address:b885cfcd-019e-62b7-937a-1fa28385d172} {Suffrage:Voter ID:server-99f94e4e-1ab1-64de-b525-ddfe174636b4 Address:99f94e4e-1ab1-64de-b525-ddfe174636b4} {Suffrage:Voter ID:server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 Address:2bf9493e-0309-447d-e2ee-b856d2a9ddd8}]"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.446Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: entering follower state: follower="Node at 99f94e4e-1ab1-64de-b525-ddfe174636b4 [Follower]" leader=
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.446Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: entering follower state: follower="Node at 99f94e4e-1ab1-64de-b525-ddfe174636b4 [Follower]" leader=
> testLoggerAdapter verbose: server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: 2020-12-08T22:13:14.446Z [INFO] server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b885cfcd-019e-62b7-937a-1fa28385d172 Address:b885cfcd-019e-62b7-937a-1fa28385d172} {Suffrage:Voter ID:server-99f94e4e-1ab1-64de-b525-ddfe174636b4 Address:99f94e4e-1ab1-64de-b525-ddfe174636b4} {Suffrage:Voter ID:server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 Address:2bf9493e-0309-447d-e2ee-b856d2a9ddd8}]"
> testing.go:149: server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: 2020-12-08T22:13:14.446Z [INFO] server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b885cfcd-019e-62b7-937a-1fa28385d172 Address:b885cfcd-019e-62b7-937a-1fa28385d172} {Suffrage:Voter ID:server-99f94e4e-1ab1-64de-b525-ddfe174636b4 Address:99f94e4e-1ab1-64de-b525-ddfe174636b4} {Suffrage:Voter ID:server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 Address:2bf9493e-0309-447d-e2ee-b856d2a9ddd8}]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:14.446Z [INFO] starting stability test: raft-state=Follower
> testing.go:149: cluster: 2020-12-08T22:13:14.446Z [INFO] starting stability test: raft-state=Follower
> testLoggerAdapter verbose: server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: 2020-12-08T22:13:14.446Z [INFO] server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: entering follower state: follower="Node at 2bf9493e-0309-447d-e2ee-b856d2a9ddd8 [Follower]" leader=
> testing.go:149: server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: 2020-12-08T22:13:14.446Z [INFO] server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: entering follower state: follower="Node at 2bf9493e-0309-447d-e2ee-b856d2a9ddd8 [Follower]" leader=
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.510Z [WARN] server-b885cfcd-019e-62b7-937a-1fa28385d172: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.510Z [WARN] server-b885cfcd-019e-62b7-937a-1fa28385d172: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.510Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: entering candidate state: node="Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Candidate]" term=2
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.510Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: entering candidate state: node="Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Candidate]" term=2
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.510Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: election won: tally=2
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.510Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: election won: tally=2
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.510Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: entering leader state: leader="Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Leader]"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.510Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: entering leader state: leader="Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Leader]"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.511Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: added peer, starting replication: peer=server-99f94e4e-1ab1-64de-b525-ddfe174636b4
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.511Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: added peer, starting replication: peer=server-99f94e4e-1ab1-64de-b525-ddfe174636b4
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.511Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: added peer, starting replication: peer=server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.511Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: added peer, starting replication: peer=server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.511Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: pipelining replication: peer="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.511Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: pipelining replication: peer="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.511Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: pipelining replication: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.511Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: pipelining replication: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:14.616Z [INFO] stable state for Follower reached at 2020-12-08 22:13:14.510994282 +0000 UTC m=+6.068154857 (2 nodes), 64.530229ms from start of poll, 65.061022ms from cluster start. Timeout at 2020-12-08 22:13:14.616103495 +0000 UTC m=+6.173264053, 105.109196ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:14.616Z [INFO] stable state for Follower reached at 2020-12-08 22:13:14.510994282 +0000 UTC m=+6.068154857 (2 nodes), 64.530229ms from start of poll, 65.061022ms from cluster start. Timeout at 2020-12-08 22:13:14.616103495 +0000 UTC m=+6.173264053, 105.109196ms after stability
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:14.616Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:14.616Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:14.721Z [INFO] stable state for Leader reached at 2020-12-08 22:13:14.616203797 +0000 UTC m=+6.173364379 (1 nodes), 3.523µs from start of poll, 170.270544ms from cluster start. Timeout at 2020-12-08 22:13:14.721307934 +0000 UTC m=+6.278468504, 105.104125ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:14.721Z [INFO] stable state for Leader reached at 2020-12-08 22:13:14.616203797 +0000 UTC m=+6.173364379 (1 nodes), 3.523µs from start of poll, 170.270544ms from cluster start. Timeout at 2020-12-08 22:13:14.721307934 +0000 UTC m=+6.278468504, 105.104125ms after stability
> raft_test.go:336: [INFO] Disconnecting Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Leader]
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.733Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=2bf9493e-0309-447d-e2ee-b856d2a9ddd8 error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.733Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=2bf9493e-0309-447d-e2ee-b856d2a9ddd8 error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.736Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to pipeline appendEntries: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}" error="append pipeline closed"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.736Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to pipeline appendEntries: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}" error="append pipeline closed"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.736Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: aborting pipeline replication: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.736Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: aborting pipeline replication: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.736Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to pipeline appendEntries: peer="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}" error="append pipeline closed"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:14.736Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.736Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to pipeline appendEntries: peer="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}" error="append pipeline closed"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.736Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: aborting pipeline replication: peer="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}"
> testing.go:149: cluster: 2020-12-08T22:13:14.736Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.736Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: aborting pipeline replication: peer="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.737Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=99f94e4e-1ab1-64de-b525-ddfe174636b4 error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.737Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=99f94e4e-1ab1-64de-b525-ddfe174636b4 error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.743Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}" error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.743Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}" error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.746Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}" error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.746Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}" error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.750Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=2bf9493e-0309-447d-e2ee-b856d2a9ddd8 error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.750Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=2bf9493e-0309-447d-e2ee-b856d2a9ddd8 error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.757Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=99f94e4e-1ab1-64de-b525-ddfe174636b4 error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.757Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=99f94e4e-1ab1-64de-b525-ddfe174636b4 error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.759Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}" error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.759Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}" error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.763Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}" error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.763Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}" error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.770Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=2bf9493e-0309-447d-e2ee-b856d2a9ddd8 error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.770Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=2bf9493e-0309-447d-e2ee-b856d2a9ddd8 error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.776Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=99f94e4e-1ab1-64de-b525-ddfe174636b4 error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.776Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=99f94e4e-1ab1-64de-b525-ddfe174636b4 error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.778Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}" error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.778Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}" error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.779Z [WARN] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to contact: server-id=server-99f94e4e-1ab1-64de-b525-ddfe174636b4 time=50.09033ms
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.779Z [WARN] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to contact: server-id=server-99f94e4e-1ab1-64de-b525-ddfe174636b4 time=50.09033ms
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.780Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}" error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.780Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}" error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.789Z [WARN] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to contact: server-id=server-99f94e4e-1ab1-64de-b525-ddfe174636b4 time=60.215193ms
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.789Z [WARN] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to contact: server-id=server-99f94e4e-1ab1-64de-b525-ddfe174636b4 time=60.215193ms
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.790Z [WARN] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to contact: server-id=server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 time=59.616371ms
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.790Z [WARN] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to contact: server-id=server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 time=59.616371ms
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.790Z [WARN] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to contact quorum of nodes, stepping down
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.790Z [WARN] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to contact quorum of nodes, stepping down
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.790Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: entering follower state: follower="Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Follower]" leader=
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.790Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: entering follower state: follower="Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Follower]" leader=
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.798Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=2bf9493e-0309-447d-e2ee-b856d2a9ddd8 error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.798Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=2bf9493e-0309-447d-e2ee-b856d2a9ddd8 error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.803Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=99f94e4e-1ab1-64de-b525-ddfe174636b4 error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.803Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to heartbeat to: peer=99f94e4e-1ab1-64de-b525-ddfe174636b4 error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.804Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}" error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.804Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}" error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.806Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}" error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.806Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to appendEntries to: peer="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}" error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.835Z [WARN] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: heartbeat timeout reached, starting election: last-leader=b885cfcd-019e-62b7-937a-1fa28385d172
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.835Z [WARN] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: heartbeat timeout reached, starting election: last-leader=b885cfcd-019e-62b7-937a-1fa28385d172
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.835Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: entering candidate state: node="Node at 99f94e4e-1ab1-64de-b525-ddfe174636b4 [Candidate]" term=3
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.835Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: entering candidate state: node="Node at 99f94e4e-1ab1-64de-b525-ddfe174636b4 [Candidate]" term=3
> testLoggerAdapter verbose: server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: 2020-12-08T22:13:14.835Z [WARN] server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: rejecting vote request since we have a leader: from=99f94e4e-1ab1-64de-b525-ddfe174636b4 leader=b885cfcd-019e-62b7-937a-1fa28385d172
> testing.go:149: server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: 2020-12-08T22:13:14.835Z [WARN] server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: rejecting vote request since we have a leader: from=99f94e4e-1ab1-64de-b525-ddfe174636b4 leader=b885cfcd-019e-62b7-937a-1fa28385d172
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.835Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to make requestVote RPC: target="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}" error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.835Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to make requestVote RPC: target="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}" error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testLoggerAdapter verbose: server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: 2020-12-08T22:13:14.840Z [WARN] server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: heartbeat timeout reached, starting election: last-leader=b885cfcd-019e-62b7-937a-1fa28385d172
> testing.go:149: server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: 2020-12-08T22:13:14.840Z [WARN] server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: heartbeat timeout reached, starting election: last-leader=b885cfcd-019e-62b7-937a-1fa28385d172
> testLoggerAdapter verbose: server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: 2020-12-08T22:13:14.840Z [INFO] server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: entering candidate state: node="Node at 2bf9493e-0309-447d-e2ee-b856d2a9ddd8 [Candidate]" term=3
> testing.go:149: server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: 2020-12-08T22:13:14.840Z [INFO] server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: entering candidate state: node="Node at 2bf9493e-0309-447d-e2ee-b856d2a9ddd8 [Candidate]" term=3
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.840Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: duplicate requestVote for same term: term=3
> testLoggerAdapter verbose: server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: 2020-12-08T22:13:14.840Z [ERROR] server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: failed to make requestVote RPC: target="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}" error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.840Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: duplicate requestVote for same term: term=3
> testing.go:149: server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: 2020-12-08T22:13:14.840Z [ERROR] server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: failed to make requestVote RPC: target="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}" error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.868Z [WARN] server-b885cfcd-019e-62b7-937a-1fa28385d172: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.868Z [WARN] server-b885cfcd-019e-62b7-937a-1fa28385d172: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.868Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: entering candidate state: node="Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Candidate]" term=3
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.868Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: entering candidate state: node="Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Candidate]" term=3
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.868Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to make requestVote RPC: target="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}" error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.868Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to make requestVote RPC: target="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}" error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.868Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to make requestVote RPC: target="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}" error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.868Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to make requestVote RPC: target="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}" error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.901Z [WARN] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: Election timeout reached, restarting election
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.901Z [WARN] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: Election timeout reached, restarting election
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.901Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: entering candidate state: node="Node at 99f94e4e-1ab1-64de-b525-ddfe174636b4 [Candidate]" term=4
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.901Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: entering candidate state: node="Node at 99f94e4e-1ab1-64de-b525-ddfe174636b4 [Candidate]" term=4
> testLoggerAdapter verbose: server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: 2020-12-08T22:13:14.901Z [INFO] server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: entering follower state: follower="Node at 2bf9493e-0309-447d-e2ee-b856d2a9ddd8 [Follower]" leader=
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.901Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to make requestVote RPC: target="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}" error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testing.go:149: server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: 2020-12-08T22:13:14.901Z [INFO] server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8: entering follower state: follower="Node at 2bf9493e-0309-447d-e2ee-b856d2a9ddd8 [Follower]" leader=
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.901Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to make requestVote RPC: target="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}" error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.901Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: election won: tally=2
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.901Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: election won: tally=2
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.901Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: entering leader state: leader="Node at 99f94e4e-1ab1-64de-b525-ddfe174636b4 [Leader]"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.901Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: entering leader state: leader="Node at 99f94e4e-1ab1-64de-b525-ddfe174636b4 [Leader]"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.901Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: added peer, starting replication: peer=server-b885cfcd-019e-62b7-937a-1fa28385d172
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.901Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: added peer, starting replication: peer=server-b885cfcd-019e-62b7-937a-1fa28385d172
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.901Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: added peer, starting replication: peer=server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.901Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: added peer, starting replication: peer=server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.902Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to appendEntries to: peer="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}" error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.902Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to appendEntries to: peer="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}" error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.902Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: pipelining replication: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.902Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: pipelining replication: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.908Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to heartbeat to: peer=b885cfcd-019e-62b7-937a-1fa28385d172 error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.908Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to heartbeat to: peer=b885cfcd-019e-62b7-937a-1fa28385d172 error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.921Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to appendEntries to: peer="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}" error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.921Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to appendEntries to: peer="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}" error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.924Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to heartbeat to: peer=b885cfcd-019e-62b7-937a-1fa28385d172 error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.924Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to heartbeat to: peer=b885cfcd-019e-62b7-937a-1fa28385d172 error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.938Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to appendEntries to: peer="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}" error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.938Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to appendEntries to: peer="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}" error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.944Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to heartbeat to: peer=b885cfcd-019e-62b7-937a-1fa28385d172 error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.944Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to heartbeat to: peer=b885cfcd-019e-62b7-937a-1fa28385d172 error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.947Z [WARN] server-b885cfcd-019e-62b7-937a-1fa28385d172: Election timeout reached, restarting election
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.947Z [WARN] server-b885cfcd-019e-62b7-937a-1fa28385d172: Election timeout reached, restarting election
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.947Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: entering candidate state: node="Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Candidate]" term=4
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.947Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: entering candidate state: node="Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Candidate]" term=4
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.947Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to make requestVote RPC: target="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}" error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.947Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to make requestVote RPC: target="{Voter server-99f94e4e-1ab1-64de-b525-ddfe174636b4 99f94e4e-1ab1-64de-b525-ddfe174636b4}" error="failed to connect to peer: 99f94e4e-1ab1-64de-b525-ddfe174636b4"
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.947Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to make requestVote RPC: target="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}" error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:14.947Z [ERROR] server-b885cfcd-019e-62b7-937a-1fa28385d172: failed to make requestVote RPC: target="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}" error="failed to connect to peer: 2bf9493e-0309-447d-e2ee-b856d2a9ddd8"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.952Z [WARN] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to contact: server-id=server-b885cfcd-019e-62b7-937a-1fa28385d172 time=50.128036ms
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.952Z [WARN] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to contact: server-id=server-b885cfcd-019e-62b7-937a-1fa28385d172 time=50.128036ms
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.963Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to appendEntries to: peer="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}" error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.963Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to appendEntries to: peer="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}" error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.970Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to heartbeat to: peer=b885cfcd-019e-62b7-937a-1fa28385d172 error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.970Z [ERROR] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to heartbeat to: peer=b885cfcd-019e-62b7-937a-1fa28385d172 error="failed to connect to peer: b885cfcd-019e-62b7-937a-1fa28385d172"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.996Z [WARN] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to contact: server-id=server-b885cfcd-019e-62b7-937a-1fa28385d172 time=94.280374ms
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:14.996Z [WARN] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: failed to contact: server-id=server-b885cfcd-019e-62b7-937a-1fa28385d172 time=94.280374ms
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:15.007Z [INFO] stable state for Leader reached at 2020-12-08 22:13:14.902015771 +0000 UTC m=+6.459176339 (1 nodes), 165.161633ms from start of poll, 456.082504ms from cluster start. Timeout at 2020-12-08 22:13:15.007126778 +0000 UTC m=+6.564287340, 105.111001ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:15.007Z [INFO] stable state for Leader reached at 2020-12-08 22:13:14.902015771 +0000 UTC m=+6.459176339 (1 nodes), 165.161633ms from start of poll, 456.082504ms from cluster start. Timeout at 2020-12-08 22:13:15.007126778 +0000 UTC m=+6.564287340, 105.111001ms after stability
> raft_test.go:371: [INFO] Reconnecting Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Candidate]
> testLoggerAdapter verbose: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:15.009Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: entering follower state: follower="Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Follower]" leader=99f94e4e-1ab1-64de-b525-ddfe174636b4
> testing.go:149: server-b885cfcd-019e-62b7-937a-1fa28385d172: 2020-12-08T22:13:15.009Z [INFO] server-b885cfcd-019e-62b7-937a-1fa28385d172: entering follower state: follower="Node at b885cfcd-019e-62b7-937a-1fa28385d172 [Follower]" leader=99f94e4e-1ab1-64de-b525-ddfe174636b4
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:15.009Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: pipelining replication: peer="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:15.009Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: pipelining replication: peer="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:15.015Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: aborting pipeline replication: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:15.015Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: aborting pipeline replication: peer="{Voter server-2bf9493e-0309-447d-e2ee-b856d2a9ddd8 2bf9493e-0309-447d-e2ee-b856d2a9ddd8}"
> testLoggerAdapter verbose: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:15.015Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: aborting pipeline replication: peer="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}"
> testing.go:149: server-99f94e4e-1ab1-64de-b525-ddfe174636b4: 2020-12-08T22:13:15.015Z [INFO] server-99f94e4e-1ab1-64de-b525-ddfe174636b4: aborting pipeline replication: peer="{Voter server-b885cfcd-019e-62b7-937a-1fa28385d172 b885cfcd-019e-62b7-937a-1fa28385d172}"
> --- PASS: TestRaft_LeaderFail (0.57s)
> === RUN TestRaft_BehindFollower
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.016Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19ced2bd-343d-3f5b-4583-a53c3cf74e12 Address:19ced2bd-343d-3f5b-4583-a53c3cf74e12} {Suffrage:Voter ID:server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 Address:6f6c5846-6819-6e8a-be2c-b22f0f26a603} {Suffrage:Voter ID:server-783b3a96-9e3e-2065-613e-98ee198ee1a2 Address:783b3a96-9e3e-2065-613e-98ee198ee1a2}]"
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.016Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19ced2bd-343d-3f5b-4583-a53c3cf74e12 Address:19ced2bd-343d-3f5b-4583-a53c3cf74e12} {Suffrage:Voter ID:server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 Address:6f6c5846-6819-6e8a-be2c-b22f0f26a603} {Suffrage:Voter ID:server-783b3a96-9e3e-2065-613e-98ee198ee1a2 Address:783b3a96-9e3e-2065-613e-98ee198ee1a2}]"
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.016Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: entering follower state: follower="Node at 19ced2bd-343d-3f5b-4583-a53c3cf74e12 [Follower]" leader=
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.016Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: entering follower state: follower="Node at 19ced2bd-343d-3f5b-4583-a53c3cf74e12 [Follower]" leader=
> testLoggerAdapter verbose: server-6f6c5846-6819-6e8a-be2c-b22f0f26a603: 2020-12-08T22:13:15.016Z [INFO] server-6f6c5846-6819-6e8a-be2c-b22f0f26a603: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19ced2bd-343d-3f5b-4583-a53c3cf74e12 Address:19ced2bd-343d-3f5b-4583-a53c3cf74e12} {Suffrage:Voter ID:server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 Address:6f6c5846-6819-6e8a-be2c-b22f0f26a603} {Suffrage:Voter ID:server-783b3a96-9e3e-2065-613e-98ee198ee1a2 Address:783b3a96-9e3e-2065-613e-98ee198ee1a2}]"
> testing.go:149: server-6f6c5846-6819-6e8a-be2c-b22f0f26a603: 2020-12-08T22:13:15.016Z [INFO] server-6f6c5846-6819-6e8a-be2c-b22f0f26a603: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19ced2bd-343d-3f5b-4583-a53c3cf74e12 Address:19ced2bd-343d-3f5b-4583-a53c3cf74e12} {Suffrage:Voter ID:server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 Address:6f6c5846-6819-6e8a-be2c-b22f0f26a603} {Suffrage:Voter ID:server-783b3a96-9e3e-2065-613e-98ee198ee1a2 Address:783b3a96-9e3e-2065-613e-98ee198ee1a2}]"
> testLoggerAdapter verbose: server-6f6c5846-6819-6e8a-be2c-b22f0f26a603: 2020-12-08T22:13:15.016Z [INFO] server-6f6c5846-6819-6e8a-be2c-b22f0f26a603: entering follower state: follower="Node at 6f6c5846-6819-6e8a-be2c-b22f0f26a603 [Follower]" leader=
> testing.go:149: server-6f6c5846-6819-6e8a-be2c-b22f0f26a603: 2020-12-08T22:13:15.016Z [INFO] server-6f6c5846-6819-6e8a-be2c-b22f0f26a603: entering follower state: follower="Node at 6f6c5846-6819-6e8a-be2c-b22f0f26a603 [Follower]" leader=
> testLoggerAdapter verbose: server-783b3a96-9e3e-2065-613e-98ee198ee1a2: 2020-12-08T22:13:15.016Z [INFO] server-783b3a96-9e3e-2065-613e-98ee198ee1a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19ced2bd-343d-3f5b-4583-a53c3cf74e12 Address:19ced2bd-343d-3f5b-4583-a53c3cf74e12} {Suffrage:Voter ID:server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 Address:6f6c5846-6819-6e8a-be2c-b22f0f26a603} {Suffrage:Voter ID:server-783b3a96-9e3e-2065-613e-98ee198ee1a2 Address:783b3a96-9e3e-2065-613e-98ee198ee1a2}]"
> testing.go:149: server-783b3a96-9e3e-2065-613e-98ee198ee1a2: 2020-12-08T22:13:15.016Z [INFO] server-783b3a96-9e3e-2065-613e-98ee198ee1a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19ced2bd-343d-3f5b-4583-a53c3cf74e12 Address:19ced2bd-343d-3f5b-4583-a53c3cf74e12} {Suffrage:Voter ID:server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 Address:6f6c5846-6819-6e8a-be2c-b22f0f26a603} {Suffrage:Voter ID:server-783b3a96-9e3e-2065-613e-98ee198ee1a2 Address:783b3a96-9e3e-2065-613e-98ee198ee1a2}]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:15.017Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:15.017Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-783b3a96-9e3e-2065-613e-98ee198ee1a2: 2020-12-08T22:13:15.017Z [INFO] server-783b3a96-9e3e-2065-613e-98ee198ee1a2: entering follower state: follower="Node at 783b3a96-9e3e-2065-613e-98ee198ee1a2 [Follower]" leader=
> testing.go:149: server-783b3a96-9e3e-2065-613e-98ee198ee1a2: 2020-12-08T22:13:15.017Z [INFO] server-783b3a96-9e3e-2065-613e-98ee198ee1a2: entering follower state: follower="Node at 783b3a96-9e3e-2065-613e-98ee198ee1a2 [Follower]" leader=
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [WARN] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [WARN] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: entering candidate state: node="Node at 19ced2bd-343d-3f5b-4583-a53c3cf74e12 [Candidate]" term=2
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: entering candidate state: node="Node at 19ced2bd-343d-3f5b-4583-a53c3cf74e12 [Candidate]" term=2
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: election won: tally=2
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: election won: tally=2
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: entering leader state: leader="Node at 19ced2bd-343d-3f5b-4583-a53c3cf74e12 [Leader]"
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: entering leader state: leader="Node at 19ced2bd-343d-3f5b-4583-a53c3cf74e12 [Leader]"
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: added peer, starting replication: peer=server-6f6c5846-6819-6e8a-be2c-b22f0f26a603
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: added peer, starting replication: peer=server-6f6c5846-6819-6e8a-be2c-b22f0f26a603
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: added peer, starting replication: peer=server-783b3a96-9e3e-2065-613e-98ee198ee1a2
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: added peer, starting replication: peer=server-783b3a96-9e3e-2065-613e-98ee198ee1a2
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: pipelining replication: peer="{Voter server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 6f6c5846-6819-6e8a-be2c-b22f0f26a603}"
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: pipelining replication: peer="{Voter server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 6f6c5846-6819-6e8a-be2c-b22f0f26a603}"
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: pipelining replication: peer="{Voter server-783b3a96-9e3e-2065-613e-98ee198ee1a2 783b3a96-9e3e-2065-613e-98ee198ee1a2}"
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.093Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: pipelining replication: peer="{Voter server-783b3a96-9e3e-2065-613e-98ee198ee1a2 783b3a96-9e3e-2065-613e-98ee198ee1a2}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:15.198Z [INFO] stable state for Leader reached at 2020-12-08 22:13:15.093652509 +0000 UTC m=+6.650813112 (1 nodes), 76.603972ms from start of poll, 77.013749ms from cluster start. Timeout at 2020-12-08 22:13:15.198772177 +0000 UTC m=+6.755932739, 105.119627ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:15.198Z [INFO] stable state for Leader reached at 2020-12-08 22:13:15.093652509 +0000 UTC m=+6.650813112 (1 nodes), 76.603972ms from start of poll, 77.013749ms from cluster start. Timeout at 2020-12-08 22:13:15.198772177 +0000 UTC m=+6.755932739, 105.119627ms after stability
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:15.198Z [INFO] starting stability test: raft-state=Follower
> testing.go:149: cluster: 2020-12-08T22:13:15.198Z [INFO] starting stability test: raft-state=Follower
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:15.303Z [INFO] stable state for Follower reached at 2020-12-08 22:13:15.198844573 +0000 UTC m=+6.756005143 (2 nodes), 920ns from start of poll, 182.20578ms from cluster start. Timeout at 2020-12-08 22:13:15.303922284 +0000 UTC m=+6.861082854, 105.077711ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:15.303Z [INFO] stable state for Follower reached at 2020-12-08 22:13:15.198844573 +0000 UTC m=+6.756005143 (2 nodes), 920ns from start of poll, 182.20578ms from cluster start. Timeout at 2020-12-08 22:13:15.303922284 +0000 UTC m=+6.861082854, 105.077711ms after stability
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.304Z [ERROR] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: failed to pipeline appendEntries: peer="{Voter server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 6f6c5846-6819-6e8a-be2c-b22f0f26a603}" error="append pipeline closed"
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.304Z [ERROR] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: failed to pipeline appendEntries: peer="{Voter server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 6f6c5846-6819-6e8a-be2c-b22f0f26a603}" error="append pipeline closed"
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.304Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: aborting pipeline replication: peer="{Voter server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 6f6c5846-6819-6e8a-be2c-b22f0f26a603}"
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.304Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: aborting pipeline replication: peer="{Voter server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 6f6c5846-6819-6e8a-be2c-b22f0f26a603}"
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.304Z [ERROR] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: failed to appendEntries to: peer="{Voter server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 6f6c5846-6819-6e8a-be2c-b22f0f26a603}" error="failed to connect to peer: 6f6c5846-6819-6e8a-be2c-b22f0f26a603"
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.304Z [ERROR] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: failed to appendEntries to: peer="{Voter server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 6f6c5846-6819-6e8a-be2c-b22f0f26a603}" error="failed to connect to peer: 6f6c5846-6819-6e8a-be2c-b22f0f26a603"
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.304Z [ERROR] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: failed to heartbeat to: peer=6f6c5846-6819-6e8a-be2c-b22f0f26a603 error="failed to connect to peer: 6f6c5846-6819-6e8a-be2c-b22f0f26a603"
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.304Z [ERROR] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: failed to heartbeat to: peer=6f6c5846-6819-6e8a-be2c-b22f0f26a603 error="failed to connect to peer: 6f6c5846-6819-6e8a-be2c-b22f0f26a603"
> raft_test.go:420: [INFO] Finished apply without behind follower
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.314Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: pipelining replication: peer="{Voter server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 6f6c5846-6819-6e8a-be2c-b22f0f26a603}"
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.314Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: pipelining replication: peer="{Voter server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 6f6c5846-6819-6e8a-be2c-b22f0f26a603}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:15.327Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:15.327Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:15.433Z [INFO] stable state for Leader reached at 2020-12-08 22:13:15.327878015 +0000 UTC m=+6.885038592 (1 nodes), 787ns from start of poll, 311.239229ms from cluster start. Timeout at 2020-12-08 22:13:15.433023198 +0000 UTC m=+6.990183766, 105.145174ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:15.433Z [INFO] stable state for Leader reached at 2020-12-08 22:13:15.327878015 +0000 UTC m=+6.885038592 (1 nodes), 787ns from start of poll, 311.239229ms from cluster start. Timeout at 2020-12-08 22:13:15.433023198 +0000 UTC m=+6.990183766, 105.145174ms after stability
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.433Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: aborting pipeline replication: peer="{Voter server-783b3a96-9e3e-2065-613e-98ee198ee1a2 783b3a96-9e3e-2065-613e-98ee198ee1a2}"
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.433Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: aborting pipeline replication: peer="{Voter server-783b3a96-9e3e-2065-613e-98ee198ee1a2 783b3a96-9e3e-2065-613e-98ee198ee1a2}"
> testLoggerAdapter verbose: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.433Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: aborting pipeline replication: peer="{Voter server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 6f6c5846-6819-6e8a-be2c-b22f0f26a603}"
> testing.go:149: server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: 2020-12-08T22:13:15.433Z [INFO] server-19ced2bd-343d-3f5b-4583-a53c3cf74e12: aborting pipeline replication: peer="{Voter server-6f6c5846-6819-6e8a-be2c-b22f0f26a603 6f6c5846-6819-6e8a-be2c-b22f0f26a603}"
> --- PASS: TestRaft_BehindFollower (0.42s)
> === RUN TestRaft_ApplyNonLeader
> testLoggerAdapter verbose: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.434Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081 Address:5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081} {Suffrage:Voter ID:server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482 Address:7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482} {Suffrage:Voter ID:server-23ad3f20-3176-5ff3-252c-7f808ae074ab Address:23ad3f20-3176-5ff3-252c-7f808ae074ab}]"
> testing.go:149: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.434Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081 Address:5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081} {Suffrage:Voter ID:server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482 Address:7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482} {Suffrage:Voter ID:server-23ad3f20-3176-5ff3-252c-7f808ae074ab Address:23ad3f20-3176-5ff3-252c-7f808ae074ab}]"
> testLoggerAdapter verbose: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.435Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: entering follower state: follower="Node at 5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081 [Follower]" leader=
> testing.go:149: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.435Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: entering follower state: follower="Node at 5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081 [Follower]" leader=
> testLoggerAdapter verbose: server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482: 2020-12-08T22:13:15.435Z [INFO] server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081 Address:5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081} {Suffrage:Voter ID:server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482 Address:7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482} {Suffrage:Voter ID:server-23ad3f20-3176-5ff3-252c-7f808ae074ab Address:23ad3f20-3176-5ff3-252c-7f808ae074ab}]"
> testing.go:149: server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482: 2020-12-08T22:13:15.435Z [INFO] server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081 Address:5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081} {Suffrage:Voter ID:server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482 Address:7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482} {Suffrage:Voter ID:server-23ad3f20-3176-5ff3-252c-7f808ae074ab Address:23ad3f20-3176-5ff3-252c-7f808ae074ab}]"
> testLoggerAdapter verbose: server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482: 2020-12-08T22:13:15.435Z [INFO] server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482: entering follower state: follower="Node at 7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482 [Follower]" leader=
> testing.go:149: server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482: 2020-12-08T22:13:15.435Z [INFO] server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482: entering follower state: follower="Node at 7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482 [Follower]" leader=
> testLoggerAdapter verbose: server-23ad3f20-3176-5ff3-252c-7f808ae074ab: 2020-12-08T22:13:15.435Z [INFO] server-23ad3f20-3176-5ff3-252c-7f808ae074ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081 Address:5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081} {Suffrage:Voter ID:server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482 Address:7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482} {Suffrage:Voter ID:server-23ad3f20-3176-5ff3-252c-7f808ae074ab Address:23ad3f20-3176-5ff3-252c-7f808ae074ab}]"
> testing.go:149: server-23ad3f20-3176-5ff3-252c-7f808ae074ab: 2020-12-08T22:13:15.435Z [INFO] server-23ad3f20-3176-5ff3-252c-7f808ae074ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081 Address:5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081} {Suffrage:Voter ID:server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482 Address:7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482} {Suffrage:Voter ID:server-23ad3f20-3176-5ff3-252c-7f808ae074ab Address:23ad3f20-3176-5ff3-252c-7f808ae074ab}]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:15.435Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:15.435Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-23ad3f20-3176-5ff3-252c-7f808ae074ab: 2020-12-08T22:13:15.435Z [INFO] server-23ad3f20-3176-5ff3-252c-7f808ae074ab: entering follower state: follower="Node at 23ad3f20-3176-5ff3-252c-7f808ae074ab [Follower]" leader=
> testing.go:149: server-23ad3f20-3176-5ff3-252c-7f808ae074ab: 2020-12-08T22:13:15.435Z [INFO] server-23ad3f20-3176-5ff3-252c-7f808ae074ab: entering follower state: follower="Node at 23ad3f20-3176-5ff3-252c-7f808ae074ab [Follower]" leader=
> testLoggerAdapter verbose: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [WARN] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [WARN] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: entering candidate state: node="Node at 5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081 [Candidate]" term=2
> testing.go:149: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: entering candidate state: node="Node at 5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081 [Candidate]" term=2
> testLoggerAdapter verbose: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: election won: tally=2
> testing.go:149: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: election won: tally=2
> testLoggerAdapter verbose: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: entering leader state: leader="Node at 5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081 [Leader]"
> testing.go:149: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: entering leader state: leader="Node at 5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081 [Leader]"
> testLoggerAdapter verbose: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: added peer, starting replication: peer=server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482
> testing.go:149: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: added peer, starting replication: peer=server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482
> testLoggerAdapter verbose: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: added peer, starting replication: peer=server-23ad3f20-3176-5ff3-252c-7f808ae074ab
> testing.go:149: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: added peer, starting replication: peer=server-23ad3f20-3176-5ff3-252c-7f808ae074ab
> testLoggerAdapter verbose: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: pipelining replication: peer="{Voter server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482 7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482}"
> testing.go:149: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: pipelining replication: peer="{Voter server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482 7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482}"
> testLoggerAdapter verbose: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: pipelining replication: peer="{Voter server-23ad3f20-3176-5ff3-252c-7f808ae074ab 23ad3f20-3176-5ff3-252c-7f808ae074ab}"
> testing.go:149: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.488Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: pipelining replication: peer="{Voter server-23ad3f20-3176-5ff3-252c-7f808ae074ab 23ad3f20-3176-5ff3-252c-7f808ae074ab}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:15.593Z [INFO] stable state for Leader reached at 2020-12-08 22:13:15.488734454 +0000 UTC m=+7.045895012 (1 nodes), 53.461929ms from start of poll, 53.898583ms from cluster start. Timeout at 2020-12-08 22:13:15.593793321 +0000 UTC m=+7.150953887, 105.058875ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:15.593Z [INFO] stable state for Leader reached at 2020-12-08 22:13:15.488734454 +0000 UTC m=+7.045895012 (1 nodes), 53.461929ms from start of poll, 53.898583ms from cluster start. Timeout at 2020-12-08 22:13:15.593793321 +0000 UTC m=+7.150953887, 105.058875ms after stability
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:15.593Z [INFO] starting stability test: raft-state=Follower
> testing.go:149: cluster: 2020-12-08T22:13:15.593Z [INFO] starting stability test: raft-state=Follower
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:15.698Z [INFO] stable state for Follower reached at 2020-12-08 22:13:15.593886272 +0000 UTC m=+7.151046866 (2 nodes), 1.28µs from start of poll, 159.050437ms from cluster start. Timeout at 2020-12-08 22:13:15.698943678 +0000 UTC m=+7.256104248, 105.057382ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:15.698Z [INFO] stable state for Follower reached at 2020-12-08 22:13:15.593886272 +0000 UTC m=+7.151046866 (2 nodes), 1.28µs from start of poll, 159.050437ms from cluster start. Timeout at 2020-12-08 22:13:15.698943678 +0000 UTC m=+7.256104248, 105.057382ms after stability
> testLoggerAdapter verbose: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.699Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: aborting pipeline replication: peer="{Voter server-23ad3f20-3176-5ff3-252c-7f808ae074ab 23ad3f20-3176-5ff3-252c-7f808ae074ab}"
> testing.go:149: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.699Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: aborting pipeline replication: peer="{Voter server-23ad3f20-3176-5ff3-252c-7f808ae074ab 23ad3f20-3176-5ff3-252c-7f808ae074ab}"
> testLoggerAdapter verbose: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.699Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: aborting pipeline replication: peer="{Voter server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482 7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482}"
> testing.go:149: server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: 2020-12-08T22:13:15.699Z [INFO] server-5c73acdc-5036-8b5e-a9fd-e3e1a7ed6081: aborting pipeline replication: peer="{Voter server-7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482 7ae1e5a2-b9d2-c2e7-62b9-0e40b9808482}"
> --- PASS: TestRaft_ApplyNonLeader (0.27s)
> === RUN TestRaft_ApplyConcurrent
> testLoggerAdapter verbose: server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc: 2020-12-08T22:13:15.700Z [INFO] server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc Address:ff9a89e8-f27a-e9ab-c743-15e2a11f73cc} {Suffrage:Voter ID:server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d Address:83f4f5ab-413c-7a10-4da3-4dd50d2e648d} {Suffrage:Voter ID:server-5732fea8-2ac6-e525-4fc8-d618d39d0e50 Address:5732fea8-2ac6-e525-4fc8-d618d39d0e50}]"
> testing.go:149: server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc: 2020-12-08T22:13:15.700Z [INFO] server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc Address:ff9a89e8-f27a-e9ab-c743-15e2a11f73cc} {Suffrage:Voter ID:server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d Address:83f4f5ab-413c-7a10-4da3-4dd50d2e648d} {Suffrage:Voter ID:server-5732fea8-2ac6-e525-4fc8-d618d39d0e50 Address:5732fea8-2ac6-e525-4fc8-d618d39d0e50}]"
> testLoggerAdapter verbose: server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc: 2020-12-08T22:13:15.700Z [INFO] server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc: entering follower state: follower="Node at ff9a89e8-f27a-e9ab-c743-15e2a11f73cc [Follower]" leader=
> testing.go:149: server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc: 2020-12-08T22:13:15.700Z [INFO] server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc: entering follower state: follower="Node at ff9a89e8-f27a-e9ab-c743-15e2a11f73cc [Follower]" leader=
> testLoggerAdapter verbose: server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d: 2020-12-08T22:13:15.701Z [INFO] server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc Address:ff9a89e8-f27a-e9ab-c743-15e2a11f73cc} {Suffrage:Voter ID:server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d Address:83f4f5ab-413c-7a10-4da3-4dd50d2e648d} {Suffrage:Voter ID:server-5732fea8-2ac6-e525-4fc8-d618d39d0e50 Address:5732fea8-2ac6-e525-4fc8-d618d39d0e50}]"
> testing.go:149: server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d: 2020-12-08T22:13:15.701Z [INFO] server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc Address:ff9a89e8-f27a-e9ab-c743-15e2a11f73cc} {Suffrage:Voter ID:server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d Address:83f4f5ab-413c-7a10-4da3-4dd50d2e648d} {Suffrage:Voter ID:server-5732fea8-2ac6-e525-4fc8-d618d39d0e50 Address:5732fea8-2ac6-e525-4fc8-d618d39d0e50}]"
> testLoggerAdapter verbose: server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d: 2020-12-08T22:13:15.701Z [INFO] server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d: entering follower state: follower="Node at 83f4f5ab-413c-7a10-4da3-4dd50d2e648d [Follower]" leader=
> testing.go:149: server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d: 2020-12-08T22:13:15.701Z [INFO] server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d: entering follower state: follower="Node at 83f4f5ab-413c-7a10-4da3-4dd50d2e648d [Follower]" leader=
> testLoggerAdapter verbose: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.702Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc Address:ff9a89e8-f27a-e9ab-c743-15e2a11f73cc} {Suffrage:Voter ID:server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d Address:83f4f5ab-413c-7a10-4da3-4dd50d2e648d} {Suffrage:Voter ID:server-5732fea8-2ac6-e525-4fc8-d618d39d0e50 Address:5732fea8-2ac6-e525-4fc8-d618d39d0e50}]"
> testing.go:149: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.702Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc Address:ff9a89e8-f27a-e9ab-c743-15e2a11f73cc} {Suffrage:Voter ID:server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d Address:83f4f5ab-413c-7a10-4da3-4dd50d2e648d} {Suffrage:Voter ID:server-5732fea8-2ac6-e525-4fc8-d618d39d0e50 Address:5732fea8-2ac6-e525-4fc8-d618d39d0e50}]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:15.702Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:15.702Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.702Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: entering follower state: follower="Node at 5732fea8-2ac6-e525-4fc8-d618d39d0e50 [Follower]" leader=
> testing.go:149: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.702Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: entering follower state: follower="Node at 5732fea8-2ac6-e525-4fc8-d618d39d0e50 [Follower]" leader=
> testLoggerAdapter verbose: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [WARN] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [WARN] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: entering candidate state: node="Node at 5732fea8-2ac6-e525-4fc8-d618d39d0e50 [Candidate]" term=2
> testing.go:149: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: entering candidate state: node="Node at 5732fea8-2ac6-e525-4fc8-d618d39d0e50 [Candidate]" term=2
> testLoggerAdapter verbose: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: election won: tally=2
> testing.go:149: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: election won: tally=2
> testLoggerAdapter verbose: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: entering leader state: leader="Node at 5732fea8-2ac6-e525-4fc8-d618d39d0e50 [Leader]"
> testing.go:149: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: entering leader state: leader="Node at 5732fea8-2ac6-e525-4fc8-d618d39d0e50 [Leader]"
> testLoggerAdapter verbose: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: added peer, starting replication: peer=server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc
> testing.go:149: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: added peer, starting replication: peer=server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc
> testLoggerAdapter verbose: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: added peer, starting replication: peer=server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d
> testing.go:149: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: added peer, starting replication: peer=server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d
> testLoggerAdapter verbose: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: pipelining replication: peer="{Voter server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc ff9a89e8-f27a-e9ab-c743-15e2a11f73cc}"
> testing.go:149: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: pipelining replication: peer="{Voter server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc ff9a89e8-f27a-e9ab-c743-15e2a11f73cc}"
> testLoggerAdapter verbose: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: pipelining replication: peer="{Voter server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d 83f4f5ab-413c-7a10-4da3-4dd50d2e648d}"
> testing.go:149: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:15.823Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: pipelining replication: peer="{Voter server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d 83f4f5ab-413c-7a10-4da3-4dd50d2e648d}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:16.028Z [INFO] stable state for Leader reached at 2020-12-08 22:13:15.823628503 +0000 UTC m=+7.380789073 (1 nodes), 120.909953ms from start of poll, 122.990297ms from cluster start. Timeout at 2020-12-08 22:13:16.028731435 +0000 UTC m=+7.585892003, 205.10293ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:16.028Z [INFO] stable state for Leader reached at 2020-12-08 22:13:15.823628503 +0000 UTC m=+7.380789073 (1 nodes), 120.909953ms from start of poll, 122.990297ms from cluster start. Timeout at 2020-12-08 22:13:16.028731435 +0000 UTC m=+7.585892003, 205.10293ms after stability
> testLoggerAdapter verbose: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:16.040Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: aborting pipeline replication: peer="{Voter server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d 83f4f5ab-413c-7a10-4da3-4dd50d2e648d}"
> testing.go:149: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:16.040Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: aborting pipeline replication: peer="{Voter server-83f4f5ab-413c-7a10-4da3-4dd50d2e648d 83f4f5ab-413c-7a10-4da3-4dd50d2e648d}"
> testLoggerAdapter verbose: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:16.040Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: aborting pipeline replication: peer="{Voter server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc ff9a89e8-f27a-e9ab-c743-15e2a11f73cc}"
> testing.go:149: server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: 2020-12-08T22:13:16.040Z [INFO] server-5732fea8-2ac6-e525-4fc8-d618d39d0e50: aborting pipeline replication: peer="{Voter server-ff9a89e8-f27a-e9ab-c743-15e2a11f73cc ff9a89e8-f27a-e9ab-c743-15e2a11f73cc}"
> --- PASS: TestRaft_ApplyConcurrent (0.34s)
> === RUN TestRaft_ApplyConcurrent_Timeout
> testLoggerAdapter verbose: server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: 2020-12-08T22:13:16.041Z [INFO] server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d873d7d2-ffbd-4310-f0b5-b63289310ad8 Address:d873d7d2-ffbd-4310-f0b5-b63289310ad8}]"
> testing.go:149: server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: 2020-12-08T22:13:16.041Z [INFO] server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d873d7d2-ffbd-4310-f0b5-b63289310ad8 Address:d873d7d2-ffbd-4310-f0b5-b63289310ad8}]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:16.041Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:16.041Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: 2020-12-08T22:13:16.041Z [INFO] server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: entering follower state: follower="Node at d873d7d2-ffbd-4310-f0b5-b63289310ad8 [Follower]" leader=
> testing.go:149: server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: 2020-12-08T22:13:16.041Z [INFO] server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: entering follower state: follower="Node at d873d7d2-ffbd-4310-f0b5-b63289310ad8 [Follower]" leader=
> testLoggerAdapter verbose: server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: 2020-12-08T22:13:16.160Z [WARN] server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: 2020-12-08T22:13:16.160Z [WARN] server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: 2020-12-08T22:13:16.160Z [INFO] server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: entering candidate state: node="Node at d873d7d2-ffbd-4310-f0b5-b63289310ad8 [Candidate]" term=2
> testing.go:149: server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: 2020-12-08T22:13:16.160Z [INFO] server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: entering candidate state: node="Node at d873d7d2-ffbd-4310-f0b5-b63289310ad8 [Candidate]" term=2
> testLoggerAdapter verbose: server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: 2020-12-08T22:13:16.160Z [INFO] server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: election won: tally=1
> testing.go:149: server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: 2020-12-08T22:13:16.160Z [INFO] server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: election won: tally=1
> testLoggerAdapter verbose: server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: 2020-12-08T22:13:16.160Z [INFO] server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: entering leader state: leader="Node at d873d7d2-ffbd-4310-f0b5-b63289310ad8 [Leader]"
> testing.go:149: server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: 2020-12-08T22:13:16.160Z [INFO] server-d873d7d2-ffbd-4310-f0b5-b63289310ad8: entering leader state: leader="Node at d873d7d2-ffbd-4310-f0b5-b63289310ad8 [Leader]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:16.361Z [INFO] stable state for Leader reached at 2020-12-08 22:13:16.160568409 +0000 UTC m=+7.717728977 (1 nodes), 119.001856ms from start of poll, 119.172519ms from cluster start. Timeout at 2020-12-08 22:13:16.361667751 +0000 UTC m=+7.918828319, 201.099342ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:16.361Z [INFO] stable state for Leader reached at 2020-12-08 22:13:16.160568409 +0000 UTC m=+7.717728977 (1 nodes), 119.001856ms from start of poll, 119.172519ms from cluster start. Timeout at 2020-12-08 22:13:16.361667751 +0000 UTC m=+7.918828319, 201.099342ms after stability
> --- PASS: TestRaft_ApplyConcurrent_Timeout (0.32s)
> === RUN TestRaft_JoinNode
> testLoggerAdapter verbose: server-9338dcca-7438-79f4-a869-f589c6d68a8f: 2020-12-08T22:13:16.362Z [INFO] server-9338dcca-7438-79f4-a869-f589c6d68a8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9338dcca-7438-79f4-a869-f589c6d68a8f Address:9338dcca-7438-79f4-a869-f589c6d68a8f} {Suffrage:Voter ID:server-f9925231-de33-36a1-6ad0-0863e17f0fed Address:f9925231-de33-36a1-6ad0-0863e17f0fed}]"
> testing.go:149: server-9338dcca-7438-79f4-a869-f589c6d68a8f: 2020-12-08T22:13:16.362Z [INFO] server-9338dcca-7438-79f4-a869-f589c6d68a8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9338dcca-7438-79f4-a869-f589c6d68a8f Address:9338dcca-7438-79f4-a869-f589c6d68a8f} {Suffrage:Voter ID:server-f9925231-de33-36a1-6ad0-0863e17f0fed Address:f9925231-de33-36a1-6ad0-0863e17f0fed}]"
> testLoggerAdapter verbose: server-9338dcca-7438-79f4-a869-f589c6d68a8f: 2020-12-08T22:13:16.362Z [INFO] server-9338dcca-7438-79f4-a869-f589c6d68a8f: entering follower state: follower="Node at 9338dcca-7438-79f4-a869-f589c6d68a8f [Follower]" leader=
> testing.go:149: server-9338dcca-7438-79f4-a869-f589c6d68a8f: 2020-12-08T22:13:16.362Z [INFO] server-9338dcca-7438-79f4-a869-f589c6d68a8f: entering follower state: follower="Node at 9338dcca-7438-79f4-a869-f589c6d68a8f [Follower]" leader=
> testLoggerAdapter verbose: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.362Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9338dcca-7438-79f4-a869-f589c6d68a8f Address:9338dcca-7438-79f4-a869-f589c6d68a8f} {Suffrage:Voter ID:server-f9925231-de33-36a1-6ad0-0863e17f0fed Address:f9925231-de33-36a1-6ad0-0863e17f0fed}]"
> testing.go:149: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.362Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9338dcca-7438-79f4-a869-f589c6d68a8f Address:9338dcca-7438-79f4-a869-f589c6d68a8f} {Suffrage:Voter ID:server-f9925231-de33-36a1-6ad0-0863e17f0fed Address:f9925231-de33-36a1-6ad0-0863e17f0fed}]"
> testLoggerAdapter verbose: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.362Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: entering follower state: follower="Node at f9925231-de33-36a1-6ad0-0863e17f0fed [Follower]" leader=
> testing.go:149: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.362Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: entering follower state: follower="Node at f9925231-de33-36a1-6ad0-0863e17f0fed [Follower]" leader=
> testLoggerAdapter verbose: server-5c75af35-be69-e987-13d6-49d9c85af40b: 2020-12-08T22:13:16.363Z [INFO] server-5c75af35-be69-e987-13d6-49d9c85af40b: initial configuration: index=0 servers=[]
> testing.go:149: server-5c75af35-be69-e987-13d6-49d9c85af40b: 2020-12-08T22:13:16.363Z [INFO] server-5c75af35-be69-e987-13d6-49d9c85af40b: initial configuration: index=0 servers=[]
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:16.363Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:16.363Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-5c75af35-be69-e987-13d6-49d9c85af40b: 2020-12-08T22:13:16.363Z [INFO] server-5c75af35-be69-e987-13d6-49d9c85af40b: entering follower state: follower="Node at 5c75af35-be69-e987-13d6-49d9c85af40b [Follower]" leader=
> testing.go:149: server-5c75af35-be69-e987-13d6-49d9c85af40b: 2020-12-08T22:13:16.363Z [INFO] server-5c75af35-be69-e987-13d6-49d9c85af40b: entering follower state: follower="Node at 5c75af35-be69-e987-13d6-49d9c85af40b [Follower]" leader=
> testLoggerAdapter verbose: server-5c75af35-be69-e987-13d6-49d9c85af40b: 2020-12-08T22:13:16.434Z [WARN] server-5c75af35-be69-e987-13d6-49d9c85af40b: no known peers, aborting election
> testing.go:149: server-5c75af35-be69-e987-13d6-49d9c85af40b: 2020-12-08T22:13:16.434Z [WARN] server-5c75af35-be69-e987-13d6-49d9c85af40b: no known peers, aborting election
> testLoggerAdapter verbose: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.456Z [WARN] server-f9925231-de33-36a1-6ad0-0863e17f0fed: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.456Z [WARN] server-f9925231-de33-36a1-6ad0-0863e17f0fed: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.456Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: entering candidate state: node="Node at f9925231-de33-36a1-6ad0-0863e17f0fed [Candidate]" term=2
> testing.go:149: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.456Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: entering candidate state: node="Node at f9925231-de33-36a1-6ad0-0863e17f0fed [Candidate]" term=2
> testLoggerAdapter verbose: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.456Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: election won: tally=2
> testing.go:149: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.456Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: election won: tally=2
> testLoggerAdapter verbose: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.456Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: entering leader state: leader="Node at f9925231-de33-36a1-6ad0-0863e17f0fed [Leader]"
> testing.go:149: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.456Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: entering leader state: leader="Node at f9925231-de33-36a1-6ad0-0863e17f0fed [Leader]"
> testLoggerAdapter verbose: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.456Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: added peer, starting replication: peer=server-9338dcca-7438-79f4-a869-f589c6d68a8f
> testing.go:149: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.456Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: added peer, starting replication: peer=server-9338dcca-7438-79f4-a869-f589c6d68a8f
> testLoggerAdapter verbose: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.456Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: pipelining replication: peer="{Voter server-9338dcca-7438-79f4-a869-f589c6d68a8f 9338dcca-7438-79f4-a869-f589c6d68a8f}"
> testing.go:149: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.456Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: pipelining replication: peer="{Voter server-9338dcca-7438-79f4-a869-f589c6d68a8f 9338dcca-7438-79f4-a869-f589c6d68a8f}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:16.562Z [INFO] stable state for Leader reached at 2020-12-08 22:13:16.456915472 +0000 UTC m=+8.014076042 (1 nodes), 93.654061ms from start of poll, 94.277625ms from cluster start. Timeout at 2020-12-08 22:13:16.562000311 +0000 UTC m=+8.119160875, 105.084833ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:16.562Z [INFO] stable state for Leader reached at 2020-12-08 22:13:16.456915472 +0000 UTC m=+8.014076042 (1 nodes), 93.654061ms from start of poll, 94.277625ms from cluster start. Timeout at 2020-12-08 22:13:16.562000311 +0000 UTC m=+8.119160875, 105.084833ms after stability
> testLoggerAdapter verbose: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.562Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: updating configuration: command=AddStaging server-id=server-5c75af35-be69-e987-13d6-49d9c85af40b server-addr=5c75af35-be69-e987-13d6-49d9c85af40b servers="[{Suffrage:Voter ID:server-9338dcca-7438-79f4-a869-f589c6d68a8f Address:9338dcca-7438-79f4-a869-f589c6d68a8f} {Suffrage:Voter ID:server-f9925231-de33-36a1-6ad0-0863e17f0fed Address:f9925231-de33-36a1-6ad0-0863e17f0fed} {Suffrage:Voter ID:server-5c75af35-be69-e987-13d6-49d9c85af40b Address:5c75af35-be69-e987-13d6-49d9c85af40b}]"
> testing.go:149: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.562Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: updating configuration: command=AddStaging server-id=server-5c75af35-be69-e987-13d6-49d9c85af40b server-addr=5c75af35-be69-e987-13d6-49d9c85af40b servers="[{Suffrage:Voter ID:server-9338dcca-7438-79f4-a869-f589c6d68a8f Address:9338dcca-7438-79f4-a869-f589c6d68a8f} {Suffrage:Voter ID:server-f9925231-de33-36a1-6ad0-0863e17f0fed Address:f9925231-de33-36a1-6ad0-0863e17f0fed} {Suffrage:Voter ID:server-5c75af35-be69-e987-13d6-49d9c85af40b Address:5c75af35-be69-e987-13d6-49d9c85af40b}]"
> testLoggerAdapter verbose: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.562Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: added peer, starting replication: peer=server-5c75af35-be69-e987-13d6-49d9c85af40b
> testing.go:149: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.562Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: added peer, starting replication: peer=server-5c75af35-be69-e987-13d6-49d9c85af40b
> testLoggerAdapter verbose: server-5c75af35-be69-e987-13d6-49d9c85af40b: 2020-12-08T22:13:16.562Z [WARN] server-5c75af35-be69-e987-13d6-49d9c85af40b: failed to get previous log: previous-index=3 last-index=0 error="log not found"
> testing.go:149: server-5c75af35-be69-e987-13d6-49d9c85af40b: 2020-12-08T22:13:16.562Z [WARN] server-5c75af35-be69-e987-13d6-49d9c85af40b: failed to get previous log: previous-index=3 last-index=0 error="log not found"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:16.562Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.562Z [WARN] server-f9925231-de33-36a1-6ad0-0863e17f0fed: appendEntries rejected, sending older logs: peer="{Voter server-5c75af35-be69-e987-13d6-49d9c85af40b 5c75af35-be69-e987-13d6-49d9c85af40b}" next=1
> testing.go:149: cluster: 2020-12-08T22:13:16.562Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.562Z [WARN] server-f9925231-de33-36a1-6ad0-0863e17f0fed: appendEntries rejected, sending older logs: peer="{Voter server-5c75af35-be69-e987-13d6-49d9c85af40b 5c75af35-be69-e987-13d6-49d9c85af40b}" next=1
> testLoggerAdapter verbose: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.562Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: pipelining replication: peer="{Voter server-5c75af35-be69-e987-13d6-49d9c85af40b 5c75af35-be69-e987-13d6-49d9c85af40b}"
> testing.go:149: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.562Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: pipelining replication: peer="{Voter server-5c75af35-be69-e987-13d6-49d9c85af40b 5c75af35-be69-e987-13d6-49d9c85af40b}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:16.667Z [INFO] stable state for Leader reached at 2020-12-08 22:13:16.562286642 +0000 UTC m=+8.119447216 (1 nodes), 625ns from start of poll, 199.648799ms from cluster start. Timeout at 2020-12-08 22:13:16.667384599 +0000 UTC m=+8.224545166, 105.09795ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:16.667Z [INFO] stable state for Leader reached at 2020-12-08 22:13:16.562286642 +0000 UTC m=+8.119447216 (1 nodes), 625ns from start of poll, 199.648799ms from cluster start. Timeout at 2020-12-08 22:13:16.667384599 +0000 UTC m=+8.224545166, 105.09795ms after stability
> testLoggerAdapter verbose: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.667Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: aborting pipeline replication: peer="{Voter server-9338dcca-7438-79f4-a869-f589c6d68a8f 9338dcca-7438-79f4-a869-f589c6d68a8f}"
> testing.go:149: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.667Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: aborting pipeline replication: peer="{Voter server-9338dcca-7438-79f4-a869-f589c6d68a8f 9338dcca-7438-79f4-a869-f589c6d68a8f}"
> testLoggerAdapter verbose: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.667Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: aborting pipeline replication: peer="{Voter server-5c75af35-be69-e987-13d6-49d9c85af40b 5c75af35-be69-e987-13d6-49d9c85af40b}"
> testing.go:149: server-f9925231-de33-36a1-6ad0-0863e17f0fed: 2020-12-08T22:13:16.667Z [INFO] server-f9925231-de33-36a1-6ad0-0863e17f0fed: aborting pipeline replication: peer="{Voter server-5c75af35-be69-e987-13d6-49d9c85af40b 5c75af35-be69-e987-13d6-49d9c85af40b}"
> --- PASS: TestRaft_JoinNode (0.31s)
> === RUN TestRaft_JoinNode_ConfigStore
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.668Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28 Address:c5a5c794-32f5-dd61-6a96-3b1309fc1d28}]"
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.668Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28 Address:c5a5c794-32f5-dd61-6a96-3b1309fc1d28}]"
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.668Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: entering follower state: follower="Node at c5a5c794-32f5-dd61-6a96-3b1309fc1d28 [Follower]" leader=
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.668Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: entering follower state: follower="Node at c5a5c794-32f5-dd61-6a96-3b1309fc1d28 [Follower]" leader=
> testLoggerAdapter verbose: server-9bb9c775-6a74-8664-1a37-920916003d93: 2020-12-08T22:13:16.669Z [INFO] server-9bb9c775-6a74-8664-1a37-920916003d93: initial configuration: index=0 servers=[]
> testing.go:149: server-9bb9c775-6a74-8664-1a37-920916003d93: 2020-12-08T22:13:16.669Z [INFO] server-9bb9c775-6a74-8664-1a37-920916003d93: initial configuration: index=0 servers=[]
> testLoggerAdapter verbose: server-9bb9c775-6a74-8664-1a37-920916003d93: 2020-12-08T22:13:16.669Z [INFO] server-9bb9c775-6a74-8664-1a37-920916003d93: entering follower state: follower="Node at 9bb9c775-6a74-8664-1a37-920916003d93 [Follower]" leader=
> testing.go:149: server-9bb9c775-6a74-8664-1a37-920916003d93: 2020-12-08T22:13:16.669Z [INFO] server-9bb9c775-6a74-8664-1a37-920916003d93: entering follower state: follower="Node at 9bb9c775-6a74-8664-1a37-920916003d93 [Follower]" leader=
> testLoggerAdapter verbose: server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: 2020-12-08T22:13:16.669Z [INFO] server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: initial configuration: index=0 servers=[]
> testing.go:149: server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: 2020-12-08T22:13:16.669Z [INFO] server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: initial configuration: index=0 servers=[]
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:16.669Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:16.669Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: 2020-12-08T22:13:16.669Z [INFO] server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: entering follower state: follower="Node at ef2ebefb-e313-7954-5be7-d38b1a7eb947 [Follower]" leader=
> testing.go:149: server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: 2020-12-08T22:13:16.669Z [INFO] server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: entering follower state: follower="Node at ef2ebefb-e313-7954-5be7-d38b1a7eb947 [Follower]" leader=
> testLoggerAdapter verbose: server-9bb9c775-6a74-8664-1a37-920916003d93: 2020-12-08T22:13:16.727Z [WARN] server-9bb9c775-6a74-8664-1a37-920916003d93: no known peers, aborting election
> testing.go:149: server-9bb9c775-6a74-8664-1a37-920916003d93: 2020-12-08T22:13:16.727Z [WARN] server-9bb9c775-6a74-8664-1a37-920916003d93: no known peers, aborting election
> testLoggerAdapter verbose: server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: 2020-12-08T22:13:16.756Z [WARN] server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: no known peers, aborting election
> testing.go:149: server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: 2020-12-08T22:13:16.756Z [WARN] server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: no known peers, aborting election
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.763Z [WARN] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.763Z [WARN] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.763Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: entering candidate state: node="Node at c5a5c794-32f5-dd61-6a96-3b1309fc1d28 [Candidate]" term=2
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.763Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: entering candidate state: node="Node at c5a5c794-32f5-dd61-6a96-3b1309fc1d28 [Candidate]" term=2
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.763Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: election won: tally=1
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.763Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: election won: tally=1
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.763Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: entering leader state: leader="Node at c5a5c794-32f5-dd61-6a96-3b1309fc1d28 [Leader]"
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.763Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: entering leader state: leader="Node at c5a5c794-32f5-dd61-6a96-3b1309fc1d28 [Leader]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:16.869Z [INFO] stable state for Leader reached at 2020-12-08 22:13:16.763928461 +0000 UTC m=+8.321089027 (1 nodes), 94.452563ms from start of poll, 95.278201ms from cluster start. Timeout at 2020-12-08 22:13:16.869040574 +0000 UTC m=+8.426201146, 105.112119ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:16.869Z [INFO] stable state for Leader reached at 2020-12-08 22:13:16.763928461 +0000 UTC m=+8.321089027 (1 nodes), 94.452563ms from start of poll, 95.278201ms from cluster start. Timeout at 2020-12-08 22:13:16.869040574 +0000 UTC m=+8.426201146, 105.112119ms after stability
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.869Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: updating configuration: command=AddStaging server-id=server-9bb9c775-6a74-8664-1a37-920916003d93 server-addr=9bb9c775-6a74-8664-1a37-920916003d93 servers="[{Suffrage:Voter ID:server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28 Address:c5a5c794-32f5-dd61-6a96-3b1309fc1d28} {Suffrage:Voter ID:server-9bb9c775-6a74-8664-1a37-920916003d93 Address:9bb9c775-6a74-8664-1a37-920916003d93}]"
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.869Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: updating configuration: command=AddStaging server-id=server-9bb9c775-6a74-8664-1a37-920916003d93 server-addr=9bb9c775-6a74-8664-1a37-920916003d93 servers="[{Suffrage:Voter ID:server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28 Address:c5a5c794-32f5-dd61-6a96-3b1309fc1d28} {Suffrage:Voter ID:server-9bb9c775-6a74-8664-1a37-920916003d93 Address:9bb9c775-6a74-8664-1a37-920916003d93}]"
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.870Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: added peer, starting replication: peer=server-9bb9c775-6a74-8664-1a37-920916003d93
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.870Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: added peer, starting replication: peer=server-9bb9c775-6a74-8664-1a37-920916003d93
> testLoggerAdapter verbose: server-9bb9c775-6a74-8664-1a37-920916003d93: 2020-12-08T22:13:16.870Z [WARN] server-9bb9c775-6a74-8664-1a37-920916003d93: failed to get previous log: previous-index=3 last-index=0 error="log not found"
> testing.go:149: server-9bb9c775-6a74-8664-1a37-920916003d93: 2020-12-08T22:13:16.870Z [WARN] server-9bb9c775-6a74-8664-1a37-920916003d93: failed to get previous log: previous-index=3 last-index=0 error="log not found"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:16.870Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:16.870Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.870Z [WARN] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: appendEntries rejected, sending older logs: peer="{Voter server-9bb9c775-6a74-8664-1a37-920916003d93 9bb9c775-6a74-8664-1a37-920916003d93}" next=1
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.870Z [WARN] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: appendEntries rejected, sending older logs: peer="{Voter server-9bb9c775-6a74-8664-1a37-920916003d93 9bb9c775-6a74-8664-1a37-920916003d93}" next=1
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.870Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: pipelining replication: peer="{Voter server-9bb9c775-6a74-8664-1a37-920916003d93 9bb9c775-6a74-8664-1a37-920916003d93}"
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.870Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: pipelining replication: peer="{Voter server-9bb9c775-6a74-8664-1a37-920916003d93 9bb9c775-6a74-8664-1a37-920916003d93}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:16.976Z [INFO] stable state for Leader reached at 2020-12-08 22:13:16.870894999 +0000 UTC m=+8.428055565 (1 nodes), 631ns from start of poll, 202.244739ms from cluster start. Timeout at 2020-12-08 22:13:16.976003777 +0000 UTC m=+8.533164414, 105.108849ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:16.976Z [INFO] stable state for Leader reached at 2020-12-08 22:13:16.870894999 +0000 UTC m=+8.428055565 (1 nodes), 631ns from start of poll, 202.244739ms from cluster start. Timeout at 2020-12-08 22:13:16.976003777 +0000 UTC m=+8.533164414, 105.108849ms after stability
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.976Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: updating configuration: command=AddStaging server-id=server-ef2ebefb-e313-7954-5be7-d38b1a7eb947 server-addr=ef2ebefb-e313-7954-5be7-d38b1a7eb947 servers="[{Suffrage:Voter ID:server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28 Address:c5a5c794-32f5-dd61-6a96-3b1309fc1d28} {Suffrage:Voter ID:server-9bb9c775-6a74-8664-1a37-920916003d93 Address:9bb9c775-6a74-8664-1a37-920916003d93} {Suffrage:Voter ID:server-ef2ebefb-e313-7954-5be7-d38b1a7eb947 Address:ef2ebefb-e313-7954-5be7-d38b1a7eb947}]"
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.976Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: updating configuration: command=AddStaging server-id=server-ef2ebefb-e313-7954-5be7-d38b1a7eb947 server-addr=ef2ebefb-e313-7954-5be7-d38b1a7eb947 servers="[{Suffrage:Voter ID:server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28 Address:c5a5c794-32f5-dd61-6a96-3b1309fc1d28} {Suffrage:Voter ID:server-9bb9c775-6a74-8664-1a37-920916003d93 Address:9bb9c775-6a74-8664-1a37-920916003d93} {Suffrage:Voter ID:server-ef2ebefb-e313-7954-5be7-d38b1a7eb947 Address:ef2ebefb-e313-7954-5be7-d38b1a7eb947}]"
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.976Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: added peer, starting replication: peer=server-ef2ebefb-e313-7954-5be7-d38b1a7eb947
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.976Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: added peer, starting replication: peer=server-ef2ebefb-e313-7954-5be7-d38b1a7eb947
> testLoggerAdapter verbose: server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: 2020-12-08T22:13:16.976Z [WARN] server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: failed to get previous log: previous-index=4 last-index=0 error="log not found"
> testing.go:149: server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: 2020-12-08T22:13:16.976Z [WARN] server-ef2ebefb-e313-7954-5be7-d38b1a7eb947: failed to get previous log: previous-index=4 last-index=0 error="log not found"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:16.976Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:16.976Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.976Z [WARN] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: appendEntries rejected, sending older logs: peer="{Voter server-ef2ebefb-e313-7954-5be7-d38b1a7eb947 ef2ebefb-e313-7954-5be7-d38b1a7eb947}" next=1
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.976Z [WARN] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: appendEntries rejected, sending older logs: peer="{Voter server-ef2ebefb-e313-7954-5be7-d38b1a7eb947 ef2ebefb-e313-7954-5be7-d38b1a7eb947}" next=1
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.976Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: pipelining replication: peer="{Voter server-ef2ebefb-e313-7954-5be7-d38b1a7eb947 ef2ebefb-e313-7954-5be7-d38b1a7eb947}"
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:16.976Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: pipelining replication: peer="{Voter server-ef2ebefb-e313-7954-5be7-d38b1a7eb947 ef2ebefb-e313-7954-5be7-d38b1a7eb947}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:17.081Z [INFO] stable state for Leader reached at 2020-12-08 22:13:16.976324154 +0000 UTC m=+8.533484722 (1 nodes), 672ns from start of poll, 307.673896ms from cluster start. Timeout at 2020-12-08 22:13:17.081423844 +0000 UTC m=+8.638584408, 105.099686ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:17.081Z [INFO] stable state for Leader reached at 2020-12-08 22:13:16.976324154 +0000 UTC m=+8.533484722 (1 nodes), 672ns from start of poll, 307.673896ms from cluster start. Timeout at 2020-12-08 22:13:17.081423844 +0000 UTC m=+8.638584408, 105.099686ms after stability
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:17.081Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: aborting pipeline replication: peer="{Voter server-ef2ebefb-e313-7954-5be7-d38b1a7eb947 ef2ebefb-e313-7954-5be7-d38b1a7eb947}"
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:17.081Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: aborting pipeline replication: peer="{Voter server-ef2ebefb-e313-7954-5be7-d38b1a7eb947 ef2ebefb-e313-7954-5be7-d38b1a7eb947}"
> testLoggerAdapter verbose: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:17.081Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: aborting pipeline replication: peer="{Voter server-9bb9c775-6a74-8664-1a37-920916003d93 9bb9c775-6a74-8664-1a37-920916003d93}"
> testing.go:149: server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: 2020-12-08T22:13:17.081Z [INFO] server-c5a5c794-32f5-dd61-6a96-3b1309fc1d28: aborting pipeline replication: peer="{Voter server-9bb9c775-6a74-8664-1a37-920916003d93 9bb9c775-6a74-8664-1a37-920916003d93}"
> --- PASS: TestRaft_JoinNode_ConfigStore (0.41s)
> === RUN TestRaft_RemoveFollower
> testLoggerAdapter verbose: server-98f59485-0a8c-a101-6742-e40e6792784e: 2020-12-08T22:13:17.083Z [INFO] server-98f59485-0a8c-a101-6742-e40e6792784e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98f59485-0a8c-a101-6742-e40e6792784e Address:98f59485-0a8c-a101-6742-e40e6792784e} {Suffrage:Voter ID:server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb Address:6390f18f-f27c-8d36-d4ed-c493ead7cdfb} {Suffrage:Voter ID:server-f18ab3cd-e87f-5b16-b696-443b93f39eca Address:f18ab3cd-e87f-5b16-b696-443b93f39eca}]"
> testing.go:149: server-98f59485-0a8c-a101-6742-e40e6792784e: 2020-12-08T22:13:17.083Z [INFO] server-98f59485-0a8c-a101-6742-e40e6792784e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98f59485-0a8c-a101-6742-e40e6792784e Address:98f59485-0a8c-a101-6742-e40e6792784e} {Suffrage:Voter ID:server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb Address:6390f18f-f27c-8d36-d4ed-c493ead7cdfb} {Suffrage:Voter ID:server-f18ab3cd-e87f-5b16-b696-443b93f39eca Address:f18ab3cd-e87f-5b16-b696-443b93f39eca}]"
> testLoggerAdapter verbose: server-98f59485-0a8c-a101-6742-e40e6792784e: 2020-12-08T22:13:17.083Z [INFO] server-98f59485-0a8c-a101-6742-e40e6792784e: entering follower state: follower="Node at 98f59485-0a8c-a101-6742-e40e6792784e [Follower]" leader=
> testing.go:149: server-98f59485-0a8c-a101-6742-e40e6792784e: 2020-12-08T22:13:17.083Z [INFO] server-98f59485-0a8c-a101-6742-e40e6792784e: entering follower state: follower="Node at 98f59485-0a8c-a101-6742-e40e6792784e [Follower]" leader=
> testLoggerAdapter verbose: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.083Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98f59485-0a8c-a101-6742-e40e6792784e Address:98f59485-0a8c-a101-6742-e40e6792784e} {Suffrage:Voter ID:server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb Address:6390f18f-f27c-8d36-d4ed-c493ead7cdfb} {Suffrage:Voter ID:server-f18ab3cd-e87f-5b16-b696-443b93f39eca Address:f18ab3cd-e87f-5b16-b696-443b93f39eca}]"
> testing.go:149: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.083Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98f59485-0a8c-a101-6742-e40e6792784e Address:98f59485-0a8c-a101-6742-e40e6792784e} {Suffrage:Voter ID:server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb Address:6390f18f-f27c-8d36-d4ed-c493ead7cdfb} {Suffrage:Voter ID:server-f18ab3cd-e87f-5b16-b696-443b93f39eca Address:f18ab3cd-e87f-5b16-b696-443b93f39eca}]"
> testLoggerAdapter verbose: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.083Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: entering follower state: follower="Node at 6390f18f-f27c-8d36-d4ed-c493ead7cdfb [Follower]" leader=
> testing.go:149: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.083Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: entering follower state: follower="Node at 6390f18f-f27c-8d36-d4ed-c493ead7cdfb [Follower]" leader=
> testLoggerAdapter verbose: server-f18ab3cd-e87f-5b16-b696-443b93f39eca: 2020-12-08T22:13:17.083Z [INFO] server-f18ab3cd-e87f-5b16-b696-443b93f39eca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98f59485-0a8c-a101-6742-e40e6792784e Address:98f59485-0a8c-a101-6742-e40e6792784e} {Suffrage:Voter ID:server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb Address:6390f18f-f27c-8d36-d4ed-c493ead7cdfb} {Suffrage:Voter ID:server-f18ab3cd-e87f-5b16-b696-443b93f39eca Address:f18ab3cd-e87f-5b16-b696-443b93f39eca}]"
> testing.go:149: server-f18ab3cd-e87f-5b16-b696-443b93f39eca: 2020-12-08T22:13:17.083Z [INFO] server-f18ab3cd-e87f-5b16-b696-443b93f39eca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98f59485-0a8c-a101-6742-e40e6792784e Address:98f59485-0a8c-a101-6742-e40e6792784e} {Suffrage:Voter ID:server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb Address:6390f18f-f27c-8d36-d4ed-c493ead7cdfb} {Suffrage:Voter ID:server-f18ab3cd-e87f-5b16-b696-443b93f39eca Address:f18ab3cd-e87f-5b16-b696-443b93f39eca}]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:17.083Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:17.083Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-f18ab3cd-e87f-5b16-b696-443b93f39eca: 2020-12-08T22:13:17.083Z [INFO] server-f18ab3cd-e87f-5b16-b696-443b93f39eca: entering follower state: follower="Node at f18ab3cd-e87f-5b16-b696-443b93f39eca [Follower]" leader=
> testing.go:149: server-f18ab3cd-e87f-5b16-b696-443b93f39eca: 2020-12-08T22:13:17.083Z [INFO] server-f18ab3cd-e87f-5b16-b696-443b93f39eca: entering follower state: follower="Node at f18ab3cd-e87f-5b16-b696-443b93f39eca [Follower]" leader=
> testLoggerAdapter verbose: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.143Z [WARN] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.143Z [WARN] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.144Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: entering candidate state: node="Node at 6390f18f-f27c-8d36-d4ed-c493ead7cdfb [Candidate]" term=2
> testing.go:149: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.144Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: entering candidate state: node="Node at 6390f18f-f27c-8d36-d4ed-c493ead7cdfb [Candidate]" term=2
> testLoggerAdapter verbose: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.144Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: election won: tally=2
> testing.go:149: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.144Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: election won: tally=2
> testLoggerAdapter verbose: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.144Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: entering leader state: leader="Node at 6390f18f-f27c-8d36-d4ed-c493ead7cdfb [Leader]"
> testing.go:149: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.144Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: entering leader state: leader="Node at 6390f18f-f27c-8d36-d4ed-c493ead7cdfb [Leader]"
> testLoggerAdapter verbose: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.144Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: added peer, starting replication: peer=server-98f59485-0a8c-a101-6742-e40e6792784e
> testing.go:149: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.144Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: added peer, starting replication: peer=server-98f59485-0a8c-a101-6742-e40e6792784e
> testLoggerAdapter verbose: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.144Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: added peer, starting replication: peer=server-f18ab3cd-e87f-5b16-b696-443b93f39eca
> testing.go:149: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.144Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: added peer, starting replication: peer=server-f18ab3cd-e87f-5b16-b696-443b93f39eca
> testLoggerAdapter verbose: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.144Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: pipelining replication: peer="{Voter server-98f59485-0a8c-a101-6742-e40e6792784e 98f59485-0a8c-a101-6742-e40e6792784e}"
> testing.go:149: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.144Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: pipelining replication: peer="{Voter server-98f59485-0a8c-a101-6742-e40e6792784e 98f59485-0a8c-a101-6742-e40e6792784e}"
> testLoggerAdapter verbose: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.144Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: pipelining replication: peer="{Voter server-f18ab3cd-e87f-5b16-b696-443b93f39eca f18ab3cd-e87f-5b16-b696-443b93f39eca}"
> testing.go:149: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.144Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: pipelining replication: peer="{Voter server-f18ab3cd-e87f-5b16-b696-443b93f39eca f18ab3cd-e87f-5b16-b696-443b93f39eca}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:17.249Z [INFO] stable state for Leader reached at 2020-12-08 22:13:17.144174441 +0000 UTC m=+8.701335002 (1 nodes), 60.305209ms from start of poll, 60.865149ms from cluster start. Timeout at 2020-12-08 22:13:17.249276446 +0000 UTC m=+8.806437016, 105.102014ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:17.249Z [INFO] stable state for Leader reached at 2020-12-08 22:13:17.144174441 +0000 UTC m=+8.701335002 (1 nodes), 60.305209ms from start of poll, 60.865149ms from cluster start. Timeout at 2020-12-08 22:13:17.249276446 +0000 UTC m=+8.806437016, 105.102014ms after stability
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:17.254Z [INFO] starting stability test: raft-state=Follower
> testing.go:149: cluster: 2020-12-08T22:13:17.254Z [INFO] starting stability test: raft-state=Follower
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:17.359Z [INFO] stable state for Follower reached at 2020-12-08 22:13:17.254467242 +0000 UTC m=+8.811627806 (2 nodes), 813ns from start of poll, 171.157953ms from cluster start. Timeout at 2020-12-08 22:13:17.359532612 +0000 UTC m=+8.916693174, 105.065368ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:17.359Z [INFO] stable state for Follower reached at 2020-12-08 22:13:17.254467242 +0000 UTC m=+8.811627806 (2 nodes), 813ns from start of poll, 171.157953ms from cluster start. Timeout at 2020-12-08 22:13:17.359532612 +0000 UTC m=+8.916693174, 105.065368ms after stability
> testLoggerAdapter verbose: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.359Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: updating configuration: command=RemoveServer server-id=server-98f59485-0a8c-a101-6742-e40e6792784e server-addr= servers="[{Suffrage:Voter ID:server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb Address:6390f18f-f27c-8d36-d4ed-c493ead7cdfb} {Suffrage:Voter ID:server-f18ab3cd-e87f-5b16-b696-443b93f39eca Address:f18ab3cd-e87f-5b16-b696-443b93f39eca}]"
> testing.go:149: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.359Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: updating configuration: command=RemoveServer server-id=server-98f59485-0a8c-a101-6742-e40e6792784e server-addr= servers="[{Suffrage:Voter ID:server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb Address:6390f18f-f27c-8d36-d4ed-c493ead7cdfb} {Suffrage:Voter ID:server-f18ab3cd-e87f-5b16-b696-443b93f39eca Address:f18ab3cd-e87f-5b16-b696-443b93f39eca}]"
> testLoggerAdapter verbose: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.359Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: removed peer, stopping replication: peer=server-98f59485-0a8c-a101-6742-e40e6792784e last-index=3
> testing.go:149: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.359Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: removed peer, stopping replication: peer=server-98f59485-0a8c-a101-6742-e40e6792784e last-index=3
> testLoggerAdapter verbose: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.359Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: aborting pipeline replication: peer="{Voter server-98f59485-0a8c-a101-6742-e40e6792784e 98f59485-0a8c-a101-6742-e40e6792784e}"
> testing.go:149: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.359Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: aborting pipeline replication: peer="{Voter server-98f59485-0a8c-a101-6742-e40e6792784e 98f59485-0a8c-a101-6742-e40e6792784e}"
> testLoggerAdapter verbose: server-98f59485-0a8c-a101-6742-e40e6792784e: 2020-12-08T22:13:17.431Z [WARN] server-98f59485-0a8c-a101-6742-e40e6792784e: heartbeat timeout reached, starting election: last-leader=6390f18f-f27c-8d36-d4ed-c493ead7cdfb
> testing.go:149: server-98f59485-0a8c-a101-6742-e40e6792784e: 2020-12-08T22:13:17.431Z [WARN] server-98f59485-0a8c-a101-6742-e40e6792784e: heartbeat timeout reached, starting election: last-leader=6390f18f-f27c-8d36-d4ed-c493ead7cdfb
> testLoggerAdapter verbose: server-98f59485-0a8c-a101-6742-e40e6792784e: 2020-12-08T22:13:17.431Z [INFO] server-98f59485-0a8c-a101-6742-e40e6792784e: entering candidate state: node="Node at 98f59485-0a8c-a101-6742-e40e6792784e [Candidate]" term=3
> testing.go:149: server-98f59485-0a8c-a101-6742-e40e6792784e: 2020-12-08T22:13:17.431Z [INFO] server-98f59485-0a8c-a101-6742-e40e6792784e: entering candidate state: node="Node at 98f59485-0a8c-a101-6742-e40e6792784e [Candidate]" term=3
> testLoggerAdapter verbose: server-f18ab3cd-e87f-5b16-b696-443b93f39eca: 2020-12-08T22:13:17.431Z [WARN] server-f18ab3cd-e87f-5b16-b696-443b93f39eca: rejecting vote request since we have a leader: from=98f59485-0a8c-a101-6742-e40e6792784e leader=6390f18f-f27c-8d36-d4ed-c493ead7cdfb
> testing.go:149: server-f18ab3cd-e87f-5b16-b696-443b93f39eca: 2020-12-08T22:13:17.431Z [WARN] server-f18ab3cd-e87f-5b16-b696-443b93f39eca: rejecting vote request since we have a leader: from=98f59485-0a8c-a101-6742-e40e6792784e leader=6390f18f-f27c-8d36-d4ed-c493ead7cdfb
> testLoggerAdapter verbose: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.431Z [WARN] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: rejecting vote request since we have a leader: from=98f59485-0a8c-a101-6742-e40e6792784e leader=6390f18f-f27c-8d36-d4ed-c493ead7cdfb
> testing.go:149: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.431Z [WARN] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: rejecting vote request since we have a leader: from=98f59485-0a8c-a101-6742-e40e6792784e leader=6390f18f-f27c-8d36-d4ed-c493ead7cdfb
> testLoggerAdapter verbose: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.464Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: aborting pipeline replication: peer="{Voter server-f18ab3cd-e87f-5b16-b696-443b93f39eca f18ab3cd-e87f-5b16-b696-443b93f39eca}"
> testing.go:149: server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: 2020-12-08T22:13:17.464Z [INFO] server-6390f18f-f27c-8d36-d4ed-c493ead7cdfb: aborting pipeline replication: peer="{Voter server-f18ab3cd-e87f-5b16-b696-443b93f39eca f18ab3cd-e87f-5b16-b696-443b93f39eca}"
> --- PASS: TestRaft_RemoveFollower (0.38s)
> === RUN TestRaft_RemoveLeader
> testLoggerAdapter verbose: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.466Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31af14f7-d3bf-e562-cebb-ddc265dfa27b Address:31af14f7-d3bf-e562-cebb-ddc265dfa27b} {Suffrage:Voter ID:server-d122db75-13a4-8434-a7ee-287f8d9b28c7 Address:d122db75-13a4-8434-a7ee-287f8d9b28c7} {Suffrage:Voter ID:server-23a44568-c72a-ca56-b42b-2cad679ade4a Address:23a44568-c72a-ca56-b42b-2cad679ade4a}]"
> testing.go:149: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.466Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31af14f7-d3bf-e562-cebb-ddc265dfa27b Address:31af14f7-d3bf-e562-cebb-ddc265dfa27b} {Suffrage:Voter ID:server-d122db75-13a4-8434-a7ee-287f8d9b28c7 Address:d122db75-13a4-8434-a7ee-287f8d9b28c7} {Suffrage:Voter ID:server-23a44568-c72a-ca56-b42b-2cad679ade4a Address:23a44568-c72a-ca56-b42b-2cad679ade4a}]"
> testLoggerAdapter verbose: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.466Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: entering follower state: follower="Node at 31af14f7-d3bf-e562-cebb-ddc265dfa27b [Follower]" leader=
> testing.go:149: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.466Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: entering follower state: follower="Node at 31af14f7-d3bf-e562-cebb-ddc265dfa27b [Follower]" leader=
> testLoggerAdapter verbose: server-d122db75-13a4-8434-a7ee-287f8d9b28c7: 2020-12-08T22:13:17.466Z [INFO] server-d122db75-13a4-8434-a7ee-287f8d9b28c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31af14f7-d3bf-e562-cebb-ddc265dfa27b Address:31af14f7-d3bf-e562-cebb-ddc265dfa27b} {Suffrage:Voter ID:server-d122db75-13a4-8434-a7ee-287f8d9b28c7 Address:d122db75-13a4-8434-a7ee-287f8d9b28c7} {Suffrage:Voter ID:server-23a44568-c72a-ca56-b42b-2cad679ade4a Address:23a44568-c72a-ca56-b42b-2cad679ade4a}]"
> testing.go:149: server-d122db75-13a4-8434-a7ee-287f8d9b28c7: 2020-12-08T22:13:17.466Z [INFO] server-d122db75-13a4-8434-a7ee-287f8d9b28c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31af14f7-d3bf-e562-cebb-ddc265dfa27b Address:31af14f7-d3bf-e562-cebb-ddc265dfa27b} {Suffrage:Voter ID:server-d122db75-13a4-8434-a7ee-287f8d9b28c7 Address:d122db75-13a4-8434-a7ee-287f8d9b28c7} {Suffrage:Voter ID:server-23a44568-c72a-ca56-b42b-2cad679ade4a Address:23a44568-c72a-ca56-b42b-2cad679ade4a}]"
> testLoggerAdapter verbose: server-d122db75-13a4-8434-a7ee-287f8d9b28c7: 2020-12-08T22:13:17.466Z [INFO] server-d122db75-13a4-8434-a7ee-287f8d9b28c7: entering follower state: follower="Node at d122db75-13a4-8434-a7ee-287f8d9b28c7 [Follower]" leader=
> testing.go:149: server-d122db75-13a4-8434-a7ee-287f8d9b28c7: 2020-12-08T22:13:17.466Z [INFO] server-d122db75-13a4-8434-a7ee-287f8d9b28c7: entering follower state: follower="Node at d122db75-13a4-8434-a7ee-287f8d9b28c7 [Follower]" leader=
> testLoggerAdapter verbose: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.466Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31af14f7-d3bf-e562-cebb-ddc265dfa27b Address:31af14f7-d3bf-e562-cebb-ddc265dfa27b} {Suffrage:Voter ID:server-d122db75-13a4-8434-a7ee-287f8d9b28c7 Address:d122db75-13a4-8434-a7ee-287f8d9b28c7} {Suffrage:Voter ID:server-23a44568-c72a-ca56-b42b-2cad679ade4a Address:23a44568-c72a-ca56-b42b-2cad679ade4a}]"
> testing.go:149: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.466Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31af14f7-d3bf-e562-cebb-ddc265dfa27b Address:31af14f7-d3bf-e562-cebb-ddc265dfa27b} {Suffrage:Voter ID:server-d122db75-13a4-8434-a7ee-287f8d9b28c7 Address:d122db75-13a4-8434-a7ee-287f8d9b28c7} {Suffrage:Voter ID:server-23a44568-c72a-ca56-b42b-2cad679ade4a Address:23a44568-c72a-ca56-b42b-2cad679ade4a}]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:17.466Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:17.466Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.466Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: entering follower state: follower="Node at 23a44568-c72a-ca56-b42b-2cad679ade4a [Follower]" leader=
> testing.go:149: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.466Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: entering follower state: follower="Node at 23a44568-c72a-ca56-b42b-2cad679ade4a [Follower]" leader=
> testLoggerAdapter verbose: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.533Z [WARN] server-23a44568-c72a-ca56-b42b-2cad679ade4a: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.533Z [WARN] server-23a44568-c72a-ca56-b42b-2cad679ade4a: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.533Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: entering candidate state: node="Node at 23a44568-c72a-ca56-b42b-2cad679ade4a [Candidate]" term=2
> testing.go:149: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.533Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: entering candidate state: node="Node at 23a44568-c72a-ca56-b42b-2cad679ade4a [Candidate]" term=2
> testLoggerAdapter verbose: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.533Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: election won: tally=2
> testing.go:149: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.533Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: election won: tally=2
> testLoggerAdapter verbose: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.533Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: entering leader state: leader="Node at 23a44568-c72a-ca56-b42b-2cad679ade4a [Leader]"
> testing.go:149: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.533Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: entering leader state: leader="Node at 23a44568-c72a-ca56-b42b-2cad679ade4a [Leader]"
> testLoggerAdapter verbose: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.533Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: added peer, starting replication: peer=server-31af14f7-d3bf-e562-cebb-ddc265dfa27b
> testing.go:149: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.533Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: added peer, starting replication: peer=server-31af14f7-d3bf-e562-cebb-ddc265dfa27b
> testLoggerAdapter verbose: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.534Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: added peer, starting replication: peer=server-d122db75-13a4-8434-a7ee-287f8d9b28c7
> testing.go:149: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.534Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: added peer, starting replication: peer=server-d122db75-13a4-8434-a7ee-287f8d9b28c7
> testLoggerAdapter verbose: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.534Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: pipelining replication: peer="{Voter server-31af14f7-d3bf-e562-cebb-ddc265dfa27b 31af14f7-d3bf-e562-cebb-ddc265dfa27b}"
> testing.go:149: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.534Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: pipelining replication: peer="{Voter server-31af14f7-d3bf-e562-cebb-ddc265dfa27b 31af14f7-d3bf-e562-cebb-ddc265dfa27b}"
> testLoggerAdapter verbose: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.534Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: pipelining replication: peer="{Voter server-d122db75-13a4-8434-a7ee-287f8d9b28c7 d122db75-13a4-8434-a7ee-287f8d9b28c7}"
> testing.go:149: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.534Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: pipelining replication: peer="{Voter server-d122db75-13a4-8434-a7ee-287f8d9b28c7 d122db75-13a4-8434-a7ee-287f8d9b28c7}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:17.639Z [INFO] stable state for Leader reached at 2020-12-08 22:13:17.53397454 +0000 UTC m=+9.091135112 (1 nodes), 67.062885ms from start of poll, 67.504092ms from cluster start. Timeout at 2020-12-08 22:13:17.639045117 +0000 UTC m=+9.196205683, 105.070571ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:17.639Z [INFO] stable state for Leader reached at 2020-12-08 22:13:17.53397454 +0000 UTC m=+9.091135112 (1 nodes), 67.062885ms from start of poll, 67.504092ms from cluster start. Timeout at 2020-12-08 22:13:17.639045117 +0000 UTC m=+9.196205683, 105.070571ms after stability
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:17.644Z [INFO] starting stability test: raft-state=Follower
> testing.go:149: cluster: 2020-12-08T22:13:17.644Z [INFO] starting stability test: raft-state=Follower
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:17.749Z [INFO] stable state for Follower reached at 2020-12-08 22:13:17.644239401 +0000 UTC m=+9.201399969 (2 nodes), 860ns from start of poll, 177.768949ms from cluster start. Timeout at 2020-12-08 22:13:17.74933517 +0000 UTC m=+9.306495716, 105.095747ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:17.749Z [INFO] stable state for Follower reached at 2020-12-08 22:13:17.644239401 +0000 UTC m=+9.201399969 (2 nodes), 860ns from start of poll, 177.768949ms from cluster start. Timeout at 2020-12-08 22:13:17.74933517 +0000 UTC m=+9.306495716, 105.095747ms after stability
> testLoggerAdapter verbose: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.749Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: updating configuration: command=RemoveServer server-id=server-23a44568-c72a-ca56-b42b-2cad679ade4a server-addr= servers="[{Suffrage:Voter ID:server-31af14f7-d3bf-e562-cebb-ddc265dfa27b Address:31af14f7-d3bf-e562-cebb-ddc265dfa27b} {Suffrage:Voter ID:server-d122db75-13a4-8434-a7ee-287f8d9b28c7 Address:d122db75-13a4-8434-a7ee-287f8d9b28c7}]"
> testing.go:149: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.749Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: updating configuration: command=RemoveServer server-id=server-23a44568-c72a-ca56-b42b-2cad679ade4a server-addr= servers="[{Suffrage:Voter ID:server-31af14f7-d3bf-e562-cebb-ddc265dfa27b Address:31af14f7-d3bf-e562-cebb-ddc265dfa27b} {Suffrage:Voter ID:server-d122db75-13a4-8434-a7ee-287f8d9b28c7 Address:d122db75-13a4-8434-a7ee-287f8d9b28c7}]"
> testLoggerAdapter verbose: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.749Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: removed ourself, shutting down
> testing.go:149: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.749Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: removed ourself, shutting down
> testLoggerAdapter verbose: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.749Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: aborting pipeline replication: peer="{Voter server-d122db75-13a4-8434-a7ee-287f8d9b28c7 d122db75-13a4-8434-a7ee-287f8d9b28c7}"
> testing.go:149: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.749Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: aborting pipeline replication: peer="{Voter server-d122db75-13a4-8434-a7ee-287f8d9b28c7 d122db75-13a4-8434-a7ee-287f8d9b28c7}"
> testLoggerAdapter verbose: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.749Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: aborting pipeline replication: peer="{Voter server-31af14f7-d3bf-e562-cebb-ddc265dfa27b 31af14f7-d3bf-e562-cebb-ddc265dfa27b}"
> testing.go:149: server-23a44568-c72a-ca56-b42b-2cad679ade4a: 2020-12-08T22:13:17.749Z [INFO] server-23a44568-c72a-ca56-b42b-2cad679ade4a: aborting pipeline replication: peer="{Voter server-31af14f7-d3bf-e562-cebb-ddc265dfa27b 31af14f7-d3bf-e562-cebb-ddc265dfa27b}"
> testLoggerAdapter verbose: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.844Z [WARN] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: heartbeat timeout reached, starting election: last-leader=23a44568-c72a-ca56-b42b-2cad679ade4a
> testing.go:149: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.844Z [WARN] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: heartbeat timeout reached, starting election: last-leader=23a44568-c72a-ca56-b42b-2cad679ade4a
> testLoggerAdapter verbose: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.844Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: entering candidate state: node="Node at 31af14f7-d3bf-e562-cebb-ddc265dfa27b [Candidate]" term=3
> testing.go:149: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.844Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: entering candidate state: node="Node at 31af14f7-d3bf-e562-cebb-ddc265dfa27b [Candidate]" term=3
> testLoggerAdapter verbose: server-d122db75-13a4-8434-a7ee-287f8d9b28c7: 2020-12-08T22:13:17.844Z [WARN] server-d122db75-13a4-8434-a7ee-287f8d9b28c7: rejecting vote request since we have a leader: from=31af14f7-d3bf-e562-cebb-ddc265dfa27b leader=23a44568-c72a-ca56-b42b-2cad679ade4a
> testing.go:149: server-d122db75-13a4-8434-a7ee-287f8d9b28c7: 2020-12-08T22:13:17.844Z [WARN] server-d122db75-13a4-8434-a7ee-287f8d9b28c7: rejecting vote request since we have a leader: from=31af14f7-d3bf-e562-cebb-ddc265dfa27b leader=23a44568-c72a-ca56-b42b-2cad679ade4a
> testLoggerAdapter verbose: server-d122db75-13a4-8434-a7ee-287f8d9b28c7: 2020-12-08T22:13:17.859Z [WARN] server-d122db75-13a4-8434-a7ee-287f8d9b28c7: heartbeat timeout reached, starting election: last-leader=23a44568-c72a-ca56-b42b-2cad679ade4a
> testing.go:149: server-d122db75-13a4-8434-a7ee-287f8d9b28c7: 2020-12-08T22:13:17.859Z [WARN] server-d122db75-13a4-8434-a7ee-287f8d9b28c7: heartbeat timeout reached, starting election: last-leader=23a44568-c72a-ca56-b42b-2cad679ade4a
> testLoggerAdapter verbose: server-d122db75-13a4-8434-a7ee-287f8d9b28c7: 2020-12-08T22:13:17.859Z [INFO] server-d122db75-13a4-8434-a7ee-287f8d9b28c7: entering candidate state: node="Node at d122db75-13a4-8434-a7ee-287f8d9b28c7 [Candidate]" term=3
> testing.go:149: server-d122db75-13a4-8434-a7ee-287f8d9b28c7: 2020-12-08T22:13:17.859Z [INFO] server-d122db75-13a4-8434-a7ee-287f8d9b28c7: entering candidate state: node="Node at d122db75-13a4-8434-a7ee-287f8d9b28c7 [Candidate]" term=3
> testLoggerAdapter verbose: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.859Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: duplicate requestVote for same term: term=3
> testing.go:149: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.859Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: duplicate requestVote for same term: term=3
> testLoggerAdapter verbose: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.900Z [WARN] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: Election timeout reached, restarting election
> testing.go:149: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.900Z [WARN] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: Election timeout reached, restarting election
> testLoggerAdapter verbose: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.900Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: entering candidate state: node="Node at 31af14f7-d3bf-e562-cebb-ddc265dfa27b [Candidate]" term=4
> testing.go:149: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.900Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: entering candidate state: node="Node at 31af14f7-d3bf-e562-cebb-ddc265dfa27b [Candidate]" term=4
> testLoggerAdapter verbose: server-d122db75-13a4-8434-a7ee-287f8d9b28c7: 2020-12-08T22:13:17.900Z [INFO] server-d122db75-13a4-8434-a7ee-287f8d9b28c7: entering follower state: follower="Node at d122db75-13a4-8434-a7ee-287f8d9b28c7 [Follower]" leader=
> testing.go:149: server-d122db75-13a4-8434-a7ee-287f8d9b28c7: 2020-12-08T22:13:17.900Z [INFO] server-d122db75-13a4-8434-a7ee-287f8d9b28c7: entering follower state: follower="Node at d122db75-13a4-8434-a7ee-287f8d9b28c7 [Follower]" leader=
> testLoggerAdapter verbose: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.900Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: election won: tally=2
> testing.go:149: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.900Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: election won: tally=2
> testLoggerAdapter verbose: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.900Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: entering leader state: leader="Node at 31af14f7-d3bf-e562-cebb-ddc265dfa27b [Leader]"
> testing.go:149: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.900Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: entering leader state: leader="Node at 31af14f7-d3bf-e562-cebb-ddc265dfa27b [Leader]"
> testLoggerAdapter verbose: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.900Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: added peer, starting replication: peer=server-d122db75-13a4-8434-a7ee-287f8d9b28c7
> testing.go:149: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.900Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: added peer, starting replication: peer=server-d122db75-13a4-8434-a7ee-287f8d9b28c7
> testLoggerAdapter verbose: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.900Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: pipelining replication: peer="{Voter server-d122db75-13a4-8434-a7ee-287f8d9b28c7 d122db75-13a4-8434-a7ee-287f8d9b28c7}"
> testing.go:149: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:17.900Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: pipelining replication: peer="{Voter server-d122db75-13a4-8434-a7ee-287f8d9b28c7 d122db75-13a4-8434-a7ee-287f8d9b28c7}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:17.959Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:17.959Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:18.065Z [INFO] stable state for Leader reached at 2020-12-08 22:13:17.959932238 +0000 UTC m=+9.517092800 (1 nodes), 31.145µs from start of poll, 493.46178ms from cluster start. Timeout at 2020-12-08 22:13:18.065039853 +0000 UTC m=+9.622200419, 105.107619ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:18.065Z [INFO] stable state for Leader reached at 2020-12-08 22:13:17.959932238 +0000 UTC m=+9.517092800 (1 nodes), 31.145µs from start of poll, 493.46178ms from cluster start. Timeout at 2020-12-08 22:13:18.065039853 +0000 UTC m=+9.622200419, 105.107619ms after stability
> testLoggerAdapter verbose: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:18.065Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: aborting pipeline replication: peer="{Voter server-d122db75-13a4-8434-a7ee-287f8d9b28c7 d122db75-13a4-8434-a7ee-287f8d9b28c7}"
> testing.go:149: server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: 2020-12-08T22:13:18.065Z [INFO] server-31af14f7-d3bf-e562-cebb-ddc265dfa27b: aborting pipeline replication: peer="{Voter server-d122db75-13a4-8434-a7ee-287f8d9b28c7 d122db75-13a4-8434-a7ee-287f8d9b28c7}"
> --- PASS: TestRaft_RemoveLeader (0.60s)
> === RUN TestRaft_RemoveLeader_NoShutdown
> testLoggerAdapter verbose: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.066Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83e13462-3df4-c52d-1e67-4006c27566eb Address:83e13462-3df4-c52d-1e67-4006c27566eb} {Suffrage:Voter ID:server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b Address:82b73bdb-8a93-345a-f2ca-d45a2e2e246b} {Suffrage:Voter ID:server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef Address:08dfaa51-9538-f3d7-c25b-f6856dbb5eef}]"
> testing.go:149: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.066Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83e13462-3df4-c52d-1e67-4006c27566eb Address:83e13462-3df4-c52d-1e67-4006c27566eb} {Suffrage:Voter ID:server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b Address:82b73bdb-8a93-345a-f2ca-d45a2e2e246b} {Suffrage:Voter ID:server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef Address:08dfaa51-9538-f3d7-c25b-f6856dbb5eef}]"
> testLoggerAdapter verbose: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.066Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: entering follower state: follower="Node at 83e13462-3df4-c52d-1e67-4006c27566eb [Follower]" leader=
> testing.go:149: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.066Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: entering follower state: follower="Node at 83e13462-3df4-c52d-1e67-4006c27566eb [Follower]" leader=
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.067Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83e13462-3df4-c52d-1e67-4006c27566eb Address:83e13462-3df4-c52d-1e67-4006c27566eb} {Suffrage:Voter ID:server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b Address:82b73bdb-8a93-345a-f2ca-d45a2e2e246b} {Suffrage:Voter ID:server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef Address:08dfaa51-9538-f3d7-c25b-f6856dbb5eef}]"
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.067Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83e13462-3df4-c52d-1e67-4006c27566eb Address:83e13462-3df4-c52d-1e67-4006c27566eb} {Suffrage:Voter ID:server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b Address:82b73bdb-8a93-345a-f2ca-d45a2e2e246b} {Suffrage:Voter ID:server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef Address:08dfaa51-9538-f3d7-c25b-f6856dbb5eef}]"
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.067Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: entering follower state: follower="Node at 82b73bdb-8a93-345a-f2ca-d45a2e2e246b [Follower]" leader=
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.067Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: entering follower state: follower="Node at 82b73bdb-8a93-345a-f2ca-d45a2e2e246b [Follower]" leader=
> testLoggerAdapter verbose: server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: 2020-12-08T22:13:18.067Z [INFO] server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83e13462-3df4-c52d-1e67-4006c27566eb Address:83e13462-3df4-c52d-1e67-4006c27566eb} {Suffrage:Voter ID:server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b Address:82b73bdb-8a93-345a-f2ca-d45a2e2e246b} {Suffrage:Voter ID:server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef Address:08dfaa51-9538-f3d7-c25b-f6856dbb5eef}]"
> testing.go:149: server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: 2020-12-08T22:13:18.067Z [INFO] server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83e13462-3df4-c52d-1e67-4006c27566eb Address:83e13462-3df4-c52d-1e67-4006c27566eb} {Suffrage:Voter ID:server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b Address:82b73bdb-8a93-345a-f2ca-d45a2e2e246b} {Suffrage:Voter ID:server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef Address:08dfaa51-9538-f3d7-c25b-f6856dbb5eef}]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:18.067Z [INFO] starting stability test: raft-state=Follower
> testing.go:149: cluster: 2020-12-08T22:13:18.067Z [INFO] starting stability test: raft-state=Follower
> testLoggerAdapter verbose: server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: 2020-12-08T22:13:18.067Z [INFO] server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: entering follower state: follower="Node at 08dfaa51-9538-f3d7-c25b-f6856dbb5eef [Follower]" leader=
> testing.go:149: server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: 2020-12-08T22:13:18.067Z [INFO] server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: entering follower state: follower="Node at 08dfaa51-9538-f3d7-c25b-f6856dbb5eef [Follower]" leader=
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [WARN] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [WARN] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: entering candidate state: node="Node at 82b73bdb-8a93-345a-f2ca-d45a2e2e246b [Candidate]" term=2
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: entering candidate state: node="Node at 82b73bdb-8a93-345a-f2ca-d45a2e2e246b [Candidate]" term=2
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: election won: tally=2
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: election won: tally=2
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: entering leader state: leader="Node at 82b73bdb-8a93-345a-f2ca-d45a2e2e246b [Leader]"
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: entering leader state: leader="Node at 82b73bdb-8a93-345a-f2ca-d45a2e2e246b [Leader]"
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: added peer, starting replication: peer=server-83e13462-3df4-c52d-1e67-4006c27566eb
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: added peer, starting replication: peer=server-83e13462-3df4-c52d-1e67-4006c27566eb
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: added peer, starting replication: peer=server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: added peer, starting replication: peer=server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: pipelining replication: peer="{Voter server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef 08dfaa51-9538-f3d7-c25b-f6856dbb5eef}"
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: pipelining replication: peer="{Voter server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef 08dfaa51-9538-f3d7-c25b-f6856dbb5eef}"
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: pipelining replication: peer="{Voter server-83e13462-3df4-c52d-1e67-4006c27566eb 83e13462-3df4-c52d-1e67-4006c27566eb}"
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.120Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: pipelining replication: peer="{Voter server-83e13462-3df4-c52d-1e67-4006c27566eb 83e13462-3df4-c52d-1e67-4006c27566eb}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:18.225Z [INFO] stable state for Follower reached at 2020-12-08 22:13:18.12036507 +0000 UTC m=+9.677525642 (2 nodes), 53.033718ms from start of poll, 53.554737ms from cluster start. Timeout at 2020-12-08 22:13:18.225477208 +0000 UTC m=+9.782637776, 105.112134ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:18.225Z [INFO] stable state for Follower reached at 2020-12-08 22:13:18.12036507 +0000 UTC m=+9.677525642 (2 nodes), 53.033718ms from start of poll, 53.554737ms from cluster start. Timeout at 2020-12-08 22:13:18.225477208 +0000 UTC m=+9.782637776, 105.112134ms after stability
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:18.225Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:18.225Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:18.330Z [INFO] stable state for Leader reached at 2020-12-08 22:13:18.225549183 +0000 UTC m=+9.782709747 (1 nodes), 559ns from start of poll, 158.738842ms from cluster start. Timeout at 2020-12-08 22:13:18.330599515 +0000 UTC m=+9.887760077, 105.05033ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:18.330Z [INFO] stable state for Leader reached at 2020-12-08 22:13:18.225549183 +0000 UTC m=+9.782709747 (1 nodes), 559ns from start of poll, 158.738842ms from cluster start. Timeout at 2020-12-08 22:13:18.330599515 +0000 UTC m=+9.887760077, 105.05033ms after stability
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.332Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: updating configuration: command=RemoveServer server-id=server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b server-addr= servers="[{Suffrage:Voter ID:server-83e13462-3df4-c52d-1e67-4006c27566eb Address:83e13462-3df4-c52d-1e67-4006c27566eb} {Suffrage:Voter ID:server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef Address:08dfaa51-9538-f3d7-c25b-f6856dbb5eef}]"
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.332Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: updating configuration: command=RemoveServer server-id=server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b server-addr= servers="[{Suffrage:Voter ID:server-83e13462-3df4-c52d-1e67-4006c27566eb Address:83e13462-3df4-c52d-1e67-4006c27566eb} {Suffrage:Voter ID:server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef Address:08dfaa51-9538-f3d7-c25b-f6856dbb5eef}]"
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.333Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: removed ourself, transitioning to follower
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.333Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: removed ourself, transitioning to follower
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.333Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: entering follower state: follower="Node at 82b73bdb-8a93-345a-f2ca-d45a2e2e246b [Follower]" leader=
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.333Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: entering follower state: follower="Node at 82b73bdb-8a93-345a-f2ca-d45a2e2e246b [Follower]" leader=
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.333Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: aborting pipeline replication: peer="{Voter server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef 08dfaa51-9538-f3d7-c25b-f6856dbb5eef}"
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.333Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: aborting pipeline replication: peer="{Voter server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef 08dfaa51-9538-f3d7-c25b-f6856dbb5eef}"
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.333Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: aborting pipeline replication: peer="{Voter server-83e13462-3df4-c52d-1e67-4006c27566eb 83e13462-3df4-c52d-1e67-4006c27566eb}"
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.333Z [INFO] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: aborting pipeline replication: peer="{Voter server-83e13462-3df4-c52d-1e67-4006c27566eb 83e13462-3df4-c52d-1e67-4006c27566eb}"
> testLoggerAdapter verbose: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.388Z [WARN] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: not part of stable configuration, aborting election
> testing.go:149: server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: 2020-12-08T22:13:18.388Z [WARN] server-82b73bdb-8a93-345a-f2ca-d45a2e2e246b: not part of stable configuration, aborting election
> testLoggerAdapter verbose: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.405Z [WARN] server-83e13462-3df4-c52d-1e67-4006c27566eb: heartbeat timeout reached, starting election: last-leader=82b73bdb-8a93-345a-f2ca-d45a2e2e246b
> testing.go:149: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.405Z [WARN] server-83e13462-3df4-c52d-1e67-4006c27566eb: heartbeat timeout reached, starting election: last-leader=82b73bdb-8a93-345a-f2ca-d45a2e2e246b
> testLoggerAdapter verbose: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.405Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: entering candidate state: node="Node at 83e13462-3df4-c52d-1e67-4006c27566eb [Candidate]" term=3
> testing.go:149: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.405Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: entering candidate state: node="Node at 83e13462-3df4-c52d-1e67-4006c27566eb [Candidate]" term=3
> testLoggerAdapter verbose: server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: 2020-12-08T22:13:18.405Z [WARN] server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: rejecting vote request since we have a leader: from=83e13462-3df4-c52d-1e67-4006c27566eb leader=82b73bdb-8a93-345a-f2ca-d45a2e2e246b
> testing.go:149: server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: 2020-12-08T22:13:18.405Z [WARN] server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: rejecting vote request since we have a leader: from=83e13462-3df4-c52d-1e67-4006c27566eb leader=82b73bdb-8a93-345a-f2ca-d45a2e2e246b
> testLoggerAdapter verbose: server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: 2020-12-08T22:13:18.416Z [WARN] server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: heartbeat timeout reached, starting election: last-leader=82b73bdb-8a93-345a-f2ca-d45a2e2e246b
> testing.go:149: server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: 2020-12-08T22:13:18.416Z [WARN] server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: heartbeat timeout reached, starting election: last-leader=82b73bdb-8a93-345a-f2ca-d45a2e2e246b
> testLoggerAdapter verbose: server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: 2020-12-08T22:13:18.416Z [INFO] server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: entering candidate state: node="Node at 08dfaa51-9538-f3d7-c25b-f6856dbb5eef [Candidate]" term=3
> testing.go:149: server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: 2020-12-08T22:13:18.416Z [INFO] server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: entering candidate state: node="Node at 08dfaa51-9538-f3d7-c25b-f6856dbb5eef [Candidate]" term=3
> testLoggerAdapter verbose: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.416Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: duplicate requestVote for same term: term=3
> testing.go:149: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.416Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: duplicate requestVote for same term: term=3
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:18.439Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:18.439Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.458Z [WARN] server-83e13462-3df4-c52d-1e67-4006c27566eb: Election timeout reached, restarting election
> testing.go:149: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.458Z [WARN] server-83e13462-3df4-c52d-1e67-4006c27566eb: Election timeout reached, restarting election
> testLoggerAdapter verbose: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.458Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: entering candidate state: node="Node at 83e13462-3df4-c52d-1e67-4006c27566eb [Candidate]" term=4
> testing.go:149: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.458Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: entering candidate state: node="Node at 83e13462-3df4-c52d-1e67-4006c27566eb [Candidate]" term=4
> testLoggerAdapter verbose: server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: 2020-12-08T22:13:18.458Z [INFO] server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: entering follower state: follower="Node at 08dfaa51-9538-f3d7-c25b-f6856dbb5eef [Follower]" leader=
> testing.go:149: server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: 2020-12-08T22:13:18.458Z [INFO] server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef: entering follower state: follower="Node at 08dfaa51-9538-f3d7-c25b-f6856dbb5eef [Follower]" leader=
> testLoggerAdapter verbose: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.458Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: election won: tally=2
> testing.go:149: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.458Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: election won: tally=2
> testLoggerAdapter verbose: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.458Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: entering leader state: leader="Node at 83e13462-3df4-c52d-1e67-4006c27566eb [Leader]"
> testing.go:149: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.458Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: entering leader state: leader="Node at 83e13462-3df4-c52d-1e67-4006c27566eb [Leader]"
> testLoggerAdapter verbose: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.458Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: added peer, starting replication: peer=server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef
> testing.go:149: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.458Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: added peer, starting replication: peer=server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef
> testLoggerAdapter verbose: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.458Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: pipelining replication: peer="{Voter server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef 08dfaa51-9538-f3d7-c25b-f6856dbb5eef}"
> testing.go:149: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.458Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: pipelining replication: peer="{Voter server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef 08dfaa51-9538-f3d7-c25b-f6856dbb5eef}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:18.563Z [INFO] stable state for Leader reached at 2020-12-08 22:13:18.458677288 +0000 UTC m=+10.015837854 (1 nodes), 19.514446ms from start of poll, 391.866949ms from cluster start. Timeout at 2020-12-08 22:13:18.56379132 +0000 UTC m=+10.120951896, 105.114042ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:18.563Z [INFO] stable state for Leader reached at 2020-12-08 22:13:18.458677288 +0000 UTC m=+10.015837854 (1 nodes), 19.514446ms from start of poll, 391.866949ms from cluster start. Timeout at 2020-12-08 22:13:18.56379132 +0000 UTC m=+10.120951896, 105.114042ms after stability
> testLoggerAdapter verbose: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.669Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: aborting pipeline replication: peer="{Voter server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef 08dfaa51-9538-f3d7-c25b-f6856dbb5eef}"
> testing.go:149: server-83e13462-3df4-c52d-1e67-4006c27566eb: 2020-12-08T22:13:18.669Z [INFO] server-83e13462-3df4-c52d-1e67-4006c27566eb: aborting pipeline replication: peer="{Voter server-08dfaa51-9538-f3d7-c25b-f6856dbb5eef 08dfaa51-9538-f3d7-c25b-f6856dbb5eef}"
> --- PASS: TestRaft_RemoveLeader_NoShutdown (0.60s)
> === RUN TestRaft_RemoveFollower_SplitCluster
> testLoggerAdapter verbose: server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: 2020-12-08T22:13:18.671Z [INFO] server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb Address:9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb} {Suffrage:Voter ID:server-25c24e87-266e-53c2-cd90-d1e602e04be9 Address:25c24e87-266e-53c2-cd90-d1e602e04be9} {Suffrage:Voter ID:server-92d1a3de-b4a1-82b9-7618-81cd0533e196 Address:92d1a3de-b4a1-82b9-7618-81cd0533e196} {Suffrage:Voter ID:server-421e9756-26d0-058c-4410-657494051d12 Address:421e9756-26d0-058c-4410-657494051d12}]"
> testing.go:149: server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: 2020-12-08T22:13:18.671Z [INFO] server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb Address:9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb} {Suffrage:Voter ID:server-25c24e87-266e-53c2-cd90-d1e602e04be9 Address:25c24e87-266e-53c2-cd90-d1e602e04be9} {Suffrage:Voter ID:server-92d1a3de-b4a1-82b9-7618-81cd0533e196 Address:92d1a3de-b4a1-82b9-7618-81cd0533e196} {Suffrage:Voter ID:server-421e9756-26d0-058c-4410-657494051d12 Address:421e9756-26d0-058c-4410-657494051d12}]"
> testLoggerAdapter verbose: server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: 2020-12-08T22:13:18.671Z [INFO] server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: entering follower state: follower="Node at 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb [Follower]" leader=
> testing.go:149: server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: 2020-12-08T22:13:18.671Z [INFO] server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: entering follower state: follower="Node at 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb [Follower]" leader=
> testLoggerAdapter verbose: server-25c24e87-266e-53c2-cd90-d1e602e04be9: 2020-12-08T22:13:18.671Z [INFO] server-25c24e87-266e-53c2-cd90-d1e602e04be9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb Address:9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb} {Suffrage:Voter ID:server-25c24e87-266e-53c2-cd90-d1e602e04be9 Address:25c24e87-266e-53c2-cd90-d1e602e04be9} {Suffrage:Voter ID:server-92d1a3de-b4a1-82b9-7618-81cd0533e196 Address:92d1a3de-b4a1-82b9-7618-81cd0533e196} {Suffrage:Voter ID:server-421e9756-26d0-058c-4410-657494051d12 Address:421e9756-26d0-058c-4410-657494051d12}]"
> testing.go:149: server-25c24e87-266e-53c2-cd90-d1e602e04be9: 2020-12-08T22:13:18.671Z [INFO] server-25c24e87-266e-53c2-cd90-d1e602e04be9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb Address:9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb} {Suffrage:Voter ID:server-25c24e87-266e-53c2-cd90-d1e602e04be9 Address:25c24e87-266e-53c2-cd90-d1e602e04be9} {Suffrage:Voter ID:server-92d1a3de-b4a1-82b9-7618-81cd0533e196 Address:92d1a3de-b4a1-82b9-7618-81cd0533e196} {Suffrage:Voter ID:server-421e9756-26d0-058c-4410-657494051d12 Address:421e9756-26d0-058c-4410-657494051d12}]"
> testLoggerAdapter verbose: server-25c24e87-266e-53c2-cd90-d1e602e04be9: 2020-12-08T22:13:18.671Z [INFO] server-25c24e87-266e-53c2-cd90-d1e602e04be9: entering follower state: follower="Node at 25c24e87-266e-53c2-cd90-d1e602e04be9 [Follower]" leader=
> testing.go:149: server-25c24e87-266e-53c2-cd90-d1e602e04be9: 2020-12-08T22:13:18.671Z [INFO] server-25c24e87-266e-53c2-cd90-d1e602e04be9: entering follower state: follower="Node at 25c24e87-266e-53c2-cd90-d1e602e04be9 [Follower]" leader=
> testLoggerAdapter verbose: server-92d1a3de-b4a1-82b9-7618-81cd0533e196: 2020-12-08T22:13:18.671Z [INFO] server-92d1a3de-b4a1-82b9-7618-81cd0533e196: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb Address:9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb} {Suffrage:Voter ID:server-25c24e87-266e-53c2-cd90-d1e602e04be9 Address:25c24e87-266e-53c2-cd90-d1e602e04be9} {Suffrage:Voter ID:server-92d1a3de-b4a1-82b9-7618-81cd0533e196 Address:92d1a3de-b4a1-82b9-7618-81cd0533e196} {Suffrage:Voter ID:server-421e9756-26d0-058c-4410-657494051d12 Address:421e9756-26d0-058c-4410-657494051d12}]"
> testing.go:149: server-92d1a3de-b4a1-82b9-7618-81cd0533e196: 2020-12-08T22:13:18.671Z [INFO] server-92d1a3de-b4a1-82b9-7618-81cd0533e196: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb Address:9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb} {Suffrage:Voter ID:server-25c24e87-266e-53c2-cd90-d1e602e04be9 Address:25c24e87-266e-53c2-cd90-d1e602e04be9} {Suffrage:Voter ID:server-92d1a3de-b4a1-82b9-7618-81cd0533e196 Address:92d1a3de-b4a1-82b9-7618-81cd0533e196} {Suffrage:Voter ID:server-421e9756-26d0-058c-4410-657494051d12 Address:421e9756-26d0-058c-4410-657494051d12}]"
> testLoggerAdapter verbose: server-92d1a3de-b4a1-82b9-7618-81cd0533e196: 2020-12-08T22:13:18.671Z [INFO] server-92d1a3de-b4a1-82b9-7618-81cd0533e196: entering follower state: follower="Node at 92d1a3de-b4a1-82b9-7618-81cd0533e196 [Follower]" leader=
> testing.go:149: server-92d1a3de-b4a1-82b9-7618-81cd0533e196: 2020-12-08T22:13:18.671Z [INFO] server-92d1a3de-b4a1-82b9-7618-81cd0533e196: entering follower state: follower="Node at 92d1a3de-b4a1-82b9-7618-81cd0533e196 [Follower]" leader=
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.671Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb Address:9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb} {Suffrage:Voter ID:server-25c24e87-266e-53c2-cd90-d1e602e04be9 Address:25c24e87-266e-53c2-cd90-d1e602e04be9} {Suffrage:Voter ID:server-92d1a3de-b4a1-82b9-7618-81cd0533e196 Address:92d1a3de-b4a1-82b9-7618-81cd0533e196} {Suffrage:Voter ID:server-421e9756-26d0-058c-4410-657494051d12 Address:421e9756-26d0-058c-4410-657494051d12}]"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.671Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb Address:9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb} {Suffrage:Voter ID:server-25c24e87-266e-53c2-cd90-d1e602e04be9 Address:25c24e87-266e-53c2-cd90-d1e602e04be9} {Suffrage:Voter ID:server-92d1a3de-b4a1-82b9-7618-81cd0533e196 Address:92d1a3de-b4a1-82b9-7618-81cd0533e196} {Suffrage:Voter ID:server-421e9756-26d0-058c-4410-657494051d12 Address:421e9756-26d0-058c-4410-657494051d12}]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:18.671Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:18.671Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.671Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: entering follower state: follower="Node at 421e9756-26d0-058c-4410-657494051d12 [Follower]" leader=
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.671Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: entering follower state: follower="Node at 421e9756-26d0-058c-4410-657494051d12 [Follower]" leader=
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.723Z [WARN] server-421e9756-26d0-058c-4410-657494051d12: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.723Z [WARN] server-421e9756-26d0-058c-4410-657494051d12: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.723Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: entering candidate state: node="Node at 421e9756-26d0-058c-4410-657494051d12 [Candidate]" term=2
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.723Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: entering candidate state: node="Node at 421e9756-26d0-058c-4410-657494051d12 [Candidate]" term=2
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: election won: tally=3
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: election won: tally=3
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: entering leader state: leader="Node at 421e9756-26d0-058c-4410-657494051d12 [Leader]"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: entering leader state: leader="Node at 421e9756-26d0-058c-4410-657494051d12 [Leader]"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: added peer, starting replication: peer=server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: added peer, starting replication: peer=server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: added peer, starting replication: peer=server-25c24e87-266e-53c2-cd90-d1e602e04be9
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: added peer, starting replication: peer=server-25c24e87-266e-53c2-cd90-d1e602e04be9
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: added peer, starting replication: peer=server-92d1a3de-b4a1-82b9-7618-81cd0533e196
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: added peer, starting replication: peer=server-92d1a3de-b4a1-82b9-7618-81cd0533e196
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: pipelining replication: peer="{Voter server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb}"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: pipelining replication: peer="{Voter server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb}"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: pipelining replication: peer="{Voter server-92d1a3de-b4a1-82b9-7618-81cd0533e196 92d1a3de-b4a1-82b9-7618-81cd0533e196}"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: pipelining replication: peer="{Voter server-92d1a3de-b4a1-82b9-7618-81cd0533e196 92d1a3de-b4a1-82b9-7618-81cd0533e196}"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: pipelining replication: peer="{Voter server-25c24e87-266e-53c2-cd90-d1e602e04be9 25c24e87-266e-53c2-cd90-d1e602e04be9}"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:18.724Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: pipelining replication: peer="{Voter server-25c24e87-266e-53c2-cd90-d1e602e04be9 25c24e87-266e-53c2-cd90-d1e602e04be9}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:18.829Z [INFO] stable state for Leader reached at 2020-12-08 22:13:18.724158657 +0000 UTC m=+10.281319254 (1 nodes), 52.663719ms from start of poll, 53.262645ms from cluster start. Timeout at 2020-12-08 22:13:18.829276466 +0000 UTC m=+10.386437034, 105.11778ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:18.829Z [INFO] stable state for Leader reached at 2020-12-08 22:13:18.724158657 +0000 UTC m=+10.281319254 (1 nodes), 52.663719ms from start of poll, 53.262645ms from cluster start. Timeout at 2020-12-08 22:13:18.829276466 +0000 UTC m=+10.386437034, 105.11778ms after stability
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:18.934Z [INFO] starting stability test: raft-state=Follower
> testing.go:149: cluster: 2020-12-08T22:13:18.934Z [INFO] starting stability test: raft-state=Follower
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:19.039Z [INFO] stable state for Follower reached at 2020-12-08 22:13:18.934434373 +0000 UTC m=+10.491594937 (3 nodes), 1.256µs from start of poll, 263.538328ms from cluster start. Timeout at 2020-12-08 22:13:19.039543653 +0000 UTC m=+10.596704221, 105.109284ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:19.039Z [INFO] stable state for Follower reached at 2020-12-08 22:13:18.934434373 +0000 UTC m=+10.491594937 (3 nodes), 1.256µs from start of poll, 263.538328ms from cluster start. Timeout at 2020-12-08 22:13:19.039543653 +0000 UTC m=+10.596704221, 105.109284ms after stability
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.039Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: updating configuration: command=RemoveServer server-id=server-92d1a3de-b4a1-82b9-7618-81cd0533e196 server-addr= servers="[{Suffrage:Voter ID:server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb Address:9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb} {Suffrage:Voter ID:server-25c24e87-266e-53c2-cd90-d1e602e04be9 Address:25c24e87-266e-53c2-cd90-d1e602e04be9} {Suffrage:Voter ID:server-421e9756-26d0-058c-4410-657494051d12 Address:421e9756-26d0-058c-4410-657494051d12}]"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.039Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: updating configuration: command=RemoveServer server-id=server-92d1a3de-b4a1-82b9-7618-81cd0533e196 server-addr= servers="[{Suffrage:Voter ID:server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb Address:9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb} {Suffrage:Voter ID:server-25c24e87-266e-53c2-cd90-d1e602e04be9 Address:25c24e87-266e-53c2-cd90-d1e602e04be9} {Suffrage:Voter ID:server-421e9756-26d0-058c-4410-657494051d12 Address:421e9756-26d0-058c-4410-657494051d12}]"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.039Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: removed peer, stopping replication: peer=server-92d1a3de-b4a1-82b9-7618-81cd0533e196 last-index=3
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.039Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: removed peer, stopping replication: peer=server-92d1a3de-b4a1-82b9-7618-81cd0533e196 last-index=3
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.039Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to pipeline appendEntries: peer="{Voter server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb}" error="append pipeline closed"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.039Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to pipeline appendEntries: peer="{Voter server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb}" error="append pipeline closed"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.039Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: aborting pipeline replication: peer="{Voter server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb}"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.039Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: aborting pipeline replication: peer="{Voter server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb}"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.039Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to pipeline appendEntries: peer="{Voter server-25c24e87-266e-53c2-cd90-d1e602e04be9 25c24e87-266e-53c2-cd90-d1e602e04be9}" error="append pipeline closed"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.039Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to pipeline appendEntries: peer="{Voter server-25c24e87-266e-53c2-cd90-d1e602e04be9 25c24e87-266e-53c2-cd90-d1e602e04be9}" error="append pipeline closed"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.039Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: aborting pipeline replication: peer="{Voter server-25c24e87-266e-53c2-cd90-d1e602e04be9 25c24e87-266e-53c2-cd90-d1e602e04be9}"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.039Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: aborting pipeline replication: peer="{Voter server-25c24e87-266e-53c2-cd90-d1e602e04be9 25c24e87-266e-53c2-cd90-d1e602e04be9}"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.039Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: aborting pipeline replication: peer="{Voter server-92d1a3de-b4a1-82b9-7618-81cd0533e196 92d1a3de-b4a1-82b9-7618-81cd0533e196}"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.039Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: aborting pipeline replication: peer="{Voter server-92d1a3de-b4a1-82b9-7618-81cd0533e196 92d1a3de-b4a1-82b9-7618-81cd0533e196}"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.045Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to heartbeat to: peer=9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb error="failed to connect to peer: 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.045Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to heartbeat to: peer=9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb error="failed to connect to peer: 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.047Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to heartbeat to: peer=25c24e87-266e-53c2-cd90-d1e602e04be9 error="failed to connect to peer: 25c24e87-266e-53c2-cd90-d1e602e04be9"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.047Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to heartbeat to: peer=25c24e87-266e-53c2-cd90-d1e602e04be9 error="failed to connect to peer: 25c24e87-266e-53c2-cd90-d1e602e04be9"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.049Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb}" error="failed to connect to peer: 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.049Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb}" error="failed to connect to peer: 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.049Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-25c24e87-266e-53c2-cd90-d1e602e04be9 25c24e87-266e-53c2-cd90-d1e602e04be9}" error="failed to connect to peer: 25c24e87-266e-53c2-cd90-d1e602e04be9"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.049Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-25c24e87-266e-53c2-cd90-d1e602e04be9 25c24e87-266e-53c2-cd90-d1e602e04be9}" error="failed to connect to peer: 25c24e87-266e-53c2-cd90-d1e602e04be9"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.061Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to heartbeat to: peer=9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb error="failed to connect to peer: 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.061Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to heartbeat to: peer=9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb error="failed to connect to peer: 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.065Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-25c24e87-266e-53c2-cd90-d1e602e04be9 25c24e87-266e-53c2-cd90-d1e602e04be9}" error="failed to connect to peer: 25c24e87-266e-53c2-cd90-d1e602e04be9"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.065Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-25c24e87-266e-53c2-cd90-d1e602e04be9 25c24e87-266e-53c2-cd90-d1e602e04be9}" error="failed to connect to peer: 25c24e87-266e-53c2-cd90-d1e602e04be9"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.066Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to heartbeat to: peer=25c24e87-266e-53c2-cd90-d1e602e04be9 error="failed to connect to peer: 25c24e87-266e-53c2-cd90-d1e602e04be9"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.066Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to heartbeat to: peer=25c24e87-266e-53c2-cd90-d1e602e04be9 error="failed to connect to peer: 25c24e87-266e-53c2-cd90-d1e602e04be9"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.068Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb}" error="failed to connect to peer: 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.068Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb}" error="failed to connect to peer: 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.078Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to heartbeat to: peer=9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb error="failed to connect to peer: 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.078Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to heartbeat to: peer=9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb error="failed to connect to peer: 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.084Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-25c24e87-266e-53c2-cd90-d1e602e04be9 25c24e87-266e-53c2-cd90-d1e602e04be9}" error="failed to connect to peer: 25c24e87-266e-53c2-cd90-d1e602e04be9"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.084Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-25c24e87-266e-53c2-cd90-d1e602e04be9 25c24e87-266e-53c2-cd90-d1e602e04be9}" error="failed to connect to peer: 25c24e87-266e-53c2-cd90-d1e602e04be9"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.084Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to heartbeat to: peer=25c24e87-266e-53c2-cd90-d1e602e04be9 error="failed to connect to peer: 25c24e87-266e-53c2-cd90-d1e602e04be9"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.084Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to heartbeat to: peer=25c24e87-266e-53c2-cd90-d1e602e04be9 error="failed to connect to peer: 25c24e87-266e-53c2-cd90-d1e602e04be9"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.087Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb}" error="failed to connect to peer: 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.087Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb}" error="failed to connect to peer: 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.089Z [WARN] server-421e9756-26d0-058c-4410-657494051d12: failed to contact: server-id=server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb time=50.066483ms
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.089Z [WARN] server-421e9756-26d0-058c-4410-657494051d12: failed to contact: server-id=server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb time=50.066483ms
> testLoggerAdapter verbose: server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: 2020-12-08T22:13:19.093Z [WARN] server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: heartbeat timeout reached, starting election: last-leader=421e9756-26d0-058c-4410-657494051d12
> testing.go:149: server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: 2020-12-08T22:13:19.093Z [WARN] server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: heartbeat timeout reached, starting election: last-leader=421e9756-26d0-058c-4410-657494051d12
> testLoggerAdapter verbose: server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: 2020-12-08T22:13:19.093Z [INFO] server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: entering candidate state: node="Node at 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb [Candidate]" term=3
> testing.go:149: server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: 2020-12-08T22:13:19.093Z [INFO] server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: entering candidate state: node="Node at 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb [Candidate]" term=3
> testLoggerAdapter verbose: server-25c24e87-266e-53c2-cd90-d1e602e04be9: 2020-12-08T22:13:19.093Z [WARN] server-25c24e87-266e-53c2-cd90-d1e602e04be9: rejecting vote request since we have a leader: from=9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb leader=421e9756-26d0-058c-4410-657494051d12
> testLoggerAdapter verbose: server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: 2020-12-08T22:13:19.093Z [ERROR] server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: failed to make requestVote RPC: target="{Voter server-421e9756-26d0-058c-4410-657494051d12 421e9756-26d0-058c-4410-657494051d12}" error="failed to connect to peer: 421e9756-26d0-058c-4410-657494051d12"
> testing.go:149: server-25c24e87-266e-53c2-cd90-d1e602e04be9: 2020-12-08T22:13:19.093Z [WARN] server-25c24e87-266e-53c2-cd90-d1e602e04be9: rejecting vote request since we have a leader: from=9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb leader=421e9756-26d0-058c-4410-657494051d12
> testing.go:149: server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: 2020-12-08T22:13:19.093Z [ERROR] server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: failed to make requestVote RPC: target="{Voter server-421e9756-26d0-058c-4410-657494051d12 421e9756-26d0-058c-4410-657494051d12}" error="failed to connect to peer: 421e9756-26d0-058c-4410-657494051d12"
> testLoggerAdapter verbose: server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: 2020-12-08T22:13:19.093Z [ERROR] server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: failed to make requestVote RPC: target="{Voter server-92d1a3de-b4a1-82b9-7618-81cd0533e196 92d1a3de-b4a1-82b9-7618-81cd0533e196}" error="failed to connect to peer: 92d1a3de-b4a1-82b9-7618-81cd0533e196"
> testing.go:149: server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: 2020-12-08T22:13:19.093Z [ERROR] server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb: failed to make requestVote RPC: target="{Voter server-92d1a3de-b4a1-82b9-7618-81cd0533e196 92d1a3de-b4a1-82b9-7618-81cd0533e196}" error="failed to connect to peer: 92d1a3de-b4a1-82b9-7618-81cd0533e196"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.099Z [WARN] server-421e9756-26d0-058c-4410-657494051d12: failed to contact: server-id=server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb time=60.212973ms
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.099Z [WARN] server-421e9756-26d0-058c-4410-657494051d12: failed to contact: server-id=server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb time=60.212973ms
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.099Z [WARN] server-421e9756-26d0-058c-4410-657494051d12: failed to contact: server-id=server-25c24e87-266e-53c2-cd90-d1e602e04be9 time=59.703795ms
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.099Z [WARN] server-421e9756-26d0-058c-4410-657494051d12: failed to contact: server-id=server-25c24e87-266e-53c2-cd90-d1e602e04be9 time=59.703795ms
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.099Z [WARN] server-421e9756-26d0-058c-4410-657494051d12: failed to contact quorum of nodes, stepping down
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.099Z [WARN] server-421e9756-26d0-058c-4410-657494051d12: failed to contact quorum of nodes, stepping down
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.099Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: entering follower state: follower="Node at 421e9756-26d0-058c-4410-657494051d12 [Follower]" leader=
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.099Z [INFO] server-421e9756-26d0-058c-4410-657494051d12: entering follower state: follower="Node at 421e9756-26d0-058c-4410-657494051d12 [Follower]" leader=
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.099Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-25c24e87-266e-53c2-cd90-d1e602e04be9 25c24e87-266e-53c2-cd90-d1e602e04be9}" error="failed to connect to peer: 25c24e87-266e-53c2-cd90-d1e602e04be9"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.099Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-25c24e87-266e-53c2-cd90-d1e602e04be9 25c24e87-266e-53c2-cd90-d1e602e04be9}" error="failed to connect to peer: 25c24e87-266e-53c2-cd90-d1e602e04be9"
> testLoggerAdapter verbose: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.099Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb}" error="failed to connect to peer: 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb"
> testing.go:149: server-421e9756-26d0-058c-4410-657494051d12: 2020-12-08T22:13:19.099Z [ERROR] server-421e9756-26d0-058c-4410-657494051d12: failed to appendEntries to: peer="{Voter server-9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb}" error="failed to connect to peer: 9099d2b7-5a82-e798-6e5f-5c2d0b2ec7fb"
> --- PASS: TestRaft_RemoveFollower_SplitCluster (0.43s)
> === RUN TestRaft_AddKnownPeer
> testLoggerAdapter verbose: server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4: 2020-12-08T22:13:19.101Z [INFO] server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 Address:423a08e9-df19-8a46-9bb4-2d9ba7ce02b4} {Suffrage:Voter ID:server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b Address:b37e2d83-350e-f2f7-f3cb-6ba0418ae91b} {Suffrage:Voter ID:server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de Address:7ce530f5-feab-3702-e3fa-d45f5ef2e8de}]"
> testing.go:149: server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4: 2020-12-08T22:13:19.101Z [INFO] server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 Address:423a08e9-df19-8a46-9bb4-2d9ba7ce02b4} {Suffrage:Voter ID:server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b Address:b37e2d83-350e-f2f7-f3cb-6ba0418ae91b} {Suffrage:Voter ID:server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de Address:7ce530f5-feab-3702-e3fa-d45f5ef2e8de}]"
> testLoggerAdapter verbose: server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4: 2020-12-08T22:13:19.101Z [INFO] server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4: entering follower state: follower="Node at 423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 [Follower]" leader=
> testing.go:149: server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4: 2020-12-08T22:13:19.101Z [INFO] server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4: entering follower state: follower="Node at 423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 [Follower]" leader=
> testLoggerAdapter verbose: server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b: 2020-12-08T22:13:19.101Z [INFO] server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 Address:423a08e9-df19-8a46-9bb4-2d9ba7ce02b4} {Suffrage:Voter ID:server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b Address:b37e2d83-350e-f2f7-f3cb-6ba0418ae91b} {Suffrage:Voter ID:server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de Address:7ce530f5-feab-3702-e3fa-d45f5ef2e8de}]"
> testing.go:149: server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b: 2020-12-08T22:13:19.101Z [INFO] server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 Address:423a08e9-df19-8a46-9bb4-2d9ba7ce02b4} {Suffrage:Voter ID:server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b Address:b37e2d83-350e-f2f7-f3cb-6ba0418ae91b} {Suffrage:Voter ID:server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de Address:7ce530f5-feab-3702-e3fa-d45f5ef2e8de}]"
> testLoggerAdapter verbose: server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b: 2020-12-08T22:13:19.101Z [INFO] server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b: entering follower state: follower="Node at b37e2d83-350e-f2f7-f3cb-6ba0418ae91b [Follower]" leader=
> testing.go:149: server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b: 2020-12-08T22:13:19.101Z [INFO] server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b: entering follower state: follower="Node at b37e2d83-350e-f2f7-f3cb-6ba0418ae91b [Follower]" leader=
> testLoggerAdapter verbose: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.101Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 Address:423a08e9-df19-8a46-9bb4-2d9ba7ce02b4} {Suffrage:Voter ID:server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b Address:b37e2d83-350e-f2f7-f3cb-6ba0418ae91b} {Suffrage:Voter ID:server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de Address:7ce530f5-feab-3702-e3fa-d45f5ef2e8de}]"
> testing.go:149: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.101Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 Address:423a08e9-df19-8a46-9bb4-2d9ba7ce02b4} {Suffrage:Voter ID:server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b Address:b37e2d83-350e-f2f7-f3cb-6ba0418ae91b} {Suffrage:Voter ID:server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de Address:7ce530f5-feab-3702-e3fa-d45f5ef2e8de}]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:19.101Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:19.101Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.101Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: entering follower state: follower="Node at 7ce530f5-feab-3702-e3fa-d45f5ef2e8de [Follower]" leader=
> testing.go:149: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.101Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: entering follower state: follower="Node at 7ce530f5-feab-3702-e3fa-d45f5ef2e8de [Follower]" leader=
> testLoggerAdapter verbose: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [WARN] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [WARN] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: entering candidate state: node="Node at 7ce530f5-feab-3702-e3fa-d45f5ef2e8de [Candidate]" term=2
> testing.go:149: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: entering candidate state: node="Node at 7ce530f5-feab-3702-e3fa-d45f5ef2e8de [Candidate]" term=2
> testLoggerAdapter verbose: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: election won: tally=2
> testing.go:149: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: election won: tally=2
> testLoggerAdapter verbose: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: entering leader state: leader="Node at 7ce530f5-feab-3702-e3fa-d45f5ef2e8de [Leader]"
> testing.go:149: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: entering leader state: leader="Node at 7ce530f5-feab-3702-e3fa-d45f5ef2e8de [Leader]"
> testLoggerAdapter verbose: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: added peer, starting replication: peer=server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4
> testing.go:149: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: added peer, starting replication: peer=server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4
> testLoggerAdapter verbose: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: added peer, starting replication: peer=server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b
> testing.go:149: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: added peer, starting replication: peer=server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b
> testLoggerAdapter verbose: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: pipelining replication: peer="{Voter server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 423a08e9-df19-8a46-9bb4-2d9ba7ce02b4}"
> testing.go:149: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: pipelining replication: peer="{Voter server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 423a08e9-df19-8a46-9bb4-2d9ba7ce02b4}"
> testLoggerAdapter verbose: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: pipelining replication: peer="{Voter server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b b37e2d83-350e-f2f7-f3cb-6ba0418ae91b}"
> testing.go:149: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.180Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: pipelining replication: peer="{Voter server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b b37e2d83-350e-f2f7-f3cb-6ba0418ae91b}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:19.286Z [INFO] stable state for Leader reached at 2020-12-08 22:13:19.180860541 +0000 UTC m=+10.738021111 (1 nodes), 79.060308ms from start of poll, 79.518992ms from cluster start. Timeout at 2020-12-08 22:13:19.285967643 +0000 UTC m=+10.843128209, 105.107098ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:19.286Z [INFO] stable state for Leader reached at 2020-12-08 22:13:19.180860541 +0000 UTC m=+10.738021111 (1 nodes), 79.060308ms from start of poll, 79.518992ms from cluster start. Timeout at 2020-12-08 22:13:19.285967643 +0000 UTC m=+10.843128209, 105.107098ms after stability
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:19.286Z [INFO] starting stability test: raft-state=Follower
> testing.go:149: cluster: 2020-12-08T22:13:19.286Z [INFO] starting stability test: raft-state=Follower
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:19.391Z [INFO] stable state for Follower reached at 2020-12-08 22:13:19.286087548 +0000 UTC m=+10.843248124 (2 nodes), 1.061µs from start of poll, 184.746005ms from cluster start. Timeout at 2020-12-08 22:13:19.391191757 +0000 UTC m=+10.948352327, 105.104203ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:19.391Z [INFO] stable state for Follower reached at 2020-12-08 22:13:19.286087548 +0000 UTC m=+10.843248124 (2 nodes), 1.061µs from start of poll, 184.746005ms from cluster start. Timeout at 2020-12-08 22:13:19.391191757 +0000 UTC m=+10.948352327, 105.104203ms after stability
> testLoggerAdapter verbose: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.391Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: updating configuration: command=AddStaging server-id=server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 server-addr=423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 servers="[{Suffrage:Voter ID:server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 Address:423a08e9-df19-8a46-9bb4-2d9ba7ce02b4} {Suffrage:Voter ID:server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b Address:b37e2d83-350e-f2f7-f3cb-6ba0418ae91b} {Suffrage:Voter ID:server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de Address:7ce530f5-feab-3702-e3fa-d45f5ef2e8de}]"
> testing.go:149: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.391Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: updating configuration: command=AddStaging server-id=server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 server-addr=423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 servers="[{Suffrage:Voter ID:server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 Address:423a08e9-df19-8a46-9bb4-2d9ba7ce02b4} {Suffrage:Voter ID:server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b Address:b37e2d83-350e-f2f7-f3cb-6ba0418ae91b} {Suffrage:Voter ID:server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de Address:7ce530f5-feab-3702-e3fa-d45f5ef2e8de}]"
> testLoggerAdapter verbose: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.391Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: aborting pipeline replication: peer="{Voter server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b b37e2d83-350e-f2f7-f3cb-6ba0418ae91b}"
> testing.go:149: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.391Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: aborting pipeline replication: peer="{Voter server-b37e2d83-350e-f2f7-f3cb-6ba0418ae91b b37e2d83-350e-f2f7-f3cb-6ba0418ae91b}"
> testLoggerAdapter verbose: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.391Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: aborting pipeline replication: peer="{Voter server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 423a08e9-df19-8a46-9bb4-2d9ba7ce02b4}"
> testing.go:149: server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: 2020-12-08T22:13:19.391Z [INFO] server-7ce530f5-feab-3702-e3fa-d45f5ef2e8de: aborting pipeline replication: peer="{Voter server-423a08e9-df19-8a46-9bb4-2d9ba7ce02b4 423a08e9-df19-8a46-9bb4-2d9ba7ce02b4}"
> --- PASS: TestRaft_AddKnownPeer (0.29s)
> === RUN TestRaft_RemoveUnknownPeer
> testLoggerAdapter verbose: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.393Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3048ddc3-06ee-ab22-3552-6e04e64c5515 Address:3048ddc3-06ee-ab22-3552-6e04e64c5515} {Suffrage:Voter ID:server-7412e29e-f250-48a1-8de3-af64beb2d08e Address:7412e29e-f250-48a1-8de3-af64beb2d08e} {Suffrage:Voter ID:server-600f2594-bc05-690f-c10d-433dcb46798b Address:600f2594-bc05-690f-c10d-433dcb46798b}]"
> testing.go:149: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.393Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3048ddc3-06ee-ab22-3552-6e04e64c5515 Address:3048ddc3-06ee-ab22-3552-6e04e64c5515} {Suffrage:Voter ID:server-7412e29e-f250-48a1-8de3-af64beb2d08e Address:7412e29e-f250-48a1-8de3-af64beb2d08e} {Suffrage:Voter ID:server-600f2594-bc05-690f-c10d-433dcb46798b Address:600f2594-bc05-690f-c10d-433dcb46798b}]"
> testLoggerAdapter verbose: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.393Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: entering follower state: follower="Node at 3048ddc3-06ee-ab22-3552-6e04e64c5515 [Follower]" leader=
> testing.go:149: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.393Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: entering follower state: follower="Node at 3048ddc3-06ee-ab22-3552-6e04e64c5515 [Follower]" leader=
> testLoggerAdapter verbose: server-7412e29e-f250-48a1-8de3-af64beb2d08e: 2020-12-08T22:13:19.393Z [INFO] server-7412e29e-f250-48a1-8de3-af64beb2d08e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3048ddc3-06ee-ab22-3552-6e04e64c5515 Address:3048ddc3-06ee-ab22-3552-6e04e64c5515} {Suffrage:Voter ID:server-7412e29e-f250-48a1-8de3-af64beb2d08e Address:7412e29e-f250-48a1-8de3-af64beb2d08e} {Suffrage:Voter ID:server-600f2594-bc05-690f-c10d-433dcb46798b Address:600f2594-bc05-690f-c10d-433dcb46798b}]"
> testing.go:149: server-7412e29e-f250-48a1-8de3-af64beb2d08e: 2020-12-08T22:13:19.393Z [INFO] server-7412e29e-f250-48a1-8de3-af64beb2d08e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3048ddc3-06ee-ab22-3552-6e04e64c5515 Address:3048ddc3-06ee-ab22-3552-6e04e64c5515} {Suffrage:Voter ID:server-7412e29e-f250-48a1-8de3-af64beb2d08e Address:7412e29e-f250-48a1-8de3-af64beb2d08e} {Suffrage:Voter ID:server-600f2594-bc05-690f-c10d-433dcb46798b Address:600f2594-bc05-690f-c10d-433dcb46798b}]"
> testLoggerAdapter verbose: server-7412e29e-f250-48a1-8de3-af64beb2d08e: 2020-12-08T22:13:19.393Z [INFO] server-7412e29e-f250-48a1-8de3-af64beb2d08e: entering follower state: follower="Node at 7412e29e-f250-48a1-8de3-af64beb2d08e [Follower]" leader=
> testing.go:149: server-7412e29e-f250-48a1-8de3-af64beb2d08e: 2020-12-08T22:13:19.393Z [INFO] server-7412e29e-f250-48a1-8de3-af64beb2d08e: entering follower state: follower="Node at 7412e29e-f250-48a1-8de3-af64beb2d08e [Follower]" leader=
> testLoggerAdapter verbose: server-600f2594-bc05-690f-c10d-433dcb46798b: 2020-12-08T22:13:19.393Z [INFO] server-600f2594-bc05-690f-c10d-433dcb46798b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3048ddc3-06ee-ab22-3552-6e04e64c5515 Address:3048ddc3-06ee-ab22-3552-6e04e64c5515} {Suffrage:Voter ID:server-7412e29e-f250-48a1-8de3-af64beb2d08e Address:7412e29e-f250-48a1-8de3-af64beb2d08e} {Suffrage:Voter ID:server-600f2594-bc05-690f-c10d-433dcb46798b Address:600f2594-bc05-690f-c10d-433dcb46798b}]"
> testing.go:149: server-600f2594-bc05-690f-c10d-433dcb46798b: 2020-12-08T22:13:19.393Z [INFO] server-600f2594-bc05-690f-c10d-433dcb46798b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3048ddc3-06ee-ab22-3552-6e04e64c5515 Address:3048ddc3-06ee-ab22-3552-6e04e64c5515} {Suffrage:Voter ID:server-7412e29e-f250-48a1-8de3-af64beb2d08e Address:7412e29e-f250-48a1-8de3-af64beb2d08e} {Suffrage:Voter ID:server-600f2594-bc05-690f-c10d-433dcb46798b Address:600f2594-bc05-690f-c10d-433dcb46798b}]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:19.393Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:19.393Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-600f2594-bc05-690f-c10d-433dcb46798b: 2020-12-08T22:13:19.393Z [INFO] server-600f2594-bc05-690f-c10d-433dcb46798b: entering follower state: follower="Node at 600f2594-bc05-690f-c10d-433dcb46798b [Follower]" leader=
> testing.go:149: server-600f2594-bc05-690f-c10d-433dcb46798b: 2020-12-08T22:13:19.393Z [INFO] server-600f2594-bc05-690f-c10d-433dcb46798b: entering follower state: follower="Node at 600f2594-bc05-690f-c10d-433dcb46798b [Follower]" leader=
> testLoggerAdapter verbose: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [WARN] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [WARN] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: entering candidate state: node="Node at 3048ddc3-06ee-ab22-3552-6e04e64c5515 [Candidate]" term=2
> testing.go:149: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: entering candidate state: node="Node at 3048ddc3-06ee-ab22-3552-6e04e64c5515 [Candidate]" term=2
> testLoggerAdapter verbose: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: election won: tally=2
> testing.go:149: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: election won: tally=2
> testLoggerAdapter verbose: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: entering leader state: leader="Node at 3048ddc3-06ee-ab22-3552-6e04e64c5515 [Leader]"
> testing.go:149: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: entering leader state: leader="Node at 3048ddc3-06ee-ab22-3552-6e04e64c5515 [Leader]"
> testLoggerAdapter verbose: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: added peer, starting replication: peer=server-7412e29e-f250-48a1-8de3-af64beb2d08e
> testing.go:149: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: added peer, starting replication: peer=server-7412e29e-f250-48a1-8de3-af64beb2d08e
> testLoggerAdapter verbose: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: added peer, starting replication: peer=server-600f2594-bc05-690f-c10d-433dcb46798b
> testing.go:149: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: added peer, starting replication: peer=server-600f2594-bc05-690f-c10d-433dcb46798b
> testLoggerAdapter verbose: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: pipelining replication: peer="{Voter server-7412e29e-f250-48a1-8de3-af64beb2d08e 7412e29e-f250-48a1-8de3-af64beb2d08e}"
> testing.go:149: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: pipelining replication: peer="{Voter server-7412e29e-f250-48a1-8de3-af64beb2d08e 7412e29e-f250-48a1-8de3-af64beb2d08e}"
> testLoggerAdapter verbose: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: pipelining replication: peer="{Voter server-600f2594-bc05-690f-c10d-433dcb46798b 600f2594-bc05-690f-c10d-433dcb46798b}"
> testing.go:149: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.461Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: pipelining replication: peer="{Voter server-600f2594-bc05-690f-c10d-433dcb46798b 600f2594-bc05-690f-c10d-433dcb46798b}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:19.566Z [INFO] stable state for Leader reached at 2020-12-08 22:13:19.461178796 +0000 UTC m=+11.018339397 (1 nodes), 67.483879ms from start of poll, 68.08833ms from cluster start. Timeout at 2020-12-08 22:13:19.566292719 +0000 UTC m=+11.123453292, 105.113895ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:19.566Z [INFO] stable state for Leader reached at 2020-12-08 22:13:19.461178796 +0000 UTC m=+11.018339397 (1 nodes), 67.483879ms from start of poll, 68.08833ms from cluster start. Timeout at 2020-12-08 22:13:19.566292719 +0000 UTC m=+11.123453292, 105.113895ms after stability
> testLoggerAdapter verbose: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.566Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: updating configuration: command=RemoveServer server-id=993e7c97-b7b6-0817-aab6-55c2e6b18fab server-addr= servers="[{Suffrage:Voter ID:server-3048ddc3-06ee-ab22-3552-6e04e64c5515 Address:3048ddc3-06ee-ab22-3552-6e04e64c5515} {Suffrage:Voter ID:server-7412e29e-f250-48a1-8de3-af64beb2d08e Address:7412e29e-f250-48a1-8de3-af64beb2d08e} {Suffrage:Voter ID:server-600f2594-bc05-690f-c10d-433dcb46798b Address:600f2594-bc05-690f-c10d-433dcb46798b}]"
> testing.go:149: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.566Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: updating configuration: command=RemoveServer server-id=993e7c97-b7b6-0817-aab6-55c2e6b18fab server-addr= servers="[{Suffrage:Voter ID:server-3048ddc3-06ee-ab22-3552-6e04e64c5515 Address:3048ddc3-06ee-ab22-3552-6e04e64c5515} {Suffrage:Voter ID:server-7412e29e-f250-48a1-8de3-af64beb2d08e Address:7412e29e-f250-48a1-8de3-af64beb2d08e} {Suffrage:Voter ID:server-600f2594-bc05-690f-c10d-433dcb46798b Address:600f2594-bc05-690f-c10d-433dcb46798b}]"
> testLoggerAdapter verbose: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.566Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: aborting pipeline replication: peer="{Voter server-600f2594-bc05-690f-c10d-433dcb46798b 600f2594-bc05-690f-c10d-433dcb46798b}"
> testing.go:149: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.566Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: aborting pipeline replication: peer="{Voter server-600f2594-bc05-690f-c10d-433dcb46798b 600f2594-bc05-690f-c10d-433dcb46798b}"
> testLoggerAdapter verbose: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.566Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: aborting pipeline replication: peer="{Voter server-7412e29e-f250-48a1-8de3-af64beb2d08e 7412e29e-f250-48a1-8de3-af64beb2d08e}"
> testing.go:149: server-3048ddc3-06ee-ab22-3552-6e04e64c5515: 2020-12-08T22:13:19.566Z [INFO] server-3048ddc3-06ee-ab22-3552-6e04e64c5515: aborting pipeline replication: peer="{Voter server-7412e29e-f250-48a1-8de3-af64beb2d08e 7412e29e-f250-48a1-8de3-af64beb2d08e}"
> --- PASS: TestRaft_RemoveUnknownPeer (0.18s)
> === RUN TestRaft_SnapshotRestore
> testLoggerAdapter verbose: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.568Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90 Address:ec2d14b0-b57f-e1cc-9f21-03815fa09e90}]"
> testing.go:149: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.568Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90 Address:ec2d14b0-b57f-e1cc-9f21-03815fa09e90}]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:19.568Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:19.568Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.568Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: entering follower state: follower="Node at ec2d14b0-b57f-e1cc-9f21-03815fa09e90 [Follower]" leader=
> testing.go:149: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.568Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: entering follower state: follower="Node at ec2d14b0-b57f-e1cc-9f21-03815fa09e90 [Follower]" leader=
> testLoggerAdapter verbose: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.626Z [WARN] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.626Z [WARN] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.626Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: entering candidate state: node="Node at ec2d14b0-b57f-e1cc-9f21-03815fa09e90 [Candidate]" term=2
> testing.go:149: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.626Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: entering candidate state: node="Node at ec2d14b0-b57f-e1cc-9f21-03815fa09e90 [Candidate]" term=2
> testLoggerAdapter verbose: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.626Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: election won: tally=1
> testing.go:149: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.626Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: election won: tally=1
> testLoggerAdapter verbose: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.626Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: entering leader state: leader="Node at ec2d14b0-b57f-e1cc-9f21-03815fa09e90 [Leader]"
> testing.go:149: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.626Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: entering leader state: leader="Node at ec2d14b0-b57f-e1cc-9f21-03815fa09e90 [Leader]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:19.731Z [INFO] stable state for Leader reached at 2020-12-08 22:13:19.626829926 +0000 UTC m=+11.183990498 (1 nodes), 58.713652ms from start of poll, 58.924571ms from cluster start. Timeout at 2020-12-08 22:13:19.73194487 +0000 UTC m=+11.289105434, 105.114936ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:19.731Z [INFO] stable state for Leader reached at 2020-12-08 22:13:19.626829926 +0000 UTC m=+11.183990498 (1 nodes), 58.713652ms from start of poll, 58.924571ms from cluster start. Timeout at 2020-12-08 22:13:19.73194487 +0000 UTC m=+11.289105434, 105.114936ms after stability
> testLoggerAdapter verbose: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.733Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: starting snapshot up to: index=102
> testing.go:149: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.733Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: starting snapshot up to: index=102
> testing.go:153: 2020-12-08T22:13:19.733Z [INFO] creating new snapshot: path=/tmp/raft589034968/snapshots/2-102-1607465599733.tmp
> testLoggerAdapter verbose: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.858Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: compacting logs: from=1 to=92
> testing.go:149: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.858Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: compacting logs: from=1 to=92
> testLoggerAdapter verbose: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.858Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: snapshot complete up to: index=102
> testing.go:149: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.858Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: snapshot complete up to: index=102
> testLoggerAdapter verbose: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.859Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: restored from snapshot: id=2-102-1607465599733
> testing.go:149: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.859Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: restored from snapshot: id=2-102-1607465599733
> testLoggerAdapter verbose: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.859Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90 Address:ec2d14b0-b57f-e1cc-9f21-03815fa09e90}]"
> testing.go:149: server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: 2020-12-08T22:13:19.859Z [INFO] server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec2d14b0-b57f-e1cc-9f21-03815fa09e90 Address:ec2d14b0-b57f-e1cc-9f21-03815fa09e90}]"
> --- PASS: TestRaft_SnapshotRestore (0.29s)
> === RUN TestRaft_NoRestoreOnStart
> testLoggerAdapter verbose: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:19.861Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f3ab57c-bef2-3ed9-000f-a392304c44a0 Address:7f3ab57c-bef2-3ed9-000f-a392304c44a0}]"
> testing.go:149: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:19.861Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f3ab57c-bef2-3ed9-000f-a392304c44a0 Address:7f3ab57c-bef2-3ed9-000f-a392304c44a0}]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:19.861Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:19.861Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:19.861Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: entering follower state: follower="Node at 7f3ab57c-bef2-3ed9-000f-a392304c44a0 [Follower]" leader=
> testing.go:149: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:19.861Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: entering follower state: follower="Node at 7f3ab57c-bef2-3ed9-000f-a392304c44a0 [Follower]" leader=
> testLoggerAdapter verbose: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:19.917Z [WARN] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:19.917Z [WARN] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:19.917Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: entering candidate state: node="Node at 7f3ab57c-bef2-3ed9-000f-a392304c44a0 [Candidate]" term=2
> testing.go:149: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:19.917Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: entering candidate state: node="Node at 7f3ab57c-bef2-3ed9-000f-a392304c44a0 [Candidate]" term=2
> testLoggerAdapter verbose: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:19.917Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: election won: tally=1
> testing.go:149: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:19.917Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: election won: tally=1
> testLoggerAdapter verbose: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:19.917Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: entering leader state: leader="Node at 7f3ab57c-bef2-3ed9-000f-a392304c44a0 [Leader]"
> testing.go:149: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:19.917Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: entering leader state: leader="Node at 7f3ab57c-bef2-3ed9-000f-a392304c44a0 [Leader]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:20.022Z [INFO] stable state for Leader reached at 2020-12-08 22:13:19.917726222 +0000 UTC m=+11.474886802 (1 nodes), 55.820713ms from start of poll, 56.001138ms from cluster start. Timeout at 2020-12-08 22:13:20.022850338 +0000 UTC m=+11.580010914, 105.124112ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:20.022Z [INFO] stable state for Leader reached at 2020-12-08 22:13:19.917726222 +0000 UTC m=+11.474886802 (1 nodes), 55.820713ms from start of poll, 56.001138ms from cluster start. Timeout at 2020-12-08 22:13:20.022850338 +0000 UTC m=+11.580010914, 105.124112ms after stability
> testLoggerAdapter verbose: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:20.024Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: starting snapshot up to: index=102
> testing.go:149: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:20.024Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: starting snapshot up to: index=102
> testing.go:153: 2020-12-08T22:13:20.024Z [INFO] creating new snapshot: path=/tmp/raft065915082/snapshots/2-102-1607465600024.tmp
> testLoggerAdapter verbose: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:20.194Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: compacting logs: from=1 to=92
> testing.go:149: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:20.194Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: compacting logs: from=1 to=92
> testLoggerAdapter verbose: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:20.194Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: snapshot complete up to: index=102
> testing.go:149: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:20.194Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: snapshot complete up to: index=102
> testLoggerAdapter verbose: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:20.194Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f3ab57c-bef2-3ed9-000f-a392304c44a0 Address:7f3ab57c-bef2-3ed9-000f-a392304c44a0}]"
> testing.go:149: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:20.194Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f3ab57c-bef2-3ed9-000f-a392304c44a0 Address:7f3ab57c-bef2-3ed9-000f-a392304c44a0}]"
> --- PASS: TestRaft_NoRestoreOnStart (0.33s)
> === RUN TestRaft_SnapshotRestore_PeerChange
> raft_test.go:1026: Disabled unreliable test
> testLoggerAdapter verbose: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:20.194Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: entering follower state: follower="Node at 7f3ab57c-bef2-3ed9-000f-a392304c44a0 [Follower]" leader=
> --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
> === RUN TestRaft_AutoSnapshot
> raft_test.go:1132: Disabled unreliable test
> --- SKIP: TestRaft_AutoSnapshot (0.00s)
> === RUN TestRaft_UserSnapshot
> testLoggerAdapter verbose: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.195Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dabe5a9b-eaaa-78b5-e319-57ddd206998d Address:dabe5a9b-eaaa-78b5-e319-57ddd206998d}]"
> testing.go:149: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.195Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dabe5a9b-eaaa-78b5-e319-57ddd206998d Address:dabe5a9b-eaaa-78b5-e319-57ddd206998d}]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:20.195Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: cluster: 2020-12-08T22:13:20.195Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.195Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: entering follower state: follower="Node at dabe5a9b-eaaa-78b5-e319-57ddd206998d [Follower]" leader=
> testing.go:149: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.195Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: entering follower state: follower="Node at dabe5a9b-eaaa-78b5-e319-57ddd206998d [Follower]" leader=
> testLoggerAdapter verbose: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:20.248Z [WARN] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:20.248Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: entering candidate state: node="Node at 7f3ab57c-bef2-3ed9-000f-a392304c44a0 [Candidate]" term=3
> testLoggerAdapter verbose: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:20.248Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: election won: tally=1
> testLoggerAdapter verbose: server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: 2020-12-08T22:13:20.248Z [INFO] server-7f3ab57c-bef2-3ed9-000f-a392304c44a0: entering leader state: leader="Node at 7f3ab57c-bef2-3ed9-000f-a392304c44a0 [Leader]"
> testLoggerAdapter verbose: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.254Z [WARN] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.254Z [WARN] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.254Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: entering candidate state: node="Node at dabe5a9b-eaaa-78b5-e319-57ddd206998d [Candidate]" term=2
> testing.go:149: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.254Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: entering candidate state: node="Node at dabe5a9b-eaaa-78b5-e319-57ddd206998d [Candidate]" term=2
> testLoggerAdapter verbose: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.254Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: election won: tally=1
> testing.go:149: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.254Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: election won: tally=1
> testLoggerAdapter verbose: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.254Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: entering leader state: leader="Node at dabe5a9b-eaaa-78b5-e319-57ddd206998d [Leader]"
> testing.go:149: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.254Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: entering leader state: leader="Node at dabe5a9b-eaaa-78b5-e319-57ddd206998d [Leader]"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:20.359Z [INFO] stable state for Leader reached at 2020-12-08 22:13:20.254239181 +0000 UTC m=+11.811399753 (1 nodes), 58.987578ms from start of poll, 59.158446ms from cluster start. Timeout at 2020-12-08 22:13:20.359348027 +0000 UTC m=+11.916508591, 105.108838ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:20.359Z [INFO] stable state for Leader reached at 2020-12-08 22:13:20.254239181 +0000 UTC m=+11.811399753 (1 nodes), 58.987578ms from start of poll, 59.158446ms from cluster start. Timeout at 2020-12-08 22:13:20.359348027 +0000 UTC m=+11.916508591, 105.108838ms after stability
> testLoggerAdapter verbose: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.359Z [ERROR] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: failed to take snapshot: error="nothing new to snapshot"
> testing.go:149: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.359Z [ERROR] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: failed to take snapshot: error="nothing new to snapshot"
> testLoggerAdapter verbose: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.359Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: starting snapshot up to: index=12
> testing.go:149: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.359Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: starting snapshot up to: index=12
> testing.go:153: 2020-12-08T22:13:20.359Z [INFO] creating new snapshot: path=/tmp/raft705135756/snapshots/2-12-1607465600359.tmp
> testLoggerAdapter verbose: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.478Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: compacting logs: from=1 to=2
> testing.go:149: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.478Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: compacting logs: from=1 to=2
> testLoggerAdapter verbose: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.478Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: snapshot complete up to: index=12
> testing.go:149: server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: 2020-12-08T22:13:20.478Z [INFO] server-dabe5a9b-eaaa-78b5-e319-57ddd206998d: snapshot complete up to: index=12
> --- PASS: TestRaft_UserSnapshot (0.28s)
> === RUN TestRaft_UserRestore
> testLoggerAdapter verbose: server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8: 2020-12-08T22:13:20.480Z [INFO] server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8 Address:6f1be988-b9f6-6f45-fd12-bc330f0a78f8} {Suffrage:Voter ID:server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39 Address:a2228fa1-6313-0f9b-a48b-c77c09ef0e39} {Suffrage:Voter ID:server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f Address:21e73cca-eb8b-115a-3a4e-2f42ddfc230f}]"
> testing.go:149: server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8: 2020-12-08T22:13:20.480Z [INFO] server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8 Address:6f1be988-b9f6-6f45-fd12-bc330f0a78f8} {Suffrage:Voter ID:server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39 Address:a2228fa1-6313-0f9b-a48b-c77c09ef0e39} {Suffrage:Voter ID:server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f Address:21e73cca-eb8b-115a-3a4e-2f42ddfc230f}]"
> testLoggerAdapter verbose: server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8: 2020-12-08T22:13:20.480Z [INFO] server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8: entering follower state: follower="Node at 6f1be988-b9f6-6f45-fd12-bc330f0a78f8 [Follower]" leader=
> testing.go:149: server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8: 2020-12-08T22:13:20.480Z [INFO] server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8: entering follower state: follower="Node at 6f1be988-b9f6-6f45-fd12-bc330f0a78f8 [Follower]" leader=
> testLoggerAdapter verbose: server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39: 2020-12-08T22:13:20.480Z [INFO] server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8 Address:6f1be988-b9f6-6f45-fd12-bc330f0a78f8} {Suffrage:Voter ID:server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39 Address:a2228fa1-6313-0f9b-a48b-c77c09ef0e39} {Suffrage:Voter ID:server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f Address:21e73cca-eb8b-115a-3a4e-2f42ddfc230f}]"
> testing.go:149: server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39: 2020-12-08T22:13:20.480Z [INFO] server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8 Address:6f1be988-b9f6-6f45-fd12-bc330f0a78f8} {Suffrage:Voter ID:server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39 Address:a2228fa1-6313-0f9b-a48b-c77c09ef0e39} {Suffrage:Voter ID:server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f Address:21e73cca-eb8b-115a-3a4e-2f42ddfc230f}]"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.480Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8 Address:6f1be988-b9f6-6f45-fd12-bc330f0a78f8} {Suffrage:Voter ID:server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39 Address:a2228fa1-6313-0f9b-a48b-c77c09ef0e39} {Suffrage:Voter ID:server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f Address:21e73cca-eb8b-115a-3a4e-2f42ddfc230f}]"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.480Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8 Address:6f1be988-b9f6-6f45-fd12-bc330f0a78f8} {Suffrage:Voter ID:server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39 Address:a2228fa1-6313-0f9b-a48b-c77c09ef0e39} {Suffrage:Voter ID:server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f Address:21e73cca-eb8b-115a-3a4e-2f42ddfc230f}]"
> testLoggerAdapter verbose: server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39: 2020-12-08T22:13:20.480Z [INFO] server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39: entering follower state: follower="Node at a2228fa1-6313-0f9b-a48b-c77c09ef0e39 [Follower]" leader=
> testing.go:149: server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39: 2020-12-08T22:13:20.480Z [INFO] server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39: entering follower state: follower="Node at a2228fa1-6313-0f9b-a48b-c77c09ef0e39 [Follower]" leader=
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:20.480Z [INFO] starting stability test: raft-state=Leader
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.480Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: entering follower state: follower="Node at 21e73cca-eb8b-115a-3a4e-2f42ddfc230f [Follower]" leader=
> testing.go:149: cluster: 2020-12-08T22:13:20.480Z [INFO] starting stability test: raft-state=Leader
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.480Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: entering follower state: follower="Node at 21e73cca-eb8b-115a-3a4e-2f42ddfc230f [Follower]" leader=
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [WARN] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: heartbeat timeout reached, starting election: last-leader=
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [WARN] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: heartbeat timeout reached, starting election: last-leader=
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: entering candidate state: node="Node at 21e73cca-eb8b-115a-3a4e-2f42ddfc230f [Candidate]" term=2
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: entering candidate state: node="Node at 21e73cca-eb8b-115a-3a4e-2f42ddfc230f [Candidate]" term=2
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: election won: tally=2
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: election won: tally=2
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: entering leader state: leader="Node at 21e73cca-eb8b-115a-3a4e-2f42ddfc230f [Leader]"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: entering leader state: leader="Node at 21e73cca-eb8b-115a-3a4e-2f42ddfc230f [Leader]"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: added peer, starting replication: peer=server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: added peer, starting replication: peer=server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: added peer, starting replication: peer=server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: added peer, starting replication: peer=server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: pipelining replication: peer="{Voter server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39 a2228fa1-6313-0f9b-a48b-c77c09ef0e39}"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: pipelining replication: peer="{Voter server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39 a2228fa1-6313-0f9b-a48b-c77c09ef0e39}"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: pipelining replication: peer="{Voter server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8 6f1be988-b9f6-6f45-fd12-bc330f0a78f8}"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.534Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: pipelining replication: peer="{Voter server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8 6f1be988-b9f6-6f45-fd12-bc330f0a78f8}"
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:20.639Z [INFO] stable state for Leader reached at 2020-12-08 22:13:20.534400908 +0000 UTC m=+12.091561474 (1 nodes), 53.59037ms from start of poll, 54.076972ms from cluster start. Timeout at 2020-12-08 22:13:20.639495885 +0000 UTC m=+12.196656451, 105.094977ms after stability
> testing.go:149: cluster: 2020-12-08T22:13:20.639Z [INFO] stable state for Leader reached at 2020-12-08 22:13:20.534400908 +0000 UTC m=+12.091561474 (1 nodes), 53.59037ms from start of poll, 54.076972ms from cluster start. Timeout at 2020-12-08 22:13:20.639495885 +0000 UTC m=+12.196656451, 105.094977ms after stability
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.639Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: starting snapshot up to: index=12
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.639Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: starting snapshot up to: index=12
> testing.go:153: 2020-12-08T22:13:20.639Z [INFO] creating new snapshot: path=/tmp/raft033519538/snapshots/2-12-1607465600639.tmp
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.783Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: snapshot complete up to: index=12
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.783Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: snapshot complete up to: index=12
> testing.go:153: 2020-12-08T22:13:20.785Z [INFO] creating new snapshot: path=/tmp/raft033519538/snapshots/2-23-1607465600785.tmp
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.900Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: copied to local snapshot: bytes=71
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.900Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: copied to local snapshot: bytes=71
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.900Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: restored user snapshot: index=1
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.900Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: restored user snapshot: index=1
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.900Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to get log: index=23 error="log not found"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.900Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to get log: index=23 error="log not found"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.900Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to get log: index=23 error="log not found"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.900Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to get log: index=23 error="log not found"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.901Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: aborting pipeline replication: peer="{Voter server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8 6f1be988-b9f6-6f45-fd12-bc330f0a78f8}"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.901Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: aborting pipeline replication: peer="{Voter server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8 6f1be988-b9f6-6f45-fd12-bc330f0a78f8}"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.901Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: aborting pipeline replication: peer="{Voter server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39 a2228fa1-6313-0f9b-a48b-c77c09ef0e39}"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.901Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: aborting pipeline replication: peer="{Voter server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39 a2228fa1-6313-0f9b-a48b-c77c09ef0e39}"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.907Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to get log: index=23 error="log not found"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.907Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to get log: index=23 error="log not found"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.907Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to get log: index=23 error="log not found"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.907Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to get log: index=23 error="log not found"
> testing.go:153: 2020-12-08T22:13:20.908Z [INFO] creating new snapshot: path=/tmp/raft170116736/snapshots/2-23-1607465600908.tmp
> testing.go:153: 2020-12-08T22:13:20.908Z [INFO] creating new snapshot: path=/tmp/raft984975774/snapshots/2-23-1607465600908.tmp
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.956Z [WARN] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to contact: server-id=server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8 time=50.051759ms
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.956Z [WARN] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to contact: server-id=server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8 time=50.051759ms
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.956Z [WARN] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to contact: server-id=server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39 time=53.342325ms
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.956Z [WARN] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to contact: server-id=server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39 time=53.342325ms
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.956Z [WARN] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to contact quorum of nodes, stepping down
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.956Z [WARN] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to contact quorum of nodes, stepping down
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.956Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: entering follower state: follower="Node at 21e73cca-eb8b-115a-3a4e-2f42ddfc230f [Follower]" leader=
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.956Z [INFO] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: entering follower state: follower="Node at 21e73cca-eb8b-115a-3a4e-2f42ddfc230f [Follower]" leader=
> testLoggerAdapter verbose: cluster: 2020-12-08T22:13:20.956Z [ERROR] Restore failed: leadership lost while committing log
> testing.go:149: cluster: 2020-12-08T22:13:20.956Z [ERROR] Restore failed: leadership lost while committing log
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.959Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=a2228fa1-6313-0f9b-a48b-c77c09ef0e39 error="command timed out"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.959Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=a2228fa1-6313-0f9b-a48b-c77c09ef0e39 error="command timed out"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.965Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=6f1be988-b9f6-6f45-fd12-bc330f0a78f8 error="command timed out"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:20.965Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=6f1be988-b9f6-6f45-fd12-bc330f0a78f8 error="command timed out"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.026Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=a2228fa1-6313-0f9b-a48b-c77c09ef0e39 error="command timed out"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.026Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=a2228fa1-6313-0f9b-a48b-c77c09ef0e39 error="command timed out"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.033Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=6f1be988-b9f6-6f45-fd12-bc330f0a78f8 error="command timed out"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.033Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=6f1be988-b9f6-6f45-fd12-bc330f0a78f8 error="command timed out"
> testLoggerAdapter verbose: server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39: 2020-12-08T22:13:21.052Z [INFO] server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39: copied to local snapshot: bytes=71
> testing.go:149: server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39: 2020-12-08T22:13:21.052Z [INFO] server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39: copied to local snapshot: bytes=71
> testLoggerAdapter verbose: server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8: 2020-12-08T22:13:21.052Z [INFO] server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8: copied to local snapshot: bytes=71
> testing.go:149: server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8: 2020-12-08T22:13:21.052Z [INFO] server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8: copied to local snapshot: bytes=71
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.052Z [WARN] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: installSnapshot rejected to: peer="{Voter server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39 a2228fa1-6313-0f9b-a48b-c77c09ef0e39}"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.052Z [WARN] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: installSnapshot rejected to: peer="{Voter server-a2228fa1-6313-0f9b-a48b-c77c09ef0e39 a2228fa1-6313-0f9b-a48b-c77c09ef0e39}"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.093Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=a2228fa1-6313-0f9b-a48b-c77c09ef0e39 error="command timed out"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.093Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=a2228fa1-6313-0f9b-a48b-c77c09ef0e39 error="command timed out"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.101Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=6f1be988-b9f6-6f45-fd12-bc330f0a78f8 error="command timed out"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.101Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=6f1be988-b9f6-6f45-fd12-bc330f0a78f8 error="command timed out"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.179Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=6f1be988-b9f6-6f45-fd12-bc330f0a78f8 error="command timed out"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.179Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=6f1be988-b9f6-6f45-fd12-bc330f0a78f8 error="command timed out"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.277Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=6f1be988-b9f6-6f45-fd12-bc330f0a78f8 error="command timed out"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.277Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=6f1be988-b9f6-6f45-fd12-bc330f0a78f8 error="command timed out"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.408Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to install snapshot: id=2-23-1607465600785 error="command timed out"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.408Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to install snapshot: id=2-23-1607465600785 error="command timed out"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.408Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to send snapshot to: peer="{Voter server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8 6f1be988-b9f6-6f45-fd12-bc330f0a78f8}" error="command timed out"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.408Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to send snapshot to: peer="{Voter server-6f1be988-b9f6-6f45-fd12-bc330f0a78f8 6f1be988-b9f6-6f45-fd12-bc330f0a78f8}" error="command timed out"
> testLoggerAdapter verbose: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.414Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=6f1be988-b9f6-6f45-fd12-bc330f0a78f8 error="command timed out"
> testing.go:149: server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: 2020-12-08T22:13:21.414Z [ERROR] server-21e73cca-eb8b-115a-3a4e-2f42ddfc230f: failed to heartbeat to: peer=6f1be988-b9f6-6f45-fd12-bc330f0a78f8 error="command timed out"
> panic: timed out waiting for shutdown
>
> goroutine 1030 [running]:
> github.com/hashicorp/raft.(*cluster).Close.func1()
> /<<PKGBUILDDIR>>/obj-powerpc64le-linux-gnu/src/github.com/hashicorp/raft/testing.go:260 +0x3c
> created by time.goFunc
> /usr/lib/go-1.15/src/time/sleep.go:167 +0x44
> FAIL github.com/hashicorp/raft 17.519s
> ? github.com/hashicorp/raft/bench [no test files]
> FAIL
> dh_auto_test: error: cd obj-powerpc64le-linux-gnu && go test -vet=off -v -p 160 github.com/hashicorp/raft github.com/hashicorp/raft/bench returned exit code 1
The full build log is available from:
http://qa-logs.debian.net/2020/12/09/golang-github-hashicorp-raft_1.1.2-1_unstable.log
A list of current common problems and possible solutions is available at
http://wiki.debian.org/qa.debian.org/FTBFS . You're welcome to contribute!
If you reassign this bug to another package, please marking it as 'affects'-ing
this package. See https://www.debian.org/Bugs/server-control#affects
If you fail to reproduce this, please provide a build log and diff it with me
so that we can identify if something relevant changed in the meantime.
About the archive rebuild: The rebuild was done on a Power8 cluster part of the
Grid'5000 testbed. Hardware specs: https://www.grid5000.fr/w/Grenoble:Hardware#drac
More information about the Pkg-go-maintainers
mailing list