在2022-11-21用户反馈说私有部署的joggle项目出现了经常掉线问题,通过日志排查,问题日志如下:
Nov 21 02:43:25 VM-12-12-ubuntu systemd[1]: Stopped bullet service. Nov 21 02:43:25 VM-12-12-ubuntu systemd[1]: Started bullet service. Nov 21 02:43:25 VM-12-12-ubuntu ngrokd[46886]: Reading configuration file %s /opt/ngrok/conf/server.yml Nov 21 02:43:25 VM-12-12-ubuntu ngrokd[46886]: Reading configuration file %s /opt/ngrok/conf/server-tunnels.yml Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: panic: concurrent write to websocket connection Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: goroutine 188 [running]: Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc00031c720, 0x1, {0x0, 0x7fa4cbc0c5b8, 0x0}) Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: /home/marker/go/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:610 +0x52b Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: github.com/gorilla/websocket.(*messageWriter).Close(0x0) Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: /home/marker/go/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:724 +0x45 Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: github.com/gorilla/websocket.(*Conn).beginMessage(0xc0000c22c0, 0xc0000b4870, 0x2) Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: /home/marker/go/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:473 +0x42 Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: github.com/gorilla/websocket.(*Conn).NextWriter(0xc0000c22c0, 0x2) Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: /home/marker/go/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:513 +0x45 Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: github.com/gorilla/websocket.(*Conn).WriteMessage(0xc0000b46c0, 0x7416e0, {0xc00023c000, 0x0, 0x768e60}) Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: /home/marker/go/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:766 +0x165 Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: bullet/ngrok/server.(*WebsocketControl).SendMessage(0xc000286690, {0x82f378, 0xc0000b46c0}) Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: /opt/build/ngrok/ngrok/server/websocket.go:127 +0xf9 Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: bullet/ngrok/server.uploadMetrics4Websocket(0xc0002d8000, 0xc0002707b8) Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: /opt/build/ngrok/ngrok/server/tunnel.go:431 +0x6e Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: created by bullet/ngrok/server.(*Tunnel).HandlePublicConnection Nov 21 02:49:00 VM-12-12-ubuntu ngrokd[46886]: /opt/build/ngrok/ngrok/server/tunnel.go:421 +0x56f Nov 21 02:49:00 VM-12-12-ubuntu systemd[1]: joggled.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Nov 21 02:49:00 VM-12-12-ubuntu systemd[1]: joggled.service: Failed with result 'exit-code'. Nov 21 02:49:01 VM-12-12-ubuntu systemd[1]: joggled.service: Scheduled restart job, restart counter is at 33.
是在上报链接相关数据出现了并发写入导致Wesocket出现异常然后直接panic
目前项目在使用github.com/gorilla/websocket库。
报”concurrent write to websocket connection”的错
原因是该库不支持并发向websocket链接写数据,详情请参考:https://github.com/gorilla/websocket/issues/380
使用golang中的锁来解决。
type WebsocketControl struct { *websocket.Conn wsMux sync.RWMutex } websocketControl.wsMux.Lock() //加锁 err = websocketControl.conn.WriteMessage(websocket.TextMessage,msgByte) websocketControl.wsMux.Unlock() //解锁
来源: 雨林博客(pancha.oss-cn-chengdu.aliyuncs.com)