CLN crash and gossip store corruption

[MegalithicBTC]

After running this image of CLN without (apparent) problems for about two weeks....

from elementsproject/lightningd:v24.11

CLN suddenly crashed with ...

lightning_connectd: common/gossmap.c:121: map_copy: Assertion `offset + len <= map->map_size' failed.
lightning_gossipd: common/gossmap.c:121: map_copy: Assertion `offset + len <= map->map_size' failed.
lightning_connectd: FATAL SIGNAL 6 (version v24.11)
lightning_gossipd: FATAL SIGNAL 6 (version v24.11)
0x581daff576d8 send_backtrace
	common/daemon.c:33

A more complete log showing the crash and the creation of gossip_store.corrupt is attached

cln-gossip-store-corrupt.txt

[JssDWt]

**BROKEN** connectd: STATUS_FAIL_INTERNAL_ERROR: FATAL SIGNAL
0x6260c41d8b69 map_copy
  common/gossmap.c:121
0x6260c41d8bab map_be16
  common/gossmap.c:142
0x6260c41daa45 map_catchup
  common/gossmap.c:705
0x6260c41dab95 gossmap_refresh_mayfail
  common/gossmap.c:1192
0x6260c41daca6 gossmap_refresh
  common/gossmap.c:1213
0x6260c41cee32 gossmap_manage_get_gossmap
  gossipd/gossmap_manage.c:1314
0x6260c41d0686 gossmap_manage_new_block
  gossipd/gossmap_manage.c:1221
0x6260c41cbfdd new_blockheight
  gossipd/gossipd.c:473
0x6260c41cc363 recv_req
  gossipd/gossipd.c:584
0x6260c41d6b1d handle_read
  common/daemon_conn.c:35
0x6260c43175b5 next_plan
  ccan/ccan/io/io.c:60
0x6260c4317a40 do_plan
  ccan/ccan/io/io.c:422
0x6260c4317af9 io_ready
  ccan/ccan/io/io.c:439
0x6260c4319446 io_loop
  ccan/ccan/io/poll.c:455
0x6260c41cccf4 main
  gossipd/gossipd.c:665
0x70508e88bd79 ???
  ???:0
0x6260c41c9d99 ???
  ???:0
0xffffffffffffffff ???
  ???:0
lightning_gossipd: FATAL SIGNAL (version v24.11)
0x6260c41d682a send_backtrace
  common/daemon.c:33
0x6260c41e098b status_failed
  common/status.c:221
0x6260c41e0b41 status_backtrace_exit
  common/subdaemon.c:18
0x6260c41d68b8 crashdump
  common/daemon.c:78
0x70508ea6913f ???
  ???:0
0x70508e8a0d51 ???
  ???:0
0x70508e88a536 ???
  ???:0
0x70508e88a40e ???
  ???:0
0x70508e8996d1 ???
  ???:0
0x6260c41d8b69 map_copy
  common/gossmap.c:121
0x6260c41d8bab map_be16
  common/gossmap.c:142
0x6260c41daa45 map_catchup
  common/gossmap.c:705
0x6260c41dab95 gossmap_refresh_mayfail
  common/gossmap.c:1192
0x6260c41daca6 gossmap_refresh
  common/gossmap.c:1213
0x6260c41cee32 gossmap_manage_get_gossmap
  gossipd/gossmap_manage.c:1314
0x6260c41d0686 gossmap_manage_new_block
  gossipd/gossmap_manage.c:1221
0x6260c41cbfdd new_blockheight
  gossipd/gossipd.c:473
0x6260c41cc363 recv_req
  gossipd/gossipd.c:584
0x6260c41d6b1d handle_read
  common/daemon_conn.c:35
0x6260c43175b5 next_plan
  ccan/ccan/io/io.c:60
0x6260c4317a40 do_plan
  ccan/ccan/io/io.c:422
0x6260c4317af9 io_ready
  ccan/ccan/io/io.c:439
0x6260c4319446 io_loop
  ccan/ccan/io/poll.c:455
0x6260c41cccf4 main
  gossipd/gossipd.c:665
0x70508e88bd79 ???
  ???:0
0x6260c41c9d99 ???
  ???:0
0xffffffffffffffff ???
  ???:0
lightningd: connectd failed (exit status 242), exiting.

[MegalithicBTC]

I think it happened again:

lightning_gossipd: gossip_store: get delete entry offset 3771339/105791801 (version v24.11)
0x577413b1182a send_backtrace
  common/daemon.c:33
0x577413b1b98b status_failed
  common/status.c:221
0x577413b08b8b gossip_store_get_with_hdr
  gossipd/gossip_store.c:466
0x577413b0911c gossip_store_set_timestamp
  gossipd/gossip_store.c:592
0x577413b0a7b4 process_channel_update
  gossipd/gossmap_manage.c:777
0x577413b0b10b gossmap_manage_channel_update
  gossipd/gossmap_manage.c:905
0x577413b07942 handle_recv_gossip
  gossipd/gossipd.c:210
0x577413b07a48 connectd_req
  gossipd/gossipd.c:302
0x577413b11b1d handle_read
  common/daemon_conn.c:35
0x577413c525b5 next_plan
  ccan/ccan/io/io.c:60
0x577413c52a40 do_plan
  ccan/ccan/io/io.c:422
0x577413c52af9 io_ready
  ccan/ccan/io/io.c:439
0x577413c54446 io_loop
  ccan/ccan/io/poll.c:455
0x577413b07cf4 main
  gossipd/gossipd.c:665
0x731bcb65dd79 ???
  ???:0
0x577413b04d99 ???
  ???:0
0xffffffffffffffff ???
  ???:0
2025-01-25T15:26:19.784Z **BROKEN** gossipd: gossip_store: get delete entry offset 3771339/105791801 (version v24.11)
2025-01-25T15:26:19.784Z **BROKEN** gossipd: backtrace: common/daemon.c:38 (send_backtrace) 0x577413b11872
2025-01-25T15:26:19.784Z **BROKEN** gossipd: backtrace: common/status.c:221 (status_failed) 0x577413b1b98b
2025-01-25T15:26:19.784Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:466 (gossip_store_get_with_hdr) 0x577413b08b8b
2025-01-25T15:26:19.784Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:592 (gossip_store_set_timestamp) 0x577413b0911c
2025-01-25T15:26:19.785Z **BROKEN** gossipd: backtrace: gossipd/gossmap_manage.c:777 (process_channel_update) 0x577413b0a7b4
2025-01-25T15:26:19.785Z **BROKEN** gossipd: backtrace: gossipd/gossmap_manage.c:905 (gossmap_manage_channel_update) 0x577413b0b10b
2025-01-25T15:26:19.785Z **BROKEN** gossipd: backtrace: gossipd/gossipd.c:210 (handle_recv_gossip) 0x577413b07942
2025-01-25T15:26:19.785Z **BROKEN** gossipd: backtrace: gossipd/gossipd.c:302 (connectd_req) 0x577413b07a48
2025-01-25T15:26:19.785Z **BROKEN** gossipd: backtrace: common/daemon_conn.c:35 (handle_read) 0x577413b11b1d
2025-01-25T15:26:19.785Z **BROKEN** gossipd: backtrace: ccan/ccan/io/io.c:60 (next_plan) 0x577413c525b5
2025-01-25T15:26:19.785Z **BROKEN** gossipd: backtrace: ccan/ccan/io/io.c:422 (do_plan) 0x577413c52a40
2025-01-25T15:26:19.785Z **BROKEN** gossipd: backtrace: ccan/ccan/io/io.c:439 (io_ready) 0x577413c52af9
2025-01-25T15:26:19.785Z **BROKEN** gossipd: backtrace: ccan/ccan/io/poll.c:455 (io_loop) 0x577413c54446
2025-01-25T15:26:19.785Z **BROKEN** gossipd: backtrace: gossipd/gossipd.c:665 (main) 0x577413b07cf4
2025-01-25T15:26:19.785Z **BROKEN** gossipd: backtrace: (null):0 ((null)) 0x731bcb65dd79
2025-01-25T15:26:19.786Z **BROKEN** gossipd: backtrace: (null):0 ((null)) 0x577413b04d99
2025-01-25T15:26:19.786Z **BROKEN** gossipd: backtrace: (null):0 ((null)) 0xffffffffffffffff
2025-01-25T15:26:19.786Z **BROKEN** gossipd: STATUS_FAIL_INTERNAL_ERROR: gossip_store: get delete entry offset 3771339/105791801