Skip to content

Commit

Permalink
Switch to pure-go zstd decoder for snapshot imports
Browse files Browse the repository at this point in the history
The benchmarking performed as part of #12852 shows that the pure-go
implementation of zstd decoder is fast-enough for snapshot import.

Switch to it motivated by the desire to reduce CGO dependencies across
Lotus.

Fixes #12852
  • Loading branch information
masih committed Jan 29, 2025
1 parent 99dcb12 commit 7ec3245
Show file tree
Hide file tree
Showing 3 changed files with 6 additions and 10 deletions.
13 changes: 6 additions & 7 deletions cli/lotus/daemon.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ import (
"runtime/pprof"
"strings"

"github.com/DataDog/zstd"
"github.com/cheggaaa/pb/v3"
metricsprom "github.com/ipfs/go-metrics-prometheus"
"github.com/klauspost/compress/zstd"
"github.com/mitchellh/go-homedir"
"github.com/multiformats/go-multiaddr"
"github.com/urfave/cli/v2"
Expand Down Expand Up @@ -579,12 +579,11 @@ func ImportChain(ctx context.Context, r repo.Repo, fname string, snapshot bool)
var ir io.Reader = br

if string(header[1:]) == "\xB5\x2F\xFD" { // zstd
zr := zstd.NewReader(br)
defer func() {
if err := zr.Close(); err != nil {
log.Errorw("closing zstd reader", "error", err)
}
}()
zr, err := zstd.NewReader(br)
if err != nil {
return xerrors.Errorf("instantiating zstd reader: %w", err)
}
defer zr.Close()
ir = zr
}

Expand Down
1 change: 0 additions & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ replace github.com/filecoin-project/filecoin-ffi => ./extern/filecoin-ffi // pro
require (
contrib.go.opencensus.io/exporter/prometheus v0.4.2
github.com/BurntSushi/toml v1.3.2
github.com/DataDog/zstd v1.4.5
github.com/GeertJohan/go.rice v1.0.3
github.com/Gurpartap/async v0.0.0-20180927173644-4f7f499dd9ee
github.com/Kubuxu/imtui v0.0.0-20210401140320-41663d68d0fa
Expand Down
2 changes: 0 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,6 @@ github.com/BurntSushi/toml v1.3.2 h1:o7IhLm0Msx3BaB+n3Ag7L8EVlByGnpq14C4YWiu/gL8
github.com/BurntSushi/toml v1.3.2/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/DataDog/zstd v1.4.1/go.mod h1:1jcaCB/ufaK+sKp1NBhlGmpz41jOoPQ35bpF36t7BBo=
github.com/DataDog/zstd v1.4.5 h1:EndNeuB0l9syBZhut0wns3gV1hL8zX8LIu6ZiVHWLIQ=
github.com/DataDog/zstd v1.4.5/go.mod h1:1jcaCB/ufaK+sKp1NBhlGmpz41jOoPQ35bpF36t7BBo=
github.com/GeertJohan/go.incremental v1.0.0 h1:7AH+pY1XUgQE4Y1HcXYaMqAI0m9yrFqo/jt0CW30vsg=
github.com/GeertJohan/go.incremental v1.0.0/go.mod h1:6fAjUhbVuX1KcMD3c8TEgVUqmo4seqhv0i0kdATSkM0=
github.com/GeertJohan/go.rice v1.0.3 h1:k5viR+xGtIhF61125vCE1cmJ5957RQGXG6dmbaWZSmI=
Expand Down

0 comments on commit 7ec3245

Please sign in to comment.