Commit 7fa77eb4 authored by craig[bot]'s avatar craig[bot]

Merge #50768 #50831 #50839

50768: colexec: use Get methods on native type slices r=jordanlewis a=jordanlewis

Also, add //gcassert:inline assertions on all Get methods but the
DatumVec ones and the gcassert linter, to prove that all of these Get
methods are inlined.

Here are some benchmarks. I didn't run them all, because I believe that
the guaranteed inlining proven by gcassert implies that any benchmark
differences must be due to "binary layout" or other ghosts.

```
Distinct/Ordered/hasNulls=false/newTupleProbability=0.010/rows=65536/cols=2/ordCols=2-24     468µs ± 0%     462µs ± 0%  -1.26%  (p=0.000 n=10+10)

name                                                                                      old speed      new speed      delta
Distinct/Ordered/hasNulls=false/newTupleProbability=0.010/rows=65536/cols=2/ordCols=2-24  2.24GB/s ± 0%  2.27GB/s ± 0%  +1.28%  (p=0.000 n=10+10)

CastOp/useSel=true/hasNulls=true/int_to_float-24      5.18µs ± 2%    4.60µs ± 1%  -11.21%  (p=0.000 n=9+9)
CastOp/useSel=true/hasNulls=false/int_to_float-24     1.06µs ± 7%    1.03µs ± 5%     ~     (p=0.101 n=10+10)
CastOp/useSel=false/hasNulls=true/int_to_float-24     5.33µs ± 1%    5.47µs ± 2%   +2.53%  (p=0.000 n=9+10)
CastOp/useSel=false/hasNulls=false/int_to_float-24    1.08µs ± 0%    1.08µs ± 0%     ~     (p=0.136 n=10+10)

name                                                old speed      new speed      delta
CastOp/useSel=true/hasNulls=true/int_to_float-24    1.58GB/s ± 2%  1.78GB/s ± 1%  +12.62%  (p=0.000 n=9+9)
CastOp/useSel=true/hasNulls=false/int_to_float-24   7.73GB/s ± 7%  7.93GB/s ± 5%     ~     (p=0.089 n=10+10)
CastOp/useSel=false/hasNulls=true/int_to_float-24   1.54GB/s ± 1%  1.50GB/s ± 2%   -2.46%  (p=0.000 n=9+10)
CastOp/useSel=false/hasNulls=false/int_to_float-24  7.60GB/s ± 0%  7.59GB/s ± 0%     ~     (p=0.143 n=10+10)
```

Release note: None

50831: kv/kvserver: Remove buffering from SSTSnapshotStorageFile r=itsbilal a=itsbilal

When investigating a durability violation around ingested sstables
that disappeared after a power outage, we realized that
SSTSnapshotStorageFile does not flush its buffer before doing
a Sync(). When used with a pebble SSTWriter, we expect Sync()
to ensure all bytes written to that file are synced to disk,
which is not what was happening here for the last couple bytes
written to the buffer.

This change removes buffering from SSTSnapshotStorageFile
as pebble's SST writer already does its own buffering. This
makes Sync() do what it is expected to do.

Release note (bug fix): Fix a bug where a badly timed power outage
or system crash could result in an error upon process restart.

50839: partitionccl,kvserver: skip some flaky tests r=rytaft a=rytaft

This commit skips 4 tests as part of the test-infra-team flaky
test cleanup:

TestGossipHandlesReplacedNode
TestStoreRangeMergeConcurrentRequests
TestCheckConsistencyInconsistent
TestRepartitioning

Informs #50024
Informs #50795
Informs #50830
Informs #49112

Release note: None
Co-authored-by: default avatarYahor Yuzefovich <[email protected]>
Co-authored-by: default avatarJordan Lewis <[email protected]>
Co-authored-by: default avatarBilal Akhtar <[email protected]>
Co-authored-by: default avatarRebecca Taft <[email protected]>
......@@ -92,6 +92,7 @@ require (
github.com/jackc/pgx v3.6.2+incompatible
github.com/jaegertracing/jaeger v1.17.0
github.com/jcmturner/gofork v1.0.0 // indirect
github.com/jordanlewis/gcassert v0.0.0-20200630054945-8ff878e72753
github.com/kevinburke/go-bindata v3.13.0+incompatible
github.com/kisielk/errcheck v1.2.0
github.com/kisielk/gotool v1.0.0
......@@ -138,13 +139,14 @@ require (
github.com/shopspring/decimal v1.2.0 // indirect
github.com/spf13/cobra v0.0.5
github.com/spf13/pflag v1.0.5
github.com/stretchr/testify v1.5.1
github.com/stretchr/testify v1.6.1
github.com/twpayne/go-geom v1.3.0
github.com/wadey/gocovmerge v0.0.0-20160331181800-b5bfa59ec0ad
go.etcd.io/etcd v0.0.0-00010101000000-000000000000
golang.org/x/crypto v0.0.0-20200323165209-0ec3e9974c59
golang.org/x/exp v0.0.0-20200513190911-00229845015e
golang.org/x/lint v0.0.0-20200130185559-910be7a94367
golang.org/x/mod v0.3.0 // indirect
golang.org/x/net v0.0.0-20200602114024-627f9648deb9
golang.org/x/oauth2 v0.0.0-20190115181402-5dab4167f31c
golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852
......@@ -152,7 +154,7 @@ require (
golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae
golang.org/x/text v0.3.2
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4
golang.org/x/tools v0.0.0-20200509030707-2212a7e161a5
golang.org/x/tools v0.0.0-20200626171337-aa94e735be7f
google.golang.org/api v0.1.0
google.golang.org/genproto v0.0.0-20200218151345-dad8c97a84f5 // indirect
google.golang.org/grpc v1.29.1
......
......@@ -376,6 +376,8 @@ github.com/jcmturner/gofork v1.0.0/go.mod h1:MK8+TM0La+2rjBD4jE12Kj1pCCxK7d2LK/U
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af h1:pmfjZENx5imkbgOkpRUYLnmbU7UEFbjtDA2hxJ1ichM=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
github.com/jordanlewis/gcassert v0.0.0-20200630054945-8ff878e72753 h1:FXipXC++4uw5v41AxCRFoxoGnQJBuEsObAy9q3a8uww=
github.com/jordanlewis/gcassert v0.0.0-20200630054945-8ff878e72753/go.mod h1:Qc93dJSt1iLNJCuG9Gy9ds0k/oh4ckhXGkD4AI3cEtM=
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.7/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
......@@ -592,6 +594,8 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.5.1 h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
github.com/twpayne/go-geom v1.3.0 h1:jwhmbnQv5VTfAgohevKBuzisQSdXGjfquZ7FC8aVUzU=
github.com/twpayne/go-geom v1.3.0/go.mod h1:90yvs0wf/gyT5eQ9W4v5WOZ9w/Xnrj5RMlA9XNKqxyA=
......@@ -654,6 +658,8 @@ golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzB
golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0 h1:KU7oHjnv3XNWfa5COkzUifxZmxp1TyI7ImMXqFxLwvQ=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20180218175443-cbe0f9307d01/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
......@@ -736,6 +742,8 @@ golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapK
golang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200509030707-2212a7e161a5 h1:MeC2gMlMdkd67dn17MEby3rGXRxZtWeiRXOnISfTQ74=
golang.org/x/tools v0.0.0-20200509030707-2212a7e161a5/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200626171337-aa94e735be7f h1:JcoF/bowzCDI+MXu1yLqQGNO3ibqWsWq+Sk7pOT218w=
golang.org/x/tools v0.0.0-20200626171337-aa94e735be7f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
......@@ -800,6 +808,8 @@ gopkg.in/src-d/go-billy.v4 v4.3.0 h1:KtlZ4c1OWbIs4jCv5ZXrTqG8EQocr0g/d4DjNg70aek
gopkg.in/src-d/go-billy.v4 v4.3.0/go.mod h1:tm33zBoOwxjYHZIE+OV8bxTWFMJLrconzFMd38aARFk=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gotest.tools v2.2.0+incompatible h1:VsBPFP1AI068pPrMxtb/S8Zkgf9xEmTLJjfM+P5UIEo=
gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw=
honnef.co/go/tools v0.0.0-20180728063816-88497007e858/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
......
......@@ -1284,6 +1284,9 @@ func TestSelectPartitionExprs(t *testing.T) {
func TestRepartitioning(t *testing.T) {
defer leaktest.AfterTest(t)()
// Skipping as part of test-infra-team flaky test cleanup.
t.Skip("https://github.com/cockroachdb/cockroach/issues/49112")
// This test configures many sub-tests and is too slow to run under nightly
// race stress.
if testutils.NightlyStress() && util.RaceEnabled {
......
......@@ -40,3 +40,27 @@ type Times []time.Time
// Durations is a slice of duration.Duration.
type Durations []duration.Duration
// Get returns the element at index idx of the vector.
func (c Bools) Get(idx int) bool { return c[idx] }
// Get returns the element at index idx of the vector.
func (c Int16s) Get(idx int) int16 { return c[idx] }
// Get returns the element at index idx of the vector.
func (c Int32s) Get(idx int) int32 { return c[idx] }
// Get returns the element at index idx of the vector.
func (c Int64s) Get(idx int) int64 { return c[idx] }
// Get returns the element at index idx of the vector.
func (c Float64s) Get(idx int) float64 { return c[idx] }
// Get returns the element at index idx of the vector.
func (c Decimals) Get(idx int) apd.Decimal { return c[idx] }
// Get returns the element at index idx of the vector.
func (c Times) Get(idx int) time.Time { return c[idx] }
// Get returns the element at index idx of the vector.
func (c Durations) Get(idx int) duration.Duration { return c[idx] }
This diff was suppressed by a .gitattributes entry.
......@@ -1518,6 +1518,9 @@ func TestStoreRangeMergeRHSLeaseExpiration(t *testing.T) {
func TestStoreRangeMergeConcurrentRequests(t *testing.T) {
defer leaktest.AfterTest(t)()
// Skipping as part of test-infra-team flaky test cleanup.
t.Skip("https://github.com/cockroachdb/cockroach/issues/50795")
ctx := context.Background()
storeCfg := kvserver.TestStoreConfig(nil)
storeCfg.TestingKnobs.DisableSplitQueue = true
......
......@@ -211,6 +211,9 @@ func TestCheckConsistencyReplay(t *testing.T) {
func TestCheckConsistencyInconsistent(t *testing.T) {
defer leaktest.AfterTest(t)()
// Skipping as part of test-infra-team flaky test cleanup.
t.Skip("https://github.com/cockroachdb/cockroach/issues/50830")
const numStores = 3
testKnobs := kvserver.StoreTestingKnobs{}
......
......@@ -140,6 +140,10 @@ func TestGossipFirstRange(t *testing.T) {
// restarted after losing its data) without the cluster breaking.
func TestGossipHandlesReplacedNode(t *testing.T) {
defer leaktest.AfterTest(t)()
// Skipping as part of test-infra-team flaky test cleanup.
t.Skip("https://github.com/cockroachdb/cockroach/issues/50024")
if testing.Short() {
// As of Nov 2018 it takes 3.6s.
t.Skip("short")
......
......@@ -81,18 +81,18 @@ func (s *SSTSnapshotStorageScratch) createDir() error {
// NewFile adds another file to SSTSnapshotStorageScratch. This file is lazily
// created when the file is written to the first time. A nonzero value for
// chunkSize buffers up writes until the buffer is greater than chunkSize.
// syncSize calls Sync after syncSize bytes have been written since last sync.
func (s *SSTSnapshotStorageScratch) NewFile(
ctx context.Context, chunkSize int64,
ctx context.Context, syncSize int64,
) (*SSTSnapshotStorageFile, error) {
id := len(s.ssts)
filename := s.filename(id)
s.ssts = append(s.ssts, filename)
f := &SSTSnapshotStorageFile{
scratch: s,
filename: filename,
ctx: ctx,
chunkSize: chunkSize,
scratch: s,
filename: filename,
ctx: ctx,
syncSize: syncSize,
}
return f, nil
}
......@@ -116,6 +116,9 @@ func (s *SSTSnapshotStorageScratch) WriteSST(ctx context.Context, data []byte) e
if _, err := f.Write(data); err != nil {
return err
}
if err := f.Sync(); err != nil {
return err
}
return f.Close()
}
......@@ -132,13 +135,13 @@ func (s *SSTSnapshotStorageScratch) Clear() error {
// SSTSnapshotStorageFile is an SST file managed by a
// SSTSnapshotStorageScratch.
type SSTSnapshotStorageFile struct {
scratch *SSTSnapshotStorageScratch
created bool
file fs.File
filename string
ctx context.Context
chunkSize int64
buffer []byte
scratch *SSTSnapshotStorageScratch
created bool
file fs.File
filename string
ctx context.Context
bytesSinceSync int64
syncSize int64
}
func (f *SSTSnapshotStorageFile) openFile() error {
......@@ -173,23 +176,18 @@ func (f *SSTSnapshotStorageFile) Write(contents []byte) (int, error) {
return 0, err
}
limitBulkIOWrite(f.ctx, f.scratch.storage.limiter, len(contents))
if f.chunkSize > 0 {
if int64(len(contents)+len(f.buffer)) < f.chunkSize {
// Don't write to file yet - buffer write until next time.
f.buffer = append(f.buffer, contents...)
return len(contents), nil
} else if len(f.buffer) > 0 {
// Write buffered writes and then empty the buffer.
if _, err := f.file.Write(f.buffer); err != nil {
return 0, err
}
f.buffer = f.buffer[:0]
}
}
if _, err := f.file.Write(contents); err != nil {
return 0, err
}
return len(contents), f.file.Sync()
var err error
if f.syncSize > 0 {
f.bytesSinceSync += int64(len(contents))
if f.bytesSinceSync >= f.syncSize {
f.bytesSinceSync = 0
err = f.Sync()
}
}
return len(contents), err
}
// Close closes the file. Calling this function multiple times is idempotent.
......@@ -203,13 +201,6 @@ func (f *SSTSnapshotStorageFile) Close() error {
if f.file == nil {
return nil
}
if len(f.buffer) > 0 {
// Write out any buffered data.
if _, err := f.file.Write(f.buffer); err != nil {
return err
}
f.buffer = f.buffer[:0]
}
if err := f.file.Close(); err != nil {
return err
}
......
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
......@@ -451,11 +451,13 @@ func (b *argWidthOverloadBase) GoTypeSliceName() string {
}
func get(family types.Family, target, i string) string {
switch family {
case types.BytesFamily, typeconv.DatumVecCanonicalTypeFamily:
return fmt.Sprintf("%s.Get(%s)", target, i)
// Assert that the .Get call is inlined by Go, except in the case
// of the DatumVec .Get which isn't inlinable.
maybeInline := " //gcassert:inline"
if family == typeconv.DatumVecCanonicalTypeFamily {
maybeInline = ""
}
return fmt.Sprintf("%s[%s]", target, i)
return fmt.Sprintf("%s.Get(%s)%s", target, i, maybeInline)
}
// Get is a function that should only be used in templates.
......
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
......@@ -29,6 +29,7 @@ import (
_ "github.com/cockroachdb/cockroach/pkg/testutils/buildutil"
"github.com/cockroachdb/errors"
"github.com/ghemawat/stream"
"github.com/jordanlewis/gcassert"
"golang.org/x/tools/go/packages"
)
......@@ -1622,6 +1623,18 @@ func TestLint(t *testing.T) {
}
})
t.Run("TestGCAssert", func(t *testing.T) {
t.Parallel()
var buf strings.Builder
if err := gcassert.GCAssert("../../sql/colexec", &buf); err != nil {
t.Fatal(err)
}
output := buf.String()
if len(output) > 0 {
t.Fatalf("failed gcassert:\n%s", output)
}
})
t.Run("TestTypesSlice", func(t *testing.T) {
t.Parallel()
cmd, stderr, filter, err := dirCmd(
......
Subproject commit 9c42e4cee71e0a680e9b05087bce111c96c4aa53
Subproject commit 51544aebe7ad27d8fe0aebdbf92f860ed3d76669
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment