Commit d244a6d9 authored by craig[bot]'s avatar craig[bot]

Merge #48542 #48787

48542: build/packer: Update TeamCity agent disk image r=bobvawter a=bobvawter

Several of the upstream packages that our TeamCity builder image depends on
have been removed or obsoleted. Specifically, Oracle Java 8 is no longer
available to download and has been replaced by OpenJDK 11. The golang backport
has been upgraded to a modern version as well.

The golang update necessitates changing a few invocation of the go tool to
unset the GOFLAGS environment variable provided by TeamCity that globally
enables json output.

I have verified that this new image passes the tests on a fork of the "GitHub
CI" suite.

The base disk image size has been increased to 256Gb.  A subsequent change will
be made to the image to enable LRU deletion of Docker images, which have been
contributing to disk-space exhaustion.

Release note: None

48787: cli/interactive_tests: deflake test_missing_log_output r=tbg,petermattis a=knz

Fixes #48413

(Logging is hard!)

The proper fix would be to redirect exit errors to the main log file, instead of whichever file the erroring logger is managing. I think this will come for free when I rework the sink logic a bit later.
Co-authored-by: default avatarBob Vawter <[email protected]>
Co-authored-by: default avatarRaphael 'kena' Poss <[email protected]>
......@@ -42,7 +42,8 @@ if [ "$(uname)" = "Darwin" ]; then
cached_volume_mode=:cached
fi
GOPATH=$(go env GOPATH)
# We don't want this to emit -json output.
GOPATH=$(GOFLAGS=; go env GOPATH)
gopath0=${GOPATH%%:*}
gocache=${GOCACHEPATH-$gopath0}
......
......@@ -8,11 +8,11 @@
"project_id": "cockroach-teamcity",
"source_image_family": "ubuntu-1804-lts",
"zone": "us-east1-b",
"machine_type": "n1-highcpu-32",
"machine_type": "n1-standard-32",
"image_name": "{{user `image_id`}}",
"image_description": "{{user `image_id`}}",
"ssh_username": "packer",
"disk_size": 50,
"disk_size": 256,
"disk_type": "pd-ssd"
}],
......
......@@ -15,17 +15,13 @@ EOF
# Avoid saving any Bash history.
HISTSIZE=0
# At the time of writing we really want 1.11, but that doesn't
# exist in the PPA yet.
GOVERS=1.10
# Add third-party APT repositories.
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 0EBFCD88
cat > /etc/apt/sources.list.d/docker.list <<EOF
deb https://download.docker.com/linux/ubuntu xenial stable
deb https://download.docker.com/linux/ubuntu bionic stable
EOF
apt-add-repository ppa:webupd8team/java
add-apt-repository ppa:gophers/archive
# Per https://github.com/golang/go/wiki/Ubuntu
add-apt-repository ppa:longsleep/golang-backports
# Git 2.7, which ships with Xenial, has a bug where submodule metadata sometimes
# uses absolute paths instead of relative paths, which means the affected
# submodules cannot be mounted in Docker containers. Use the latest version of
......@@ -33,27 +29,20 @@ add-apt-repository ppa:gophers/archive
add-apt-repository ppa:git-core/ppa
apt-get update --yes
# Auto-accept the Oracle Java license agreement.
debconf-set-selections <<< "oracle-java8-installer shared/accepted-oracle-license-v1-1 select true"
# Install the necessary dependencies. Keep this list small!
apt-get install --yes \
docker-ce \
docker-compose \
gnome-keyring \
git \
golang-${GOVERS} \
oracle-java8-installer \
golang-go \
openjdk-11-jre-headless \
unzip
# Installing gnome-keyring prevents the error described in
# https://github.com/moby/moby/issues/34048
# Link Go into the PATH; the PPA installs it into /usr/lib/go-1.x/bin.
ln -s /usr/lib/go-${GOVERS}/bin/go /usr/bin/go
# Add a user for the TeamCity agent with Docker rights.
adduser agent --disabled-password
adduser agent docker
# Give the user for the TeamCity agent Docker rights.
usermod -a -G docker agent
# Download the TeamCity agent code and install its configuration.
# N.B.: This must be done as the agent user.
......
......@@ -12,7 +12,8 @@ tc_start_block "Prepare environment for acceptance tests"
# that it exists before running the test.
export TMPDIR=$PWD/artifacts/acceptance
mkdir -p "$TMPDIR"
type=$(go env GOOS)
# Disable global -json flag.
type=$(GOFLAGS=; go env GOOS)
tc_end_block "Prepare environment for acceptance tests"
tc_start_block "Compile CockroachDB"
......
......@@ -7,7 +7,8 @@ source "$(dirname "${0}")/teamcity-support.sh"
tc_prepare
tc_start_block "Prepare environment for compose tests"
type=$(go env GOOS)
# Disable global -json flag.
type=$(GOFLAGS=; go env GOOS)
tc_end_block "Prepare environment for compose tests"
tc_start_block "Compile CockroachDB"
......
......@@ -20,7 +20,8 @@ artifacts=$PWD/artifacts
mkdir -p "${artifacts}"
chmod o+rwx "${artifacts}"
export PATH=$PATH:$(go env GOPATH)/bin
# Disable global -json flag.
export PATH=$PATH:$(GOFLAGS=; go env GOPATH)/bin
make bin/workload bin/roachtest bin/roachprod > "${artifacts}/build.txt" 2>&1 || cat "${artifacts}/build.txt"
......
......@@ -21,14 +21,21 @@ eexpect ":/# "
end_test
start_test "Check that a broken stderr prints a message to the log files."
send "$argv start-single-node -s=path=logs/db --insecure --logtostderr --vmodule=*=1 2>&1 | cat\r"
# We use --log-file-max-size to avoid rotations during the test.
send "$argv start-single-node -s=path=logs/db --log-file-max-size=100M --insecure --logtostderr --vmodule=*=1 2>&1 | cat\r"
eexpect "CockroachDB node starting"
system "killall cat"
eexpect ":/# "
# NB: we can't just grep for the broken pipe output, because it may take
# a while for the server to initiate the next log line where it will detect
# the broken pipe error.
send "tail -f logs/db/logs/cockroach.log\r"
#
# We use -F and not -f because the log file may be rotated during the test.
#
# We also watch all the log files in the directory, because the error
# is only reported on the logger which is writing first after stderr
# is has been broken, and that may be the secondary logger.
send "tail -F `find logs/db/logs -type l`\r"
eexpect "log: exiting because of error: write /dev/stderr: broken pipe"
interrupt
eexpect ":/# "
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment