2017-01-09 02:30:57 -07:00
|
|
|
#!/bin/bash
|
|
|
|
# check for errors under valgrind or sanitizers.
|
|
|
|
|
|
|
|
error()
|
|
|
|
{
|
|
|
|
echo "instrumented testing failed on line $1"
|
|
|
|
exit 1
|
|
|
|
}
|
|
|
|
trap 'error ${LINENO}' ERR
|
|
|
|
|
|
|
|
# define suitable post and prefixes for testing options
|
|
|
|
case $1 in
|
|
|
|
--valgrind)
|
|
|
|
echo "valgrind testing started"
|
|
|
|
prefix=''
|
2021-06-03 11:18:24 -06:00
|
|
|
exeprefix='valgrind --error-exitcode=42 --errors-for-leak-kinds=all --leak-check=full'
|
2017-01-09 02:30:57 -07:00
|
|
|
postfix='1>/dev/null'
|
Fix four data races.
the nodes, tbHits, rootDepth and lastInfoTime variables are read by multiple threads, but not declared atomic, leading to data races as found by -fsanitize=thread. This patch fixes this issue. It is based on top of the CI-threading branch (PR #1129), and should fix the corresponding CI error messages.
The patch passed an STC check for no regression:
http://tests.stockfishchess.org/tests/view/5925d5590ebc59035df34b9f
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 169597 W: 29938 L: 30066 D: 109593
Whereas rootDepth and lastInfoTime are not performance critical, nodes and tbHits are. Indeed, an earlier version using relaxed atomic updates on the latter two variables failed STC testing (http://tests.stockfishchess.org/tests/view/592001700ebc59035df34924), which can be shown to be due to x86-32 (http://tests.stockfishchess.org/tests/view/592330ac0ebc59035df34a89). Indeed, the latter have no instruction to atomically update a 64bit variable. The proposed solution thus uses a variable in Position that is accessed only by one thread, which is copied every few thousand nodes to the shared variable in Thread.
No functional change.
Closes #1130
Closes #1129
2017-06-21 14:36:53 -06:00
|
|
|
threads="1"
|
2017-01-09 02:30:57 -07:00
|
|
|
;;
|
2017-08-31 14:11:09 -06:00
|
|
|
--valgrind-thread)
|
|
|
|
echo "valgrind-thread testing started"
|
|
|
|
prefix=''
|
2020-09-30 13:22:36 -06:00
|
|
|
exeprefix='valgrind --fair-sched=try --error-exitcode=42'
|
2017-08-31 14:11:09 -06:00
|
|
|
postfix='1>/dev/null'
|
|
|
|
threads="2"
|
|
|
|
;;
|
Fix four data races.
the nodes, tbHits, rootDepth and lastInfoTime variables are read by multiple threads, but not declared atomic, leading to data races as found by -fsanitize=thread. This patch fixes this issue. It is based on top of the CI-threading branch (PR #1129), and should fix the corresponding CI error messages.
The patch passed an STC check for no regression:
http://tests.stockfishchess.org/tests/view/5925d5590ebc59035df34b9f
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 169597 W: 29938 L: 30066 D: 109593
Whereas rootDepth and lastInfoTime are not performance critical, nodes and tbHits are. Indeed, an earlier version using relaxed atomic updates on the latter two variables failed STC testing (http://tests.stockfishchess.org/tests/view/592001700ebc59035df34924), which can be shown to be due to x86-32 (http://tests.stockfishchess.org/tests/view/592330ac0ebc59035df34a89). Indeed, the latter have no instruction to atomically update a 64bit variable. The proposed solution thus uses a variable in Position that is accessed only by one thread, which is copied every few thousand nodes to the shared variable in Thread.
No functional change.
Closes #1130
Closes #1129
2017-06-21 14:36:53 -06:00
|
|
|
--sanitizer-undefined)
|
2017-08-31 14:11:09 -06:00
|
|
|
echo "sanitizer-undefined testing started"
|
2017-01-09 02:30:57 -07:00
|
|
|
prefix='!'
|
|
|
|
exeprefix=''
|
2019-01-09 08:14:34 -07:00
|
|
|
postfix='2>&1 | grep -A50 "runtime error:"'
|
Fix four data races.
the nodes, tbHits, rootDepth and lastInfoTime variables are read by multiple threads, but not declared atomic, leading to data races as found by -fsanitize=thread. This patch fixes this issue. It is based on top of the CI-threading branch (PR #1129), and should fix the corresponding CI error messages.
The patch passed an STC check for no regression:
http://tests.stockfishchess.org/tests/view/5925d5590ebc59035df34b9f
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 169597 W: 29938 L: 30066 D: 109593
Whereas rootDepth and lastInfoTime are not performance critical, nodes and tbHits are. Indeed, an earlier version using relaxed atomic updates on the latter two variables failed STC testing (http://tests.stockfishchess.org/tests/view/592001700ebc59035df34924), which can be shown to be due to x86-32 (http://tests.stockfishchess.org/tests/view/592330ac0ebc59035df34a89). Indeed, the latter have no instruction to atomically update a 64bit variable. The proposed solution thus uses a variable in Position that is accessed only by one thread, which is copied every few thousand nodes to the shared variable in Thread.
No functional change.
Closes #1130
Closes #1129
2017-06-21 14:36:53 -06:00
|
|
|
threads="1"
|
|
|
|
;;
|
|
|
|
--sanitizer-thread)
|
2017-08-31 14:11:09 -06:00
|
|
|
echo "sanitizer-thread testing started"
|
Fix four data races.
the nodes, tbHits, rootDepth and lastInfoTime variables are read by multiple threads, but not declared atomic, leading to data races as found by -fsanitize=thread. This patch fixes this issue. It is based on top of the CI-threading branch (PR #1129), and should fix the corresponding CI error messages.
The patch passed an STC check for no regression:
http://tests.stockfishchess.org/tests/view/5925d5590ebc59035df34b9f
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 169597 W: 29938 L: 30066 D: 109593
Whereas rootDepth and lastInfoTime are not performance critical, nodes and tbHits are. Indeed, an earlier version using relaxed atomic updates on the latter two variables failed STC testing (http://tests.stockfishchess.org/tests/view/592001700ebc59035df34924), which can be shown to be due to x86-32 (http://tests.stockfishchess.org/tests/view/592330ac0ebc59035df34a89). Indeed, the latter have no instruction to atomically update a 64bit variable. The proposed solution thus uses a variable in Position that is accessed only by one thread, which is copied every few thousand nodes to the shared variable in Thread.
No functional change.
Closes #1130
Closes #1129
2017-06-21 14:36:53 -06:00
|
|
|
prefix='!'
|
|
|
|
exeprefix=''
|
2019-01-09 08:14:34 -07:00
|
|
|
postfix='2>&1 | grep -A50 "WARNING: ThreadSanitizer:"'
|
Fix four data races.
the nodes, tbHits, rootDepth and lastInfoTime variables are read by multiple threads, but not declared atomic, leading to data races as found by -fsanitize=thread. This patch fixes this issue. It is based on top of the CI-threading branch (PR #1129), and should fix the corresponding CI error messages.
The patch passed an STC check for no regression:
http://tests.stockfishchess.org/tests/view/5925d5590ebc59035df34b9f
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 169597 W: 29938 L: 30066 D: 109593
Whereas rootDepth and lastInfoTime are not performance critical, nodes and tbHits are. Indeed, an earlier version using relaxed atomic updates on the latter two variables failed STC testing (http://tests.stockfishchess.org/tests/view/592001700ebc59035df34924), which can be shown to be due to x86-32 (http://tests.stockfishchess.org/tests/view/592330ac0ebc59035df34a89). Indeed, the latter have no instruction to atomically update a 64bit variable. The proposed solution thus uses a variable in Position that is accessed only by one thread, which is copied every few thousand nodes to the shared variable in Thread.
No functional change.
Closes #1130
Closes #1129
2017-06-21 14:36:53 -06:00
|
|
|
threads="2"
|
|
|
|
|
|
|
|
cat << EOF > tsan.supp
|
2021-02-26 02:02:13 -07:00
|
|
|
race:Stockfish::TTEntry::move
|
|
|
|
race:Stockfish::TTEntry::depth
|
|
|
|
race:Stockfish::TTEntry::bound
|
|
|
|
race:Stockfish::TTEntry::save
|
|
|
|
race:Stockfish::TTEntry::value
|
|
|
|
race:Stockfish::TTEntry::eval
|
|
|
|
race:Stockfish::TTEntry::is_pv
|
|
|
|
|
|
|
|
race:Stockfish::TranspositionTable::probe
|
|
|
|
race:Stockfish::TranspositionTable::hashfull
|
Fix four data races.
the nodes, tbHits, rootDepth and lastInfoTime variables are read by multiple threads, but not declared atomic, leading to data races as found by -fsanitize=thread. This patch fixes this issue. It is based on top of the CI-threading branch (PR #1129), and should fix the corresponding CI error messages.
The patch passed an STC check for no regression:
http://tests.stockfishchess.org/tests/view/5925d5590ebc59035df34b9f
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 169597 W: 29938 L: 30066 D: 109593
Whereas rootDepth and lastInfoTime are not performance critical, nodes and tbHits are. Indeed, an earlier version using relaxed atomic updates on the latter two variables failed STC testing (http://tests.stockfishchess.org/tests/view/592001700ebc59035df34924), which can be shown to be due to x86-32 (http://tests.stockfishchess.org/tests/view/592330ac0ebc59035df34a89). Indeed, the latter have no instruction to atomically update a 64bit variable. The proposed solution thus uses a variable in Position that is accessed only by one thread, which is copied every few thousand nodes to the shared variable in Thread.
No functional change.
Closes #1130
Closes #1129
2017-06-21 14:36:53 -06:00
|
|
|
|
|
|
|
EOF
|
|
|
|
|
|
|
|
export TSAN_OPTIONS="suppressions=./tsan.supp"
|
|
|
|
|
2017-01-09 02:30:57 -07:00
|
|
|
;;
|
|
|
|
*)
|
|
|
|
echo "unknown testing started"
|
|
|
|
prefix=''
|
|
|
|
exeprefix=''
|
|
|
|
postfix=''
|
Fix four data races.
the nodes, tbHits, rootDepth and lastInfoTime variables are read by multiple threads, but not declared atomic, leading to data races as found by -fsanitize=thread. This patch fixes this issue. It is based on top of the CI-threading branch (PR #1129), and should fix the corresponding CI error messages.
The patch passed an STC check for no regression:
http://tests.stockfishchess.org/tests/view/5925d5590ebc59035df34b9f
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 169597 W: 29938 L: 30066 D: 109593
Whereas rootDepth and lastInfoTime are not performance critical, nodes and tbHits are. Indeed, an earlier version using relaxed atomic updates on the latter two variables failed STC testing (http://tests.stockfishchess.org/tests/view/592001700ebc59035df34924), which can be shown to be due to x86-32 (http://tests.stockfishchess.org/tests/view/592330ac0ebc59035df34a89). Indeed, the latter have no instruction to atomically update a 64bit variable. The proposed solution thus uses a variable in Position that is accessed only by one thread, which is copied every few thousand nodes to the shared variable in Thread.
No functional change.
Closes #1130
Closes #1129
2017-06-21 14:36:53 -06:00
|
|
|
threads="1"
|
2017-01-09 02:30:57 -07:00
|
|
|
;;
|
|
|
|
esac
|
|
|
|
|
|
|
|
# simple command line testing
|
|
|
|
for args in "eval" \
|
|
|
|
"go nodes 1000" \
|
|
|
|
"go depth 10" \
|
|
|
|
"go movetime 1000" \
|
|
|
|
"go wtime 8000 btime 8000 winc 500 binc 500" \
|
This commit enables a mixed bench, to improve CI and allow for PGO (profile-build) of the NNUE part of the code.
Joint work gvreuls / vondele
* Download the default NNUE net in AppVeyor
* Download net in travis CI `make net`
* Adjust tests to cover more archs, speedup instrumented testing
* Introduce 'mixed' bench as default, with further options:
classical, NNUE, mixed.
mixed (default) and NNUE require the default net to be present,
which can be obtained with
```
make net
```
Further examples (first is equivalent to `./stockfish bench`):
```
./stockfish bench 16 1 13 default depth mixed
./stockfish bench 16 1 13 default depth classical
./stockfish bench 16 1 13 default depth NNUE
```
The net is now downloaded automatically if needed for `profile-build`
(usual `build` works fine without net present)
PGO gives a nice speedup on fishtest:
passed STC:
LLR: 2.93 (-2.94,2.94) {-0.50,1.50}
Total: 3360 W: 469 L: 343 D: 2548
Ptnml(0-2): 20, 246, 1030, 356, 28
https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569
passed LTC:
LLR: 2.97 (-2.94,2.94) {0.25,1.75}
Total: 8824 W: 609 L: 502 D: 7713
Ptnml(0-2): 8, 430, 3438, 519, 17
https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c
closes https://github.com/official-stockfish/Stockfish/pull/2931
fixes https://github.com/official-stockfish/Stockfish/issues/2907
requires fishtest updates before commit
Bench: 4290577
2020-08-07 09:07:46 -06:00
|
|
|
"bench 128 $threads 8 default depth"
|
2017-01-09 02:30:57 -07:00
|
|
|
do
|
|
|
|
|
|
|
|
echo "$prefix $exeprefix ./stockfish $args $postfix"
|
|
|
|
eval "$prefix $exeprefix ./stockfish $args $postfix"
|
|
|
|
|
|
|
|
done
|
|
|
|
|
|
|
|
# more general testing, following an uci protocol exchange
|
|
|
|
cat << EOF > game.exp
|
This commit enables a mixed bench, to improve CI and allow for PGO (profile-build) of the NNUE part of the code.
Joint work gvreuls / vondele
* Download the default NNUE net in AppVeyor
* Download net in travis CI `make net`
* Adjust tests to cover more archs, speedup instrumented testing
* Introduce 'mixed' bench as default, with further options:
classical, NNUE, mixed.
mixed (default) and NNUE require the default net to be present,
which can be obtained with
```
make net
```
Further examples (first is equivalent to `./stockfish bench`):
```
./stockfish bench 16 1 13 default depth mixed
./stockfish bench 16 1 13 default depth classical
./stockfish bench 16 1 13 default depth NNUE
```
The net is now downloaded automatically if needed for `profile-build`
(usual `build` works fine without net present)
PGO gives a nice speedup on fishtest:
passed STC:
LLR: 2.93 (-2.94,2.94) {-0.50,1.50}
Total: 3360 W: 469 L: 343 D: 2548
Ptnml(0-2): 20, 246, 1030, 356, 28
https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569
passed LTC:
LLR: 2.97 (-2.94,2.94) {0.25,1.75}
Total: 8824 W: 609 L: 502 D: 7713
Ptnml(0-2): 8, 430, 3438, 519, 17
https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c
closes https://github.com/official-stockfish/Stockfish/pull/2931
fixes https://github.com/official-stockfish/Stockfish/issues/2907
requires fishtest updates before commit
Bench: 4290577
2020-08-07 09:07:46 -06:00
|
|
|
set timeout 240
|
2017-01-09 02:30:57 -07:00
|
|
|
spawn $exeprefix ./stockfish
|
|
|
|
|
|
|
|
send "uci\n"
|
|
|
|
expect "uciok"
|
|
|
|
|
Fix four data races.
the nodes, tbHits, rootDepth and lastInfoTime variables are read by multiple threads, but not declared atomic, leading to data races as found by -fsanitize=thread. This patch fixes this issue. It is based on top of the CI-threading branch (PR #1129), and should fix the corresponding CI error messages.
The patch passed an STC check for no regression:
http://tests.stockfishchess.org/tests/view/5925d5590ebc59035df34b9f
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 169597 W: 29938 L: 30066 D: 109593
Whereas rootDepth and lastInfoTime are not performance critical, nodes and tbHits are. Indeed, an earlier version using relaxed atomic updates on the latter two variables failed STC testing (http://tests.stockfishchess.org/tests/view/592001700ebc59035df34924), which can be shown to be due to x86-32 (http://tests.stockfishchess.org/tests/view/592330ac0ebc59035df34a89). Indeed, the latter have no instruction to atomically update a 64bit variable. The proposed solution thus uses a variable in Position that is accessed only by one thread, which is copied every few thousand nodes to the shared variable in Thread.
No functional change.
Closes #1130
Closes #1129
2017-06-21 14:36:53 -06:00
|
|
|
send "setoption name Threads value $threads\n"
|
|
|
|
|
2017-01-09 02:30:57 -07:00
|
|
|
send "ucinewgame\n"
|
|
|
|
send "position startpos\n"
|
|
|
|
send "go nodes 1000\n"
|
|
|
|
expect "bestmove"
|
|
|
|
|
|
|
|
send "position startpos moves e2e4 e7e6\n"
|
|
|
|
send "go nodes 1000\n"
|
|
|
|
expect "bestmove"
|
|
|
|
|
2017-03-25 18:57:07 -06:00
|
|
|
send "position fen 5rk1/1K4p1/8/8/3B4/8/8/8 b - - 0 1\n"
|
2021-06-18 04:03:03 -06:00
|
|
|
send "go depth 10\n"
|
2017-03-25 18:57:07 -06:00
|
|
|
expect "bestmove"
|
|
|
|
|
2017-01-09 02:30:57 -07:00
|
|
|
send "quit\n"
|
|
|
|
expect eof
|
|
|
|
|
|
|
|
# return error code of the spawned program, useful for valgrind
|
|
|
|
lassign [wait] pid spawnid os_error_flag value
|
|
|
|
exit \$value
|
|
|
|
EOF
|
|
|
|
|
2018-03-30 02:21:59 -06:00
|
|
|
#download TB as needed
|
|
|
|
if [ ! -d ../tests/syzygy ]; then
|
|
|
|
curl -sL https://api.github.com/repos/niklasf/python-chess/tarball/9b9aa13f9f36d08aadfabff872882f4ab1494e95 | tar -xzf -
|
|
|
|
mv niklasf-python-chess-9b9aa13 ../tests/syzygy
|
|
|
|
fi
|
|
|
|
|
|
|
|
cat << EOF > syzygy.exp
|
|
|
|
set timeout 240
|
|
|
|
spawn $exeprefix ./stockfish
|
|
|
|
send "uci\n"
|
|
|
|
send "setoption name SyzygyPath value ../tests/syzygy/\n"
|
|
|
|
expect "info string Found 35 tablebases" {} timeout {exit 1}
|
This commit enables a mixed bench, to improve CI and allow for PGO (profile-build) of the NNUE part of the code.
Joint work gvreuls / vondele
* Download the default NNUE net in AppVeyor
* Download net in travis CI `make net`
* Adjust tests to cover more archs, speedup instrumented testing
* Introduce 'mixed' bench as default, with further options:
classical, NNUE, mixed.
mixed (default) and NNUE require the default net to be present,
which can be obtained with
```
make net
```
Further examples (first is equivalent to `./stockfish bench`):
```
./stockfish bench 16 1 13 default depth mixed
./stockfish bench 16 1 13 default depth classical
./stockfish bench 16 1 13 default depth NNUE
```
The net is now downloaded automatically if needed for `profile-build`
(usual `build` works fine without net present)
PGO gives a nice speedup on fishtest:
passed STC:
LLR: 2.93 (-2.94,2.94) {-0.50,1.50}
Total: 3360 W: 469 L: 343 D: 2548
Ptnml(0-2): 20, 246, 1030, 356, 28
https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569
passed LTC:
LLR: 2.97 (-2.94,2.94) {0.25,1.75}
Total: 8824 W: 609 L: 502 D: 7713
Ptnml(0-2): 8, 430, 3438, 519, 17
https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c
closes https://github.com/official-stockfish/Stockfish/pull/2931
fixes https://github.com/official-stockfish/Stockfish/issues/2907
requires fishtest updates before commit
Bench: 4290577
2020-08-07 09:07:46 -06:00
|
|
|
send "bench 128 1 8 default depth\n"
|
2018-03-30 02:21:59 -06:00
|
|
|
send "quit\n"
|
|
|
|
expect eof
|
|
|
|
|
|
|
|
# return error code of the spawned program, useful for valgrind
|
|
|
|
lassign [wait] pid spawnid os_error_flag value
|
|
|
|
exit \$value
|
|
|
|
EOF
|
|
|
|
|
|
|
|
for exp in game.exp syzygy.exp
|
2017-01-09 02:30:57 -07:00
|
|
|
do
|
|
|
|
|
2018-03-30 02:21:59 -06:00
|
|
|
echo "$prefix expect $exp $postfix"
|
|
|
|
eval "$prefix expect $exp $postfix"
|
Fix four data races.
the nodes, tbHits, rootDepth and lastInfoTime variables are read by multiple threads, but not declared atomic, leading to data races as found by -fsanitize=thread. This patch fixes this issue. It is based on top of the CI-threading branch (PR #1129), and should fix the corresponding CI error messages.
The patch passed an STC check for no regression:
http://tests.stockfishchess.org/tests/view/5925d5590ebc59035df34b9f
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 169597 W: 29938 L: 30066 D: 109593
Whereas rootDepth and lastInfoTime are not performance critical, nodes and tbHits are. Indeed, an earlier version using relaxed atomic updates on the latter two variables failed STC testing (http://tests.stockfishchess.org/tests/view/592001700ebc59035df34924), which can be shown to be due to x86-32 (http://tests.stockfishchess.org/tests/view/592330ac0ebc59035df34a89). Indeed, the latter have no instruction to atomically update a 64bit variable. The proposed solution thus uses a variable in Position that is accessed only by one thread, which is copied every few thousand nodes to the shared variable in Thread.
No functional change.
Closes #1130
Closes #1129
2017-06-21 14:36:53 -06:00
|
|
|
|
2018-03-30 02:21:59 -06:00
|
|
|
rm $exp
|
2017-01-09 02:30:57 -07:00
|
|
|
|
|
|
|
done
|
|
|
|
|
Fix four data races.
the nodes, tbHits, rootDepth and lastInfoTime variables are read by multiple threads, but not declared atomic, leading to data races as found by -fsanitize=thread. This patch fixes this issue. It is based on top of the CI-threading branch (PR #1129), and should fix the corresponding CI error messages.
The patch passed an STC check for no regression:
http://tests.stockfishchess.org/tests/view/5925d5590ebc59035df34b9f
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 169597 W: 29938 L: 30066 D: 109593
Whereas rootDepth and lastInfoTime are not performance critical, nodes and tbHits are. Indeed, an earlier version using relaxed atomic updates on the latter two variables failed STC testing (http://tests.stockfishchess.org/tests/view/592001700ebc59035df34924), which can be shown to be due to x86-32 (http://tests.stockfishchess.org/tests/view/592330ac0ebc59035df34a89). Indeed, the latter have no instruction to atomically update a 64bit variable. The proposed solution thus uses a variable in Position that is accessed only by one thread, which is copied every few thousand nodes to the shared variable in Thread.
No functional change.
Closes #1130
Closes #1129
2017-06-21 14:36:53 -06:00
|
|
|
rm -f tsan.supp
|
2017-01-09 02:30:57 -07:00
|
|
|
|
|
|
|
echo "instrumented testing OK"
|