deepcrayon
/
stockfish
1
0
Fork 0
Code Releases Activity
stockfish/src/benchmark.cpp

169 lines
6.4 KiB
C++
Raw Normal View History

Initial import of Glaurung 2.1
2008-08-31 23:59:13 -06:00
/*
Update copyright info Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2008-10-19 10:56:28 -06:00
Stockfish, a UCI chess playing engine derived from Glaurung 2.1
Add NNUE evaluation This patch ports the efficiently updatable neural network (NNUE) evaluation to Stockfish. Both the NNUE and the classical evaluations are available, and can be used to assign a value to a position that is later used in alpha-beta (PVS) search to find the best move. The classical evaluation computes this value as a function of various chess concepts, handcrafted by experts, tested and tuned using fishtest. The NNUE evaluation computes this value with a neural network based on basic inputs. The network is optimized and trained on the evalutions of millions of positions at moderate search depth. The NNUE evaluation was first introduced in shogi, and ported to Stockfish afterward. It can be evaluated efficiently on CPUs, and exploits the fact that only parts of the neural network need to be updated after a typical chess move. [The nodchip repository](https://github.com/nodchip/Stockfish) provides additional tools to train and develop the NNUE networks. This patch is the result of contributions of various authors, from various communities, including: nodchip, ynasu87, yaneurao (initial port and NNUE authors), domschl, FireFather, rqs, xXH4CKST3RXx, tttak, zz4032, joergoster, mstembera, nguyenpham, erbsenzaehler, dorzechowski, and vondele. This new evaluation needed various changes to fishtest and the corresponding infrastructure, for which tomtor, ppigazzini, noobpwnftw, daylen, and vondele are gratefully acknowledged. The first networks have been provided by gekkehenker and sergiovieri, with the latter net (nn-97f742aaefcd.nnue) being the current default. The evaluation function can be selected at run time with the `Use NNUE` (true/false) UCI option, provided the `EvalFile` option points the the network file (depending on the GUI, with full path). The performance of the NNUE evaluation relative to the classical evaluation depends somewhat on the hardware, and is expected to improve quickly, but is currently on > 80 Elo on fishtest: 60000 @ 10+0.1 th 1 https://tests.stockfishchess.org/tests/view/5f28fe6ea5abc164f05e4c4c ELO: 92.77 +-2.1 (95%) LOS: 100.0% Total: 60000 W: 24193 L: 8543 D: 27264 Ptnml(0-2): 609, 3850, 9708, 10948, 4885 40000 @ 20+0.2 th 8 https://tests.stockfishchess.org/tests/view/5f290229a5abc164f05e4c58 ELO: 89.47 +-2.0 (95%) LOS: 100.0% Total: 40000 W: 12756 L: 2677 D: 24567 Ptnml(0-2): 74, 1583, 8550, 7776, 2017 At the same time, the impact on the classical evaluation remains minimal, causing no significant regression: sprt @ 10+0.1 th 1 https://tests.stockfishchess.org/tests/view/5f2906a2a5abc164f05e4c5b LLR: 2.94 (-2.94,2.94) {-6.00,-4.00} Total: 34936 W: 6502 L: 6825 D: 21609 Ptnml(0-2): 571, 4082, 8434, 3861, 520 sprt @ 60+0.6 th 1 https://tests.stockfishchess.org/tests/view/5f2906cfa5abc164f05e4c5d LLR: 2.93 (-2.94,2.94) {-6.00,-4.00} Total: 10088 W: 1232 L: 1265 D: 7591 Ptnml(0-2): 49, 914, 3170, 843, 68 The needed networks can be found at https://tests.stockfishchess.org/nns It is recommended to use the default one as indicated by the `EvalFile` UCI option. Guidelines for testing new nets can be found at https://github.com/glinscott/fishtest/wiki/Creating-my-first-test#nnue-net-tests Integration has been discussed in various issues: https://github.com/official-stockfish/Stockfish/issues/2823 https://github.com/official-stockfish/Stockfish/issues/2728 The integration branch will be closed after the merge: https://github.com/official-stockfish/Stockfish/pull/2825 https://github.com/official-stockfish/Stockfish/tree/nnue-player-wip closes https://github.com/official-stockfish/Stockfish/pull/2912 This will be an exciting time for computer chess, looking forward to seeing the evolution of this approach. Bench: 4746616
2020-08-05 09:11:15 -06:00
Copyright (C) 2004-2020 The Stockfish developers (see AUTHORS file)
Initial import of Glaurung 2.1
2008-08-31 23:59:13 -06:00
Update copyright info Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2008-10-19 10:56:28 -06:00
Stockfish is free software: you can redistribute it and/or modify
Initial import of Glaurung 2.1
2008-08-31 23:59:13 -06:00
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
Big trailing whitespace cleanup part 1 Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2009-01-07 06:26:58 -07:00
Update copyright info Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2008-10-19 10:56:28 -06:00
Stockfish is distributed in the hope that it will be useful,
Initial import of Glaurung 2.1
2008-08-31 23:59:13 -06:00
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
Big trailing whitespace cleanup part 1 Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2009-01-07 06:26:58 -07:00
Initial import of Glaurung 2.1
2008-08-31 23:59:13 -06:00
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
Teach Benchmark to read positions from a file This comes handy to test the engine on different sets of positions, not only the 15 default ones. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2008-09-18 02:55:35 -06:00
#include <fstream>
Retire move.cpp Move its functions where they belong. No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2011-01-07 05:00:25 -07:00
#include <iostream>
Use bench to implement UI 'perft' command Now that we can call bench on current position we can directly use it to perform our perft. No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2012-04-07 06:23:43 -06:00
#include <istream>
Teach Benchmark to read positions from a file This comes handy to test the engine on different sets of positions, not only the 15 default ones. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2008-09-18 02:55:35 -06:00
#include <vector>
Initial import of Glaurung 2.1
2008-08-31 23:59:13 -06:00
Tidy up benchmark.cpp Node count is different just becuase now we don't log on "bench.txt" file anymore so that we avoid some calls to pretty_pv() that calls Position::do_move(). Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2011-04-25 03:46:52 -06:00
#include "position.h"
Initial import of Glaurung 2.1
2008-08-31 23:59:13 -06:00
Use string instead of std::string And others small code style touches. No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2009-05-10 10:58:53 -06:00
using namespace std;
Initial import of Glaurung 2.1
2008-08-31 23:59:13 -06:00
Consistent use of anonymous namespace Objects that are only accessible at file-scope should be put in the anonymous namespace. This is what the C++ standard recommends, rather than using static, which is really C-style and results in static linkage. Stockfish already does this throughout the code. So let's weed out the few exceptions, because... they have no reason to be exceptional. No functional change. Resolves #84
2014-11-01 14:35:10 -06:00
namespace {
Import C++11 branch Import C++11 branch from: https://github.com/mcostalba/Stockfish/tree/c++11 The version imported is teh last one as of today: https://github.com/mcostalba/Stockfish/commit/6670e93e50a7a4e739ac2ac8b6026ffa3a12150a Branch is fully equivalent with master but syzygy tablebases that are missing (but will be added with next commit). bench: 8080602
2015-01-18 00:00:50 -07:00
const vector<string> Defaults = {
Wide bench coverage Add tests for: - Positions with move list - Chess960 positions Now bench covers almost all cases, only few endgames are still out of reach (verified with lcov) It is a non functionality patch, but bench changed because we added new test positions. bench: 5479946
2017-08-15 03:51:14 -06:00
"setoption name UCI_Chess960 value false",
Fix a missing comma in BenchmarkPositions[] An old bug introduced in 3e0dc9ee8477 almost one month ago. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2008-10-19 04:55:53 -06:00
"rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1",
Teach to_fen() about Halfmove and Fullmove number And also fix the last '/' at the end of the piece placement field. No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2011-07-03 04:32:34 -06:00
"r3k2r/p1ppqpb1/bn2pnp1/3PN3/1p2P3/2N2Q1p/PPPBBPPP/R3K2R w KQkq - 0 10",
"8/2p5/3p4/KP5r/1R3p1k/8/4P1P1/8 w - - 0 11",
Initial import of Glaurung 2.1
2008-08-31 23:59:13 -06:00
"4rrk1/pp1n3p/3q2pQ/2p1pb2/2PP4/2P3N1/P2B2PP/4RRK1 b - - 7 19",
Wide bench coverage Add tests for: - Positions with move list - Chess960 positions Now bench covers almost all cases, only few endgames are still out of reach (verified with lcov) It is a non functionality patch, but bench changed because we added new test positions. bench: 5479946
2017-08-15 03:51:14 -06:00
"rq3rk1/ppp2ppp/1bnpb3/3N2B1/3NP3/7P/PPPQ1PP1/2KR3R w - - 7 14 moves d4e6",
"r1bq1r1k/1pp1n1pp/1p1p4/4p2Q/4Pp2/1BNP4/PPP2PPP/3R1RK1 w - - 2 14 moves g2g4",
Initial import of Glaurung 2.1
2008-08-31 23:59:13 -06:00
"r3r1k1/2p2ppp/p1p1bn2/8/1q2P3/2NPQN2/PPP3PP/R4RK1 b - - 2 15",
"r1bbk1nr/pp3p1p/2n5/1N4p1/2Np1B2/8/PPP2PPP/2KR1B1R w kq - 0 13",
"r1bq1rk1/ppp1nppp/4n3/3p3Q/3P4/1BP1B3/PP1N2PP/R4RK1 w - - 1 16",
"4r1k1/r1q2ppp/ppp2n2/4P3/5Rb1/1N1BQ3/PPP3PP/R5K1 w - - 1 17",
"2rqkb1r/ppp2p2/2npb1p1/1N1Nn2p/2P1PP2/8/PP2B1PP/R1BQK2R b KQ - 0 11",
"r1bq1r1k/b1p1npp1/p2p3p/1p6/3PP3/1B2NN2/PP3PPP/R2Q1RK1 w - - 1 16",
"3r1rk1/p5pp/bpp1pp2/8/q1PP1P2/b3P3/P2NQRPP/1R2B1K1 b - - 6 22",
"r1q2rk1/2p1bppp/2Pp4/p6b/Q1PNp3/4B3/PP1R1PPP/2K4R w - - 2 18",
Rewrite Position::from_fen() Complete rewrite the function and extend compatibility also to X-FEN notation for Chess960. We are now able to read standard FEN, Shredder-FEN and X-FEN. No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2010-07-23 02:38:19 -06:00
"4k2r/1pb2ppp/1p2p3/1R1p4/3P4/2r1PN2/P4PPP/1R4K1 b - - 3 22",
Add more depth/positions to bench Increase bench default depth from 12 to 13 and add 15 new endgame positions to have broader coverage and also more reliable nps calulcation used for fishtest framework. Due to the new endgame positions, where nps is higher, the total nps is increased of about 15%. Thanks to Lucas and Jörg for the suggestions. No functional change, but bench number is now: bench: 8336338
2013-09-29 01:43:10 -06:00
"3q2k1/pb3p1p/4pbp1/2r5/PpN2N2/1P2P2P/5PP1/Q2R2K1 b - - 4 26",
"6k1/6p1/6Pp/ppp5/3pn2P/1P3K2/1PP2P2/3N4 b - - 0 1",
"3b4/5kp1/1p1p1p1p/pP1PpP1P/P1P1P3/3KN3/8/8 w - - 0 1",
Wide bench coverage Add tests for: - Positions with move list - Chess960 positions Now bench covers almost all cases, only few endgames are still out of reach (verified with lcov) It is a non functionality patch, but bench changed because we added new test positions. bench: 5479946
2017-08-15 03:51:14 -06:00
"2K5/p7/7P/5pR1/8/5k2/r7/8 w - - 0 1 moves g5g6 f3e3 g6g5 e3f3",
Add more depth/positions to bench Increase bench default depth from 12 to 13 and add 15 new endgame positions to have broader coverage and also more reliable nps calulcation used for fishtest framework. Due to the new endgame positions, where nps is higher, the total nps is increased of about 15%. Thanks to Lucas and Jörg for the suggestions. No functional change, but bench number is now: bench: 8336338
2013-09-29 01:43:10 -06:00
"8/6pk/1p6/8/PP3p1p/5P2/4KP1q/3Q4 w - - 0 1",
"7k/3p2pp/4q3/8/4Q3/5Kp1/P6b/8 w - - 0 1",
"8/2p5/8/2kPKp1p/2p4P/2P5/3P4/8 w - - 0 1",
"8/1p3pp1/7p/5P1P/2k3P1/8/2K2P2/8 w - - 0 1",
"8/pp2r1k1/2p1p3/3pP2p/1P1P1P1P/P5KR/8/8 w - - 0 1",
"8/3p4/p1bk3p/Pp6/1Kp1PpPp/2P2P1P/2P5/5B2 b - - 0 1",
"5k2/7R/4P2p/5K2/p1r2P1p/8/8/8 b - - 0 1",
"6k1/6p1/P6p/r1N5/5p2/7P/1b3PP1/4R1K1 w - - 0 1",
"1r3k2/4q3/2Pp3b/3Bp3/2Q2p2/1p1P2P1/1P2KP2/3N4 w - - 0 1",
"6k1/4pp1p/3p2p1/P1pPb3/R7/1r2P1PP/3B1P2/6K1 w - - 0 1",
Add some tablebase positions to bench This makes it easier to check for regressions in the tablebase code. Bench: 9489202 5-man bench: 8943906 Resolves #145
2014-12-07 16:55:53 -07:00
"8/3p3B/5p2/5P2/p7/PP5b/k7/6K1 w - - 0 1",
Add four positions to bench The current bench is missing a position with high 50 moves rule counter, making most 'shuffle' tests based on 50mr > N seem non-functional. This patch adds one FEN with high 50mr counter to address this issue (taken from a recent tcec game). Four new FENs: - position with high 50mr counter - tactical position with many captures, checks, extensions, fails high/low - two losses by Stockfish in the S16 bonus games against Houdini See the pull request for nice comments by @Alayan-stk-2 about each position in bench: https://github.com/official-stockfish/Stockfish/pull/2338 Bench: 4590210
2019-10-01 14:11:12 -06:00
"5rk1/q6p/2p3bR/1pPp1rP1/1P1Pp3/P3B1Q1/1K3P2/R7 w - - 93 90",
"4rrk1/1p1nq3/p7/2p1P1pp/3P2bp/3Q1Bn1/PPPB4/1K2R1NR w - - 40 21",
"r3k2r/3nnpbp/q2pp1p1/p7/Pp1PPPP1/4BNN1/1P5P/R2Q1RK1 w kq - 0 16",
"3Qb1k1/1r2ppb1/pN1n2q1/Pp1Pp1Pr/4P2p/4BP2/4B1R1/1R5K b - - 11 40",
Equations for edges and corners. This is a functional simplification that removes the large arrays in endgames.cpp. It also fixes a recently introduced bug (960d59d54143d84aab26deae65279a611fc989f4) in KNBvK, now using flip_file() instead of ~. One fen added to bench to increase endgame coverage. STC LLR: 2.94 (-2.94,2.94) {-1.50,0.50} Total: 174724 W: 33325 L: 33404 D: 107995 Ptnml(0-2): 2503, 19607, 43181, 19608, 2463 http://tests.stockfishchess.org/tests/view/5e6448ffe42a5c3b3ca2e287 LTC LLR: 2.95 (-2.94,2.94) {-1.50,0.50} Total: 35640 W: 4679 L: 4621 D: 26340 Ptnml(0-2): 189, 2991, 11424, 3005, 211 http://tests.stockfishchess.org/tests/view/5e650b24e42a5c3b3ca2e2d8 closes https://github.com/official-stockfish/Stockfish/pull/2577 Bench: 5527957
2020-03-09 15:11:08 -06:00
"4k3/3q1r2/1N2r1b1/3ppN2/2nPP3/1B1R2n1/2R1Q3/3K4 w - - 5 1",
Add some tablebase positions to bench This makes it easier to check for regressions in the tablebase code. Bench: 9489202 5-man bench: 8943906 Resolves #145
2014-12-07 16:55:53 -07:00
// 5-man positions
Assorted nitpicking code-style No functional change.
2014-12-08 00:23:09 -07:00
"8/8/8/8/5kp1/P7/8/1K1N4 w - - 0 1", // Kc2 - mate
"8/8/8/5N2/8/p7/8/2NK3k w - - 0 1", // Na2 - mate
"8/3k4/8/8/8/4B3/4KB2/2B5 w - - 0 1", // draw
Add some tablebase positions to bench This makes it easier to check for regressions in the tablebase code. Bench: 9489202 5-man bench: 8943906 Resolves #145
2014-12-07 16:55:53 -07:00
// 6-man positions
Assorted nitpicking code-style No functional change.
2014-12-08 00:23:09 -07:00
"8/8/1P6/5pr1/8/4R3/7k/2K5 w - - 0 1", // Re5 - mate
"8/2p4P/8/kr6/6R1/8/8/1K6 w - - 0 1", // Ka2 - mate
"8/8/3P3k/8/1p6/8/1P6/1K3n2 b - - 0 1", // Nd2 - draw
Add some tablebase positions to bench This makes it easier to check for regressions in the tablebase code. Bench: 9489202 5-man bench: 8943906 Resolves #145
2014-12-07 16:55:53 -07:00
// 7-man positions
Fix crash when passing a mate/stalemate position Both Tablebases::filter_root_moves() and extract_ponder_from_tt(9 were unable to handle a mate/stalemate position. Spotted and reported by Dann Corbit. Added some mate/stalemate positions to bench so to early catch this regression in the future. No functional change.
2016-09-23 23:30:37 -06:00
"8/R7/2q5/8/6k1/8/1P5p/K6R w - - 0 124", // Draw
// Mate and stalemate positions
Wide bench coverage Add tests for: - Positions with move list - Chess960 positions Now bench covers almost all cases, only few endgames are still out of reach (verified with lcov) It is a non functionality patch, but bench changed because we added new test positions. bench: 5479946
2017-08-15 03:51:14 -06:00
"6k1/3b3r/1p1p4/p1n2p2/1PPNpP1q/P3Q1p1/1R1RB1P1/5K2 b - - 0 1",
"r2r1n2/pp2bk2/2p1p2p/3q4/3PN1QP/2P3R1/P4PP1/5RK1 w - - 0 1",
Fix crash when passing a mate/stalemate position Both Tablebases::filter_root_moves() and extract_ponder_from_tt(9 were unable to handle a mate/stalemate position. Spotted and reported by Dann Corbit. Added some mate/stalemate positions to bench so to early catch this regression in the future. No functional change.
2016-09-23 23:30:37 -06:00
"8/8/8/8/8/6k1/6p1/6K1 w - -",
Wide bench coverage Add tests for: - Positions with move list - Chess960 positions Now bench covers almost all cases, only few endgames are still out of reach (verified with lcov) It is a non functionality patch, but bench changed because we added new test positions. bench: 5479946
2017-08-15 03:51:14 -06:00
"7k/7P/6K1/8/3B4/8/8/8 b - -",
// Chess 960
"setoption name UCI_Chess960 value true",
Small cleanups closes https://github.com/official-stockfish/Stockfish/pull/2772 No functional change
2020-07-09 14:01:06 -06:00
"bbqnnrkr/pppppppp/8/8/8/8/PPPPPPPP/BBQNNRKR w HFhf - 0 1 moves g2g3 d7d5 d2d4 c8h3 c1g5 e8d6 g5e7 f7f6",
Wide bench coverage Add tests for: - Positions with move list - Chess960 positions Now bench covers almost all cases, only few endgames are still out of reach (verified with lcov) It is a non functionality patch, but bench changed because we added new test positions. bench: 5479946
2017-08-15 03:51:14 -06:00
"setoption name UCI_Chess960 value false"
Initial import of Glaurung 2.1
2008-08-31 23:59:13 -06:00
};
Big trailing whitespace cleanup part 1 Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2009-01-07 06:26:58 -07:00
Consistent use of anonymous namespace Objects that are only accessible at file-scope should be put in the anonymous namespace. This is what the C++ standard recommends, rather than using static, which is really C-style and results in static linkage. Stockfish already does this throughout the code. So let's weed out the few exceptions, because... they have no reason to be exceptional. No functional change. Resolves #84
2014-11-01 14:35:10 -06:00
} // namespace
Initial import of Glaurung 2.1
2008-08-31 23:59:13 -06:00
Rewrite benchmark First step in improving bench to handle arbitrary UCI commands so to test many more code paths. This first patch just set the new code structure. No functional change.
2017-08-14 11:41:04 -06:00
/// setup_bench() builds a list of UCI commands to be run by bench. There
Restore perft Rewrite perft to be placed naturally inside new bench code. In particular we don't have special custom code to run perft anymore but perft is just a new parameter of 'go' command. So user API is now changed, old style command: $perft 5 becomes $go perft 4 No functional change.
2017-08-15 02:05:22 -06:00
/// are five parameters: TT size in MB, number of search threads that
/// should be used, the limit value spent for each position, a file name
This commit enables a mixed bench, to improve CI and allow for PGO (profile-build) of the NNUE part of the code. Joint work gvreuls / vondele * Download the default NNUE net in AppVeyor * Download net in travis CI `make net` * Adjust tests to cover more archs, speedup instrumented testing * Introduce 'mixed' bench as default, with further options: classical, NNUE, mixed. mixed (default) and NNUE require the default net to be present, which can be obtained with ``` make net ``` Further examples (first is equivalent to `./stockfish bench`): ``` ./stockfish bench 16 1 13 default depth mixed ./stockfish bench 16 1 13 default depth classical ./stockfish bench 16 1 13 default depth NNUE ``` The net is now downloaded automatically if needed for `profile-build` (usual `build` works fine without net present) PGO gives a nice speedup on fishtest: passed STC: LLR: 2.93 (-2.94,2.94) {-0.50,1.50} Total: 3360 W: 469 L: 343 D: 2548 Ptnml(0-2): 20, 246, 1030, 356, 28 https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569 passed LTC: LLR: 2.97 (-2.94,2.94) {0.25,1.75} Total: 8824 W: 609 L: 502 D: 7713 Ptnml(0-2): 8, 430, 3438, 519, 17 https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c closes https://github.com/official-stockfish/Stockfish/pull/2931 fixes https://github.com/official-stockfish/Stockfish/issues/2907 requires fishtest updates before commit Bench: 4290577
2020-08-07 09:07:46 -06:00
/// where to look for positions in FEN format, the type of the limit:
/// depth, perft, nodes and movetime (in millisecs), and evaluation type
/// mixed (default), classical, NNUE.
Restore perft Rewrite perft to be placed naturally inside new bench code. In particular we don't have special custom code to run perft anymore but perft is just a new parameter of 'go' command. So user API is now changed, old style command: $perft 5 becomes $go perft 4 No functional change.
2017-08-15 02:05:22 -06:00
///
/// bench -> search default positions up to depth 13
/// bench 64 1 15 -> search default positions up to depth 15 (TT = 64MB)
/// bench 64 4 5000 current movetime -> search current position with 4 threads for 5 sec
/// bench 64 1 100000 default nodes -> search default positions for 100K nodes each
/// bench 16 1 5 default perft -> run a perft 5 on default positions
Initial import of Glaurung 2.1
2008-08-31 23:59:13 -06:00
Wide bench coverage Add tests for: - Positions with move list - Chess960 positions Now bench covers almost all cases, only few endgames are still out of reach (verified with lcov) It is a non functionality patch, but bench changed because we added new test positions. bench: 5479946
2017-08-15 03:51:14 -06:00
vector<string> setup_bench(const Position& current, istream& is) {
Reformat benchmark interface Prepare to following patches, still no functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2008-09-18 02:54:44 -06:00
Rewrite benchmark First step in improving bench to handle arbitrary UCI commands so to test many more code paths. This first patch just set the new code structure. No functional change.
2017-08-14 11:41:04 -06:00
vector<string> fens, list;
string go, token;
Initial import of Glaurung 2.1
2008-08-31 23:59:13 -06:00
Tidy up benchmark.cpp Node count is different just becuase now we don't log on "bench.txt" file anymore so that we avoid some calls to pretty_pv() that calls Position::do_move(). Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2011-04-25 03:46:52 -06:00
// Assign default values to missing arguments
Fix Hash in bench The compiler tries to cast Options["Hash"] into a string, using: Option::operator std::string() const { assert(type == "string"); return currentValue; } And, as expected, the assert() fails. std::to_string() would be the right solution, but it's C++11. And using a stringstream is too much code to achieve so little. Let's keep it the way it was: hardcoded (ie. default hash defined in two places). No functional change.
2014-08-07 04:51:07 -06:00
string ttSize = (is >> token) ? token : "16";
Process 'bench' also from SF prompt It is possible to start with 'stockfish', then from command prompt type 'bench' and SF will do what you expect. Old behaviour is anyhow preserved. As a bonus we can now start from command line any UCI command understood by Stockfish. The difference is that after execution of a command from arguments SF quits, while at the end of the same command from prompt SF stays in UCI loop. No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2012-04-01 02:25:07 -06:00
string threads = (is >> token) ? token : "1";
Add more depth/positions to bench Increase bench default depth from 12 to 13 and add 15 new endgame positions to have broader coverage and also more reliable nps calulcation used for fishtest framework. Due to the new endgame positions, where nps is higher, the total nps is increased of about 15%. Thanks to Lucas and Jörg for the suggestions. No functional change, but bench number is now: bench: 8336338
2013-09-29 01:43:10 -06:00
string limit = (is >> token) ? token : "13";
Process 'bench' also from SF prompt It is possible to start with 'stockfish', then from command prompt type 'bench' and SF will do what you expect. Old behaviour is anyhow preserved. As a bonus we can now start from command line any UCI command understood by Stockfish. The difference is that after execution of a command from arguments SF quits, while at the end of the same command from prompt SF stays in UCI loop. No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2012-04-01 02:25:07 -06:00
string fenFile = (is >> token) ? token : "default";
string limitType = (is >> token) ? token : "depth";
This commit enables a mixed bench, to improve CI and allow for PGO (profile-build) of the NNUE part of the code. Joint work gvreuls / vondele * Download the default NNUE net in AppVeyor * Download net in travis CI `make net` * Adjust tests to cover more archs, speedup instrumented testing * Introduce 'mixed' bench as default, with further options: classical, NNUE, mixed. mixed (default) and NNUE require the default net to be present, which can be obtained with ``` make net ``` Further examples (first is equivalent to `./stockfish bench`): ``` ./stockfish bench 16 1 13 default depth mixed ./stockfish bench 16 1 13 default depth classical ./stockfish bench 16 1 13 default depth NNUE ``` The net is now downloaded automatically if needed for `profile-build` (usual `build` works fine without net present) PGO gives a nice speedup on fishtest: passed STC: LLR: 2.93 (-2.94,2.94) {-0.50,1.50} Total: 3360 W: 469 L: 343 D: 2548 Ptnml(0-2): 20, 246, 1030, 356, 28 https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569 passed LTC: LLR: 2.97 (-2.94,2.94) {0.25,1.75} Total: 8824 W: 609 L: 502 D: 7713 Ptnml(0-2): 8, 430, 3438, 519, 17 https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c closes https://github.com/official-stockfish/Stockfish/pull/2931 fixes https://github.com/official-stockfish/Stockfish/issues/2907 requires fishtest updates before commit Bench: 4290577
2020-08-07 09:07:46 -06:00
string evalType = (is >> token) ? token : "mixed";
Let benchmark to default to depth 12 And also simplify a lot the code. No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2010-11-08 03:08:01 -07:00
Extend bench to static evaluations this patch extends bench to print static evaluations. ./stockfish bench 16 1 1 filename eval will now print the evaluations for all fens in the file. This complements the various 'go' flavors for bench and might be useful for debugging and/or tuning. No functional change.
2019-11-27 11:03:23 -07:00
go = limitType == "eval" ? "eval" : "go " + limitType + " " + limit;
Teach Benchmark to read positions from a file This comes handy to test the engine on different sets of positions, not only the 15 default ones. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2008-09-18 02:55:35 -06:00
Use bench to implement UI 'perft' command Now that we can call bench on current position we can directly use it to perform our perft. No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2012-04-07 06:23:43 -06:00
if (fenFile == "default")
Import C++11 branch Import C++11 branch from: https://github.com/mcostalba/Stockfish/tree/c++11 The version imported is teh last one as of today: https://github.com/mcostalba/Stockfish/commit/6670e93e50a7a4e739ac2ac8b6026ffa3a12150a Branch is fully equivalent with master but syzygy tablebases that are missing (but will be added with next commit). bench: 8080602
2015-01-18 00:00:50 -07:00
fens = Defaults;
Use bench to implement UI 'perft' command Now that we can call bench on current position we can directly use it to perform our perft. No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2012-04-07 06:23:43 -06:00
else if (fenFile == "current")
Don't access std::cout from Position::print() Let the caller to decide where to redirect (cout or cerr) the ASCII representation of the position. Rename the function to reflect this. Renamed also from_fen() and to_fen() to set() and fen() respectively. No functional change.
2012-11-11 05:18:10 -07:00
fens.push_back(current.fen());
Use bench to implement UI 'perft' command Now that we can call bench on current position we can directly use it to perform our perft. No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2012-04-07 06:23:43 -06:00
else
Teach Benchmark to read positions from a file This comes handy to test the engine on different sets of positions, not only the 15 default ones. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2008-09-18 02:55:35 -06:00
{
Tidy up benchmark.cpp Node count is different just becuase now we don't log on "bench.txt" file anymore so that we avoid some calls to pretty_pv() that calls Position::do_move(). Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2011-04-25 03:46:52 -06:00
string fen;
Import C++11 branch Import C++11 branch from: https://github.com/mcostalba/Stockfish/tree/c++11 The version imported is teh last one as of today: https://github.com/mcostalba/Stockfish/commit/6670e93e50a7a4e739ac2ac8b6026ffa3a12150a Branch is fully equivalent with master but syzygy tablebases that are missing (but will be added with next commit). bench: 8080602
2015-01-18 00:00:50 -07:00
ifstream file(fenFile);
Tidy up benchmark.cpp Node count is different just becuase now we don't log on "bench.txt" file anymore so that we avoid some calls to pretty_pv() that calls Position::do_move(). Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2011-04-25 03:46:52 -06:00
Assorted cleanups in benchmark.cpp No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2011-12-30 06:40:35 -07:00
if (!file.is_open())
Teach Benchmark to read positions from a file This comes handy to test the engine on different sets of positions, not only the 15 default ones. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2008-09-18 02:55:35 -06:00
{
Tidy up benchmark.cpp No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2011-07-24 01:26:35 -06:00
cerr << "Unable to open file " << fenFile << endl;
Rewrite benchmark First step in improving bench to handle arbitrary UCI commands so to test many more code paths. This first patch just set the new code structure. No functional change.
2017-08-14 11:41:04 -06:00
exit(EXIT_FAILURE);
Teach Benchmark to read positions from a file This comes handy to test the engine on different sets of positions, not only the 15 default ones. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2008-09-18 02:55:35 -06:00
}
Tidy up benchmark.cpp No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2011-07-24 01:26:35 -06:00
Assorted cleanups in benchmark.cpp No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2011-12-30 06:40:35 -07:00
while (getline(file, fen))
Tidy up benchmark.cpp No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2011-07-24 01:26:35 -06:00
if (!fen.empty())
Assorted cleanups in benchmark.cpp No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2011-12-30 06:40:35 -07:00
fens.push_back(fen);
Tidy up benchmark.cpp No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2011-07-24 01:26:35 -06:00
Assorted cleanups in benchmark.cpp No functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2011-12-30 06:40:35 -07:00
file.close();
Tidy up benchmark.cpp Node count is different just becuase now we don't log on "bench.txt" file anymore so that we avoid some calls to pretty_pv() that calls Position::do_move(). Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2011-04-25 03:46:52 -06:00
}
Reformat benchmark interface Prepare to following patches, still no functional change. Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2008-09-18 02:54:44 -06:00
Rewrite benchmark First step in improving bench to handle arbitrary UCI commands so to test many more code paths. This first patch just set the new code structure. No functional change.
2017-08-14 11:41:04 -06:00
list.emplace_back("setoption name Threads value " + threads);
list.emplace_back("setoption name Hash value " + ttSize);
Try to get a more precise bench time (#2211) Initialization of larger hash sizes can take some time. Don't include this time in the bench by resetting the timer after Search::clear(). Also move 'ucinewgame' command down in the list, so that it is processed after the configuration of Threads and Hash size. No functional change.
2019-06-30 07:16:20 -06:00
list.emplace_back("ucinewgame");
Add support for saving timing file during benchmark Add a new argument to bench to specify the name of the file where timing information will be saved for each benchmark session. This argument is optional, if not specified file will not be created. Original patch by Heinz van Saanen Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2009-07-04 03:32:51 -06:00
This commit enables a mixed bench, to improve CI and allow for PGO (profile-build) of the NNUE part of the code. Joint work gvreuls / vondele * Download the default NNUE net in AppVeyor * Download net in travis CI `make net` * Adjust tests to cover more archs, speedup instrumented testing * Introduce 'mixed' bench as default, with further options: classical, NNUE, mixed. mixed (default) and NNUE require the default net to be present, which can be obtained with ``` make net ``` Further examples (first is equivalent to `./stockfish bench`): ``` ./stockfish bench 16 1 13 default depth mixed ./stockfish bench 16 1 13 default depth classical ./stockfish bench 16 1 13 default depth NNUE ``` The net is now downloaded automatically if needed for `profile-build` (usual `build` works fine without net present) PGO gives a nice speedup on fishtest: passed STC: LLR: 2.93 (-2.94,2.94) {-0.50,1.50} Total: 3360 W: 469 L: 343 D: 2548 Ptnml(0-2): 20, 246, 1030, 356, 28 https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569 passed LTC: LLR: 2.97 (-2.94,2.94) {0.25,1.75} Total: 8824 W: 609 L: 502 D: 7713 Ptnml(0-2): 8, 430, 3438, 519, 17 https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c closes https://github.com/official-stockfish/Stockfish/pull/2931 fixes https://github.com/official-stockfish/Stockfish/issues/2907 requires fishtest updates before commit Bench: 4290577
2020-08-07 09:07:46 -06:00
size_t posCounter = 0;
Rewrite benchmark First step in improving bench to handle arbitrary UCI commands so to test many more code paths. This first patch just set the new code structure. No functional change.
2017-08-14 11:41:04 -06:00
for (const string& fen : fens)
Wide bench coverage Add tests for: - Positions with move list - Chess960 positions Now bench covers almost all cases, only few endgames are still out of reach (verified with lcov) It is a non functionality patch, but bench changed because we added new test positions. bench: 5479946
2017-08-15 03:51:14 -06:00
if (fen.find("setoption") != string::npos)
list.emplace_back(fen);
else
{
This commit enables a mixed bench, to improve CI and allow for PGO (profile-build) of the NNUE part of the code. Joint work gvreuls / vondele * Download the default NNUE net in AppVeyor * Download net in travis CI `make net` * Adjust tests to cover more archs, speedup instrumented testing * Introduce 'mixed' bench as default, with further options: classical, NNUE, mixed. mixed (default) and NNUE require the default net to be present, which can be obtained with ``` make net ``` Further examples (first is equivalent to `./stockfish bench`): ``` ./stockfish bench 16 1 13 default depth mixed ./stockfish bench 16 1 13 default depth classical ./stockfish bench 16 1 13 default depth NNUE ``` The net is now downloaded automatically if needed for `profile-build` (usual `build` works fine without net present) PGO gives a nice speedup on fishtest: passed STC: LLR: 2.93 (-2.94,2.94) {-0.50,1.50} Total: 3360 W: 469 L: 343 D: 2548 Ptnml(0-2): 20, 246, 1030, 356, 28 https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569 passed LTC: LLR: 2.97 (-2.94,2.94) {0.25,1.75} Total: 8824 W: 609 L: 502 D: 7713 Ptnml(0-2): 8, 430, 3438, 519, 17 https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c closes https://github.com/official-stockfish/Stockfish/pull/2931 fixes https://github.com/official-stockfish/Stockfish/issues/2907 requires fishtest updates before commit Bench: 4290577
2020-08-07 09:07:46 -06:00
if (evalType == "classical" || (evalType == "mixed" && posCounter % 2 == 0))
list.emplace_back("setoption name Use NNUE value false");
else if (evalType == "NNUE" || (evalType == "mixed" && posCounter % 2 != 0))
list.emplace_back("setoption name Use NNUE value true");
Wide bench coverage Add tests for: - Positions with move list - Chess960 positions Now bench covers almost all cases, only few endgames are still out of reach (verified with lcov) It is a non functionality patch, but bench changed because we added new test positions. bench: 5479946
2017-08-15 03:51:14 -06:00
list.emplace_back("position fen " + fen);
list.emplace_back(go);
This commit enables a mixed bench, to improve CI and allow for PGO (profile-build) of the NNUE part of the code. Joint work gvreuls / vondele * Download the default NNUE net in AppVeyor * Download net in travis CI `make net` * Adjust tests to cover more archs, speedup instrumented testing * Introduce 'mixed' bench as default, with further options: classical, NNUE, mixed. mixed (default) and NNUE require the default net to be present, which can be obtained with ``` make net ``` Further examples (first is equivalent to `./stockfish bench`): ``` ./stockfish bench 16 1 13 default depth mixed ./stockfish bench 16 1 13 default depth classical ./stockfish bench 16 1 13 default depth NNUE ``` The net is now downloaded automatically if needed for `profile-build` (usual `build` works fine without net present) PGO gives a nice speedup on fishtest: passed STC: LLR: 2.93 (-2.94,2.94) {-0.50,1.50} Total: 3360 W: 469 L: 343 D: 2548 Ptnml(0-2): 20, 246, 1030, 356, 28 https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569 passed LTC: LLR: 2.97 (-2.94,2.94) {0.25,1.75} Total: 8824 W: 609 L: 502 D: 7713 Ptnml(0-2): 8, 430, 3438, 519, 17 https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c closes https://github.com/official-stockfish/Stockfish/pull/2931 fixes https://github.com/official-stockfish/Stockfish/issues/2907 requires fishtest updates before commit Bench: 4290577
2020-08-07 09:07:46 -06:00
++posCounter;
Wide bench coverage Add tests for: - Positions with move list - Chess960 positions Now bench covers almost all cases, only few endgames are still out of reach (verified with lcov) It is a non functionality patch, but bench changed because we added new test positions. bench: 5479946
2017-08-15 03:51:14 -06:00
}
Add support for saving timing file during benchmark Add a new argument to bench to specify the name of the file where timing information will be saved for each benchmark session. This argument is optional, if not specified file will not be created. Original patch by Heinz van Saanen Signed-off-by: Marco Costalba <mcostalba@gmail.com>
2009-07-04 03:32:51 -06:00
Rewrite benchmark First step in improving bench to handle arbitrary UCI commands so to test many more code paths. This first patch just set the new code structure. No functional change.
2017-08-14 11:41:04 -06:00
return list;
Initial import of Glaurung 2.1
2008-08-31 23:59:13 -06:00
}