13x13 Scalability study

58851 games played so far.
Computed by bayeselo

Last update: Tue Apr 29 10:27:34 EDT 2008

The purpose of this study is to determine the effects of additional computing power for scalable 13x13 Computer Go programs. The programs that are being tested use a popular new technique based on Monte Carlo simulations (randomly played games in this case) combined with best first tree searching called UCT.

This study tests programs at many different level in a competition for ELO rating points. Elo ratings are a direct measure of playing strength. Each version of a particular program spends twice as much effort playing it's move as the previous version, as measured by the number of random games played to choose a move.

Matches are scheduled between players using a random scheduling algorithm where players of similar strength are much more likely to play each other, but any pairing is still possible. This works far better for assessment purposes than, for instance, a round robin where matches between players of considerably different skill are common.

The amount of computing effort required to test these programs at higher levels is substantial and several people have contributed computing resources to this effort. On a single computer this study would require many months of computing effort in order to get enough data to be statistically meaningful.

Currently, two programs are participating in this study, Mogo, and Leela. Mogo is one of the strongest, if not strongest, 13x13 go playing program in the world at this time (early 2008.) Leela is also one of the top playing programs on the CGOS server at small board sizes.

One additional program plays as a reference point, a popular open source program called "Gnugo" which plays at a fixed level of 1800 ELO.

In the following table, there are various versions of mogo, labeled 01, 02, 03, .... Mogo_01 does only 128 monte carlo simulations in the evaluation porition of the tree search. Mogo_02 doubles this and each subsequent version doubles the previous in number of simulations.

Likewise, Leela is labelled similarly. Both Leela and Mogo do the same number of simulations at the coresponding levels, i.e. 128 monte carlo simulation at level 01.

We can compute the number of simulations for any level as:

Mogo and Leela simulations at level N = 128 * 2^(N-1)

Rating Chart

Rank Name Elo + Games score opponent
1 Mogo_13_15 2880 21 20 2340 84% 2457
2 Mogo_13_14 2808 19 18 2614 79% 2396
3 Mogo_13_13 2717 16 16 3039 70% 2423
4 Leela2_13 2638 21 21 2367 80% 2191
5 Mogo_13_12 2610 16 16 3051 60% 2428
6 Leela2_12 2548 20 20 2451 75% 2159
7 Mogo_13_11 2493 15 15 3656 61% 2331
8 Leela2_11 2437 20 20 2458 70% 2106
9 Mogo_13_10 2380 14 15 3733 50% 2330
10 Leela2_10 2332 18 18 2625 63% 2113
11 Mogo_13_09 2270 13 13 4789 54% 2218
12 Leela2_09 2213 19 19 2613 61% 2008
13 Mogo_13_08 2147 13 13 4788 44% 2195
14 Leela2_08 2085 19 19 2656 58% 1925
15 Mogo_13_07 2022 13 13 4693 43% 2100
16 Leela2_07 1932 18 19 2658 56% 1818
17 Mogo_13_06 1903 13 13 4586 43% 1967
18 Leela2_06 1806 18 18 2688 56% 1687
19 Gnugo-3.7.11 1800 12 11 6506 57% 1751
20 Mogo_13_05 1765 13 13 4542 45% 1802
21 LightLeela_13 1745 28 28 969 48% 1802
22 LightLeela_12 1696 28 28 964 47% 1764
23 LightLeela_11 1631 29 29 974 46% 1705
24 Leela2_05 1621 19 19 2772 51% 1583
25 Mogo_13_04 1599 12 12 6046 45% 1631
26 LightLeela_10 1551 30 30 967 46% 1613
27 LightLeela_09 1462 30 30 969 44% 1559
28 Leela2_04 1412 20 20 2785 47% 1458
29 LightLeela_08 1357 31 31 979 44% 1473
30 Mogo_13_03 1338 15 15 5538 49% 1359
31 LightLeela_07 1223 31 31 1021 41% 1384
32 Leela2_03 1169 20 20 2811 41% 1334
33 LightLeela_06 1124 31 31 1034 40% 1322
34 Mogo_13_02 1041 13 13 5500 54% 1082
35 LightLeela_05 946 31 31 1020 39% 1190
36 Leela2_02 913 19 20 2783 31% 1236
37 LightLeela_04 824 30 31 1027 33% 1152
38 Mogo_13_01 745 15 16 4879 16% 1185
39 Leela2_01 693 21 22 2729 16% 1233
40 LightLeela_03 643 32 32 1023 26% 1102
41 LightLeela_02 468 34 35 1027 16% 1062
42 LightLeela_01 370 38 41 1032 9% 1094

Command line program invocation (weakest level)
Mogomogo --13 --nbTotalSimulations 128 --playsAgainstHuman 0
Leelaleela 7       note: level N = 2 ^ N
Leela LiteleelaLight 7       note: level N = 2 ^ N
Gnugo-3.7.11gnugo --mode gtp --capture-all-dead --chinese-rules --min-level 10 --max-level 10 --positional-superko

Rating plot

X axis:Each CPU Doubling
Y axis:ELO Rating

Rating plot with bezeir smoothing