Last update: Tue Apr 29 10:27:34 EDT 2008

The purpose of this study is to determine the effects of additional computing power for scalable 13x13 Computer Go programs. The programs that are being tested use a popular new technique based on Monte Carlo simulations (randomly played games in this case) combined with best first tree searching called UCT.

This study tests programs at many different level in a competition for ELO rating points. Elo ratings are a direct measure of playing strength. Each version of a particular program spends twice as much effort playing it's move as the previous version, as measured by the number of random games played to choose a move.

Matches are scheduled between players using a random scheduling algorithm where players of similar strength are much more likely to play each other, but any pairing is still possible. This works far better for assessment purposes than, for instance, a round robin where matches between players of considerably different skill are common.

The amount of computing effort required to test these programs at higher levels is substantial and several people have contributed computing resources to this effort. On a single computer this study would require many months of computing effort in order to get enough data to be statistically meaningful.

Currently, two programs are participating in this study, Mogo, and Leela. Mogo is one of the strongest, if not strongest, 13x13 go playing program in the world at this time (early 2008.) Leela is also one of the top playing programs on the CGOS server at small board sizes.

One additional program plays as a reference point, a popular open source program called "Gnugo" which plays at a fixed level of 1800 ELO.

In the following table, there are various versions of mogo, labeled 01, 02, 03, .... Mogo_01 does only 128 monte carlo simulations in the evaluation porition of the tree search. Mogo_02 doubles this and each subsequent version doubles the previous in number of simulations.

Likewise, Leela is labelled similarly. Both Leela and Mogo do the same number of simulations at the coresponding levels, i.e. 128 monte carlo simulation at level 01.

We can compute the number of simulations for any level as:

Mogo and Leela simulations at level N = 128 * 2^(N-1)

Rating Chart

Rank	Name	Elo	+	−	Games	score	opponent
1	Mogo_13_15	2880	21	20	2340	84%	2457
2	Mogo_13_14	2808	19	18	2614	79%	2396
3	Mogo_13_13	2717	16	16	3039	70%	2423
4	Leela2_13	2638	21	21	2367	80%	2191
5	Mogo_13_12	2610	16	16	3051	60%	2428
6	Leela2_12	2548	20	20	2451	75%	2159
7	Mogo_13_11	2493	15	15	3656	61%	2331
8	Leela2_11	2437	20	20	2458	70%	2106
9	Mogo_13_10	2380	14	15	3733	50%	2330
10	Leela2_10	2332	18	18	2625	63%	2113
11	Mogo_13_09	2270	13	13	4789	54%	2218
12	Leela2_09	2213	19	19	2613	61%	2008
13	Mogo_13_08	2147	13	13	4788	44%	2195
14	Leela2_08	2085	19	19	2656	58%	1925
15	Mogo_13_07	2022	13	13	4693	43%	2100
16	Leela2_07	1932	18	19	2658	56%	1818
17	Mogo_13_06	1903	13	13	4586	43%	1967
18	Leela2_06	1806	18	18	2688	56%	1687
19	Gnugo-3.7.11	1800	12	11	6506	57%	1751
20	Mogo_13_05	1765	13	13	4542	45%	1802
21	LightLeela_13	1745	28	28	969	48%	1802
22	LightLeela_12	1696	28	28	964	47%	1764
23	LightLeela_11	1631	29	29	974	46%	1705
24	Leela2_05	1621	19	19	2772	51%	1583
25	Mogo_13_04	1599	12	12	6046	45%	1631
26	LightLeela_10	1551	30	30	967	46%	1613
27	LightLeela_09	1462	30	30	969	44%	1559
28	Leela2_04	1412	20	20	2785	47%	1458
29	LightLeela_08	1357	31	31	979	44%	1473
30	Mogo_13_03	1338	15	15	5538	49%	1359
31	LightLeela_07	1223	31	31	1021	41%	1384
32	Leela2_03	1169	20	20	2811	41%	1334
33	LightLeela_06	1124	31	31	1034	40%	1322
34	Mogo_13_02	1041	13	13	5500	54%	1082
35	LightLeela_05	946	31	31	1020	39%	1190
36	Leela2_02	913	19	20	2783	31%	1236
37	LightLeela_04	824	30	31	1027	33%	1152
38	Mogo_13_01	745	15	16	4879	16%	1185
39	Leela2_01	693	21	22	2729	16%	1233
40	LightLeela_03	643	32	32	1023	26%	1102
41	LightLeela_02	468	34	35	1027	16%	1062
42	LightLeela_01	370	38	41	1032	9%	1094

Command line program invocation (weakest level)

Mogo	mogo --13 --nbTotalSimulations 128 --playsAgainstHuman 0
Leela	leela 7       note: level N = 2 ^ N
Leela Lite	leelaLight 7       note: level N = 2 ^ N
Gnugo-3.7.11	gnugo --mode gtp --capture-all-dead --chinese-rules --min-level 10 --max-level 10 --positional-superko

Rating plot

X axis:	Each CPU Doubling
Y axis:	ELO Rating

Rating plot with bezeir smoothing