Patterns Game benchmark file

Everything about Sudoku that doesn't fit in one of the other sections

Patterns Game benchmark file

Postby m_b_metcalf » Thu Nov 19, 2015 8:28 am

Up until the end of September, the Patterns Game had received 26855 puzzle submissions (excluding 'mores'). Their ratings are spread rather uniformly across the whole range of SE ratings, tailing off, of course, towards the high end. A randomized file of those puzzles, with ratings and authors, would perhaps be useful as a benchmark for solvers. However, the file I have prepared cannot be attached here because, even zipped, it is about 800KB long.

Is there anyone who could and would host the file?

A 'taster' file with 1000 puzzles selected at random is attached. (The ratings are multiplied by 10 and given as integers.)

Regards,

Mike Metcalf


P.S. The complete file corresponds to the ratings data base posted here.
Attachments
PG1000.txt
PG1000
(103.52 KiB) Downloaded 53 times
User avatar
m_b_metcalf
2017 Supporter
 
Posts: 7938
Joined: 15 May 2006
Location: Berlin

Re: Patterns Game benchmark file

Postby champagne » Thu Nov 19, 2015 10:00 am

Hi mike,

I can add that file to others located in a visible part of my google drive (as for example the data base of potential hardest).
As it is not a very big file, I can also put it in my web site.

If nobody has a better idea, we can organize that.

I agree with you that it is a very good sample file for a solver.
champagne
2017 Supporter
 
Posts: 5445
Joined: 02 August 2007
Location: France Brittany

Re: Patterns Game benchmark file

Postby champagne » Thu Nov 19, 2015 2:08 pm

Hi mike,

Just a question on your file that I just open.

I have in my own old and hand collected file that puzzle

010000020230000045000205000006732400000806000007954600000308000850000036040000090

the only one rating 1.2 1.0 1.0

I can not find it in your file.

My question :

What games are collected in your file (I restarted from the old game thread years ago when I prepared my own file).
champagne
2017 Supporter
 
Posts: 5445
Joined: 02 August 2007
Location: France Brittany

Re: Patterns Game benchmark file

Postby m_b_metcalf » Thu Nov 19, 2015 3:19 pm

champagne wrote:What games are collected in your file (I restarted from the old game thread years ago when I prepared my own file).

Well, I really don't remember how this started, but in principle it contains all games from the point where the results were published in the 'modern' format and, indeed, it contains game 14. The idea was to match g.r.emlins's records, which also start then. However, the beginning of my source file is not in chronological order, the first entry coming from game 68 (with hindsight I should have added the game numbers to the file). Your puzzle comes from game 7.

Regards,

Mike
Last edited by m_b_metcalf on Thu Nov 19, 2015 5:56 pm, edited 1 time in total.
User avatar
m_b_metcalf
2017 Supporter
 
Posts: 7938
Joined: 15 May 2006
Location: Berlin

Re: Patterns Game benchmark file

Postby champagne » Thu Nov 19, 2015 4:35 pm

I loaded Mike's file in my site.

It can be down loaded from games_sample
In addition to the file supplied by mike, I added a smal file containing some rating EP=ED=1.0
champagne
2017 Supporter
 
Posts: 5445
Joined: 02 August 2007
Location: France Brittany

Re: Patterns Game benchmark file

Postby dobrichev » Thu Nov 19, 2015 5:16 pm

I would use such file for benchmarking and agree this is a good idea that makes sense.
On the other hand I wouldn't optimize a solver for this puzzle collection because
a) all puzzles are valid and minimal
b) the patterns are biased
c) the complexity distribution between the solving steps is extremely biased
dobrichev
2016 Supporter
 
Posts: 1295
Joined: 24 May 2010

Re: Patterns Game benchmark file

Postby m_b_metcalf » Thu Nov 19, 2015 6:52 pm

dobrichev wrote:I would use such file for benchmarking and agree this is a good idea that makes sense.
On the other hand I wouldn't optimize a solver for this puzzle collection because
a) all puzzles are valid and minimal
b) the patterns are biased
c) the complexity distribution between the solving steps is extremely biased


One can, of course:

a) in each puzzle delete a random clue to create a puzzle with multiple solutions; add a valid clue to create a non-minimal puzzle; and add an invalid clue to create an invalid puzzle.

b) yes.

c) yes, but it's a test of a program to find one and only one solution to each puzzle (I have already located a bug in one of my solvers by running the complete set through it).

Regards,

Mike Metcalf
User avatar
m_b_metcalf
2017 Supporter
 
Posts: 7938
Joined: 15 May 2006
Location: Berlin

Re: Patterns Game benchmark file

Postby champagne » Thu Nov 19, 2015 8:17 pm

dobrichev wrote:I would use such file for benchmarking and agree this is a good idea that makes sense.
On the other hand I wouldn't optimize a solver for this puzzle collection because


For me this is not a file for benchmarks, but more a (partial) collection of all classical rules that a solver must use.

Perfect for a first cleaning of a code prepared to "clone" serate, but a very good start for any solver.

Many "classical" rules are missing in serate. For example

The UR analysis is relatively limited,
Basis chains as sky scrapers, kites, ... are not clearly identified
...
and surely this is not a sample for exotic patterns,
but knowing the difficulty to build a sample file, it is a very good tool as start.



BTW, "optimizing a solver" has surely as many definitions as "rating a puzzle".
champagne
2017 Supporter
 
Posts: 5445
Joined: 02 August 2007
Location: France Brittany


Return to General

cron