Patterns Game benchmark file

Everything about Sudoku that doesn't fit in one of the other sections

Patterns Game benchmark file

Postby m_b_metcalf » Thu Nov 19, 2015 8:28 am

Up until the end of September, the Patterns Game had received 26855 puzzle submissions (excluding 'mores'). Their ratings are spread rather uniformly across the whole range of SE ratings, tailing off, of course, towards the high end. A randomized file of those puzzles, with ratings and authors, would perhaps be useful as a benchmark for solvers. However, the file I have prepared cannot be attached here because, even zipped, it is about 800KB long.

Is there anyone who could and would host the file?

A 'taster' file with 1000 puzzles selected at random is attached. (The ratings are multiplied by 10 and given as integers.)

Regards,

Mike Metcalf


P.S. The complete file corresponds to the ratings data base posted here.
Attachments
PG1000.txt
PG1000
(103.52 KiB) Downloaded 156 times
User avatar
m_b_metcalf
2017 Supporter
 
Posts: 10984
Joined: 15 May 2006
Location: Berlin

Re: Patterns Game benchmark file

Postby champagne » Thu Nov 19, 2015 10:00 am

Hi mike,

I can add that file to others located in a visible part of my google drive (as for example the data base of potential hardest).
As it is not a very big file, I can also put it in my web site.

If nobody has a better idea, we can organize that.

I agree with you that it is a very good sample file for a solver.
champagne
2017 Supporter
 
Posts: 7154
Joined: 02 August 2007
Location: France Brittany

Re: Patterns Game benchmark file

Postby champagne » Thu Nov 19, 2015 2:08 pm

Hi mike,

Just a question on your file that I just open.

I have in my own old and hand collected file that puzzle

010000020230000045000205000006732400000806000007954600000308000850000036040000090

the only one rating 1.2 1.0 1.0

I can not find it in your file.

My question :

What games are collected in your file (I restarted from the old game thread years ago when I prepared my own file).
champagne
2017 Supporter
 
Posts: 7154
Joined: 02 August 2007
Location: France Brittany

Re: Patterns Game benchmark file

Postby m_b_metcalf » Thu Nov 19, 2015 3:19 pm

champagne wrote:What games are collected in your file (I restarted from the old game thread years ago when I prepared my own file).

Well, I really don't remember how this started, but in principle it contains all games from the point where the results were published in the 'modern' format and, indeed, it contains game 14. The idea was to match g.r.emlins's records, which also start then. However, the beginning of my source file is not in chronological order, the first entry coming from game 68 (with hindsight I should have added the game numbers to the file). Your puzzle comes from game 7.

Regards,

Mike
Last edited by m_b_metcalf on Thu Nov 19, 2015 5:56 pm, edited 1 time in total.
User avatar
m_b_metcalf
2017 Supporter
 
Posts: 10984
Joined: 15 May 2006
Location: Berlin

Re: Patterns Game benchmark file

Postby champagne » Thu Nov 19, 2015 4:35 pm

I loaded Mike's file in my site.

It can be down loaded from games_sample
In addition to the file supplied by mike, I added a smal file containing some rating EP=ED=1.0
champagne
2017 Supporter
 
Posts: 7154
Joined: 02 August 2007
Location: France Brittany

Re: Patterns Game benchmark file

Postby dobrichev » Thu Nov 19, 2015 5:16 pm

I would use such file for benchmarking and agree this is a good idea that makes sense.
On the other hand I wouldn't optimize a solver for this puzzle collection because
a) all puzzles are valid and minimal
b) the patterns are biased
c) the complexity distribution between the solving steps is extremely biased
dobrichev
2016 Supporter
 
Posts: 1784
Joined: 24 May 2010

Re: Patterns Game benchmark file

Postby m_b_metcalf » Thu Nov 19, 2015 6:52 pm

dobrichev wrote:I would use such file for benchmarking and agree this is a good idea that makes sense.
On the other hand I wouldn't optimize a solver for this puzzle collection because
a) all puzzles are valid and minimal
b) the patterns are biased
c) the complexity distribution between the solving steps is extremely biased


One can, of course:

a) in each puzzle delete a random clue to create a puzzle with multiple solutions; add a valid clue to create a non-minimal puzzle; and add an invalid clue to create an invalid puzzle.

b) yes.

c) yes, but it's a test of a program to find one and only one solution to each puzzle (I have already located a bug in one of my solvers by running the complete set through it).

Regards,

Mike Metcalf
User avatar
m_b_metcalf
2017 Supporter
 
Posts: 10984
Joined: 15 May 2006
Location: Berlin

Re: Patterns Game benchmark file

Postby champagne » Thu Nov 19, 2015 8:17 pm

dobrichev wrote:I would use such file for benchmarking and agree this is a good idea that makes sense.
On the other hand I wouldn't optimize a solver for this puzzle collection because


For me this is not a file for benchmarks, but more a (partial) collection of all classical rules that a solver must use.

Perfect for a first cleaning of a code prepared to "clone" serate, but a very good start for any solver.

Many "classical" rules are missing in serate. For example

The UR analysis is relatively limited,
Basis chains as sky scrapers, kites, ... are not clearly identified
...
and surely this is not a sample for exotic patterns,
but knowing the difficulty to build a sample file, it is a very good tool as start.



BTW, "optimizing a solver" has surely as many definitions as "rating a puzzle".
champagne
2017 Supporter
 
Posts: 7154
Joined: 02 August 2007
Location: France Brittany

4281 unique patterns game puzzles

Postby 1to9only » Wed Sep 18, 2019 10:33 pm

File contains 4281 unique ER/EP/ED puzzles submitted in the patterns games 1 to 360 inclusive of :::more::: entries.


The first puzzle submitted (in each rating) is listed, with game number, submission ordinal, ER/EP/ED (as submitted), rarity/more flag, and puzzle submitter shown. I have verified the ratings are correct (using SE 1.2.1.3).

[Edit 21/09] I've temporarily removed the file, as I've found at least 1 error in it (using SE 1.2.1.3):
Code: Select all
...3..2.....7.8.1.....59..424.....5...1...7...6.....898..51.....3.4.7.....6..3... # 0154 206    4.9/4.2/2.6 ED=4.6/4.2/2.6

I'll try to create a new list, and validate it before posting! :oops:
.
Last edited by 1to9only on Sat Sep 21, 2019 9:12 am, edited 1 time in total.
1to9only
 
Posts: 1807
Joined: 04 April 2018

Re: 4281 unique patterns game puzzles

Postby tarek » Thu Sep 19, 2019 1:02 pm

1to9only wrote:File contains 4281 unique ER/EP/ED puzzles submitted in the patterns games 1 to 360 inclusive of :::more::: entries.

pg-0001-0360-uniques.zip

The first puzzle submitted (in each rating) is listed, with game number, submission ordinal, ER/EP/ED (as submitted), rarity/more flag, and puzzle submitter shown. I have verified the ratings are correct (using SE 1.2.1.3).
.

Very helpful. thanks

tarek
User avatar
tarek
 
Posts: 3726
Joined: 05 January 2006

4279 unique patterns game puzzles

Postby 1to9only » Sun Sep 22, 2019 12:26 pm

Apologies to all who downloaded (and used) the 4281 puzzles file - I think it contained between 4-6 misrated puzzles.
New file contains 4279 unique ER/EP/ED puzzles submitted in the patterns games 1 to 360 inclusive of :::more::: entries.
I have re-verified the ratings are correct (using SE 1.2.1.3 (304,777 bytes)) - this took some time!
A custom SukakuExplainer built (a few days ago) from recent github source but WITHOUT the lksudoku's BUG2 and UR/UL fixes was also used - this rated the puzzles same as SE.
I have not tried with the SukakuExplainer 1.4.0 released earlier today.

[Edit] updated 4353 unique patterns game puzzles is here.
.
Last edited by 1to9only on Tue May 26, 2020 12:53 pm, edited 1 time in total.
1to9only
 
Posts: 1807
Joined: 04 April 2018

Re: Patterns Game benchmark file

Postby champagne » Mon Sep 30, 2019 2:10 pm

My file of puzzles published in the pattern games has now 36890 records.
This includes 'more' puzzles.

As far as I remember, only one very old puzzle was published with a rating not in line with the referee current rating.
I update this file on a regular basis to catch the potential hardest published and to have a sample file to test clones of serate.
If anybody is interested, I'll load it in my google drive.

For sure, there will be redundancy with other files
here after an example of the file structure

Code: Select all
.....1..2.....345...4.5..67...2...8...8.34...52..8...4.3......9.813...2.7.9..26..;1.2;1.2;1.2; ; m_b_metcalf
.....1.23..2.....4.5....6......1..37...4.58..2...9..6...6.7..4.5..8.69..87.9.....;1.2;1.2;1.2;+; m_b_metcalf  <<< here a 'more' puzzle
champagne
2017 Supporter
 
Posts: 7154
Joined: 02 August 2007
Location: France Brittany

Re: Patterns Game benchmark file

Postby rjamil » Sun Apr 26, 2020 8:26 pm

Hi experts,

May I ask, how you people collect puzzles published in the pattern games? Specially 'more' puzzles?

I have written a small c language program recently and sharing here for public use freely. It reads raw data copied from up till now 26 Patterns Game Results pages and display the filtered puzzle lines on console/screen. (Just open the first page; then press Ctrl+A (or Command+C on MAC) to select the entire text; open and paste it in to blank PGResult.txt file in Notepad (or TextEdit on MAC) by pressing Ctrl+V (or Command+V on MAC). Open second page; copy entire text; paste in to PGResult.txt at bottom of the file. Repeat copy/paste steps for all 26 pages)

Copy below given c source code and paste it in to RJPGR.c program file and save it to the same folder/directory where PGResult.txt file created/saved. Compile with any suitable c compiler as per normal practice and run it from same directory through DOS command window. The program will display the filtered puzzle lines on console/screen that will be easily redirected to any file to save:

Code: Select all
#include <stdio.h>

int main (void)
{
  char a[1024];

  FILE *f = fopen ("PGResult.txt", "r");

  if (f == NULL)
    printf ("Error: Unable to open PGResult.txt file for read !!\n");
  else
    while (fgets (a, 1024, f) != NULL)
      if (a[0] >= '0' && a[0] <= '9')
        printf ("%s", a);
  if (fclose (f) == EOF)
    printf ("Error: Unable to close PGResult.txt file !!");
}


For your kind information, I have saved all 26 pages in to RJResult.txt file, run RJPGR.exe program and get total 39,540 puzzles till now.

However, not confirm about whether 'more' puzzles are included or not.

R. Jamil
rjamil
 
Posts: 573
Joined: 15 October 2014
Location: Karachi, Pakistan

Re: Patterns Game benchmark file

Postby 1to9only » Mon Apr 27, 2020 10:23 am

Here's what I did and do:

Add '&view=print' to the url, e.g. http://forum.enjoysudoku.com/viewtopic.php?f=8&t=6291&view=print - use copy&paste to save the contents to a text file. Repeat for 26 pages.

I think I did this process once a while back, nowadays I only pick up the results of the last game to append to the results file.
There are some missing puzzles (around games 12-13), these can be obtained from the patterns game pages.
I re-rated some of the early puzzles (games 1-14) to obtain ER/EP/ED ratings. I did this once for the results file.

The results file is processed into an intermediate file:

Pick out the line 'game NNNN YYYY-MM-DD+HH:MM:SS-0000 ...' for the game number.
Convert empty cells '0' to dots '.'.
Include the game number in the output line, e.g.
..1...2...2...4.1.6...8...4..86.2..5....1....4..3.79..5...9...7.1.5...3...3...1.. # 0382 1 1.5/1.2/1.2 - champagne

For games 0001-0381 inclusive, I have 39,128 entries. This includes updated games 12-13 and all more entries.
I do further processing on this intermediate file, e.g. for rare entries, unique entries, etc.
.
1to9only
 
Posts: 1807
Joined: 04 April 2018

Re: Patterns Game benchmark file

Postby m_b_metcalf » Mon Apr 27, 2020 11:23 am

1to9only wrote:Here's what I did and do:

I do it roughly the same way, starting from 2014. My big mistake was not recording the game number with the data, but I have written a utility program that gets round that when necessary.

If I were to start again, I would investigate web scraping.

Mike
User avatar
m_b_metcalf
2017 Supporter
 
Posts: 10984
Joined: 15 May 2006
Location: Berlin

Next

Return to General