Low/Hi Clue Thresholds

Everything about Sudoku that doesn't fit in one of the other sections

Re: Low/Hi Clue Thresholds

Postby coloin » Thu Aug 08, 2019 1:46 pm

Yes i tend to agree with you
looking at the batch run a single 19C puzzle was generated x7 to 750,000 puzzles in 4 hours, but a breakdown of solution grids was only 250,000. So indeed your program is 15x quicker !!!
When the LCT-20 completes i will use your program ...
coloin
 
Posts: 1790
Joined: 05 May 2005

Re: Low/Hi Clue Thresholds

Postby Mathimagics » Sat Aug 10, 2019 12:59 pm

Hi coloin,

You (or anybody) can start right now, in fact!

Setup instructions:

  • download Win64.zip (below), create a directory, eg LCT19, and unzip the files into that
  • in that directory, run Gen19C. It will verify that the inboard solver works on your system, or produce an error message otherwise.
  • if solver is ok, fire up 1 or more workers as follows:

    • open a new CMD window
    • go to a Worker directory (eg Worker1)
    • run ..\Gen19C (or, if you prefer, copy Gen19C.exe into each worker directory first)

That's it. I have provided 4 x Worker directories, but you can create extra ones should you get really enthusiastic. Typically a worker will produce a batch of 3 to 6 million puzzles every 3-6 hours. Each batch will be assigned a new batch id number, and the batch files will be deposited in the BatchFiles directory.

You can kill / restart the workers at any time.

The worker process starts by generating 10 x 19C puzzles using random grids and reducing them. This can take several minutes. These are used to seed the morphing logic - each pass generates all simple morphs from the previous pass, until a pass produces 3 million or more. This then becomes a batch, and we start over.

I will discuss batch delivery options in a separate post.

Cheers
MM

[EDIT] The zip files now contain two executables, Gen19C.exe and Gen19J.exe. If Gen19C fails the verification test, remove it and use Gen19J instead. Gen19J uses a CPU-independent solver (JCZsolver) and is only a tad slower than Gen19C.

If you run the workers on different PC's, then in order to ensure all batch numbers are unique, please edit the file "LastBatchN.txt" before running the workers, replacing the "1" with a "2" on the second PC.

[EDIT] 13 Aug 2019: Fixed a problem reported by 1to9only.

Worker32.zip
Worker19C (Win32)
(144.68 KiB) Not downloaded yet

Worker64.zip
Worker19C (Win64)
(162.62 KiB) Downloaded 7 times
Last edited by Mathimagics on Tue Aug 13, 2019 12:06 pm, edited 6 times in total.
User avatar
Mathimagics
2017 Supporter
 
Posts: 1274
Joined: 27 May 2015
Location: Canberra

Re: Low/Hi Clue Thresholds

Postby coloin » Sat Aug 10, 2019 6:11 pm

my oldish but not that old spare dual computor ...is 32 bit i will use it !!!

but my 4 core -64 reported no issues !!!

Code: Select all
Solver verified, ready for work!
....great ! :D

well done in the cricket ! :roll:
Last edited by coloin on Sat Aug 10, 2019 7:12 pm, edited 1 time in total.
coloin
 
Posts: 1790
Joined: 05 May 2005

Re: Low/Hi Clue Thresholds

Postby Mathimagics » Sun Aug 11, 2019 9:02 am

I have updated the zip files (Worker32.zip and Worker64.zip) above to include alternative binaries in the event that Gen19C fails the verification test, or crashes.

An alternative version Gen19J.exe is provided, which uses JCZsolver rather than fsss2, and so should run on any Windows platform. It is only marginally slower than Gen19C.

Cheers,
MM
User avatar
Mathimagics
2017 Supporter
 
Posts: 1274
Joined: 27 May 2015
Location: Canberra

Re: Low/Hi Clue Thresholds

Postby coloin » Sun Aug 11, 2019 10:05 am

Excellent that did the trick.
Now up and running with the Gen19j.exe - 2 workers more.
coloin
 
Posts: 1790
Joined: 05 May 2005

Re: Low/Hi Clue Thresholds

Postby 1to9only » Sun Aug 11, 2019 10:57 am

I probably have a little bit of spare processing capacity in between patern games! Might download and try later!

Out of curiousity, and after a very long run, are we not just generating the same random grids? (a bit like in the patterns game!). I imagine that some (many) batches will overlap, but the goal is to scan the whole puzzle space (again a bit like the patterns game!).
1to9only
 
Posts: 1010
Joined: 04 April 2018

Re: Low/Hi Clue Thresholds

Postby Mathimagics » Sun Aug 11, 2019 12:53 pm

1to9only wrote:... are we not just generating the same random grids? (a bit like in the patterns game!).

I imagine that some (many) batches will overlap, but the goal is to scan the whole puzzle space (again a bit like the patterns game!).


The morph process concentrates on ED grids - so starting from a set of 19C puzzles on ED grids, it generates {-1,+1} morphs, but converts each one found to CF, and only those ones that are "new" are retained as seeds for the next pass.

This proved effective for LCT-20, with some overlap between the batches produced, but no more than 10%. For LCT-19 this should (hopefully) still be the case.

The chances of the puzzles being on new grids, when the batches are applied to the catalog, do naturally diminish over time, but you can be assured that for the first couple of months, at least, any batches that your workers produce will be worthwhile!
User avatar
Mathimagics
2017 Supporter
 
Posts: 1274
Joined: 27 May 2015
Location: Canberra

Re: Low/Hi Clue Thresholds

Postby 1to9only » Mon Aug 12, 2019 5:59 am

Can I obtain the source code for Gen19C.exe? To help debug 2 Worker deaths so far, each after some 9 hours of running!

Code: Select all
18:17:55: Generating initial seeds
...
19:06:53: Processing new items - pass #10
21:45:36: Items in = 717721, items out = 5896101
21:45:41: Sort/merge new items
21:45:42: Pass complete: ED grids/puzzles =   3411795
21:45:42: Processing new items - pass #11
03:22:56: Items in = 1556406, items out = 12936647
03:23:10: Sort/merge new items

Gen19C.App.Err.2.jpg
Gen19C.App.Err.2.jpg (21.37 KiB) Viewed 73 times
1to9only
 
Posts: 1010
Joined: 04 April 2018

Re: Low/Hi Clue Thresholds

Postby 1to9only » Mon Aug 12, 2019 6:04 am

Mathimagics wrote:I'm also looking at a custom compression tool that would give smaller zip files.

7zip produces smaller files than zip.

Batch001002.txt 654,407,275 (639MB)
Batch001002.zip 141,311,440 (138MB) <- using Zip 3.0 (July 5th 2008) from http://infozip.sourceforge.net/
Batch001002.7z 103,461,289 (101MB) <- using 7-Zip 19.00 (2019-02-21) from http://7-zip.org/
.
1to9only
 
Posts: 1010
Joined: 04 April 2018

Re: Low/Hi Clue Thresholds

Postby Mathimagics » Mon Aug 12, 2019 7:51 am

My apologies! :oops:

I think that I have identified the probable cause and will provide an update, and also the source code, ASAP.
Last edited by Mathimagics on Mon Aug 12, 2019 8:30 am, edited 1 time in total.
User avatar
Mathimagics
2017 Supporter
 
Posts: 1274
Joined: 27 May 2015
Location: Canberra

Re: Low/Hi Clue Thresholds

Postby Mathimagics » Mon Aug 12, 2019 8:25 am

Updated exes now available above, and here is the source package. It includes batch files for building Gen19C/J (64-bit), and all additional objects needed for linking.

WorkerSrc.zip
Worker19C (Source pack)
(75.75 KiB) Downloaded 4 times


[EDIT] 13 Aug 2019: Fixed a problem reported by 1to9only.
Last edited by Mathimagics on Tue Aug 13, 2019 12:08 pm, edited 3 times in total.
User avatar
Mathimagics
2017 Supporter
 
Posts: 1274
Joined: 27 May 2015
Location: Canberra

Re: Low/Hi Clue Thresholds

Postby 1to9only » Mon Aug 12, 2019 1:01 pm

I've had 3 Worker crashes in the 1st 24 hrs! I'm running the latest Gen19C.exe now. I had a quick look at the source code, could not spot anything wrong in ConsolidateList() where I think the crash occurred, likely fixed already. I'll run for another 24 hrs, and hopefully no more mishaps...
1to9only
 
Posts: 1010
Joined: 04 April 2018

Re: Low/Hi Clue Thresholds

Postby Mathimagics » Mon Aug 12, 2019 1:46 pm

It's odd that neither coloin nor me have seen this problem ...

If the problem persists, you might try reducing the batch size check at line 1189, eg change "3" to "2":
Code: Select all
      if (NMT > 0x3FFFFF) break;
User avatar
Mathimagics
2017 Supporter
 
Posts: 1274
Joined: 27 May 2015
Location: Canberra

Re: Low/Hi Clue Thresholds

Postby 1to9only » Mon Aug 12, 2019 7:42 pm

This time ALL 4 Workers crashed. Processing is stopped. I'll do some investigations when time permits.

Worker 1: The instruction at 0x0000000000403154 referenced memory at 0x00000001A6012000. The memory could not be written.
Worker 2: The instruction at 0x0000000000403154 referenced memory at 0x00000001A601D000. The memory could not be written.
Worker 3: The instruction at 0x00000000004034A8 referenced memory at 0x00000001A6016000. The memory could not be written.
Worker 4: The instruction at 0x0000000000403154 referenced memory at 0x00000001A601F000. The memory could not be written.

Worker 3 stopped at a different code address.

I've had similar (occasional) failures (The memory could not be read.) when one of my programs usually ran 99.99% percent of the time, on this particular machine, and on other machines as well!
.
1to9only
 
Posts: 1010
Joined: 04 April 2018

Re: Low/Hi Clue Thresholds

Postby blue » Tue Aug 13, 2019 7:34 am

1to9only wrote:II had a quick look at the source code, could not spot anything wrong in ConsolidateList() where I think the crash occurred, likely fixed already.

Mathimagics wrote:It's odd that neither coloin nor me have seen this problem ...

If the problem persists, you might try reducing the batch size check at line 1189, eg change "3" to "2":
Code: Select all
      if (NMT > 0x3FFFFF) break;

It's amazing, that neither of you has seen a crash.
There's a definite problem in ConsolidateList().

Code: Select all
   void AddPuzzleN() {
      memcpy(XT, NT, 166); XT += 166;
      nx++;
      if (nx >= MAXN) {printf("[ERROR] merge table\n"); exit(0);}
      if (nc >= MAXC) return;
      memcpy(NC, NT, 166); NC += 166; nc++; 
      }

   void AddPuzzleM() {
      memcpy(XT, MT, 166);   XT += 166;
      nx++;
      if (nx >= MAXN) {printf("[ERROR] merge table\n"); exit(0);}
      }

A good start, would be to move the "[ERROR]" check lines to in front of the 'memcpy' lines that are (no doubt) causing the crashes.
Also MAXN should be changed to either TABSIZE or MAXC (for versions where MAXN is 12936648 and TABSIZE is only 0x7FFFFF).

The crash can happen in the following circumstance:
  1. the incoming MTABLE has a lot of entries, but not enough to close out the "batch".
  2. The incoming NTABLE, if it was collapsed to have only one puzzle per grid, and then stripped of puzzles for grids that appear in MTABLE ... would still be large enough so that combining it with MTABLE would cause writes past the end of XTABLE.
The issues that I saw with the ConsolidateList() code, were:
  • There is no check for XTABLE being full, before writing to it -- related to the above.
    (The only check, before writing, is for CTABLE).
  • endM should be set to 1 on entry, when MTABLE is empty, as (it seems) it is on the 1st call for a new batch.
  • Entries in MTABLE that are "larger" than every entry in NTABLE, are lost in the merge.
blue
 
Posts: 840
Joined: 11 March 2013

PreviousNext

Return to General