The New Sudoku Players' Forum

[Mathimagics]

Thanks blue!

That's more grist for the mill ...

[Mathimagics]

Todays LCT-20 status:

Code: Select all

 Puzzles: 4726407895 (  86.36%)
 Pending:  746322643 (  13.64%)
          ==========
   Grids: 5472730538


Just 746 million to go (Aug 15 still looks like the completion date) ... no sign yet of any new 668-pattern failures ...

[Mathimagics]

Todays LCT-20 status:

Code: Select all

 Puzzles: 4909685286 (  89.71%)
 Pending:  563045252 (  10.29%)
          ==========
   Grids: 5472730538


It looks like LCT-19 will be a much longer task! blue reports that BlueMagic19 will take 4 to 5 times longer per grid.

And now I have run a few 19C batch generations using the stage 1 worker process adapted for 19C. It looks like it will take about 3 times longer to produce batches of the same size than before. That would probably mean at least 3 times as long (6 months at least) for the full job.

I will try and find ways to speed up the batch generation ...

Cheers
MM

[Mathimagics]

Perhaps there are casual readers who could easily help out with spare unused computer time. !
Perhaps I could put up a software guide where some could email you zipped reliable batches of up to 1M puzzles at a time.. ?

That would certainly be useful, and would be much appreciated.

There is a better way, I think, than using the gsf tool. Any volunteer on a Windows system can be supplied with a Worker19 binary that will do the generation exactly as LCT-19 does it, and this would be fully automated, minimal intervention required. And, I think, this would produce more ED grid/puzzles than the gsf tool.

Unix users could be given source code to compile and run.

For file delivery, I think that a better idea would be for me to provide a cloud storage location. Batch files produced by the worker could be zipped and uploaded to that location. I'm also looking at a custom compression tool that would give smaller zip files.

Anybody who thinks they can spare a cpu core/thread (or two) over an extended period can PM me (please use subject "LCT-19").

Cheers
MM

PS: LCT-20 has passed the 5 billion grid mark:

Code: Select all

 Puzzles: 5029787238 (  91.91%)
 Pending:  442943300 (   8.09%)
          ==========
   Grids: 5472730538


[Mathimagics]

Re: gsf's Sudoku.exe tool:

Using coloin's 19C example (from here), I obtained 283,716 x 19C puzzles in 36 minutes. But this yielded only 36,480 ED grids. So the net yield with this method (for that example) is roughly 1000 grids/min.

Worker19C can produce 3 million ED grids/puzzles in about 200 minutes, which is a yield of roughly 15,000 grids/min.

So our volunteer workforce, should it materialise, should probably be using Worker19C.

Cheers
MM

[Mathimagics]

Hi coloin,

You (or anybody) can start right now, in fact!

Setup instructions:

• download Win64.zip (below), create a directory, eg LCT19, and unzip the files into that

• in that directory, run Gen19C. It will verify that the inboard solver works on your system, or produce an error message otherwise.

• if solver is ok, fire up 1 or more workers as follows:
  
  
  • open a new CMD window
  • go to a Worker directory (eg Worker1)
  • run ..\Gen19C (or, if you prefer, copy Gen19C.exe into each worker directory first)

That's it. I have provided 4 x Worker directories, but you can create extra ones should you get really enthusiastic. Typically a worker will produce a batch of 3 to 6 million puzzles every 3-6 hours. Each batch will be assigned a new batch id number, and the batch files will be deposited in the BatchFiles directory.

You can kill / restart the workers at any time.

The worker process starts by generating 10 x 19C puzzles using random grids and reducing them. This can take several minutes. These are used to seed the morphing logic - each pass generates all simple morphs from the previous pass, until a pass produces 3 million or more. This then becomes a batch, and we start over.

I will discuss batch delivery options in a separate post.

Cheers
MM

[EDIT] 21 Oct 2019: zip files deleted, now obsolete[/b].

[Mathimagics]

I have updated the zip files (Worker32.zip and Worker64.zip) above to include alternative binaries in the event that Gen19C fails the verification test, or crashes.

An alternative version Gen19J.exe is provided, which uses JCZsolver rather than fsss2, and so should run on any Windows platform. It is only marginally slower than Gen19C.

Cheers,
MM

[Mathimagics]

... are we not just generating the same random grids? (a bit like in the patterns game!).

I imagine that some (many) batches will overlap, but the goal is to scan the whole puzzle space (again a bit like the patterns game!).

The morph process concentrates on ED grids - so starting from a set of 19C puzzles on ED grids, it generates {-1,+1} morphs, but converts each one found to CF, and only those ones that are "new" are retained as seeds for the next pass.

This proved effective for LCT-20, with some overlap between the batches produced, but no more than 10%. For LCT-19 this should (hopefully) still be the case.

The chances of the puzzles being on new grids, when the batches are applied to the catalog, do naturally diminish over time, but you can be assured that for the first couple of months, at least, any batches that your workers produce will be worthwhile!

[Mathimagics]

My apologies!

I think that I have identified the probable cause and will provide an update, and also the source code, ASAP.

[Mathimagics]

Updated exes now available above, and here is the source package. It includes batch files for building Gen19C/J (64-bit), and all additional objects needed for linking.

[EDIT] 21 Oct 2019: zip file removed, now obsolete.

[Mathimagics]

It's odd that neither coloin nor me have seen this problem ...

If the problem persists, you might try reducing the batch size check at line 1189, eg change "3" to "2":

Code: Select all

      if (NMT > 0x3FFFFF) break;

[Mathimagics]

Thanks blue!

I had arrived at the same conclusion myself and have patched all the files. Sloppy work on my part ...

The patched source is available above, could you cast your eye over it? I have verified it with a small batch run (see line #17 in the source) and it works ok.

Cheers
MM

PS: LCT-20 BlueMagic processing will be complete in 24 hours ...

[Mathimagics]

Still no new grids without a 668 puzzle ?

No, not a sausage! Looks very much like you already have the complete list ... (and that there are only those 4 cases of "No20C" grids)

Then I'll work on an LCT-19 replacement for BlueMagic.

Excellent!

[Mathimagics]

1to9only: I think that I have changed the source since you last downloaded it.

There should be no spare bytes, as all table allocations are done in multiples (via the defines MAXN and TABSIZE) of 166 bytes. This is the standard table entry format:

• ED grid (81)
• a colon + a space (2)
• 19C puzzle (81)
• CR + LF (2)

Sort/merge comparisons are done only on the grid portion. blue pointed out that I was using the wrong comparison (my shame ) in the AddPuzzleM/N procs, which I have now corrected and posted above. These were the root cause of the crashes, I think.

Cheers,
MM

[EDIT] Not all batch-runs fill the larger table, but here is the log of one which I did overnight that does fill it (in pass #10), and lives to tell the tale:

Gen19C log: Show

Code: Select all

21:57:31: Generating initial seeds
  1: .23...7......89........1.........46...8...2..9.1.........64..2....32.6..8........
  2: 12....7.......9.3....2......8...4.9..74..............5....6.4..5..72......9...3..
  3: .2........5.7....3...1..5.....36..........27..8....4......9...8....2.9...31.4....
  4: ......7.9456.8......8......2.....4.....3...1....9...3......5....71.....6....34...
  5: ....5..........1...8.1....4.6...4..3...92..5..3.....2....6...785...1......2......
  6: ....56.....718.....9......4.......6......2.97..1...8......1....5..9.....7.6.....2
  7: .23...7...5............1.46.7.5.........4..6...8...3..3....4...6......9....3.7...
  8: 12............9..66...3.4.5..5.....4..1972......8.......4.........6...2..9.......
  9: .2....7..4....92..6......1......7...5.16...........9.........6..7..42......3.1.5.
 10: 1.3.5......7.8...6........4...6......7....8.39....5....6.9..5......3.9....4......
22:02:19: Added 10 new seeds
22:02:19: Processing new items - pass #1
22:02:19: Items in = 10, items out = 42
22:02:19: Sort/merge new items
22:02:19: Pass complete: ED grids/puzzles =        43
22:02:19: Processing new items - pass #2
22:02:19: Items in = 42, items out = 162
22:02:19: Sort/merge new items
22:02:19: Pass complete: ED grids/puzzles =       162
22:02:19: Processing new items - pass #3
22:02:22: Items in = 162, items out = 559
22:02:22: Sort/merge new items
22:02:22: Pass complete: ED grids/puzzles =       559
22:02:22: Processing new items - pass #4
22:02:29: Items in = 559, items out = 2282
22:02:29: Sort/merge new items
22:02:29: Pass complete: ED grids/puzzles =      2012
22:02:29: Processing new items - pass #5
22:02:53: Items in = 2012, items out = 11025
22:02:53: Sort/merge new items
22:02:53: Pass complete: ED grids/puzzles =      7815
22:02:53: Processing new items - pass #6
22:04:26: Items in = 7815, items out = 51085
22:04:26: Sort/merge new items
22:04:26: Pass complete: ED grids/puzzles =     32955
22:04:26: Processing new items - pass #7
22:11:03: Items in = 32955, items out = 242899
22:11:03: Sort/merge new items
22:11:03: Pass complete: ED grids/puzzles =    150805
22:11:03: Processing new items - pass #8
22:41:23: Items in = 150805, items out = 1189410
22:41:24: Sort/merge new items
22:41:24: Pass complete: ED grids/puzzles =    723467
22:41:24: Processing new items - pass #9
01:04:53: Items in = 723467, items out = 5978588
01:04:59: Sort/merge new items
01:05:01: Pass complete: ED grids/puzzles =   3572019
01:05:01: Processing new items - pass #10
06:08:12: Items in = 1544485, items out = 12936647
06:09:36: Sort/merge new items
06:09:42: Pass complete: ED grids/puzzles =   8388607
06:09:42: Creating batch file
06:09:45:  Waiting for Batch# lock
        1 file(s) moved.
06:09:45:  Saved as Batch000011.txt


[Mathimagics]

LCT-20 Final Report

Code: Select all

     17C:      46300
     18C:   10658721
     19C:  133785077
     20C: 5328240436
     21C:          4
          ----------
 Puzzles: 5472730538 ( 100.00%)


So, every ED grid (except for 4) has a 20C puzzle, the only exceptions being those 4 grids previously identified by blue and confirmed by dobrichev:

Code: Select all

123456789456789123789123456214897365365214897897365214541632978632978541978541632
123456789456789123789132465218967534564213978937548216391875642645321897872694351
123456789457189326689327154216534897745891632938672541361245978574918263892763415
123456789457189326689327154216534897745891632938672541392765418574918263861243975


blue's discovery of these grids was originally reported here: Grids containing a 21 but no 20

Also, blue's table of "668 failures" is complete, no new cases were discovered.

A pass of the catalog was made to check for redundant clues in the recorded puzzle, and this resulted in about 1/40 of 20C's being "promoted" to 19C, and also turned up a few million extra 18C's.

A big thank you to blue for the BlueMagic20 tool!

And so, onward with LCT-19 ...