The New Sudoku Players' Forum

[coloin]

Yes i tend to agree with you
looking at the batch run a single 19C puzzle was generated x7 to 750,000 puzzles in 4 hours, but a breakdown of solution grids was only 250,000. So indeed your program is 15x quicker !!!
When the LCT-20 completes i will use your program ...

[Mathimagics]

Hi coloin,

You (or anybody) can start right now, in fact!

Setup instructions:

• download Win64.zip (below), create a directory, eg LCT19, and unzip the files into that

• in that directory, run Gen19C. It will verify that the inboard solver works on your system, or produce an error message otherwise.

• if solver is ok, fire up 1 or more workers as follows:
  
  
  • open a new CMD window
  • go to a Worker directory (eg Worker1)
  • run ..\Gen19C (or, if you prefer, copy Gen19C.exe into each worker directory first)

That's it. I have provided 4 x Worker directories, but you can create extra ones should you get really enthusiastic. Typically a worker will produce a batch of 3 to 6 million puzzles every 3-6 hours. Each batch will be assigned a new batch id number, and the batch files will be deposited in the BatchFiles directory.

You can kill / restart the workers at any time.

The worker process starts by generating 10 x 19C puzzles using random grids and reducing them. This can take several minutes. These are used to seed the morphing logic - each pass generates all simple morphs from the previous pass, until a pass produces 3 million or more. This then becomes a batch, and we start over.

I will discuss batch delivery options in a separate post.

Cheers
MM

[EDIT] 21 Oct 2019: zip files deleted, now obsolete[/b].

[coloin]

my oldish but not that old spare dual computor ...is 32 bit i will use it !!!

but my 4 core -64 reported no issues !!!

Code: Select all

Solver verified, ready for work!

....great !

well done in the cricket !

[Mathimagics]

I have updated the zip files (Worker32.zip and Worker64.zip) above to include alternative binaries in the event that Gen19C fails the verification test, or crashes.

An alternative version Gen19J.exe is provided, which uses JCZsolver rather than fsss2, and so should run on any Windows platform. It is only marginally slower than Gen19C.

Cheers,
MM

[coloin]

Excellent that did the trick.
Now up and running with the Gen19j.exe - 2 workers more.

[1to9only]

I probably have a little bit of spare processing capacity in between patern games! Might download and try later!

Out of curiousity, and after a very long run, are we not just generating the same random grids? (a bit like in the patterns game!). I imagine that some (many) batches will overlap, but the goal is to scan the whole puzzle space (again a bit like the patterns game!).

[Mathimagics]

... are we not just generating the same random grids? (a bit like in the patterns game!).

I imagine that some (many) batches will overlap, but the goal is to scan the whole puzzle space (again a bit like the patterns game!).

The morph process concentrates on ED grids - so starting from a set of 19C puzzles on ED grids, it generates {-1,+1} morphs, but converts each one found to CF, and only those ones that are "new" are retained as seeds for the next pass.

This proved effective for LCT-20, with some overlap between the batches produced, but no more than 10%. For LCT-19 this should (hopefully) still be the case.

The chances of the puzzles being on new grids, when the batches are applied to the catalog, do naturally diminish over time, but you can be assured that for the first couple of months, at least, any batches that your workers produce will be worthwhile!

[1to9only]

Can I obtain the source code for Gen19C.exe? To help debug 2 Worker deaths so far, each after some 9 hours of running!

Code: Select all

18:17:55: Generating initial seeds
...
19:06:53: Processing new items - pass #10
21:45:36: Items in = 717721, items out = 5896101
21:45:41: Sort/merge new items
21:45:42: Pass complete: ED grids/puzzles =   3411795
21:45:42: Processing new items - pass #11
03:22:56: Items in = 1556406, items out = 12936647
03:23:10: Sort/merge new items


Edit: [Windows exception] image deleted.

[1to9only]

I'm also looking at a custom compression tool that would give smaller zip files.

7zip produces smaller files than zip.

Batch001002.txt 654,407,275 (639MB)
Batch001002.zip 141,311,440 (138MB) <- using Zip 3.0 (July 5th 2008) from http://infozip.sourceforge.net/
Batch001002.7z 103,461,289 (101MB) <- using 7-Zip 19.00 (2019-02-21) from http://7-zip.org/
.

[Mathimagics]

My apologies!

I think that I have identified the probable cause and will provide an update, and also the source code, ASAP.

[Mathimagics]

Updated exes now available above, and here is the source package. It includes batch files for building Gen19C/J (64-bit), and all additional objects needed for linking.

[EDIT] 21 Oct 2019: zip file removed, now obsolete.

[1to9only]

I've had 3 Worker crashes in the 1st 24 hrs! I'm running the latest Gen19C.exe now. I had a quick look at the source code, could not spot anything wrong in ConsolidateList() where I think the crash occurred, likely fixed already. I'll run for another 24 hrs, and hopefully no more mishaps...

[Mathimagics]

It's odd that neither coloin nor me have seen this problem ...

If the problem persists, you might try reducing the batch size check at line 1189, eg change "3" to "2":

Code: Select all

      if (NMT > 0x3FFFFF) break;

[1to9only]

This time ALL 4 Workers crashed. Processing is stopped. I'll do some investigations when time permits.

Worker 1: The instruction at 0x0000000000403154 referenced memory at 0x00000001A6012000. The memory could not be written.
Worker 2: The instruction at 0x0000000000403154 referenced memory at 0x00000001A601D000. The memory could not be written.
Worker 3: The instruction at 0x00000000004034A8 referenced memory at 0x00000001A6016000. The memory could not be written.
Worker 4: The instruction at 0x0000000000403154 referenced memory at 0x00000001A601F000. The memory could not be written.

Worker 3 stopped at a different code address.

I've had similar (occasional) failures (The memory could not be read.) when one of my programs usually ran 99.99% percent of the time, on this particular machine, and on other machines as well!
.

[blue]

II had a quick look at the source code, could not spot anything wrong in ConsolidateList() where I think the crash occurred, likely fixed already.

It's odd that neither coloin nor me have seen this problem ...

If the problem persists, you might try reducing the batch size check at line 1189, eg change "3" to "2":

Code: Select all

      if (NMT > 0x3FFFFF) break;

It's amazing, that neither of you has seen a crash.
There's a definite problem in ConsolidateList().

Code: Select all

   void AddPuzzleN() {
      memcpy(XT, NT, 166); XT += 166;
      nx++;
      if (nx >= MAXN) {printf("[ERROR] merge table\n"); exit(0);}
      if (nc >= MAXC) return;
      memcpy(NC, NT, 166); NC += 166; nc++; 
      }

   void AddPuzzleM() {
      memcpy(XT, MT, 166);   XT += 166;
      nx++;
      if (nx >= MAXN) {printf("[ERROR] merge table\n"); exit(0);}
      }


A good start, would be to move the "[ERROR]" check lines to in front of the 'memcpy' lines that are (no doubt) causing the crashes.
Also MAXN should be changed to either TABSIZE or MAXC (for versions where MAXN is 12936648 and TABSIZE is only 0x7FFFFF).

The crash can happen in the following circumstance:

1. the incoming MTABLE has a lot of entries, but not enough to close out the "batch".
2. The incoming NTABLE, if it was collapsed to have only one puzzle per grid, and then stripped of puzzles for grids that appear in MTABLE ... would still be large enough so that combining it with MTABLE would cause writes past the end of XTABLE.

The issues that I saw with the ConsolidateList() code, were:

• There is no check for XTABLE being full, before writing to it -- related to the above.
  (The only check, before writing, is for CTABLE).
• endM should be set to 1 on entry, when MTABLE is empty, as (it seems) it is on the 1st call for a new batch.
• Entries in MTABLE that are "larger" than every entry in NTABLE, are lost in the merge.