Thankyou !
Well it seems we will know eventually if there are any more 21 but no 20 grids.
And excellent computation with the double bands .....i knew it was big
I remember a contributor computing the 5e9 grids row by row, but i have failed to locate where it was in the forum - so we could confirm your calculation if it could be located
And its such a shame [ for me ] that the MC grid does have double band with 12 clues .....I will have to edit accordingly !
If there are any double bands which need 13 clues to complete they would surely be in the known grids with bands needing 6 clues.... and we know the status of these grids anyway.
Low/Hi Clue Thresholds
Re: Low/Hi Clue Thresholds
by coloin » Tue Jul 23, 2019 4:12 pm
- coloin
- Posts: 2494
- Joined: 05 May 2005
- Location: Devon
LCT-20 Progress Report
by Mathimagics » Thu Jul 25, 2019 6:35 am
A minor milestone has been passed, the number of grids resolvedis now 3,880,139,870, which is a shade under 71%.
And the Stage 1 workers are still producing net yields that make me reluctant to kill them off and switch to Stage 2 (explicit grid testing with BlueMagic).
Productivity measures are quite rubbery here at the atomic level - the net grid rate can only be estimated, for various reasons. Administration is also complicated, again for various reasons. These vagaries will all go away when we switch to Stage 2 mode. We will have much more straight-forward DB admin and accurate timings available. But the overall progress is absolute, and indisputable ...
Despite dwindling net yields (only 20% of grids produced are new ones), these hard-working souls have delivered 460 million new grid resolutions since July 20, 5 days ago. That's the equivalent of 18 million per day per worker. This looks to have declined to 130 million for the past 2 days, but I can't kill off these guys just yet, not with those figures. And so, the execution is delayed for another 24/48 hours!
Will they survive long enough to see us safely though to the 4 billion mark? The entire planet holds its collective breath ...
And the Stage 1 workers are still producing net yields that make me reluctant to kill them off and switch to Stage 2 (explicit grid testing with BlueMagic).
Productivity measures are quite rubbery here at the atomic level - the net grid rate can only be estimated, for various reasons. Administration is also complicated, again for various reasons. These vagaries will all go away when we switch to Stage 2 mode. We will have much more straight-forward DB admin and accurate timings available. But the overall progress is absolute, and indisputable ...
Despite dwindling net yields (only 20% of grids produced are new ones), these hard-working souls have delivered 460 million new grid resolutions since July 20, 5 days ago. That's the equivalent of 18 million per day per worker. This looks to have declined to 130 million for the past 2 days, but I can't kill off these guys just yet, not with those figures. And so, the execution is delayed for another 24/48 hours!
Will they survive long enough to see us safely though to the 4 billion mark? The entire planet holds its collective breath ...
-
Mathimagics - 2017 Supporter
- Posts: 1926
- Joined: 27 May 2015
- Location: Canberra
LCT-20 Progress
by Mathimagics » Sat Jul 27, 2019 9:28 pm
We now have 4,065,435,695 ( 74.3%) of grids resolved.
blue has kindly provided a library interface to the BlueMagic function which will greatly simplify the system admin side of the crossover to explicit grid testing (LCT-20 Stage 2). The old (Stage 1) worker processes will be replaced by new ones over the next couple of days.
Cheers
MM
blue has kindly provided a library interface to the BlueMagic function which will greatly simplify the system admin side of the crossover to explicit grid testing (LCT-20 Stage 2). The old (Stage 1) worker processes will be replaced by new ones over the next couple of days.
Cheers
MM
-
Mathimagics - 2017 Supporter
- Posts: 1926
- Joined: 27 May 2015
- Location: Canberra
LCT-20 End stage 1
by Mathimagics » Sun Jul 28, 2019 4:26 pm
Ok, it's official! LCT-20 Stage 1 is over, with 75% of grids resolved: 4,107,224,495
The work force have been retired, given 3 weeks leave (unpaid), and will return for the LCT-19 job. They have resolved an impressive total of 800 million grids per week since the commencement (around 22 June).
Stage 2 has now commenced, with 5 workers using BlueMagic to find puzzles for the 1,365,506,043 unresolved grids.
Each should process 14-16 million grids a day, so that's at most 100 x worker-days to complete, with an eta of around 15 August looking very likely.
It is interesting that some of the bigger bands have quite low resolution rates, perhaps 50-55%, while most other bands have 80%+. Not sure if there's anything going on there ...
A full breakdown by band is included below (ng = grids in band, nu = grids unresolved). Note that Band files 250 - 300 are all multiband files (eg Band300 has grids with band index = 300 to 416).
Cheers
MM
The work force have been retired, given 3 weeks leave (unpaid), and will return for the LCT-19 job. They have resolved an impressive total of 800 million grids per week since the commencement (around 22 June).
Stage 2 has now commenced, with 5 workers using BlueMagic to find puzzles for the 1,365,506,043 unresolved grids.
Each should process 14-16 million grids a day, so that's at most 100 x worker-days to complete, with an eta of around 15 August looking very likely.
It is interesting that some of the bigger bands have quite low resolution rates, perhaps 50-55%, while most other bands have 80%+. Not sure if there's anything going on there ...
A full breakdown by band is included below (ng = grids in band, nu = grids unresolved). Note that Band files 250 - 300 are all multiband files (eg Band300 has grids with band index = 300 to 416).
Cheers
MM
- Code: Select all
Totals:
17C: 46300
18C: 10221135
19C: 5626788
20C: 4091330268
21c: 4
----------
Puzzles: 4107224495 ( 75.05%)
unk: 1365506043 ( 24.95%)
----------
Grids: 5472730538
Band status: Show
-
Mathimagics - 2017 Supporter
- Posts: 1926
- Joined: 27 May 2015
- Location: Canberra
LCT-20 Progress
by Mathimagics » Wed Jul 31, 2019 4:21 pm
72 hours gone, and stage 2 has found 267 million 20C puzzles (all with 668-pattern). With 5 workers, this is 17.8 million grids per worker/day. Hats off to blue for that! That's roughly 1/5 of the 1,365,506,043 candidates, so the completion date might be 13 August.
Interesting to note that NO new cases of "no 668 20C puzzle" have been found. Might blue's table above in fact be complete? We'll just have to wait and see ...
Interesting to note that NO new cases of "no 668 20C puzzle" have been found. Might blue's table above in fact be complete? We'll just have to wait and see ...
-
Mathimagics - 2017 Supporter
- Posts: 1926
- Joined: 27 May 2015
- Location: Canberra
Re: Low/Hi Clue Thresholds
by coloin » Wed Jul 31, 2019 5:34 pm
Great ..
.. will blue’s magic program be slower or quicker at finding 667-19Cs .... just wondering !
.. will blue’s magic program be slower or quicker at finding 667-19Cs .... just wondering !
- coloin
- Posts: 2494
- Joined: 05 May 2005
- Location: Devon
Re: Low/Hi Clue Thresholds
by blue » Thu Aug 01, 2019 9:07 pm
Hi Colin,
It's slower by a factor of 27 -- about 129 milliseconds per grid, for random grids, on my machine.
For finding a 19C for a grid, it's faster to look first, for puzzles with >= 8 clues a band/stack.
The best order I've hit on so far, takes ~22.6 milliseconds per grid, for random grids.
On my machine, that's ~3,800,000 grids per day per core, and 1432 core*days to do the whole catalog.
Mathimagics' puzzle morphing code, will probably do a lot better than that, for a long time.
coloin wrote:will blue’s magic program be slower or quicker at finding 667-19Cs .... just wondering !
It's slower by a factor of 27 -- about 129 milliseconds per grid, for random grids, on my machine.
For finding a 19C for a grid, it's faster to look first, for puzzles with >= 8 clues a band/stack.
The best order I've hit on so far, takes ~22.6 milliseconds per grid, for random grids.
On my machine, that's ~3,800,000 grids per day per core, and 1432 core*days to do the whole catalog.
Mathimagics' puzzle morphing code, will probably do a lot better than that, for a long time.
- blue
- Posts: 1045
- Joined: 11 March 2013
Re: Low/Hi Clue Thresholds
by Mathimagics » Fri Aug 02, 2019 4:47 am
blue wrote:For finding a 19C for a grid, it's faster to look first, for puzzles with >= 8 clues a band/stack.
Thanks blue for those timings. You anticipated my questions just as I was contemplating them ...
Guesstimating times for LCT-19 is more complicated than it was LCT-20.
Let's say BlueMagic has these 2 functions, Find19C-8 for finding 19C puzzles with 8 or more clues per band/stack, and Find19C-7 for finding {667} puzzles.
So, with X = price of a Find20C call, the price of 19C testing is probably going to be Find19C-8 ~= 5X, and Find19C-7 ~= 27X.
Reducing the pool size (grids that need to be tested with Find19C) will be crucial. So, the morphing workers will clearly have to do most of the heavy lifting. How they will actually perform in the 19C context is harder to predict, but I don't expect major degradation there - I do need to run some preliminary tests, however.
Cheers
MM
PS: blue, can you do some sampling and give estimates of the probability that a grid will (a) fail the 19C-8 test, (b) fail both tests?
PPS: LCT-20 is motoring through the unresolved grids, only 972,353,033 to go ...
-
Mathimagics - 2017 Supporter
- Posts: 1926
- Joined: 27 May 2015
- Location: Canberra
Re: Low/Hi Clue Thresholds
by blue » Fri Aug 02, 2019 6:07 am
Mathimagics wrote:PPS: LCT-20 is motoring through the unresolved grids, only 972,353,033 to go ...
Nice !
Mathimagics wrote:PS: blue, can you do some sampling and give estimates of the probability that a grid will (a) fail the 19C-8 test, (b) fail both tests?
I have these results from my earlier test of 10 million "no NTA" grids:
- Code: Select all
3198/10,000,000 -- failed the "19-8c" test (limited to only one band having >= 8 clues)
505/10,000,000 -- failed the "19-8c" and 667 tests
461/10,000,000 -- failed the "19-8c", 667 and 577+757 tests
461/10,000,000 -- failed those and 289/388 tests.
The "19-8c" testing took almost all of the time ... ~25.6 ms per grid.
The change to being it down to 22.6 ms, was to look for 559 puzzles (for up to 6 bands/stacks) first,
and skip that combination when the other "19-8c" tests run.
In a shorter run for the "559 first" method (earlier today):
- Code: Select all
71905/1,000,000 -- failed the "559" test
321/1,000,000 -- failed the rest of the "19-8c" test
50/1,000,000 -- failed the "19-8c" and 667 tests
48/1,000,000 -- failed the "19-8c", 667 and 577+757 tests
48/1,000,000 -- failed those and 289/388 tests.
- blue
- Posts: 1045
- Joined: 11 March 2013
Re: Low/Hi Clue Thresholds
by Mathimagics » Fri Aug 02, 2019 7:29 am
Thanks blue!
That's more grist for the mill ...
That's more grist for the mill ...
-
Mathimagics - 2017 Supporter
- Posts: 1926
- Joined: 27 May 2015
- Location: Canberra
Re: Low/Hi Clue Thresholds
by Mathimagics » Sun Aug 04, 2019 6:07 pm
Todays LCT-20 status:
Just 746 million to go (Aug 15 still looks like the completion date) ... no sign yet of any new 668-pattern failures ...
- Code: Select all
Puzzles: 4726407895 ( 86.36%)
Pending: 746322643 ( 13.64%)
==========
Grids: 5472730538
Just 746 million to go (Aug 15 still looks like the completion date) ... no sign yet of any new 668-pattern failures ...
-
Mathimagics - 2017 Supporter
- Posts: 1926
- Joined: 27 May 2015
- Location: Canberra
Re: Low/Hi Clue Thresholds
by Mathimagics » Tue Aug 06, 2019 8:14 pm
Todays LCT-20 status:
It looks like LCT-19 will be a much longer task! blue reports that BlueMagic19 will take 4 to 5 times longer per grid.
And now I have run a few 19C batch generations using the stage 1 worker process adapted for 19C. It looks like it will take about 3 times longer to produce batches of the same size than before. That would probably mean at least 3 times as long (6 months at least) for the full job.
I will try and find ways to speed up the batch generation ...
Cheers
MM
- Code: Select all
Puzzles: 4909685286 ( 89.71%)
Pending: 563045252 ( 10.29%)
==========
Grids: 5472730538
It looks like LCT-19 will be a much longer task! blue reports that BlueMagic19 will take 4 to 5 times longer per grid.
And now I have run a few 19C batch generations using the stage 1 worker process adapted for 19C. It looks like it will take about 3 times longer to produce batches of the same size than before. That would probably mean at least 3 times as long (6 months at least) for the full job.
I will try and find ways to speed up the batch generation ...
Cheers
MM
-
Mathimagics - 2017 Supporter
- Posts: 1926
- Joined: 27 May 2015
- Location: Canberra
Re: Low/Hi Clue Thresholds
by coloin » Wed Aug 07, 2019 9:52 am
Such is the scale of the task ....
Perhaps there are casual readers who could easily help out with spare unused computer time. !
Perhaps I could put up a software guide where some could email you zipped reliable batches of up to 1M puzzles at a time.. ?
I hope you dont mind ive done it here - to see if there is any interest !!
However the 18C as we know is the stumbling block.
I’ve been looking at /generating the remote ( untouchable) puzzles which are isolated (-1+1) or existing in small pockets of isolated groups ... as of course there is
The big group of 1 billion 18C puzzles which are connected (-1+1) are there to be found easily - but not easy to collate!
Perhaps there are casual readers who could easily help out with spare unused computer time. !
Perhaps I could put up a software guide where some could email you zipped reliable batches of up to 1M puzzles at a time.. ?
I hope you dont mind ive done it here - to see if there is any interest !!
However the 18C as we know is the stumbling block.
I’ve been looking at /generating the remote ( untouchable) puzzles which are isolated (-1+1) or existing in small pockets of isolated groups ... as of course there is
The big group of 1 billion 18C puzzles which are connected (-1+1) are there to be found easily - but not easy to collate!
- coloin
- Posts: 2494
- Joined: 05 May 2005
- Location: Devon
Re: Low/Hi Clue Thresholds
by Mathimagics » Thu Aug 08, 2019 5:41 am
coloin wrote:Perhaps there are casual readers who could easily help out with spare unused computer time. !
Perhaps I could put up a software guide where some could email you zipped reliable batches of up to 1M puzzles at a time.. ?
That would certainly be useful, and would be much appreciated.
There is a better way, I think, than using the gsf tool. Any volunteer on a Windows system can be supplied with a Worker19 binary that will do the generation exactly as LCT-19 does it, and this would be fully automated, minimal intervention required. And, I think, this would produce more ED grid/puzzles than the gsf tool.
Unix users could be given source code to compile and run.
For file delivery, I think that a better idea would be for me to provide a cloud storage location. Batch files produced by the worker could be zipped and uploaded to that location. I'm also looking at a custom compression tool that would give smaller zip files.
Anybody who thinks they can spare a cpu core/thread (or two) over an extended period can PM me (please use subject "LCT-19").
Cheers
MM
PS: LCT-20 has passed the 5 billion grid mark:
- Code: Select all
Puzzles: 5029787238 ( 91.91%)
Pending: 442943300 ( 8.09%)
==========
Grids: 5472730538
-
Mathimagics - 2017 Supporter
- Posts: 1926
- Joined: 27 May 2015
- Location: Canberra
Re: Low/Hi Clue Thresholds
by Mathimagics » Thu Aug 08, 2019 8:02 am
Re: gsf's Sudoku.exe tool:
Using coloin's 19C example (from here), I obtained 283,716 x 19C puzzles in 36 minutes. But this yielded only 36,480 ED grids. So the net yield with this method (for that example) is roughly 1000 grids/min.
Worker19C can produce 3 million ED grids/puzzles in about 200 minutes, which is a yield of roughly 15,000 grids/min.
So our volunteer workforce, should it materialise, should probably be using Worker19C.
Cheers
MM
Using coloin's 19C example (from here), I obtained 283,716 x 19C puzzles in 36 minutes. But this yielded only 36,480 ED grids. So the net yield with this method (for that example) is roughly 1000 grids/min.
Worker19C can produce 3 million ED grids/puzzles in about 200 minutes, which is a yield of roughly 15,000 grids/min.
So our volunteer workforce, should it materialise, should probably be using Worker19C.
Cheers
MM
-
Mathimagics - 2017 Supporter
- Posts: 1926
- Joined: 27 May 2015
- Location: Canberra
