Hi Denis, thanks for your questions and comments.
To explain my
modus operandi, let me first explain how it emerged.
I got interested in sudokus about ten years ago. I learned a lot from Andrew Stuart's website “sudokuwiki.org” and wrote my first Basic solver along his concept of basic strategies.
Later I discovered on internet an early version of a database of 'hard sudokus' : HardestDatabase110626, with golden Nugget, Kolk, Imam bayildi etc.
I studied relationships between individual puzzles with my equivalence analyser. I noticed how small differences between puzzles could increase the ratings.
At first I built a sudoku generator placing random numbers on random positions in an emtpy 9x9 table in a Word document respecting Sudoku rules. Valid puzzles were further analysed and in subsequent steps made harder with the techniques discovered as indicated above.
It took me a couple of years to come to the conclusion that this was not a very efficient technique.The maximum rating reached was SER 11.4.
In the meantime, I discovered this forum and Champagne's database ph_1710 and later ph_1910. I perfected my 'modifier' scripts by applying them on those databases. What struck me was that when I modified a puzzle in this way, the resulting puzzle, with a higher (or lower) rating, was practically always already present in the database. My conclusion was that the overwhelming majority of puzzles in those databases are derivatives from pre-existing puzzles. Nevertheless I found one new 11.8 that I submitted with my first post in this thread in April 2021.
Regarding your question about
algorithms: indeed, one type of them consists in adding clues (from the solution) and subsequently identifying minimals. This type of algorithm does not modify the solution, so all the resulting puzzles are in the same tree (expression used by mith). Another type of algorithm changes the solution (f.i by changing one or more clues) and the resulting puzzles thus belong to a different tree. This second type of algorithms can have a more profound effect, positive or negative, on the rating. It is also my experience that most of the interesting results come from a combination of both types of algorithms.
About
minimals: you have a point there. Indeed, one could limit the novelty to the expanded version only. But there is the following observation keeping in mind the preceding alinea: a combination of the two types of algorithms leads sometimes to a remarkable result (f.i. an 11.9 in my previous batch last September), but it does so only for a single one of the minimals. This means that in my
modus operandi, I would have to test all minimals anyway in combination with the other type of algorithm (modifying the solution). The question remains therefore: should all the relevant minimals be published or not? If not, everyone wanting to 'build' on the expanded version should identify separately the relevant minimals. Sudokus are forever and perhaps in the future, a clever young developer may with the right modifications on the right minimal open the door to a new class of puzzles, that is if AI will not do it first
Regarding your question on
filters: I first use stage 2 (of 4) in my Logical solver, which I suppose is akin to T&E(1). It is fast and, if it doesn't solve the puzzle, skfr is used as a second filter before proceeding to SER, the ultimate filter. I put no limits on the number of clues or candidates.
Finally about your consideration
in fine: what I also have learned about hard sudokus is that the large majority of them are not generated from scratch, but are derived from precursors, applying suitable modifications. I have observed that the potential for deriving hard puzzles from puzzles in previous databases, of which ph_2010 was the most recent, was practically exhausted, at least with the 'modifiers' I use. When last year new puzzles became available, there was a new potential for deriving other hard sudokus with my optimized modifiers. If the original puzzle happens to be in T&E(3) then there is a reasonable chance that some derivatives keep that characteristic, which explains why a (not so large) substantial part of the puzzles in my latest results remain in T&E(3). Mith has opened a rich source for further derivations in the years to come. We may all be grateful to him.