To test the dfa pattern matcher, first build the program with
'configure --enable-dfa', then go to './patterns/minipat2/'
and type ./generate_owl_dfa.sh This script will compile
owl_attackpats.db and owl_defendpats.db into owl_attackpat.dfa
and owl_defendpat.dfa and copy them into ./regression/

When lauching gnugo in regression, it will load the owl dfa files and
say "found <name-of-file>.dfa". 'mkpat -d20' generates a
special binary file : .dfa file.

owl_defendpat.dfa and owl_attackpat.dfa must be in the
regression directory when you do your benchmark otherwise
gnugo do the old algorithm with check_pattern penality.

In the new_matchpat.c file, when no dfa file is found in the
execution directory, the old pattern matcher algorithm does
the work in a loop checking each pattern of the base. But with
a little penalty due to the fact that the function
check_pattern() is called at each iteration instead of being
directly written in the loop as in matchpat.c One possiblity
to suppress this penalty is to define a macro or an inline
function instead of a call to the function check_pattern().
When a dfa file is found for a given pattern database, gnugo
says "found pattern_dbase.dfa file" at startup and use the dfa
to "filter" patterns.

======================================================

Here we describe the basic principle of the 
dfa-pattern matcher:

1) Order
2) What is a DFA
3) Pattern matching with  DFAs
4) Size of the automaton
5) Cache consideration
6) The Joys of determinism


1)  Order
    -----

first we must choose an order to scan the board:

example
(i,j)-->(i+1,j)-->(i+1,j+1)-->(i,j+1)-->(i+2,j+0)-->(i+2,j+1)-->(i+2,j+2)...

Given a set of gnugo patterns we can choose an order to minimize
the size of the dfa (see later).

The order is given by two arrays of integers order_i[k] and order_j[k]
that gives the offset where to read the values on the board.

example
order_i[]={ 0, 1, 1, 0, 0, 2, 2, ...};
order_j[]={ 0, 0, 1, 1, 0, 1, 2, ...};

From the order we convert the patterns into a one dimention arrays
of integers.

example:

this patterns:

?X?
.O?
?OO

scanned in the order


149
238
567

(i,j)-->(i+1,j)-->(i+1,j+1)-->(i,j+1)-->(i+2,j+0)-->(i+2,j+1)-->(i+2,j+2)...


gives the string

"?.OX?OO??"


understood ? Ok, 
so let's forget the 2 dimentions patterns
for a time and focuse ourselve on the
1 dimention patterns.


2) What is a DFA
   -------------

A DFA is a Deterministic Finite state Automaton. See 
http://www.eti.pg.gda.pl/~jandac/thesis/node12.html
or Hopcroft & Ullman "Introduction to Language Theory" 
for more details.


A deterministic state is constituted by 

1) at most one transtion to another state by possible value 
(in go EMPTY, WHITE, BLACK, OUT_BOARD).
2) a list of Attributes (in Go the attributes are pattern indexes).

/* dfa state */

typedef struct state
{
  int next[4];
  attrib_t *att;
}
state_t;

A deterministic automaton is a finite set of deterministic states with two
special states:

1) the "start" state (here startState)
2) the "error" state (here zero)

/* dfa */

typedef struct dfa
{
  state_t *states;
  int maxStates;
  int lastState;
  int startState;
}
dfa_t;


3) Pattern matching with  DFAs
   ---------------------------

Recognizing with a dfa is very simple. That's why dfa are very fast in 
pattern matching. (See dfa_scan() in the engine/new_matchpat.c file)

Starting from the "start" state you only have to scan the entry string and
jump from states to states following the transtions and collect the attributes
on the way.

If we reach the "error" state (zero), it means that no more patterns will be
matched.

**********************************************************
Here is an example of scan:
**********************************************************

First we build a minimal dfa that recognize
"X..X" "X???" "X.OX" "X?oX"

Note that special letters "?ox" gives multiple out-transtions.

----------------------------------------------------
 state  |   .    |   O    |   X    |   #    |  att
----------------------------------------------------
      1 |      0 |      0 |      2 |      0 |
      2 |      3 |     10 |     10 |      0 |
      3 |      4 |      7 |      9 |      0 |
      4 |      5 |      5 |      6 |      0 |
      5 |      0 |      0 |      0 |      0 |    2
      6 |      0 |      0 |      0 |      0 |    4    2    1
      7 |      5 |      5 |      8 |      0 |
      8 |      0 |      0 |      0 |      0 |    4    2    3
      9 |      5 |      5 |      5 |      0 |
     10 |     11 |     11 |      9 |      0 |
     11 |      5 |      5 |     12 |      0 |
     12 |      0 |      0 |      0 |      0 |    4    2
----------------------------------------------------

Then we perform the scan of the string 
"X..XXO...." starting from state 1:

Actual state:     1, substring to scan : X..XXO....

We read an 'X' value, so from state 1 we must go to
state 2.

Actual state:     2, substring to scan : ..XXO....

We read a '.' value, so from state 2 we must go to
state 3 and so on ...

Actual state:     3, substring to scan : .XXO....
Actual state:     4, substring to scan : XXO....
Actual state:     6, substring to scan : XO....
Found pattern 4
Found pattern 2
Found pattern 1                 

After having reached the state number 6 where we mached the patterns
1,2 and 4, there is no out-transitions so we simply stop the matching.


4) Size of the automaton
   ---------------------

The first proposed version was bad, because the construction algorithm
did'nt reduce the automaton, the size of the automaton was exponential
in the number of wildcards '?', 'o' and 'x' in the pattern database.
I was near to think that dfa were not usable in real life :(
for example ".?????" was generating a 2^5 state dfa !!
This was really stupid!

This new version uses an "on the fly" algorithm to generate the 
minimal dfa's by synchronisation products using the acyclic property
of the graph. (see Hopcroft & Ullman "Introduction to Language Theory" for
more details).

It is still possible to construct a special pattern database that generates an
huge automaton.

For example a database constituted by the patterns
"X????","?X???","??X??" and "???X?"

needs 46 states for only 20 letters.

But in real life this doesn't occur as the probability to have a "bad" pattern
database is quasi null.  In this example if we add the pattern "O..O" no state
will be added.

5) Cache consideration
   -------------------

The dfa is constructed to minimize cache misses by 
making some assumptions about the frequencies of the values:
the EMPTY value is supposed to appear often on the board, so
the the '.' transition are almost always successors in memory.

The OUT_BOARD are supposed to be rare, so
'#' transitions will almost always imply a cache miss.

So "######" is hard to scan, but "......" is very nice to scan.


5) Optimal Order
   -------------

There is basicly two ways of optimization:

Given a set of patterns, we can compute the order of scan that minimize the
number of prefixes wildcards '?', 'o' or 'x'.

Example:
"XO????" is better than
"?????X" as it gives a smaller dfa.

We can also choose an order that "separates" better the patterns
for example
"XXO", "O.." and "OO." are better separated by the second position:
the value of the second position is different for each pattern so
we immediatly know which pattern we may have by reading this value.


6) The Joys of determinism
   -------------------------

As our automaton is deterministic, scanning the same string will reach the
same state every time ...  so what ?  If we store the path used during the
last scan in a stack, we only have to restart our scan from the position of the
last move.





