Change the name of the "unicode" tokenizer to "unicode61" to emphasize that the case folding and separator-character identification routines are based on unicode version 6.1.

FossilOrigin-Name: 8f3e60aa2253f21bcee5d03982cfdd7f16c00060
2012-05-26 14:54:50 +00:00
parent 1c7016c9a5
commit ab322bd21e
5 changed files with 107 additions and 12 deletions
@@ -11,7 +11,7 @@
    );

  The built-in tokenizers (valid values to pass as <tokenizer name>) are
-  "simple" and "porter".
+  "simple", "porter" and "unicode".

  <tokenizer-args> should consist of zero or more white-space separated
  arguments to pass to the selected tokenizer implementation. The