dan 754d3adf7c Have the FTS unicode61 strip out diacritics when tokenizing text. This can be disabled by specifying the tokenizer option "remove_diacritics=0". ...

FossilOrigin-Name: 790f76a5898dad1a955d40edddf11f7b0fec0ccd

2012-06-06 19:30:38 +00:00

..

CaseFolding.txt

Add an experimental tokenizer to fts4 - "unicode". This tokenizer works in the same way except that it understands unicode "simple case folding" and recognizes all characters not classified as "Letters" or "Numbers" by unicode as token separators.

2012-05-25 17:50:19 +00:00

mkunicode.tcl

Have the FTS unicode61 strip out diacritics when tokenizing text. This can be disabled by specifying the tokenizer option "remove_diacritics=0".

2012-06-06 19:30:38 +00:00

UnicodeData.txt

Add an experimental tokenizer to fts4 - "unicode". This tokenizer works in the same way except that it understands unicode "simple case folding" and recognizes all characters not classified as "Letters" or "Numbers" by unicode as token separators.

2012-05-25 17:50:19 +00:00