lucene does not search cyrillic text

demi · #1 (**permalink**) 03-24-2008, 05:07 PM

Here is program in which I create index with some example data:

PHP Code:


			
        $itemId = 3245;

        $title = 'деякий екземпловий текст'; // here is some cyrillic text

        

        setlocale(LC_CTYPE, 'uk_UA.UTF-8');

        

        Zend_Search_Lucene_Analysis_Analyzer::setDefault(

            new Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8());

        

        Zend_Loader::loadClass('Zend_Search_Lucene');

        

        $index = Zend_Search_Lucene::create('tmp/index');

        $doc = new Zend_Search_Lucene_Document();

        $doc->addField(Zend_Search_Lucene_Field::UnIndexed('itemId', $itemId));

        $doc->addField(Zend_Search_Lucene_Field::Text('url', $url));

        $doc->addField(Zend_Search_Lucene_Field::Keyword('title', $title));

        $doc->addField(Zend_Search_Lucene_Field::UnStored('contents', $contentText)); // $contentText is variable comes with cyrillic text

        $index->addDocument($doc);

After executing this code, try to find indexed text:

PHP Code:


			
        $index = Zend_Search_Lucene::open('tmp/index');

        $hits = $index->find($this->getRequest()->query); // contains cyrillic word which is also contains in indexed text

        foreach($this->items as $item)

        {

            echo $item->title;

            echo '<br />';

            echo $item->url;

        }

But $index->find(), returns empty result! Please, help me understand what wrong? Program works fine if operates with text contains latin symbols. Why here is important latin or cyrillic symbols I use?

Thank you in advance!

hedonism · #2 (**permalink**) 04-16-2008, 02:02 PM

Set lucene fields to utf-8 encoding like this:

PHP Code:


			
$doc->addField(Zend_Search_Lucene_Field::Keyword('title', $title, 'utf-8'));

Add this lines before searching indexes:

PHP Code:


			
Zend_Search_Lucene_Analysis_Analyzer::setDefault(
    new Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8());

And finally do this:

PHP Code:


			
$query = Zend_Search_Lucene_Search_QueryParser::parse($this->getRequest()->query,'utf-8');
$hits = $index->find($query);

With me and Bulgarian characters works fine

Cheers!

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode