Zend_Search_Lucene and umlauts

TobiasPl · #1 (**permalink**) 04-03-2008, 03:50 PM

Hi!
I have a strange problem:

I am trying to index a website that is in UTF-8 format.
Like this:

PHP Code:


			
Zend_Search_Lucene_Analysis_Analyzer::setDefault(

               new Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8()

        );

        $searchindex = Zend_Search_Lucene::create($dir); // $dir is set correctly

        foreach($urls as $url) // $urls holds a list of URLs

        {

            $doc = Zend_Search_Lucene_Document_Html::loadHTMLFile($url['url']);

            $doc->addField(Zend_Search_Lucene_Field::Text('url', $url['url']));

            $searchindex->addDocument($doc);

        }

No error, but trying to search for a word with an umlaut always failed.
I have checked the index using this function:

PHP Code:


			
header('Content-Type: text/html; charset=UTF-8');

/* ... */

Zend_Search_Lucene_Analysis_Analyzer::setDefault(

               new Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8()

        );

        Zend_Search_Lucene_Search_QueryParser::setDefaultEncoding('utf-8');



        $searchindex = Zend_Search_Lucene::open($dir);



        foreach ($searchindex->terms() as $term) {

            echo ($term->key())."<br />";

        }

The result:
...
body�Accounts
body�Achtung
body�AdministrationsoberflÃ
body�Administratoren
body�Adressbestand
body�AdressbestÃ
body�Adressdaten
...

Well...
The word "AdministrationsoberflÃ" actually is "Administrationsoberfläche", and "AdressbestÃ" actually is "Adressbestände". In every word with umlauts, the word is cut after the fist umlaut which, additionally, is unreadable.
What am I doing wrong?
Any ideas?
Thank you very much.
Tobias

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode