Non-english characters aren't displayed correctly

[gospodin.horoshiy]

Hi.

What I have is Ubuntu Server 8.04 with Zabbix 1.6 on it.

I also have lots or Windows Servers which are Russian version. So when I receive an Event log from such a server, event log looks like this:

Severity: High
Windows ?? ??????? ???????? ?????? ? ????? GPT.INI ??? ???????

I installed Ubuntu Server with no default Russian language support.

So after the installation I installed the following packs to add some support:
sudo apt-get install language-pack-ru
sudo apt-get install language-pack-ru-base
sudo apt-get install language-support-ru

It changed nothing.

Any idea what I need to install to be able to receive Russian characters from Windows Server RUS?


I'm not experienced in Linux systems actually, so I don't really know where to start digging, can someone help me out, please?

[Aly]

With DISPLAYING may be this one helps: http://www.zabbix.com/forum/showthread.php?t=10367
but, PROCESSING is little bit different thing..

[gospodin.horoshiy]

Aly, thanks for the reply. I will try this patch shortly.

Has anyone experienced simillar issue? I bet there is someone)

[gospodin.horoshiy]

First, I changed mysql database zabbix to UTF-8 charset. All tables.
Next, I added the following to the /etc/mysql/my.cnf:

[mysqld]
default-character-set=utf8
character-set-server=utf8
collation-server=utf8_general_ci
init-connect="SET NAMES utf8"
skip-character-set-client-handshake

[mysqldump]
default-character-set=utf8

Didn't work. Information in zabbix.history_log is full of ??????? question marks instead of cyrilic symbols. In which charset the information is sent by zabbix_agent? What i'm doing wrong? )

[Aly]

I think, nothing. Only frontend was made unicode ready. Server still have issues with processing string containing multibyte chars.

[gospodin.horoshiy]

Damn) Any way to fix this? )

[Aly]

In what encoding log is written?

[gospodin.horoshiy]

CP1251, I would reckon (Standard for all Russian versions of Windows OS)

P.S. Yes, I also recreated MySQL DB using CP1251 everywhere - still ???? instead of Russian letters

[Aly]

I would try to add two queries just after Db connection is made. Both on server side and in frontend:

SET CHARACTER SET "cp1251";
SET NAMES "cp1251";

This should keep db and server on same encoding. After that I would set cp1251 as default charset for front end in "include/locales/ru_ru.inc.php", but that may require to convert this file from utf8 to cp1251.

And even after that I can't guaranty that it will process cyrillic logs correctly.

[stalker]

encodings other than utf-8 very bad thing!

Я не вижу никаких проблем при показе русскоязычных event'ов во frontend'e. А логи это не так критично. UTF-8.

BTW: when frontend switched to imageTTFtext to properly display non-latin subscriptions on graphs?

[Aly]

I think when server and db will support unicode. Basically it's based on using proper fonts..

[gospodin.horoshiy]

Ok, I've got a solution, but not so sure how good it is though.

Here is what I've done:

In the Zabbix's source there is a zabbix-1.6\src\libs\zbxdb\db.c which contains if(ZBX_DB_OK == ret) DBexecute("SET CHARACTER SET utf8").
So this actually forces any data coming from Zabbix agents to be interpretated as UTF8 data. But I had CP1251!

So first, I removed this line from db.c and rebuilt Zabbix from the scratch.

Then I recreated a MySQL database using UTF-8 Charset.

I created new mysql user called zabbix. Gave him full permissions on zabbix database but no more access.(BTW, is this enough?)

Then I used root/password credentials for Zabbix frontend
and I used zabbix/password credentials for Zabbix_server

Then, I applied db_set_utf8.patch
(Thanks Aly)

I also put these lines in /etc/mysql/my.cnf:
[client]

default-character-set=utf8
[mysqld]
default-character-set=utf8
character-set-server=utf8
collation-server=utf8_general_ci
init-connect = "SET CHARACTER SET cp1251"




Here is the trick:
When client( in my case it's frontend or zabbix_server) connects to the database, what "SET CHARACTER SET cp1251" does is says to the DB that data which is going to come is in cp1251 charset. So MySQL will take then this cp1251-coded string , CONVERT it to the UTF-8 and put it to the DB. It also will return UTF-8-coded data in cp1251 back to the client just doing the same in reverse order. Good.

However, since root is the SUPERUSER, it ignores init-connect option according to the MySQL manual.

So cp1251 data coming from Windows will be converted to the UTF8 because init-connect will be triggered.
On the other hand, frontend will comunicate in UTF-8 to UTF-8 fashion as init-connect option will be ignored.

The result of all this is Frontend is in UTF-8. Database is UTF-8. Data coming from Windows is converted to UTF-8. All cyrillic symbols are visible in DB and Frontend. Sweet.

Still I have bunch of questions If I may:
1) I altered this db.c file. What problems I could have in future? For example while upgrading zabbix to newer versions?
2) Calling to all who successfully use cyrillic symbols to tell me easier way
3) I created this new DB user called 'zabbix' (used by zabbix_server) that has full permissions only on 'zabbix' DB. Is this enough?
4) Any other possible problems I can run into with this configuration?
5) To Zabbix developers: this line if(ZBX_DB_OK == ret) DBexecute("SET CHARACTER SET utf8"). why is it there in first place?
Thnx

[gospodin.horoshiy]

Now I have new problem, the same as Lufa here

http://www.zabbix.com/forum/showthread.php?t=10367&highlight=cp1252

Can't see cyrillic in e-mails(action notifications).

But, in my e-mail client I can see that E-Mail header/charset is UTF-8
MySQL is UTF-8
Frontend is standard: UTF-8

Using MySQL Query Browser I can see all other different UTF-8 cyrillic data just fine, but in zabbix.actions table, r_longdata column (Data type BLOB) - it's all messed up: I can see Latin chars, but no Cyrillic - I see them as squares.

So I think it's an issue how the data goes from Frontend Action creation php-form to the DB...

Server is 1.6, db_set_utf8.patch is applied.

Please help)

[Aly]

E-mails are sent by server, so by your logic charset for email should be cp1251

P.S. for retrieving data from db , I would use phpmyAdmin, as far as I know this tool better handles charsets problems.

[gospodin.horoshiy]

Aly , You are right! Just ve checked that(
What else zabbix_server does? except sending mails and talking to the agents?