Ad Widget

Collapse

Zabbix history syncer processes more than 75% busy

Collapse
X
Collapse
 
  • Page of 2
  • Time
  • Show
Clear All
new posts
Previous 1 2 template Next
  • MaksimsE
    Junior Member
    • Nov 2017
    • 23
    #1

    Zabbix history syncer processes more than 75% busy

    Всем привет,

    Вопрос простой. Как правильно подобрать значения CacheSize и HistoryCacheSize чтобы триггер history syncer не срабатывал? Если не только эти параметры отвечают за этот триггер, то что еще порекомендуете смотреть?
    Сейчас я вижу, что данные на графике отображаются с задержкой более 10-и минут.

    Конфиг и состояние Zabbix Server:

    1115.05 NVPS

    Code:
    LogFile=/var/log/zabbix/zabbix_server.log
LogFileSize=100
DebugLevel=3
PidFile=/var/run/zabbix/zabbix_server.pid
DBName=zabbix
DBUser=zabbix
StartPollers=750
StartPreprocessors=16
StartPollersUnreachable=150
StartTrappers=1
StartPingers=20
StartDiscoverers=250
StartTimers=3
SNMPTrapperFile=/var/log/snmptrap/snmptrap.log
HousekeepingFrequency=0
MaxHousekeeperDelete=1000
CacheSize=4096M
CacheUpdateFrequency=360
StartDBSyncers=2
HistoryCacheSize=1024M
HistoryIndexCacheSize=1024M
TrendCacheSize=1024M
ValueCacheSize=4096M
Timeout=30
UnreachablePeriod=360
UnavailableDelay=600
UnreachableDelay=120
AlertScriptsPath=/usr/lib/zabbix/alertscripts
ExternalScripts=/usr/lib/zabbix/externalscripts
LogSlowQueries=3000
    Code:
    #               total        used        free      shared  buff/cache   available
Mem:       65696748    42042644    18209472     3704492     5444632    19174456
Swap:       5915644           0     5915644 #
    Конфиг mariadb /etc/my.cnf.d/server.cnf:

    Code:
    innodb_flush_log_at_trx_commit=0
skip-name-resolve
datadir=/db/mysql
slow_query_log = 1
long_query_time = 10
log_output = FILE
connect_timeout = 60
wait_timeout = 28800
max_connections = 20000
max_allowed_packet = 512M
max_connect_errors = 2000
tmp_table_size = 512M
max_heap_table_size = 256M
table_cache = 1024
default_storage_engine=InnoDB
innodb_file_per_table = 1
innodb_status_file = 1
innodb_additional_mem_pool_size = 128M
innodb_buffer_pool_instances = 35
innodb_buffer_pool_size = 20G 
innodb_flush_method = O_DIRECT
innodb_io_capacity = 2000
innodb_flush_log_at_trx_commit = 2
innodb_support_xa = 0
innodb_log_buffer_size = 8G
innodb_write_io_threads = 8
innodb_read_io_threads = 8
event_scheduler = 1
query_cache_type = 1
query_cache_size = 64M
join_buffer_size=16M
query_cache_limit=16M
sort_buffer_size=2M
read_buffer_size=2M
read_rnd_buffer_size=4M
key_buffer = 256M
key_buffer_size=64M
open_files_limit = 100000
thread_cache_size = 100
    Code:
    top - 16:40:10 up 1 day, 18:01,  1 user,  load average: 1.67, 2.08, 2.12
Tasks: 1465 total,   1 running, 1464 sleeping,   0 stopped,   0 zombie
%Cpu0  :  3.3 us,  0.7 sy,  0.0 ni, 96.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  4.6 us,  1.0 sy,  0.0 ni, 94.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  2.7 us,  0.7 sy,  0.0 ni, 96.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  8.0 us,  0.7 sy,  0.0 ni, 81.9 id,  9.4 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  :  5.0 us,  1.3 sy,  0.0 ni, 93.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  :  1.3 us,  1.0 sy,  0.0 ni, 97.3 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu6  :  2.0 us,  0.7 sy,  0.0 ni, 97.0 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu7  :  8.9 us,  1.3 sy,  0.0 ni, 89.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu8  :  4.6 us,  0.7 sy,  0.0 ni, 94.4 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu9  :  6.6 us,  1.0 sy,  0.0 ni, 92.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu10 :  6.3 us,  0.7 sy,  0.0 ni, 93.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu11 :  3.0 us,  0.3 sy,  0.0 ni, 96.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu12 :  1.3 us,  0.7 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu13 :  4.0 us,  0.3 sy,  0.0 ni, 95.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu14 :  1.7 us,  0.3 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu15 :  1.7 us,  0.3 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 65696748 total, 18100692 free, 42087120 used,  5508936 buff/cache
KiB Swap:  5915644 total,  5915644 free,        0 used. 19086636 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
22938 mysql     20   0 34.869g 0.032t  12816 S  18.2 52.2 353:00.11 mysqld
29313 zabbix    20   0 11.200g 3.318g 3.255g S   0.0  5.3  31:24.00 zabbix_server
29314 zabbix    20   0 11.191g 3.309g 3.255g S   0.3  5.3  31:32.86 zabbix_server
28814 zabbix    20   0 11.172g 805088 769212 S   0.0  1.2   0:12.26 zabbix_server
30626 zabbix    20   0 11.229g 735772 638772 S   1.7  1.1  26:47.43 zabbix_server
29035 zabbix    20   0 11.139g 723516 721108 S   0.0  1.1  19:01.28 zabbix_server
30603 zabbix    20   0 11.185g 486376 450360 S   0.0  0.7   0:41.45 zabbix_server
30013 zabbix    20   0 11.274g 416432 377420 S   0.0  0.6   0:10.81 zabbix_server
30288 zabbix    20   0 11.274g 416416 377348 S   0.0  0.6   0:11.65 zabbix_server
30213 zabbix    20   0 11.274g 416320 377232 S   0.0  0.6   0:10.60 zabbix_server
30090 zabbix    20   0 11.274g 416164 377168 S   0.0  0.6   0:11.27 zabbix_server
30248 zabbix    20   0 11.274g 416132 377044 S   0.0  0.6   0:10.70 zabbix_server
29876 zabbix    20   0 11.274g 416096 377028 S   0.0  0.6   0:12.00 zabbix_server
30195 zabbix    20   0 11.274g 416048 377048 S   0.0  0.6   0:11.04 zabbix_server
29800 zabbix    20   0 11.274g 416040 376956 S   0.0  0.6   0:10.80 zabbix_server
30313 zabbix    20   0 11.274g 416028 376960 S   0.0  0.6   0:10.92 zabbix_server
30192 zabbix    20   0 11.274g 416020 376956 S   0.0  0.6   0:11.30 zabbix_server
30038 zabbix    20   0 11.274g 416016 376952 S   0.0  0.6   0:11.73 zabbix_server
30120 zabbix    20   0 11.274g 416016 376932 S   0.0  0.6   0:10.49 zabbix_server
29408 zabbix    20   0 11.274g 416008 377024 S   0.0  0.6   0:11.77 zabbix_server
29766 zabbix    20   0 11.274g 415984 376920 S   0.0  0.6   0:10.85 zabbix_server
29385 zabbix    20   0 11.274g 415968 376888 S   0.0  0.6   0:10.68 zabbix_server
29805 zabbix    20   0 11.274g 415952 376868 S   0.0  0.6   0:10.64 zabbix_server
30356 zabbix    20   0 11.274g 415928 376832 S   0.0  0.6   0:10.90 zabbix_server
29801 zabbix    20   0 11.274g 415908 376840 S   0.0  0.6   0:10.13 zabbix_server
29855 zabbix    20   0 11.274g 415900 376820 S   0.0  0.6   0:10.97 zabbix_server
30208 zabbix    20   0 11.274g 415888 376828 S   0.0  0.6   0:10.15 zabbix_server
29973 zabbix    20   0 11.274g 415880 376816 S   0.0  0.6   0:11.44 zabbix_server
29391 zabbix    20   0 11.274g 415876 376812 S   0.0  0.6   0:10.53 zabbix_server
29916 zabbix    20   0 11.274g 415876 376792 S   0.0  0.6   0:10.15 zabbix_server
30092 zabbix    20   0 11.274g 415848 376784 S   0.0  0.6   0:11.61 zabbix_server
30393 zabbix    20   0 11.274g 415840 376760 S   0.0  0.6   0:11.52 zabbix_server
29988 zabbix    20   0 11.274g 415832 376768 S   0.0  0.6   0:10.77 zabbix_server
    Last edited by MaksimsE; 21-11-2017, 13:55.
    Tags: None
    • astrix89
      Senior Member
      • Jun 2017
      • 149
      #2
      Дисковая подсистема помимо заббикса чем-нибудь еще занета? И что там за диски стоят у вас.
      Судя по NVPS нагрузка мизерная, обычный десктоп, с небольшим тюнингом, тянет порядка 3k NVPS.

        Comment

        • MaksimsE
          Junior Member
          • Nov 2017
          • 23
          #3
          Сегодня ночью в 02:36 Zabbix перестал нормально записывать данные в базу.
          Вот что я вижу сейчас:

          30626 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 28468519, processed 0 values, idle 0.000000 sec during 5.042466 sec]

          То есть очередь явно выросла. С того момента загрузка одного из ядер 100%. В логах ничего не нашел.

          На графике сам момент тоже хорошо виден.

          iostat -xt:
          Code:
          avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.35    0.00    0.72    0.57    0.00   94.36

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.06    0.08    0.82     1.59     7.99    21.23     0.02   20.56    5.86   21.95  12.36   1.12
sdb               0.00     0.06    0.07    0.82     1.13     7.99    20.51     0.02   17.56    3.44   18.68  11.56   1.03
sdc               0.40    32.41   40.88  183.18   660.07  6226.20    61.47     0.57    2.56    0.66    2.98   1.53  34.24
sdd               0.18    32.23    5.93  183.35    97.69  6226.20    66.82     0.57    3.01    3.43    2.99   1.79  33.92
md127             0.00     0.00    0.09    0.57     2.28     7.54    30.00     0.00    0.00    0.00    0.00   0.00   0.00
md126             0.00     0.00    0.00    0.00     0.05     0.08   125.42     0.00    0.00    0.00    0.00   0.00   0.00
md125             0.00     0.00   47.38  204.46   757.73  6212.74    55.35     0.00    0.00    0.00    0.00   0.00   0.00
md2               0.00     0.00    0.05    0.01     0.33     0.14    14.28     0.00    0.00    0.00    0.00   0.00   0.00
          iotop:
          Code:
          Total DISK READ :       0.00 B/s | Total DISK WRITE :     939.52 K/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:      77.11 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
19125 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.17 % mysqld
19129 be/4 mysql       0.00 B/s  553.08 K/s  0.00 %  0.16 % mysqld
19117 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.02 % mysqld
19214 be/4 mysql       0.00 B/s   10.64 K/s  0.00 %  0.00 % mysqld
19279 be/4 mysql       0.00 B/s   14.18 K/s  0.00 %  0.00 % mysqld
19299 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
19338 be/4 mysql       0.00 B/s    3.55 K/s  0.00 %  0.00 % mysqld
19359 be/4 mysql       0.00 B/s    3.55 K/s  0.00 %  0.00 % mysqld
19418 be/4 mysql       0.00 B/s    3.55 K/s  0.00 %  0.00 % mysqld
19440 be/4 mysql       0.00 B/s   10.64 K/s  0.00 %  0.00 % mysqld
19479 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
19572 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
19726 be/4 mysql       0.00 B/s   14.18 K/s  0.00 %  0.00 % mysqld
19824 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
19858 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
20031 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
20095 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
20097 be/4 mysql       0.00 B/s    3.55 K/s  0.00 %  0.00 % mysqld
20140 be/4 mysql       0.00 B/s    3.55 K/s  0.00 %  0.00 % mysqld
20148 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
20198 be/4 mysql       0.00 B/s   10.64 K/s  0.00 %  0.00 % mysqld
20213 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
20219 be/4 mysql       0.00 B/s    3.55 K/s  0.00 %  0.00 % mysqld
20282 be/4 mysql       0.00 B/s   17.73 K/s  0.00 %  0.00 % mysqld
20330 be/4 mysql       0.00 B/s   17.73 K/s  0.00 %  0.00 % mysqld
20340 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
20369 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
20082 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
30607 be/4 zabbix      0.00 B/s    3.55 K/s  0.00 %  0.00 % zabbix_server: icmp pinger #4 [pinging hosts]
          Диски, на которых база:

          Code:
          Samsung SSD 850 PRO 1TB износ 48% uptime 1.5 года
Samsung SSD 850 PRO 1TB износ 47% uptime 1.5 года
Оба в RAID1
          Система вместе с логами и резервными копиями на:

          Code:
          Hitachi/HGST Ultrastar 7K4000 2TB uptime 2.5 года
Hitachi/HGST Ultrastar 7K4000 2TB uptime 2.5 года
          Last edited by MaksimsE; 21-11-2017, 13:55.

            Comment

            • astrix89
              Senior Member
              • Jun 2017
              • 149
              #4
              zabbix_server -V можно?
              preprocessing manager сколько памяти отъедает когда очередь растет?
              Обновления были?
              Есть подозрения на утечку памяти в preprocessing manager, была такая проблема у меня, помогло обновление.

              Есть такая утилита, mytop, показывает какие запросы идут в базу в реальном времени, посмортите во время проблемы что там происходит.

                Comment

                • MaksimsE
                  Junior Member
                  • Nov 2017
                  • 23
                  #5
                  zabbix_server -V:

                  Code:
                  zabbix_server (Zabbix) 3.4.3
Revision 73588 17 October 2017, compilation time: Oct 17 2017 16:43:26

Copyright (C) 2017 Zabbix SIA
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it according to
the license. There is NO WARRANTY, to the extent permitted by law.
                  Используемая память препроцессором:

                  30626 zabbix 20 0 21.193g 0.011t 1.529g R 100.0 18.5 576:27.44 zabbix_server

                  18.5%, что есть 11.84G

                  Обновления чего именно? Вообще yum update нет, сам zabbix обновлял на прошлой неделе. Сперва была проблема в том, что препроцессор был 1 по умолчанию, затем по рекомендации суппорта поднял его до количества ядер, т.е. 16. Проблема была такая же, только сейчас она появилась позже.

                  В Mytop постоянно висит запрос show full preprocesslist, проблема ли это?

                  Code:
                  MySQL on localhost (10.1.28-MariaDB)                                                                                                                                         up 0+17:06:20 [11:42:39]
 Queries: 39.2M  qps:  667 Slow:   99.7k         Se/In/Up/De(%):    81/01/09/00
             qps now:  420 Slow qps: 0.0  Threads: 1160 (   2/  18) 104/01/01/00
 Cache Hits: 1.9M  Hits/s: 32.7 Hits now:  86.1  Ratio:  6.0% Ratio now: 19.8%
 Key Efficiency: 60.0%  Bps in/out: 134.2k/ 1.0M   Now in/out: 43.2k/412.4k

      Id      User         Host/IP         DB      Time    Cmd Query or State
       --      ----         -------         --      ----    --- ----------
        9    zabbix       localhost     zabbix         0  Sleep
      176    zabbix       localhost     zabbix         0  Sleep
      203    zabbix       localhost     zabbix         0  Sleep
      245    zabbix       localhost     zabbix         0  Sleep
      328    zabbix       localhost     zabbix         0  Sleep
      332    zabbix       localhost     zabbix         0  Sleep
      347    zabbix       localhost     zabbix         0  Sleep
      425    zabbix       localhost     zabbix         0  Sleep
      451    zabbix       localhost     zabbix         0  Sleep
      601    zabbix       localhost     zabbix         0  Sleep
      678    zabbix       localhost     zabbix         0  Sleep
      807    zabbix       localhost     zabbix         0  Sleep
      812    zabbix       localhost     zabbix         0  Sleep
      890    zabbix       localhost     zabbix         0  Sleep
      900    zabbix       localhost     zabbix         0  Sleep
      954    zabbix       localhost     zabbix         0  Sleep
      970    zabbix       localhost     zabbix         0  Sleep
     1010    zabbix       localhost     zabbix         0  Sleep
     1168    zabbix       localhost     zabbix         0  Sleep
    80526    zabbix       localhost     zabbix         0  Query show full processlist
      214    zabbix       localhost     zabbix         1  Sleep
      282    zabbix       localhost     zabbix         1  Sleep
      300    zabbix       localhost     zabbix         1  Sleep
      339    zabbix       localhost     zabbix         1  Sleep
      396    zabbix       localhost     zabbix         1  Sleep
      407    zabbix       localhost     zabbix         1  Sleep
      486    zabbix       localhost     zabbix         1  Sleep
      537    zabbix       localhost     zabbix         1  Sleep
      579    zabbix       localhost     zabbix         1  Sleep
      682    zabbix       localhost     zabbix         1  Sleep
      705    zabbix       localhost     zabbix         1  Sleep
      725    zabbix       localhost     zabbix         1  Sleep
      843    zabbix       localhost     zabbix         1  Sleep
      884    zabbix       localhost     zabbix         1  Sleep
      936    zabbix       localhost     zabbix         1  Sleep

                    Comment

                    • astrix89
                      Senior Member
                      • Jun 2017
                      • 149
                      #6
                      В mytop нажмите i, задержка обновления s и число в секундах.

                        Comment

                        • MaksimsE
                          Junior Member
                          • Nov 2017
                          • 23
                          #7
                          MySQL on localhost (10.1.28-MariaDB) up 0+17:16:25 [11:52:44]
                          Queries: 39.5M qps: 667 Slow: 99.7k Se/In/Up/De(%): 81/01/09/00
                          qps now: 572 Slow qps: 0.0 Threads: 1160 ( 1/ 18) 88/00/03/00
                          Cache Hits: 1.9M Hits/s: 32.7 Hits now: 30.1 Ratio: 6.0% Ratio now: 6.0%
                          Key Efficiency: 60.0% Bps in/out: 133.9k/ 1.0M Now in/out: 91.2k/485.4k

                          Id User Host/IP DB Time Cmd Query or State
                          -- ---- ------- -- ---- --- ----------
                          80846 zabbix localhost zabbix 0 Query show full processlist

                            Comment

                            • astrix89
                              Senior Member
                              • Jun 2017
                              • 149
                              #8
                              Когда будет расти очередь, смотрите что пишется в базу, в особенности в какие таблицы.
                              Вариантов тут не много:
                              1. Если в очереди будет висеть таблицы trends и trends_uint, ваша система загибается при построении трендов(если вообще их делаете).
                              2. Если в очереди будет висеть таблицы history и history_uint, ваша система загибается при лавинообразном поступлении данных, дисковая подсистема не успевает это все записать.
                              3. Если в очереди ни чего не будет висеть... Значит проблема не с ssd/hdd

                              А вообще, посмотрел я по внимательнее на ваши первые скриншоты, такое было когда заканчивалось место на диске, падал IOPS, и привет лаги... Как только место освобождалось, все приходило в норму буквально минут за 5 или меньше.
                              Last edited by astrix89; 08-11-2017, 12:18.

                                Comment

                                • MaksimsE
                                  Junior Member
                                  • Nov 2017
                                  • 23
                                  #9
                                  Очередь растет на глазах:

                                  Code:
                                  Wed Nov  8 12:40:42 EET 2017
Redirecting to /bin/systemctl status zabbix-server.service
           ├─30626 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 36936613, processed 0 values, idle 0.000000 sec during 5.006135 sec
Wed Nov  8 12:40:43 EET 2017
Redirecting to /bin/systemctl status zabbix-server.service
           ├─30626 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 36936613, processed 0 values, idle 0.000000 sec during 5.006135 sec
Wed Nov  8 12:40:44 EET 2017
Redirecting to /bin/systemctl status zabbix-server.service
           ├─30626 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 36936613, processed 0 values, idle 0.000000 sec during 5.006135 sec
Wed Nov  8 12:40:45 EET 2017
Redirecting to /bin/systemctl status zabbix-server.service
           ├─30626 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 36936613, processed 0 values, idle 0.000000 sec during 5.006135 sec
Wed Nov  8 12:40:46 EET 2017
Redirecting to /bin/systemctl status zabbix-server.service
           ├─30626 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 36941347, processed 0 values, idle 0.000000 sec during 5.012215 sec
Wed Nov  8 12:40:47 EET 2017
Redirecting to /bin/systemctl status zabbix-server.service
           ├─30626 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 36941347, processed 0 values, idle 0.000000 sec during 5.012215 sec
Wed Nov  8 12:40:48 EET 2017
Redirecting to /bin/systemctl status zabbix-server.service
           ├─30626 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 36941347, processed 0 values, idle 0.000000 sec during 5.012215 sec
Wed Nov  8 12:40:49 EET 2017
Redirecting to /bin/systemctl status zabbix-server.service
           ├─30626 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 36941347, processed 0 values, idle 0.000000 sec during 5.012215 sec
                                  При этом в mytop я вижу:

                                  Code:
                                  MySQL on localhost (10.1.28-MariaDB)                                                                                                                                         up 0+18:06:42 [12:43:01]
 Queries: 41.2M  qps:  663 Slow:   99.7k         Se/In/Up/De(%):    81/01/09/00
             qps now:  457 Slow qps: 0.0  Threads: 1160 (   1/  18) 86/00/03/00
 Cache Hits: 2.0M  Hits/s: 32.1 Hits now:  14.5  Ratio:  5.9% Ratio now:  3.7%
 Key Efficiency: 70.0%  Bps in/out: 132.4k/ 1.0M   Now in/out: 81.5k/452.5k

      Id      User         Host/IP         DB      Time    Cmd Query or State
       --      ----         -------         --      ----    --- ----------
    82282    zabbix       localhost     zabbix         0  Query show full processlist
                                  Следовательно, проблему можно не искать в дисках?
                                  Если так, где бы Вы искали?

                                    Comment

                                    • astrix89
                                      Senior Member
                                      • Jun 2017
                                      • 149
                                      #10
                                      Возможно какой-то глючный хост, можно пачками отключать.
                                      Если используете прокси, отключайте по одной.

                                      А что вообще за данные у вас в очереди скапливаются? Посмотреть можно Администрирование->очередь, там есть детальный обзор.

                                      StartDBSyncers=2 увеличивали?
                                      StartTrappers=1 наверное тоже стоит увеличить.
                                      Last edited by astrix89; 08-11-2017, 12:56.

                                        Comment

                                        • MaksimsE
                                          Junior Member
                                          • Nov 2017
                                          • 23
                                          #11
                                          Возможно какой-то глючный хост, можно пачками отключать.
                                          Если используете прокси, отключайте по одной.

                                          прокси не используем

                                          А что вообще за данные у вас в очереди скапливаются? Посмотреть можно Администрирование->очередь, там есть детальный обзор.

                                          в очереди только SNMPv2 agent и SNMPv1 agent, но не более 30 секунд.

                                          StartDBSyncers=2 увеличивали?

                                          нет, т.к. где-то рекомендуют 1 syncer/1000 nvps.
                                          Вы думаете стоит увеличить?

                                          StartTrappers=1 наверное тоже стоит увеличить.

                                          у нас нет ни активных проверок, ни zabbix sender где либо


                                          У меня есть подозрение на discovery и timeout. Уменьшил сейчас timeout с 30 секунд до 5 и уберу лишние правила дискавери, посмотрим что будет.

                                          Code:
                                          Redirecting to /bin/systemctl status zabbix-server.service
           ├─29082 /usr/sbin/zabbix_server: discoverer #38 [processed 1 rules in 81316.963152 sec, performing discovery
           ├─29109 /usr/sbin/zabbix_server: discoverer #65 [processed 1 rules in 25202.996321 sec, performing discovery
           ├─29110 /usr/sbin/zabbix_server: discoverer #66 [processed 1 rules in 28126.188467 sec, performing discovery
           ├─29111 /usr/sbin/zabbix_server: discoverer #67 [processed 1 rules in 26357.790457 sec, performing discovery
           ├─29118 /usr/sbin/zabbix_server: discoverer #74 [processed 1 rules in 70600.069327 sec, performing discovery
           ├─29119 /usr/sbin/zabbix_server: discoverer #75 [processed 1 rules in 42981.116679 sec, performing discovery
           ├─29120 /usr/sbin/zabbix_server: discoverer #76 [processed 1 rules in 41835.078660 sec, performing discovery
           ├─29121 /usr/sbin/zabbix_server: discoverer #77 [processed 1 rules in 17297.848522 sec, performing discovery
           ├─29122 /usr/sbin/zabbix_server: discoverer #78 [processed 1 rules in 60064.122587 sec, performing discovery
           ├─29123 /usr/sbin/zabbix_server: discoverer #79 [processed 1 rules in 33676.250530 sec, performing discovery
           ├─29124 /usr/sbin/zabbix_server: discoverer #80 [processed 1 rules in 42390.735919 sec, performing discovery
           ├─29125 /usr/sbin/zabbix_server: discoverer #81 [processed 1 rules in 27787.531799 sec, performing discovery
           ├─29126 /usr/sbin/zabbix_server: discoverer #82 [processed 1 rules in 27480.026395 sec, performing discovery
           ├─29127 /usr/sbin/zabbix_server: discoverer #83 [processed 1 rules in 15307.259473 sec, performing discovery
           ├─29128 /usr/sbin/zabbix_server: discoverer #84 [processed 1 rules in 20892.406963 sec, performing discovery
           ├─29129 /usr/sbin/zabbix_server: discoverer #85 [processed 1 rules in 31830.075783 sec, performing discovery
           ├─29130 /usr/sbin/zabbix_server: discoverer #86 [processed 1 rules in 42953.851083 sec, performing discovery
           ├─29131 /usr/sbin/zabbix_server: discoverer #87 [processed 1 rules in 21284.307118 sec, performing discovery
           ├─29132 /usr/sbin/zabbix_server: discoverer #88 [processed 1 rules in 40119.390920 sec, performing discovery
           ├─29133 /usr/sbin/zabbix_server: discoverer #89 [processed 1 rules in 21070.987304 sec, performing discovery
           ├─29134 /usr/sbin/zabbix_server: discoverer #90 [processed 1 rules in 28783.359823 sec, performing discovery
           ├─29135 /usr/sbin/zabbix_server: discoverer #91 [processed 1 rules in 44506.155894 sec, performing discovery
           ├─29232 /usr/sbin/zabbix_server: discoverer #188 [processed 1 rules in 20360.848667 sec, performing discovery
           ├─29233 /usr/sbin/zabbix_server: discoverer #189 [processed 1 rules in 14467.798476 sec, performing discovery
           ├─29234 /usr/sbin/zabbix_server: discoverer #190 [processed 1 rules in 14766.505182 sec, performing discovery
           ├─29235 /usr/sbin/zabbix_server: discoverer #191 [processed 1 rules in 48769.864897 sec, performing discovery
           ├─29236 /usr/sbin/zabbix_server: discoverer #192 [processed 1 rules in 57624.336098 sec, performing discovery
           ├─29237 /usr/sbin/zabbix_server: discoverer #193 [processed 1 rules in 56714.511533 sec, performing discovery
           ├─29238 /usr/sbin/zabbix_server: discoverer #194 [processed 1 rules in 26164.136517 sec, performing discovery
           ├─29241 /usr/sbin/zabbix_server: discoverer #197 [processed 1 rules in 51256.672664 sec, performing discovery
           ├─29242 /usr/sbin/zabbix_server: discoverer #198 [processed 1 rules in 56805.099653 sec, performing discovery
           ├─29243 /usr/sbin/zabbix_server: discoverer #199 [processed 1 rules in 57385.029700 sec, performing discovery
           ├─29244 /usr/sbin/zabbix_server: discoverer #200 [processed 1 rules in 14406.476498 sec, performing discovery
           ├─29246 /usr/sbin/zabbix_server: discoverer #202 [processed 1 rules in 27831.401260 sec, performing discovery
           ├─29247 /usr/sbin/zabbix_server: discoverer #203 [processed 1 rules in 28126.014803 sec, performing discovery
           ├─29248 /usr/sbin/zabbix_server: discoverer #204 [processed 1 rules in 25512.802465 sec, performing discovery
           ├─29249 /usr/sbin/zabbix_server: discoverer #205 [processed 1 rules in 106630.359209 sec, performing discovery
           ├─29250 /usr/sbin/zabbix_server: discoverer #206 [processed 1 rules in 108338.401020 sec, performing discovery
           ├─29251 /usr/sbin/zabbix_server: discoverer #207 [processed 1 rules in 56562.586089 sec, performing discovery
           ├─29252 /usr/sbin/zabbix_server: discoverer #208 [processed 1 rules in 30375.069189 sec, performing discovery
           ├─29253 /usr/sbin/zabbix_server: discoverer #209 [processed 1 rules in 15006.548462 sec, performing discovery

                                            Comment

                                            • astrix89
                                              Senior Member
                                              • Jun 2017
                                              • 149
                                              #12
                                              Думаю стоит, хуже точно не будет. С другой стороны данные быстрее будут в базу попадать, скажем так, не 2 потока, а куда больше.
                                              можно еще ps ax | grep "history syncer" ?

                                                Comment

                                                • MaksimsE
                                                  Junior Member
                                                  • Nov 2017
                                                  • 23
                                                  #13
                                                  Количество StartDBSyncers поднял с 2 до 10.

                                                  ps ax | grep "history syncer" в течение 10 секунд:

                                                  Code:
                                                  Wed Nov  8 14:02:12 EET 2017
18604 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #1 [synced 403 items in 0.040444 sec, idle 1 sec]
18605 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #2 [synced 4 items in 0.001283 sec, idle 1 sec]
18606 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #3 [synced 0 items in 0.000032 sec, idle 1 sec]
18607 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #4 [synced 63 items in 0.004622 sec, idle 1 sec]
18608 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #5 [synced 3 items in 0.000610 sec, idle 1 sec]
18609 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #6 [synced 66 items in 0.007265 sec, idle 1 sec]
18610 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #7 [synced 99 items in 0.007877 sec, idle 1 sec]
18611 ?        S      0:03 /usr/sbin/zabbix_server: history syncer #8 [synced 7 items in 0.001608 sec, idle 1 sec]
18612 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #9 [synced 0 items in 0.000003 sec, idle 1 sec]
18613 ?        S      0:06 /usr/sbin/zabbix_server: history syncer #10 [synced 5 items in 0.001492 sec, idle 1 sec]
25984 pts/0    S+     0:00 grep --color=auto history syncer
Wed Nov  8 14:02:13 EET 2017
18604 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #1 [synced 32 items in 0.013892 sec, idle 1 sec]
18605 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #2 [synced 0 items in 0.000001 sec, idle 1 sec]
18606 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #3 [synced 0 items in 0.000001 sec, idle 1 sec]
18607 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #4 [synced 833 items in 0.044559 sec, idle 1 sec]
18608 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #5 [synced 0 items in 0.000001 sec, idle 1 sec]
18609 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #6 [synced 1 items in 0.004153 sec, idle 1 sec]
18610 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #7 [synced 1 items in 0.000319 sec, idle 1 sec]
18611 ?        S      0:03 /usr/sbin/zabbix_server: history syncer #8 [synced 0 items in 0.000001 sec, idle 1 sec]
18612 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #9 [synced 0 items in 0.000001 sec, idle 1 sec]
18613 ?        S      0:06 /usr/sbin/zabbix_server: history syncer #10 [synced 0 items in 0.000001 sec, idle 1 sec]
25996 pts/0    S+     0:00 grep --color=auto history syncer
Wed Nov  8 14:02:14 EET 2017
18604 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000001 sec, idle 1 sec]
18605 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #2 [synced 0 items in 0.000464 sec, idle 1 sec]
18606 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #3 [synced 9 items in 0.003058 sec, idle 1 sec]
18607 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #4 [synced 1748 items in 0.091142 sec, idle 1 sec]
18608 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #5 [synced 0 items in 0.000397 sec, idle 1 sec]
18609 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #6 [synced 9 items in 0.001586 sec, idle 1 sec]
18610 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #7 [synced 0 items in 0.000001 sec, idle 1 sec]
18611 ?        S      0:03 /usr/sbin/zabbix_server: history syncer #8 [synced 2 items in 0.025958 sec, idle 1 sec]
18612 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #9 [synced 0 items in 0.000065 sec, idle 1 sec]
18613 ?        S      0:06 /usr/sbin/zabbix_server: history syncer #10 [synced 0 items in 0.000025 sec, idle 1 sec]
26003 pts/0    S+     0:00 grep --color=auto history syncer
Wed Nov  8 14:02:15 EET 2017
18604 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #1 [synced 438 items in 0.057068 sec, idle 1 sec]
18605 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #2 [synced 0 items in 0.000009 sec, idle 1 sec]
18606 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #3 [synced 230 items in 0.014997 sec, idle 1 sec]
18607 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #4 [synced 159 items in 0.022618 sec, idle 1 sec]
18608 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #5 [synced 0 items in 0.000032 sec, idle 1 sec]
18609 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #6 [synced 2 items in 0.000896 sec, idle 1 sec]
18610 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #7 [synced 20 items in 0.003781 sec, idle 1 sec]
18611 ?        S      0:03 /usr/sbin/zabbix_server: history syncer #8 [synced 84 items in 0.007108 sec, idle 1 sec]
18612 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #9 [synced 0 items in 0.000007 sec, idle 1 sec]
18613 ?        S      0:06 /usr/sbin/zabbix_server: history syncer #10 [synced 31 items in 0.007141 sec, idle 1 sec]
26011 pts/0    S+     0:00 grep --color=auto history syncer
Wed Nov  8 14:02:17 EET 2017
18604 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #1 [synced 408 items in 0.062994 sec, idle 1 sec]
18605 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #2 [synced 74 items in 0.006163 sec, idle 1 sec]
18606 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #3 [synced 0 items in 0.000001 sec, idle 1 sec]
18607 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #4 [synced 0 items in 0.000001 sec, idle 1 sec]
18608 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #5 [synced 0 items in 0.000005 sec, idle 1 sec]
18609 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #6 [synced 0 items in 0.000001 sec, idle 1 sec]
18610 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #7 [synced 0 items in 0.000001 sec, idle 1 sec]
18611 ?        S      0:03 /usr/sbin/zabbix_server: history syncer #8 [synced 0 items in 0.000001 sec, idle 1 sec]
18612 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #9 [synced 0 items in 0.000004 sec, idle 1 sec]
18613 ?        S      0:06 /usr/sbin/zabbix_server: history syncer #10 [synced 0 items in 0.000001 sec, idle 1 sec]
26024 pts/0    S+     0:00 grep --color=auto history syncer
Wed Nov  8 14:02:18 EET 2017
18604 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #1 [synced 206 items in 0.013181 sec, idle 1 sec]
18605 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #2 [synced 0 items in 0.000057 sec, idle 1 sec]
18606 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #3 [synced 0 items in 0.000166 sec, idle 1 sec]
18607 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #4 [synced 12 items in 0.002281 sec, idle 1 sec]
18608 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #5 [synced 1127 items in 0.071832 sec, idle 1 sec]
18609 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #6 [synced 0 items in 0.000029 sec, idle 1 sec]
18610 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #7 [synced 6 items in 0.001410 sec, idle 1 sec]
18611 ?        S      0:03 /usr/sbin/zabbix_server: history syncer #8 [synced 0 items in 0.000007 sec, idle 1 sec]
18612 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #9 [synced 26 items in 0.004897 sec, idle 1 sec]
18613 ?        S      0:06 /usr/sbin/zabbix_server: history syncer #10 [synced 71 items in 0.011904 sec, idle 1 sec]
26039 pts/0    S+     0:00 grep --color=auto history syncer
Wed Nov  8 14:02:19 EET 2017
18604 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #1 [synced 2 items in 0.000227 sec, idle 1 sec]
18605 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #2 [synced 0 items in 0.000004 sec, idle 1 sec]
18606 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #3 [synced 0 items in 0.000001 sec, idle 1 sec]
18607 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #4 [synced 0 items in 0.000001 sec, idle 1 sec]
18608 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #5 [synced 0 items in 0.000001 sec, idle 1 sec]
18609 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #6 [synced 0 items in 0.000000 sec, idle 1 sec]
18610 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #7 [synced 0 items in 0.000000 sec, idle 1 sec]
18611 ?        S      0:03 /usr/sbin/zabbix_server: history syncer #8 [synced 57 items in 0.005112 sec, idle 1 sec]
18612 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #9 [synced 9 items in 0.001476 sec, idle 1 sec]
18613 ?        S      0:06 /usr/sbin/zabbix_server: history syncer #10 [synced 0 items in 0.000001 sec, idle 1 sec]
26057 pts/0    S+     0:00 grep --color=auto history syncer
Wed Nov  8 14:02:20 EET 2017
18604 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #1 [synced 51 items in 0.003877 sec, idle 1 sec]
18605 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #2 [synced 1029 items in 0.062145 sec, idle 1 sec]
18606 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #3 [synced 5 items in 0.002117 sec, idle 1 sec]
18607 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #4 [synced 0 items in 0.000042 sec, idle 1 sec]
18608 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #5 [synced 0 items in 0.000001 sec, idle 1 sec]
18609 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #6 [synced 0 items in 0.000341 sec, idle 1 sec]
18610 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #7 [synced 0 items in 0.000001 sec, idle 1 sec]
18611 ?        S      0:03 /usr/sbin/zabbix_server: history syncer #8 [synced 0 items in 0.000001 sec, idle 1 sec]
18612 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #9 [synced 765 items in 0.045897 sec, idle 1 sec]
18613 ?        S      0:06 /usr/sbin/zabbix_server: history syncer #10 [synced 13 items in 0.009548 sec, idle 1 sec]
26065 pts/0    S+     0:00 grep --color=auto history syncer
Wed Nov  8 14:02:21 EET 2017
18604 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #1 [synced 525 items in 0.025105 sec, idle 1 sec]
18605 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #2 [synced 0 items in 0.000001 sec, idle 1 sec]
18606 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #3 [synced 6 items in 0.001096 sec, idle 1 sec]
18607 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #4 [synced 0 items in 0.000004 sec, idle 1 sec]
18608 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #5 [synced 0 items in 0.000005 sec, idle 1 sec]
18609 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #6 [synced 11 items in 0.002780 sec, idle 1 sec]
18610 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #7 [synced 44 items in 0.003041 sec, idle 1 sec]
18611 ?        S      0:03 /usr/sbin/zabbix_server: history syncer #8 [synced 20 items in 0.001863 sec, idle 1 sec]
18612 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #9 [synced 158 items in 0.013358 sec, idle 1 sec]
18613 ?        S      0:06 /usr/sbin/zabbix_server: history syncer #10 [synced 36 items in 0.004651 sec, idle 1 sec]
26077 pts/0    S+     0:00 grep --color=auto history syncer
Wed Nov  8 14:02:22 EET 2017
18604 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #1 [synced 732 items in 0.051497 sec, idle 1 sec]
18605 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #2 [synced 0 items in 0.000001 sec, idle 1 sec]
18606 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #3 [synced 0 items in 0.000001 sec, idle 1 sec]
18607 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #4 [synced 0 items in 0.000001 sec, idle 1 sec]
18608 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #5 [synced 0 items in 0.000001 sec, idle 1 sec]
18609 ?        S      0:04 /usr/sbin/zabbix_server: history syncer #6 [synced 43 items in 0.003872 sec, idle 1 sec]
18610 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #7 [synced 0 items in 0.000001 sec, idle 1 sec]
18611 ?        S      0:03 /usr/sbin/zabbix_server: history syncer #8 [synced 0 items in 0.000001 sec, idle 1 sec]
18612 ?        S      0:05 /usr/sbin/zabbix_server: history syncer #9 [synced 0 items in 0.000001 sec, idle 1 sec]
18613 ?        S      0:06 /usr/sbin/zabbix_server: history syncer #10 [synced 0 items in 0.000001 sec, idle 1 sec]

                                                    Comment

                                                    • astrix89
                                                      Senior Member
                                                      • Jun 2017
                                                      • 149
                                                      #14
                                                      Даже не знаю, идей пока ни каких нет по этому поводу.
                                                      Вариант с временным отключением хостов стоит рассмотреть.
                                                      Еще стоит по всем итемам пройтись на предмет сложности преобразования данных, возможно там что будет.

                                                        Comment

                                                        • MaksimsE
                                                          Junior Member
                                                          • Nov 2017
                                                          • 23
                                                          #15
                                                          Поднял уже до 20 DBSyncer. Все еще срабатывает триггер, но теперь его "отпускает" значительно раньше. Подниму до 50, посмотрим что будет...

                                                            Comment

                                                            Previous 1 2 template Next
                                                            Working...