Ad Widget

Collapse

Memory leak in trapper

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • alj
    Senior Member
    • Aug 2006
    • 188

    #1

    Memory leak in trapper

    Zabbix trapper has 2 problems (1.4.2):

    1. It stops working after someone restarts mysql (need to restart zabbix after that), by "stops working" i mean it accepts connection but does not do anything - zabbix_sender is hanging forever.
    2. Zabbix trapper has some nasty memory leak. The memory gets released if mysql gets restarted indicating that memory is leaking somewhere in the database connector code.

    Here's some evidence (look some processes uses small amount of memory but trapper processes completely went out of control):

    Code:
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    zabbix   27213  0.0  0.0  33364  1352 ?        SN   Nov09   0:00 /usr/sbin/zabbix_server
    zabbix   27214  0.0  0.0  36108  2504 ?        SN   Nov09   4:28 /usr/sbin/zabbix_server
    zabbix   27215  0.1  0.0  36108  2568 ?        SN   Nov09   6:22 /usr/sbin/zabbix_server
    zabbix   27216  0.0  0.0  36108  2512 ?        SN   Nov09   5:51 /usr/sbin/zabbix_server
    zabbix   27217  0.0  0.0  36108  2540 ?        SN   Nov09   6:04 /usr/sbin/zabbix_server
    zabbix   27218  0.0  0.0  36112  2576 ?        SN   Nov09   5:49 /usr/sbin/zabbix_server
    zabbix   27219  0.0  0.0  36108  2532 ?        SN   Nov09   5:17 /usr/sbin/zabbix_server
    zabbix   27220  0.0  0.0  36108  2528 ?        SN   Nov09   5:31 /usr/sbin/zabbix_server
    zabbix   27221  0.0  0.0  36108  2532 ?        SN   Nov09   4:13 /usr/sbin/zabbix_server
    zabbix   27222  0.0  0.0  36148  2544 ?        SN   Nov09   5:01 /usr/sbin/zabbix_server
    zabbix   27223  0.0  0.0  36136  2536 ?        SN   Nov09   4:57 /usr/sbin/zabbix_server
    zabbix   27224  0.0  0.0  36152  2516 ?        SN   Nov09   4:17 /usr/sbin/zabbix_server
    zabbix   27225  0.0  0.0  36112  2544 ?        SN   Nov09   5:59 /usr/sbin/zabbix_server
    zabbix   27226  0.0  0.0  36148  2548 ?        SN   Nov09   5:06 /usr/sbin/zabbix_server
    zabbix   27227  0.0  0.0  36108  2524 ?        SN   Nov09   4:58 /usr/sbin/zabbix_server
    zabbix   27228  0.0  0.0  36108  2540 ?        SN   Nov09   5:10 /usr/sbin/zabbix_server
    zabbix   27229  0.0  0.0  36108  2540 ?        SN   Nov09   4:31 /usr/sbin/zabbix_server
    zabbix   27230  0.0  0.0  36108  2548 ?        SN   Nov09   5:16 /usr/sbin/zabbix_server
    zabbix   27231  0.0  0.0  36108  2500 ?        SN   Nov09   4:41 /usr/sbin/zabbix_server
    zabbix   27232  0.0  0.0  36160  2516 ?        SN   Nov09   4:40 /usr/sbin/zabbix_server
    zabbix   27233  0.0  0.0  36108  2508 ?        SN   Nov09   4:45 /usr/sbin/zabbix_server
    zabbix   27234  0.0  1.7 1461392 294392 ?      SN   Nov09   2:08 /usr/sbin/zabbix_server
    zabbix   27235  0.0  1.7 1454320 291428 ?      SN   Nov09   2:12 /usr/sbin/zabbix_server
    zabbix   27236  0.0  1.7 1453648 290288 ?      SN   Nov09   2:08 /usr/sbin/zabbix_server
    zabbix   27237  0.0  1.7 1451564 290160 ?      SN   Nov09   2:12 /usr/sbin/zabbix_server
    zabbix   27238  0.0  1.7 1451332 292232 ?      SN   Nov09   2:10 /usr/sbin/zabbix_server
    zabbix   27239  0.0  1.7 1449808 289296 ?      SN   Nov09   2:09 /usr/sbin/zabbix_server
    zabbix   27240  0.0  1.7 1452012 293712 ?      SN   Nov09   2:10 /usr/sbin/zabbix_server
    zabbix   27241  0.0  1.7 1449904 295028 ?      SN   Nov09   2:09 /usr/sbin/zabbix_server
    zabbix   27242  0.0  1.7 1457904 293332 ?      SN   Nov09   2:08 /usr/sbin/zabbix_server
    zabbix   27243  0.0  1.7 1450320 291992 ?      SN   Nov09   2:10 /usr/sbin/zabbix_server
    zabbix   27244  0.0  1.7 1449304 291320 ?      SN   Nov09   2:11 /usr/sbin/zabbix_server
    zabbix   27245  0.0  1.7 1464636 294604 ?      SN   Nov09   2:09 /usr/sbin/zabbix_server
    zabbix   27246  0.0  1.7 1454592 291944 ?      SN   Nov09   2:10 /usr/sbin/zabbix_server
    zabbix   27247  0.0  1.7 1460532 293232 ?      SN   Nov09   2:09 /usr/sbin/zabbix_server
    zabbix   27248  0.0  1.8 1452132 297060 ?      SN   Nov09   2:10 /usr/sbin/zabbix_server
    zabbix   27249  0.0  1.7 1458280 292416 ?      SN   Nov09   2:10 /usr/sbin/zabbix_server
    zabbix   27250  0.0  1.7 1456060 293064 ?      SN   Nov09   2:09 /usr/sbin/zabbix_server
    zabbix   27251  0.0  1.7 1459548 292708 ?      SN   Nov09   2:10 /usr/sbin/zabbix_server
    zabbix   27252  0.0  1.7 1453864 292100 ?      SN   Nov09   2:10 /usr/sbin/zabbix_server
    zabbix   27253  0.0  1.7 1453840 292764 ?      SN   Nov09   2:10 /usr/sbin/zabbix_server
    zabbix   27254  0.0  1.7 1447996 291824 ?      SN   Nov09   2:13 /usr/sbin/zabbix_server
    zabbix   27255  0.0  1.7 1458616 292036 ?      SN   Nov09   2:11 /usr/sbin/zabbix_server
    zabbix   27256  0.0  1.7 1452404 292508 ?      SN   Nov09   2:10 /usr/sbin/zabbix_server
    zabbix   27257  0.0  1.7 1457528 292592 ?      SN   Nov09   2:09 /usr/sbin/zabbix_server
    zabbix   27258  0.0  1.7 1456252 291016 ?      SN   Nov09   2:10 /usr/sbin/zabbix_server
    zabbix   27259  0.0  0.0  33380  1204 ?        SN   Nov09   1:11 /usr/sbin/zabbix_server
    zabbix   27260  0.0  0.0  33364  1192 ?        SN   Nov09   0:00 /usr/sbin/zabbix_server
    zabbix   27261  0.0  0.0  33364   992 ?        SN   Nov09   0:10 /usr/sbin/zabbix_server
    zabbix   27262  0.0  0.0  33368  1160 ?        SN   Nov09   0:00 /usr/sbin/zabbix_server
    zabbix   27263  0.0  0.0  36120  2284 ?        SN   Nov09   0:03 /usr/sbin/zabbix_server
    zabbix   27264  0.0  0.0  36120  2288 ?        SN   Nov09   0:03 /usr/sbin/zabbix_server
    zabbix   27266  0.0  0.0  33364   792 ?        SN   Nov09   0:01 /usr/sbin/zabbix_server
    zabbix   27267  0.0  0.0  33364   784 ?        SN   Nov09   0:00 /usr/sbin/zabbix_server
    zabbix   27268  0.0  0.0  34668  2384 ?        SN   Nov09   0:02 /usr/sbin/zabbix_server
    zabbix   27269  0.0  0.0  34936  1692 ?        SN   Nov09   0:00 /usr/sbin/zabbix_server
  • alj
    Senior Member
    • Aug 2006
    • 188

    #2
    Since noone responds, theres a script to automatically restart zabbix if any process takes more than 400mb of virtual ram.


    Code:
    #!/usr/bin/perl
    
    use strict;
    my $totalvsz=0;
    my $totalrss=0;
    my $processvsztreshold=400000;
    my $tresholdflag=0;
    open (CMD,'ps axuw |') or die "$!";
    while (<CMD>) {
       next unless /zabbix_server/;
       my @ar=split /\s+/;
       $totalvsz+=$ar[4];
       $totalrss+=$ar[5];
       $tresholdflag++ if $ar[4] > $processvsztreshold;
    }
    close CMD;
    #print "Total VSZ: $totalvsz, RSS: $totalrss\n";
    
    if ($tresholdflag) {
       system('echo "Restarting zabbix" | mailx -s "zabbixwatch" mymail@domain');
       system('/etc/init.d/zabbix-server restart > /dev/null 2>&1');
       sleep 5;
       unless (&countzab) {
          system('echo "Restarting zabbix 2-nd time" | mailx -s "zabbixwatch" mymail@domain');
          system('/usr/sbin/invoke-rc.d --quiet zabbix-server restart > /dev/null 2>&1');
          sleep 5;
          unless (&countzab) {
    #           print "Cannot restart zabbix";
          system('echo "zabbixwatch script cannot restart zabbix server, please contact me immediately" | mailx -s "zabbixwatch" operators@mydomain');
          }
       }
    }
    1;
    ######
    #
    #
    sub countzab {
       my $num=0;
       open (CMD,'ps axuw |') or die "$!";
       while (<CMD>) {
          $num++ if /zabbix_server/;
       }
       close CMD;
       return $num;
    }

    Comment

    • Alexei
      Founder, CEO
      Zabbix Certified Trainer
      Zabbix Certified SpecialistZabbix Certified Professional
      • Sep 2004
      • 5654

      #3
      The memory leak was already fixed long time ago. It happens only in case of non-existent items received by the trapper.
      Alexei Vladishev
      Creator of Zabbix, Product manager
      New York | Tokyo | Riga
      My Twitter

      Comment

      • alj
        Senior Member
        • Aug 2006
        • 188

        #4
        That would explain it. When it was fixed? in cvs tree?
        I run 1.4.2. When 1.4.3 is out?

        Comment

        Working...