Ad Widget

Collapse

RedHat + Zabbix on VM + SAN = Disk IO is overloaded

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • tchjts1
    Senior Member
    • May 2008
    • 1605

    #1

    RedHat + Zabbix on VM + SAN = Disk IO is overloaded

    We are running Zabbix APP server and DB server on 2 VM's. These VM's have 4 vCPU's and 8GB of memory.

    On the DB server, we were experiencing very sluggish graph rendering and seeing a fair amount of swap space usage, and often getting the alert of Disk IO is overloaded on Zabbix DB Server. The DB itself resides on the SAN

    We found a setting in RHEL called "Swappiness" that basically tells the OS how aggressively to switch processes out of memory and into swap space. This also applies to Ubuntu. This setting's range is from 0 - 100. By default, it is set to 60.

    We set this to 0 and everything started to breathe much easier. The CPU cycles eased up, swap space usage went to almost zero and graph rendering got much more snappy. You can see where we changed this setting in the graphs at the bottom of the page.

    Here is a snippet of an editorial about the swappiness setting:


    The swappiness parameter controls the tendency of the kernel to move processes out of physical memory and onto the swap disk. Because disks are much slower than RAM,
    this can lead to slower response times for system and applications if processes are too aggressively moved out of memory.

    • swappiness can have a value of between 0 and 100
    • swappiness=0 tells the kernel to avoid swapping processes out of physical memory for as long as possible
    • swappiness=100 tells the kernel to aggressively swap processes out of physical memory and move them to swap cache

    The default setting in most systems including Ubuntu and RHEL is swappiness=60. Reducing the default value of swappiness will probably improve overall performance for a typical desktop installation. A value of swappiness=10 is recommended, but feel free to experiment.
    Note: Server installations have different performance requirements to desktop systems, and the default value of 60 is likely more suitable.

    To check your current running swappiness simply run:
    Code:
    cat /proc/sys/vm/swappiness
    To make the change in a temporary state (until you reboot) run: (you can make this permanent by running sysctl -p)
    Code:
     sudo sysctl vm.swappiness=10
    To make this permanent you can run:
    Code:
     vim /etc/sysctl.conf
    and add the line
    Code:
     vm.swappiness=10
    Attached Files
    Last edited by tchjts1; 10-04-2013, 19:50.
  • emmanux
    Member
    Zabbix Certified Specialist
    • Mar 2013
    • 47

    #2
    Many thanks, just what i was looking for.

    I have some other VM servers with this trigger (disk i/o) blinking out from time to time, any of them running databases like Oracle, MySQL, or Postgresql. Most of them redhats, but a debian too. Avoiding swap helps but not completely solves.

    So, can we say that it's a bad idea to run databases on virtual machines? Or they need just other metrics? Have we a hardware problem (should i look for a SAN failure)? Is it all about OS settings like swappiness? May be an ESXi bug?

    Coming again over the swappiness trick, do you recommend set it to zero for vm redhat databases?

    Many thanks in advance

    Comment

    • tchjts1
      Senior Member
      • May 2008
      • 1605

      #3
      I don't necessarily think it is a bad idea to run a DB on a VM, but it all depends on how much memory you can allocate to the VM, how large you anticipate your DB to grow to, and how well tuned you get your VM and DB.

      However, if I have a choice between a VM and a stand-alone server... I will take the stand-alone every time.

      Regarding the swappiness setting, I would definitely set it to zero. The OS will still use swap if it needs it... but as a last resort.

      There are other settings that will need tuned to help you out. I've had to adjust several others since I wrote the original post in this thread to ease that condition. Now the frontend is lightning fast overall.

      Make sure you create a screen of graphs for your Zabbix server internal checks as seen below. They will help provide clues of what needs attention.

      One of the more productive settings I changed was on the Zabbix server side in zabbix_server.conf - I changed the Timeout= value from 3 to 10.

      Also, below where you see the big drop in poller process busy %, I changed from using passive items to using Zabbix agent Active items.
      Attached Files

      Comment

      • emmanux
        Member
        Zabbix Certified Specialist
        • Mar 2013
        • 47

        #4
        Your advices are priceless. Thanks a lot!
        Last edited by emmanux; 03-04-2013, 01:51.

        Comment

        • kloczek
          Senior Member
          • Jun 2006
          • 1771

          #5
          Originally posted by tchjts1
          On the DB server, we were experiencing very sluggish graph rendering and seeing a fair amount of swap space usage, and often getting the alert of Disk IO is overloaded on Zabbix DB Server. The DB itself resides on the SAN

          We found a setting in RHEL called "Swappiness" that basically tells the OS how aggressively to switch processes out of memory and into swap space. This also applies to Ubuntu. This setting's range is from 0 - 100. By default, it is set to 60.

          We set this to 0 and everything started to breathe much easier. The CPU cycles eased up, swap space usage went to almost zero and graph rendering got much more snappy. You can see where we changed this setting in the graphs at the bottom of the page.
          If you have blocked using swap or don't want to use swap by change swappinness to low value or even 0 why you have swap dev?
          Contradiction.
          Usually Linux after few days even with swappiness=0 starts using swap dev if it can. If it can it means that you have started some processes and these processes are doing nothing (waiting on put in running queue but conditions necessary to put these processes in running queue never happens).
          Usually when system starts using swap constantly it is useless and/or it is kind of critical state.

          Remove swap dev, add more memory and review list of services started from init scripts.
          Last edited by kloczek; 06-10-2013, 22:35.
          http://uk.linkedin.com/pub/tomasz-k%...zko/6/940/430/
          https://kloczek.wordpress.com/
          zapish - Zabbix API SHell binding https://github.com/kloczek/zapish
          My zabbix templates https://github.com/kloczek/zabbix-templates

          Comment

          • tchjts1
            Senior Member
            • May 2008
            • 1605

            #6
            Originally posted by kloczek
            If you have blocked using swap or don't want to use swap by change swappinness to low value or even 0 why you have swap dev?
            Contradiction.
            I am only relating what worked for me. If you found a better way, then that's awesome.

            Comment

            Working...