Ad Widget

Collapse

Zabbix 4.4 error - [Z3005] query failed: [1114] The table 'history_uint' is full

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • csid2174
    Junior Member
    • Sep 2020
    • 4

    #1

    Zabbix 4.4 error - [Z3005] query failed: [1114] The table 'history_uint' is full

    Hello Zabbix World,

    For last few months, I've been trying to upgrade my Zabbix installation from Zabbix 2.4.5 to Zabbix 4.4 on a x32Bit Centos 6.4 Based Linux Operating System. I did a fresh install of compatible x32bit Zabbix Linux RPMs and I was successful. I managed to successfully run Zabbix 4.4 release on a x32bit OS without any issues for almost 2 weeks without monitoring any nodes.

    Note: My linux server is a HP DL360p Gen8 with RAID1 config and has Harddisk storage of 500GB approx. On the server I have 16GB RAM.

    So after waiting, for few weeks to ensure stability of the Zabbix 4.4 release on my x32 bit Centos 6.4 based linux server, I configured almost 50+ linux hosts on 28th August 2020 for monitoring via the Zabbix 4.4 frontend GUI successfully. However, after few days of monitoring I started receiving "[Z3005] query failed: [1114] The table 'history_uint' is full" error messages in zabbix_server.log as shown below.

    I've attached my configuration for zabbix_server and my.cnf(mysql - 5.7.28 release) for reference. I've been trying different settings in the configuration files after reading many zabbix related forums and I am kind of lost now, hence I desperately need help from experts on Zabbix.

    Also I am not a database guy and I'd greatly appreciate any help that I can get. I have been quite successful in managing zabbix 2.4.5 platform for last five years with some tweaking that was required in zabbix_server and mysql my.cnf configuration files initially and thereafter my life was quite comfortable.

    But ever since business requirements have changed, I really need to get Zabbix 4.4 platform on x32 bit linux platform running with acceptable stability. I am not allowed to upgrade to x64 bit OS, hence I am stuck with x32 bit OS.

    Once again I'd really appreciate any resolution that I can get with this problem and it would be nice if I don't have to use custom scripts to manage the database as I've seen in other forums but then again if I've no choice I'll have to go for the scripts .

    Thanks,


    --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    3661:20200901:113556.430 housekeeper [deleted 8112 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 records in 0.248109 sec, idle for 1 hour(s)]
    3661:20200901:123556.720 executing housekeeper
    3661:20200901:123557.057 housekeeper [deleted 8109 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 records in 0.337510 sec, idle for 1 hour(s)]
    3661:20200901:133557.343 executing housekeeper
    3661:20200901:133557.683 housekeeper [deleted 8113 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 records in 0.339456 sec, idle for 1 hour(s)]
    3661:20200901:143557.976 executing housekeeper
    3661:20200901:143558.290 housekeeper [deleted 8109 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 records in 0.313366 sec, idle for 1 hour(s)]
    3661:20200901:153558.571 executing housekeeper
    3661:20200901:153558.820 housekeeper [deleted 8112 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 records in 0.248169 sec, idle for 1 hour(s)]
    3661:20200901:163559.106 executing housekeeper
    3661:20200901:163559.456 housekeeper [deleted 8111 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 records in 0.350234 sec, idle for 1 hour(s)]
    3661:20200901:173559.749 executing housekeeper
    3661:20200901:173600.000 housekeeper [deleted 8109 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 records in 0.250889 sec, idle for 1 hour(s)]
    3667:20200901:182446.342 [Z3005] query failed: [1114] The table 'history_uint' is full [insert into history_uint (itemid,clock,ns,value) values (39996,1598984685,422702904,0),(40019,1598984685,4 22702904,2),(39939,1598984685,422702904,1),(40035, 1598984685,422702904,632),(39994,1598984685,422702 904,0),(39997,1598984685,422702904,0),(40030,15989 84685,422702904,0),(40009,1598984685,422702904,0), (40025,1598984685,422702904,0),(40022,1598984685,4 22702904,1),(40002,1598984685,422702904,0),(40024, 1598984685,422702904,0),(40017,1598984685,42270290 4,1),(34805,1598984685,422702904,0),(40011,1598984 685,422702904,576),(40028,1598984685,422702904,0), (40032,1598984685,422702904,0),(40041,1598984685,4 22702904,8),(40034,1598984685,422702904,3392),(399 50,1598984685,422702904,1),(39972,1598984685,42270 2904,1),(40027,1598984685,422702904,0),(40038,1598 984685,422702904,0),(39949,1598984685,422702904,1) ,(40023,1598984685,422702904,0),(40000,1598984685, 422702904,0),(40042,1598984685,422702904,632),(400 10,1598984685,422702904,0),(40039,1598984685,42270 2904,0),(40006,1598984685,422702904,0),(40013,1598 984685,422702904,1),(40029,1598984685,422702904,0) ,(34809,1598984685,422702904,2),(40037,1598984685, 422702904,0),(34804,1598984685,422702904,0),(39968 ,1598984685,422702904,1),(39944,1598984685,4227029 04,1),(40003,1598984685,422702904,49904),(39965,15 98984685,422702904,1),(40014,1598984685,422702904, 1),(39998,1598984685,422702904,0),(39966,159898468 5,422702904,2),(33589,1598984685,422702904,7737414 ),(39961,1598984685,422702904,954316),(40036,15989 84685,422702904,0),(35403,1598984685,422702904,2), (40015,1598984685,422702904,1),(35405,1598984685,4 22702904,2),(39999,1598984685,422702904,0),(39946, 1598984685,422702904,1),(39971,1598984685,42270290 4,1),(40005,1598984685,422702904,179136),(39963,15 98984685,422702904,1),(34808,1598984685,422702904, 2),(39943,1598984685,422702904,1),(39945,159898468 5,422702904,1),(39942,1598984685,422702904,1),(400 31,1598984685,422702904,0),(40004,1598984685,42270 2904,317064),(35404,1598984685,422702904,2),(40016 ,1598984685,422702904,2),(40007,1598984685,4227029 04,0),(39948,1598984685,422702904,1),(40021,159898 4685,422702904,1),(39969,1598984685,422702904,2),( 39947,1598984685,422702904,1),(39970,1598984685,42 2702904,2),(34806,1598984685,422702904,118),(33553 ,1598984685,422702904,7737414),(39967,1598984685,4 22702904,1),(40008,1598984685,422702904,178984),(3 9993,1598984685,422702904,0),(39995,1598984685,422 702904,0),(39962,1598984685,422702904,2435817),(40 020,1598984685,422702904,2),(39964,1598984685,4227 02904,1),(40001,1598984685,422702904,0),(33587,159 8984685,422702904,2),(40012,1598984685,422702904,3 58120),(34807,1598984685,422702904,118),(33586,159 8984685,422702904,2),(40033,1598984685,422702904,4 9904),(39941,1598984685,422702904,1),(39940,159898 4685,422702904,1),(33585,1598984685,422702904,2),( 40018,1598984685,422702904,1),(40040,1598984685,42 2702904,0),(40026,1598984685,422702904,0),(37605,1 598984685,422993941,11942912),(30713,1598984685,42 3753828,600559616),(30712,1598984685,423753828,395 177984),(30714,1598984685,423753828,31566270464),( 30710,1598984685,423753828,14388506624),(30716,159 8984685,423753828,17177763840),(30711,1598984685,4 23753828,17177763840),(30715,1598984685,423753828, 15797256192),(31444,1598984685,424315897,313520496 64),(31446,1598984685,424315897,17177763840),(3144 1,1598984685,424315897,17177763840),(31445,1598984 685,424315897,15797256192),(31442,1598984685,42431 5897,401645568),(31443,1598984685,424315897,718655 488),(31440,1598984685,424315897,14174285824),(321 74,1598984685,424869859,30923153408),(32171,159898 4685,424869859,17177763840),(32172,1598984685,4248 69859,350109696),(32176,1598984685,424869859,17177 763840),(32170,1598984685,424869859,13745389568),( 32175,1598984685,424869859,15797256192),(32173,159 8984685,424869859,1000833024),(32881,1598984685,42 5434461,14580178944),(32882,1598984685,425434461,1 7177763840),(32885,1598984685,425434461,3175794278 4),(32883,1598984685,425434461,372621312),(32887,1 598984685,425434461,17177763840),(32886,1598984685 ,425434461,15797256192),(32884,1598984685,42543446 1,506769408),(33613,1598984685,947895134,372060160 ),(33615,1598984685,947895134,32131006464),(33620, 1598984685,947895134,22),(33617,1598984685,9478951 34,17177763840),(33616,1598984685,947895134,157972 56192),(33614,1598984685,947895134,301719552),(336 12,1598984685,947895134,17177763840),(33611,159898 4685,947895134,14953242624),(40105,1598984685,9481 42005,0),(34818,1598984685,948142005,2),(40128,159 8984685,948142005,0),(40122,1598984685,948142005,1 ),(40071,1598984685,948142005,1),(40145,1598984685 ,948142005,16),(40117,1598984685,948142005,1),(400 98,1598984685,948142005,0),(40047,1598984685,94814 2005,1),(40124,1598984685,948142005,2),(40073,1598 984685,948142005,2),(40051,1598984685,948142005,1) ,(40068,1598984685,948142005,1),(40110,1598984685, 948142005,0),(34815,1598984685,948142005,112),(336 58,1598984685,948142005,2),(40052,1598984685,94814 2005,1),(40106,1598984685,948142005,0),(40140,1598 984685,948142005,0),(40123,1598984685,948142005,2) ,(40049,1598984685,948142005,1),(40132,1598984685, 948142005,0),(40131,1598984685,948142005,0),(40101 ,1598984685,948142005,0),(40130,1598984685,9481420 05,0),(40111,1598984685,948142005,0),(40127,159898 4685,948142005,0),(40118,1598984685,948142005,1),( 33662,1598984685,948142005,7737421),(40104,1598984 685,948142005,0),(40103,1598984685,948142005,0),(4 0099,1598984685,948142005,0),(35406,1598984685,948 142005,2),(40146,1598984685,948142005,616),(40116, 1598984685,948142005,352944),(40067,1598984685,948 142005,1),(40113,1598984685,948142005,0),(40143,15 98984685,948142005,0),(35407,1598984685,948142005, 2),(40120,1598984685,948142005,2),(33660,159898468 5,948142005,2),(40125,1598984685,948142005,1),(401 14,1598984685,948142005,0),(40044,1598984685,94814 2005,1),(34817,1598984685,948142005,2),(40136,1598 984685,948142005,0),(40065,1598984685,948142005,95 8019),(40142,1598984685,948142005,0),(40046,159898 4685,948142005,1),(40072,1598984685,948142005,1),( 40112,1598984685,948142005,176384),(40121,15989846 85,948142005,1),(40126,1598984685,948142005,1),(40 139,1598984685,948142005,616),(34816,1598984685,94 8142005,112),(40141,1598984685,948142005,0),(40050 ,1598984685,948142005,1),(40097,1598984685,9481420 05,0),(40135,1598984685,948142005,0),(40074,159898 4685,948142005,2),(40137,1598984685,948142005,4948 0),(40075,1598984685,948142005,1),(40053,159898468 5,948142005,1),(40043,1598984685,948142005,1),(348 13,1598984685,948142005,1444),(40119,1598984685,94 8142005,1),(40133,1598984685,948142005,0),(40115,1 598984685,948142005,576),(40107,1598984685,9481420 05,49480),(33626,1598984685,948142005,7737421),(40 054,1598984685,948142005,1),(40070,1598984685,9481 42005,2),(40144,1598984685,948142005,0),(40102,159 8984685,948142005,0),(40066,1598984685,948142005,2 665055),(40048,1598984685,948142005,1),(40108,1598 984685,948142005,314808),(40129,1598984685,9481420 05,0),(40069,1598984685,948142005,1),(40045,159898 4685,948142005,1),(40109,1598984685,948142005,1765 52),(33659,1598984685,948142005,2),(40100,15989846 85,948142005,0),(40134,1598984685,948142005,0),(35 408,1598984685,948142005,2),(40138,1598984685,9481 42005,3416),(34814,1598984685,948142005,1444),(400 76,1598984685,948142005,1),(34346,1598984685,94853 1999,15797256192),(34344,1598984685,948531999,1089 82272),(34342,1598984685,948531999,17177763840),(3 4341,1598984685,948531999,15007961088),(34345,1598 984685,948531999,32185724928),(34343,1598984685,94 8531999,459579392),(34347,1598984685,948531999,171 77763840);
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    I also observed the same behavior in the mysqld log file as shown below :

    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    2020-07-06T04:43:28.300010Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.28) starting as process 8440 ...
    2020-07-06T04:43:28.302356Z 0 [Note] InnoDB: PUNCH HOLE support available
    2020-07-06T04:43:28.302373Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2020-07-06T04:43:28.302378Z 0 [Note] InnoDB: Uses event mutexes
    2020-07-06T04:43:28.302385Z 0 [Note] InnoDB: GCC builtin __sync_synchronize() is used for memory barrier
    2020-07-06T04:43:28.302389Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
    2020-07-06T04:43:28.302395Z 0 [Note] InnoDB: Using Linux native AIO
    2020-07-06T04:43:28.302608Z 0 [Note] InnoDB: Number of pools: 1
    2020-07-06T04:43:28.302671Z 0 [Note] InnoDB: Not using CPU crc32 instructions
    2020-07-06T04:43:28.306612Z 0 [Note] InnoDB: Initializing buffer pool, total size = 1.95313G, instances = 12, chunk size = 166.667M
    2020-07-06T04:43:28.406813Z 0 [Note] InnoDB: Completed initialization of buffer pool
    2020-07-06T04:43:28.415522Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
    2020-07-06T04:43:28.446121Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
    2020-07-06T04:43:28.631261Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
    2020-07-06T04:43:28.631334Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
    2020-07-06T04:43:28.861480Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
    2020-07-06T04:43:28.862285Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
    2020-07-06T04:43:28.862297Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
    2020-07-06T04:43:28.862453Z 0 [Note] InnoDB: Waiting for purge to start
    2020-07-06T04:43:28.919492Z 0 [Note] InnoDB: 5.7.28 started; log sequence number 143765791
    2020-07-06T04:43:28.919661Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
    2020-07-06T04:43:28.919803Z 0 [Note] Plugin 'FEDERATED' is disabled.
    2020-07-06T04:43:28.967490Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
    2020-07-06T04:43:28.967521Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
    2020-07-06T04:43:28.968159Z 0 [Warning] CA certificate ca.pem is self signed.
    2020-07-06T04:43:28.968206Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
    2020-07-06T04:43:28.968705Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
    2020-07-06T04:43:28.968761Z 0 [Note] IPv6 is available.
    2020-07-06T04:43:28.968772Z 0 [Note] - '::' resolves to '::';
    2020-07-06T04:43:28.968802Z 0 [Note] Server socket created on IP: '::'.
    2020-07-06T04:43:29.029027Z 0 [Note] Event Scheduler: Loaded 0 events
    2020-07-06T04:43:29.029217Z 0 [Note] /usr/sbin/mysqld: ready for connections.
    Version: '5.7.28' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
    2020-07-06T04:43:29.314108Z 0 [Note] InnoDB: Buffer pool(s) load completed at 200706 4:43:29
    2020-07-06T04:43:29.968136Z 2 [Note] Access denied for user 'UNKNOWN_MYSQL_USER'@'localhost' (using password: NO)
    2020-09-01T18:24:46.330374Z 9 [ERROR] /usr/sbin/mysqld: The table 'history_uint' is full
    2020-09-01T18:40:17.208546Z 7 [ERROR] /usr/sbin/mysqld: The table 'history_uint' is full
    2020-09-01T18:53:13.767678Z 11 [ERROR] /usr/sbin/mysqld: The table 'history_uint' is full
    2020-09-01T19:03:33.148842Z 7 [ERROR] /usr/sbin/mysqld: The table 'history_uint' is full
    2020-09-01T19:13:54.234994Z 5 [ERROR] /usr/sbin/mysqld: The table 'history_uint' is full
    2020-09-01T19:22:17.155963Z 5 [ERROR] /usr/sbin/mysqld: The table 'history_uint' is full
    2020-09-01T19:29:30.987212Z 9 [ERROR] /usr/sbin/mysqld: The table 'history_uint' is full
    2020-09-01T19:35:55.779780Z 7 [ERROR] /usr/sbin/mysqld: The table 'history_uint' is full
    2020-09-01T19:47:37.044546Z 9 [ERROR] /usr/sbin/mysqld: The table 'history_uint' is full
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Attached Files
  • tim.mooney
    Senior Member
    • Dec 2012
    • 1427

    #2
    When you see "the table <whatever> is full" from MySQL (or MariaDB or Percona), the first two places you should look are
    1. is the volume where the database files reside full?
    2. since this is a 32 bit OS (I wouldn't recommend that, at this point) you probably need to determine if the file containing the history_uint table has reached the maximum supported size that MySQL allows on 32 bit CentOS. Since you are using "innodb_file_per_table" (that's good!), how big is the history_uint.ibd file?

    Comment


    • csid2174
      csid2174 commented
      Editing a comment
      Thank you Tim for your very useful feedback. I should actually install mysql in a larger partition, I don't know what I was thinking. I think I was having amateur moment and  it didn't quite occur to me whilst installing the packages and since you've mentioned this, I am having a light bulb moment since my /root partition is only configured to be 22GB and the damn history_uint.ibd is 30GB. Like I said, I should have been more careful, as a network guy I didn't give too much thought on these constraints while installing Zabbix 4.4, but thanks to you, I'll go to my office at some stage and do a re-install with smarter allocation of space and do the install in the system partition which is 260GB in size.

      Due to this pandemic I am unable to fix this issue immediately cause of travel restriction ,but if you don't mind can I give you my feedback in three weeks time. I am pretty certain that harddisk space is my problem as you've correctly identified and I'll rectify this issue soon-ish.

      Thanks again for your useful feedback and I really appreciate it. Also I am surprised at my ineptness since I've been managing Zabbix 2.4.5 for last 5 years and I just remembered that I kind of had the same issue in the beginning where I moved the mysql install to the larger partition after the install and used sym-links for Zabbix to access mysql directory structure. Anyhow thanks again for the help.
  • csid2174
    Junior Member
    • Sep 2020
    • 4

    #3
    Thank you Tim for your very useful feedback. I should actually install mysql in a larger partition, I don't know what I was thinking. I think I was having amateur moment and it didn't quite occur to me whilst installing the packages and since you've mentioned this, I am having a light bulb moment since my /root partition is only configured to be 22GB and the damn history_uint.ibd is 30GB. Like I said, I should have been more careful, as a network guy I really didn't give too much thought on these constraints while installing Zabbix 4.4, but thanks to you, I'll go to my office at some stage and do a re-install with smarter allocation of space and do the install in the system partition which is 260GB in size.

    Due to this pandemic I am unable to fix this issue immediately cause of travel restriction ,but if you don't mind can I give you my feedback in three weeks time. I am pretty certain that harddisk space is my problem as you've correctly identified and I'll rectify this issue soon-ish.

    Thanks again for your useful feedback and I really appreciate it. Also I am surprised at my ineptness since I've been managing Zabbix 2.4.5 for last 5 years and I just remembered that I kind of had the same issue in the beginning where I moved the mysql install to the larger partition after the install and used sym-links for Zabbix to access mysql directory structure. Anyhow thanks again for the help.

    Comment

    • tim.mooney
      Senior Member
      • Dec 2012
      • 1427

      #4
      Glad to help.

      If you can remotely access the system (I know that not every organization allows that kind of access), you may be able to do most of the fixing remotely, depending on a few factors. For example, if you're using LVM on the system and there is unconfigured free space in an appropriate LVM volume group, you may just be able to grow a volume to make more space available to it. That generally doesn't work with the / partition (usually you don't put that under LVM), but depending on how you want to arrange things, it still may be possible to grow space elsewhere and then move the database onto the new volume.

      PS: if you do end up reinstalling, remember that CentOS 6.x is end-of-life in just a couple months. If your company allows it, using 64 bit CentOS 7 or CentOS 8 puts you in better shape to be able to upgrade to recent Zabbix versions.

      Comment

      Working...