How to get Munin default with Zabbix??? First contact experience report.

[xabbiz]

As this is my first deeper dive into Zabbix 4 I would like to give the opportunity to learn from what an old school admin feels using zabbix for the first time. This impression will of course change the more I get into the system, so sometimes it is valuable to preserve that first contact experience for people who are deep inside the production process and completely blind to that perspective. I am not trying to be submissive on purpose or because I have fun with that - it is quite the opposite, I really do not like to play that role, but as someone wearing the developer hat, too, I understand how incredibly useful it can be when experienced people are writing honestly about what they think when confronted with your product for the first time. In that spirit please try not to feel offended - I am just honest!

OK, I now spend far too many hours installing and configuring Zabbix and adding a few hosts with zabbix-agent - everything seems to be up and running now. But I am not happy.

First: the installation process delivered an error - with the well-known Ondrej Sury PHP packages Zabbix seems to be confused about PHP versions and which packages it really needs. This is quite telling for somebody with some years of php knowledge - developers must be kind of far out not to know Ondrejs packages and not to test against them - this is an important indicator that also triggered me to take a really quick look into the sources, and, hmm, well, I am not completely baffled but I see there some very old php practices and some kind of refusal of modern php ideas... hmmm, well, this is NOT a code review and it really does not say anything about the code quality, however in 2018 you would expect some more openness to the general direction the PHP ecosystem is going. However, I know that sometimes it is good to stay with what you know and not change all the things, so I just notice that the code might be "interesting" and go ahead - but there is some smell. I make a notice for myself to do some research about Zabbix security history to learn more about where the devs are coming from.

Nevertheless after that little distraction there comes the next roadblock: NO NGINX CONFIG out of the box. WTF??? Moon? Dark side of that? Come on... o my god, well, ok, it is not that hard to configure a nginx vhost for some php but it eats time and this is something the devs could do in ten minutes - a really low hanging fruit and while the technical implementation is not a big problem for me it makes me wonder about the mindset of these php devs again - how on earth can you deliver an "enterprise grade" software package in 2018 without nginx support out of the box? This idea looks absurd. This again feels like some very, very remote and isolated person should really take up a little bit on current practices and tools. It also makes me remember that the zabbix debian packages wanted to install the mysql package, which is extremely deprecated... well, but ok, a quick look into the sources make my dark suspicion vanish... still, PDO does not seem to be used, I see "mysqli" statements, hmm, well, this is not a problem, but the code-smell is getting stronger. Well, it´s just a monitoring tool, I should not be too strict here with my usual requirements for php software. Meanwhile I add another "!" to the notice to myself to do some research about zabbix security.

Another small distraction: the php requirements. Wait, 16M post_max_size??? WTF is zabbix frontend sending over the wire that it needs that much data? Hmmm. Makes me wonder again. But, ok. Go ahead. Setup-screen.

After configuring NGINX I understand that the manual installation steps (setup database and import sql dump) are not optional, well, ok, no big thing, however, I already feel like in the php world of 10 years ago, so no problem - most important thing is that I can now start working on it already.

Ah, no, the setup screen can not write its configuration file - well, I am not surprised at this point, of course you do not want to write to /usr/share/zabbix and also not to /etc from a web app, to say the truth I am quite happy that this web app can not write to the filesystem - I am trying to find some information in the manual and wiki what other files Zabbix wants to write to but can not find any valuable info about such important (from the view of an enterprise admin) things and I do not feel like grepping the sources for all file writing things, I just want to get the monitoring setup done. Here I have to think about Munin for the first time. Just for fun I install it and have the graphs I want to see in a browser window after 10 seconds. Yes, I know Zabbix will give me much more features, so I just go ahead. Another idea just quickly crosses my brain: did that not-able-to-write error just happened because I setup nginx the way I think it makes sense? In other words: would a default install of Zabbix really write to /usr/share/zabbix or to /etc ??? I can not imagine that anybody would publish such a security nightmare in 2018, but I make a notice to myself to check the "appliance" and other zabbix installations at some future day. Also one more "!" for the security research item on the Zabbix todo list...

SO, then, setup screen done, login - wow - empty! Royal emptiness, one could say. Another low hanging fruit missed - if I was the dev of a monitoring and graph tool I would build a default screen for first time visitors that would just blow them away, showing off all the features on one big page with fantastic graphs that will make all the admins cry. I mean, Zabbix is already running on at least one server and there is plenty of example data that can be collected from localhost, but hey, if you are really forcing people to do a manual sql import - why not just PUT SOME EXAMPLES THERE so people can just copy-paste-clone their setup in ten minutes. Instead - emptiness. Again I wonder about the devs - are they lonely? Are they not interested in presenting the capabilities of their product? Why are they missing out on this fantastic possibility to present everything - nobody could do a better blow-me-away setup of some example graphs than the developers themselves - they totally miss out on that and instead leave the totally inexperienced user to click around in a mostly empty gui - if you think about it, you could just put a message on the login screen: "Please do not use our software right now. There is nothing to see here, you first have to study manuals for several days and then it might make sense to come back. By default we show you nothing. If you want to see graphs of your server quickly - go and install Munin!". Well, ok, yes, I know, Zabbix is a powerhouse of features, I will have patience. But why don´t they show me.

Ok, yes, there is the "Zabbix Server" item that at least allows to learn a little bit by cloning things - but I feel now like on the beginning of a very, very tedious and time-burning task, like the first 20 meters of swimming to cross the ocean. SIgh. It is incredible tedious to get at least some nice default data and graph views - I would like to see at least something that is similar to what I get with Munin by default. I know that Zabbix is a powerful tool and so I spend nearly the whole night with Zabbix and what I can see is: empty screens. Sooo tedious.

After breakfast I am still not happy. But most important: Munin still beats what I see in Zabbix. I did read some stuff in the manual and tried to construct some graphs via the web ui, but this is such a boring and stupid task - I mean this is not the first Linux server on earth, why the hell do I need to DO THAT?

I understand that Zabbix seems to deliver quite some powerful toolset, however the first contact experience is incredibly frustrating. Instead of spending so much effort to get to some reasonable default views and reports that are totally obvious everybody needs I would like to spend time on learning how to solve the SPECIAL monitoring tasks that are on my list - these are the reasons I was looking for such a tool, the non-generic things that will need some non-generic code are the things where I have a time budget for - but of course I am still expecting an enterprise level monitoring tool to spit out most of the expected data and obvious alarms (100% values on CPU, harddisks, network saturation and stuff like that) by default. I feel depressed. It is a little bit like spending a night with the wrong girl. (OK, i am just adding some drama to make it more fun to read.)

Also the manual... well. I mean, it can not be said that it is bad. But, please, why do developers stop with so moderate demands on themselves? Dokuwiki? Really? Please, take some time off and make a little journey to python-land, look at the documentation of Django or the Pyramid Framework or look at the Java Spring Framework documentation - it is not only about the amount of words, it is about the depth, the dedication, the devotion to the task to make the user not only understand, but love your product. This is missing. Zabbix docs look like written by some person who wants to spend time on something else - and you feel that reading these do-not-waste-time-i-am-annoyed style abbreviations, you feel the impatience. Also: why not simply be friendly to your users - write like you would write to your friend or mother - this is a very important attitude for technical text, because it is really boring to read that, so all the little things become very important.

Well, ok. Now I have my Zabbix instance running and I am not sure. If it will take weeks to configure what I need I would be much better off with Prometheus and Grafana, which give me a very flexible toolkit. But I have also lots of other things to waste time with - so my motivation for installing Zabbix was to GET TO THE RESULT QUICKLY - but now I am wasting time to get to some reasonable and expected defaults. Not the edge cases. And the results are modest.

I still have some hope and that is why I am asking for help - the manual is not enough. I would like to read / see a Tutorial or maybe a virtual machine with some pre-configured defaults, something that makes it easy to clone the way to your own environment. I would also take youtube videos. Anything. I am desperate. Is there anybody willing to give me some hints how to get to Munin-level display or to some Prometheus / Grafana lookalike default views quickly without spending another night on that? Thanks!

My recommendation for the devs: please much better docs - and please do not miss that opportunity to show off features - Zabbix should make every feature and graph visible with example data by default, that not only is a great presentation highlight moment, but also has incredible value for your users that might clone these examples quickly. Get rid of the empty screens on a zabbix install. It is easy to implement one button: "delete all example data", but it is incredibly hard to fight your way through an empty gui. Show what Zabbix can do and deliver all the eye candy by default! It is easy to delete that stuff later on. Think more about your users and how to make their life more easy. Try to develop real dedication and attention, that will go into all the details. You put a lot of love into the zabbix.com website - but that level has to be sustained through all areas of the product, because users will feel the contrast very strongly.

Thanks for your attention.

Any Zabbix 4 Tutorials are very welcome and will be greatly appreciated! Thanks!

[ingus.vilnis]

Hi,

This was some interesting reading. And very much true if you have encountered Zabbix for the first time. I also do hope that your lines will be read by people who make decisions on the development of the product.

Most of your statements were true, some not so but overall the idea is clear. Zabbix has a very steep learning curve. By no means it is a Next Next Finish shiny thingy. In many ways regarding the setup it relies on the user to be very experienced in command line, many custom things have to be implemented differently from the documentation and once you are done with the install you get a sample screen with one host and some graphs.

But Zabbix is not built to be easy. Yes, it has a very steep learning curve. It takes I would say years to reach a decent level of expertise. Take the server operating systems for example. So I need a server for whatever purposes. say a website. What are my options? Linux? Windows? others? Ok, I know the cool admins use Linux so I install it. And what do I get after that? Black screen with root@localhost # >_ And? Where is the beauty? Where are the bells and whistles? Imma go with Windows instead. It has a decent desktop, I can click around, it has tiles and a start menu etc.... I hope you got this example right. Zabbix is like Linux, you can do incredible things with that but at first it is black screen and you have to put some efforts to make it work the way you need. For users which just want to spin up a VM not caring about OS and DB setup that just works there is the Appliance to be downloaded. Or Docker containers if you fancy.

The packages, dependencies, PHP, Nginx... yes, I see what you mean there and agree. The problem is that from Zabbix perspective one can't know what repos will be on that server. Will there be access to the web to download them? The safest thing to do is to rely on the packages of the particular OS, and for CentOS that indeed means old php 5.4 and Apache. Because nginx and newer php is not in default packages for CentOS / RHEL. Same for other OS. Want to have something newer? Sure, but you are on your own then.

And the docs. One can't deny the efforts put in documenting that whole of functionality Zabbix has. Over the time it has also improved a lot, practical examples added. And really if you would make it even more friendly I don't know how bloated that would become. This is by far not the worst documentation of a product.

Speaking about tutorials the Internet is full with stuff about Zabbix. Also good books are available. And you may opt for a Zabbix training wherever that is happening. It will sort many things out for you within that week as learning on your own takes lots of time and sometimes you just need someone to simply explain you things.

So see you around. If you will successfully get over some specifics of the tool then you will have some great time with it. It will take time, not always things will work intuitively, you will encounter some strange concepts but overall - it is not such a bad product despite the first impressions.

[Alexei]

Thank you for sharing your first contact experience on a such detailed level. There are many things that can be improved indeed, shared with our dev team.