Application monitoring that takes early detection and early restoration to the limits, supporting the quality of the KDDI Cloud Platform Service (KCPS) cloud service provided by KDDI, and its other side
KDDI needed a flexible monitoring flow to monitor and react to different circumstances, action and possible problems of customers.
Improve work efficiency, performance and down-time.
A mature enterprise-level monitoring solution, which is able to integrate with CloudStack and other systems effortlessly.
Automatic linking for application monitoring of approximately 20 thousands of hosts efficiently.
Approximately 40 duplicated Zabbix servers were used to provide monitoring of several hundred thousand items.
Implemented a structure where Zabbix and CloudPlatform were automatically linked.
Integration with CloudStack and related systems.
Easily customizable front-end graphing capabilities.
Improvement efficiency and performance using advanced features.
Fulfilling Support framework.
KDDI is providing a high-quality, business-oriented cloud service known as the “KDDI Cloud Platform Service (KCPS)”, which sets forth a “Quality Cloud” based on the Citrix CloudPlatform. The carrier-grade quality of this service is supported by unceasing application monitoring using Zabbix. As a company that is using Zabbix and implementing early detection/early restoration, we asked KDDI about these points.
The KDDI Cloud Platform Service (KCPS) is an Infrastructure as a Service (IaaS) that KDDI has been providing to businesses since July 2012. Based on the Citrix CloudPlatform cloud software infrastructure, this is a high-quality service that reflects that various forms of know-how that KDDI has cultivated as a telecommunications carrier. As a service based on the Citrix CloudPlatform and highly rated for its quality, the KDDI Cloud Platform Service (KCPS) has grown to become among the largest of such services in Japan and is even leading the way on a global scale. (Current as of November 2014)
In response to the concerns of many Japanese companies regarding the security of using cloud services, security is ensured by providing not only for connection via the Internet, but also a standard service for connecting with an Intranet via a special closed network. Also, in addition to shared virtual servers, KDDI provides a proprietary service with exclusive use that places importance on privacy, delivering an environment that customers who are considering cloud utilization can use with peace of mind and security.
Also, while the SLA of the KDDI Cloud Platform Service (KCPS) is an operating ratio of 99.99%, it has for the past months continually exceeded that ratio, reaching an operating ratio of 99.9999%. The key point of this implementation is fine, in-depth application and monitoring. KDDI’s Mr. Takeshi Maehara (Leader at Development Section 1, the Cloud Service Planning , Platform Development Division) states: “We have not reached a level where absolutely no failures occur, but servers by their very nature as machines inevitably break down from time to time. With that in mind, the key is how quickly the server can be re-launched when a failure has occurred, and to minimize the length of time during which the server is out of order.”
*Operating ratio = Total number of operating hours in one month / (Total number of operating hours in one month + Total number of out-of-order hours in one month), not including maintenance time
The quality of the KDDI Cloud Platform Service (KCPS) is supported by “application monitoring”. Here, “Zabbix” plays an important role. In addition to the standard functions of CloudPlatform, a flexible monitoring flow reacting to different circumstances is being implemented through a combination of Zabbix and proprietary shell-script.
With the KDDI Cloud Platform Service (KCPS), approximately 40 Zabbix servers are used to provide monitoring of several hundred thousand items across infrastructure comprising approximately 20 thousand hosts. KDDI originally used Zabbix for its own internal system application, giving its operators familiarity with the operation of Zabbix, and it has been highly regarded for its flexibility, enabling free tuning and customization of monitoring items and action settings at the time of detection.
There is another major reason: the possibility of linking with CloudPlatform. As well as equipment that forms the basis of IaaS, KDDI also provides monitoring of the operating status of customers’ virtual servers when customers request this as an option. But on its own, says KDDI’s Mr. Masato Kato (Manager at Development Section 1, the Cloud Service Planning , Platform Development Division), “Even if the customer had personally restarted a virtual server for some reason, this would be deemed an abnormality and would set off an alert. That’s why we have implemented a structure where Zabbix and CloudPlatform are automatically linked so that no warning is issued when items have intentionally been stopped.” Automatic linking has also made it possible to carry out application monitoring of approximately 20 thousands of hosts efficiently, without having to check each host’s IP address or OS type individually.
This structure was realized thanks to a product from Intelligence Business Solutions (IBS), a Zabbix partner that provides installation-related consultancy and support services. Mr. Taketo Kosaka, of the OSS Technology Group at the IBS Platform Solutions Division, explains: “Zabbix uses functions that have been prepared from the outset, and has been able to pick out actions on the CloudPlatform side.” The fast-acting CloudPlatform will also continue to be upgraded with new versions.
One of the features of Zabbix that is highly regarded by KDDI is the ability to improve work efficiency by making templates for monitoring settings. Mr. Jun Endo, Assistant Manager at KDDI’s Development Section 1, Cloud Service Planning , Platform Development Division, says that “At present, Zabbix is used in the KDDI Cloud Platform Service (KCPS) for monitoring of customers’ servers and of KDDI’s servers and network equipment, and is even used for monitoring of service equipment outside of the KDDI Cloud Platform Service (KCPS), and this is being carried out efficiently by producing templates with IBS to suit the respective characteristics of each task.”
But this was no simple feat. For example, when Zabbix servers were first introduced, the number of monitoring items increased considerably, and disk capacity expanded significantly. This sometimes resulted in instability in the actual behavior of the monitoring servers. Mr. Endo adds that based on this experience “IBS now arranges parameter and maintains a state in which monitoring can be carried out smoothly by means of performance tuning”.
Looking back, Mr. Kato recalls: “In fact, there were several problems at the beginning with threshold settings, etc.” Even the slightest slippage can lead to major problems in a cloud environment with thousands of virtual servers and tens of thousands of items.
Accordingly, KDDI and IBS have produced tools to check that settings are being implemented as planned, from the generation of settings through to their actual operation, and are endeavoring to further optimize application. Mr. Kato explains: “With conventional systems, there are limits to the checking of cloud environment configuration with human eyes. Taking that into account, we are carrying out precise systematization.” Mr. Maehara concurs: “Clouds have a tremendous sense of speed, and with the KDDI Cloud Platform Service (KCPS) too there is expansion with unbelievable impetus. It is impossible to keep up without automation.”
As a system continues to be applied, the number of individual parts requiring monitoring increases, and common template parts become relatively few, but this has also been dealt with by redrawing templates in an arrangement together with IBS.
Mr. Naoyuki Kajiwara, a Senior Engineer of the OSS Technology Group at the IBS Platform Solutions Division, recalls that occasionally “We would receive very in-depth questions that were impossible to reply to immediately.” Mr. Naoya Tsuchiyama, also of the OSS Technology Group at the IBS Platform Solutions Division, states: “Rather than simply replying with product specifications, we aim to provide the best possible answer by surmising what the customer wants to achieve when carrying out monitoring.”
Together with reviewing design from the ground up in order to achieve further quality enhancement, the aim is to produce a world in which customers can see the quality of the cloud and use it with peace of mind.
Mr. Maehara emphasizes the importance of monitoring in supporting quality: “When we use the word ‘quality’ in this field, it also includes detailed monitoring and quick submission of reports. In order to provide even greater levels of quality, it is important to recognize problems as quickly as possible and carry out restoration as quickly as possible. Application monitoring and Zabbix have extremely important roles to play in achieving this.”
Also, Mr. Endo says, “We have great hopes for the structure that has been added to Zabbix so that customers in a multi-tenant environment can configure and check monitoring items on their own.” We look forward to the further growth of the KDDI Cloud Platform Service (KCPS), which has used such functions.
※Official name for ‘CloudPlatform' is 'Citrix CloudPlatform powered by Apache CloudStack’.Citrix, CloudStack and CloudPlatform are trademarks of Citrix Systems, Inc. and/or one or more of its subsidiaries, and may be registered in the U.S. Patent and Trademark Office and in other countries.
KDDI, as a comprehensive communications company offering both fixed-line and mobile communications services, strives to be a leading company during changing times.
For individual customers, KDDI offers its mobile communications (au mobile phone) and fixed-line communications (broadband Internet/telephone) services under the brand name "au", helping to realize new seamless communications environments. And for business clients, KDDI provides all services in the ICT (Information and Communication Technology) realm, from FMC (Fixed Mobile Convergence) networks to data centers, applications, and security strategies, to help clients strengthen their businesses.
Its Cloud Service Planning provides Cloud Services for business clients. The features of KDDI cloud are provided with the concept ’One-stop for cloud, network and mobile’,’ freely and through the eyes of customers’ and ’ QualityCloud’.
Fill out this Questionnaire or contact our Marketing Team for further assistance.
Zabbix is a professionally developed open-source software with no limits or hidden costs