KDDI Case Study Interview

Application monitoring that takes early detection and early restoration to the limits, supporting the quality of the KDDI Cloud Platform Service (KCPS) cloud service provided by KDDI, and its other side

KDDI logo
 

OBJECTIVE

KDDI needed a flexible monitoring flow to monitor and react to different circumstances, action and possible problems of customers.

Improve work efficiency, performance and down-time.

REQUIREMENTS

A mature enterprise-level monitoring solution, which is able to integrate with CloudStack and other systems effortlessly.

Automatic linking for application monitoring of approximately 20 thousands of hosts efficiently.

APPROACH

approximately 40 duplicated Zabbix servers were used to provide monitoring of several hundred thousand items.

Implemented a structure where Zabbix and CloudPlatform were automatically linked.

OUTCOME

Integration with CloudStack and related systems.

Easily customizable front-end graphing capabilities.

Improvement efficiency and performance using advanced features.

Fulfilling Support framework.


KDDI is providing a high-quality, business-oriented cloud service known as the “KDDI Cloud Platform Service (KCPS)”, which sets forth a “Quality Cloud” based on the Citrix CloudPlatform. The carrier-grade quality of this service is supported by unceasing application monitoring using Zabbix. As a company that is using Zabbix and implementing early detection/early restoration, we asked KDDI about these points.

Mr. Takeshi Maehara
Mr. Takeshi Maehara,
Leader
Development Section 1,
Cloud Services
Development Department,
Platform Development
Divison at KDDI

Mr. Masato Kato
Mr. Masato Kato
Manager
Development Section 1,
Cloud Services
Development Department,
Platform Development
Divison at KDDI

Mr. Jun Endo
Mr. Jun Endo
Assistant Manager
Cloud Services
Development Department,
Platform Development
Divison at KDDI

KDDI Cloud Platform Service (KCPS): a Quality Cloud boasting an operating ratio of 99.9999% (actual results during period of July to September)

The KDDI Cloud Platform Service (KCPS) is an Infrastructure as a Service (IaaS) that KDDI has been providing to businesses since July 2012. Based on the Citrix CloudPlatform cloud software infrastructure, this is a high-quality service that reflects that various forms of know-how that KDDI has cultivated as a telecommunications carrier. As a service based on the Citrix CloudPlatform and highly rated for its quality, the KDDI Cloud Platform Service (KCPS) has grown to become among the largest of such services in Japan and is even leading the way on a global scale. (Current as of November 2014)

In response to the concerns of many Japanese companies regarding the security of using cloud services, security is ensured by providing not only for connection via the Internet, but also a standard service for connecting with an Intranet via a special closed network. Also, in addition to shared virtual servers, KDDI provides a proprietary service with exclusive use that places importance on privacy, delivering an environment that customers who are considering cloud utilization can use with peace of mind and security.

Also, while the SLA of the KDDI Cloud Platform Service (KCPS) is an operating ratio of 99.99%, it has for the past months continually exceeded that ratio, reaching an operating ratio of 99.9999%. The key point of this implementation is fine, in-depth application and monitoring. KDDI’s Mr. Takeshi Maehara (Leader at Development Section 1, the Cloud Service Planning , Platform Development Divison) states: “We have not reached a level where absolutely no failures occur, but servers by their very nature as machines inevitably break down from time to time. With that in mind, the key is how quickly the server can be re-launched when a failure has occurred, and to minimize the length of time during which the server is out of order.”

*Operating ratio = Total number of operating hours in one month / (Total number of operating hours in one month + Total number of out-of-order hours in one month), not including maintenance time

Implementation of monitoring by linking Citrix CloudPlatform and Zabbix

The quality of the KDDI Cloud Platform Service (KCPS) is supported by “application monitoring”. Here, “Zabbix” plays an important role. In addition to the standard functions of CloudPlatform, a flexible monitoring flow reacting to different circumstances is being implemented through a combination of Zabbix and proprietary shell-script.

With the KDDI Cloud Platform Service (KCPS), approximately 40 Zabbix servers are used to provide monitoring of several hundred thousand items across infrastructure comprising approximately 20 thousand hosts. KDDI originally used Zabbix for its own internal system application, giving its operators familiarity with the operation of Zabbix, and it has been highly regarded for its flexibility, enabling free tuning and customization of monitoring items and action settings at the time of detection.

There is another major reason: the possibility of linking with CloudPlatform. As well as equipment that forms the basis of IaaS, KDDI also provides monitoring of the operating status of customers’ virtual servers when customers request this as an option. But on its own, says KDDI’s Mr. Masato Kato (Manager at Development Section 1, the Cloud Service Planning , Platform Development Divison), “Even if the customer had personally restarted a virtual server for some reason, this would be deemed an abnormality and would set off an alert. That’s why we have implemented a structure where Zabbix and CloudPlatform are automatically linked so that no warning is issued when items have intentionally been stopped.” Automatic linking has also made it possible to carry out application monitoring of approximately 20 thousands of hosts efficiently, without having to check each host’s IP address or OS type individually.

This structure was realized thanks to a product from Intelligence Business Solutions (IBS), a Zabbix partner that provides installation-related consultancy and support services. Mr. Taketo Kosaka, of the OSS Technology Group at the IBS Platform Solutions Division, explains: “Zabbix uses functions that have been prepared from the outset, and has been able to pick out actions on the CloudPlatform side.” The fast-acting CloudPlatform will also continue to be upgraded with new versions.

Mr. Taketo Kosaka
Mr. Taketo Kosaka
OSS Technology Group,
Platform Solutions Division
at Intelligence Business Solutions

Mr. Naoyuki Kajiwara
Mr. Naoyuki Kajiwara
a Senior Engineer
of the OSS Technology Group,
Platform Solutions Division
at Intelligence Business Solutions

Mr. Naoya Tsuchiyama
Mr. Naoya Tsuchiyama
OSS Technology Group,
Platform Solutions Division
at Intelligence Business Solutions

Promotion of making templates and automation through shared purpose and cooperation with IBS

One of the features of Zabbix that is highly regarded by KDDI is the ability to improve work efficiency by making templates for monitoring settings. Mr. Jun Endo, Assistant Manager at KDDI’s Development Section 1, Cloud Service Planning , Platform Development Divison, says that “At present, Zabbix is used in the KDDI Cloud Platform Service (KCPS) for monitoring of customers’ servers and of KDDI’s servers and network equipment, and is even used for monitoring of service equipment outside of the KDDI Cloud Platform Service (KCPS), and this is being carried out efficiently by producing templates with IBS to suit the respective characteristics of each task.”

But this was no simple feat. For example, when Zabbix servers were first introduced, the number of monitoring items increased considerably, and disk capacity expanded significantly. This sometimes resulted in instability in the actual behavior of the monitoring servers. Mr. Endo adds that based on this experience “IBS now arranges parameter and maintains a state in which monitoring can be carried out smoothly by means of performance tuning”.

Looking back, Mr. Kato recalls: “In fact, there were several problems at the beginning with threshold settings, etc.” Even the slightest slippage can lead to major problems in a cloud environment with thousands of virtual servers and tens of thousands of items.

Accordingly, KDDI and IBS have produced tools to check that settings are being implemented as planned, from the generation of settings through to their actual operation, and are endeavoring to further optimize application. Mr. Kato explains: “With conventional systems, there are limits to the checking of cloud environment configuration with human eyes. Taking that into account, we are carrying out precise systematization.” Mr. Maehara concurs: “Clouds have a tremendous sense of speed, and with the KDDI Cloud Platform Service (KCPS) too there is expansion with unbelievable impetus. It is impossible to keep up without automation.”

As a system continues to be applied, the number of individual parts requiring monitoring increases, and common template parts become relatively few, but this has also been dealt with by redrawing templates in an arrangement together with IBS.

Mr. Naoyuki Kajiwara, a Senior Engineer of the OSS Technology Group at the IBS Platform Solutions Division, recalls that occasionally “We would receive very in-depth questions that were impossible to reply to immediately.” Mr. Naoya Tsuchiyama, also of the OSS Technology Group at the IBS Platform Solutions Division, states: “Rather than simply replying with product specifications, we aim to provide the best possible answer by surmising what the customer wants to achieve when carrying out monitoring.”

Further quality improvements to be made, placing even greater importance on application monitoring

Together with reviewing design from the ground up in order to achieve further quality enhancement, the aim is to produce a world in which customers can see the quality of the cloud and use it with peace of mind.

Mr. Maehara emphasizes the importance of monitoring in supporting quality: “When we use the word ‘quality’ in this field, it also includes detailed monitoring and quick submission of reports. In order to provide even greater levels of quality, it is important to recognize problems as quickly as possible and carry out restoration as quickly as possible. Application monitoring and Zabbix have extremely important roles to play in achieving this.”

Also, Mr. Endo says, “We have great hopes for the structure that has been added to Zabbix so that customers in a multi-tenant environment can configure and check monitoring items on their own.” We look forward to the further growth of the KDDI Cloud Platform Service (KCPS), which has used such functions.


System Over view

Number of Zabbix Servers: approximately 40

Redundancy: Yes. Active-Active

Multi tenant monitoring: Yes.

Number of monitored devices: approximately 20,000

Number of triggers: approximately 800,000

Number of item: approximately 400,000

Hardware for Zabbix servers: virtual machines and fiscal servers
Number of CPU core:4 to 16 (depend on case)
Memory:16GB to 128GB (depend on case)


Internet

※Official name for ‘CloudPlatform' is 'Citrix CloudPlatform powered by Apache CloudStack’.Citrix, CloudStack and CloudPlatform are trademarks of Citrix Systems, Inc. and/or one or more of its subsidiaries, and may be registered in the U.S. Patent and Trademark Office and in other countries.


KDDI CORPORATION at Glance

HEAD OFFICE:
Tokyo, Japan

FOUNDED: 1984
EMPLOYEES: 27,073
CAPITAL: 141,85 mil.Yen
(March, 2014)

www.kddi.com

 

KDDI logo

KDDI, as a comprehensive communications company offering both fixed-line and mobile communications services, strives to be a leading company during changing times.

For individual customers, KDDI offers its mobile communications (au mobile phone) and fixed-line communications (broadband Internet/telephone) services under the brand name "au", helping to realize new seamless communications environments. And for business clients, KDDI provides all services in the ICT (Information and Communication Technology) realm, from FMC (Fixed Mobile Convergence) networks to data centers, applications, and security strategies, to help clients strengthen their businesses.

Its Cloud Service Planning provides Cloud Services for business clients. The features of KDDI cloud are provided with the concept ’One-stop for cloud, network and mobile’,’ freely and through the eyes of customers’ and ’ QualityCloud’.

 

Want to share your story of using Zabbix Monitoring Solution?

Fill out this Questionnaire

Or contact our Marketing Team for further assistance.

Information Required to Prepare Zabbix Case Study
 

* Required fields

1. Please provide us with some information about your company
a) Company's name *


b) Your name, surname *


c) Job title *


d) E-mail address *


e) Description of your company's business *


f) Year of company establishment *


g) HQ location (address) *


h) Company's phone number


i) Number of employees with reference date (e.g., 580 employees on December 31, 2013)


j) Amount of revenue for a previous period (e.g., $1.2 mill. for year 2013)


k) Name several biggest customers


l) Short information/reference on why you have chosen Zabbix Monitoring Solution (provided by the person who made the decision e.g. CEO, CTO, senior engineer or any other responsible decision-maker) *

 
 
2. Reason for your organization to use monitoring solution
a) What were the challenges for you when you started to look for a monitoring solution? (the more details you provide, the better overview of the problem we can provide in a case study) *


b) What was the last drop that pushed you to search for a monitoring solution?

 
 
3. Why did you choose Zabbix
a) Which monitoring software (if any) did you compare before choosing Zabbix?


b) What made you to choose Zabbix? *


c) What, in your opinion, are those benefits of Zabbix that make it stand out over other monitoring solutions?

 
 
4. If you were using Zabbix before you have applied for any Zabbix professional services, please describe:
a) What were the reasons for you, which made you to look for Zabbix professional services


b) Was it easy to apply for Zabbix professional services


c) How satisfied are you with Zabbix professional services, and do you plan to continue purchasing Zabbix services in the future


d) Do you believe it's worth to use Zabbix services, in terms of peace of mind and economic reasons.

 
 
5. How was the process of implementing Zabbix? Was it easy? If hard, how did you manage it? *

 
 
6. What are the outcomes of implementing Zabbix within your organization
a) Name the issues that were solved after implementing Zabbix? Did you manage to solve all the issues you hoped to? *


b) How was the performance improved after implementing Zabbix? *

 
 
7. Please specify:
a) Current parameters of your company's IT infrastructure (number of monitored devices implemented systems, topology of Zabbix installations (Zabbix Servers + Zabbix Proxies + locations (everything on 1 machine or distributed across several), etc.) *


b) Parameters of Zabbix installation (number of hosts, triggers, items, users) *


c) Information on hardware, where Zabbix is installed *


d) Your Zabbix performance data (NVPS) *

 
 
8. Additional notes
Please feel free to add any additional information you would like us to know, which was not covered by the above mentioned questions

 
 
Privacy Policy Trademark Policy