cmu monitoring tool - proto

HP Restricted

CMU Performance Monitoring tool - PrototypeIntern Name: Goutam Adwant

What does the tool do?

Purpose of this tool is to run jobs that would extract core,disk,memory & network - detail + summary data from CMU( using collect & Scripts ) for a specified time frame and then store it in MySQL DB for the purpose of reporting/graphing & analysis of the test results.

HP Restricted


Technologies:

Front end:•HTML•CSS•JQuery

Server Side:•PHP/PHPSECLIB – Used to exec/run monitoring dump commands on mgmt. server & connect to MySQL dB to store metadata of the test run. •Python – Used to extract & read dump file and connect to MySQL dB & insert the data to its corresponding detail & summary table.Tools:•HeidiSQL – To display mgmt. server MySQL data in a tabular form •CMU – To view the data of the defined metrics – Core,Disk,Memory & Network.

Build a simple front end screen through which user can input test details & time frame.

HP Restricted


Component functional view :

Test DescriptionStart TimeEnd Time

Submit

Front End Screen

PHP Script

CMU Python Script

Excel/CSVChartsReporting

Connect & Generates dump file

Returns Dump file

Triggers

Exec .py scripts

Reads the dump file & inserts into respective tables

MySQL

Entire code deployed on tomcat server of mgmt. node.AAA File

CPU & Disk Scripts

/proc/stat

Cpucore0-39

Disk1-13

Request is hit only once and the result is stored in cache so that all the core & disk data are retrieved from that.

HP Restricted


Schema Flow diagram:

Test Execution

Test metrics summary

cpu_util_all disk_metrics

Detail Level Data

Summary Level Data

1 row per each test execution

1 row/node during a test run

1 row/detail of node during a test run

Meta Data

HP Restricted


Test Execution data – captures metadata for each test execution

DB Snapshot

SchemaCREATE TABLE `test_execution` ( `test_id` VARCHAR(45) NOT NULL, `start_time` DATETIME NULL DEFAULT NULL, `end_time` DATETIME NULL DEFAULT NULL, `test_desc` VARCHAR(45) NULL DEFAULT NULL, `test_result` VARCHAR(45) NULL DEFAULT NULL, `test_user` VARCHAR(45) NULL DEFAULT NULL, `metrics_calc` TINYINT(1) NULL DEFAULT NULL, `results_captrured` TINYINT(1) NULL DEFAULT NULL, `stdout_captured` TINYINT(1) NULL DEFAULT NULL, PRIMARY KEY (`test_id`));Test id: Program generated metric. It is unique for every job executed. Acts as a foreign key to the meta & Summary table/data. start_time: User input data – start time of the data that needs to be extracted.End_time : User input data – end time of the data that needs to be extracted.Test_desc: User input data – test description of the test run that the user submits.Test_result,test_user,metrics_calc,results_captured,stdout_captured: not yet populated – to be decided.

HP Restricted


Test Execution data – Metric Details

Metric Name SourceTest_id Program generated metric- unique

for each test runStart_time User input data End_time User input data Test_desc User input data

HP Restricted


Test Metrics Summary data –

Captures node level summary data. It contains core,disk,network & memory information of each node for 5sec time span.

Schema :

CREATE TABLE `test_metrics_summary` ( `test_id` INT(11) NULL DEFAULT NULL, `servername` VARCHAR(45) NOT NULL, `date_time` VARCHAR(45) NOT NULL, `interval_time` DATETIME NOT NULL, `data_disks_read_mb` DECIMAL(10,2) NULL DEFAULT NULL, `data_disks_write_mb` DECIMAL(10,2) NULL DEFAULT NULL, `data_disks_read_iops` VARCHAR(45) NULL DEFAULT NULL, `data_disks_write_iops` VARCHAR(45) NULL DEFAULT NULL,`total_cpu` INT(11) NULL DEFAULT NULL, `sys_disks_read_mb` VARCHAR(45) NULL DEFAULT NULL,`sys_disks_write_mb` VARCHAR(45) NULL DEFAULT NULL, `sys_disks_read_iops` VARCHAR(45) NULL DEFAULT NULL,`sys_disks_write_iops` VARCHAR(45) NULL DEFAULT NULL, `nw_pkt_in` VARCHAR(45) NULL DEFAULT NULL,`nw_pkt_out` VARCHAR(45) NULL DEFAULT NULL, `memory_used` VARCHAR(45) NULL DEFAULT NULL,`total_avail_memory` VARCHAR(45) NULL DEFAULT NULL, `memory_free` VARCHAR(45) NULL DEFAULT NULL, PRIMARY KEY (`servername`, `date_time`, `interval_time`));

HP Restricted


Metric Name Sourcetest_id Program generated metric- unique for each test

runstart_time, end_time User input data data_disks_read_mb, data_disks_write_mb /opt/cmu/tools/cmu_get_det_disk -n disk –rs

/opt/cmu/tools/cmu_get_det_disk -n disk -wsdata_disks_read_iops, data_disks_write_iops /opt/cmu/tools/cmu_get_det_disk -n disk –rc

/opt/cmu/tools/cmu_get_det_disk -n disk -wc

system_disks_read_mb, system_disks_write_mb /opt/cmu/tools/cmu_get_det_disk -n sys –rs/opt/cmu/tools/cmu_get_det_disk -n sys -ws

system_disks_read_iops, system_disks_write_iops

/opt/cmu/tools/cmu_get_det_disk -n sys–rc/opt/cmu/tools/cmu_get_det_disk -n sys -wc

total_cpu /opt/cmu/tools/cmu_get_det_cpu -c -n cpunw_out_mbps, nw_in_mbps COLLECTL (nettotals.kbout) / 1024

COLLECTL (nettotals.kbin) / 1024total_memory COLLECTL (meminfo.tot)memory_free COLLECTL (meminfo.tot) - (meminfo.used)memory_used COLLECTL (meminfo.used) / (meminfo.tot) *

100

Test Metrics Summary data – Metric Details

HP Restricted


CPU/Core - Detail Data – captures core level detail

DB Snapshot

SchemaCREATE TABLE `cpu_util_all` ( `test_id` INT(11) NOT NULL, `interval_time` INT(11) NOT NULL, `servername` VARCHAR(45) NOT NULL, `date_time` VARCHAR(45) NOT NULL, `cpu_user` VARCHAR(45) NOT NULL, `cpu_core_value` VARCHAR(45) NULL DEFAULT NULL, PRIMARY KEY (`test_id`, `interval_time`, `servername`, `date_time`, `cpu_user`));

Test id: Program generated metric. It is unique for every job executed. Acts as a foreign key to the meta & Summary table/data. Interval_time: Program generated metric. For a particular test_id, it increments for every 5seconds of test result data. ( e.g.. For 1min of test data – we have 12 interval_time id’s )Servername: Generated through the dump file.Data_time: Generated through the dump file.cpu_user: Metric data generated through the AAA file.cpu_core_value: Metric data generated through the AAA file.

HP Restricted



runInterval_time Program generated metric. For a particular

test_id, it increments for every 5seconds of test result data

Servername Generated through the dump file.date_time Generated through the dump file. Increments

with 5secs from start time to end time.cpu_user Generated through detail script in AAA.

cpucore_00 to cpucore_39cpu_core_value /opt/cmu/tools/cmu_get_det_cpu -n cpu00-39

CPU/Core data – Metric Details

HP Restricted


Disk – Detail Data:

DB Snapshot

SchemaCREATE TABLE `disk_stats_all` ( `test_id` INT(11) NOT NULL, `interval_time` INT(11) NOT NULL, `servername` VARCHAR(45) NOT NULL, `date_time` VARCHAR(45) NOT NULL, `disk_name` VARCHAR(45) NOT NULL, `disk_read_mbps` VARCHAR(45) NULL DEFAULT NULL, `disk_write_mbps` VARCHAR(45) NULL DEFAULT NULL, `disk_reads` VARCHAR(45) NULL DEFAULT NULL, `disk_writes` VARCHAR(45) NULL DEFAULT NULL, PRIMARY KEY (`test_id`, `interval_time`, `servername`, `date_time`, `disk_name`));Test id: Program generated metric. It is unique for every job executed. Acts as a foreign key to the meta & Summary table/data. Interval_time: Program generated metric. For a particular test_id, it increments for every 5seconds of test result data. ( e.g.. For 1min of test data – we have 12 interval_time id’s )Servername: Generated through the dump file.Data_time: Generated through the dump file.Disk_name: Inserted through the python script for the number of disks present for a particular node.disk_read_mbps, disk_write_mbps,disk_reads,disk_writes : Metric data generated through the AAA file.

HP Restricted



runInterval_time Program generated metric. For a particular

test_id, it increments for every 5seconds of test result data

Servername Generated through the dump file.date_time Generated through the dump file. Increments

with 5secs from start time to end time.disk_name Generated through detail script in AAA.disk-1 to

disk-13disk_read_mbps,disk_write_mbps /opt/cmu/tools/cmu_get_det_disk -n disk1-13 –

rs /opt/cmu/tools/cmu_get_det_disk -n disk1-13 –ws

disk_reads,disk_writes /opt/cmu/tools/cmu_get_det_disk -n disk1-13 -rc/opt/cmu/tools/cmu_get_det_disk -n disk1-13 –wc

Disk data – Metric Details

HP Restricted


• Select `cmudata`.`test_metrics_summary`.`test_id` AS `test_id`,`cmudata`.`test_metrics_summary`.ìnterval_time` AS ìnterval_time`,avg(`cmudata`.`test_metrics_summary`.`total_cpu`) AS àvg(total_cpu)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_read_mb`) AS `sum(data_disks_read_mb)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_write_mb`) AS `sum(data_disks_write_mb)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_read_iops`) AS `sum(data_disks_read_iops)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_write_iops`) AS `sum(data_disks_write_iops)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_read_mb`) AS `sum(sys_disks_read_mb)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_write_mb`) AS `sum(sys_disks_write_mb)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_read_iops`) AS `sum(sys_disks_read_iops)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_write_iops`) AS `sum(sys_disks_write_iops)`,sum(`cmudata`.`test_metrics_summary`.`nw_pkt_in`) AS `sum(nw_pkt_in)`,sum(`cmudata`.`test_metrics_summary`.`nw_pkt_out`) AS `sum(nw_pkt_out)`,sum(`cmudata`.`test_metrics_summary`.`memory_used`) AS `sum(memory_used)`,sum(`cmudata`.`test_metrics_summary`.`total_avail_memory`) AS `sum(total_avail_memory)`,sum(`cmudata`.`test_metrics_summary`.`memory_free`) AS `sum(memory_free)` from `cmudata`.`test_metrics_summary` where (`cmudata`.`test_metrics_summary`.`servername` like 'f02wn%') group by `cmudata`.`test_metrics_summary`.`test_id`,`cmudata`.`test_metrics_summary`.ìnterval_time` order by `cmudata`.`test_metrics_summary`.`test_id`,`cmudata`.`test_metrics_summary`.ìnterval_time

Views:

test_metrics_north_aggregate : Retrieves the aggregate level data filtered only for worker nodes and ordered by test_id & interval id.

test_metrics_south_aggregate : Retrieves the aggregate level data filtered only for data nodes and ordered by test_id & interval id.

• Select `cmudata`.`test_metrics_summary`.`test_id` AS `test_id`,`cmudata`.`test_metrics_summary`.ìnterval_time` AS ìnterval_time`,avg(`cmudata`.`test_metrics_summary`.`total_cpu`) AS àvg(total_cpu)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_read_mb`) AS `sum(data_disks_read_mb)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_write_mb`) AS `sum(data_disks_write_mb)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_read_iops`) AS `sum(data_disks_read_iops)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_write_iops`) AS `sum(data_disks_write_iops)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_read_mb`) AS `sum(sys_disks_read_mb)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_write_mb`) AS `sum(sys_disks_write_mb)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_read_iops`) AS `sum(sys_disks_read_iops)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_write_iops`) AS `sum(sys_disks_write_iops)`,sum(`cmudata`.`test_metrics_summary`.`nw_pkt_in`) AS `sum(nw_pkt_in)`,sum(`cmudata`.`test_metrics_summary`.`nw_pkt_out`) AS `sum(nw_pkt_out)`,sum(`cmudata`.`test_metrics_summary`.`memory_used`) AS `sum(memory_used)`,sum(`cmudata`.`test_metrics_summary`.`total_avail_memory`) AS `sum(total_avail_memory)`,sum(`cmudata`.`test_metrics_summary`.`memory_free`) AS `sum(memory_free)` from `cmudata`.`test_metrics_summary` where (`cmudata`.`test_metrics_summary`.`servername` like 'f02dn%') group by `cmudata`.`test_metrics_summary`.`test_id`,`cmudata`.`test_metrics_summary`.ìnterval_time` order by `cmudata`.`test_metrics_summary`.`test_id`,`cmudata`.`test_metrics_summary`.ìnterval_timetest_metrics_summary : Retrieves the aggregate level data for both

the nodes and ordered by test_id & interval id

HP Restricted


Final Output:Data stored in MySQL Db. is saved as excel/csv and graphs/charts can be generated.

HP Restricted


DEMO

cmu monitoring tool - proto

Documents