Join 136,266 Programmers for FREE! Get instant access to thousands of experts, tutorials, code snippets, and more! There are 2,230 people online right now. Registration is fast and FREE... Join Now!
Hi, I am looking to manipulate some files that I have created from scripts that I have writen and placed in cron to gather stats of our MySQL databases. Now the scripts are very simple, below is an example of what I have created and is run every 24hrs
date >> /home/jcat/disk_mon.txt echo "Disk Usage" >> /home/jcat/disk_mon.txt df /var >> /home/jcat/disk_mon.txt echo "MySQL" >> /home/jcat/disk_mon.txt sudo du --max-depth=1 /var/lib/mysql >> /home/jcat/disk_mon.txt echo "Workarea" >> /home/jcat/disk_mon.txt sudo du --max-depth=1 /var/local/jc/workarea >> /home/jcat/disk_mon.txt date >> /home/jcat/mysql_data.txt sudo du -s /var/lib/mysql >> /home/jcat/mysql_data.txt date >> /home/jcat/sup_data.txt sudo du -s /var/local/jc/workarea/supplier >> /home/jcat/sup_data.txt date >> /home/jcat/tar_data.txt sudo du -s /var/local/jc/workarea/TarArchive >> /home/jcat/tar_data.txt
The most important files that I am looking to cut up so that I can place the data into an excel sheet to create some graphs is the disk_mon.txt and the mysql_data.txt files. Below is an example of the content of the disk_mon.txt file.
The data that I am most interested in is the following,
Total Disk Space: 30963708 Used Disk Space: 18103576 5164916 /var/lib/mysql/prod 5164956 /var/lib/mysql/preview 3648 /var/lib/mysql/jcatdemo 6975788 /var/lib/mysql/jcatprev 17330356 /var/lib/mysql
I would like to remove the spaces from the first lines, then place the rest into columns that are related to each topic. So all numbers and names related to prod in one column, then all data for preview in another column and so on. Now I have used sed to try to manipulate the file, but I really do not understand how it works and would like someone to try to help me with the output. The original file is below,
Ultimately I would like to know what to use and how to go about learning it so that I can go ahead and manipulate some other text files taht I have to create a decent output for excel input. Thanks in advance for your help and understanding.
First, by a var. Just few would help you out. Also, just run the command with sudo, it will save the initial question. Unless you already are, in which case you don't need them.
Second, how you do you want the data in excel? You need to really figure that out first. Here's a quick example of something that may help. Note, you do the processing on a command basis.
This overwrites an old file. Everything is tab delimited, so excel should handle it reasonably well. There's no reason you can't just tag every line with a date if you want to have a running log.
First, by a var. Just few would help you out. Also, just run the command with sudo, it will save the initial question. Unless you already are, in which case you don't need them.
Second, how you do you want the data in excel? You need to really figure that out first. Here's a quick example of something that may help. Note, you do the processing on a command basis.
This overwrites an old file. Everything is tab delimited, so excel should handle it reasonably well. There's no reason you can't just tag every line with a date if you want to have a running log.
Hope this helps.
Hey thanks for the help and advice,
I am a little confused with some of your answers, Sorry for my misunderstanding.
Your comment, "First, by a var. Just few would help you out", can you please explain this to me so that I do not make the same mistake again.
I have taken your script and saved it as start_diskmon.sh, I then ran the script with “sudo perl start_diskmon” and I received the following result.
[jcat@ujcatapp1 ~]$ cat dump.txt date\tTue Sep 2 03:51:11 CDT 2008 df /dev/mapper/local-var 1K-blocks df /dev/mapper/local-var Used df /dev/mapper/local-var Available df /dev/mapper/local-var Use% df /dev/mapper/local-var Mounted on mysql /var/lib/mysql/mysql 520 mysql /var/lib/mysql/test 4 mysql /var/lib/mysql/preview 1130560 mysql /var/lib/mysql/prod 1130560 mysql /var/lib/mysql/jcatprev 1062032 mysql /var/lib/mysql 3344200
I am a little lost as to where the $1 and $2 parameters come from?
Now as far as the excel format is concerned, below is what I think would work.
While I can't answer any of your other questions, I can help you with one of them - the script that baavgai has given you is a shell script and not a Perl script. To run it, you would want to chmod it:
While I can't answer any of your other questions, I can help you with one of them - the script that baavgai has given you is a shell script and not a Perl script. To run it, you would want to chmod it:
shell
chmod + x start_diskmon
And then execute it:
CODE
./start_diskmon
Girasquid
Thanks for the advice, it is very much appreciated.
I only used a portion of what baavgai gave me, but what he did was a really great step in the right direction.
Someone else made the following addition and it really works great for what I am looking for. Below is the script.
Your comment, "First, by a var. Just few would help you out", can you please explain this to me so that I do not make the same mistake again. ... I am a little lost as to where the $1 and $2 parameters come from? [/quote]
Sorry, "buy a var" means that using variables allows for easier code. The $LOG is a variable and by using it you can redirect your log output by just changing one value.
The $1 and $2 are a result of taking a line and spitting it up, based on white space. After the line is split up, the first non space chunk becomes $1, the second $2 and so on.
Now, on the script you've got. Congratulations, I'm glad you've got something you're happy with.
For excel, you'll have a much easier time with tab delimited rather than comma delimited. The reason is that tabs aren't going to appear in the data you're collection, commas might. When you have commas in the data, you have to allow for them with quotes; you then have to allow for quotes in you data. Trust me, tabs.
Your script doesn't look 100%; it's going to put things on different lines. Since you're unfamiliar with awk, perl sounds like a good choice instead. You'll find a lot more help for perl. ( I just assumed shell scripter because you offered a shell script. )
Here's a perl script than may suit your needs.
perl
#!/usr/bin/env perl
processCurrent('./dump.txt');
sub processCurrent() { my($logFile) = @_;
my $needHeader = !(-e $logFile); my $dt = getDateStamp(); my($used,$avail) = getVolInfo('/'); my @mysql = getMySql(qw/jcatprev preview prod/);
sub getDateStamp() { my($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time); return sprintf("%02d/%02d/%d", $mon,$mday,$year+1900); }
sub getMySql() { my(@filter) = @_; my @results = ((0) x ($#filter + 1));
foreach my $line (`sudo du --max-depth=1 /var/lib/mysql/`) { my($size,$dir) = (split(' ',$line)); $dir =~ s/^.*\///; next if ! $dir; # because if $dir is blank, it's bad foreach $i (0..$#filter) { next if $filter[$i] ne $dir; $results[$i] = $size; $filter[$i] = ""; } } return @results; }
sub getVolInfo() { my($formount) = shift; foreach my $line (`df -P /var`) { next if $line !~ /^\/dev/; my($used,$avail,$mounted) = (split(' ',$line))[2,3,5]; next if $mounted!=$formount; return ($used,$avail); } }
This code does the processing by splitting lines into arrays and then looking at the results. The mysql bit may be a little challenging, but that's only because we're trying to gather up all the matches of multiple lines into an array. Note, if you don't like my tabs, just join on commas.
1 Sep, 2008 - 07:25 AM
| 