macOS Text Parser: Text to MySQL Database

ATG said:

Use regexes. AGRegex is the best I've found (and it has my initials )

EDIT: Oh, sorry, I didn't ask which language you are using :\

Click to expand...

stndn said:

I thought you wanted to have something in Perl, and then I realized you mentioned that Perl is not necessary in your initial post ,p

iBert has the right questions, which I'm going to add to: Is this text parsing a one-time deal? Or are you going to run the script every time you want to show the output in other format?

If it's a one time deal, then you can take the Perl script as your base, make some changes, save it as filename.pl, and run the script through perl parser (one of which can be done from command line / terminal by typing: perl filename.pl).

If it's an ongoing parsing (done per request), maybe change it to something like PHP or whatnot. Not that Perl can't be used for web, but PHP can be mixed with HTML more easily / intuitively than Perl, which seems like what you're looking for (add HTML to the output).

-stndn.

Click to expand...

bobber205 said:

MySql and PHP can be installed locally. You can reach your server by going to locahost:80

I just checked my apps folder and I can't seem to find the app that does this server. Do a quick "PHP and MYSQL server for OS X" search on google.

EDIT: I tried that search and I can't find it. Damn.

Click to expand...

macfaninpdx said:

You could either install them separately, by going to their respective sites (mySQL and PHP), or you could use packaged solution such as XAMPP, which will install Apache, MySQL, PHP & PEAR, and PERL along with several other useful utilities.

The advantage to installing them individually is that you only install the pieces you need. The advantage to using something like XAMPP is that it gives you administration utilities as well.

Click to expand...

pengu said:

firstly. i suggest you install apache2, php5 and mysql5 seperately. there are very easy to follow tutorials, and you aren't likely to need to mess with them much once its set up.

Click to expand...

macfaninpdx said:

If you use PHP for you original question, you could try something like the following:

PHP:

$logfile = file("\\path\\to\\your\\file");

foreach ($logfile AS $linenum => $line) {
    $logdate = substr ($line, 0, 8);
    $logtime = substr ($line, 9, 12);
    $level = substr ($line, 22, 4);
    $id = substr ($line, 27, 29);
    $info = rtrim (substr ($line, 57) );
}

Doing it this way (with substr) means that the date, time, level and id must be the same exact length every time, or it will break the line in the wrong spot. So a RegEx expression would definitely be more robust, but since I do not know regex expressions very well , this is another solution.

Click to expand...

macfaninpdx said:

I just thought of one more solution to splitting the log file entries, although it still may not be as efficient as a properly written regex:

PHP:

$logfile = file("\\path\\to\\your\\file"); 

foreach ($logfile AS $linenum => $line) { 
    //explode line using space delimiter
    list ($logdate, $logtime, $level, $id1, $id2, $id3, $info) = explode (" ", rtrim( $line ), 7);
    //combine id variables into one
    $id = implode (" ", array ($id1, $id2, $id3) );
}

Since the explode function is using a space character to split the line, the ID field was split into three parts. I used the implode line to combine them again. Also, since there is a limit of 7 on the explode function, the remaining info in the INFO field will all be kept together (not separated).

Click to expand...

stndn said:

I've just seen the sample log files you provided, and it's safe to say that the example code with regex won't work because I used the keyword 'accept' as one of the separators towards the end.

The key with text parsing is identifying consistent section(s) of the texts you are trying to parse.

From your example log, it's safe to assume that the part that will be consistent is the timestamp. It's not clear yet (to us) whether the texts 'INFO Servlet.Engine.Transports : 0' will always be there as well. If yes, then it can be used as separator if you are going through the regex way.

If you are using the substr () or explode () way per what's given by macfaninpdx, then you need to make sure that the texts and spacings are consistent among different parts of the log.

Otherwise, you might consider mixing regex and substr/explode as well.

-stndn.

Click to expand...

stndn said:

I've just seen the sample log files you provided, and it's safe to say that the example code with regex won't work because I used the keyword 'accept' as one of the separators towards the end.

The key with text parsing is identifying consistent section(s) of the texts you are trying to parse.

From your example log, it's safe to assume that the part that will be consistent is the timestamp. It's not clear yet (to us) whether the texts 'INFO Servlet.Engine.Transports : 0' will always be there as well. If yes, then it can be used as separator if you are going through the regex way.

If you are using the substr () or explode () way per what's given by macfaninpdx, then you need to make sure that the texts and spacings are consistent among different parts of the log.

Otherwise, you might consider mixing regex and substr/explode as well.

-stndn.

Click to expand...

stndn said:

From your example log, it's safe to assume that the part that will be consistent is the timestamp. It's not clear yet (to us) whether the texts 'INFO Servlet.Engine.Transports : 0' will always be there as well. If yes, then it can be used as separator if you are going through the regex way.



-stndn.

Click to expand...

06/07/07 02:44:27.585 INFO Servlet.Engine.Transports : 0 Type: TEST
06/07/07 02:44:27.586 INFO Servlet.Engine.Transports : 0 Site User: QA_BPORTAL
06/07/07 02:44:27.586 INFO Servlet.Engine.Transports : 0 Product:ProductId: Broker Portal Name: Broker Portal
06/07/07 02:44:27.588 INFO Servlet.Engine.Transports : 0 Retrieving user ent list
06/07/07 02:44:27.619 INFO Servlet.Engine.Transports : 0 Retrieve ent names for user: 3771000016, contract: qa_bportal