Today we worked on moving some of the non-select SQL statements outside of the for loops
in an attempt to speed up the load script. We kept running into new problems due to some
cut & paste errors that we made, but it appears that we have gotten most if not all of those
out of our code for now.The only thing that I couldn't figure out was we needed to enter in
a null value into a table and couldn't seem to find a value that would work. Turns out that
just declaring a my $NULL and not initializing it to anything worked. Aparently
Perl initiallizes variables to NULL or something that works. It still takes forever to run
though, so I don't know what, if anything, is causing the holdup. We'll run a few tests to
see whats going on.
Finished cleaning up the code for load.pl today. So-far we have moved most if
not all of the non-select statements out of for loops and we have attempted to remove all of
the random generation out of the for loops also. For everything except Orders
these modifications have resulted in some improvement in wall runtime. Orders is still rather
slow in comparison though. I've been running it using SmallProf, so tomorrow, I/we should be
able to find out where the time is going.
Went through the transaction code today and made sure that things had been cleaned up in a
similar fassion to the way load.pl. That is, moving random stuff out of any
possible loops, and putting commit statements into the code and turning off AutoCommit. Made
a few more test runs of load. Late in the afternoon we discovered 1 split
statement that was taking up 2 hours of run-time to execute. We initially thought that this
was because we were operating on the Argument stack. I changed things around so that the
argument was put into a variable before being passed to slplit, and then run split on it.
We'll have to see if this makes any significant difference in our wall-time.