05-16-2006, 11:46 AM
AHH! Big avatar!
Screwing with your reality since 1998.
External sort compo?
|
05-17-2006, 05:36 AM
Antoni, Just a thought.
Here's your getform FUNCTION: FUNCTION getform FOR j = 0 TO maxi GET #1,,s1(j) IF EOF(1) THEN EXIT FOR NEXT IF j =< maxi THEN getform = j-1 ELSE getform = maxi END FUNCTION It looks like you will read one long variable into s1(j) 16k times. Since the unsort file is 4MB, using this function you will perform 1 million reads (gets) from the file. Won't this slow down the whole process? Isn't there a way you could read some larger chunks, like maybe 4000 bytes, to reduce the number of reads down to 1000? Of course you would have to parse these chunks in 4 byte increments and convert each 4 bytes to a long variable, and then store them into the s1 array. Would this be difficult? Assuming you have an extra 4000 bytes in your program's memory, wouldn't that increase the speed? *****
05-17-2006, 06:01 AM
I can read 12MB at a time, reducing the number of reads. I'm not sure if this is standadr accross the board for all disk I/O buffers, but mine is 12MB. Meaning I can read a total of 40 or 50MB per second.
Then all you wold do is split it into chunks that you can deal with.
Screwing with your reality since 1998.
05-17-2006, 11:12 AM
Quote: Isn't there a way you could read some larger chunks, like maybe 4000 bytes, to reduce the number of reads down to 1000?Unfortunately QB can only GET simple variables. The only way to reduce the number of gets would be to use a 32K string, then move the values to the array with CVL(MID$ ) in a loop. I don't know if it would increase the speed , because of the overhead in using a string function as MID$. I should try it.. BTW: FB can read a complete array in a single operation, this makes it fast!
Antoni
05-17-2006, 11:20 AM
Still awaiting a reply
Quote:The only way to reduce the number of gets would be to use a 32K string, then move the values to the array with CVL(MID$ ) in a loop. I don't know if it would increase the speed , because of the overhead in using a string function as MID$. I should try it..If the overhead on using the string function is too big, play it static. I.e. use PEEK and read the 4-byte data directly from the string instead of creating many string duplicates. Then merge the 4 bytes together using binary operations. I think if done correctly, you might reduce the overhead that you would have got if you applied the string method. - Neo
05-17-2006, 12:00 PM
I rechecked my code, I think the botleneck is not where Moneo suggests, in fact the bottleneck are the 255 different SEEKS required to save the sorted data en each pass. Each file pointer movement does flush the output buffer...
Now I think the fastest sort would be an in-memory radix sort combined with a multiway merge at the end.
Antoni
05-21-2006, 10:33 PM
Quote:Antoni, I read Knuth's section on external sorting, with special attention to Radix sorting. As with many chapters of Knuth's books, I hardly understood what he was talking about. I've been reading some Knuth during the last few days, its pretty tough going. I found an error too, and reported it, but apparently some people had to wait 15 years for their cheque because Knuth was too busy! I also found this great quote, from when Steve Jobs (of Apple) met Knuth. Quote:"It's a pleasure to meet you, Professor Knuth," Steve said. "I've read all of your books."
05-23-2006, 07:59 PM
I made a program, unfortunately when i tried running it in qb 4.5 I discovered that it only works in PDS 7.1, which i used when I was writing it.
Is it still ok if I post the program?
05-23-2006, 08:27 PM
Ok.. here we go, it's kind of spaghetti like, but I was trying to keep it simple and as fast as i could.
It's based on radix sorting. Code: DEFINT A-Z Please post comments if you have any. |
« Next Oldest | Next Newest »
|