After much analysis, i decided to go the Python route with the Amazon S3 service. While using curl is appealing, i’d still have to write a driver script with error-checking. In addition to the S3 python library provided by Amazon, i’m using the S3 BitBucket library by Mitch Garnaat. Though i still had to write some driver code to use the library, probably still came out ahead.
It was a little disconcerting to get a ‘500 Internal Server Error’ on my first run; though i wasn’t surprised from reading the forums and blog comments. I was surprised by the performance. In a bad way. I would think that the service would be well connected and upload rates would be quite high. I expected the majority of the application overhead to be the exec call to the md5 program, which is not inconsequential for the number of small files i have. I should do some throughput analysis, but i don’t have that kind of time right now.
Ugh. Just got a second ‘500 Internal Server Error’. Wrote some code to breathe for 10 seconds, then keep going. We’ll see how it goes.
0 comments:
Post a Comment