keithb's ramblings

“I’ve heard some claim Fallout 3 is just Oblivion with guns. To those people, I say this: you say that like it’s a bad thing.”

— Jeff Atwood

My sleep schedule is completely tweaked, but it’s totally worth it. Fallout 3 is incredible.

boto’s get_all_keys() results depend on your definition of “all”

If you’re using boto to connect to s3 and want to get a listing of all the keys in a bucket, you’ll probably reach for get_all_keys(). Unfortunately, Amazon truncates the list after the first 1000. There’s a post discussing the problem and a potential solution here.

My preferred method:

 def really_get_all_keys(bucket, prefix=None):
     keys = boto.resultset.ResultSet()
     last_key_count = -1
     while last_key_count != len(keys):
         last_key_count = len(keys)
         keys += bucket.get_all_keys(prefix=prefix, 
             marker=keys[-1].key if keys else None)
     return keys

Beware if you have many many keys, this can take a while.

UPDATE: RTFM. get_all_keys() is deprecated in favor of using list(). I’ll use that instead.