My goto style when dealing with python dictionaries. In my latest one, I was building a dictionary of lists: try dict[k].append(element), except dict[k] = [element].
Glad to know the EAFP fancy name that I can throw around my peers. And thanks for the good insights on the performance cuts this approach entails.
Use a defaultdict. If the key does not exist it defaults to a given type.
from collections import defaultdict
dict_of_list = defaultdict(list)
dict_of_list[key].append(element)
That causes the python vm to build a new empty list for each element, and make two function calls instead of one.
This adds up when looping, and may fragment the heap.
The post sorely missies a benchmark measuring actual performance difference. Given that message_is_compressed(message) is probably not message.startswith("{"), the actual JSON parsing should completely dominate the check, and I would be surprised if this indeed makes a measurable difference.
True, I felt the same writing the post, since since it’s from a year ago, and the reason for blogging it was a friend’s chat I didn’t have the chance to measure it.
Will try to measure it next time. But the point is -even though not measured- a string scan easily adds up in overhead over millions of messages. For ~0.02% of the cases?
What do you mean by “a string scan”? What was the nature of the check to see if a message was compressed? Was it trialing an unpacking of the message?
I ask because you’ve brought it up a lot of times, once in reference to checking a single char at the start of the string, and it feels like you’ve been bitten by something related hard enough that you’ve got a visceral reaction here, rather than measuring things out?
Clever trick, but I think that down this path lies madness. The 256kb limit hasn’t gone away; it will just creep up on you when you least expect it. Suppose you switch compression algorithms to buy more time - now every message detected as base64 will have to rotate through a bunch of decompression algorithms to find one that generates reasonable-looking output. There goes your performance improvement.
Some alternatives:
Spin up an alternative SNS topic for compressed messages, and have different lambdas processing compressed/uncompressed data. Possibly causes a maintenance explosion if you need to have multiple consumers of both topic.
Prefix your payloads with some metadata (at least a version number), in a way that’s fast to parse.
Use a faster language if speed really is that important, and probably pick up safety guarantees too given recent advances in popular typesafe languages.
Store large bodies elsewhere, like S3 or DynamoDB. Adds a lot more complexity to your messaging system.
Devil’s advocate: why impose a maintenance or complexity burden up front? If you can use a good enough solution right now, you can attack the complexity later on when you have a better idea of how the problem is going to change. If it never comes up then you avoided over designing your system.
FWIW, I worked on a system dealing with SQS’s similar limit, and we’ve also dealt with compression and multiple formats. Observations, though I’m not sure they change anything for anyone:
We could be confident that compression does what we need because we know the kind of content we put in. For us it’s batches of similar items and we get much more than gzip’s typical ~3:1 for text or such.
We temporarily ran a fork of the system where individual items could be huge, and implemented option 4: a JSON struct in SQS with some metadata and a pointer to S3. (We also switched to zstd, which can better handle repetition across large items.) There was enough work to do on each batch that the S3 latency didn’t really matter.
Hacking in a second format without an explicit version number was easier in practice than in theory. We run the producer and the consumer. Our compressed messages predictably started with gzip’s magic bytes base64’d, and our JSON pointing to S3 with {. A msg.startswith is negligible even compared to just the SQS fetch, and we still have a lot of magic-number (or other format-distinguishing) space open.
SQS and SNS also both have message attributes you could use for version numbers, S3 pointers, etc. Seems like you’re essentially having your AWS library do a serialize/deserialize for you, with no advantage in terms of size limit or billing, but they still may be convenient.
Enough of the code was the application (as opposed to the fetching/decoding) that I’m not 100% sure if we used msg.startswith or try/except and the difference is small compared to other work being done and other code.
Encoding message type in-band like this is a great way to create security vulnerabilities. I would reject this approach in review on that general principle.
The specifics of how you’re suggesting doing the detection are also weird. In addition to the likely unique prefix of the JSON that matklad notes, your compressed blob probably already has a unique prefix (e.g., a gzip or zlib header). Checking for that magic in base64 form should have negligible cost.
If these are json messages, I’d check if maybe you can compress everything. Stream compression of text is really cheap and at 256kb, I wonder if the uncompressed sending takes longer than just compressing everything.
Most of the time I would chose clarity instead of performance, if I can’t have both, and IMHO the ifs version is clearer here.
Moreover, in the case of a lambda function, I think you will spend way more time starting the python interpreter than computing an if statement !
That happens mostly with cold starts, most of the time this lambda function is warm/hot, continuously processing a considerable load of messages. A string scan easily adds up in overhead over millions of messages. For ~0.02% of the cases?
Readability counts, but it also a tradeable trade-off in such cases. And I don’t personally see a try/catch that terribly less readable.
To circumvent this limit, we’ve found that 256KB is well enough room if we compress these kind of messages. But compressing produces bytes (binary data), and AWS SNS accepts text-based messages, so we had to encode the data with base64 before sending over to SNS.
Could you store the binary value in a MessageAttribute? Then you’d save time on decoding the base64. Might work for the 99.8% of smaller messages, too: there’s probably a library for decoding json from binary files that’s faster than parsing text.
For the specific example given, you may want to consider using Ascii85 instead of Base64? SNS apparently supports at least 0x20..0x7f, and you could even swap 0x7f = DEL for e.g. 0x0a = \n if you want printable characters, so…
This is all well and good until your compression | base64 step accidentally generates valid JSON.
How likely is that to happen? I don’t know, perhaps impossible for gzip, but that might not hold for other algorithms. I wouldn’t like to be the one who has to chase down this possibility as part of a debugging run.
Regarding this comment on your website:
Use a defaultdict. If the key does not exist it defaults to a given type.
It is more readable and faster!
Or
setdefaultmethod for thedictbuilt-in:d.setdefault(key, []).append(element)That causes the python vm to build a new empty list for each element, and make two function calls instead of one. This adds up when looping, and may fragment the heap.
I believe the defaultdict simply overrides the
__missing__method of dict, effectively making it a ‘ask for forgiveness’ implementation.The post sorely missies a benchmark measuring actual performance difference. Given that
message_is_compressed(message)is probablynot message.startswith("{"), the actual JSON parsing should completely dominate the check, and I would be surprised if this indeed makes a measurable difference.True, I felt the same writing the post, since since it’s from a year ago, and the reason for blogging it was a friend’s chat I didn’t have the chance to measure it.
Will try to measure it next time. But the point is -even though not measured- a string scan easily adds up in overhead over millions of messages. For ~0.02% of the cases?
What do you mean by “a string scan”? What was the nature of the check to see if a message was compressed? Was it trialing an unpacking of the message?
I ask because you’ve brought it up a lot of times, once in reference to checking a single char at the start of the string, and it feels like you’ve been bitten by something related hard enough that you’ve got a visceral reaction here, rather than measuring things out?
Clever trick, but I think that down this path lies madness. The 256kb limit hasn’t gone away; it will just creep up on you when you least expect it. Suppose you switch compression algorithms to buy more time - now every message detected as base64 will have to rotate through a bunch of decompression algorithms to find one that generates reasonable-looking output. There goes your performance improvement.
Some alternatives:
Devil’s advocate: why impose a maintenance or complexity burden up front? If you can use a good enough solution right now, you can attack the complexity later on when you have a better idea of how the problem is going to change. If it never comes up then you avoided over designing your system.
That said, I don’t totally disagree with you.
Alternative 2 is the best idea imo. Encoding the message “kind” in a header can be very cheap and very fast.
Agreed.
We were already considering option #4, but as already mentioned by other comments, we didn’t want to add that complexity upfront at the moment.
FWIW, I worked on a system dealing with SQS’s similar limit, and we’ve also dealt with compression and multiple formats. Observations, though I’m not sure they change anything for anyone:
{. Amsg.startswithis negligible even compared to just the SQS fetch, and we still have a lot of magic-number (or other format-distinguishing) space open.Enough of the code was the application (as opposed to the fetching/decoding) that I’m not 100% sure if we used
msg.startswithortry/exceptand the difference is small compared to other work being done and other code.https://quoteinvestigator.com/2018/06/19/forgive/
It didn’t sound right that Grace Hopper coined that phrase. I’d heard it in too many contexts for her to be the origin.
Encoding message type in-band like this is a great way to create security vulnerabilities. I would reject this approach in review on that general principle.
The specifics of how you’re suggesting doing the detection are also weird. In addition to the likely unique prefix of the JSON that matklad notes, your compressed blob probably already has a unique prefix (e.g., a gzip or zlib header). Checking for that magic in base64 form should have negligible cost.
If these are json messages, I’d check if maybe you can compress everything. Stream compression of text is really cheap and at 256kb, I wonder if the uncompressed sending takes longer than just compressing everything.
Only recently did try/except become “zero-cost” in CPython. Before, the
ifmay well have been faster.If the condition check itself is fast, sure.
Yes, good point.
Forget about performance – making a habit of this prevents race conditions and TOCTOU security problems.
Most of the time I would chose clarity instead of performance, if I can’t have both, and IMHO the ifs version is clearer here. Moreover, in the case of a lambda function, I think you will spend way more time starting the python interpreter than computing an if statement !
That happens mostly with cold starts, most of the time this lambda function is warm/hot, continuously processing a considerable load of messages. A string scan easily adds up in overhead over millions of messages. For ~0.02% of the cases?
Readability counts, but it also a tradeable trade-off in such cases. And I don’t personally see a try/catch that terribly less readable.
Could you store the binary value in a MessageAttribute? Then you’d save time on decoding the base64. Might work for the 99.8% of smaller messages, too: there’s probably a library for decoding json from binary files that’s faster than parsing text.
For the specific example given, you may want to consider using Ascii85 instead of Base64? SNS apparently supports at least 0x20..0x7f, and you could even swap 0x7f = DEL for e.g. 0x0a = \n if you want printable characters, so…
This is all well and good until your compression | base64 step accidentally generates valid JSON.
How likely is that to happen? I don’t know, perhaps impossible for gzip, but that might not hold for other algorithms. I wouldn’t like to be the one who has to chase down this possibility as part of a debugging run.
Base 64 doesn’t include quotation marks or brackets, but it could be all digits. That’s extremely unlikely, but theoretically possible.
Yes, all digits was my thinking, or possibly
null