My knee-jerk reaction was to exclaim “preposterous!” in an uppity tone, but my monocle was not nearby. I’ll wait and see. They don’t try to make it sound magical, they go to great lengths to explain the constraints one has to respect if he hopes for this to work at all…
I was under the impression that there was some mathematical impossibility regarding exactly-once semantics? Or at least very prohibitive conditions? I can’t remember. Please halp?
My (definitely novice) understanding is that they’re ignoring a failure here:
If you are using the consumer API, this means ensuring that you commit changes to your application state concordant with your offsets as described here.
so they’ve gotten it pretty close to exactly once, and punted the responsibility of breaking the laws of physics to your application.
EDIT: Note the realm of failures possible here is not just software bugs, but if your code completes exactly everything it wanted to do, was exactly before “commit changes to your application state concordant with your offsets as described here” and then the hardware goes kaput.
iiuc, you always have the possibilty of duplication (double-processing) or loss (zero processing), because fundamentally your client code will do either:
process(msg);
ack(msg);
or:
ack(msg);
process(msg);
and you can always just die (power off) in between the two. This can be made a short interval but it’s always there. (If your process() records the ids of the msgs you’ve processed and ignores ones it has seen before you have the same situation but just lower down - the ack(msg) becomes record_id(msg)).
I think these improvements are reducing the failure modes where the client has called ack(msg) and the rest of the system loses the ack and can cause the msg to be replayed.
[I guess it might be possible if your ‘process’ can do its work and record the id in an atomic step (a txn)?]
I think the way it’s supposed to work is, if you commit “achieved work” alongside of wherever it is your processor is at, AND have idempotent operations, then it means that “processing twice doesn’t matter”. I’d argue that in effect this doesn’t really mean “exactly once” delivery. It’s more “exactly-once processing”, like the response linked to by @pushcx: You’d only have completely (as in, successfully) processed any given message exactly once.
That’s how I understand it at least. It’s still pretty good. It doesn’t get around the mathematical impossibility, it gives you the tools to do what you want with less chance of footgunmanship.
Yes, reading the ‘redux’ article - basically if your ‘process’ step can be a part of the same transaction as recordin the ack. you can achieve exactly onice.
I don’t think you also need the process step to be idempotent - being part of a transaction (which can also contain the id for de-dupe) is sufficient. That does mean though that the process step be reversible - so the txn can be aborted.
I think if your process step is “fire the missiles”, you’ll never have exactly once.
I don’t think you also need the process step to be idempotent - being part of a transaction (which can also contain the id for de-dupe) is sufficient. That does mean though that the process step be reversible - so the txn can be aborted.
Transactions work only if the changes they are doing can be rolled back. That is not exactly the same as requiring idempotence, but your process may need to be restarted after failing at an arbitrary point.
That is the whole disagreement of what “exactly once means”, as far as I understand this. One cannot guarantee “exactly once” on arbitrary conditions, what this (and other recent works that claim to achieve exactly once) do is to provide a set of restrictions under which you have those guarantees.
My knee-jerk reaction was to exclaim “preposterous!” in an uppity tone, but my monocle was not nearby. I’ll wait and see. They don’t try to make it sound magical, they go to great lengths to explain the constraints one has to respect if he hopes for this to work at all…
I was under the impression that there was some mathematical impossibility regarding exactly-once semantics? Or at least very prohibitive conditions? I can’t remember. Please halp?
My (definitely novice) understanding is that they’re ignoring a failure here:
so they’ve gotten it pretty close to exactly once, and punted the responsibility of breaking the laws of physics to your application.
EDIT: Note the realm of failures possible here is not just software bugs, but if your code completes exactly everything it wanted to do, was exactly before “commit changes to your application state concordant with your offsets as described here” and then the hardware goes kaput.
[Comment removed by author]
response
iiuc, you always have the possibilty of duplication (double-processing) or loss (zero processing), because fundamentally your client code will do either:
process(msg); ack(msg);
or:
ack(msg); process(msg);
and you can always just die (power off) in between the two. This can be made a short interval but it’s always there. (If your process() records the ids of the msgs you’ve processed and ignores ones it has seen before you have the same situation but just lower down - the ack(msg) becomes record_id(msg)).
I think these improvements are reducing the failure modes where the client has called ack(msg) and the rest of the system loses the ack and can cause the msg to be replayed.
[I guess it might be possible if your ‘process’ can do its work and record the id in an atomic step (a txn)?]
I think the way it’s supposed to work is, if you commit “achieved work” alongside of wherever it is your processor is at, AND have idempotent operations, then it means that “processing twice doesn’t matter”. I’d argue that in effect this doesn’t really mean “exactly once” delivery. It’s more “exactly-once processing”, like the response linked to by @pushcx: You’d only have completely (as in, successfully) processed any given message exactly once.
That’s how I understand it at least. It’s still pretty good. It doesn’t get around the mathematical impossibility, it gives you the tools to do what you want with less chance of footgunmanship.
Yes, reading the ‘redux’ article - basically if your ‘process’ step can be a part of the same transaction as recordin the ack. you can achieve exactly onice.
I don’t think you also need the process step to be idempotent - being part of a transaction (which can also contain the id for de-dupe) is sufficient. That does mean though that the process step be reversible - so the txn can be aborted.
I think if your process step is “fire the missiles”, you’ll never have exactly once.
Transactions work only if the changes they are doing can be rolled back. That is not exactly the same as requiring idempotence, but your process may need to be restarted after failing at an arbitrary point.
That is the whole disagreement of what “exactly once means”, as far as I understand this. One cannot guarantee “exactly once” on arbitrary conditions, what this (and other recent works that claim to achieve exactly once) do is to provide a set of restrictions under which you have those guarantees.