No, I read about the algorithms used in other routers and why they where an improvement. I then adapted the algorithm until it could support Werkzeug’s features. There is likely further optimization possible via profiling.
Oh cool… I definitely recommend this book, it’s thin and packed with information. One of the few programming books I can say I’ve read more or less cover to cover. Although it’s aged, Awk is still useful, and it’s one of the few languages that fits in your head!
Are there similar methods or simd tricks for multi-way merges? I have some 32-way merges that take long enough I’ve spent lots of time wishing were faster, but haven’t yet found any articles on making them faster. :)
I would add some examples for each defined term (at the end of each definition). I always get tripped up by this vocab when reading about parsing, so every little hint helps.
If Mobile Data and WiFi are not available, the app stores the data and uploads it after it gets the connectivity. That’s why the timestamp comes from a query parameter instead of the server timestamp.
Hmm. I can’t help but wonder why virtualenv + Docker is necessary in 95% of cases? Why not just install the requirements globally to the Python install… given, you’re in a container and running likely only 1 app… ?
For multi-stage Docker builds, where you have compiler etc. in first stage, and then copy over compiled code (Python C extensions etc.) into second stage image that doesn’t have gcc etc so gives you a smaller image. In this case, installing directly with pip means some files end up in /usr/bin, others in site-packages, so it’s hard to copy everything over. Virtualenv solves that since everything is one directory.
Or on top of that, why bother activating at all? You can always just give the full path to your virtualenv python binary and it’ll know where everything else is.
Was profiling used to pick routing for optimization?
No, I read about the algorithms used in other routers and why they where an improvement. I then adapted the algorithm until it could support Werkzeug’s features. There is likely further optimization possible via profiling.
I’m pretty sure Kernighan does a toy make in awk in this classic book:
https://www.amazon.com/AWK-Programming-Language-Alfred-Aho/dp/020107981X
(maybe you can find it at the library, etc.)
I transcribed it into Python many years ago (and thus learned a bunch of awk).
Very nice. Looks like internet archive has several copies:
https://archive.org/details/pdfy-MgN0H1joIoDVoIC7
Oh cool… I definitely recommend this book, it’s thin and packed with information. One of the few programming books I can say I’ve read more or less cover to cover. Although it’s aged, Awk is still useful, and it’s one of the few languages that fits in your head!
Are there similar methods or simd tricks for multi-way merges? I have some 32-way merges that take long enough I’ve spent lots of time wishing were faster, but haven’t yet found any articles on making them faster. :)
Good question! I have no idea but I did find this in google, which is set intersection: http://www.adms-conf.org/p1-SCHLEGEL.pdf
I would add some examples for each defined term (at the end of each definition). I always get tripped up by this vocab when reading about parsing, so every little hint helps.
Thanks!, I will do that.
Did you have any batching requirements (primarily to cut down on radio time), or does it always send updates in ~real-time?
If Mobile Data and WiFi are not available, the app stores the data and uploads it after it gets the connectivity. That’s why the timestamp comes from a query parameter instead of the server timestamp.
Hmm. I can’t help but wonder why virtualenv + Docker is necessary in 95% of cases? Why not just install the requirements globally to the Python install… given, you’re in a container and running likely only 1 app… ?
System python packages might occasionally conflict with packages in your virtualenv (see https://hynek.me/articles/virtualenv-lives/).
For multi-stage Docker builds, where you have compiler etc. in first stage, and then copy over compiled code (Python C extensions etc.) into second stage image that doesn’t have gcc etc so gives you a smaller image. In this case, installing directly with pip means some files end up in /usr/bin, others in site-packages, so it’s hard to copy everything over. Virtualenv solves that since everything is one directory.
This would be great context to add to the top of the post!
Yeah. People asked this a lot, so going to write another article about that specifically and will then link to it from this article.
… and why not using the already great Python images: https://docs.docker.com/samples/library/python/ ?
Or on top of that, why bother activating at all? You can always just give the full path to your virtualenv python binary and it’ll know where everything else is.
As I discuss in the article, this is definitely an option. However:
The proposed solution suffers from neither problem.
This is great. Are there any similar docs for Presto or Spark? Or docs that compare trade-offs between them?
The video presentation of the paper, is excellent:
https://www.youtube.com/watch?v=yL_-1d9OSdk