I find these kind of visualization a little misleading if not outright dangerous - I live in a city with 1.5m people and it’s probably one of the very green dots. Which means someone in this city has 650 Mbit or a GBit while probably half the city is limited to 50-100 Mbit. I know, I know, there are enough regions or countries where this sounds pretty good, but taking the max and not even average or median is just kinda horrible imho.
they are great on large scale data where you don’t want to send MBs of GeoJSON to the client.
this is a great key point worth emphasizing, you don’t only save on the data transfer, but also in performance, both to parse and to render a giant GeoJSON file.
I’m interested too! I haven’t gotten around to doing a proper comparison yet myself due to other deadline pressures, but I owe my coworkers one soon. Happened upon this article while researching options. This article indicates that cppyy works well with templates, maybe that’s the killer app? I look forward to experimenting with the options but pybind was another top contender for me. My team has a mixed C++/Python codebase so we could definitely use a binding solution that works both ways. We haven’t had a pressing need for it quite yet though.
According to the article the data comes from a company collecting precise movements using software slipped onto mobile phone apps. So maybe installing/using less apps helps somewhat…
I would assume that disabling location permission for apps would also be sufficient. I can understand why maps needs my location, but not much else.
That said, the article does say the weather channel was also at this, and I guess a lot of people would give this more trust as its a default app on ios.
the weather channel was also at this, and I guess a lot of people would give this more trust as its a default app on ios.
Is it? I know there’s a weather app by default, but I thought the Weather Channel app was a separate download (not a regular iOS user here). I uninstalled The Weather Channel on my Android device after they were outed as scraping contacts and selling that off. No reason to install anything of theirs after that and now this as well.
Interesting write up and great to learn about where the various projects are going. Has anyone tried computationally intensive tasks with this in the wild? This mentions some: https://stories.dask.org/en/latest/
I used Dask at previous job, for multiple steps of gene sequencing pipeline. Worked well for us.
Modin is similar to Dask’s Pandas support, I think. However, Dask is much more general purpose than modin: while it does have Pandas Dataframe emulation (and numpy array emulation), you can also run arbitrary calculations on arbitrary data, which is what we did: https://docs.dask.org/en/latest/bag.html
Also, Dask has very nice single-machine mode. Possibly ray (which modin uses) has this too, not sure.
I agree, compared to something like Celery it’s a much nicer development experience (of course, use-cases are slightly different). No need to setup any external services like message queues etc. And you can even choose between creating a local cluster by using processes, or threads. I’ve started to use Prefect recently, which uses Dask under the hood. I was very pleasantly surprised how little friction there was before I had something usable.
Holiday is over, back to work. We have a new machine with two RTX 5000s and plenty of memory/cores. Our project was running against the limits of our Tesla M60 (dual GPU) and Tesla K20c (an oldie, still good for training feed-forward networks or lighter RNNs). So, there is ample capacity again for new experiments. We also have two new Radeon VIIs, but these are still being set up.
I have a local Hydra instance set up for automatically testing models using some Nix on git commits, but I still have to deploy that properly on a work server (which requires some hoop jumping due to CentOS 7, storage via NFS, etc.).
any more info on the lab setup or development process?
It is mostly quite boring. We use GitHub, do PRs + reviews for everything Travis-CI for CI. Some projects are built using Nix in CI because it is easier to pull in all the dependencies (libtensorflow, Python Tensorflow module, various Rust versions, etc.). The team consists of more senior staff, PhD students, and students. I am very happy that we have been able to grow a team that it both strong scientifically, but also has a good Rust and Tensorflow background. Some public projects are e.g. our Rust packages for training and using embeddings:
Some great advice I got was its a combination of three things and you have to be happy with it. Each of these may have non tangible things beyond financial aspects, work flexibility etc.
What you think you should be paid
What the company thinks you should be paid
What the market is paying
Now given these survey the market, the best way I have done this is using the Visa application data for the US. It’s very transparent… https://h1bdata.info/index.php
Perhaps put together a vertical herb garden. My last project was a small (2’ x 2’) planter bed which went well, (except I could have used more screws perhaps) so I want to try this slightly more challenging project.
I want to install an alternator in my Toyota, first time doing that job. Also want to clean my HVAC coils. I’ll probably catch up on GSL (StarCraft II), watch Solo, cancel HBO, start Wolfenstein II. Read more Mastering PostgreSQL, that or a Python ebook. As long as I get that alternator done, I’ll be happy. Advice for a diy mechanic welcome.
How about maps? Anyone got pet peevs about it? I know Open Street Map (OSM) is OK. But also for sharing locations and routes plus calendar integration it’s hard to deplatform (maybe not a word?) from “google”.
What are the things you look for in an algorithmic open source project?
I look for how well it is documented. If it’s a library, the API must be documented (‘just read the source’ is a silly thing to tell users who might want to use the thing you made, for example). If there are examples of how to use your thing, that’s even better!
I also look for how active the project is, how commits happen over time, how responsive the developer is to user questions/issues, what the release cycle is like, what type of testing (if any) is performed prior to making releases, and what the project license is (and if it’s compatible with the license I chose for my application).
Are there any assumptions that get made that make a project less attractive?
See the first response about documentation :P. Other than that, other things that might make me walk away are (in no particular order:
if the developer seems generally hostile towards users reporting issues or trying to make contributions
if the project has lots of reported issues (that seem like issues I would care about) but it appears the developer has lost interested and is either not resolving them or is not addressing community attempts to resolve them by reviewing/merging patches from others
if the developer does not pick a license for the code (which in may areas I guess it defaults to ‘look but don’t touch’), or if they pick a license I do not agree with, or if it’s a library they pick a license that is incompatible with the license I chose for my application.
if there’s little or no testing of code before releases are tagged
What are your best practices?
Document all the things (I’m no expert here, but trying to improve)
Not all users are as familiar with your thing as you are, so don’t assume they are if their questions seem too ‘simple’ or ‘basic’. Everyone has to start from somewhere, and some users are more experienced in learning on their own than others. Try to be nice :)
Don’t let open issues linger. If you plan to fix/address it, say so. If it’s not relevant, close it. If you planned to address it but no longer plan to, say so and close it. Re-opening issues is cheap. You can also publicly ask for help with issues, but this may not always work (kind of depends on how much you might be willing to help newbies get familiar enough to contribute, etc, and maybe some luck..)
Good unit tests also helps tremendously. Contributors can easily see when something they want to contribute breaks stuff, and in some cases it can even serve as examples for how to use your thing.
Pick a good open source license that reflects your goals for the project and protects rights of users who contribute to your project. I’m partial to copyleft-leaning licenses (e.g. GPL), which attempt to make sure forks of the thing continue to be open source. Others prefer MIT-style licenses. There are lots, it helps to do some research just so you know what is out there.
Would you rather release it now (in an OK state) v. later (in an excellent state)?
There’s no better time than now! Document the shortcomings (e.g. in a TODO, readme, or by opening bugs/enhancements in your tracker of choice). You can also use things like semantic versioning to convey to folks that your thing is not ready (e.g. 0.X version), or when you introduce breaking changes. If your thing is super cool and you get a lot of folks interested, some may even help move it forward. Don’t count on this though, but it is a nice bonus sometimes when it does happen. There’s a 0% chance of getting this bonus if you don’t release anything publicly.
I find these kind of visualization a little misleading if not outright dangerous - I live in a city with 1.5m people and it’s probably one of the very green dots. Which means someone in this city has 650 Mbit or a GBit while probably half the city is limited to 50-100 Mbit. I know, I know, there are enough regions or countries where this sounds pretty good, but taking the max and not even average or median is just kinda horrible imho.
My town has widespread access to 100Mbit+ fibre, including GBit where I used to live.
The city centre however has residential access to ADSL only. In my apartment that is about 8Mbit down and 0.6 up. I had to switch to a 4G SIM
This is probably a resolution thing.
Did you try zoom in? It’s quite detailed, should be within a block or so…
The challenge is the aggregation at different zoom levels. Any suggestions on what you’d like?
Guess I should’ve taken a look on a proper machine and not something with potato graphics, I didn’t zoom in because 0.5 FPS.
mmm nice just went through trying this out the other day.
One point to mention is that they are great on large scale data where you don’t want to send MBs of GeoJSON to the client.
Here is the output I made to visualize potential broadband connectivity - http://broadband.cat/mississippi.html
this is a great key point worth emphasizing, you don’t only save on the data transfer, but also in performance, both to parse and to render a giant GeoJSON file.
Nice work, thanks for sharing!
I’ve done the ASUS ac-1900 reflash and it works nicely. You can pick these up cheap as t-mobile devices.
You can also combine multiple or a 86u for mesh wifi and cat5 backhaul work.
https://www.bayareatechpros.com/ac1900-to-ac68u/
Interested to know what features made this the choice. Tried pybind and it seemed OK too.
I’m interested too! I haven’t gotten around to doing a proper comparison yet myself due to other deadline pressures, but I owe my coworkers one soon. Happened upon this article while researching options. This article indicates that cppyy works well with templates, maybe that’s the killer app? I look forward to experimenting with the options but pybind was another top contender for me. My team has a mixed C++/Python codebase so we could definitely use a binding solution that works both ways. We haven’t had a pressing need for it quite yet though.
Nice thanks for the link.
Beautiful Visualisations here. I wonder if using non-smart phones can help in anyway.
According to the article the data comes from a company collecting precise movements using software slipped onto mobile phone apps. So maybe installing/using less apps helps somewhat…
After reading this, I deleted half the apps from my phone and disabled location services for most of the ones that were left.
I would assume that disabling location permission for apps would also be sufficient. I can understand why maps needs my location, but not much else.
That said, the article does say the weather channel was also at this, and I guess a lot of people would give this more trust as its a default app on ios.
Yes and no.
The wifi MAC address method is also becoming quite prevalent. If your phone has wifi just enabled it can be pinged. [1]
Then over the cell network your signal can be triangulated.
True, thats what skyhook does as well. Interestingly (and this might show a conflict of interest within Google) Android has started using random mac’s for wifi. https://source.android.com/devices/tech/connect/wifi-mac-randomization
Is it? I know there’s a weather app by default, but I thought the Weather Channel app was a separate download (not a regular iOS user here). I uninstalled The Weather Channel on my Android device after they were outed as scraping contacts and selling that off. No reason to install anything of theirs after that and now this as well.
The bar has been set. This is clearly a leader board… first developer to make #1 in all categories, can claim the “glory”.
Interesting write up and great to learn about where the various projects are going. Has anyone tried computationally intensive tasks with this in the wild? This mentions some: https://stories.dask.org/en/latest/
Also wondering about comparisons to https://github.com/modin-project/modin
I used Dask at previous job, for multiple steps of gene sequencing pipeline. Worked well for us.
Modin is similar to Dask’s Pandas support, I think. However, Dask is much more general purpose than modin: while it does have Pandas Dataframe emulation (and numpy array emulation), you can also run arbitrary calculations on arbitrary data, which is what we did: https://docs.dask.org/en/latest/bag.html
Also, Dask has very nice single-machine mode. Possibly ray (which modin uses) has this too, not sure.
| Also, Dask has very nice single-machine mode.
I agree, compared to something like Celery it’s a much nicer development experience (of course, use-cases are slightly different). No need to setup any external services like message queues etc. And you can even choose between creating a local cluster by using processes, or threads. I’ve started to use Prefect recently, which uses Dask under the hood. I was very pleasantly surprised how little friction there was before I had something usable.
Holiday is over, back to work. We have a new machine with two RTX 5000s and plenty of memory/cores. Our project was running against the limits of our Tesla M60 (dual GPU) and Tesla K20c (an oldie, still good for training feed-forward networks or lighter RNNs). So, there is ample capacity again for new experiments. We also have two new Radeon VIIs, but these are still being set up.
I have a local Hydra instance set up for automatically testing models using some Nix on git commits, but I still have to deploy that properly on a work server (which requires some hoop jumping due to CentOS 7, storage via NFS, etc.).
Also writing up some recent work.
any more info on the lab setup or development process? Sounds like you are upto some interesting model creation?
It is mostly quite boring. We use GitHub, do PRs + reviews for everything Travis-CI for CI. Some projects are built using Nix in CI because it is easier to pull in all the dependencies (libtensorflow, Python Tensorflow module, various Rust versions, etc.). The team consists of more senior staff, PhD students, and students. I am very happy that we have been able to grow a team that it both strong scientifically, but also has a good Rust and Tensorflow background. Some public projects are e.g. our Rust packages for training and using embeddings:
and a neural sequence labeler:
https://github.com/danieldk/sticker/
Some recent work:
And a lot more stuff cooking ;).
Excellent!
Some great advice I got was its a combination of three things and you have to be happy with it. Each of these may have non tangible things beyond financial aspects, work flexibility etc.
Now given these survey the market, the best way I have done this is using the Visa application data for the US. It’s very transparent… https://h1bdata.info/index.php
I’ve just started my summer holidays.
Is there any suggestions from lobsters on what to do in Basel, Geneva or easily accessible areas around them?
Well I’d say get up in them mountains! Check out the villages and the Jungfrau.
The Bodensee is really nice.
Zweig Media’s JS-based linear programming solver has been available on the web since 2007: https://www.zweigmedia.com/simplex/simplex.php?lang=en
Oh thanks, will check it out. Any idea about how it compares to alternatives, functionality and speed wise etc?
No, but if you learn anything on that front, I’d be interested to hear about it.
I’m off to Dublin, Ireland for the Euro30 optimization conference.
Anyone going or got some recommendations?
Perhaps put together a vertical herb garden. My last project was a small (2’ x 2’) planter bed which went well, (except I could have used more screws perhaps) so I want to try this slightly more challenging project.
NIce! Definitely share the results
Oh no not my secret recipe for…
looks through generated recipes
Super easy homemade water
uh oh I shattered the jar with the boiling vanilla salt mess…
I have the dreaded live coding exercise in a few days - I have been thinking about recursion and the Fibonacci sequence lately - thanks for sharing.
Here is a great run down of that example - https://hplgit.github.io/primer.html/doc/pub/half/book.pdf
What’s that?
Getting ready for my conference talk at INFORMS.
I want to install an alternator in my Toyota, first time doing that job. Also want to clean my HVAC coils. I’ll probably catch up on GSL (StarCraft II), watch Solo, cancel HBO, start Wolfenstein II. Read more Mastering PostgreSQL, that or a Python ebook. As long as I get that alternator done, I’ll be happy. Advice for a diy mechanic welcome.
Cool! Step 1 disconnect battery.
Everything else will depend on make and model ;)
How about maps? Anyone got pet peevs about it? I know Open Street Map (OSM) is OK. But also for sharing locations and routes plus calendar integration it’s hard to deplatform (maybe not a word?) from “google”.
A word, but usually used for like, banning some unpleasant hateful person from a platform.
Came across this - https://eprints.qut.edu.au/113630/1/article.pdf
Here are my 2 cents:
I look for how well it is documented. If it’s a library, the API must be documented (‘just read the source’ is a silly thing to tell users who might want to use the thing you made, for example). If there are examples of how to use your thing, that’s even better!
I also look for how active the project is, how commits happen over time, how responsive the developer is to user questions/issues, what the release cycle is like, what type of testing (if any) is performed prior to making releases, and what the project license is (and if it’s compatible with the license I chose for my application).
See the first response about documentation :P. Other than that, other things that might make me walk away are (in no particular order:
if the developer seems generally hostile towards users reporting issues or trying to make contributions
if the project has lots of reported issues (that seem like issues I would care about) but it appears the developer has lost interested and is either not resolving them or is not addressing community attempts to resolve them by reviewing/merging patches from others
if the developer does not pick a license for the code (which in may areas I guess it defaults to ‘look but don’t touch’), or if they pick a license I do not agree with, or if it’s a library they pick a license that is incompatible with the license I chose for my application.
if there’s little or no testing of code before releases are tagged
Document all the things (I’m no expert here, but trying to improve)
Not all users are as familiar with your thing as you are, so don’t assume they are if their questions seem too ‘simple’ or ‘basic’. Everyone has to start from somewhere, and some users are more experienced in learning on their own than others. Try to be nice :)
Don’t let open issues linger. If you plan to fix/address it, say so. If it’s not relevant, close it. If you planned to address it but no longer plan to, say so and close it. Re-opening issues is cheap. You can also publicly ask for help with issues, but this may not always work (kind of depends on how much you might be willing to help newbies get familiar enough to contribute, etc, and maybe some luck..)
Good unit tests also helps tremendously. Contributors can easily see when something they want to contribute breaks stuff, and in some cases it can even serve as examples for how to use your thing.
Pick a good open source license that reflects your goals for the project and protects rights of users who contribute to your project. I’m partial to copyleft-leaning licenses (e.g. GPL), which attempt to make sure forks of the thing continue to be open source. Others prefer MIT-style licenses. There are lots, it helps to do some research just so you know what is out there.
There’s no better time than now! Document the shortcomings (e.g. in a TODO, readme, or by opening bugs/enhancements in your tracker of choice). You can also use things like semantic versioning to convey to folks that your thing is not ready (e.g. 0.X version), or when you introduce breaking changes. If your thing is super cool and you get a lot of folks interested, some may even help move it forward. Don’t count on this though, but it is a nice bonus sometimes when it does happen. There’s a 0% chance of getting this bonus if you don’t release anything publicly.
Thanks, I like the idea of utilizing a tracker for the work to be done.
Also agree with the taking feedback from any user at any level in a positive way to be a good start.