-
What did your query plan look like? Did you consider a materialized view or any other technique that’d let you solve this in the database?
-
I also was curious if they’d exhausted the SQL options. Appears from the brief Group Builder example, they could’ve generated raw SQL queries that hit all the right indexes, without doing the unions and group by.
As a query builder, this Group Builder looks pretty slick. The technical discussion about their history and solutions is also interesting and instructive. The Go language propaganda is kind of distracting from an other wise interesting article. Not saying that Go didn’t provide the benefits described, it just seems a little irrelavant.
-
It looks to me as if they are just constructing boolean expressions in the web interface, e.g., the first screenshot corresponding to (as far is I understand it):
... WHERE gender != 'male' AND ( age > 50 OR censor_rating_r18 < 0.2 )(it’s not clear to me how they actually store/process the censor rating, so i’m just guessing here)
So it’s a matter of translating from the interface to a boolean expression with “Subgroup”s mapped to parentheses and comparisons based on exclude/include specification.
I don’t really see the necessity for unions, or am I missing something?
-
-
-
The graphs are especially interesting if one uses the Random Article feature Wikipedia offers, for example:
- 7 degrees of separation from Puma.NET to Physopyxis cristata
- 5 degrees of separation from 1979 Sun Bowl to Seoci, Croatia
- 5 degrees of separation from Zesiini to Frasne-les-Meulières
- 7 degrees of separation from COMIT to Antichloris eriphia (this one was intentionally chosen, since links between biology and CS seem to always turn out interesting)
but it’s not always that fascinating, since sometimes there just aren’t enough links:
- 5 degrees of separation from T with stroke to Multifurca zonaria
- 6 paths with 5 degrees of separation from Stephanos Sahlikis to Mohsen Ghaedpouri
The only thing I’m afraid of it that this isn’t spamming the Wiki servers too much. But from what I’m reading out of the network trace, they’re only downloading the pictures, and have the article links cached somewhere. It’s certainly better than on of the first Perl scripts I wrote to check the “Clicks to Hitler” Game, which will probably be unplayable from now on.
Edit: I missed the GitHub link, and it says there that they use a database dump wikipedia updates each month.
Edit 2: Also interesting, if one take the last link from the first list, and reverses the two, one get’s a much more boring graph: 4 degrees of separation from Antichloris eriphia to COMIT.
-
I find relations with many links boring, if you have enough links it’s obvious you can relate anything to to anything else, what I find very funny though are short links:
Anime to obesity in two steps.
-
-
-
Proving Peter Thiel right about AI:
-
-
-
The GitHub repository itself has a section with interesting combinations.
-
enplotz
6 months ago
|
link
| on:
The World's Most Mind-Bending Language Has the Best Development Environment
After browsing the wiki for a bit and discovering Jd (a J columnar RDBMS), I remembered K (and kdb+), as both the syntax and the immediate mention of a columnar RDBMS reminded me of it. So it seems, both are APL-inspired.
-
You can also locate military bases in sparsely populated areas: https://twitter.com/tobiaschneider/status/957317886112124928
-
A couple of German universities use it to give researchers easy access to cloud computing: https://www.bw-cloud.org/en/project
-
I never understood the popularity of solarized. It lacks contrast and makes my eyes hurt.
-
-
-
The implication that the goodness of something so subjective can be quantified really irks me. However, I think a lot of people ate this up, as I’ve seen people non-ironically citing this as a reason it is good.
-
I hear it’s Cave Johnson’s favorite IDE color scheme.
-
-
-
-
I think it depends a lot on lighting. I use the dark theme at evening/night, and don’t have a lot of light in the room. More contrast rich themes like Monokai hurt my eyes in that setting.
The Solarized theme that comes with Visual Studio Code actually uses a base color with more contrast than the original design. But I find that rather annoying in the light theme, especially since they also use bold.
-
More contrast rich themes like Monokai hurt my eyes in that setting.
That makes sense. It’s funny, at night I will continue using typical white-on-black high-contrast color schemes but just drop the monitor brightness a lot if I happen to be hacking away in the dark. Usually I just turn the lights on, though.
-
For me both variants of Solarized are difficult to read in the daytime on a nice display and borderline unusable at any time of day on a low-end display. On the other hand, I find high-contrast dark themes too harsh, so I tend to use dark themes that are somewhere in the middle (~#999 on ~#222) and higher-contrast light themes (~#222 on ~#f5f5f5).
-
gruvbox dark works well for me :)
-
-
-
-
Agree on the importance of contrast. Lots of color themes are happy to use tons of different colors on things that aren’t completely semantically different (a numeric literal doesn’t always need to stand out a lot) while ignoring the more subtle details such as contrast.
I want the attention to detail Solarized has, but with more contrast, and something besides an ocean or a piece of parchment as the background. I’ve been using a version of Github’s color scheme in my editor for awhile, but have yet to really find a color theme that I really like.
-
-
When my students draw diagrams (E/R, Venn, …) using Preview.app’s markup tools I cannot really blame them, as it essentially gets the job done. However, using 100 circles to fill another circle with a color (I’m not making this up) does not quite seem right to me. Although I would have preferred a proper drawing tool (actually LaTeX/TikZ), which would look much more professional in my opinion, they solved the exercise. I still find it difficult to argue why I think the presentation is as much contributing to solving the exercise as the content. After all, we are communicating thoughts/information, I guess?
-
I end up using
ncdua lot in practice, and it solves my problems.I used to use a TreeMap program (on Windows) called SpaceMonger, which was both pretty and useful. I always wanted something like that on Linux. There are dozens of similar programs, but somehow the visualization was not as good.
ncdu doesn’t really visualize much, but the key it is that it’s fast enough and shows enough to get the job done quickly (identify the very few big files you should delete).
Somewhat of a tangent, if I ever get around to writing a visual version of ncdu, I would now use flame graphs and not treemaps:
http://www.brendangregg.com/blog/2017-02-06/flamegraphs-vs-treemaps-vs-sunburst.html
I mentioned this in another comment: treemaps are really hard to get right from the user perspective. The preponderance of programs that are “not as good as SpaceMonger” shows this. I think that flame graphs will work perfectly for this. But admittedly I haven’t tried it yet.
-
I’ve thought about using flamegraphs for it before too, Brendan has an interesting post where he tried it and it seems to be a decent way to visualise it - http://www.brendangregg.com/blog/2017-02-05/file-system-flame-graph.html
-
-
-
Serious question: isn’t it super scary to run “real” things on SQLite? Or are things like backup tools also well developed for SQLite at a “reasonable load”.
-
-
-
-
The test suite is quite substantial as well.
-
-
-
[…] Stash is extremely useful when someone randomly asks you to check out another branch, but you’re right in the middle of something.[…]
I like to use
git worktreefor that. Let’s me quickly check out any other branch (or create new ones with small hotfixes) without disturbing what I am currently working with or me having to manage k other clones of the same repository.
-
owen
1 year ago
|
link
| on:
Meet the Man Behind ‘Solarized,’ the Most Important Color Scheme in Computer History
As a person with super shitty eyesight I’ve noticed that Solarized is probably the worst color scheme I’ve ever used.
Designers - don’t forget about those of us who need a little extra contrast :)
-
-
Try gruvbox, the most important color scheme in computing history.
-
-
I never understand how most make comments barely visible (even for me still having relatively good eyesight). I mean they’re like the most important part of code, for me anyways. Dozens of high-contrast colors for every keyword, then 0 contrast for comments against the background…
-
-
-
-
Another downside they forgot to mention is that vector images are harder to downsize. With bitmaps, you know exactly what value each pixel will have, whereas with vectors you have to lay them out so the image will still be crisp at small sizes after antialiasing. This is what makes good font design more constrained on low-resolution screens, and why most fonts contained a bitmap for each character before screen resolutions got bumped up.
-
-
The cert is invalid at this site (according to my browser)
-
-
That is a really nice writeup of the project!
-
-
enplotz
2 years ago
|
link
| on:
UW team stores digital images in DNA — and retrieves them perfectly
The cool thing about storage in DNA is the built in redundancy. I once heard a very interesting (although very theoretical) talk about databases in DNA. One hurdle, still, seems to be the slow sequencing speed.
-
This is a very thin wrapper around the psycopg2 library. Not sure why it’s necessary vs. using psycopg2 directly.
-
-
Did this seriously just start downloading more than a hundred megabytes as soon as I opened it? That seems pretty user-hostile.
-
It’s probably way out of the intended scope, but could Mitogen be used for basic or throwaway parallel programming or analytics? I’m imagining a scenario where a data scientist has a dataset that’s too big for their local machine to process in a reasonable time. They’re working in a Jupyter notebook, using Python already. They spin up some Amazon boxes, each of which pulls the data down from S3. Then, using Mitogen, they’re able to push out a Python function to all these boxes, and gather the results back (or perhaps uploaded to S3 when the function finishes).
It’s not /that/ far removed. Some current choices would make processing a little more restrictive than usual, and the library core can’t manage much more than 80MB/sec throughput just now, limiting its usefulness for data-heavy IO, such as large result aggregation.
I imagine a tool like you’re describing with a nice interface could easily be built on top, or maybe as a higher level module as part of the library. But I suspect right now the internal APIs are just a little too hairy and/or restrictive to plug into something like Jupyter – for example, it would have to implement its own serialization for Numpy arrays, and for very large arrays, there is no primitive in the library (yet, but soon!) to allow easy streaming of serialized chunks – either write your own streaming code or double your RAM usage, etc.
Interesting idea, and definitely not lost on me! The “infrastructure” label was primarily there to allow me to get the library up to a useful point – i.e. permits me to say “no” to myself a lot when I spot some itch I’d like to scratch :)
This might work, though I think you’d be limited to pure python code. On the initial post describing it:
If it are just simple functions you run, you could probably use pySpark in a straight-forward way to go distributed (although Spark can handle much more complicated use-cases as well).
That’s an interesting option, but presumably requires you to have Spark setup first. I’m thinking of something a bit more ad-hoc and throwaway than that :)
I was thinking that if you’re spinning up AWS instances automatically, you could probably also configure that a Spark cluster is setup with it as well, and with that you get the benefit that you neither have to worry much about memory management and function parallelization nor about recovery in case of instance failure. The performance aspect of pySpark (mainly Python object serialization/memory management) is also actively worked on transitively through pandas/pyArrow.
Yeah that’s a fair point. In fact there’s probably an AMI pre-built for this already, and a decent number of data-science people would probably be working with Spark to begin with.