1. 1

    Does it include lobste.rs users’ blogs in the search stack?

    1. 1

      Sorry, what are lobste.rs users’ blogs?

      1. 1

        Blogs written by people on lobste.rs :-) See for example here.

        1. 1

          I’ll try to, gradually :) Thanks!

    1. 1

      This is really cool, besides being useful and a good idea, I’ve been using to see what comes up when I search non-dev related things (such as band names or celebrities) and getting hilarious results

      1. 1

        I’m also slightly terrified of entering stuff like “how to kill children processes” into Google.

      1. 1

        No JS is bad feature. Instead of adding more value (autocomplete, autosuggestion, etc) you treated with small benefits of “no js”. A graceful degradation will be better. More features for normal users, less for adepts on “No Script”.

        1. 3

          That’s probably how I’ll proceed from now on. Autocomplete, keyboard shortcuts, etc via js, but never breaking the core functionality.

        1. 1

          –foo excludes word

          I think you mean

          -foo excludes word

          (Something auto-converting hyphen to en dash)

          1. 1

            Yeah, something in my toolchain auto-converts this and it drives me crazy… Fixed, deployed, thanks!

          1. 2

            This is pretty cool! It works better than I would have expected. I just made a pull request for Common Lisp.

            1. 2

              Merged! Thank you.

            1. 6

              Definitely some potential. But the search results will need some prioritization before it truly hit the mark. Searching for “mongoc_client_pool_new” I could not see the documentation page on the first page of result. And searching for “std::vector” came up with a bunch of Stack Overflow links.

              1. 4

                Agreed, some prioritization is needed, maybe even by users themselves (voting?).

                Also, have you tried narrowing the search to “docs” only?

                1. 4

                  I had not but I just did. It is better, but still needs some refinement from what I would expect.

                  Still, keep up the good work!

                  1. 1

                    I didn’t see that narrowing the search was possible at first. In Firefox, your form select { appearance: none } CSS style hides the disclosure triangle, so the “All” part of the search bar doesn’t look at all like a menu. It looks more like a button, which makes it feel janky that a pop-up menu opens after I click it.

                    If you’re going to hide the browser-default disclosure triangle, I suggest adding your own (which can match your site’s theme) so the menu still looks like a menu. Alternatively, if you really want to make the menu look like a button, I suggest giving it a darker background color and making its background color change on hover.

                    1. 1

                      Got it, I’ll work on it, thank you!

                    2. 1

                      Also, have you tried narrowing the search to “docs” only?

                      I do not understand this reasoning. If this is a search engine for programmers, then looking for “std::vector” must give you fist the most official documentation of that class, no matter what.

                  1. 1

                    Might be interesting to add devdocs.io docs collection as a source. It has a really large number of docs collected.

                      1. 2

                        It doesn’t work though. It only finds the documentation groups, but cannot find symbols in the documentation, e.g. I’d expect https://quickref.dev/search?q=site%3Adevdocs.io+godot+object.connect&type=all to point to https://devdocs.io/godot~3.1/classes/class_object#class-object-method-connect

                        1. 2

                          I could add a bang for devdocs, though. Maybe, !dd ?

                          1. 1

                            Hmm, google yields similar results with site:devdocs.io godot object.connect. Maybe devdocs loads that content dynamically and somehow doesn’t let crawlers to scrap the data… Quickref uses Bing’s index.

                      1. 2

                        Is the “explicit” vs “implicit” distinction a common one for search engines? I’m not familiar with it. I’m guessing that explicit means “literal” match, implicit means some kind of fuzzy/“guess what I mean” matching?

                        1. 5

                          Most search engines assume you’re looking for something common, and thus:

                          • “Fix” queries with typo & misspelling correction based on string edit distance and term frequency
                          • “Expand” queries by projecting your terms into some kind of topic space (embedding) and giving you results that score highly on nearby terms

                          These are great for natural language queries for popular resources, and really terrible for precise, niche queries.

                          1. 1

                            Hmm, I haven’t heard those terms being used either in this context. I definitely would find it a little more clear if it used the terminology you mentioned instead.

                            1. 4

                              Thanks, I’ve updated the terminology on the frontpage, and it should go live in a few minutes.

                          1. 32

                            There’s a huge vacuum in the search space right now since Google is so broken. This has so much potential!

                            Could you tell us a little about how the search engine is implemented?

                            1. 20

                              Sure!

                              I use Bing search services (via MS Azure). Quickref itself is a simple RoR app which loads metadata from https://github.com/freetonik/quickref.dev and talks to Azure, while maintaining privacy and not transferring any client info to MS.

                              Bing is pretty good, since I didn’t have to crawl and index the sites myself (the first iteration was built with scrapy + elasticsearch), and the results are generally ok, but there are limits to the number of sites in the index and the number of priority adjustments. So, if this project proofs to be of value, I’d probably need to go back to custom crawl/search setup, hopefully with the help of others. I’ll definitely open source the app.

                              1. 4

                                Very interesting, I had no idea that Bing provides that kind of service. I think it’s good to use their service for now; for the users of a search engine the value should be in the search results, not in how the results are computed. The results are good as far as I can see, I’m finding lots of things I never knew about!

                                If you have some thoughts about it, could you provide some guidance on what sites belong in the index and what is out of scope?

                                1. 4

                                  If you have some thoughts about it, could you provide some guidance on what sites belong in the index and what is out of scope?

                                  Yeah, I’ll put that info to the github page. In short, I think it’s a good idea to stick to these:

                                  • official websites and docs
                                  • community-driven docs
                                  • private blogs
                                  • blogs of engineering departments of companies

                                  The borders are vague though, it’s a bit tricky to define. Like, I feel like dev.to is okay, but something like programiz.com is not…

                              2. 14

                                There’s a huge vacuum in the search space right now since Google is so broken.

                                Absolutely yes. I believe that nowadays Google presents the most popular results and not the most accurate anymore. This is especially hard for programmers because we sometimes search for very narrow and specific things where accuracy actually matters. You can still find your ways around this (I call this search skills) but it seems to get harder recently.

                                Maybe it’s time to have more specialized search providers / search engines in the future for different fields or purposes.

                                1. 11

                                  One idea I had floating around my head is to seed a search engine with this human-curated content. These “awesome” lists have been appearing to fill a hole in Google, IMO.

                                  https://github.com/sindresorhus/awesome – 133 K stars

                                  https://github.com/vinta/awesome-python – 82K stars

                                  https://github.com/ziadoz/awesome-php - 24K stars

                                  https://github.com/aalhour/awesome-compilers – 5K

                                  Wikipedia is also another obvious source of human-curated content (and one that Google heavily relies on to answer many queries)


                                  It’s like a distributed Yahoo on top of Github! (if anyone even remembers what Yahoo did originally …!) Human curation rather than algorithmic.

                                  Basically the theory is that github collaboration prevents spam and is a filter for quality. Spam and dupes are the #1 problem you’ll run into when creating a search engine. And I think the web is full of pages that just BARELY meet the Google threshold of “not spam”. But those pages don’t make it onto these lists…

                                  So it should be an easier ranking problem in theory. Too bad I don’t have years to work on this :-/

                                  1. 3

                                    I mean, that’s how DMOZ and Yahoo worked in the Before Times™.

                                    1. 2

                                      If we think of a search engine as a shared view into a projected subset of the web and not a gate keeper, then the search service itself should be a form of collaboration so that groups of people can train/code/filter a specific hyperplane in a high dimensional space.

                                      There are plenty of niche industries that would be better served by industry specific search services.

                                      Combine reddit, full text search, rss/federated queries. Basically a community workbench for creating custom search and presentation services.

                                1. 2

                                  This is so cool! I just tried this for the issue I’m working on now, and it surfaced a relevant github issue I haven’t seen yet!

                                  I think this really fills a need. I’ve been finding myself using !g on DuckDuckGo a lot, since google’s results are a lot more relevant for dev related searches than DDG’s. This looks like a far more privacy-friendly option, with better results too!

                                  The only feedback I have is to pad the search results to appear in the center of the screen on wide monitors like most other search engines do.

                                  1. 1

                                    Thanks! I’ll work on improving the layout soon. Btw, maybe we should also add the !g bang :-)

                                  1. 2

                                    Is it open source?

                                    1. 7

                                      I too find it interesting, but am somewhat disappointed that only the metadata is public, while the site itself seems closed source. Do you plan to change that?

                                      1. 6

                                        Probably, but there’s not much there anyway. The search engine is Azure’s Bing services, and the app itself is just a simple RoR app.

                                        1. 4

                                          Ah, I see. For some reason I was hoping you had been working on your own search-engine implementation, but I guess that’s not someone one should expect from a side project.

                                          1. 1

                                            It is a good starting point, and changing the user expectation is already a huge gain, giving more to margin to similar projects to maneuver with later-on.

                                      1. 1

                                        Neat idea, interested in seeing where this ends up.

                                        One big thing that’s missing is issue trackers. A lot of folk knowledge only resides in various issue trackers.

                                        1. 1

                                          Atm, “repos” filter includes github, gitlab, sr.ht and sourceforge (https://github.com/freetonik/quickref.dev/blob/master/sites/repositories.txt). Are there other issue trackers I should add?

                                          1. 2

                                            Oops, sorry. I guess the issue is then “it didn’t order search results in way I expected”, then.

                                            I maintain somewhat obscure logging library named eliot, but searches for eliot logging python pandas do not return the issue in the Eliot issue tracker covering Pandas support. Plausibly too hard to get this right in a general search engine, so not sure it’s a flaw or just a hard-to-fix edge case.

                                            Apache’s JIRA might be useful.

                                            1. 1

                                              Ah, seeing that it’s driven by Bing, my guess is Bing (and definitely Google) considers individual GitHub issues in less popular repos to be not worth indexing.

                                        1. 2

                                          re video. Do you have slides or a transcript for this?

                                          re first line about limited domain/application. I barely know anything about this subject. The Humies do make me wonder if you’re right about how limited it is, though. I figured, like deep learning, it’s more limited by your hardware resources and time available than anything else.

                                          1. 2

                                            Slides here → https://speakerdeck.com/rakhim/genetic-programming-is-waiting-for-better-tools

                                            I barely know anything, too, this talk was a lot of assumptions and just thoughts. I wonder what you mean by “limit” in this context. Deep learning being limited by computing resources or time means that with enough resources it can do what?

                                            1. 1

                                              Thanks for the slides. I’ll check them out later. The comment about limited was in the Youtube description:

                                              “Genetic programming seems to be applicable in a very limited domain.”

                                              The one thing I remember most about GP was the Humies. It seemed like they applied it to a lot of areas. So, I was asking what made you think the highlighted statement was true.

                                              1. 1

                                                Just finished the slides. Ok, yeah, GP isn’t doing whole applications at the moment. That’s program synthesis. It’s definitely extremely limited in practice compared to what programmers do. Personally, I think it will be done iteratively like humans do it even if evolved. Gotta somehow do multiple layers of decomposition. Even the deep learning systems seem to do that inside.

                                                Favorite part is the Ferris Wheel of Software Engineering. That was great. :)

                                            1. 12

                                              NO tags, categories, taxonomy

                                              I get that maybe you dont like Hugo because it has too many features. In general I prefer smaller faster software, many times at the expense of features.

                                              But to me taxonomy is essential, its the killer feature, its what separates a real “SSG” from the others. Jekyll lack of taxonomy is literally the one reason I stopped using Jekyll:

                                              https://github.com/jekyll/jekyll/issues/6952

                                              It was a pain to switch to Hugo, but that one feature made it worth it. Before, I would spend so much time deciding on a category for each post. Since the categories were assigned via folders, you could only choose one category per post. Now with Hugo, I can put one, two or how many I want categories on each post.

                                              1. 7

                                                In soupault, I went for allowing the user to extract and export any metadata from pages to JSON and make their own taxonomies from it (like microformats). But, the reason it needs no front matter is that it can look inside HTML.

                                                If “no front matter” is a hard design goal, it may be tricky to apply that idea to underblog. But, maybe there are other “natural” metadata sources one can use instead? Random idea: put a page in a subdirectory to set its category, and symlink it to other subdirectories to add secondary categories.

                                                In any case, the great thing about SSGs compared to server-side web software is that you can safely use an unmaintained SSG. Their proliferation is not harmful and stopping updating/developing further is not an irresponsible act. If for some people underblog’s feature set and workflow is exactly what they want, nothing wrong with it.

                                                1. 1

                                                  Random idea: put a page in a subdirectory to set its category, and symlink it to other subdirectories to add secondary categories.

                                                  This is interesting, thank you. I do believe this sort of thing fits nicely with the underlying philosophy, because it introduces no additional complexity and is absolutely transparent. Love it. Added an issue to discuss with other contributors.

                                                  Thank you!

                                                  1. 1

                                                    Good luck getting that working on Windows…

                                                2. 3

                                                  I understand, taxonomy is essential to me too, in many cases. That’s why I’m using Hugo on three different projects.

                                                  But every once in a while I need to make a small blog. Or recommend a simple way to generate a blog from Markdown. And I can’t really recommend neither Jekyll nor Hugo when the person wants to basically make a site out of a bunch of md files. I don’t want to dump config files, front matter, layouts/themes and plugins/dependencies on them. At that point, I’d rather recommend them to try Ghost or something.

                                                  The goal of underblog is to provide a way to do it without learning anything new. No new concepts, no multi-layered processing, etc.

                                                  So, while taxonomy is essential for many projects, the mental and technical overhead of Hugo/Jekyll is also a thing to consider. Underblog is not an alternative to Jekyll or Hugo, as much as a bicycle is not an alternative to a container ship.

                                                  1. 1

                                                    Re Underblog. Love the name. :)

                                                    1. 2

                                                      Thanks :) My first idea was “shyblog”.

                                                  2. 2

                                                    Time to plug my static site generator :) https://github.com/xiaq/genblog

                                                    It is also in the realm of minimalist static site generator, but it generates an index page for each category. Besides this, there are 2 other differences with the tool in the post:

                                                    • It does not auto discover posts, but always relies on an index.toml file, which is a “manifest” of everything. This is perhaps similar to many established generators, but it goes to the extreme of not assuming any filename (other than index.toml) and require every filename to be specified in index.toml.

                                                    • It does not compile Markdown; rather, it expects each post to be HTML fragments and merely inserts them verbatim into the template. You are free to use whichever Markdown compiler you like. I am not aware of any established generators that does this; and IMO it is the correct design choice to decouple from Markdown compilation.

                                                    I haven’t written any doc for this tool yet, but if you are interested, take a look at https://github.com/elves/elvish/tree/master/_website, starting with Makefile (which does the Markdown compilation using a combination of pandoc and some adhoc macro stuff) and index.toml (the manifest file).

                                                    1. 1

                                                      Huh, I have tags and categories on my Jekyll blog. Although the tags I guess I wrote custom plugins for, ..and I also wrote a custom Generator for the category pages .. and I use jekyll-pagination. 😅 Jekyll is a bit more like Jenkins; it comes with little in core and you need plugins to make it really do anything. I don’t really have a problem with that architecture honestly.

                                                      To be fair to the author, it looks like this is a little experiment to get a custom generator up for some personal or professional site. Going back, I really wish I had written my own instead of using Jekyll. It gave me some things out of the box or with plugins, but I was wrestling a lot with it .. and liquid templates are garbage!

                                                      If this is a small personal project, I’m sure it will grow to gradually fulfill the author’s needs.

                                                      1. 0

                                                        This is pretty much what I wanted to say. Jekyll is minimalist, but, it’s Ruby, and easy enough to extend.

                                                        For categories I’ve been using this plugin0, last updated in 2012 but still works. It generates a page for each category. I also have a page with some Liquid I wrote that lists every category1.

                                                    1. 5

                                                      Welcome to the club of not-invented-here! All of us went for their own static blog generator, the more the merrier!

                                                      Don’t forget to register it to staticgen.com!

                                                      1. 2

                                                        I mean, this one will eventually rule them all :D

                                                        Thanks, wasn’t aware about staticgen.com!

                                                      1. 1

                                                        I really enjoyed this. Even though I’ve been programming for years now, I found the video really helpful, and I immediately sent it to a friend who’s currently in a boot camp. Great work! I’d love to see more in this vein.

                                                        1. 2

                                                          Thank you!

                                                          If you have any topic suggestions/ideas, please let me know.

                                                          1. 1

                                                            Yeah! Maybe something with recursion using a russian nesting doll and disguised jump cuts? Only half an idea, though.

                                                            1. 2

                                                              I actually had made a video on recursion in the past, with boxes generating other boxes based on blueprints :) https://www.youtube.com/watch?v=vLhHyGTkjCs

                                                              1. 1

                                                                Ha, beat me to it!

                                                                What about sorting algorithms with refrigerator magnets?

                                                                1. 1

                                                                  Hmm, could you elaborate, please?

                                                                  1. 1

                                                                    Sure. It’s a half-idea, but my thought was to explain something like bubble-sort, by re-arranging the order of stacked magnets on a refrigerator. Maybe something like these that have different colors, so you can see visually how the sort works based on color groupings https://www.amazon.com/Tiergrade-Assorted-Classroom-Refrigerator-Whiteboard/dp/B01K70O2HO?ref_=Oct_DotdC_3i1i9_0_09c36237&pf_rd_p=89def296-fb7d-550c-8a3f-9de7cb1ee35d&pf_rd_s=merchandised-search-5&pf_rd_t=101&pf_rd_i=3737161&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=2N3HRCNB9X0DX1K6ZZ1M&pf_rd_r=2N3HRCNB9X0DX1K6ZZ1M&pf_rd_p=89def296-fb7d-550c-8a3f-9de7cb1ee35d

                                                                    1. 2

                                                                      Interesting, I’ll think about it! Thanks!

                                                        1. 2

                                                          Thanks for sharing! It’s great to see the status-quo of text being challenged, and Lisp/Scheme/Clojure are definitely good candidates to continue the reflection on post-text programming UI. Trees as opposed to text would make so much more sense (especially for merging, as mentioned in the video) – but we do already have plenty of format to store trees: S-Expr, XML or JSON being a few good candiates.

                                                          If you don’t know about it already, I found the Pane (watch the video) to highlight quite well how purely functional languages can support live introspection of data flow.

                                                          1. 1

                                                            Pane looks very interesting, thanks for sharing. I didn’t know about it.

                                                            Someone also mentioned Jonathan Edwards’s work found here http://www.subtext-lang.org/

                                                            1. 1

                                                              Yes, it’s referenced in Pane. You might also want to look at Aardapel (and the corresponding thesis). This page also contains a treasure trove of screenshots of visual programming languages.

                                                          1. 3

                                                            I’m looking forward to listening! I’ve been an Emacs user for almost 30 years? and am always discovering new things.

                                                            1. 1

                                                              Woah!

                                                              Let me know what you think if you have a chance.

                                                              1. 1

                                                                I’m very much enjoying it, and I’ve been inspired to try ox-hugo and ivy, so far!

                                                                1. 1

                                                                  Thank you, I’m glad you liked it. Let me know if you have questions about ox-hugo. Also, its creator and maintainer Kaushal is a very nice guy who’s willing to help and solve problems.