Often, there are the same links on the front pages of Lobste.rs and Hacker News. I was curious how many, actually? This small web app fetches the links from both sites (ignoring non-link “stories”), and shows the statistics and listings. Updates every 10 minutes.
Have you considered using heuristics to normalize the links? For example, I currently see
https://avi.im/blag/2024/sqlite-past-present-futureas “Unique to Lobsters”https://avi.im/blag/2024/sqlite-past-present-future/as “Unique to Hacker News”Good point, fixed!
If you’d like a few more normalization rules, ours are here.
I’ve noticed that often a link will be on HN and then appear on lobste.rs a few days later, so it would be interesting to widen the time window rather than just looking at the front page. (Obviously that would be a lot more work, though!)
I sometimes cross post to Lobsters because I feel like this community would have a different or deeper opinion on the linked article.
Please continue — the S/N ratio in comments is much better here!
I agree. I see an article on the orange site and after a few comments things kind of turn sour for me and I head to lobsters to see the comments. Usually lobsters does not disappoint…
Without relying on server-side storage (which I’d like to avoid), it’d be great to be able to fetch some historical data. I fetch the Lobse.rs feed from https://lobste.rs/hottest.json, which I found in some random comment here; I am not aware of other endpoints. For HN, I’m using the Algolia API (https://hn.algolia.com/api/v1/search?tags=front_page), which gives a bit more flexibility.
My go-to for this sort of app is using “git scraping” ala https://simonwillison.net/2020/Oct/9/git-scraping/
I’m going to say Github here, but I’m sure alternatives work too. All you need is static hosting and an ability to run a cron that can commit to itself.
It would look something like this:
Every hour (or your interval of choice), run a Github Action that fetches the entries on the current homepage. Save them in a file in your git repo.
Then to render your site, check back through the git history for whatever overlap window (eg, 5 days might work well for HN/Lobsters). Commit the html file, and host it on Github Pages.
For something as small as “the links on Lobsters every hour”, you can easily keep decades of history in a git repo.
There’s active.json as well, and you always could use the rss links too for filtering
Nice to see some competition!
https://gerikson.com/hnlo/
you should put https://github.com/gustafe/hackernews-lobsters-pairs somewhere on the website! took me a while to track it down to give it a star :)
Thanks for the suggestion! I’ve added a link to the About page.
You didn’t ask “why”, but a while ago I noticed one of the main drivers of this is that
toddacerdotiwill quickly repost many new lobste.rs. They run a service that lets you quickly build web bots in a manner analogous to what yahoo pipes used to do, so I guess it’s built using that.Yeah, I’ve posted on here a few times and found it was reposted to HN within a few minutes multiple times. Definitely a few repost bots on there.
I complained to dang about that obvious bot only to be told “he’s definitely not a bot”.
I’m okay with repost bots if they aren’t trying to be coy about it and can’t use their enormous amounts of karma to influence votes (not saying this is happening, but it’s a strong motivator)
Wow, HN just keeps getting less and less on topic.
What stuff is in contradiction to its guidelines?
The guidelines are so open-ended it would be impossible to be off-topic, but when most of the front page looks like “WSDA, USDA announce eradication of northern giant hornet from the United States” I think it’s fair to say the site isn’t living up to its name or original intention anymore.
Which is fine, cause I think that’s what their audience wants.
HN is Facebook for STEM grads.
Thank you, I’m stealing this
i’d be curious to see kind of the opposite- which posts do well on one site but not on the other?
On HN, the success of a post is mostly random and about luck. I’ve seen some of my blog posts posted 5/6 times in 48h on HN with one getting more than 100 votes while the other getting less than 10. Sometimes, none of those posts get traction. Then, someone repost it like two years later and it receives 300 votes. (which proves that the “getting no traction” was not related to quality).
This is very random and why I don’t post my own stuff on HN. HN is a pure radom feed with “not too bad” stuff.
On lobste.rs, it is very different. You can’t really post the same URL multiple time. Each post will receive at least a least 5/6 votes and the most successful receive 70/80. Which means there’s no real “major success” on lobste.rs. Some stay a bit longer on the front page, that’s all.
Well, the sorting function is not nearly aggressive enough given the post volume. So if you have a successful post, it’s likely to stay on the front page for several days in a row. I find this pretty annoying as someone who checks lobsters at least daily. I’ve mostly switched to browsing by new…
“How bloom filters made SQLite 10x faster” is not detected as shared, but it is
Yeah, that was a problem with normalization, as per ryan-duve’s comment above. Fixed now.
While this isn’t terribly fast, could it be slower, making it cheaper for you and pushcx, while still providing a reasonable overview of the differences? Or, assuming HN has a higher rate of submissions vs here, maybe check HN every 10 min but only check Lobsters once every few cycles?
I have a tangential question for @pushcx on this: is there guidance for cross posting from HN? For example, I read the article at That’s Not an Abstraction, That’s Just a Layer of Indirection and want to see what folks here have to say on the topic.
If it’s on-topic, it’s fine. There’s no rule about it being seen on HN first makes it “tainted”.