1. 1

    trying to write a cli tool for webhook replay on local for a mail stream.

    My app https://hanami.run support webhook, email come in, we parse and fire webhook but writing webhook handle with an email workflow is tedious to test so I’m going to write a local CLI that connect to our server sent event app to tail the mail stream and fire webhook to a local endpoint.

    1. 2

      It looks like they are an email forwarding serivce, if they change the content of the email, then DKIM will be render invalid.

      How does this really work? I wonder how they made SPF/DKIM work without rewriting the message essentially sign with duck.com?

      1. 3

        I actually think that botnet and their existence is a beauty feature of the internet. The reason botnet existed is due to the amazing of the internet that enable them to operate. Think about it for a second, with all kind of lock down, rate limiting put in place, they existed.

        The real problem to me is that bandwidth costs is hight nowaday. What if we turn thing around? bandwidth is charged to requester? Say I made a website, if people visit it, they should pay me for my bandwidth cost. no? If somehow the internet operate that way, we will be welcome the botnet :-)

        1. 1

          trying to add IMAP support for my email forwarding app https://hanami.run Not going to be a full feature IMAP but with limited so people can access email through IMAP progamatically is good enough.

          1. 1

            I think JavaScript isn’t intended to use as a desktop app language and a server side language. Yet, it diverse and eat into those market. people even use it for infrastructure like AWS CDK and Pulumi guys.

            Ins’t that cool? When something is designed to be use in this but people use them in a different way.

            I run an email forwarding service which its main purpose is to forward emails but a few users use it as a tool to preview their email rendering without forwarding out so that they can inspect and fix any issue. I never intended for it to be used that way. I think it’s great when people found another way to use the thing that you design. It doesn’t mean it’s a failure, to me, It means the thing is flexible enough to warrant their workload.

            Samething with Go, they want to win over C but lots of Ruby/Python people started to pick up it.

            1. 3

              I’m not sure why the author is so convinced that no one could be doing something new or fun in Ruby land or that if they did no one would talk to them…

              1. 2

                Because he tried for many years, maybe? And observed the community for said years? ¯\_(ツ)_/¯

                1. 1

                  Ins’t stuff the Andrew https://github.com/ankane build consider cool? He did lot of ML stuff. All of his gem integrate nicely without heavy external dependencies

                  Same with https://github.com/ioquatix

                  1. 2

                    Andrew is an absolute hero indeed, and there are several other groups working on ML gems; but in the scope of the point I’ve tried to make it is important to notice that

                    • a) these gems typically have very low usage in the community (even if “considered cool”, it is more “Oh, it is a cool thing, I’ll bookmark it for the case somebody says Ruby doesn’t do ML on Reddit”); and
                    • b) it is more of a “chasing the leader” kind of work (“Ruby now can do this and that too”).

                    Nothing wrong with chasing the leader, and it is work that needs to be done.

                    But my argument was about the fact that it is very hard to do something new in Ruby to be noticeable in the industry in general, not just mentioned in the “Ruby newsletter” or upvoted on /r/ruby—while Ruby’s traits are very suitable for this kind of work (inventing something “completely new” out of the blue).

              1. 1

                I tried to add a key-value store as embeded buffer on top of our Postgres for my hanami email forwarding app

                Sometimes I want to do database maintenance but being a mail service I cannot have downtime, I therefore try to add LevelDB embeded in our mail server, accept mail, write it to local disk and have another thread process them and write metadata to Postgres(and re-try if postgres down).

                This will allow me to upgrade/change Postgres all day long. Essentially a cheap version of Kafka.

                Another thing is to migrate our Ruby webhook fanout to Go or Rust. The app growth fast and currently sending more than one webhook per seconds for our user.

                1. 1

                  I’m trying to make my app aware of database being down and buffer write to a local badger db then sync the change back to db once our db is backed up.

                  My app is an email forwarding app so I cannot have downtime, I have enough domains to manage that I’m processing about 20 emails per days so I want to decouple the mail server from the postgres database. It’s ok for the webapp to be down when we maintain db but mail server cannot be down.

                  Luckily a majority of change between mail server <-> postgres is append only so I can buffer write to badger db,and write back to postgres later on.

                  1. 3

                    Essentially any email forwarding services will help you there. You can configure a catch-all.

                    To me, email is a great protocol where it’s distributed. If you only use any email service as a dumb client, as in don’t rely on their web ui to read email, then it’s very easy to migrate to another platform.

                    In case of email forwarding, emails are stored at the final destination so it’s even easier to migrate, you just point MX record of your domain to another platform.

                    1. Privacy Really at the end of day it’s all about who you trust. All service will claim they won’t read your email and obviously they won’t. But let’s say they are working on a spam filtering, they have to monitor it, and at least the operator has to check a field thing such as origin IP, domain, and subject line at least to ensure spam filtering work. So this is a very grey area. If Privacy is about not leaking the original emails are almost all email forwarding services are good here

                    2. Cost When we acquire a domain, we pay about $10-20 depend on provider or domian type, so I generally assume cost about $12 for email forwarding is good here.

                    3. Reliability/Longevity of the service In my experience, icloud/hotmail has kind of dumb/sensitive spam filtering sometime may flagged your legitimates email as spam. Esepcially if you used to receive email from someone directly at your domain, or your real inbox is new.

                    Many provider are support ARC now so we will have better delivery(because SPF will fail when forwarding email) and ARC is a must.

                    1. Features Some services offer webhook(so you can example, fire a webhook to process your receipt emails- this is what I do and essentially build an API for my bank on top of google sheet api :-) will open source it soon too. Some offer regex routing and URL redirection too. Some offer maillog so you an look into the mailqueue to know if an email is flagged as spam or never come to your final inbox

                    FULL DISCLOUSE: I run an hanami, an email forwarding service

                    1. 1

                      This is only on edge mean it isn’t release yet right? I would love to use this. I store DKIM private key and currently use some homegrown skim to store it and kind of a mess now :(.

                      1. 5

                        Dive more into Kubernetes. Maybe someone has good resources?

                        1. 2

                          https://learnk8s.io/ is a good one

                          However, in K8S, I think the best is hand-on experience, do it and. you will understand why a certain thing has to do that way. To recep some concept inside out:

                          • A Deployment to run our app
                          • A service to group pod and its port
                          • An ingress(load balancer) to route traffic to those pod
                          • An ingress controller to create that ingress
                          • Secret to map env to our container
                          • Config map to share file config or simple keyvalue store which can be mounted as file/directory, yes, crazy but imagine you can YAML a json blob and expose it as a single file
                          1. 2

                            I highly recommend Kubernetes in Action. I used it to ramp up on Kube when I joined a new team at my current job.

                            1. 1

                              I followed a couple of pluralsight videos, and while I generally prefer written material - at 1.5x speed it was a good way to get a quick overview over k8s (eg: things like ingress not being standardized).

                              Apparently the courses I took has been retired, and replaced by courses that we need a different licence/plan to access. I guess maybe we’ll drop our subscription, if it turns out it’s stopped being useful. It’s not even obvious why the new courses aren’t part of our subscription, or what upgrade options we have… Sigh.

                              So, I guess I cannot recommend pluralsight after all.

                              I did see this - but I have not taken it (or any edx courses) - but I’m definitely considering it for myself: https://www.edx.org/course/introduction-to-kubernetes

                              Ed: having signed up and had a brief look, this seems frustratingly slow paced - I’d be happy if anyone have other resources to recommend.

                            1. 3

                              Trying to get spam in control for my app an email forwarding service. I have been dealing with spam in last 2 weeks and I learn that gmail is the best when dealing with spam. They are very smart and filter by content. They implement burst-able so if you suddenly has a spike they slow you down but will eventually accept your email and analyze it. On the other hand, hotmail/icloud are the dumbest. They blindly flagged the IP and called it a day.

                              My actionable items:

                              • Improve rspamd
                              • Train rspamd somehow to have better filtering
                              1. 1

                                Trying to implement rspamd for hanami, my email forwarding service and add team supports.

                                1. 1

                                  Thanks so much for this. I want to get into Swift and I think this small app is enough to boostrap me.

                                  1. 1

                                    I’m relatively new to Swift too, so it’s probably not the most idiomatic Swift project out there, but I’m glad it’s helpful!

                                  1. 1

                                    Is my understanding correct that this is just for the default, and I can restore the old behavior for my websites by including <meta name="referrer" content="unsafe-url">? If so I am fine with it, except for the fact that name is unfortunate: what is unsafe about unsafe-url?

                                    1. 4

                                      It doesn’t matter 90% of time but it may leak a few sensitive info such as access token.

                                      Site like docusign/dropbox generate links like that has something like ?token=unique-token-here anyone know that token can view that content. If browser send full referrer header, now the third party service has that token in their log.

                                      So it’s always a good practice to trim down. Obviously if you are sure you have no sensitive information in the URL then you’re fine.

                                      One way to fix this is once you see ?token=, you set it in session and immediately redirect to another page. Never ever render a full HTML page with sensitive token.

                                      1. 3

                                        Obviously if you are sure you have no sensitive information in the URL then you’re fine.

                                        It is a bad idea to be sure of this. You don’t know what kind of situation your reader is in - they could be a domestic violence situation, politically persecuted, or otherwise in a dangerous environment. Leaking what page they were reading on your site might be a huge problem for them, even if you (or I!) can’t see what problem that might be. Err on the side of caution.

                                        If you absolutely insist on overriding Firefox’s new default, then at least please don’t set it to send a full Referrer even over plaintext connections, where any casual adversary on the network could get at it.

                                      2. 4

                                        It leaks information about your visitors to third-parties.

                                        1. 3

                                          I want to “leak” that information so that third parties are informed where they are linked from.

                                          1. 14

                                            I think the point is that your readers may not want the same thing.

                                            1. 8

                                              Yeah, it’s not unsafe for the website author. It’s unsafe for the user.

                                      1. 2

                                        Adding team support to an existing SaaS. https://hanami.run an email forwarding service.

                                        The challenge is that my app is paid-only with free trial, no freemium so everyone has to have an active subscription in the original code, a bad assumption I must say. So when a user join a team, they have no subscription at all but inherit that from their team…So all the assumption I made about every user will have an active subscription to operate the site are wrong now and I have to fix this legacy :-).


                                        PS: If you are curious what the tech stack looks like, I’m on youtube talking about it https://www.youtube.com/watch?v=O1p2crPpFIc

                                        1. 1

                                          Was about to whip up mine when working on my (email forwarding app)[https://hanami.run] and found out this great library. I think it’s useful to anyone working on building automation around email flow such as support tickets(parse email /signature to get content)

                                          1. 14

                                            I’m a big fan of storing developer documentation in a doc directory in the git repo, as plain text files in $your_favourite_format. MAny hosting tools will render Markdown (or RST, ASCIIDoc, etc.) But you can also add a static site generator in front of it, which is pretty easy to set up.

                                            It’s simple, easy, can be reviewed in code reviews, the documentation will always match the code version, can be searched easily, you get a good history with decent tooling (I missed “blame” on Wikipedia many times), anyone can use their $favourite editor/IDE to write documentation, has basically 0 operational overhead/barrier of entry, etc.

                                            I have yet to see a system that improves on this for developer documentation. Sometimes things don’t need to be complex.

                                            1. 3

                                              There was even this cool concept named artifact (now abandoned) that expanded this to provide a link between documentation and implementation of features described in the specification. That sound like almost perfect solution for documenting projects.

                                              1. 3

                                                I found this also keep the docs and code in one place so people are easier to update it. Example, you submit a PR and someone pointed out you have to update the doc.

                                                If we move the doc out to say Google Docs or Dropbox Paper then it’s harder to review them.

                                                1. 2

                                                  I missed “blame” on Wikipedia many times

                                                  There’s a tool for a git-blame-like usage on Wikipedia http://wikipedia.ramselehof.de/wikiblame.php?lang=en&article=Main_Page (this alternative one also), linked in the History tab for each article.

                                                  Also, putting documentation in the repo makes it easily discoverable with grep for example.

                                                  1. 2

                                                    Ah good to know; thanks. It’s been quite a few years since I did serious editing and I tried to find something like this, but wasn’t able to find it at the time.

                                                  2. 1

                                                    Dropbox Paper worked well at one place, re: operations. Searchable, without having to work out which git repo to look at. Easier to make small edits, without coming up with a edit reason. Editing a wiki page, I could picture getting interrupted, and totally losing the WIP.

                                                    1. 1

                                                      I am also generally a fan of this approach, but unfortunately it still suffers from the same issues as other tree-based solutions. Its fine if it is tied 1-to-1 with the code, because then the hierarchy is determined. But for information that doesn’t fit this mold it has the same issues as any tree-store will.

                                                      Additionally, I have found this approach works until it gets unmaintainable (because tree-based documentation always does), and then someone higher up decides the issue is that the documentation is stored in Git and not the structure of the documentation. I’ve seen this happen twice, and while switching to a system like Confluence makes things worse it is unfortunately how things have gone in my experience.

                                                      Gittit is a cool project which provides a graph/wiki interface on top of a Git repo which I think gets the best of both worlds. As far as I know it doesnt store images in Git which is a little unfortunate though. I would love to see some more solutions in this space, and I’ve been somewhat working on my own on the side.

                                                      1. 3

                                                        I don’t think it needs to be a tree-based approach. I rarely use subdirectories for these things (preferring filenames to order things, like email-development.markdown, email-sendgrid.markdown, etc.) and you can link to other documents with [email-sendgrid] or [See the sendgrid docs](email-sendgrid] (Markdown syntax) which should show up in the rendered version.

                                                        With things like Gittit you lose some of the advantages of storing the docs in the repo, like reviews. We used GitHub wiki at a previous job (which is also stored in git) and I found the only real advantage is the ability to edit files with Vim, which is nice (for me, anyway), but other than that I didn’t really see any clear advantages over Confluence or whatnot.

                                                        The biggest downside of editing Markdown files like this is the lack of direct preview by the way. If you make a typo in the above then you won’t know until after you push, although this can be solved with a doc generator/lint tool locally to some degree (there are other mistakes you can make too, like forgetting to close a * which are not so easily caught by a linter).

                                                        1. 1

                                                          This is super interesting, thanks for explaining! Using a flat-directory approach with markdown files does seem like a good solution, and I was unaware that Markdown could link between itself that easily. It is certainly something that I will have to try out myself. Have you used this system with many developers at once? I’m curious how easy it is to get everyone on-board.

                                                          I agree with your points on Gittit; it is not a perfect solution. The underlying behavior is very similar to your flat-directory approach, but the Git integration is a little lacking (for instance, symmetric syncing doesn’t seem completely figured out, and PRs are not a thing).

                                                    1. 14

                                                      It feels so amazing about their raspberry pi offering. I can see myself order that as well.

                                                      1. 4

                                                        Yeah, that’s a really interesting managed-but-on-prem model! For more “compute-heavy” stuff, I could see something like a NUC being a pretty viable replacement as well. Very cool!

                                                      1. 1

                                                        This post is tagged rant, so please allow me…

                                                        >2020+1
                                                        >they still use webmail instead of regular desktop/mobile email clients
                                                        
                                                        1. 1

                                                          The iOS client is a native app, not a web view.

                                                          I’m pretty sure that goes for Android too.

                                                          1. 1

                                                            But why do you need to have any sort of specialized “app” for each e-mail service where you can use generic IMAP client? There’s nothing you can do in Gmail UI which you can’t do over IMAP in regards of sending and receiving emails.

                                                            Yes, I know you can “bind” other email accounts to the Gmail acoount, but then you let all your mails get through Google servers after all, so you can just throw these accounts and use the Gmail only anyways.

                                                            1. 1

                                                              Hi,

                                                              Agree with what you said. I always use a Mail App client. This post was written when one of client of my service asked my why my email is showing “via way.hanami.run”. This inspired me to write this post.