1. 4

    Nice review. I think the most important of these is setting ids in your fixtures which helps keep parameterisations readable when there are a lot of them. Once you go down the path of parameterising fixtures you can end with with tests that have a lot of parameters and it can be hard to tell what’s failed at a glance.

    A few more…though perhaps people know all of these:

    If you set log_level=DEBUG in your pytest config you will get logging output formatted from the test run which can help debug failures. If you need to print stuff to debug a test failure perhaps it’s something for a log (possibly at DEBUG level). I always set this config first on every project because logs you use in testing are often handy in prod.

    --ff (failures first), -x (stop on first failure) and --pdb (enter debugger on assertion fail or exception), ‘-k xyz’ (run tests matching “xyz”) are key flags to use (--sw is handy when doing big refactorings). Having your test enter the debugger when something goes wrong makes debugging really easy.

    Plugins are extremely easy to write and to begin with you can just put them into your conftest.py. I once developed a plugin for a television set top box marker than send test-run telemetry to a REST API for posterity (among other things).

    The caplog fixture allows for asserting log messages were logged - important for testing error handling in server software. I wouldn’t assert on every log but there are certain circumstances where you want to ensure that your important message is being logged.

    And one other tip: tear down the database before your tests, not at the end. If the tests fail it’s nice to be able to inspect the state afterwards…

    Great tool, probably a bigger reason why I like working in Python.

    1. 2

      Good ones, thanks for the list! As you already started, here are a few more gems that might be interesting :)

      pytest-socket

      A library that prevents internet usage during test execution.

      pytest-mock

      A library that brings you a mocker fixture which enables very convenient mocking.

      pytest-sugar

      Visually more appealing test results and early failure report (you don’t have to wait for all your tests to finish to already see the error traces).

      set autouse=True for fixtures

      Automatically applies fixtures to test cases. This is very convenient when you have certain actions you don’t want to have enabled in your test suite (i.e. mocking certain API calls or expensive document generations).

      1. 2

        Didn’t know about pytest-socket, will have to try it out. For mocking I usually just use the new unittest.mock stdlib module. In the past would use pytest’s monkeypatch fixture.

        Another great plugin is pytest-cov which outputs code coverage reports.

        1. 1

          I haven’t used monkeypatch, but I do like mocker. It mirrors the unittest.mock API except it’s a pytest fixture. I find it helps cleanup tests significantly.

          Another pattern that I do, similar to parameterization, is to have the fixture return a function. e.g.

          def test_foo(bar):
              bar_42 = bar(42)
              bar_57 = bar(57)
              assert bar_42.thingy() == (bar_57.thingy() - 15)
          
          def bar(mocker):
              def get_bar_mock(value):
                  bar_mock = Bar()
                  mocker.patch.object(bar_mock, 'thingy', return_value=value)
                  return bar_mock
              return get_bar_mock
          
          1. 1

            I agree that mocker yields a cleaner interface. It just happened too often to me that tests failed in mysterious ways until I realised that the signature of my test needs to change because I added or removed a mock.patch decorator…

            I like the idea of returning a function in the fixture. I’ll see if I can put that to use somehow :)

    1. 13

      It’s missing a crucial point:

      • Be ready to give up good HiDPI screen support

      I have a 4K monitor because it’s much more enjoyable to use for development, but it becomes a pure nightmare on Linux. I ended up installing an Hackintosh.

      1. 8

        Not necessarily. You can look at this as purely anecdotal, but my Dell Precision 5510, which I’ve had for about four years, has a HiDPI display on par with what I have on the MBP I use for work. Maybe this is down to the fact that I run Ubuntu on it, but I’m just using stock Intel and Nvidia drivers on it, depending on my use case.

        1. 3

          Just curious, what makes it a nightmare?

          1. 10

            My most recent encounter with this is fractional scaling:

            I have a 27’’ 4k monitor. I find that at this resolution, 1.5x scaling works best as 2x is too big and 1x is too small. With MacOS or even Windows, this is not a problem at all, with Linux it’s a can of worms:

            • Xorg doesn’t natively support fractional scaling, Instead you have to rely on hacks or only scaling the fonts (which quite frankly, looks like shit). I never managed to make any of those hacks work reliably and consistently in all apps, eventually you’ll open that one app that uses Qt and you require yet another hack.

            • Wayland does support fractional scaling, too bad that not all apps support Wayland, most notably Firefox. I tried running Firefox with the experimental Wayland backend with fractional scaling on, and everything looked blurry, it’s just not there yet.

            Now, I’m sure for all of those issues there are 10 different workarounds to try and things to tweak to make things better, but I can’t be bothered to do any of that when in macOS (or even Windows) it Just Works™.

            1. 7

              Just the fact that the ArchWiki page on HiDPI has a comprehensive list of required hacks is another example of what I’m referring to.

              1. 3

                Have you tried with KDE? I’m curious because I’m planning on using HiDPI with 1.25x scaling.

                1. 2

                  Yes, with KDE it’s slightly better but still not as good or smooth as Windows and macOS in my opinion.

                2. 2

                  Latest versions of Gnome in Ubuntu 20 have fractional scaling and it’s been set and forget for me.

                  1. 2

                    Ubuntu ’s fractional scaling is using significant more CPU power compare to 2x scaling.

              2. 3

                I tried switching to Ubuntu recently (not for the first time) and ended up going the Hackintosh route, too. For me the breaking straw was not being able to adjust mouse wheel scroll settings in a way that would work everywhere and didn’t seem to come with caveats or be labelled as a hack.

                Setting up a Hackintosh certainly wasn’t without its hassles, but having got there I’m very happy with it. I also have a Macbook and an iPhone, so that is another motivation to stay in the Apple camp (vendor lock-in, I guess?)

                1. 1

                  I’m surprised you had mouse wheel woes. Do you have a special or fancy mouse or something? Or want very specific wheel behaviour?

                  1. 1

                    I just have a normal mouse, and TBH I’m a little fuzzy on what happened now. It could very well have been that if I has been using Gnome (or KDE, whichever one I wasn’t using) then it wouldn’t have been a problem at all, but what I really wanted was MacOS anyway, so I just did the Hackintosh thing instead. I kind of took “well I can’t get this basic thing to work right” as an omen.

                2. 3

                  I use Xubuntu 20.04 on my X1 carbon and desktop with a 27” 4K monitor. Both work totally fine with HiDPI. The main issue I’ve had was when I plugged my laptop into non-HiDPI monitors, I had to lower the DPI for the monitors and that made everything on the laptop small.

                  1. 1

                    The distros the article mentions are ElementaryOS, Pop!_OS, and Fedora Workstation. I believe all of those have had great native HiDPI support “out of the box” for a few years now.

                    I totally get not loving the aesthetic of GNOME or whatever, especially bumped up at x2, but I think it’s cool that Linux lets us customize everything end to end to our heart’s desire. If we’re going for a more custom setup, totally agree that it can require doing some custom tweaks to get a good HiDPI experience and it won’t be completely automagic.

                    I don’t think it’s fair to say it’s all a pure nightmare on Linux as a whole. Beginners can have a good experience, experts can have a good experience, crossing that valley can be painful.

                    Also worth noting that most of the ArchWiki page on HiDPI refers to outdated workarounds that are no longer required, though not all.

                    1. 0

                      Honestly, yeah. I purposely bought a normal DPI display just to avoid this pain. It’s never gonna be perfect. Apps aren’t gonna scale right, even with fractional scaling. Other apps might scale right, but be blurry. It’s gonna drive you crazy.

                      Just get over the loss aversion, buy a good normal DPI monitor, and get back to work. I’m 100% satisfied with normal DPI.

                    1. 1

                      I use Google Keep. It works for simple notes and is seamless between web and mobile. Have been curious about switching to Notion though.

                      1. 1

                        Does it support indentation via tab key yet? Google Keep would be 1000x more useful it handled markdown

                        1. 2

                          No, it doesn’t support tabbed indentation and it would be nice if it supported markdown. It does automatically do ordered and unorded lists, and the checkbox notes work for me as to-do lists.

                      1. 2

                        For work I’m migrating a service from Elasticsearch 2.3 to Elasticsearch 7.1. We are running the two clusters side by side and will retire Elasticsearch 2.3 once the queries go to Elasticsearch 7.1 Also working on migrating a different service from Python 2 to 3.

                        1. 3

                          Probable list of things

                          • Upgrade machines to Xubuntu 20.04, which was released yesterday
                          • Cook & smoke a rack of ribs on the grill (first time)
                          • Continue looking into “Bug Out Bag” equipment, particularly first aid because I’ve ordered most of the other gear. I’m using this: https://theprepared.com/bug-out-bags/guides/bug-out-bag-list/. Also need to make a basic list and plan of what to do in this kind of emergency. Some of the gear should be arriving this weekend so I’ll have a chance to check it out. I’m totally new to camping/backpacking so lots to learn.

                          FWIW I wouldn’t call myself a “prepper” and I don’t think I’ll need to use this stuff anytime soon. But I do live in NYC so I figured why not just have the gear and view it as insurance, or if I have the urge to really go camping when it’s a better time. I don’t live very close to immediate family so having the option to camp out somewhere and survive for a few days puts me at ease. A lot of camping gear is on sale right now too.

                          1. 2

                            On the subject of a “Bug Out Bag”. I find that it only makes sense to “But Out” if you live alone (it seems like you do, but you aren’t specific enough). I’m a big fan of the “Get Home Bag” concept in contrast. I recommend Warrior Poet Society’s videos on the subject: https://www.youtube.com/watch?v=FtcH73kXxZY (original) https://www.youtube.com/watch?v=DZmSX6LHZQk (update)

                            1. 2

                              Wow, he just released another video 2 hours ago, “Why I ditched my Bug-Out bag”: https://www.youtube.com/watch?v=RVdMHDaiaFs

                              1. 2

                                This is a great video! I agree with him that from a security perspective, bugging out probably isn’t the best idea in most situations.

                                Given my situation where I live with my girlfriend and we don’t have kids, it’s easier for us to bug out compared to most people in NYC. Our immediate family is between 5-20H away. So getting to them would likely be a priority but it’s difficult to leave your community and friends many of whom are still here. Plus the journey could be dangerous. If I lived in the suburbs or owned a home with family closer by, then I don’t think bugging out would be a very good option.

                          1. 5

                            Probably gonna branch out and see if I can cook more than spaghetti, stir fry, rice and other simple things like that.

                            I’m also having trouble waking up and coffee is having little effect. I think I might be going through the periodic quarantine “hell zone”.

                            1. 3

                              Our mothership (based in Minnesota USA) gave everyone the day off on Mon to ease the pressure a bit.

                              1. 1

                                That’s awesome. What company? :-)

                              2. 3

                                I’ve had trouble with coffee recently. I normally drink it twice a day and now I’m cutting the afternoon coffee with a couple scoops of decaf. (Down from 4 cups a day.) Only having coffee once in a day every now and then helps me de-load a bit. (Though I’m not sure why that would be. I think the half life of caffeine is only a few hours.) YMMV

                                1. 2

                                  Started doing the same. I cut out afternoon coffee and instead make green tea.

                                  1. 2

                                    If it makes you feel any better, I’ve 100% stopped caffinated coffee consumption and only drink decaf and green tea also. I feel way more consistently stable.

                                    1. 1

                                      Have you found a good brand of decaf? I find it generally doesn’t have as much flavor.

                                      1. 3

                                        Of all the ones, McDonald’s Decaf has been the best so far. Unfortunately there is not much choice here, maybe 5 brands or less total.

                                        1. 1

                                          Interesting. I would not have guessed that.

                                  2. 3

                                    I suggest pizza, anything with beans, and wraps. Since I’ve been cooking for myself for years, you begin to see everything is the same just presented in many different ways. Every dish is essentially wheat, veggies and meat.

                                    The weekly menu I’ve created is essentially this in any order:

                                    • Monday: veggie-based souvlaki-based wraps (salsa, cheese, banana peppers, red onion…etc)
                                    • Tuesday: rice, veggies, chicken
                                    • Wednesday: chicken, mash potatoes, peas
                                    • Thursday: homemade pizza base + toppings
                                    • Friday: pasta (everything pre-done…it takes too many resources to create from scratch.)
                                    • Saturday, Sunday: up in the air. Sunday usually something lighter.

                                    Last night I took a bunch of recipes I’ve been refining on cooking-stained paper and put them into a recipes.txt, like this:

                                    Pizza dough (2): 0F 0min (1 1/2c warm water * 2tbs sugar * 8g/1pkt yeast) * 5min * (2tbsp oil) * (4 1/2c flour) * 65min * (salt)

                                    It makes it super easy to write, read and share. 1 line = 1 recipe. * is left-associative :)

                                    It would be cool for people to share theirs online in a distributed way. Like a web ring: https://mysite.com/recipes.txt , and at the end of each recipes.txt are links to more sites.

                                    1. 1
                                    2. 1

                                      Check out Sam the Cooking Guy on YouTube. Entertaining, lots of swearing, while cooking very simple yet delicious food. He’s inspired me to try quite a few new things and our family is enjoying mixing up the home cooking.

                                    1. 1

                                      I use Xubuntu. My use of Linux started in 2016 when I started my current job and decided to use Linux on a Lenovo X1 Carbon. I picked Debian with XFCE as the desktop env which I really liked compared to GNOME because it used less resources and looked decent. Then I upgraded to a newer X1 Carbon and installed Xubuntu because I didn’t want to deal with Debian’s janky installer. I also use Xubuntu on my home desktop/gaming machine. Xubuntu has worked great for me and I’m always amazed at how easily it installs. Though I wish disk encryption were easier to set up from the get-go because it requires formatting the entire disk.

                                      1. 5

                                        I have the Dell UltraSharp U2718Q and find it to be totally fine, even with gaming. Wirecutter says it’s their budget 4K screen. Would be nice if it had USB-C but not something I need.

                                        1. 1

                                          Yeah, I have the 2715Q. Bought two for $435 each in 2018 for work. I have been very happy with them for office use. Anything less than UHD is just horrible on the eyes after using retina screens for almost a decade now. I don’t get how anyone stands it (Google doesn’t issue UHD monitors to staff which I think is pretty nuts).

                                          1. 1

                                            Have three of these (computing family), they’re great and I’m totally satisfied. However, does not meet the refresh rate you mentioned.

                                            Older macbook pro drives two of them very nicely except for videoconf. All three have a problem wake from sleep, both OSs. I have to reset a monitor about once a month, which I can now do in less than a minute. Still wouldn’t hesitate to recommend. They’re beautiful and relatively affordable.

                                            1. 1

                                              Same display here. I love it. I have the HDMI cable connected to my (personal) Mac mini and a mini-DP cable loose to plug in my work laptop when I want to work.

                                              I’ve heard this year’s model has USB-C.

                                            1. 1

                                              On my previous team we had one team member who was remote. Doing standup in the open office was difficult because they would be on a screen and we’d need to connect one of those disk shaped conference microphones so that they could hear us and we could hear them. It wasn’t the most comfortable because of ambient noise in the office so we resorted to reserving a conference room with a TV screen, camera, microphones. Always having a room for standup was a challenge though, and if we didn’t get it, the ambient noise and awkward huddle around a desk made standup difficult. So we started doing “Slack standup” here and there. Other than that I haven’t had a bad experience collaborating informally through Slack and Zoom with this teammate or working remotely myself. Zoom always works except it uses a ton of CPU. I miss ScreenHero.

                                              1. 10

                                                Doing standup in the open office was difficult because they would be on a screen and we’d need to connect one of those disk shaped conference microphones so that they could hear us and we could hear them.

                                                That’s not remote work. That’s local work, done remotely. If you’re videoconferencing with someone and trying to include them in inherently local activities like a standup then it’s not remote work.

                                              1. 1

                                                Catching up on sleep and hanging out with friends & GF. Working on a side project for work to parse JS in our huge SPA codebase and find all the remaining angularjs definitions along with their dependencies so that we can map out the remaining migration effort. Maybe try to finish the parser chapter of FP in Scala which has been challenging. Definitely playing some Assetto Corsa Competizione here and there as well :)

                                                1. 1

                                                  ELK stack running on AWS with a separate index per service. Kibana has some bad parts but overall it’s pretty useable for me. We have a team that owns it but it’s been running without much intervention for the past ~2 years. At my previous job we used Graylog and found the experience to be awful.

                                                  1. 8

                                                    This is both interesting and embarrassing. At work we’re heavy users of Redis’ HyperLogLog type for analytics and it runs on AWS ElastiCache. However we haven’t been the biggest fans because it’s used a lot of memory. One day our ElastiCache cluster ran out of memory. We didn’t have an alert so we didn’t notice until customer data started looking funny because Redis was deleting keys to make room for new keys to be written. So we spun up a new ElastiCache instance with more memory, which was also two major versions newer from the old one which hadn’t been upgraded in two years. I thought, well there are no breaking changes to Redis for our usecase so this should be fine. Then we pointed our batch job at the new ElastiCache instance, ran it, inspected the data, looked ok, and then cut over to the new ElastiCache instance. Everything was was back to normal. At this point I poke around in CloudWatch and notice that the memory usage is 5x lower than I expected and I become very worried. I wonder if something was wrong with the batch job and it didn’t do a full rebuild of the data? Then I looked at the number of keys in the new instance and it was slightly more than the old one. Ok, what if the keys don’t have the correct values? I check again in our product the data looks correct, there’s no anomalies. Finally the thought that upgrading Redis caused some kind of memory usage improvement crossed my mind. I do some googling and find two big improvements to Redis:

                                                    The first was technically a bug in Redis where merging HLLs would use more memory than necessary. The second is a new algorithm within HLL itself which is an amazing improvement. I don’t think the postgresql-hll extension has this yet. Since we upgraded we were able to scale down our ElastiCache cluster to a much smaller size than previously used saving a lot of $$$.

                                                    The main lessons learned:

                                                    • Have usage monitoring for all your production data stores. It’s embarassing but we didn’t have it for Redis even though all of our other databases had these checks in place
                                                    • Stay tuned on changes in new versions of your datastores, even if they’re as boring as Redis. Someone could contribute something really useful
                                                    1. 6

                                                      My PC is named Phoenix because it has basically been upgraded in bits and pieces, Ship of Theseus style, continuously since I originally built it in 2006. It is currently incarnated as a Ryzen R7 1700 with an ASRock motherboard, 16 GB of RAM, an NVidia GTX 660 GPU, and a 500 GB SSD. (Most of my actual data lives on a NAS.)

                                                      Wish I could do that with a laptop. Building your own PC is cheaper and better than buying off the shelf IMO, but I also enjoy doing it even now.

                                                      1. 2

                                                        how good is AMD compared to Intel processors ? and for gaming in general ?

                                                        1. 2

                                                          AMD has really stepped up recently. They’ve been beating Intel on performance and cost across their most recent line of CPUs. https://www.youtube.com/watch?v=stM2CPF9YAY https://www.pcmag.com/review/371925/amd-ryzen-9-3950x

                                                          1. 8

                                                            It’s also quite nice that AMD isn’t constantly changing their CPU sockets; every Ryzen generation has kept the same socket so far, and so (assuming your motherboard manufacturer keeps its firmware up-to-date) you can theoretically just keep upgrading the CPU as better ones come out… Whereas Intel seems to change their sockets every 1-2 years, locking you into specific generations unless you’re willing to buy a new motherboard and basically reassemble the entire PC from scratch. With the Ryzen line so far a CPU upgrade has been about as easy as a GPU upgrade, whereas with Intel you’re often stuck with whatever processor generation you bought into.

                                                            That being said, apparently AMD’s going to do a socket change in 2020, so there’s not a lot of difference if you’re buying right now. If you’re willing to wait for the next socket to come out, they’ll probably support it for a lot longer than Intel, though.

                                                            Currently AMD has Intel pretty soundly beat on price for performance, and in some benchmarks (here’s the Anandtech benchmarking for the 3950x, similar to the PCMag one linked above, but I find Anandtech even more thorough, and they test specific games as well) they beat Intel even when ignoring price.

                                                            My current gaming PC is Intel; I built it a while ago though, before Ryzen came out. My Linux desktop is AMD, and I’d probably go with AMD if I were rebuilding the gaming PC today. (And at some point presumably I’ll do it, since I’ll have to replace the gaming PC’s motherboard anyway due to the socket changing.)

                                                          2. 2

                                                            Every benchmark I’ve seen has them quite a long way ahead on price/performance.

                                                            My current PC build is running ubuntu on the cheapest available Threadripper (24 cores) and cheapest RTX-class GPU, and came in at about 2/3rds the price of my $WORK macbook pro.

                                                            It absolutely screams along (eg: $WORK test suite on my 15’ macbook pro took 48m, on desktop it takes 9m, and the desktop remains responsive while it’s running).

                                                            I’ve never been super interested in current-gen AAA titles, but the few I’ve tried have all worked on high settings, and it drives my VR setup very easily.

                                                            1. 2

                                                              In my experience, AMD’s Ryzen line of processors is basically 95% of the performance for 75% of the price of their Intel equivalents. This applies to gaming too. Their basic mid-range desktop CPU, the X600 (where X = 1, 2 or 3 depending on generation), is also 6 cores instead of the more typical 4, so for multithreaded workloads, like compiling code, it’s more like 110% of the performance for 75% of the price. I am a thorough fan.

                                                              Because it’s a good story… In 2011 AMD released their then-next-gen CPU architecture, which turned out to be a giant architectural dead-end. AMD CPU’s were just tragically bad for a long long time after that, as they had to develop an entirely model of processor core without going broke in the process, and they’re a much smaller company than Intel that can’t afford to just eat development costs and get on with life. Intel naturally took the opportunity to become the de-facto standard in all but the lowest-performance markets, and made money hand over fist. In mid 2017 AMD finally managed to dig themselves out of their technological hole and released the Ryzen line of CPU’s which are once again competitive. And lo and behold, since they did that Intel CPU’s have quite suddenly gotten cheaper, more powerful and more flexible as well.

                                                          1. 3

                                                            I built this AMD Ryzen 7 2700X rig in March earlier this year: https://pcpartpicker.com/user/jjmalina/saved/#view=gFqCLk

                                                            Photo of the battlestation: https://twitter.com/jjmalina/status/1150199875088322560

                                                            It’s running Xubuntu and Windows 10. I use it for half gaming half coding. For work I have a Lenovo ThinkPad X1 Carbon, which has acceptable performance but nothing like the desktop. Now I want a desktop at the work office.

                                                            1. 1

                                                              Looks really cool, thank you for sharing!

                                                            1. 11

                                                              I love dhall as a non turing complete config language.

                                                              1. 3

                                                                Sort of related: I had no idea what Dhall was for but recently listened to the CoRecursive podcast with the creator and found it interesting https://podcasts.google.com/?feed=aHR0cHM6Ly9jb3JlY3Vyc2l2ZS5jb20vZmVlZA&episode=Y2I0NmZhNmZjOTQ5NGY1OTkxZWE2ZDZkNzU2MDM0Mjk&hl=en&ved=2ahUKEwjlo8OLnu_lAhUGrVkKHXOiAOkQieUEegQIABAG&ep=6&at=1573924629200

                                                                The CoRecursive site isn’t working for some reason so I shared the google podcasts link instead.

                                                                1. 2

                                                                  This episode of CoRecursive mentions this SERadio podcast episode about Dhall too: https://www.se-radio.net/2019/08/episode-375-gabriel-gonzalez-on-configuration/

                                                                2. 3

                                                                  Yeah! Starlark (mentioned in the article as well) is also not Turing complete. It’s a dialect of Python which makes it a bit easier to adopt if you’re familiar with Python. I’m using the Rust implementation for a small personal project that needs configuring.

                                                                1. 2

                                                                  Cool article! I think it really shows how close you can get, despite the syntactic overhead of Python’s lambda.

                                                                  The only thing I may add is a word of warning that streams a la Scala are a very bad idea in general.

                                                                  They basically combine

                                                                  • a generator that can create an infinite amount of elements
                                                                  • a memoized, singly-linked list-like data-structure
                                                                  • with deep integration into the collections API.

                                                                  This means it comes with

                                                                  • the added possibility of non-termination in every collection operation on more general types like Seq,
                                                                  • a massive danger of memory leaks if the head isn’t dropped in each and every place it is required,
                                                                  • a horrid amount of implementation complexity, not only because you have to play defense against every implementation supertypes “helpfully” provide, but also because it’s really hard to avoid evaluating thunks unnecessarily.

                                                                  Scala’s Stream should have been deprecated and removed years ago.

                                                                  1. 1

                                                                    Thanks!

                                                                    Interesting take on why streams are a bad idea. I haven’t used them in practice in Scala, but in other languages like Python generators are very common (in Python 3 range, filter, map, etc. became lazy by default), and I think in node streams are used quite often. So I’m curious in Scala if streams are to be avoided, then whats the alternative? What about Haskell?

                                                                    I do realize the danger of memory leaks is something to be aware of. Avoiding thunk evaluation was also something I didn’t try since the first call on filter would always be evaluated. So I can see that the full implementation might get complicated.

                                                                    1. 2

                                                                      I haven’t used them in practice in Scala, but in other languages like Python generators are very common […]

                                                                      There is nothing wrong with generators in isolation – Scala even ships with a lot generator functionality on the Iterator object.

                                                                      The problem is that Scala married functions that can generate an infinite amount of values with a memoizing data structure.

                                                                      The Stream companion object comes with pretty much the same “generator” feature set that Iterator provides, except that while everything Iterator produces is ephemeral, everything that Stream produces is retained if no care is taken to drop the head at the right places (which is non-trivial).

                                                                      So I’m curious in Scala if streams are to be avoided, then what’s the alternative?

                                                                      I’d stay away from the eagerly evaluated collections operations and try to use views; they have been too buggy before 2.13 to be usable, but things may have improved after the rewrite in 2.13.

                                                                      The exact brand of laziness Scala’s Stream offers is hardly ever useful – I haven’t found a single use-case during the time I worked with/on the language.

                                                                  1. 4

                                                                    This worries me a lot. The web is the only platform you can easily control as a consumer. When you close a web page, everything is gone. There’s no background tracking or annoying permissions to deal with. The tradeoff is native apps are faster and JS is not the best designed language.

                                                                    1. 9

                                                                      I paid off all of my loans and have some savings for emergency. So, I’m going to look into investing in some stocks. I am a total newbie when it comes to investment. So, I don’t want to start it without knowing what I am getting into.

                                                                      On technical side, I am going to try Google’s Sentiment Analysis APIs for one of my private projects.

                                                                      1. 11

                                                                        I work in finance, and most people who actively manage portfolios buy passive stuff (like ETFs or low-fee funds) for themselves with their “important” money.

                                                                        With then some amount of play money they often invest in the latest fad like cannabis stocks or some other bullshit, and they usually lose as much as they gain on average.

                                                                        Point is, keep it boring, with fees <70 basis points if you just want to leave your money alone for a decade or two. Don’t stock pick unless you would be willing drop the same amount on blackjack or poker.

                                                                        1. 2

                                                                          Robo-advisors (like Schwab or Betterment) offer a compelling solution, investing your moneys in index funds (passive stuff), rebalancing automatically and in certain situations, ‘tax-harvesting’ on losses.

                                                                          Still not an excuse to not understand what’s going on, but reduces labor (and potential mistakes).

                                                                          1. 2

                                                                            My problem with robo advisors for taxable investment accounts is that they have a certain amount of lock in - transferring shares out in kind creates a mess that’s fairly difficult to manage by hand. TLH/rebalancing is also quite easy to do manually once or twice a year on a small ETF or mutual fund portfolio.

                                                                          2. 1

                                                                            I like the idea of keeping it boring. Like I said, I am really new to investment. So, stocks might not be the only options I want to pursue. Also, I am not making a lot of money right now but I though it would be a good idea to do some study when I have time. Thanks for you suggestions.

                                                                            1. 3

                                                                              One piece of advice that got my attention was to contribute to your employer 401k program - specifically contribute as much as your employer would match. That way you know you’re getting a 100% return on that money, which is basically unmatched anywhere else. The catch is that contributing more than what your employer matches actually brings that % return down.

                                                                              The idea that marginal returns are what matters was particularly powerful to me - basically paying down your credit card debt at 18% interest rate is equivalent to investing in a stock with 18% return. So unless you can guarantee that you can get an 18%+ return on your investment, you may as well just “invest” in paying off your credit card.

                                                                              1. 1

                                                                                I am contributing to my 401k as much as my employer matches right now. I plan to increase my contribution in future though.

                                                                          3. 3

                                                                            Some books I recommend

                                                                            I agree with the no stock picking / buy an index advice even though these are about stock picking :) I’ve heard Mastering the Market Cycle book is a good introduction to investing.

                                                                            1. 3

                                                                              Congratulations!

                                                                              This is a ~16 page book that I recommend for new investors. It packs a bunch of useful advice in a very short volume: https://www.etf.com/docs/IfYouCan.pdf

                                                                              Past that, the Boglegeads wiki and forum has a wealth of information and people who can help if you want to learn more or ask about your specific situation.

                                                                              1. 3

                                                                                As a novice myself I found A Random Walk Down Wall Street refreshing.

                                                                                1. 2

                                                                                  Thanks for the recommendation. I will definitely check it out.

                                                                                2. 2

                                                                                  Good work getting to debt free!

                                                                                  1. 1

                                                                                    Thanks. I did not have huge debt. It was mostly student loan and my car installment. Still, it was a huge relief.

                                                                                  2. 2

                                                                                    Congrats on paying off your loans! One piece of advice (among the hundreds of unsolicited pieces of advice you’ll get when talking about investment online) is to only buy risky¹ investments with money that you don’t need to survive with. If you imagine that pile of money disappearing, would you still be able to pay for food, rent, and other daily expenses? It’s also prudent to plan for the occasional emergency expense such as last-minute travel for a funeral. You obviously can’t plan for everything (nor should you), but you don’t want to be put in a position where you have to sell stocks at a loss to pay for a plane ticket.

                                                                                    ¹ “risky” in this context means anything that only makes sense as a long-term multi-year investment, such as stocks

                                                                                    1. 2

                                                                                      Congrats on paying off all your loans!

                                                                                      1. 2

                                                                                        On the off-chance that you end up looking at peer-to-peer investment sites like LendingClub and Prosper, be aware that your money stays locked up for years in those loans. With LendingClub you may be able to sell some of your loan investments on a third party site (FolioFN) but Prosper does not have this. You won’t be able to get your money out easily (or at all) if you need it - you’ll get it back (hopefully with a profit) in a trickle as loans are (hopefully) repaid. I see these platforms now as essentially a medium term bet on economic stability, and I am personally not a sophisticated enough investor (or maybe I’m just too risk averse) to make that bet. I’m gradually withdrawing my investments as they’re repaid (that’s not intended as investment advice 😀)

                                                                                      1. 1

                                                                                        Google Keep. My coworkers use Notion so I’m considering switching to that.

                                                                                        1. 3

                                                                                          Awesome work! Thanks for sharing. We have a similar infrastructure at NewsCred. We export data out of BigQuery (collected by Google Analytics), then aggregate it in Spark and use Redis for HLLs, which an HTTP API service queries and returns. The downside is adding more keys uses more memory and Redis becomes expensive. Spark now has an interoperable HLL format in a library they’ve published. A side project of mine I haven’t gotten around to yet would be to get Spark to output this to Postgres which supposedly should work with this HLL format.