1. 2

    A “ghetto hack” to workaround this limitation in the PayPal subscriptions API.

      1. 3

        Tangential - if anyone wants to export their Google/other search history, the easiest way that worked for me a few months ago was to export all browser history to a list of newline-separated URLs and then to run this (series of) commands from the Linux terminal to extract all “q=” parameters (also trims leading/trailing whitespace and removes any duplicates):

        cat chrome-history-urls.txt | grep -Po '[?&]q=.*?(&|$)' | perl -pe 's/.*[?&]q=(.*)?(&|$)/\1/' | perl -pe 's/&//s' | perl -pe 's/\+/ /g' | perl -pe 's/\%(\w\w)/chr hex $1/ge' | uniq
        
        1. 5

          This post was made roughly a month ago.

          After experimenting with the various options for headless web crawling:

          • Due to it’s command-line flags, Headless Chrome is probably the best for extremely basic web crawling for use cases where you only need to render the page with JS, but don’t actually need to cause any custom interactions with the page directly (e.g., just execute the webpage’s built-in JS and then get the resulting DOM as HTML).

          • Despite no longer being maintained, PhantomJS is a very solid setup for smooth/minimalist web crawling with fairly light scripting/JS interaction. In contrast to Headless Chrome, it’s easy to interact with webpages without bringing in external libraries like Selenium. E.g., if you want to load a webpage, press a button, and then get the resulting HTML.

          • Headless FireFox is solidly adequate. There were no specific areas that stood out while using it. PhantomJS feels lighter-weight and quicker to get up-and-running for small-to-medium sized scripts, and the Headless Chrome CLI was superior for small/quick one-liners; if you’re writing a medium-to-large size web crawling tool, those considerations are likely going to be less relevant, and in such cases Headless Firefox probably works just as well as the others.

          1. 6

            I do a fair bit of low-volume, high-fidelity scraping.

            I wouldn’t recommend PhantomJS any more, because if the page you’re scraping happens to use any recent JS features (eg ‘does not work on IE10’) it’s not going to work on PhantomJS either.

            You probably already have chrome or firefox installed on your computer. The selenium bindings are not that heavyweight, and using a real browser makes a huge difference in the amount of time you spend faffing about with obscure script errors.

            Grab the selenium bindings for your favorite language (there’s a ton of options) - I’m partial to the capybara DSL for ruby - and go to town.

            For instance, childcare posts photos of my kids using an app that (among other things) sets up your view based on a session adjusted by making GET requests. The images are only displayed using a JS carousel. This ruby/capybara script lets me archive them for myself:

            visit 'https://web.myxplor.com'
            
            fill_in 'email_address', with: '<REDACTED>'
            fill_in 'password', with: '<REDACTED>'
            click_button 'Login'
            
            # Select which kid to get pictures of, via a GET request. Yuck.
            visit "https://web.myxplor.com/parent/child_timeline/<REDACTED"
            
            # Go to the page with the pictures
            visit "https://web.myxplor.com/observations"
            
            # Load all the pictures
            while page.has_button?('Load More')
                click_button 'Load More'
                sleep 4 # If it's stupid but it works, it isn't stupid...
            end
            
            all("a.fancyboxNew").each do |link|
                outfile = sanitize_filename(link['title']) '.jpg'
            
                # Don't re-download files
                next if File.exists?(outfile)
                `wget -O #{Shellwords.escape outfile} #{Shellwords.escape link['href']}`
                $?.success? || warn("Failed to fetch #{link['href']}")
            end
            
            
            1. 2

              The use cases that I’ve been using PhantomJS for consist primarily of:

              • One-off scripts (i.e., that aren’t intended to be maintained beyond like three days into the future);
              • That are under 50 lines; and
              • That are generally being used as part of Bash/shell scripts where input is piped into other shell utilities

              Even with the flood of comments from the previous thread noting PhantomJS’ limitations, it’s still the cleanest and easiest setup for the use case and scope of those scripts.

              Selenium is easy enough. And there’s an argument to be made that, if recommendations were being made to a novice programmer, it might be better to use that kind of setup even for the use cases mentioned above, since they might be unfamiliar with the limitations of using a library that is no longer being maintained.

              Regardless, PhantomJS has continued to be the most efficient setup for a number of lightweight use cases, particularly where the main concern is setup time, rather than task-complexity or long-term maintainability.

          1. 67

            There are a few key factors I think that come into play. I should note that I only have direct experience with the US market (although I know many people who do or have worked in other parts of the world).

            The first thing I think is to point out that levels.fyi and other sites that focus on salaries at FAANG/tier 1 companies greatly distort the perception of the market. Most developers don’t make anything near the total comp that you see at those companies. Even folks with comparable base salaries aren’t making nearly the amount in equity, even out on the west coast.

            Those salaries are also often for jobs that are clustered in the most expensive parts of the country. For folks living outside of the Bay Area, salaries are much lower in general. The discrepancy in cost of living is astounding.

            Equity is another big part of it. Equity isn’t normal money. Even at large public companies where you can easily sell your shares, the accounting is different. You have to wait for those shares to vest, and you can only sell them at certain times, you don’t know the exact value your getting when you accept a job with an equity component. From the companies perspective issuing equity doesn’t have the same cost as issuing cash, and most people will never actually get all of their equity in reality (they will leave with some equity unvested for example).

            Even with all of that, you might wonder why salaries are so high. Supply and demand is a big part of that. If you look at the trend of companies investing in boot camps and other training programs it’s obvious that they see increasing supply of developers as a strategic way of lowering salaries. In particular I’ve noticed that the salaries for some very common sorts of web development have plummeted over the last 5 years as hoards of bootcampers who’ve been trained in those very specific skills (but intentionally not given the skills to be mobile in the industry or get better jobs) have entered the market and started turning that part of the development field more blue collar.

            Related to supply and demand, I think one strategy that has been inflating developers salaries is that some large companies are hiring everyone they can to starve the market of talent. A lot of developers are paid a lot of money just to prevent them from starting up a competing company, or going to work for a competitor. The laws in California against non-competed has probably helped this some, but even if you can prevent your employees from doing exactly the same work they have been doing, there’s a lot of strategic value in depriving your competitors of talent.

            The last factor I think is the immaturity of the industry. As time goes on and the industry matures, we are going to see salaries for developers depressed and more compensation going to investors and executives (this has already happened with startup equity. Nobody gets rich from being an early employee anymore because VCs have sucked the marrow from that bone already). I personally think a well organized union for software developers to represent our interests is the only way to stop the complete commoditization of our jobs over the next decade, but for it to be effective we’d have to do it now- and the strong undercurrent of right leaning and libertarian leaning culture in tech is likely to prevent that kind of organizing before it’s far too late to be effective.

            1. 14

              If you want accurate statistics about pay for different jobs in the US, the Bureau of Labor Statistics provides very good ones: https://www.bls.gov/oes/current/oes_nat.htm I think it would be a good idea to use these numbers rather than the ones from levels.fyi

              1. 2

                I’d never heard of this website before, so I suppose this post was good advertisement for it. I’m especially intrigued by the $2.5M salaries at Microsoft. I agree with your statement. A good place for actual numbers is also the H1B database, say https://h1bdata.info/index.php.

                1. 1

                  Very true those numbers are more representitive of reality, BUT even my state/metro, I know many developers who are way above the median for the area. I am myself. Of course that’s why it’s a ‘median’ but on the ground pay can be substantially higher than those numbers indicate.

                2. 12

                  I think one strategy that has been inflating developers salaries is that some large companies are hiring everyone they can to starve the market of talent.

                  This is a hard claim to swallow. It requires that very few companies have so much market share that they can overpay developers, and make it up by monopolizing their niches.

                  Maybe this is true of specialized niches, perhaps subfields of machine learning or search, where there are three or fewer companies operating at massive scale. But for “generic” web/application/systems/distributed systems/ programmers[0], you have at least Apple, Amazon, Microsoft, Google, Facebook, and “everybody else” competing for talent. Even if you very generously suppose that “everyone else” only is as big as one of those five, that gives you six players.

                  So with 6+ players, I think that’s too many for a stable overpaying strategy to be going on–someone would start cutting salaries and would save lots of money.

                  [0] I’m “generic”, so don’t take that as a criticism.

                  1. 4

                    They don’t want all the talent as in every developer. She might have meant all the best hires, such as top universities. They’re flooding toward FAANG for the big bucks and prestige.

                    1. 1

                      I was talking about the top of the market, hence my references to GAMFA.

                      They’re a significant minority of overall hiring, but probably a much larger chunk of the very high salaries.

                  2. 7

                    Overall I think that’s a good summary, but it’s missing possibly the most important aspect: the revenue per employee of the top companies is unprecedented. It’s worth paying someone $500k/yr when they’re making you $1m/yr.

                    I personally think a well organized union for software developers to represent our interests is the only way to stop the complete commoditization of our jobs over the next decade

                    I’m not sure how unionization would help. Unions I have been involved with (mostly federal government) are as, or more, guilty of treating people as replaceable cogs in the machine than “management”. In their drive to push for equal treatment, they often end up pushing the fallacy of equal ability. I’ve had discussions with union reps who were literally saying “an employee at pay level X should be able to do the job of any other employee at pay level X”, completely disregarding specialist fields, let alone actual skill.

                    What will stop the (complete) commoditization of software dev is quality. I’m yet to see “commodity dev shops” (including most big name consultancies, TBH) deliver products that are fit for use or maintainable.

                    1. 16

                      Profit per employee isn’t unimportant, but I see it more as something that enables the other factors, rather than a cause in and of itself. Absent the other factors that are driving salaries up, I think more companies would prefer to have a larger profit margin rather than pass that money on to developers. It’s also very hard to know how much profit actual developers are generating in a large company. Even when the story is ostensibly clear (I made code change X that brought our AWS spend down from 100k/month to 50k/month) the complexities of the business make direct attribution fizzy at best.

                      Regarding unions- I think that people tend to look at the most degenerate cases of union behavior and use that to explain why we don’t need one, without considering what a union that was organized specifically for tech workers could bring. Groups like the Screen Actors Guild might be a better starting place than something like a teachers union, because it has to scale across a much broader range of talent and demand. Personally I’d love to be part of collective bargaining for better terms for equity offered by early stage startups (like a longer exercise window), or an organization that helps create guidelines so that “unlimited vacation” doesn’t effectively because “no vacation”. Tech workers are doing good right now and I don’t want to make it sound like things are terrible- but the best time to organize and collectively bargain is when you have leverage.

                      1. 3

                        I think more companies would prefer to have a larger profit margin rather than pass that money on to developers.

                        Of course, that’s where competition comes into it. But fundamentally, without high revenue per employee, none of the other factors come into play.

                        I think that people tend to look at the most degenerate cases of union behavior and use that to explain why we don’t need one

                        I’m open the the idea that there are possibly good, useful unions. They’re just not something I’ve seen in the real world. The main union I have experience with represents 10s of thousands of skilled workers across many disciplines and still struggles to be, in my estimation, a net positive.

                        The collectivist nature of unions means that they often act too much in the interest of the collective, regardless of what that means for individuals. That can translate into being more concerned with maintaining their power base than necessarily improving member outcomes (though you’d hope those two things would be at least partly aligned).

                        1. 4

                          It’s also very hard to know how much profit actual developers are generating in a large company.

                          And irrelevant. Companies don’t pay what people are worth, they pay the minimum they believe they can while keeping the person on staff and productive. It is all about the competitive marketplace and individual skill (both in the trade and in negotiation).

                          Right now, it is easy to leave a company for a large salary boost, so to retain people you have to pay them enough to make taking the risk of leaving not worth it – that is all.

                          1. 1

                            It is all about the competitive marketplace and individual skill (both in the trade and in negotiation).

                            It is for the low-level, production workers. The executives get routinely overpaid with all kinds of special deals and protections. I think either they should be forced to compete in a race to the bottom like the rest of us (free market) or we get protectionism and/or profit sharing, too.

                          2. 1

                            It’s also very hard to know how much profit actual developers are generating in a large company.

                            Divide the total profit by the number of employees.

                            This isn’t a perfect solution, but it gets you pretty close.

                        2. 4

                          Thanks for the detailed answer. I am not that knowledgeable about the job market being a junior CS student myself, but the point about bootcamps intentionally not teaching the skills to be mobile and get better jobs caught my attention. I am aware that learning ‘basic web development’ is perhaps the fastest way into the job market, so most bootcamps focus on that, but what are those other skills you mentioned that bootcamps should be teaching but aren’t?

                          1. 4

                            but what are those other skills you mentioned that bootcamps should be teaching but aren’t?

                            Mostly; math, algebra’s, algorithms, data structures, complexity theory, analysis, logic, set theory, consultancy skills, scientific methods and everything that allows you to build better software and frameworks than there are currently out there. You need some hardcore coding skills and formal skills to build something better than what’s already out there, but it’s totally doable with the right education, because most of the industry is just selling “hot air”. Basically: You should be able to read and understand the four volumes of “The art of computer programming” without to much trouble. If you can do that, you’re there.

                            That being said: It’s more about you picking your educators carefully, than it is about getting a degree. You’d still need that degree, but where it comes from is way more important.

                          2. 2

                            I can’t upvote this enough! Great explanation.

                            1. 1

                              In particular I’ve noticed that the salaries for some very common sorts of web development have plummeted over the last 5 years as hoards of bootcampers who’ve been trained in those very specific skills (but intentionally not given the skills to be mobile in the industry or get better jobs) have entered the market and started turning that part of the development field more blue collar.

                              One consideration with web development is that the amount of money you can make scales with development speed much more than total cost-per-project/website made.

                              Small/local business websites generally go for somewhere in the range of $1k to $3k, but the difference in development speed to get those sites live is massive between developers. There are developers in India et., al. that will take 150 hours to complete a semi-custom WordPress site (which would be around $10 per hour) and there are developers that can build the same site at the same level of quality in 15 hours (which would be $100 per hour).

                              Also, accessibility to CMS tools, plugins, and so on is another reason why that type of work has become more blue-collar-y. You can make very solid small-business type websites with little-to-no programming experience in Current Year.

                              This is from a contract-based/freelancer point-of-view. It’s possible that the scenario is different in the corporate/employee-basis web development world.

                                    1. 6

                                      There’s also headless Firefox, and a phantomjs equivalent… in case you don’t want to perpetuate the chrome monopoly

                                      https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Headless_mode

                                      https://github.com/laurentj/slimerjs

                                      1. 1

                                        Perfect – Interested to see how the various headless libraries compare. FF seems like a better option than Chrome; will find out.

                                  1. 2
                                    1. 1

                                      Tru. Am basically using it to grab a page’s post-JS-executed HTML source to then pipe into other utilities in BASH, which seems to work fine. Will see if it catches fire with more complicated scripts.

                                      There’s also this:

                                    1. 2

                                      Looks pretty interesting. One of my friends was trying to rig an online poll at my school, so he went through the trouble of getting Python to open up his web browser and interact with his screen. It was a very finicky thing to do since he effectively couldn’t use his computer while it was running. With this, it would have been a heck of a lot easier. Sad to see that development is effectively over at this point.

                                      1. 2

                                        The other option that might have worked there is Selenium WebDriver:

                                        It was a very finicky thing to do since he effectively couldn’t use his computer while it was running

                                        Using a VM/container also helps with preventing scripts that emulate the browser from constantly taking focus when windows are opened/closed/other.

                                        1. 1

                                          Selenium is a ball ache[1] but mostly can be called upon by creating a python virtual environment just with:

                                          virtualenv --python=python3
                                          ./bin/pip install selenium
                                          

                                          You can then grind through the usual python tutorials to solve your problem[2]; PhantomJS supporting a webdriver socket helps too.

                                          IIRC you can make Firefox/Chrome less crap if you specify a profile directory to use so that it does not piggyback and couple its-self to any existing instances of it already running.

                                          [1] though probably more due to my experience of dealing with BrowserStack being a PoS

                                          [2] I personally then drive Python via a pipe using a less-awful language…such as Erlang or Perl :)

                                          1. 1

                                            It shouldn’t create visible windows if Chrome is started with --headless and Firefox with -headless.

                                        1. 3

                                          Quality YouTube channel. One of the best videos I’ve seen posted on lobste.rs.

                                          Went through the channel and found this, which is part of a series:

                                          1. 2

                                            The issue with the robots.txt standard is that while it’s hypothetically useful, it’s hard to justify creating one, since implementing a mandatory, non-suggestion solution via a .htaccess file (or similar) is superior in basically every possible way.

                                            It’s a useful tool for E-Z blocking of “nice” bots, but too reliant on not only bots actually caring enough to read it at all, but also to care enough to parse it correctly.

                                            (Good post though)

                                            1. 2

                                              but also to care enough to parse it correctly

                                              Well, luckily Goog released their robots.txt parser just today :)

                                              https://opensource.googleblog.com/2019/07/googles-robotstxt-parser-is-now-open.html

                                            1. 1

                                              Spent a week or so on and off winging it and it’s alive. Tried to document everything fairly extensively while figuring out the circuitry, GPIO interactions, and so on.

                                              Raspberry Pi 4 came out partway through writing this post recently. Perhaps some of you have picked up various neat Rasp. Pi projects recently as well.

                                              Loosely inspired by this guide from around 2007.

                                              1. 1

                                                Full single-page HTML to test the code snippet in that file below. Make sure to include the JQuery UI library in addition to the base JQuery library:

                                                <!DOCTYPE html>
                                                <html>
                                                <head>
                                                  <title>Page Title</title>
                                                
                                                  <link href="http://code.jquery.com/ui/1.10.2/themes/smoothness/jquery-ui.css" rel="Stylesheet"></link>
                                                  <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script>
                                                  <script src="http://code.jquery.com/ui/1.10.2/jquery-ui.js" ></script>
                                                
                                                </head>
                                                <body>
                                                
                                                <h1>This is a Heading</h1>
                                                <p>This is a paragraph.</p>
                                                <input type="text" name="search" id="search" placeholder="Search Here" />
                                                
                                                <script>
                                                var suggestCallBack; // global var for autocomplete jsonp
                                                
                                                $(document).ready(function () {
                                                    $("#search").autocomplete({
                                                            source: function(request, response) {
                                                                    $.getJSON("http://suggestqueries.google.com/complete/search?callback=?",
                                                                            {
                                                                              "hl":"en", // Language
                                                                              "ds":"yt", // Restrict lookup to youtube
                                                                              "jsonp":"suggestCallBack", // jsonp callback function name
                                                                              "q":request.term, // query term
                                                                              "client":"youtube" // force youtube style response, i.e. jsonp
                                                                            }
                                                                    );
                                                                    suggestCallBack = function (data) {
                                                                            var suggestions = [];
                                                                            $.each(data[1], function(key, val) {
                                                                                    suggestions.push({"value":val[0]});
                                                                            });
                                                                            suggestions.length = 5; // prune suggestions list to only 5 items
                                                                            response(suggestions);
                                                                    };
                                                            },
                                                    });
                                                });
                                                </script>
                                                </body>
                                                </html>
                                                
                                                1. 1

                                                  Previously submitted a similar post here:

                                                  Title truncated slightly due to 100 character title limit.

                                                  1. 1

                                                    I would link to a specific post, but this site/author (Jason Brownlee) publishes multiple high-quality posts per week. Too many things to link.

                                                    1. 15

                                                      Because PC gaming is largely a windows monoculture.

                                                      1. 2

                                                        This + video games are one of relatively few applications that run directly on the end user’s machine, rather than through a browser, which makes that distinction particularly relevant for developers.

                                                        1. 1

                                                          I remember there were attempts to change this, though. There was the Games Kitchen back in the early Power Mac days which was allegedly a trial balloon to attract more ports and hopefully first-party titles to the Mac (if you go to the About window in Mac Dark Forces, there’s a shout out). But Apple never had their heart in it and this lasted well into the Jobs era. Gabe Newell when asked why Valve’s Mac support was so poor back then famously complained that they would find problems they wanted fixed, and someone would say they would, and that someone would end up working somewhere else, and their problems never got repaired because then it was a new someone else who was unaware of them. I’m not sure if that’s why they never released Half-Life for the Mac, despite the fact that Logicware pretty much had it completed, but it was certainly why there was never a Half-Life II ( http://archive.videogamesdaily.com/features/gabenewell_valve_iv_sep07_p1.asp ). But Valve never cracked the formula themselves either. The Linux-based Steam Machines didn’t exactly sell off the shelves.

                                                          At the end, I guess it’s where the market is.

                                                        1. 6

                                                          Vim seems to have an incredibly long phase where you can call yourself “intermediate”, so that two experienced users can have two disjunct sets of things they know, “but most people don’t”.

                                                          1. 1

                                                            True. Am not surprised that a number of others in this thread already knew some of these tricks. I used Vim for ~5 years before discovering :autoindent, after which I promptly turned it off.

                                                          1. 20

                                                            I knew four. I feel cheated by a clickbait title.

                                                            And about the one I didn’t know, I think it’s better to understand how to paste from the system clipboard using the verb for paste and the noun referring to the system clipboard.

                                                            1. 6

                                                              This gives me a post idea: “At least two things in vim you don’t know.” The post covers 30 different things. Statistically speaking, you don’t know at least two of them!

                                                              1. 2

                                                                Might steal that idea from you.

                                                                1. 1

                                                                  I’d bet money that this llama would know all 30.

                                                                2. 1

                                                                  I kinda feel cheated too. Actually, I feel mocked. Showing the line numbers? Seriously?

                                                                1. 4

                                                                  Animated spinners are possible in CSS as well. No need to chuck a gif in there!

                                                                  1. 2

                                                                    Yeah, but title says “5 Tasks You Didn’t Know Could be Done…” and animated spinners in CSS is something I believe to be common. The 5 in the article are, at least for me, some gems.

                                                                    1. 1

                                                                      Good point. Probably even worth it, since would be like 1kb of CSS instead of the 50 kb gif.