I thought I did, but I didn’t realize I had posted that one here, this was more of a re-write to capture the core ideas from that. I will go ahead and delete this one + check if there’s anything there I hadn’t answered.
Why are hardware advances linked to RISC-V? I understand why it is nice to have an open platform, but I am not sure how this helps Machine Learning. Is it because custom new AI hardware will be possible to manufacture without license? NVIDIA is already investing so much in ML, why are they not already making what people want?
5 people with capital “P” and maybe some “C” in their job titles
What does this mean? P’s are presidents? What is C?
I guess I’m pointing to it more as an example of open-source hardware potentially leading to medium-sized shops being able to have their own custom-made ml chips that are more than an FPGA but less than investing a few hundred billion into being the next intel/amd/nvidia/ibm.
The link itself is more like “food for thought, this might be one such thing”, but maybe a bit misleading.
As for that quip with C, see what /u/FRIGN is saying
I bought an open EIT device to hopefully take images of my insides, turns out it’s harder than I thought, will keep working on getting meaningful data out of it.
Maybe I am missing the point of the post, but I don’t agree with some of the points you raise.
Human brains are shit at certain tasks, things like finding the strongest correlation with some variables in an n-million times n-million valued matrix. Or heck, even finding the most productive categories to quantify a spreadsheet with a few dozen categorical columns and a few thousand rows.
That might be true of individual brains, but you can put together a lot of people and get to the same result. See how mathematical computations were carried out by humans computers before the introduction of electronic computers.
Similarly, I don’t agree with the overall framing of your critique. You can say something like “Human bodies are shit at certain tasks, things like digging a 100ft long 10ft deep trench. Or heck, even walking more than a few miles a day or even run a few hundred feet”. We have excavators and cars for a reason. Similarly, we have computers to do “boring” computation work for us.
This is not to say that the scientific establishment is doomed or anything, it’s just slow at using new technologies, especially those that shift the onus of what a researcher ought to be doing.
Well, isn’t there the saying that suggests scientific progress happens one funeral at at time? It feels to me that the problem is that researchers tend to ask questions that they can reasonably answer. The questions that they ask are informed by their methods and theoretical preparation. So, if a researcher doesn’t have a good understanding of machine learning models and what they could do, they will probably not think of a research line that will leverage these new methods to move the field forward. Instead, they will probably just apply them toy to an old problem (which will probably be solved by an old boring method just as well).
I wonder if we are at the intersection of “old” and “new”. While we are at the intersection, we are still applying the “new” models to the “old” problems. Our problem space hasn’t opened up yet to the possibilities that the “new” methods opened (maybe because we haven’t figured them out yet). That makes me wonder whether we need to push ourselves to imagine and tackle new, unexplored problem spaces that machine learning (or any new technology) has opened for us instead of trying to solve old problems with the new methods.
That might be true of individual brains, but you can put together a lot of people and get to the same result. See how mathematical computations were carried out by humans computers before the introduction of electronic computers.
That being said I don’t think I have an argument against this, but my intuition based on how many applications nns have found in terms of outdoing equations in hard sciences like physics and chemistry is that we aren’t super good at this, even if it’s 100k of us working on it together.
Similarly, I don’t agree with the overall framing of your critique. You can say something like “Human bodies are shit at certain tasks, things like digging a 100ft long 10ft deep trench. Or heck, even walking more than a few miles a day or even run a few hundred feet”. We have excavators and cars for a reason. Similarly, we have computers to do “boring” computation work for us.
In hindsight, yeah, I think the framing is a bit off.
I have a theory that popularity of vision and language is due to data. There are endless amount of unlabeled data, and labeled data can be crowdsourced cheaply.
People like Horvath are hailed as genius-level polymaths in molecular biology for calling 4 scikit-learn functions on a tiny dataset.
Horvath spent over 4 years collecting publicly available Illumina DNA methylation data… (snip) The age estimator was developed using 8,000 samples from 82 Illumina DNA methylation array datasets.
It is both true that 8,000 samples are tiny, and it took 4 years to collect. Majority of machine learning effort is data collection, not data modeling. Data collection is easier with vision and language, even though data modeling is higher impact elsewhere.
I have a theory that popularity of vision and language is due to data.
Well to add to this, CV and NLP are old fields. They’ve been collecting and cleaning datasets for decades now. Applying newer NN-based ML techniques was as easy as taking an academic dataset and training the nets. Other fields don’t have nearly the same history of pedagogy and so they’re probably going to take a lot longer to collect data (usefully) for.
Bio is also special…. Consider a genome. Most of your genome is the same as everyone else’s, so even though it’s a lot of data, the degrees of freedom are considerably lower. Also, even if we were to sequence every American and european, that’s only 750M data points (most of which self correlate)… Wiipedia alone kas 3.9 billion words.
This would be true for just DNA and genetics. If you include epigenetic information and start going into gene expression in different cell types/tissues, there’s probably a lot more variation, but I don’t think we’ve quantified it yet.
I would attribute the popularity of language and vision more to problems that can be well modeled on GPUs w/ neural nets, as well as having massive amounts of labelled data. ImageNet existed well before the computer vision breakthrough by Hinton’s team. It was applying GPUs for process + neural networks that did the trick.
I think that the human-level intelligence is also partially a way to enable more people to “access” APIs, especially the people with less affinity towards technology.
Just as an example, consider ordering a pizza. If you are able to order a pizza via a website I have never not seen the option to add custom text regarding the order. You simply will have people that do not have their needs met by the ordering mask or do not find the option they really want. In this case it can make sense to have a computer attempt to decipher the customers wish.
This could also be applied to ordering via phone, which is still very common in Germany. This would be another avenue where voice recognition could make sense and improve efficiency. But ultimately, those problems are intricately connected to UX and not necessarily to fancy ML models, too.
In the end these types of ML models reach more people and it’s important that they do not feel left behind, which can cause rejection and opposition to modern technology. That is a very social problem and I am not convinced that there is a technological solution to it.
I think I agree with this, it might be that the “left behind” angle is one I don’t consider enough when thinking about this, ultimately the vast majority of people, can’t use google maps, heck, even many professionals like cab drivers which would benefit immensely from it.
I guess I’m not sure how much of this can be fixed by “education” once technology becomes to useful to be ignored, but it’s the above google map example that made me hedgy on this, after all it’s a prime example of a life-transforming technology that many people just ignore.
Maybe in some cases it’s just more fun to live life without technology? I’ve heard this argument made by literate people that preferred using paper maps and compass navigation just because it was exciting. But under that scenario I’m not sure simpler UX would help, or that any help is needed per se
I have a personal theory that many people firmly believe, in a deep, maybe even unconscious way, that they are entitled to stop learning new things at some point in their lives. It’s something I’ve been noticing more and more with the pandemic cutting off so many “non-tech” ways of doing things.
I have noticed this too. It really pains me. I personally favour the hypothesis that our education systems are to blame. All humans start off with curiosity as a primary motivator, perhaps the primary motivator after survival stuff. It seems there is a strong correlation between attending a few years of school and this no longer being a strong motivator for many people. As far as causation goes there are reasons to favour this hypothesis - the degree to which curiosity is actively suppressed in most education systems, and also the degree to which education is made to be associated with bland, boring and generally unpleasant hard work, rather than discovery. Unfortunately there is not a lot of good data regarding well-educated individuals that have not been subjected to a classical western style education. There are alternative systems like Montessori but there is no clear way to quantify cognitive ability, we use western school performance for that in most instances. The Finnish school system is an interesting case study though.
This may all seem off topic, but to tie it in to the main theme: It is actually trivial to create a human level intelligence. Many people do it by accident when they are drunk. This makes creating them artificially doubly redundant. I think that parallel to improving inhuman AI as the article suggests, we should be looking into improving the human level intelligence we have easy access to, which is mainly a software problem. Most of the software that is widely available for the human brain is decades out of date and designed for applications that are no longer needed.
To some extent though this might be a good conservative instinct.
Don’t get me wrong, I like being in some “privileged” 0.01% of the world’s population that knows the technology well enough to make 6 figures writing stuff on a laptop, I’m sure most people here feel the same way.
But if I had to choose between being in the top 20% or in the bottom 20% I’d choose to bottom.
Technology very quickly becomes a tool that most people end up using to fulfil destructive patterns more than anything.
That’s not to say it has no uses, it does, but it’s way too easy to get into pitfalls if you spend too much time on computers and don’t understand how good certain programs are at manipulating your preferences.
I’ve noticed it myself with myself. After 30 years in the computer industry, I’m just sick and tired of having to relearn the hot new development methodology. Again. For the umteenth time. Can’t things just settle down for a few years? It seems as if the singularity has hit and no one can keep up.
I suppose to some people it is more fun. I write my day-to-day plans still on paper because I prefer it to electronic bookkeeping. The battery of a sheet of paper cannot run out, although my phone hardly does either these days. But there still is something that makes me prefer to write them by hand, it also helps me remember what I actually wrote down. To some extent it is also control over the medium; I know that I am responsible if I lose it or it somehow is destroyed. I guess the aspect of control and being in charge can also translate to navigation or in general other aspects of life (although I have to agree with you, navigating via online maps is much better).
Potentially education could help with furthering usage of technology. But it could also be that most people are just stubborn and want to continue with their way of life indefinitely. Depending on how big that share of people is, it is important to make new non-human intelligence backwards compatible, so to say. Then once most people stop relying on the old technology (die out?) it can be deprecated.
Trains, assembly lines, and planes are not fully automated in the way that self-driving cars are automated; they have human staff on hand who can manage irregularities and small problems. If we use an apples-to-apples comparison, where the cars are attended by humans but centrally coordinated and powered, then the Commercial Railway seems to have opened in 1840. Certainly, by the 1880s, streetcars and trams were appearing in many cities. In general, the history of trams and light rail is part of the history of trains. I think that bridging the gap between this sort of transit and commuters is a last-mile problem, not necessarily an automation problem.
That’s actually a good point, but the automation is responsible for flagging irregularities for the human operator to look at. So in the case of a car, this could essentially mean “slow down, and stop”.
Planes and most trains do usually have on-board operators but this seems to be mainly a regulation issue.
Chesterton’s Fence applies here; the regulations have a rationale. We want heavy machinery to have operators in case of emergency or exceptional situations which go beyond the capabilities of automation.
With e.g. trains there are fully automated systems that work just fine.
For a plane you could indeed argue cost of a pilot is small enough to justify it as conservative safety, same for a train that goes over unusual and varried terrain (as opposed to, say, a metro)
I wonder if there aren’t examples of the AI->Automation process which have already occured? Maybe that’s just called automation?
I’m thinking about machine-readable data formats and CPU or database dependant operations gaining acceptance over time (e.g. the standardization of VINs in the 80’s).
I don’t think that AI is special in this regard… if anything I would expect it to accelerate the process in all cases where it is present.
I wonder if there aren’t examples of the AI->Automation process which have already occured? Maybe that’s just called automation?
I can think of: companies introduced AI in their customer support, then realized they could just automate it. For a successful example see Amazon, for a failed one Revolut. At least that’s my reading of the situation but I can’t be sure.
But it’s not a very good example, I’ll try to look into this.
I don’t think that AI is special in this regard… if anything I would expect it to accelerate the process in all cases where it is present.
If you believe the thesis of the article, OCR is AI made necessary only by our civilization’s inability to digitize everything, while automation would do away with both papers and the need for OCR.
You might be confused because AI is often seen as one kind of automation. I believe the article tries to distinguish them by correctness (without actually using that word). OCR and other “AI” can make mistakes and require human intervention, while “automation” can be proven correct.
I think it’s self evident that a world without humans is easy to manage by shifting data to memory regions.
We all know printers are devilspawn.
Getting computers to work with us instead of us working with them, that’s the trick, and I’m not sure automation solves that problem. In fact I might be so bold as to say AI gets computers to work with us, and automation helps us work with computers.
I defined “AI” for the purpose of this as “using the term to mean some heuristic for learning equations from data or simulations rather than writing them out by hand”.
So I think OCR fits as AI, unless there’s a way to do OCR without training data.
A while ago I got this problem of the fuzziness of the term “AI” stuck in my head and thought about it a lot. I think “a computer program that takes on the world on its own terms (without the situation being arranged to suit the computer program)” is somewhat accurate. I think this matches your definition of “AI” and the parenthetical fits how you contrast it against “automation”.
The real question here is, I feel, why did the writer this not do it on their own and never bothered anyone?
Like, not calling her lazy or anything, but why wasn’t there an incentive for him to just get it done on his own? Since it’s obviously possible.
Indeed, getting anything “simple” to the level of “management handled issue” will by default add a standard amount of overhead (e.g. 10 mins in a startup, 1hr in round B, few weeks in a megacorp). But that’s just the nature of consensus making in large orgs… however a lot of things can be resolved exactly by not having to get them to management, by just doing them yourself (or yourself + a group of friends that don’t mind asking for help)
First, I believe the “guy” writing at the “Rachel by the Bay” blog is a woman.
Second, they did write an initial dashboard by themselves, but then a “dashboard team” came along and insisted they’ll provide said dashboard—and then promptly decided they had more important things to do for 5 months. Here’s paragraph 3 and 4 of the post:
January 1: we put up a terrible hack: a shell script that runs a Python script that talks to the service to get the status and then dumps out raw HTML to a file in someone’s public_html path.
January 29, early: there’s this team that nominally owns dashboards, and they got wind of us wanting a dashboard. They want to be the ones to do it, so we meet with them to convey the request. We make a mockup of the list and the eventual big red button to give them some idea of what it should look like.
First, I believe the “guy” writing at the “Rachel by the Bay” blog is a woman.
Ups, edited
Second, they did write an initial dashboard by themselves, but then a “dashboard team” came along and insisted they’ll provide said dashboard—and then promptly decided they had more important things to do for 5 months. Here’s paragraph 3 and 4 of the post:
What I was saying here was that, i.e. “WHY IS THERE a dashboard team!?” that seems like the fundamental question to tackle here. Why does the company not provide incentives to just leave the stupid python script up, or get the devs to spend the few days they did writing the dashboard without implicating a whole other department.
I think the author might also think the same thing here, I was trying to paraphrase my takeaway, not criticize.
“WHY IS THERE a dashboard team!?” that seems like the fundamental question to tackle here.
I can answer this question as if it was the BigCorp for which I’m currently working. Please take my answer with a grain of salt, as this is most likely biased by my “grunt” point of view.
Up high, the CEO most likely decided they wanted to reduce cost on the technology.
They gave this task to a Senior VP. This senior VP discussed with a VP and a few directors, and they came up with one of the conclusions that the technology is too fragmented across the organisation, and too many people solve the same problems.
The task got split up into smaller chunk “consolidate server technology”, “consolidate dashboard technology”, etc. At some point, some director got assigned the objective to “consolidate the dashboard technology”
This director within their budget created a “dashboard team”
Now you have a team which sole purpose is consolidate the dashboards, their performance is measured every quarter in terms of: how many dashboard were consolidated, and how many new dashboards are created by this consolidation team.
This team now hunts down anybody who is using a dashboard or creating a new one, the team existence depends on it. (If they don’t perform well they will get reassigned, and members will loose the opportunity to be promoted quickly, and/or face the possibility to made redundant.)
Her name is Rachel and she is not a guy. If you want to use gendered language you should do your research on not jump to conclusions. The name is even visible in the domain.
Gendering someone based on a personal name that you found in the domain name of the site where you found an article sounds like a lot of making assumptions to me. And in the glorious twenty-first century, anyone and everyone is free to be a “guy”, just like in the 90s we were all dudes.
Overview/review of the alpha fold paper that doesn’t require a lot of machine learning of molecular biology expertise to understand and can be read in 10 minutes.
I’m not sure I succeeded, it may well staddle the line poorly enough that it’s both uninteresting to someone familiar with the subject and unpenetrable to someone that isn’t.
But I almost wasted a whole day reading, writing, arguing, and editing to get this done. So I decided I have to post it somewhere to see what people think.
If you believe it’s too unrelated/abstract/bad then do let me know and I’ll delete it, I think I’ve been posting some mediocre articles here lately and I feel bad about it and want to stop, I appreciate the relative quality of this board as much as anyone. But it’s kind of hard to tell before I see people’s reactions, especially with stuff that you spend a lot of time writing.
I would write one or two sentence at the very beginning detailing your background, basically making clear what your focus will be on (more computer science or biology focused?). This should help the reader set his expectations accordingly. You can so write explicitly what you think the reader should ideally know already.
in section ii you describe the model somewhat closely. I think it would make sense to cite the paper in more detail, where the first pipeline is mentioned or where the evotransformer is first defined. It seems like you oriented yourself on the paper, but the reader will most likely not have it open alongside the article.
I haven’t read the alphafold paper yet, but your impressions are helpful to me personally, so thanks for writing it up. I liked the personal take and especially admitting what you don’t understand is nice to read, because reading papers is hard and not often acknowledged.
Think for a second of a future where we can open a browser and run tf/torch, starcraft, sqlite, postgres, clang, brew, wayland, R studio, a bsd or linux VM, octave, the .NET ecosystem, vim, vscode, another browser.
NetBSD’s rump kernel has been able to run in a browser for well over a decade (perhaps two?) And I’ve seen Gameboy and Nintendo emulators that run in a browser.
Just how complex browsers are these days boggles the mind. They’re also nigh impossible to secure.
NetBSD’s rump kernel has been able to run in a browser for well over a decade (perhaps two?)
Can you give some more detials here? Do you mean in userspace or as part of browser’s code?
I guess I should have mentioned I was referring soley to userspace. If that’s the case… I don’t really see the reason why you might want to do that. Maybe to run it on in-browser simulated hardware?
NetBSD has pretty good documentation on rumpkernels. Effectively, it’s akin to running the kernel as a program in userspace. Rumpkernels allow one to develop and test new TCP/IP stacks and filesystem drives, for example. Pretty cool methodology of OS/systems development.
It doesn’t seem to me like it’s running in a web browser, but rather that it’s running the web browser.
Relevant section being:
This section explains how to run any dynamically linked networking program against a rump TCP/IP stack without requiring any modifications to the application, including no recompilation. The application we use in this example is the Firefox browser. It is an interesting application for multiple reasons. Segregating the web browser to its own TCP/IP stack is an easy way to increase monitoring and control over what kind of connections the web browser makes. It is also an easy way to get some increased privacy protection (assuming the additional TCP/IP stack can have its own external IP). Finally, a web browser is largely “connectionless”, meaning that once a page has been loaded a TCP/IP connection can be discarded. We use this property to demonstrate killing and restarting the TCP/IP stack from under the application.
Those demos showing the rumpkernel booting in the browser itself seem to be no longer available on the internet. That’s a shame, because rumpkernels-in-js are awesome.
This. This is why independent and different implementations matter. If you understand the web as an application platform you will realize that some browsers should not be able to dictate how it works. And there is little independence out there…
The problem boils down to something like “who will pay for independence” ?
As in, dontaions is not a reliable model, and ads are blurrying the line of ethics.
Personally I always though there’s an audience for a premium browser that includes other cost saving features (e.g. banning spammy websites, currated suggestions, built-in unblocking for papers, easy download for all mutlimedia, built in search engine that behaves based on your stated preferences rather than your behaviour-revealed preferences etc). But the problem is that most of the audience that would understand why such a browser is necessary are the peole that need it least.
One could also hope that some crypto billionare might just give a 10B endownment to a foundation to build a competitive browser under very strict decentralized supervision, but afacit that hasn’t happened yet.
This is a really interesting view into the future – and I’ve become, oddly, a bit less worried about it as time goes on.
Google Docs recently announced – and we discussed that they’ll be doing canvas-based rendering of documents. Does this further lock us into the Google monopoly as they now do everything server-side? Possibly, but is it truly any worse than the webasm/minified JS monstrosities that are modern “web apps”?
It feels like the web is starting to split: on one side a large amount of data is still following the hypertext vision we’ve traditionally chased, alternative decentralized networks like Mastodon and the Tildeverse are taking root, and hell – we’ve got people writing Gopher pages! In this century! On the other side, we have video games using OpenGL in your browser.
Stuck in the middle is the last independent browser trying to decide which of the competing standards to chase. Perhaps a hard fork between “the simpler, hypertext web” and “the web as a way to deliver arbitrary, sandboxed applications” helps them decide where they want to spend their more limited resources.
I dunno. Maybe I’m just the frog that’s been boiled slowly enough that I’m not as worried about this future, but I’m not convinced it’s entirely terrible.
Does this further lock us into the Google monopoly as they now do everything server-side?
What do you mean, server-side? My assumption is that the only change here is that the ‘rendering’ step now just calls the canvas draw methods as opposed to generating HTML, along with accompanying changes in how they figure out what user input should do (since now every click will hit the same element). It definitely makes it harder to pull out the textual content from extensions and the like, but I don’t think this really increases lock-in.
And I think the problem is that there is no actual hard divide between “simple hypertext” and “arbitrary sandboxed applications”. It’s funny you mention Mastodon, because the default Mastodon frontend is, conceptually, much closer to the application model. Clicking profiles doesn’t navigate you to separate pages, it does an XHR and then renders that in a new ‘column’. It even literally embeds an OCR engine that’s compiled to wasm for generating image captions!
What I’m unclear on is whether or not there’s an actual split to be talked about.
As in, you could still support html+css in a browser as VMs setup, at most they’d have to include something like an HTML+CSS “special compiler” of sorts that adds some pre-defined rendering logic.
And I doubt any browsers will want to loss e.g. wikipedia or the blogosphere. More interestingly though, if javascript becomes something closer to flash (unsafe, redundant, nobody uses this anymore kinda thing) it might prompt minimal-javascript websites to switch to being fully static.
Lobster might be a good example of this where (while I can’t speak for the devs) I could see it being easy to port back into raw HTML, and given enough js related issues some people might just say fuck it and go back to HTML instead of chasing compatibility. The fact that HTML has added many quality of life elements and tags also helps.
I think a big reason Flash had the (justified) reputation it did is that it fundamentally had less of a sandbox mechanism than JS, and the developers just were not as security-focused as browser developers were/are. And as JS grew more and more powerful, there was less and less reason to use Flash. I don’t see this happening to JS unless something new comes to replace it, and I don’t think this will happen; in particular, I think the ability to just mess around in the web console is incredibly useful, and you just don’t get that with wasm.
Why don’t you get that with WASM? As in, I get that you don’t get that today but in principle, nobody is stopping an addon that compiles wasm to js and runs it in the console. And/or direct vendor support if it gets popular enough… it’s a bit annoying since you’d have to specify to the browser a path to the compiler of choice (since you’re not going to ship browsers with all of them). But that’s a one-time setup.
Or do you mean that people won’t be able to mess with the code of the websites they are using?
That also seems untrue, given that EOD WASM just modifies the DOM. You’ll still be able to use the console to add functionality. And most websites, even “puritan” ones ala lobsters come with compiled javascript that’s impossible to mod.
There’s a dying breed of dev (e.g. myself) that enjoy shipping readable js to the user, but that’s so niche as not to matter.
This is an interesting article. I have a few “big picture” issues with it that I discuss below.
The argument in the introduction seems somewhat all over the place. The points that the author raises are interesting, but I am not sure if I agree with the conclusion. For example, the author claims that
Furthermore, the complexity of the statistical apparatus being used leads to the general public being unable to understand scientific research.
as a critique of “classical statistical” methods. While I agree with the author that scientific and statistical literacy is a problem, I am at a loss of the eventual conclusion that statistical learning models, that are even less interpretable than classical models, are a solution for this problem.
I am also not sure if I follow how these three conclusions that the author highlights for the post follow from the article:
Simplicity; It can be explained to an average 14-year-old (without obfuscating its shortcomings)
Inferential power; It allows for more insight to be extracted out of data
Safety against data manipulation; It disallows techniques that permit clever uses of “classical statistics” to generate fallacious conclusions
As far as 1, a 14 year old in a regular algebra class will be able to carry out a simple linear regression with 1 variable. That’s because the methods are rather simple and it is easy enough to understand the difference between the line of best fit and the observed points. I am not completely sure if the same is true for even the simplest statistical learning models.
Maybe I am not understanding what “inferential power” means here, but my understanding of machine learning is that those models excel at predictions, as in out of sample predictions, rather than inferences. I am also fuzzy about the kind of insights that the author is going to discuss.
Finally, I am not sure how data manipulation (as in data cleaning, for example) is connected to data analysis and how statistical learning “fixes” them. All the ideas that are discussed further in the post seem to be on the data analysis side, so I don’t see the connection between this last insight to the rest of the post.
As a final comment, most research is done to answer specific research questions. Usually a researcher asks a question and picks the best methods to answer it. The author seems to advocate moving from inferential questions (e.g., that is the difference between a group that received X and a group that didn’t) towards predictive questions (e.g., what will happen if this group receive X, maybe?). I wooden if a better motivation for the post needs to be an epistemological first rather than a methodological/technical one.
A few more fine-grained questions:
In point iii), isn’t the author just describing jackknifing/leave-one-out? The same idea can be applied to classical methods to calculate non-parametric standard errors/p-values. What am I missing here?
In point vi), there seem to be a strong assumption that all confounding variables are observed and that the deconfounding technique that the author describes can be applied to the data (by the way, if I understood the procedure that is described, it should be the same idea behind classical multiple regression). The most difficult thing about deconfounding is trying to account for the effects of unobserved variables (e.g., omitted variable bias). I am not sure if machine learning methods are better or worse than classical methods to address these issues.
that are even less interpretable than classical models, are a solution for this problem.
I agree that ML models are less interpretable. But what I think lends itself to explainability is performing CV and reporting accuracy as a function of that (easy to get, relatively speaking) and making the error-function explicit, i.e. informing people of what you’re trying to optimize and comparing your model to a “base” assumption or previous models on that metric.
You don’t need to understand how the model operates to get the above, but in statistics, the underlying model (assumptions about it’s error distribution) is actually relevant to interpreting e.g. the p-values.
All the ideas that are discussed further in the post seem to be on the data analysis side, so I don’t see the connection between this last insight to the rest of the post.
I focus on how data analysis can be done under this paradigm because a lot of the strong argument pro using outdated statistical model seems to focus on their power of “explaining” the data while making predictions at the same time. So I’m trying to showcase why that also works just as well if not better with more complex models.
I agree it’s in part a different point.
I wooden if a better motivation for the post needs to be an epistemological first rather than a methodological/technical one.
In part the post came as part of a long series of post about why a predictive based epistemology is the only way to actually make sense of the world, and why e.g. a “equation first” epistemology is just a subset of that which happens to work decently in some domains (e.g. medieval physics).
But I guess the article specifically was meant mostly towards people that already agree a predictive epistemology makes more sense to pursue knowledge about the world, to actually lay out how it could be done better than now.
In part my fault for not tackling this assumption, but at that point I’d have had a literal book instead of a way-too-long article.
In point iii), isn’t the author just describing jackknifing/leave-one-out? The same idea can be applied to classical methods to calculate non-parametric standard errors/p-values. What am I missing here?
You aren’t missing anything. Classical methods would work just fine under this paradigm and I assume for many problems a linear regression would still end up being a sufficiently complex model to capture all relevant connections.
In point vi), there seem to be a strong assumption that all confounding variables are observed and that the deconfounding technique that the author describes can be applied to the data (by the way, if I understood the procedure that is described, it should be the same idea behind classical multiple regression). The most difficult thing about deconfounding is trying to account for the effects of unobserved variables (e.g., omitted variable bias). I am not sure if machine learning methods are better or worse than classical methods to address these issues.
I’d argue accounting for the effect of unobserved variables is essentially impossible without making ridiculous assumptions about the external world (see psychology and degrees of freedom). I agree that ML models don’t add anything new here.
Reader: feel free to read the article, but you won’t find anything valuable. The parts that are radical aren’t accurate, and the parts that are accurate aren’t radical.
See the end of the article, 50% of what I’m saying is a rehash of ideas that existed since the 70s, I even gave a link to a 73 paper presented to the royal society outlining basically the same CV method I propose, but were never implemented widely enough or (to my knowledge) explained in modern terminology for a layman to understand.
I’m glad most things are obvious and not radical, that’s my whole point, these things are obvious and are method that should have been used everywhere since forever.
I would love to hear what parts aren’t accurate though, as it stands this just seems to be a drive-by sneer.
Replacing statistics by clueless button pushing, aka machine learning, would be a disaster.
I understand that statistics is unpopular: it requires people to think and carefully collect lots of data. But it can be used to provide insight about how things work.
Machine learning is a black box. Chuck in lots of data, push some buttons and sprout clueless techno-babble.
I understand that statistics is unpopular: it requires people to think and carefully collect lots of data. But it can be used to provide insight about how things work.
That’s not why statistics are different from ML methods. The term you’re looking for is “explainability”, the ability for an inferred model to actually explain the situation being observed to the practitioner. You’re strawmanning statistics and ML while trying to make your point.
Machine learning is a black box. Chuck in lots of data, push some buttons and sprout clueless techno-babble.
Could you please actually respond to the article instead of grinding your axe? Or, just don’t respond at all. I think axe grinding just decreases the quality of commentary on the forum. Not answering and not voting is a way to deal with topics that you don’t have a substantive critique for but you still dislike.
The links you provided are non-specific whining related to… ahm, some talks about ML at UCL and software effort estimation research is wrong. I’m unsure how this is at all related to the OP.
Replacing statistics by clueless button pushing, aka machine learning, would be a disaster.
There is no hard line between the two, but IF you grant that people are clueless about stat models, which they are, we might as well switch to a paradigm that generates a more powerful model that doesn’t require people to understand the model to begin with in order to ensure valid results.
Machine learning is a black box. Chuck in lots of data, push some buttons and sprout clueless techno-babble.
So are statistical models, unless you understand them, in which case they are obviously outright wrong in a lot of cases, and thus they are worst than a black box, they are a fake model of reality (see, for example: assuming everything fits a Gaussian or making binaries/multiclass issues out of continuous problems). So if people are already taking a black-box approach, we might as well useless biased black boxes that generate more powerful models with validation techniques that don’t require strong assumptions.
On the whole, I’d love to actually understand your issues with the content, other than the title, because based on the posts you linked I don’t think our views of “proper research” are actually very different.
The chief complaint with Fortran is that it’s unreadable but also that its buggy and slow, which it is. Being less readable than ASM is just the cherry on top of the mound of ****.
The very benchmarks the author cites show that Fortran is slower than C++ and Rust in all tasks and slower than C in all but one. Conceptually Rust, C++, Fortran and C can aim to have about the same speed, C and C++ have the advantage in terms of native APIs and ABIs designed with them in mind, Rust has the advantage of safety when doing weird stuff, C++ has the advantage of library ecosystem, compile-time abstraction and abstraction in general, C has simplicity. Fortran is inferior to all 3 in any way imaginable.
It should be noted that the speed difference here is very significant, with fortran being 2-10x times slower than the leader on virtually all problems and when the results are taken in aggregate, it barely beats C# (you know, the whole .NET language made to develop web apps?): https://benchmarksgame-team.pages.debian.net/benchmarksgame/which-programs-are-fastest.html
And again, these are the benchmarks the author reference, so I assume they are better than average.
Fortran only gets written in fields that are status-driven, where skill and merit are irrelevant, it’s used to keep the “old guard” into academic positions.
If you want to know what the guys that need efficiency are doing (AWS, GCP, Cloudflare, Dropbox… etc), they are doing C++ and some C and some are considering using some Rust, if you want to know what the guys that need fast complex math are doing (Hedge funds, exchanges, brokers, applied ML services… etc) they are doing C++ and some C.
Nobody is arguing for turning Fortran code into python, they are arguing for turning it into code that works as intended, can be read by normal humans and compiles to something that is 2-10x times as fast.
The chief complaint with Fortran is that it’s unreadable but also that its buggy and slow, which it is. Being less readable than ASM is just the cherry on top of the mound of ****.
The very benchmarks the author cites show that Fortran is slower than C++ and Rust in all tasks and slower than C in all but one. Conceptually Rust, C++, Fortran and C can aim to have about the same speed, C and C++ have the advantage in terms of native APIs and ABIs designed with them in mind, Rust has the advantage of safety when doing weird stuff, C++ has the advantage of library ecosystem, compile-time abstraction and abstraction in general, C has simplicity. Fortran is inferior to all 3 in any way imaginable.
It should be noted that the speed difference here is very significant, with fortran being 2-10x times slower than the leader on virtually all problems and when the results are taken in aggregate, it barely beats C# (you know, the whole .NET language made to develop web apps?): https://benchmarksgame-team.pages.debian.net/benchmarksgame/which-programs-are-fastest.html
You know that LAPACK (itself based on BLAS which is also written in Fortran) is written in Fortran 90 right? Almost all scientific code running on every platform used by every academy, government, or corporation is using LAPACK through their programming language/runtime of choice. If you want to push scientific computing out of the Fortran orbit, there’s a lot of work to be done.
LAPACK is not very widely used for compute intensive problems, CUDA doesn’t support it, it support CUBLAS (which si written in C/ASM), so that’s 90% of high speed computing ruled out.
Then you have CPU based stuff, for which LAPACK will be used, but not the original implementation, but rather e.g. intel’s MK, or whatever the guys with supercomputers are using. LAPACK is more of a standard than a library, and people have their own implementations.
The fortran one is indeed used by… ahm? Who exactly? A few procedures on Android phones that lack GPUs?
I’m not talking about SOTA. I’m talking about, well, every other application of linear algebra. If you think everything but Android has moved to using CUDA and MKL, I think you need to take a broader look at academia and industry. ARM cores don’t even have MKL available.
I’m not sure if you’re intentionally dense, but let me try to take a stab at this again:
For applications where speed matters and where complex linear algebra is needed (e.g. what the author is talking about), NOBODY is using the default LAPACK implementation.
For applications where you need 2-3 LAPACK function calls every hour, LAPACK is a reasonable choice, since it’s lightweight and compatible on a lot of platforms and hard enough to write that nobody bothered to try and rewrite a free and open-source cross-hardware version.
However, 99% of usage is not 99% of devices, so the fact that LAPACK is running on most devices doesn’t matter, since this device run <1% of all LAPACK-interface based computations.
This goes back to my point of Fortran is slow and buggy, so people that have skin in the game don’t use it for mathematical applications. It might be reasonable for a simple decision tree used in bejewelled but not, e.g., for meteorological models (at least not if the meteorologists were concerned about speed or accuracy, but see the problem of responsibility/skin-in-the-game).
You seem to be attacking a strawman of my position here. I’m not arguing that libraries written in fortran don’t work anymore, just that they are very bad, very bad in that they are slower than alternatives written in Rust, C++ and C, and more on part with fast GCed language (e.g. C#, Java, Switf)
I’m not sure if you’re intentionally dense, but let me try to take a stab at this again:
Not everyone on the internet is trying to fight you, please relax. Assume good faith.
You seem to be attacking a strawman of my position here. I’m not arguing that libraries written in fortran don’t work anymore, just that they are very bad, very bad in that they are slower than alternatives written in Rust, C++ and C, and more on part with fast GCed language (e.g. C#, Java, Switf)
I was responding in particular to your claim that we should just rewrite these routines out of Fortran. When we need to use specific CPU or GPU features, we reach for MKL or CUBLAS. For applications that need the speed or CPU/GPU optimizations, this is fine. For all other applications, Fortran is sufficient. I’m only saying that rewriting Fortran into other languages is a lot of work for dubious gain. Folks that need the extra speed can get it. It sounds like we largely agree on this.
I am a bit dubious about your claim that Fortran is chased by academic status-seekers, because it makes the assumption that industry is smart and efficient while academia is bureaucratic and not-efficient and uses results from the Benchmarks game to highlight it. I realize TFA linked to these benchmarks, but I took a cursory glance at them, and some of them don’t even parallelize across CPUs and some don’t even compile! Given that Fortran is a niche language, an online benchmark shootout is probably not the best place. That said, I don’t have a justified idea of Fortran’s numbers either so I can’t do anything more than say I’m dubious.
Based in part on the comment orib made, and looking at some of the code myself, I actually agree with you that those benchmarks might not be a fair comparison.
I guess the thing I disagree on still is that there’s little purpose for most scientific applications if people can’t easily read and understand their code. Especially in domains where practical applications are 30-years away, I feel like it would make sense for 3rd parties to be able to validate the logic as easily as possible.
Much like I would protest to someone using mathematical notation from the late 19th century, even though in principal it would be sufficient for a vast swath of all equations being represented in physics.
The chief complaint with Fortran is that it’s unreadable but also that its buggy and slow, which it is.
Prove it, how is it buggy and slow for its domain? Give 3 examples for each of buggy and slow, should be easy for you with how determined your claim is. Showing benchmarkgames isn’t showing real world code/simulations in any language. Matrix multiplication is what you want to benchmark, why are you using benchmark “games”? Those “games” are hardly readable after people golf them to insanity. If your argument is that I should trust anything from those “games”, well I’ve got ice cubes in antarctica to sell you.
And demonstrate how the restrict keyword in c, which is relatively recent mind you can overcome what fortran can guarantee inherently. And how that affects aliasing and ultimately generated code and how your statement of fortran being inferior to all of c/c++/rust. Have you ever talked to a scientist writing fortran for simualtion? Ever ask them why? Methinks the answer to those last two questions is no. Because I have, and the people writing fortran for simulations are insanely smart, and bear in mind the goal isn’t 100% “it benchmarks faster than language xyz”, its “I run this simulation for $N months/weeks on end, it needs to be fast enough, and I’ll profile it to see what could be improved”. And for each supercomputer they rebuild and retune every library and object for the new processor as needed. I gotta be honest the amount of ignorance of the general computer programming population about how traditional HPC works is disheartening.
And how are simulations that are being run by scientists using the literal fortran COMPLEX type by definition not “guys that need fast complex math”? Fortran specifically handles complex math better than c. My mind boggles at some of the statements here.
Less supposition and hearsay and more demonstrable proof please, this comment is atrocious for basically being without any actual depth or substance to its claims.
I provided examples of fortran being bad within the terms the author set for himself. Have you heard of arbitrary demands for rigor? If every time an opinion you hold is challenge you start clamouring for more evidence instead of considering whether or not you have any counter-evidence, well, you see the problem.
Are academics with significant clout even writing code these days? I assume they should have a small team of grad students and postdocs to do that in their lab for them…. is is the fact that everything is written in Fortran the cause of them being forced to maintain old code?
Fortran only gets written in fields that are status-driven
Our industry isn’t immune to status-driven endeavors. Think of all the “bad” technology that gets adopted because FAANG uses it or the proliferation of flavor-of-the-month frameworks in the web space.
Even if his assertion were true, assuming that all ‘complex math’ is the same and a language that is good for one is good for another is a pretty good indication that you should disregard the post. For example, a lot of financial regulation describes permitted errors in terms of decimal digits and so to comply you must perform the calculations in BCD. If you’re doing scientific computing in BCD, you have serious problems. Conversely, a lot of scientific computing needs the full range of IEEE floating point rounding modes and so on, yet this is rarely if ever needed for financial analysis.
With C99, C grew the full support for floating point weirdness that you need to be able to port Fortan code to C, but so few people care about this in C that most C compilers are only now starting to grow acceptable support (GCC had it for a while, LLVM is gaining it now, in both cases the majority of the work is driven by the needs of the Fortran front end and then just exposed in C). Fortran has much stricter restrictions on aliasing, which make things like autovectorisation far easier than for C/C++. C/C++ developers now use modern OpenMP vectorisation pragmas to tell the compiler what assumptions it is allowed to make (and is unable to check, so miscompiles may happen if you get it wrong), whereas a half-decent Fortan compiler can get the same information directly from the source.
I admit I’m a bit of a dilettante here, in that I only used ORMs for like 2 large projects, the scope of my work didn’t ever intersect their use-case.
But might it not be worth giving a few more details here (other than slapping “active record”, that is) as to what differentiates this from other python ORMs?
Since the space is so crowded and 99% of people using them are probably familiar with at least one of the biggest 2 or 3 libraries, a comparison would help a lot in explaining why one should actually want to use this.
I agree, it needs a comparision with their two biggest competitors: the Django ORM and SQLAlchemy.
Django also uses the Active Record pattern, while SQLAlchemy follows the Data Mapper… but there are so many other details that go beyond that.
Given this is mostly the same text as Machine learning could be fundamentally unexplainable, I think you should have addressed some of the critique raised in the comments to that piece.
I thought I did, but I didn’t realize I had posted that one here, this was more of a re-write to capture the core ideas from that. I will go ahead and delete this one + check if there’s anything there I hadn’t answered.
Why are hardware advances linked to RISC-V? I understand why it is nice to have an open platform, but I am not sure how this helps Machine Learning. Is it because custom new AI hardware will be possible to manufacture without license? NVIDIA is already investing so much in ML, why are they not already making what people want?
What does this mean? P’s are presidents? What is C?
C as in “chief” (CEO, CTO, CFO, etc.)
My understanding is that NVIDIA is already using RISC-V in a limited capacity: https://riscv.org/wp-content/uploads/2017/05/Tue1345pm-NVIDIA-Sijstermans.pdf
I guess I’m pointing to it more as an example of open-source hardware potentially leading to medium-sized shops being able to have their own custom-made ml chips that are more than an FPGA but less than investing a few hundred billion into being the next intel/amd/nvidia/ibm.
The link itself is more like “food for thought, this might be one such thing”, but maybe a bit misleading.
As for that quip with C, see what /u/FRIGN is saying
I bought an open EIT device to hopefully take images of my insides, turns out it’s harder than I thought, will keep working on getting meaningful data out of it.
Maybe I am missing the point of the post, but I don’t agree with some of the points you raise.
That might be true of individual brains, but you can put together a lot of people and get to the same result. See how mathematical computations were carried out by humans computers before the introduction of electronic computers.
Similarly, I don’t agree with the overall framing of your critique. You can say something like “Human bodies are shit at certain tasks, things like digging a 100ft long 10ft deep trench. Or heck, even walking more than a few miles a day or even run a few hundred feet”. We have excavators and cars for a reason. Similarly, we have computers to do “boring” computation work for us.
Well, isn’t there the saying that suggests scientific progress happens one funeral at at time? It feels to me that the problem is that researchers tend to ask questions that they can reasonably answer. The questions that they ask are informed by their methods and theoretical preparation. So, if a researcher doesn’t have a good understanding of machine learning models and what they could do, they will probably not think of a research line that will leverage these new methods to move the field forward. Instead, they will probably just apply them toy to an old problem (which will probably be solved by an old boring method just as well).
I wonder if we are at the intersection of “old” and “new”. While we are at the intersection, we are still applying the “new” models to the “old” problems. Our problem space hasn’t opened up yet to the possibilities that the “new” methods opened (maybe because we haven’t figured them out yet). That makes me wonder whether we need to push ourselves to imagine and tackle new, unexplored problem spaces that machine learning (or any new technology) has opened for us instead of trying to solve old problems with the new methods.
A 2-layer neural network can learn the tricks of 200 years of research into signal processing from a bunch of domain-specific training data: https://www.youtube.com/watch?v=HtFZ9uwlscE&feature=youtu.be
That being said I don’t think I have an argument against this, but my intuition based on how many applications nns have found in terms of outdoing equations in hard sciences like physics and chemistry is that we aren’t super good at this, even if it’s 100k of us working on it together.
In hindsight, yeah, I think the framing is a bit off.
I have a theory that popularity of vision and language is due to data. There are endless amount of unlabeled data, and labeled data can be crowdsourced cheaply.
Looking at https://en.wikipedia.org/wiki/Epigenetic_clock, I read:
It is both true that 8,000 samples are tiny, and it took 4 years to collect. Majority of machine learning effort is data collection, not data modeling. Data collection is easier with vision and language, even though data modeling is higher impact elsewhere.
Well to add to this, CV and NLP are old fields. They’ve been collecting and cleaning datasets for decades now. Applying newer NN-based ML techniques was as easy as taking an academic dataset and training the nets. Other fields don’t have nearly the same history of pedagogy and so they’re probably going to take a lot longer to collect data (usefully) for.
Bio is also special…. Consider a genome. Most of your genome is the same as everyone else’s, so even though it’s a lot of data, the degrees of freedom are considerably lower. Also, even if we were to sequence every American and european, that’s only 750M data points (most of which self correlate)… Wiipedia alone kas 3.9 billion words.
This would be true for just DNA and genetics. If you include epigenetic information and start going into gene expression in different cell types/tissues, there’s probably a lot more variation, but I don’t think we’ve quantified it yet.
I agree with this, for hard problems you can’t just use mturk to label commonly available data
I would attribute the popularity of language and vision more to problems that can be well modeled on GPUs w/ neural nets, as well as having massive amounts of labelled data. ImageNet existed well before the computer vision breakthrough by Hinton’s team. It was applying GPUs for process + neural networks that did the trick.
I think that the human-level intelligence is also partially a way to enable more people to “access” APIs, especially the people with less affinity towards technology.
Just as an example, consider ordering a pizza. If you are able to order a pizza via a website I have never not seen the option to add custom text regarding the order. You simply will have people that do not have their needs met by the ordering mask or do not find the option they really want. In this case it can make sense to have a computer attempt to decipher the customers wish.
This could also be applied to ordering via phone, which is still very common in Germany. This would be another avenue where voice recognition could make sense and improve efficiency. But ultimately, those problems are intricately connected to UX and not necessarily to fancy ML models, too.
In the end these types of ML models reach more people and it’s important that they do not feel left behind, which can cause rejection and opposition to modern technology. That is a very social problem and I am not convinced that there is a technological solution to it.
I think I agree with this, it might be that the “left behind” angle is one I don’t consider enough when thinking about this, ultimately the vast majority of people, can’t use google maps, heck, even many professionals like cab drivers which would benefit immensely from it.
I guess I’m not sure how much of this can be fixed by “education” once technology becomes to useful to be ignored, but it’s the above google map example that made me hedgy on this, after all it’s a prime example of a life-transforming technology that many people just ignore.
Maybe in some cases it’s just more fun to live life without technology? I’ve heard this argument made by literate people that preferred using paper maps and compass navigation just because it was exciting. But under that scenario I’m not sure simpler UX would help, or that any help is needed per se
I have a personal theory that many people firmly believe, in a deep, maybe even unconscious way, that they are entitled to stop learning new things at some point in their lives. It’s something I’ve been noticing more and more with the pandemic cutting off so many “non-tech” ways of doing things.
I have noticed this too. It really pains me. I personally favour the hypothesis that our education systems are to blame. All humans start off with curiosity as a primary motivator, perhaps the primary motivator after survival stuff. It seems there is a strong correlation between attending a few years of school and this no longer being a strong motivator for many people. As far as causation goes there are reasons to favour this hypothesis - the degree to which curiosity is actively suppressed in most education systems, and also the degree to which education is made to be associated with bland, boring and generally unpleasant hard work, rather than discovery. Unfortunately there is not a lot of good data regarding well-educated individuals that have not been subjected to a classical western style education. There are alternative systems like Montessori but there is no clear way to quantify cognitive ability, we use western school performance for that in most instances. The Finnish school system is an interesting case study though.
This may all seem off topic, but to tie it in to the main theme: It is actually trivial to create a human level intelligence. Many people do it by accident when they are drunk. This makes creating them artificially doubly redundant. I think that parallel to improving inhuman AI as the article suggests, we should be looking into improving the human level intelligence we have easy access to, which is mainly a software problem. Most of the software that is widely available for the human brain is decades out of date and designed for applications that are no longer needed.
To some extent though this might be a good conservative instinct.
Don’t get me wrong, I like being in some “privileged” 0.01% of the world’s population that knows the technology well enough to make 6 figures writing stuff on a laptop, I’m sure most people here feel the same way.
But if I had to choose between being in the top 20% or in the bottom 20% I’d choose to bottom.
Technology very quickly becomes a tool that most people end up using to fulfil destructive patterns more than anything.
That’s not to say it has no uses, it does, but it’s way too easy to get into pitfalls if you spend too much time on computers and don’t understand how good certain programs are at manipulating your preferences.
I’ve noticed it myself with myself. After 30 years in the computer industry, I’m just sick and tired of having to relearn the hot new development methodology. Again. For the umteenth time. Can’t things just settle down for a few years? It seems as if the singularity has hit and no one can keep up.
I suppose to some people it is more fun. I write my day-to-day plans still on paper because I prefer it to electronic bookkeeping. The battery of a sheet of paper cannot run out, although my phone hardly does either these days. But there still is something that makes me prefer to write them by hand, it also helps me remember what I actually wrote down. To some extent it is also control over the medium; I know that I am responsible if I lose it or it somehow is destroyed. I guess the aspect of control and being in charge can also translate to navigation or in general other aspects of life (although I have to agree with you, navigating via online maps is much better).
Potentially education could help with furthering usage of technology. But it could also be that most people are just stubborn and want to continue with their way of life indefinitely. Depending on how big that share of people is, it is important to make new non-human intelligence backwards compatible, so to say. Then once most people stop relying on the old technology (die out?) it can be deprecated.
Trains, assembly lines, and planes are not fully automated in the way that self-driving cars are automated; they have human staff on hand who can manage irregularities and small problems. If we use an apples-to-apples comparison, where the cars are attended by humans but centrally coordinated and powered, then the Commercial Railway seems to have opened in 1840. Certainly, by the 1880s, streetcars and trams were appearing in many cities. In general, the history of trams and light rail is part of the history of trains. I think that bridging the gap between this sort of transit and commuters is a last-mile problem, not necessarily an automation problem.
That’s actually a good point, but the automation is responsible for flagging irregularities for the human operator to look at. So in the case of a car, this could essentially mean “slow down, and stop”.
Planes and most trains do usually have on-board operators but this seems to be mainly a regulation issue.
Chesterton’s Fence applies here; the regulations have a rationale. We want heavy machinery to have operators in case of emergency or exceptional situations which go beyond the capabilities of automation.
I mean, to a point, yes.
With e.g. trains there are fully automated systems that work just fine.
For a plane you could indeed argue cost of a pilot is small enough to justify it as conservative safety, same for a train that goes over unusual and varried terrain (as opposed to, say, a metro)
I wonder if there aren’t examples of the AI->Automation process which have already occured? Maybe that’s just called automation?
I’m thinking about machine-readable data formats and CPU or database dependant operations gaining acceptance over time (e.g. the standardization of VINs in the 80’s).
I don’t think that AI is special in this regard… if anything I would expect it to accelerate the process in all cases where it is present.
I can think of: companies introduced AI in their customer support, then realized they could just automate it. For a successful example see Amazon, for a failed one Revolut. At least that’s my reading of the situation but I can’t be sure.
But it’s not a very good example, I’ll try to look into this.
I agree
How does OCR work, AI or automation?
If you believe the thesis of the article, OCR is AI made necessary only by our civilization’s inability to digitize everything, while automation would do away with both papers and the need for OCR.
You might be confused because AI is often seen as one kind of automation. I believe the article tries to distinguish them by correctness (without actually using that word). OCR and other “AI” can make mistakes and require human intervention, while “automation” can be proven correct.
I think it’s self evident that a world without humans is easy to manage by shifting data to memory regions.
We all know printers are devilspawn.
Getting computers to work with us instead of us working with them, that’s the trick, and I’m not sure automation solves that problem. In fact I might be so bold as to say AI gets computers to work with us, and automation helps us work with computers.
It’s a sliding scale. https://en.m.wikipedia.org/wiki/OCR-A
I defined “AI” for the purpose of this as “using the term to mean some heuristic for learning equations from data or simulations rather than writing them out by hand”.
So I think OCR fits as AI, unless there’s a way to do OCR without training data.
Basically what axelsvensson said.
A while ago I got this problem of the fuzziness of the term “AI” stuck in my head and thought about it a lot. I think “a computer program that takes on the world on its own terms (without the situation being arranged to suit the computer program)” is somewhat accurate. I think this matches your definition of “AI” and the parenthetical fits how you contrast it against “automation”.
The real question here is, I feel, why did the writer this not do it on their own and never bothered anyone?
Like, not calling her lazy or anything, but why wasn’t there an incentive for him to just get it done on his own? Since it’s obviously possible.
Indeed, getting anything “simple” to the level of “management handled issue” will by default add a standard amount of overhead (e.g. 10 mins in a startup, 1hr in round B, few weeks in a megacorp). But that’s just the nature of consensus making in large orgs… however a lot of things can be resolved exactly by not having to get them to management, by just doing them yourself (or yourself + a group of friends that don’t mind asking for help)
First, I believe the “guy” writing at the “Rachel by the Bay” blog is a woman.
Second, they did write an initial dashboard by themselves, but then a “dashboard team” came along and insisted they’ll provide said dashboard—and then promptly decided they had more important things to do for 5 months. Here’s paragraph 3 and 4 of the post:
Ups, edited
What I was saying here was that, i.e. “WHY IS THERE a dashboard team!?” that seems like the fundamental question to tackle here. Why does the company not provide incentives to just leave the stupid python script up, or get the devs to spend the few days they did writing the dashboard without implicating a whole other department.
I think the author might also think the same thing here, I was trying to paraphrase my takeaway, not criticize.
I can answer this question as if it was the BigCorp for which I’m currently working. Please take my answer with a grain of salt, as this is most likely biased by my “grunt” point of view.
Her name is Rachel and she is not a guy. If you want to use gendered language you should do your research on not jump to conclusions. The name is even visible in the domain.
Gendering someone based on a personal name that you found in the domain name of the site where you found an article sounds like a lot of making assumptions to me. And in the glorious twenty-first century, anyone and everyone is free to be a “guy”, just like in the 90s we were all dudes.
Overview/review of the alpha fold paper that doesn’t require a lot of machine learning of molecular biology expertise to understand and can be read in 10 minutes.
I’m not sure I succeeded, it may well staddle the line poorly enough that it’s both uninteresting to someone familiar with the subject and unpenetrable to someone that isn’t.
But I almost wasted a whole day reading, writing, arguing, and editing to get this done. So I decided I have to post it somewhere to see what people think.
If you believe it’s too unrelated/abstract/bad then do let me know and I’ll delete it, I think I’ve been posting some mediocre articles here lately and I feel bad about it and want to stop, I appreciate the relative quality of this board as much as anyone. But it’s kind of hard to tell before I see people’s reactions, especially with stuff that you spend a lot of time writing.
Since you asked for feedback:
I haven’t read the alphafold paper yet, but your impressions are helpful to me personally, so thanks for writing it up. I liked the personal take and especially admitting what you don’t understand is nice to read, because reading papers is hard and not often acknowledged.
Cheers, let me get those edits in. Glad at least one person got a bit out of it.
NetBSD’s rump kernel has been able to run in a browser for well over a decade (perhaps two?) And I’ve seen Gameboy and Nintendo emulators that run in a browser.
Just how complex browsers are these days boggles the mind. They’re also nigh impossible to secure.
Can you give some more detials here? Do you mean in userspace or as part of browser’s code?
I guess I should have mentioned I was referring soley to userspace. If that’s the case… I don’t really see the reason why you might want to do that. Maybe to run it on in-browser simulated hardware?
NetBSD has pretty good documentation on rumpkernels. Effectively, it’s akin to running the kernel as a program in userspace. Rumpkernels allow one to develop and test new TCP/IP stacks and filesystem drives, for example. Pretty cool methodology of OS/systems development.
It doesn’t seem to me like it’s running in a web browser, but rather that it’s running the web browser.
Relevant section being:
Years ago, you could run the rumpkernel inside the browser, exactly similar to emulating a gameboy or NES in the browser. Similar to this: https://bellard.org/jslinux/index.html
Those demos showing the rumpkernel booting in the browser itself seem to be no longer available on the internet. That’s a shame, because rumpkernels-in-js are awesome.
Curiosity got me and I think I found a copy of one of those demos:
http://197.155.77.5/NetBSD/misc/pooka/rump.js/
This. This is why independent and different implementations matter. If you understand the web as an application platform you will realize that some browsers should not be able to dictate how it works. And there is little independence out there…
The problem boils down to something like “who will pay for independence” ?
As in, dontaions is not a reliable model, and ads are blurrying the line of ethics.
Personally I always though there’s an audience for a premium browser that includes other cost saving features (e.g. banning spammy websites, currated suggestions, built-in unblocking for papers, easy download for all mutlimedia, built in search engine that behaves based on your stated preferences rather than your behaviour-revealed preferences etc). But the problem is that most of the audience that would understand why such a browser is necessary are the peole that need it least.
One could also hope that some crypto billionare might just give a 10B endownment to a foundation to build a competitive browser under very strict decentralized supervision, but afacit that hasn’t happened yet.
TBF Mozilla continues to make it structurally impossible to donate directly to Firefox development for some reason.
I’d love to pay/donate to Mozilla on a monthly base, if it could be done in a way that made it clear I’m not in support of their stupid shit.
This is a really interesting view into the future – and I’ve become, oddly, a bit less worried about it as time goes on.
Google Docs recently announced – and we discussed that they’ll be doing canvas-based rendering of documents. Does this further lock us into the Google monopoly as they now do everything server-side? Possibly, but is it truly any worse than the webasm/minified JS monstrosities that are modern “web apps”?
It feels like the web is starting to split: on one side a large amount of data is still following the hypertext vision we’ve traditionally chased, alternative decentralized networks like Mastodon and the Tildeverse are taking root, and hell – we’ve got people writing Gopher pages! In this century! On the other side, we have video games using OpenGL in your browser.
Stuck in the middle is the last independent browser trying to decide which of the competing standards to chase. Perhaps a hard fork between “the simpler, hypertext web” and “the web as a way to deliver arbitrary, sandboxed applications” helps them decide where they want to spend their more limited resources.
I dunno. Maybe I’m just the frog that’s been boiled slowly enough that I’m not as worried about this future, but I’m not convinced it’s entirely terrible.
What do you mean, server-side? My assumption is that the only change here is that the ‘rendering’ step now just calls the canvas draw methods as opposed to generating HTML, along with accompanying changes in how they figure out what user input should do (since now every click will hit the same element). It definitely makes it harder to pull out the textual content from extensions and the like, but I don’t think this really increases lock-in.
And I think the problem is that there is no actual hard divide between “simple hypertext” and “arbitrary sandboxed applications”. It’s funny you mention Mastodon, because the default Mastodon frontend is, conceptually, much closer to the application model. Clicking profiles doesn’t navigate you to separate pages, it does an XHR and then renders that in a new ‘column’. It even literally embeds an OCR engine that’s compiled to wasm for generating image captions!
What I’m unclear on is whether or not there’s an actual split to be talked about.
As in, you could still support html+css in a browser as VMs setup, at most they’d have to include something like an HTML+CSS “special compiler” of sorts that adds some pre-defined rendering logic.
And I doubt any browsers will want to loss e.g. wikipedia or the blogosphere. More interestingly though, if javascript becomes something closer to flash (unsafe, redundant, nobody uses this anymore kinda thing) it might prompt minimal-javascript websites to switch to being fully static.
Lobster might be a good example of this where (while I can’t speak for the devs) I could see it being easy to port back into raw HTML, and given enough js related issues some people might just say fuck it and go back to HTML instead of chasing compatibility. The fact that HTML has added many quality of life elements and tags also helps.
I think a big reason Flash had the (justified) reputation it did is that it fundamentally had less of a sandbox mechanism than JS, and the developers just were not as security-focused as browser developers were/are. And as JS grew more and more powerful, there was less and less reason to use Flash. I don’t see this happening to JS unless something new comes to replace it, and I don’t think this will happen; in particular, I think the ability to just mess around in the web console is incredibly useful, and you just don’t get that with wasm.
Wait… I was following you until the last bit.
Why don’t you get that with WASM? As in, I get that you don’t get that today but in principle, nobody is stopping an addon that compiles wasm to js and runs it in the console. And/or direct vendor support if it gets popular enough… it’s a bit annoying since you’d have to specify to the browser a path to the compiler of choice (since you’re not going to ship browsers with all of them). But that’s a one-time setup.
Or do you mean that people won’t be able to mess with the code of the websites they are using?
That also seems untrue, given that EOD WASM just modifies the DOM. You’ll still be able to use the console to add functionality. And most websites, even “puritan” ones ala lobsters come with compiled javascript that’s impossible to mod.
There’s a dying breed of dev (e.g. myself) that enjoy shipping readable js to the user, but that’s so niche as not to matter.
This is an interesting article. I have a few “big picture” issues with it that I discuss below.
The argument in the introduction seems somewhat all over the place. The points that the author raises are interesting, but I am not sure if I agree with the conclusion. For example, the author claims that
as a critique of “classical statistical” methods. While I agree with the author that scientific and statistical literacy is a problem, I am at a loss of the eventual conclusion that statistical learning models, that are even less interpretable than classical models, are a solution for this problem.
I am also not sure if I follow how these three conclusions that the author highlights for the post follow from the article:
As far as 1, a 14 year old in a regular algebra class will be able to carry out a simple linear regression with 1 variable. That’s because the methods are rather simple and it is easy enough to understand the difference between the line of best fit and the observed points. I am not completely sure if the same is true for even the simplest statistical learning models.
Maybe I am not understanding what “inferential power” means here, but my understanding of machine learning is that those models excel at predictions, as in out of sample predictions, rather than inferences. I am also fuzzy about the kind of insights that the author is going to discuss.
Finally, I am not sure how data manipulation (as in data cleaning, for example) is connected to data analysis and how statistical learning “fixes” them. All the ideas that are discussed further in the post seem to be on the data analysis side, so I don’t see the connection between this last insight to the rest of the post.
As a final comment, most research is done to answer specific research questions. Usually a researcher asks a question and picks the best methods to answer it. The author seems to advocate moving from inferential questions (e.g., that is the difference between a group that received X and a group that didn’t) towards predictive questions (e.g., what will happen if this group receive X, maybe?). I wooden if a better motivation for the post needs to be an epistemological first rather than a methodological/technical one.
A few more fine-grained questions:
I agree that ML models are less interpretable. But what I think lends itself to explainability is performing CV and reporting accuracy as a function of that (easy to get, relatively speaking) and making the error-function explicit, i.e. informing people of what you’re trying to optimize and comparing your model to a “base” assumption or previous models on that metric.
You don’t need to understand how the model operates to get the above, but in statistics, the underlying model (assumptions about it’s error distribution) is actually relevant to interpreting e.g. the p-values.
I focus on how data analysis can be done under this paradigm because a lot of the strong argument pro using outdated statistical model seems to focus on their power of “explaining” the data while making predictions at the same time. So I’m trying to showcase why that also works just as well if not better with more complex models.
I agree it’s in part a different point.
In part the post came as part of a long series of post about why a predictive based epistemology is the only way to actually make sense of the world, and why e.g. a “equation first” epistemology is just a subset of that which happens to work decently in some domains (e.g. medieval physics).
But I guess the article specifically was meant mostly towards people that already agree a predictive epistemology makes more sense to pursue knowledge about the world, to actually lay out how it could be done better than now.
In part my fault for not tackling this assumption, but at that point I’d have had a literal book instead of a way-too-long article.
You aren’t missing anything. Classical methods would work just fine under this paradigm and I assume for many problems a linear regression would still end up being a sufficiently complex model to capture all relevant connections.
I’d argue accounting for the effect of unobserved variables is essentially impossible without making ridiculous assumptions about the external world (see psychology and degrees of freedom). I agree that ML models don’t add anything new here.
Reader: feel free to read the article, but you won’t find anything valuable. The parts that are radical aren’t accurate, and the parts that are accurate aren’t radical.
See the end of the article, 50% of what I’m saying is a rehash of ideas that existed since the 70s, I even gave a link to a 73 paper presented to the royal society outlining basically the same CV method I propose, but were never implemented widely enough or (to my knowledge) explained in modern terminology for a layman to understand.
I’m glad most things are obvious and not radical, that’s my whole point, these things are obvious and are method that should have been used everywhere since forever.
I would love to hear what parts aren’t accurate though, as it stands this just seems to be a drive-by sneer.
Replacing statistics by clueless button pushing, aka machine learning, would be a disaster.
I understand that statistics is unpopular: it requires people to think and carefully collect lots of data. But it can be used to provide insight about how things work.
Machine learning is a black box. Chuck in lots of data, push some buttons and sprout clueless techno-babble.
Machine learning in SE research is a train wreck
Software effort estimation is mostly fake research
That’s not why statistics are different from ML methods. The term you’re looking for is “explainability”, the ability for an inferred model to actually explain the situation being observed to the practitioner. You’re strawmanning statistics and ML while trying to make your point.
Could you please actually respond to the article instead of grinding your axe? Or, just don’t respond at all. I think axe grinding just decreases the quality of commentary on the forum. Not answering and not voting is a way to deal with topics that you don’t have a substantive critique for but you still dislike.
The links you provided are non-specific whining related to… ahm, some talks about ML at UCL and software effort estimation research is wrong. I’m unsure how this is at all related to the OP.
So are statistical models, unless you understand them, in which case they are obviously outright wrong in a lot of cases, and thus they are worst than a black box, they are a fake model of reality (see, for example: assuming everything fits a Gaussian or making binaries/multiclass issues out of continuous problems). So if people are already taking a black-box approach, we might as well useless biased black boxes that generate more powerful models with validation techniques that don’t require strong assumptions.
On the whole, I’d love to actually understand your issues with the content, other than the title, because based on the posts you linked I don’t think our views of “proper research” are actually very different.
Using statistics does not guarantee that the model is correct. No technique guarantees correctness.
The appropriate adage is: “All models are wrong, but some are useful”.
Exactly, which is why any modelling effort requires:
The statistical approaches a lot of fields currently use don’t do either very well, hence, the article.
This article is attacking a strawman using lies.
The chief complaint with Fortran is that it’s unreadable but also that its buggy and slow, which it is. Being less readable than ASM is just the cherry on top of the mound of ****.
The very benchmarks the author cites show that Fortran is slower than C++ and Rust in all tasks and slower than C in all but one. Conceptually Rust, C++, Fortran and C can aim to have about the same speed, C and C++ have the advantage in terms of native APIs and ABIs designed with them in mind, Rust has the advantage of safety when doing weird stuff, C++ has the advantage of library ecosystem, compile-time abstraction and abstraction in general, C has simplicity. Fortran is inferior to all 3 in any way imaginable.
It should be noted that the speed difference here is very significant, with fortran being 2-10x times slower than the leader on virtually all problems and when the results are taken in aggregate, it barely beats C# (you know, the whole .NET language made to develop web apps?): https://benchmarksgame-team.pages.debian.net/benchmarksgame/which-programs-are-fastest.html
And again, these are the benchmarks the author reference, so I assume they are better than average.
Fortran only gets written in fields that are status-driven, where skill and merit are irrelevant, it’s used to keep the “old guard” into academic positions.
If you want to know what the guys that need efficiency are doing (AWS, GCP, Cloudflare, Dropbox… etc), they are doing C++ and some C and some are considering using some Rust, if you want to know what the guys that need fast complex math are doing (Hedge funds, exchanges, brokers, applied ML services… etc) they are doing C++ and some C.
Nobody is arguing for turning Fortran code into python, they are arguing for turning it into code that works as intended, can be read by normal humans and compiles to something that is 2-10x times as fast.
You know that LAPACK (itself based on BLAS which is also written in Fortran) is written in Fortran 90 right? Almost all scientific code running on every platform used by every academy, government, or corporation is using LAPACK through their programming language/runtime of choice. If you want to push scientific computing out of the Fortran orbit, there’s a lot of work to be done.
LAPACK is not very widely used for compute intensive problems, CUDA doesn’t support it, it support CUBLAS (which si written in C/ASM), so that’s 90% of high speed computing ruled out.
Then you have CPU based stuff, for which LAPACK will be used, but not the original implementation, but rather e.g. intel’s MK, or whatever the guys with supercomputers are using. LAPACK is more of a standard than a library, and people have their own implementations.
The fortran one is indeed used by… ahm? Who exactly? A few procedures on Android phones that lack GPUs?
I’m not talking about SOTA. I’m talking about, well, every other application of linear algebra. If you think everything but Android has moved to using CUDA and MKL, I think you need to take a broader look at academia and industry. ARM cores don’t even have MKL available.
I’m not sure if you’re intentionally dense, but let me try to take a stab at this again:
This goes back to my point of Fortran is slow and buggy, so people that have skin in the game don’t use it for mathematical applications. It might be reasonable for a simple decision tree used in bejewelled but not, e.g., for meteorological models (at least not if the meteorologists were concerned about speed or accuracy, but see the problem of responsibility/skin-in-the-game).
You seem to be attacking a strawman of my position here. I’m not arguing that libraries written in fortran don’t work anymore, just that they are very bad, very bad in that they are slower than alternatives written in Rust, C++ and C, and more on part with fast GCed language (e.g. C#, Java, Switf)
Not everyone on the internet is trying to fight you, please relax. Assume good faith.
I was responding in particular to your claim that we should just rewrite these routines out of Fortran. When we need to use specific CPU or GPU features, we reach for MKL or CUBLAS. For applications that need the speed or CPU/GPU optimizations, this is fine. For all other applications, Fortran is sufficient. I’m only saying that rewriting Fortran into other languages is a lot of work for dubious gain. Folks that need the extra speed can get it. It sounds like we largely agree on this.
I am a bit dubious about your claim that Fortran is chased by academic status-seekers, because it makes the assumption that industry is smart and efficient while academia is bureaucratic and not-efficient and uses results from the Benchmarks game to highlight it. I realize TFA linked to these benchmarks, but I took a cursory glance at them, and some of them don’t even parallelize across CPUs and some don’t even compile! Given that Fortran is a niche language, an online benchmark shootout is probably not the best place. That said, I don’t have a justified idea of Fortran’s numbers either so I can’t do anything more than say I’m dubious.
Based in part on the comment orib made, and looking at some of the code myself, I actually agree with you that those benchmarks might not be a fair comparison.
I guess the thing I disagree on still is that there’s little purpose for most scientific applications if people can’t easily read and understand their code. Especially in domains where practical applications are 30-years away, I feel like it would make sense for 3rd parties to be able to validate the logic as easily as possible.
Much like I would protest to someone using mathematical notation from the late 19th century, even though in principal it would be sufficient for a vast swath of all equations being represented in physics.
Prove it, how is it buggy and slow for its domain? Give 3 examples for each of buggy and slow, should be easy for you with how determined your claim is. Showing benchmarkgames isn’t showing real world code/simulations in any language. Matrix multiplication is what you want to benchmark, why are you using benchmark “games”? Those “games” are hardly readable after people golf them to insanity. If your argument is that I should trust anything from those “games”, well I’ve got ice cubes in antarctica to sell you.
And demonstrate how the restrict keyword in c, which is relatively recent mind you can overcome what fortran can guarantee inherently. And how that affects aliasing and ultimately generated code and how your statement of fortran being inferior to all of c/c++/rust. Have you ever talked to a scientist writing fortran for simualtion? Ever ask them why? Methinks the answer to those last two questions is no. Because I have, and the people writing fortran for simulations are insanely smart, and bear in mind the goal isn’t 100% “it benchmarks faster than language xyz”, its “I run this simulation for $N months/weeks on end, it needs to be fast enough, and I’ll profile it to see what could be improved”. And for each supercomputer they rebuild and retune every library and object for the new processor as needed. I gotta be honest the amount of ignorance of the general computer programming population about how traditional HPC works is disheartening.
And how are simulations that are being run by scientists using the literal fortran COMPLEX type by definition not “guys that need fast complex math”? Fortran specifically handles complex math better than c. My mind boggles at some of the statements here.
Less supposition and hearsay and more demonstrable proof please, this comment is atrocious for basically being without any actual depth or substance to its claims.
Think its time for me to call lobsters quits.
FWIW, every other commenter, except for the comment you replied to, agreed with your position on Fortran.
I provided examples of fortran being bad within the terms the author set for himself. Have you heard of arbitrary demands for rigor? If every time an opinion you hold is challenge you start clamouring for more evidence instead of considering whether or not you have any counter-evidence, well, you see the problem.
Are academics with significant clout even writing code these days? I assume they should have a small team of grad students and postdocs to do that in their lab for them…. is is the fact that everything is written in Fortran the cause of them being forced to maintain old code?
Our industry isn’t immune to status-driven endeavors. Think of all the “bad” technology that gets adopted because FAANG uses it or the proliferation of flavor-of-the-month frameworks in the web space.
https://blog.janestreet.com/why-ocaml/
Together with the two parallel comments I’m going to conclude you don’t know what you’re talking about.
Even if his assertion were true, assuming that all ‘complex math’ is the same and a language that is good for one is good for another is a pretty good indication that you should disregard the post. For example, a lot of financial regulation describes permitted errors in terms of decimal digits and so to comply you must perform the calculations in BCD. If you’re doing scientific computing in BCD, you have serious problems. Conversely, a lot of scientific computing needs the full range of IEEE floating point rounding modes and so on, yet this is rarely if ever needed for financial analysis.
With C99, C grew the full support for floating point weirdness that you need to be able to port Fortan code to C, but so few people care about this in C that most C compilers are only now starting to grow acceptable support (GCC had it for a while, LLVM is gaining it now, in both cases the majority of the work is driven by the needs of the Fortran front end and then just exposed in C). Fortran has much stricter restrictions on aliasing, which make things like autovectorisation far easier than for C/C++. C/C++ developers now use modern OpenMP vectorisation pragmas to tell the compiler what assumptions it is allowed to make (and is unable to check, so miscompiles may happen if you get it wrong), whereas a half-decent Fortan compiler can get the same information directly from the source.
I admit I’m a bit of a dilettante here, in that I only used ORMs for like 2 large projects, the scope of my work didn’t ever intersect their use-case.
But might it not be worth giving a few more details here (other than slapping “active record”, that is) as to what differentiates this from other python ORMs?
Since the space is so crowded and 99% of people using them are probably familiar with at least one of the biggest 2 or 3 libraries, a comparison would help a lot in explaining why one should actually want to use this.
I agree, it needs a comparision with their two biggest competitors: the Django ORM and SQLAlchemy. Django also uses the Active Record pattern, while SQLAlchemy follows the Data Mapper… but there are so many other details that go beyond that.