A mathematician friend once suggested that starting with limits is bad, but so is starting with derivatives. He proposed we start with integrals, as they’re more conceptually intuitive for beginners. Then you introduce limits and derivatives as tools for doing integrals.
Beyond HS algebra & trig, most of the math I know I’ve learned on my own, from books. Bartlett’s Calculus from the Ground Up is the only calculus book that ever made sense to me. It basically starts with differentials  (after a brief discussion of derivatives).
 Jonathan Bartlett, “Simplifying and Refactoring Introductory Calculus” (https://arxiv.org/abs/1811.03459)
I’ve seen that perspective before, that Apostol got it right in 1967 and everyone else got it wrong.
I haven’t read the article yet, but I think that limits are not necessarily hard to grasp. I think most people would grasp it pretty quickly when then are presented with a time-distance graph and the task to find the speed at some point.
The plus side of starting with limits is that it’s easier to define everything right from the start (which might or might not be important – I personally think this is not needed in most cases that are not a formal presentation that is as short as possible).
Edit: Read the article… After reading the article I’m still not really sure how you would define derivatives, prove the product rule, etc. without limits. I would be really interested to see how it can be teached without introducing limits first.
I just could not learn calculus with limits, I had to go back to the old infinitesimals books to wrap my head around calculus.
I ended up with a book from the early 1900s where everything makes much more sense (to me).
You may enjoy nonstandard analysis! I am also very fond of infinitesimals. I’ve looked at the Keisler books and also a while back I was enjoying going through ‘Lectures on the hyperreals’ by Goldblatt.
The Intermediate Value Theorem is valuable because of what it implies about the continuous nature of the underlying numbers. Consider this classic riddle: A basketball player has a shot record of exactly 0.75. They practice and work until their record has improved to over 0.95. Must their record, at some point, be exactly 0.9? This question about intermediate values depends on the different shape of the Intermediate Value Theorem for rationals compared to reals.
We cannot really introduce derivatives without first justifying limits. Perhaps instead we should start with infinitesimal values and build up a geometric understanding first, as in 3blue1brown’s series on calculus. This is extremely valuable to programmers in particular because of the connection to the dual numbers, a straightforward and cheap way to implement forward-mode automatic differentiation.