In my experience, with anything of any significant complexity, this sort of approach ends up being more work than straight LaTeX. On top of that, the conversion tools change too much inbetween versions to guarantee longevity.
Having worked on a 1,200 page book I am going to concur. I started out using markdown+Pandoc and it became a time loss pretty quickly.
I would love to be able to use something similar (org-mode instead of markdown) but so far I’ve never gotten satisfactory results. To me the ideal case would be that I am able to easily export the source document into multiple formats so that I can distribute it as a pdf and html document.
Where this fails is most often the interoperability of external tools and to some extent my own laziness. The process of setting up a framework to cite references exist in org-mode (and I believe in pandoc through cite-proc) it is cumbersome to set up. Another issue is the visualization. Since the fonts are most often different I would need to generate multiple versions of the same plot, which becomes even more work if I want to use TikZ in LaTeX and for example matplotlib for web publishing.
While the tufte layout is great, what should I do with overlapping margin notes? They need to be manually corrected in LaTeX, at least as far as I know.
Another thing: How can I generate a glossary? For LaTeX the glossaries package exists but you cannot use that straightforward in markdown.
While this probably reads very negative, I would love it, if something like this existed! I just do not see that happening so far. And yet another point would be getting other people on board for collaboration.
My experience with Pandoc was not excellent to convert Markdown to LaTeX. Pandoc is a bit complicated to extend as well.
What I did was use a Markdown parser(1) producing an AST(2), plugging into the parsing to extend Markdown syntax to support for instance the glossary use case(3), then write a LaTeX stringifier(4) for this AST. (In this Markdown AST -> LaTeX stringifier I only support standard Markdown syntax, this abbr plugin I wrote gets stringified through yet another plugins package(5) I wrote. It’s straightforward: 6.) Then I simply put the latex doc into some sort of latex template using a custom class, in the latex template you’d have your tableofcontents, glossaries, etc.
I had a bunch of lecture slides written up in org that I was dumping to LaTeX, which was always a bit fragile, and then a change in org-mode broke everything. In the end I wasted more time fixing everything than if I’d just written LaTeX to start with.
I think the best thing is to leverage smart editing (e.g. AUCTeX) as much as possible, and not deal with trying to convert from something else into LaTeX.