TeX itself is not really complex. The implementation is fine. The hard part is the ecosystem.
Any contender is going to face a daunting task of explaining why adopters should throw away 50 years of collective work of thousands of people that went into TeX packages in CTAN. Math notation is at best 1% of the reasons why people use TeX. Packages are.
The 66-page "Breaking Paragraphs into Lines" is a research paper, and much of it is is the "complexity" of the problem itself, rather than TeX's elegant solution in particular (e.g. pages 48–59 are about history). In a sense, the generality shown in the paper (the various different things that TeX's algorithm can accomplish) shows how TeX reduces the real-world complexity of typesetting with a simple approach that covers all of them.
The book "Digital Typography" is, somewhat misleadingly, not a book that was written about digital typography, but simply a collection (https://cs.stanford.edu/~knuth/selected.html) of several papers that Knuth wrote about TeX and Metafont and some related topics, over several decades.
In any case, I agree with the comment you're replying to: if you use TeX itself without dragging behind you the entire ecosystem and all the packages (in particular, use plain TeX rather than LaTeX) you'll see it's not really complex, and the implementation is fine. Especially today, with things like LuaTeX scripting (and, say, opTeX) it is feasible to bypass LaTeX and packages and do things yourself — that is, unless external circumstances require you to use LaTeX and packages etc, which is still often the case (e.g. submitting papers to journals).
I think the argument comes down to complexity. Getting Markdown --> Pandoc featureset up to a reasonable fidelity is the next stage IMO, especially if Markdown can be made to support the most common downloaded TeX extension packages.
Not going to happen. If you look at the CommonMark discussions, you will find people adamant that X features not be included for varying reasons.
I think the best future option is going to be djot[0]. It is being created by the author of pandoc, who might possibly be the most qualified person in the world to appreciate all if the nuances of marking up text and parsing it.
Clean slate design. I think pandoc markdown is a superset of markdown, which must parse the ambiguous parts of the spec. Djot has a goal to eliminate the corner cases and be able to represent all things without requiring the html escape hatch (eg bold text within a word).
Thanks, that makes sense. Although I’d like to point out that one of the listed problems with commonmark, and the problem you mention, are not problems with Pandoc markdown:
You make markdown support TeX by running it though a tex compiler. The other way around is never in a million years going to happen, because markdown is all about not adding additional markup like \commands, or extensions without which a markdown renderer wouldn't be able to generate the content it's supposed to.
(which is why markdown still doesn't officially have any maths support, only certain non-standard flavours do)
Any contender is going to face a daunting task of explaining why adopters should throw away 50 years of collective work of thousands of people that went into TeX packages in CTAN. Math notation is at best 1% of the reasons why people use TeX. Packages are.