I've been going through the book "But How Do It Know" (not a typo) and it's a great introduction on how to build a CPU from scratch starting from basic transistor working up higher and higher levels of abstraction using a bottom up approach. You don't need a lot of math to understand it at all and is a fairly easy read.