Probability theory expositions, especially for [software] engineers, would be better served if they were well typed. What is the type of a random variable, E[Y|X], E[E[Y|X]]? Hint, a random variable is not a scalar, but rather a function, the probability distribution.
Hmm, a random variable (in the sense of measure theory, as in OP) is indeed a function - but it's not a probability distribution.
An R.V. is a measurable function from the sample space into the reals. A probability distribution is a function assigning probabilities to measurable sets, formally, a function from the sigma-algebra into [0,1].
So in particular, a R.V. (like a gaussian) can take on negative values. A probability distribution cannot.
Also, the domain of the R.V. is the sample space. But the domain of the probability distribution is the sigma-algebra over that sample space.
A distribution is a real valued function of a real variable. The domain of the function is the whole real line.
Note: Below, borrowing from D. Knuth's TeX, we use the underscore character '_' to denote the start of a subscript.
Details: For real valued random variable X, probability measure P, and the set of real numbers R, the cumulative distribution of X is the function F_X: R --> R where, for x in R, F_X(x) = P(X <= x).
If F_X is differentiable, then the probability density distribution of X is the real valued function of a real variable f_X: R --> R where, for all x in R, f_X(x) = d/dx F_X(x) where d/dx is the calculus derivative.
For the connections with sigma algebras, that is more advanced than most engineers care about, but here are some of the details:
For real numbers a and b with a < b, there is the open interval
(a,b) = {x|a < x < b}
A topology on R is a collection of subsets regarded as open and that satisfy the axioms for a topology -- the sets in a topology are closed under finite intersections and arbitrary unions and both R and the empty set are open. The usual topology on R is the smallest (a short argument shows that this "smallest" is well defined) topology that has each open interval an element of the topology -- right, the topology regards the open intervals as open.
The usual reason to discuss a topology is to have a means of defining continuous functions, a means more general than from the usual "for each epsilon greater than zero, there exists a delta greater than zero such that ..." or in terms of limits of sequences. Indeed, there are advanced situations where we can use topologies to define continuous functions where epsilon and delta and where converging sequences don't work. If curious, look up Moore-Smith convergence, nets, and filters or just Kelley, General Topology.
Well, a sigma algebra is like a topology, that is, is a collection of subsets: A sigma algebra is closed under countable unions and relative complements. Right, we avoid uncountable unions because otherwise we will get stuck in a big mud hole. It is an early exercise that there are no countably infinite sigma algebras.
The reason for sigma algebras is to permit defining a measurable function, that is, one where we can apply the Lebesgue integration theory. The integral of calculus is due to B. Riemann and is the Riemann integral. W. Rudin, Principles of Mathematical Analysis shows that for a continuous real valued function with domain a compact set (closed and bounded, where closed is the complement of an open set) has a Riemann integral. Well in this case, the Lebesgue integral gives the same numerical answer -- same thing. The advantage of the Lebesgue approach is that the function can be even bizarre and its domain can be much more general. Indeed, in probability theory, expectation is just the Lebesgue integral. In simple terms, Riemann partitioned on the X axis, and Lebesgue partitioned on the Y axis.
Well, given the usual topology on R, we can ask for the smallest sigma algebra on R that has the topology as a subset. That sigma algebra is the Borel sets of R. Uh, Lebesgue was a student of E. Borel. In Rudin will find the Heine-Borel theorem.
So, in probability theory, we have a sample space. Each point in the sample space is a trial, i.e., essentially a real world experimental trial (note: really our attitude is that in all the universe we see only one such trial -- if this seems far out, then blame the Russians, e.g., A. Kolmogorov, E. Dynkin, etc.!). Well, an event is a subset of the sample space, that is, a set of trials. So, flip a coin. Let H be the event, the set of all trials where, that the coin comes up heads.
Well, to apply Lebesgue's theory of integration, we want the set of all events to be a sigma algebra.
Then a probability measure is a measure in the sense of Lebesgue's measure theory, that is, a real value function, in the case of probability taking values in [0,1], and with domain the sigma algebra of events. So, for the event H, we can ask for the probability of H, that is, P(H), which is a number in [0,1]. For a fair coin tossed by an honest member of the FBI we have P(H) = 1/2.
Then a real valued random variable X is just a real valued function with domain the sample space and also measurable: This part about being measurable is that for each Borel set A, a subset of R, the set of all trials w so that X(w) is in A is an event, that is, an element of the sigma algebra on the set of trials. That is, the inverse image under X of the Borel sets are events, elements of the sigma algebra on the sample space.
So, with X measurable in this way, we have a near perfect shot at defining the expectation of X, E[X]. For this we have a little two step dance:
First we look at X^+ ('^' denotes a superscript) where X is >= 0. So, X^+ is the positive part of X. Similarly X^- is the negative part of X. So, X = X^+ + X^-. Uh, I'm working quickly from memory; maybe we want X^- to be -X where X < 0 and 0 otherwise. So both X^+ and X^- are >= 0 and we have X = X*+ - X^-. Either way.
Well, we can use Lebesgue's theory to integrate X^+ and X^-. Biggie stuff: The X need only be measurable, and that admits lots of really wildly bizarre functions. We've got great generality, and that's good to have in various limiting arguments. Uh, we like limiting arguments because that is our main way to approximate which our main way to being healthy, wealthy, and wise!
So the Lebesgue integral of X^+ we write as E[X^+]. Similarly for X^-. Now no way do we want to be subtracting one infinity from another since permitting that would trash the usual laws of arithmetic.
So, for our second step, in the case X^- >= 0, if at least one of E[X^+] and E[X^-] is finite, then we define E[X] = E[X^+] - E[X^-].
Now we've defined expectation ("average") of a real random variable X. Our definition is just the Lebesgue integral. For the Lebesgue integral, we wanted the sigma algebras.
On the real line we can consider the sigma algebra of Lebesgue measurable sets; that's larger than the Borel sets. Then we just ask, assume, assert, believe, ..., that our random variables are measurable with respect to the sigma algebra of Lebesgue sets and the sigma algebra of the events. Uh, right, Lebesgue measure on R assigns Lebesgue measure b - a to interval (a,b) and extends from there. Fine details are in various texts by Rudin, Royden, etc.
That's the beginnings of the role of sigma algebras in advanced approaches to probability, statistics, and stochastic processes. It turns out, the sigma algebra approach is for several parts of what we want in probability, much nicer, e.g., for defining independence and conditional expectation. E.g., if we want to know that some set of uncountably infinitely many random variables are independent, we can. Same for conditioning on uncountably infinitely many random variables, e.g., the past history of a stochastic process.
below. In short, the answer is that a real valued random variable X is a real valued function. The domain of the function is a set of trials. So, for a trial w (usually written as lower case omega), X(w) is a real number.
Then the event for real number x
X <= x
is really shorthand notation for
{w|X(w) <= x}
So, typically we don't mention the w.
Moreover, typically for all but grad school mathematicians taking a course in "graduate probability" we don't mention that X is a function. Instead we just say something like, X is the number we get from running an experimental trial, one of all the numbers we "might have gotten" considering the probability distribution of X.
You are correct: You sense some mushy ground under the foundations of probability theory, and you are not nearly the first to so sense.
Long an answer was, "it works great in practice" which is doesn't make the mush any more firm.
Well, in 1933 A. Kolmogorov gave a solid mathematical foundation for probability theory. That's the usual foundations for advanced work in probability, statistics, and stochastic processes. My post
Some of the consequences are surprising, but I omit those. And we end up assuming that in all the universe all we ever see is just some one trial and don't say anything about the other trials but imagine about them a lot. That point may be hard to swallow.
IIRC, one line of argument is just that in probability there are lots of possibilities we just don't distinguish. E.g, maybe the police have long since concluded that nearly everyone driving a car with custom installed, hidden compartments is a drug dealer and then conclude that a person with such compartment is "likely" a drug dealer. Well, of course, actually, they might not be a drug dealer and have the car and its compartments for some other reason. So, the police are putting all owners of cars with hidden compartments in a box and refusing to distinguish them, insisting that they all be treated the same until there is evidence otherwise. It may be that more can be said. For now, make of such lines of thought what you will.
