From microstates to Gibbs measures
How should a probability distribution be chosen when the microscopic state of a system is unknown, but its average energy is prescribed?
Let \(\Omega\) be a nonempty finite set, called the configuration space. Its elements \(\omega\in\Omega\) are the microstates of the system. A Hamiltonian is a function
Thus \(H(\omega)\) assigns an energy to each microscopic configuration \(\omega\).
We equip \(\Omega\) with the discrete \(\sigma\)-algebra \(\mathcal F=\mathcal P(\Omega)\). An observable is any map \(A:\Omega\to\mathbb R\).
A statistical state is a probability measure \(\nu\) on \((\Omega,\mathcal F)\). We write
\[ \mathcal M_1(\Omega) :=\left\{\nu:\Omega\to[0,1]: \sum_{\omega\in\Omega}\nu(\omega)=1\right\}. \]Once \(\nu\) has been specified, \((\Omega,\mathcal F,\nu)\) is a probability space. In particular, the Hamiltonian \(H\) is an observable and therefore a random variable on that probability space.
For an observable \(A:\Omega\to\mathbb R\), we use the following standard quantities.
There is an important distinction here: \(H\) is part of the description of the system, while \(\nu\) describes how likely we consider each configuration to be. Which probability measure should we choose? To answer this question, we need a measure of uncertainty.
To a statistical state \(\nu\in\mathcal M_1(\Omega)\), we associate its Shannon entropy. Informally, entropy measures how spread out the probability mass is over the accessible microstates.
For \(\nu\in\mathcal M_1(\Omega)\), define
\[ S_{\mathrm{Sh}}(\nu) :=-\sum_{\omega\in\Omega}\nu(\omega)\log\nu(\omega), \]with the convention \(0\log 0:=0\). Throughout, \(\log\) is the natural logarithm.
The function \(\psi:[0,1]\to\mathbb R\) given by
\[ \psi(x)=\begin{cases}-x\log x,&x\in(0,1],\\0,&x=0,\end{cases} \qquad\text{satisfies}\qquad S_{\mathrm{Sh}}(\nu)=\sum_{\omega\in\Omega}\psi(\nu(\omega)). \]We shall repeatedly use the fact that \(\psi\) is continuous on \([0,1]\), nonnegative there, and strictly concave.
The first result collects the properties that make entropy the appropriate quantity to maximize.
Let \(\Omega\) be a nonempty finite set. Then:
- Strict concavity. For any distinct \(\nu,\rho\in\mathcal M_1(\Omega)\) and every \(t\in(0,1)\), \[ S_{\mathrm{Sh}}(t\nu+(1-t)\rho) >tS_{\mathrm{Sh}}(\nu)+(1-t)S_{\mathrm{Sh}}(\rho). \]
- Bounds. Every \(\nu\in\mathcal M_1(\Omega)\) satisfies \[0\le S_{\mathrm{Sh}}(\nu)\le\log|\Omega|.\] The lower bound is attained precisely by point masses \(\delta_\omega\), while the upper bound is attained precisely by the uniform measure \(\nu_{\mathrm{unif}}(\omega)=1/|\Omega|\).
First observe that \(\mathcal M_1(\Omega)\) is convex. If \(\nu,\rho\in\mathcal M_1(\Omega)\) and \(0\le t\le1\), define \(\mu=t\nu+(1-t)\rho\). Then \(\mu(\omega)\ge0\) for all \(\omega\), and
\[ \sum_{\omega\in\Omega}\mu(\omega) =t\sum_{\omega\in\Omega}\nu(\omega) +(1-t)\sum_{\omega\in\Omega}\rho(\omega) =t+(1-t)=1. \]Thus \(\mu\) is again a probability measure, so the expression in the lemma is well defined.
We now establish the strict concavity of \(\psi\). For \(x>0\),
\[\psi''(x)=-\frac1x<0.\]This gives strict concavity whenever both arguments belong to \((0,1]\). The boundary case must also be checked because probabilities may vanish. If \(x=0\) and \(y>0\), then for \(t\in(0,1)\),
The case \(x>0\), \(y=0\) is analogous. If \(x=y=0\), equality holds, as it must. Consequently \(\psi\) is strictly concave on all of \([0,1]\).
Apply this inequality with \(x=\nu(\omega)\) and \(y=\rho(\omega)\), and sum over \(\omega\in\Omega\):
Because \(\nu\ne\rho\), there is at least one \(\omega\) at which their masses differ. The corresponding inequality is strict, so the sum is strictly greater.
□Since \(\psi(x)=-x\log x\ge0\) for all \(x\in[0,1]\),
\[S_{\mathrm{Sh}}(\nu)=\sum_{\omega\in\Omega}\psi(\nu(\omega))\ge0.\]For \(|\Omega|\ge2\), equality can hold only when every \(\nu(\omega)\) is either \(0\) or \(1\). Their sum being one, \(\nu\) must be a point mass. Conversely, any point mass has zero entropy. The same conclusion is immediate if \(|\Omega|=1\).
For the upper bound, use Jensen's inequality for the concave function \(\psi\):
By strict concavity, equality in Jensen's inequality requires all the masses \(\nu(\omega)\) to agree. Normalization then gives \(\nu(\omega)=1/|\Omega|\) for every \(\omega\). Conversely, this uniform measure has entropy \(\log|\Omega|\).
□The two extreme cases have an immediate interpretation. If a single configuration has probability one, we have no uncertainty about the microstate. If all configurations are equally likely, uncertainty is maximal.
How uncertain is a three-state system?
Choose a probability measure on \(\Omega=\{\omega_1,\omega_2,\omega_3\}\). The bars show its three probability masses; the numbers below show its entropy and the upper bound \(\log 3\).
The entropy of a mixture lies above the chord
Fix \(\nu=(0.9,0.1,0)\) and \(\rho=(0,0.2,0.8)\), then vary the mixing parameter \(t\).
In realistic models, however, we often know more than just the set of configurations. For example, we might know the empirically measured mean of an observable, especially of the energy. Our next task is to find the most uncertain probability measure that is compatible with that information, without introducing additional assumptions.
Fix a Hamiltonian \(H:\Omega\to\mathbb R\), assume that \(H\) is not constant, and write
\[ H_{\min}=\min_{\omega\in\Omega}H(\omega), \qquad H_{\max}=\max_{\omega\in\Omega}H(\omega). \]Suppose that the mean energy is known to have some value \(u\) strictly between these extrema. The available information is encoded by the admissible set
\[ \mathcal A_u :=\left\{\nu\in\mathcal M_1(\Omega): \mathbb E_\nu[H]=u\right\}. \]Among the measures in \(\mathcal A_u\), we wish to maximize \(S_{\mathrm{Sh}}\).
Let \(\Omega\) be a nonempty finite set and let \(H:\Omega\to\mathbb R\) be nonconstant. Fix
\[u\in(H_{\min},H_{\max}).\]Then the constrained optimization problem
\[ \max\left\{S_{\mathrm{Sh}}(\nu): \nu\in\mathcal M_1(\Omega),\ \mathbb E_\nu[H]=u\right\} \]has a unique solution. It is the Gibbs measure
\[ \boxed{\quad \mu_\beta(\omega)=\frac{e^{-\beta H(\omega)}}{Z(\beta)}, \qquad Z(\beta):=\sum_{\eta\in\Omega}e^{-\beta H(\eta)}. \quad} \]The parameter \(\beta\in\mathbb R\) is uniquely determined by the equation
\[\mathbb E_{\mu_\beta}[H]=u.\]The normalization factor \(Z(\beta)\) is called the partition function. It is finite and strictly positive because \(\Omega\) is finite. Notice that the theorem asserts more than the Gibbs form: it determines one and only one value of \(\beta\) for each admissible interior mean energy \(u\).
We divide the proof into five steps. First, we make sure there are admissible measures. Then compactness and strict concavity give existence and uniqueness of an abstract maximizer. Next, Lagrange multipliers suggest its form; we determine the parameter from the energy constraint. Finally, the supporting-hyperplane inequality for the concave entropy proves that the Gibbs candidate really is the maximizer.
Choose \(\omega_{\min},\omega_{\max}\in\Omega\) such that \(H(\omega_{\min})=H_{\min}\) and \(H(\omega_{\max})=H_{\max}\). Define
\[ t:=\frac{H_{\max}-u}{H_{\max}-H_{\min}}\in(0,1), \qquad \nu_0:=t\delta_{\omega_{\min}}+(1-t)\delta_{\omega_{\max}}. \]This is a probability measure, being a convex combination of two point masses. More importantly,
Therefore \(\nu_0\in\mathcal A_u\), proving that \(\mathcal A_u\neq\varnothing\).
Write \(\Omega=\{\omega_1,\ldots,\omega_m\}\). Every \(\nu\in\mathcal M_1(\Omega)\) can be identified with the vector \(p=(p_1,\ldots,p_m)\), where \(p_i=\nu(\omega_i)\). Thus
\[ \mathcal M_1(\Omega) \cong\left\{p\in[0,1]^m:\sum_{i=1}^{m}p_i=1\right\}. \]We give this simplex the subspace topology inherited from the Euclidean topology on \(\mathbb R^m\). Thus \(\nu_n\to\nu\) means that \(\nu_n(\omega_i)\to\nu(\omega_i)\) for every \(i=1,\ldots,m\). Equivalently, one may use the total-variation distance
These topologies agree because \(\Omega\) is finite. All compactness and continuity statements below refer to this topology.
The right-hand side is closed in the compact cube \([0,1]^m\), and therefore compact. The constraint \(\sum_i p_iH(\omega_i)=u\) is a closed affine condition. Hence
\[ \mathcal A_u =\left\{p\in[0,1]^m: \sum_{i=1}^m p_i=1,\ \sum_{i=1}^m p_iH(\omega_i)=u\right\} \]is compact as well. It is convex, since for any \(\nu,\rho\in\mathcal A_u\) and \(t\in[0,1]\),
\[ \mathbb E_{t\nu+(1-t)\rho}[H] =t\mathbb E_\nu[H]+(1-t)\mathbb E_\rho[H] =tu+(1-t)u=u. \]The entropy \(S_{\mathrm{Sh}}:\mathcal M_1(\Omega)\to\mathbb R\) is continuous for this topology (and the usual topology on \(\mathbb R\)). Indeed, \(\lim_{x\downarrow0}(-x\log x)=0\), so \(\psi\) is continuous on \([0,1]\). Therefore, if \(\nu_n\to\nu\), then
Since \(\mathcal A_u\) is nonempty and compact, Weierstrass' theorem shows that \(S_{\mathrm{Sh}}\) attains its maximum there.
Moreover, Lemma 1.3 says that \(S_{\mathrm{Sh}}\) is strictly concave. If two distinct maximizers existed, their midpoint would still belong to \(\mathcal A_u\) but would have strictly greater entropy, a contradiction. Thus the entropy maximizer exists and is unique. We now identify it.
First perform the standard stationary-point calculation in the interior of the probability simplex, where all \(p_i>0\). The normalization and energy constraints are
\[ \sum_{i=1}^m p_i=1, \qquad \sum_{i=1}^m H(\omega_i)p_i=u. \]Introduce the Lagrangian
\[ \mathcal L(p,a,b) :=-\sum_{i=1}^m p_i\log p_i -a\left(\sum_{i=1}^m p_i-1\right) -b\left(\sum_{i=1}^m H(\omega_i)p_i-u\right). \]The stationarity equations, for \(i=1,\ldots,m\), read
\[ \frac{\partial\mathcal L}{\partial p_i} =-1-\log p_i-a-bH(\omega_i)=0. \]Solving for \(p_i\) gives
\[p_i=e^{-1-a-bH(\omega_i)}.\]Normalization requires
\[ 1=e^{-1-a}\sum_{i=1}^m e^{-bH(\omega_i)} =e^{-1-a}Z(b), \quad\text{hence}\quad e^{-1-a}=\frac1{Z(b)}. \]Consequently, any interior stationary point must have the form
\[ \mu_b(\omega)=\frac{e^{-bH(\omega)}}{Z(b)}. \]Let \(b\in\mathbb R\). Define the expected energy under the Gibbs measure by
\[ U(b):=\mathbb E_{\mu_b}[H] =\frac{\sum_{\omega\in\Omega}H(\omega)e^{-bH(\omega)}}{Z(b)}. \]Since
\[ Z'(b)=-\sum_{\omega\in\Omega}H(\omega)e^{-bH(\omega)}, \]we can write
\[ \boxed{\quad U(b)=-\frac{d}{db}\log Z(b).\quad} \]Differentiating once more,
For every finite \(b\), \(\mu_b(\omega)>0\) for all \(\omega\in\Omega\). Since \(H\) is not constant, its variance under \(\mu_b\) is strictly positive. Therefore
\[U'(b)=-\operatorname{Var}_{\mu_b}(H)<0\qquad\text{for every }b\in\mathbb R.\]Thus \(U\) is continuous and strictly decreasing. Furthermore, as \(b\to-\infty\), the weights favor the configurations with maximal energy; as \(b\to+\infty\), they favor those with minimal energy. Consequently,
\[ \lim_{b\to-\infty}U(b)=H_{\max}, \qquad \lim_{b\to+\infty}U(b)=H_{\min}. \]Here is the underlying argument for the second limit. Write
\[ \mu_b(\omega)= \frac{e^{-b(H(\omega)-H_{\min})}} {\sum_{\eta\in\Omega}e^{-b(H(\eta)-H_{\min})}}. \]As \(b\to+\infty\), every term with \(H(\omega)>H_{\min}\) tends to zero, whereas the terms with \(H(\omega)=H_{\min}\) remain equal to one. Hence the measure concentrates on minimizing configurations and \(U(b)\to H_{\min}\). The other limit follows by the analogous argument with \(H_{\max}\).
Because \(u\in(H_{\min},H_{\max})\), continuity, strict monotonicity, and the intermediate value theorem show that there exists exactly one \(\beta\in\mathbb R\) such that
\[U(\beta)=\mathbb E_{\mu_\beta}[H]=u.\]Therefore \(\mu_\beta\in\mathcal A_u\), as required.
We must now prove that the Gibbs candidate obtained in Step 4 is indeed the entropy maximizer from Step 2. Notice first that \(\mu_\beta\) has full support, so \(S_{\mathrm{Sh}}\) is differentiable at \(\mu_\beta\) as a function of its \(|\Omega|\) coordinates.
For a differentiable concave function, the graph lies below each of its tangent hyperplanes. Applied to the entropy, this gives, for every \(\nu\in\mathcal M_1(\Omega)\),
\[ S_{\mathrm{Sh}}(\nu) \le S_{\mathrm{Sh}}(\mu_\beta) +\sum_{\omega\in\Omega}\big(-1-\log\mu_\beta(\omega)\big) \big(\nu(\omega)-\mu_\beta(\omega)\big). \]From the Gibbs formula,
\[ \log\mu_\beta(\omega)=-\beta H(\omega)-\log Z(\beta), \]and hence
\[ -1-\log\mu_\beta(\omega) =-1+\log Z(\beta)+\beta H(\omega). \]In particular, if \(\nu\in\mathcal A_u\), the correction term in the supporting-hyperplane inequality is
\[ \begin{aligned} &\sum_{\omega\in\Omega} (-1+\log Z(\beta)+\beta H(\omega)) (\nu(\omega)-\mu_\beta(\omega))\\ &\quad=(-1+\log Z(\beta)) \underbrace{\sum_{\omega\in\Omega}(\nu(\omega)-\mu_\beta(\omega))}_{=\,0} +\beta\underbrace{\big(\mathbb E_\nu[H] -\mathbb E_{\mu_\beta}[H]\big)}_{=\,u-u=0}=0. \end{aligned} \]We have therefore proved
\[ S_{\mathrm{Sh}}(\nu)\le S_{\mathrm{Sh}}(\mu_\beta) \qquad\text{for every }\nu\in\mathcal A_u. \]So \(\mu_\beta\) is a global maximizer. Its uniqueness follows from strict concavity, as already shown in Step 2. This concludes the proof.
□Find the maximum: it is the Gibbs distribution
Keep the three energy levels \(H(\omega_1)=0\), \(H(\omega_2)=1\), and \(H(\omega_3)=3\). Write \(p_i=\nu(\omega_i)\). We compare probability distributions with the same prescribed mean energy \(u\).
The blue point is calculated from the Gibbs formula. It lies at the unique maximum of the curve. The vertical orange segment measures the entropy lost by choosing a different admissible distribution.
Both distributions have mean energy 1.300.
Bar height = probability, on the same scale in both panels. The labels 0, 1, 3 are the energies.
How is the Gibbs parameter \(\beta\) determined?
For each chosen \(u\), we solve \(U(\beta)=u\), where \(U(\beta)=\mathbb E_{\mu_\beta}[H]\). The function \(U\) is strictly decreasing, so the horizontal line at height \(u\) meets it exactly once.
Theorem 1.4 assumed that the prescribed mean energy lies strictly between \(H_{\min}\) and \(H_{\max}\). This is essential for there to be a finite parameter \(\beta\). What if \(u=H_{\min}\)?
Define the set of energy-minimizing configurations by
\[ \Omega_{\min}:=\{\omega\in\Omega:H(\omega)=H_{\min}\}. \]If the constraint is \(\mathbb E_\nu[H]=H_{\min}\), then the unique entropy maximizer is the uniform distribution on \(\Omega_{\min}\):
\[ \nu_{\min}(\omega) =\begin{cases} \dfrac1{|\Omega_{\min}|},&\omega\in\Omega_{\min},\\[5pt] 0,&\omega\notin\Omega_{\min}. \end{cases} \]Similarly, for \(u=H_{\max}\), the unique maximizer is uniform on \(\Omega_{\max}:=\{\omega:H(\omega)=H_{\max}\}\).
A probability measure \(\nu\) is admissible for the minimum-energy constraint precisely when
Each summand is nonnegative. Their sum can therefore vanish only if every term vanishes. In particular, if \(H(\omega)>H_{\min}\), then necessarily \(\nu(\omega)=0\). Thus
\[ \mathbb E_\nu[H]=H_{\min} \quad\Longleftrightarrow\quad \nu(\Omega_{\min})=1. \]The constrained problem has reduced to maximizing the entropy of a probability measure supported on \(\Omega_{\min}\). By Lemma 1.3, the entropy on this finite set is uniquely maximized by its uniform distribution, which is exactly \(\nu_{\min}\).
The case \(H_{\max}\) is identical after replacing \(H\) by \(-H\).
□These boundary measures are also the limits of the Gibbs family:
\[ \lim_{\beta\to+\infty}\mu_\beta=\nu_{\min}, \qquad \lim_{\beta\to-\infty}\mu_\beta=\nu_{\max}. \]To see this explicitly, suppose that \(H(\omega)=c\) for every \(\omega\in\Omega\), and set \(m=|\Omega|\). For every \(\beta\in\mathbb R\),
Thus every value of \(\beta\) gives exactly the same probability measure: the uniform law. Its mean energy is
Consequently, when the prescribed mean is \(u=c\), the equation \(U(\beta)=u\) is satisfied by every \(\beta\in\mathbb R\). When \(u\ne c\), there is no admissible probability measure and no solution of that equation. Equivalently, \(U'(\beta)=-\operatorname{Var}_{\mu_\beta}(H)=0\), so the strict monotonicity used in Step 4 is lost. The entropy-maximizing measure is still unique, by Lemma 1.3, but its parameter \(\beta\) is not.
We conclude with the following result.
For every probability measure \(\nu\) on a finite nonempty set \(\Omega\),
where \(\operatorname{supp}\nu:=\{\omega\in\Omega:\nu(\omega)>0\}\) is the support of \(\nu\). Equivalently, \(S_{\mathrm{Sh}}(\nu)=0\) if and only if \(\nu=\delta_{\omega_0}\) for some \(\omega_0\in\Omega\).
For \(p\in[0,1]\), the quantity \(-p\log p\) is nonnegative and is zero exactly when \(p=0\) or \(p=1\). Therefore the finite sum
can vanish only if every summand is zero, that is, only if \(\nu(\omega)\in\{0,1\}\) for every \(\omega\in\Omega\). Since the masses sum to one, exactly one configuration \(\omega_0\) has mass one and all the others have mass zero. Hence \(\nu=\delta_{\omega_0}\).
Conversely, every Dirac measure has entropy zero. Indeed, using \(\log 1=0\) and the convention \(0\log 0=0\),
A probability measure has a one-point support exactly when it is a Dirac measure, which proves the remaining equivalence.
□If \(\nu\) has full support, then \(\operatorname{supp}\nu=\Omega\); under this additional assumption, zero entropy is equivalent to \(|\Omega|=1\). In particular, for a Gibbs measure with finite \(\beta\),
because \(\mu_\beta(\omega)>0\) for every \(\omega\in\Omega\).
Let \(I\subseteq\mathbb R\) be an interval and let \(f:I\to\mathbb R\) be concave. If \(x_1,\ldots,x_n\in I\), \(\lambda_i>0\), and \(\sum_{i=1}^n\lambda_i=1\), then
If \(f\) is strictly concave, equality holds if and only if \(x_1=\cdots=x_n\). Positivity of all weights is relevant to this equality condition.
Let \(K\subseteq\mathbb R^m\) be nonempty and compact for the Euclidean topology. Every continuous function \(F:K\to\mathbb R\) attains both its minimum and its maximum on \(K\): there are \(x_{\min},x_{\max}\in K\) such that
Let \(a<b\) and let \(f:[a,b]\to\mathbb R\) be continuous. Every value \(y\) between \(f(a)\) and \(f(b)\) is attained: there exists \(c\in[a,b]\) with \(f(c)=y\). If \(f\) is strictly monotone, this \(c\) is unique.
Let \(C\subseteq\mathbb R^m\) be convex and let \(F:C\to\mathbb R\) be concave. Suppose that \(F\) has a differentiable extension to an open neighbourhood of \(x\in C\). Then
The inequality allows \(y\) to lie on the boundary of \(C\). Indeed, concavity gives \(F(y)-F(x)\le [F(x+t(y-x))-F(x)]/t\) for \(0<t<1\); letting \(t\downarrow0\) yields the displayed inequality. For the entropy, the differentiable extension is \(-\sum_i p_i\log p_i\) on the open positive orthant, and \(x=\mu_\beta\) has all coordinates positive.