Convexity and Jensen’s Inequality (and the AM-GM Inequality)

Introduction

In this (relatively) short article, we will discuss the extremely useful Jensen’s inequality, named after the Danish mathematician Johan Jensen.

Jensen’s inequality can be viewed as a generalization of the inequality defining real-valued convex functions. As a reminder, a function

is convex if

Intuitively, this means that if you draw the graph of the function f, and pick two points of the graph and connect them with a line, then the line lies above the graph of the function, as shown in an example in the picture below.

The graph of a convex function f. [Source:https://en.wikipedia.org/wiki/Jensen%27s_inequality]

Jensen’s inequality generalizes this picture when multiple points (even infinitely many!) are involved. A clean way to state the inequality is by using the language of probability. This is because a convex combination of points can be viewed as a probability distribution over these points. For example, in the above definition of convexity, we consider the weighted sum of two points

This can be thought of as a probability distribution over these two points, where the first one is chosen with probability t, and the second one with probability 1-t.

Jensen’s inequality

If X is a random variable and f is a convex function, then

Note: If f is concave, then the inequality holds in the opposite direction.

Sketch of proof

In this section, we briefly discuss the proof. The easiest form of the inequality that one can prove is the finite form, where a finite number of points are involved. More specifically, we can prove, by using induction on the number n of points, that given a convex function f and any n points

and corresponding non-negative t-values

we have

The base case of the induction is for n = 2, and corresponds to the definition of convexity. The induction step then is to move from n to n+1 points. This is easily done as follows. Since we have n+1 points, this means that all t-values of them are strictly positive and add up to 1. We consider the first n points (by arbitrarily assigning an order to them), and we set

Since

we apply the definition of convexity and, by also using the induction hypothesis on the first n points, we get

The induction proof is now complete. In order to generalize to the more general probability setting in which we stated the inequality, one has to use some analytical arguments, which are more technical, and which we skip. The interested reader can further read the Wikipedia entry for Jensen’s inequality (which can be found here), or have a look at the multiple sources that are widely available online.

As noted above, the inequality holds in the opposite direction when dealing with concave functions. One can mimic the above argument in the context of concave functions in order to verify that this is indeed the case; we encourage the reader to work out the details.

An application: proof of the AM-GM inequality

Here, we give a very short proof of yet another very useful inequality, the so-called inequality of arithmetic and geometric means, or, in short, AM-GM inequality. The AM-GM inequality can be stated as follows.

Let

be positive real numbers. Then, we always have

where equality holds only if

In order to prove the above inequality, we will use Jensen’s inequality, and in particular, we will apply it to a concave (and not convex!) function. As noted above, the inequality holds in the opposite direction when f is concave. The concave function we use is the well-known logarithmic function f(x) = ln x. We have

where we used standard properties of logarithms (see here for more details). Using now the fact that the exponential function is an increasing function, we apply it to both sides of the obtained inequality and we get

which implies the AM-GM inequality. We are done!

Conclusion

In this article, we discussed the famous and broadly applicable to Jensen’s inequality. It is always good to keep the inequality in the back of our heads, as it describes the fundamental property of convex functions, and convex functions can be found almost everywhere! As an added bonus, we saw that the inequality can also be applied to concave functions by reversing the direction of the inequality and using this, we got a very short proof of another very useful inequality, the AM-GM inequality.

Sources