In functional analysis, a reproducing kernel Hilbert space (RKHS) is a Hilbert space of functions in which point evaluation is a continuous linear functional is then called the reproducing kernel via the inner product.

An immediate consequence of this property is that convergence in norm implies uniform convergence.

It is not entirely straightforward to construct natural examples of a Hilbert space which are not an RKHS in a non-trivial fashion. Some examples, however, have been found.

While, formally, L2 spaces are defined as Hilbert spaces of equivalence class would need to be the non-existent Dirac delta function). However, there are RKHSs in which the norm is an L2-norm, such as the space of band-limited functions (see the example below).

" can be performed by taking an inner product with a function determined by the kernel. Such a reproducing kernel exists if and only if every evaluation functional is continuous.

The reproducing kernel was first introduced in the 1907 work of Stanisław Zaremba concerning boundary value problems for harmonic and biharmonic functions. James Mercer simultaneously examined functions which satisfy the reproducing property in the theory of integral equations. The idea of the reproducing kernel remained untouched for nearly twenty years until it appeared in the dissertations of Gábor Szegő, Stefan Bergman, and Salomon Bochner. The subject was eventually systematically developed in the early 1950s by Nachman Aronszajn and Stefan Bergman.

These spaces have wide applications, including complex analysis, harmonic analysis, and quantum mechanics. Reproducing kernel Hilbert spaces are particularly important in the field of statistical learning theory because of the celebrated representer theorem which states that every function in an RKHS that minimises an empirical risk functional can be written as a linear combination of the kernel function evaluated at the training points. This is a practically useful result as it effectively simplifies the empirical risk minimization problem from an infinite dimensional to a finite dimensional optimization problem.

For ease of understanding, we provide the framework for real-valued Hilbert spaces. The theory can be easily extended to spaces of complex-valued functions and hence include the many important examples of reproducing kernel Hilbert spaces that are spaces of analytic functions.

Definition

be an arbitrary set a Hilbert space of real-valued functions, equipped with pointwise addition and pointwise scalar multiplication. The evaluation,

We say that H is a reproducing kernel Hilbert space is continuous is a bounded operator such that

.

While property (. This function is the so-called reproducing kernel from which the RKHS takes its name. More formally, the Riesz representation theorem with the reproducing property,

we have that

.

in the complex case) by

in the complex case) is both symmetric (resp. conjugate symmetric) and positive definite, i.e.

for which it is a reproducing kernel.

Examples

is the counting measure is the indicator function.

Nontrivial reproducing kernel Hilbert spaces often involve analytic functions, as we now illustrate by example. Consider the Hilbert space of bandlimited continuous functions. Fix some cutoff frequency and define the Hilbert space

is the Fourier transform. As the inner product, we use

that tend to zero at infinity, essentially by the Riemann-Lebesgue lemma of entire holomorphic functions, by the Paley–Wiener theorem.

From the Fourier inversion theorem, we have

It then follows by the Cauchy–Schwarz inequality and Plancherel's theorem,

is indeed a RKHS.

in this case is given by

defined above is given by

which is a consequence of the time-shifting property of the Fourier transform. Consequently, using Plancherel's theorem, we have

Thus we obtain the reproducing property of the kernel.

in this case is the "bandlimited version" of the Dirac delta function tends to infinity.

Moore–Aronszajn theorem

We have seen how a reproducing kernel Hilbert space defines a reproducing kernel function that is both symmetric and positive definite. The Moore–Aronszajn theorem goes in the other direction; it states that every symmetric, positive definite kernel defines a unique reproducing kernel Hilbert space. The theorem first appeared in Aronszajn's Theory of Reproducing Kernels, although he attributes it to E. H. Moore.

1. Theorem. Suppose K is a symmetric, positive definite kernel on a set X. Then there is a unique Hilbert space of functions on X for which K is a reproducing kernel.

Proof. For all x in X, define Kx = K(x, ⋅ ). Let H0 be the linear span of {Kx : x ∈ X}. Define an inner product on H0 by

. The symmetry of this inner product follows from the symmetry of K and the non-degeneracy follows from the fact that K is positive definite.

Let H be the completion of H0 with respect to this inner product. Then H consists of functions of the form

Now we can check the reproducing property ():

To prove uniqueness, let G be another Hilbert space of functions for which K is a reproducing kernel. For every x and y in X, () implies that

because G is complete and contains H0 and hence contains its completion.

Now we need to prove that every element of G is in H be an element of G. Since H is a closed subspace of G then, since K is a reproducing kernel of G and H:

belongs to H in G in G and concludes the proof.

Integral operators and Mercer's theorem

via the integral operator using Mercer's theorem be a compact space equipped with a strictly positive finite Borel measure as

.

is a reproducing kernel so that the corresponding RKHS can be defined in terms of these eigenvalues and eigenfunctions. We provide the details below.

is a compact, continuous, self-adjoint, and positive operator. The spectral theorem may be written in terms of the eigenvalues and continuous eigenfunctions as

such that

.

is given by

given by

This representation of the RKHS has application in probability and statistics, for example to the Karhunen–Loève representation for stochastic processes and kernel PCA.

Feature maps

A feature map is a Hilbert space which we will call the feature space. The first sections presented the connection between bounded/continuous evaluation functions, positive definite functions, and integral operators and in this section we provide another representation of the RKHS in terms of feature maps.

Every feature map defines a kernel via

. Conversely, every positive definite function and corresponding reproducing kernel Hilbert space has infinitely many associated feature maps such that () holds.

. Then (.

. Moreover, every feature map can naturally define a RKHS by means of the definition of a positive definite function.

Lastly, feature maps allow us to construct function spaces that reveal another perspective on the RKHS. Consider the linear space

by

. This representation implies that the elements of the RKHS are inner products of elements in the feature space and can accordingly be seen as hyperplanes. This view of the RKHS is related to the kernel trick in machine learning.

Properties

Useful properties of RKHSs:

• It then follows that

• is also a reproducing kernel.

• . Define a pseudo-metric on X as

  1. By the Cauchy–Schwarz inequality,

  2. as a measure of similarity will be closer to 0.

  .

Common examples

Bilinear kernels

corresponding to this kernel is the dual space.

Polynomial kernels

Radial basis function kernelRadial basis function kernels

. Some examples include:

• Gaussian or squared exponential kernel:

• Laplacian kernel:

  1. with this kernel is:

Bergman kernelBergman kernels

We also provide examples of Bergman kernels. Let X be finite and let H consist of all complex-valued functions on X. Then an element of H can be represented as an array of complex numbers. If the usual inner product is used, then Kx is the function whose value is 1 at x can be thought of as an identity matrix since

In this case, H.

denotes the unit disc) is more sophisticated. Here the Bergman space A^2(\mathbb{D}) is the space of square-integrable holomorphic functions is

is a RKHS with reproducing kernel

Extension to vector-valued functions

In this section we extend the definition of the RKHS to spaces of vector-valued functions as this extension is particularly important in multi-task learning and manifold regularization is a symmetric function that is now a positive semi-definite matrix

and

, another property similar to the scalar-valued case.

We can gain intuition for the vvRKHS by taking a component-wise perspective on these spaces. In particular, we find that every vvRKHS is isometrically isomorphic and the corresponding reproducing kernel

for every set of pairs

The connection to the scalar-valued RKHS can then be made by the fact that every matrix-valued kernel can be identified with a kernel of the form of () via

Moreover, every kernel with the form of ( be defined as

.

While this view of the vvRKHS can be useful in multi-task learning, this isometry does not reduce the study of the vector-valued case to that of the scalar-valued case. In fact, this isometry procedure can make both the scalar-valued kernel and the input space too difficult to work with in practice as properties of the original kernels are often lost.

An important class of matrix-valued reproducing kernels are separable-dimensional symmetric positive semi-definite matrix. In light of our previous discussion these kernels are of the form

. As the scalar-valued kernel encodes dependencies between the inputs, we can observe that the matrix-valued kernel encodes dependencies among both the inputs and the outputs.

We lastly remark that the above theory can be further extended to spaces of functions with values in function spaces but obtaining kernels for these spaces is a more difficult task.

Connection between RKHSs and the ReLU function

The ReLU function and is a mainstay in the architecture of neural networks where it is used as an activation function. One can construct a ReLU-like nonlinear function using the theory of reproducing kernel Hilbert spaces. Below, we derive this construction and show how it implies the representation power of neural networks with ReLU activations.

) derivative. It has the inner product

. The Fundamental Theorem of Calculus then gives

where

i.e.

.

has the following representations with the ReLu function:

Using this formulation, we can apply the representer theorem to the RKHS, letting one prove the optimality of using ReLU activations in neural network settings.

See also

• Positive definite kernel

• Mercer's theorem

• Kernel trick

• Kernel embedding of distributions

• Representer theorem

Notes

References

• Alvarez, Mauricio, Rosasco, Lorenzo and Lawrence, Neil, “Kernels for Vector-Valued Functions: a Review,” https://arxiv.org/abs/1106.6251, June 2011.

• Berlinet, Alain and Thomas, Christine. Reproducing kernel Hilbert spaces in Probability and Statistics, Kluwer Academic Publishers, 2004.

• De Vito, Ernest, Umanita, Veronica, and Villa, Silvia. "An extension of Mercer theorem to vector-valued measurable kernels," , June 2013.

• Durrett, Greg. 9.520 Course Notes, Massachusetts Institute of Technology, https://www.mit.edu/~9.520/scribe-notes/class03_gdurett.pdf, February 2010.

• Okutmustur, Baver. “Reproducing Kernel Hilbert Spaces,” M.S. dissertation, Bilkent University, https://users.metu.edu.tr/baver/MS.Thesis.pdf, August 2005.

• Paulsen, Vern. “An introduction to the theory of reproducing kernel Hilbert spaces,” https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=440218056738e05b5ab43679f932a9f33fccee87.

• Rosasco, Lorenzo and Poggio, Thomas. "A Regularization Tour of Machine Learning – MIT 9.520 Lecture Notes" Manuscript, Dec. 2014.

• Wahba, Grace, Spline Models for Observational Data, SIAM, 1990.