Dimension-Independent Kernel eps-Covers

Abstract

We introduce the notion of an eps-cover for a kernel range space. A kernel range space concerns a set of points X in R^d and the space of all queries by a fixed kernel (e.g., a Gaussian kernel K(p,.) = exp(-|p-.|²), where p is in \R^d). For a point set X of size n, a query returns a vector of values R_p in Rⁿ, where the i-th coordinate (R_p)_i = K(p, x_i)$ for x_i in X$ An eps-cover is a subset of points Q in R^d so for any p in R^d$ that (1/n) |R_p - R_q|₁ <= eps for some q in Q. This is a smooth analog of Haussler's notion of eps-covers for combinatorial range spaces (e.g., defined by subsets of points within a ball query) where the resulting vectors R_p are in {0, 1}ⁿ instead of [0, 1]ⁿ. The kernel versions of these range spaces show up in data analysis tasks where the coordinates may be uncertain or imprecise, and hence one wishes to add some flexibility in the notion of inside and outside of a query range.

Our main result is that, unlike combinatorial range spaces, the size of kernel eps-covers is independent of the input size n and dimension d. We obtain a bound of 2^{O~(1/\eps^2)}, where O~(f(1 / eps)) hides log factors in (1 / eps) that can depend on the kernel. This implies that by relaxing the notion of boundaries in range queries, eventually the curse of dimensionality disappears, and may help explain the success of machine learning in very high-dimensions.We also complement this result with a lower bound of almost (1/\eps)^{Omega(1/\eps)}, showing the exponential dependence on 1/eps is necessary.