We study a Rayleigh-Faber-Krahn inequality for regional fractional Laplacian operators. In particular, we show that there exists a compactly supported nonnegative Sobolev function $u_0$ that attains the infimum (which will be a positive real number) of the set \[ \left\{ \int\int_{ \{ u > 0\}\times\{ u>0\}} \frac{ |u(x) - u(y)|^2}{ |x - y|^{ n + 2 \sigma}}d x d y : u \in \mathring H^\sigma(\mathbb{ R}^n), \int_{ \mathbb{ R}^n} u^2 = 1, |\{ u > 0 \}| \leq 1\right\}. \] Unlike the corresponding problem for the usual fractional Laplacian, where the domain of the integration is $\mathbb{ R}^n \times \mathbb{ R}^n$, symmetrization techniques may not apply here. Our approach is instead based on the direct method and new a priori diameter estimates. We also present several remaining open questions concerning the regularity and shape of the minimizers, and the form of the Euler-Lagrange equations.
Categories: math.AP