This paper is dedicated to studying the following elliptic system of Hamiltonian type: $$\left\{ \begin{ array}{ ll} -\varepsilon^2\triangle u+u+V(x)v=Q(x)F_{ v}(u, v), \ \ \ \ x\in \mathbb{ R}^N,\\ -\varepsilon^2\triangle v+v+V(x)u=Q(x)F_{ u}(u, v), \ \ \ \ x\in \mathbb{ R}^N,\\ |u(x)|+|v(x)| \rightarrow 0, \ \ \mbox{ as} \ |x|\rightarrow \infty, \end{ array}\right. $$ where $N\ge 3$, $V, Q\in \mathcal{ C}(\mathbb{ R}^N, \mathbb{ R})$, $V(x)$ is allowed to be sign-changing and $\inf Q > 0$, and $F\in \mathcal{ C}^1(\mathbb{ R}^2, \mathbb{ R})$ is superquadratic at both $0$ and infinity but subcritical. Instead of the reduction approach used in [Calc Var PDE, 2014, 51: 725-760], we develop a more direct approach -- non-Nehari manifold approach to obtain stronger conclusions but under weaker assumptions than these in [Calc Var PDE, 2014, 51: 725-760]. We can find an $\varepsilon_0>0$ which is determined by terms of $N, V, Q$ and $F$, then we prove the existence of a ground state solution of Nehari-Pankov type to the coupled system for all $\varepsilon\in (0, \varepsilon_0]$.
Categories: math.AP