We construct and study Donaldson-Thomas invariants counting orthogonal and symplectic objects in linear categories, which are a generalization of the usual Donaldson-Thomas invariants from the structure groups $\mathrm{ GL} (n)$ to the groups $\mathrm{ O} (n)$ and $\mathrm{ Sp} (2n)$, and a special case of the intrinsic Donaldson-Thomas theory developed by the author, Halpern-Leistner, Ib\'a\~nez N\'u\~nez, and Kinjo. Our invariants are defined using the motivic Hall algebra and its orthosymplectic analogue, the motivic Hall module. We prove wall-crossing formulae for our invariants, which relate the invariants with respect to different stability conditions. As examples, we define Donaldson-Thomas invariants counting orthogonal and symplectic perfect complexes on a Calabi-Yau threefold, and Donaldson-Thomas invariants counting self-dual representations of a self-dual quiver with potential. In the case of quivers, we compute the invariants explicitly in some cases. We also define a motivic version of Vafa-Witten invariants counting orthogonal and symplectic Higgs complexes on a class of algebraic surfaces.
Categories: math.AG hep-th