Let G be a reductive group over a field k of characteristic ≠2, let g=Lie(G), let θ be an involutive automorphism of G and let g=k⊕p be the associated symmetric space decomposition. For the case of a ground field of characteristic zero, the action of the isotropy group G^θ on p is well understood, since the well-known paper of Kostant and Rallis [B. Kostant, S. Rallis, Orbits and representations associated with symmetric spaces, Amer. J. Math. 93 (1971) 753–809]. Such a theory in positive characteristic has proved more difficult to develop. Here we use an approach based on some tools from geometric invariant theory to establish corresponding results in (good) positive characteristic.

Among other results, we prove that the variety N of nilpotent elements of p has a dense open orbit, and that the same is true for every fibre of the quotient map p→p//G^θ. However, we show that the corresponding statement for G, conjectured by Richardson, is not true. We provide a new, (mostly) calculation-free proof of the number of irreducible components of N, extending a result of Sekiguchi for k=C. Finally, we apply a theorem of Skryabin to describe the infinitesimal invariants k[p]^k.