This paper proposes a novel design of cooperative non-orthogonal layered multicast multiple access in a heterogeneous network, where the information is encoded into the messages of high-priority (HP) and low-priority (LP). Two types
of multicast users coexist in the network: 1) regular users (RUs), which are located far away from the base-station (BS) and expect to decode only the HP message (due to the weak channels); 2) advanced users (AUs), which are located close to the BS and expect to decode both HP and LP messages. To improve the
reliability of layered multicast, we consider that the successful AUs (those AUs who successfully decode the HP and LP messages) serve as potential relays to assist other AUs/RUs. Based on this idea, two novel cooperation strategies are proposed for different cases of channel information availability. For each proposed strategy, we derive closed-form exact outage probabilities of AUs and
RUs, and then further analyze their diversity orders. Moreover, considering that the layered multicast is outage-constrained, we theoretically evaluate the energy consumption of both strategies and demonstrate their energy saving gains over the direct non-orthogonal multiple access for layered multicast. Finally, our
theoretical analysis is verified by numerical results, and the advantages of the proposed strategies are also demonstrated.