This paper presents a local spatio-temporal descriptor for action recognition from depth video sequences which is capable of distinguishing similar actions as well as coping with different speeds of actions. This descriptor is based on three processing stages. In the first stage, the shape and motion cues are captured from a weighted depth sequence by temporally overlapped depth segments, leading to three improved depth motion maps (DMMs) compared to previously introduced DMMs. In the second stage, the improved DMMs are partitioned into dense patches, from which the local binary patterns histogram features are extracted to characterize local rotation invariant texture information. In the final stage, a Fisher kernel is used for generating a compact feature representation, which is then combined with a kernel-based extreme learning machine (ELM) classifier. The developed solution is applied to five public domain datasets and is extensively evaluated. The results obtained demonstrate the effectiveness of this solution as compared to the existing approaches.