Mobile Crowdsensing (MCS) utilizes sensing data collected from users' mobile devices (MDs) to provide high-quality and personalized services, such as traffic monitoring, weather prediction, and service recommendation. In return, users who participate in crowdsensing (i.e., MCS participants) get payment from cloud service providers (CSPs) according to the quality of their shared data. Therefore, it is vital to guarantee the security of payment transactions between MCS participants and CSPs. As a distributed ledger, the blockchain technology is effective in providing secure transactions among users without a trusted third party, which has found many promising applications such as virtual currency and smart contract. In a blockchain, the proof-of-work (PoW) executed by users plays an essential role in solving consensus issues. However, the complexity of PoW severely obstructs the application of blockchain in MCS due to the limited computational capacity of MDs. To solve this issue, we propose a new framework based on Deep Reinforcement Learning (DRL) for offloading computation-intensive tasks of PoW to edge servers in a blockchain-based MCS system. The proposed framework can be used to obtain the optimal offloading policy for PoW tasks under the complex and dynamic MCS environment. Simulation results demonstrate that our method can achieve a lower weighted cost of latency and power consumption compared to benchmark methods.