deepspeed,torch训练遇到:exits with return code = -8
·
deepspeed,torch训练遇到:exits with return code = -8
通过以下代码
import faulthandler
faulthandler.enable()
发现错误是:
Fatal Python error: Floating point exception
一路找终于找到了解决方案,感谢大神!
https://www.cnblogs.com/lhx9527/p/18780404
我的环境是:
nvcc 12.2, torch 2.5.1+cu121, deepspeed 0.18.6, h20
根据大神的解决方案:
pip3 install nvidia-cublas-cu12==12.3.4.1
export LD_LIBRARY_PATH=/opt/conda/lib/python3.8/site-packages/nvidia/cublas/lib/(自己的路径)
成功解决!
更多推荐



所有评论(0)