Hi Huggingface community, I encountered the “no space left on device” error when a sagemaker training job tries to download a large model from huggingface hub. To reproduce this issue, simply replace the model id with “EleutherAI/gpt-neox-20b” in this notebook . Both g5.8xlarge and p4d.24xlarge insta…

@philschmid Thanks for the suggestion. I found that I need to also set os.environ['HF_HOME'] = '/tmp' to make it work.

"no space left on device" when downloading a large model for the Sagemaker training job

Tom June 18, 2023, 12:37am 2

@ philschmid Calling the AWS hero for help. Much appreciated!

Topic		Replies	Views
SageMaker OS Error No Space Left On Device while trying to train Falcon40B Amazon SageMaker	3	1364	August 24, 2023
"No space left on device" when using HuggingFace + SageMaker Amazon SageMaker	39	26173	October 10, 2023
"OS Errorr: No space left on device" when trying to load a trained model from S3 Amazon SageMaker	1	1380	December 28, 2023
Sagemaker endpoint - no space left on device with large models Amazon SageMaker	3	4974	February 21, 2023
Llama3-8B-Instruct finetuning - No space left on device error Models	3	758	July 20, 2024