While analysing the GPU memory footprint of a few models ‘prepared’ via accelerator.prepare(), I found the models occupying more than twice the GPU memory compared to normally loaded models on device. What can be the reason for this inflated memory footprint? I am using a single V100 32 GB GPU and …

Inflated GPU memory footprint of model prepared via accelerate

varadhbhatnagar September 15, 2023, 4:40am 5

1 Like

Topic		Replies	Views
Data Parallel Multi GPU Inference 🤗Accelerate	9	4823	September 15, 2023
Accelerator OOM 🤗Accelerate	2	1321	July 5, 2023
`Accelerator.prepare` utilize only one GPU instead of all the 8 available GPUs and raises "CUDA out of memory" 🤗Accelerate	3	2914	July 19, 2024
Loading a HF Model in Multiple GPUs and Run Inferences in those GPUs 🤗Accelerate	10	9866	October 16, 2024
Docs Clarification: Is prepare() inefficient for models that are frozen? 🤗Accelerate	0	211	January 22, 2024