I submitted an Apache Spark application to an Amazon EMR cluster. The application fails with a 'no space left on device' stage failure like this:
Short Description
Spark uses local disks on the core and task nodes to store intermediate data. If the disks run out of space, the job fails with a 'no space left on device' error. Use one of the following methods to resolve this error: Live wallpaper for mac.
- Add more Amazon Elastic Block Store (Amazon EBS) capacity.
- Add more Spark partitions.
- Use a bootstrap action to dynamically scale up storage on the core and task nodes. For more information and a recommended bootstrap action script, see Dynamically scale up storage on Amazon EMR clusters.
'no space left on the device' could also mean that the partition is out of inodes. Use df -i to check inode usage. Edit: Each file, directory, and symbolic link requires one inode. So the idea is to remove some files from the / partition. It doesn't matter how large the files are. AFAIK, two reasons may cause the 'No space left on device' error: (1) The physical disk has been used up. Run df -hk to check whether there is any free space left. (2) There is enough disk space, but you have reached the maximum number that a file system can hold, because of huge numbers of files on the disk.
Resolution
Add more EBS capacity
Such difference between the output of du -sh and df -h may happen if some large file has been deleted, but is still opened by some process. Check with the command lsof grep deleted to see which processes have opened descriptors to deleted files. You can restart the process and the space will be freed. Such difference between the output of du -sh and df -h may happen if some large file has been deleted, but is still opened by some process. Check with the command lsof grep deleted to see which processes have opened descriptors to deleted files. You can restart the process and the space will be freed. This error is triggered if there is no space left in the directory which contains the Confluence Home folder. Usually happens in large instances with a high number of pages being created.
For new clusters: use larger EBS volumes
Launch an Amazon EMR cluster and choose an Amazon Elastic Compute Cloud (Amazon EC2) instance type with larger EBS volumes. For more information about the amount of storage and number of volumes allocated for each instance type, see Default EBS Storage for Instances.
For running clusters: add more EBS volumes
1. If larger EBS volumes don't resolve the problem, attach more EBS volumes to the core and task nodes.
2. Format and mount the attached volumes. Be sure to use the correct disk number (for example, /mnt1 or /mnt2 instead of /data).
No Space Left On Device Error Detection
3. Connect to the node using SSH.
4. Create a /mnt2/yarn directory, and thenset ownership of the directory to the YARN user:
5. Add the /mnt2/yarn directory inside the yarn.nodemanager.local-dirs property of /etc/hadoop/conf/yarn-site.xml. Example:
Raag darbari tv serial youtube telugu. 6. Restart the NodeManager service:
Add more Spark partitions
Depending on how many core and task nodes are in the cluster, consider increasing the number of Spark partitions. Use the following Scala code to add more Spark partitions:
Related Information
How can I troubleshoot stage failures in Spark jobs on Amazon EMR?
Anything we could improve?
Need more help?
No Space Left On Device Error Download
When you try to install a VMware ESXi Update using esxcli, the upgrade fails with the following error message:
'[Errno 28] No space left on device'
The problem is caused by ESXi not having enough free space available to extract the installation packages. This article explains how to solve the issue by enabling swapping to a Datastore.
2. Format and mount the attached volumes. Be sure to use the correct disk number (for example, /mnt1 or /mnt2 instead of /data).
No Space Left On Device Error Detection
3. Connect to the node using SSH.
4. Create a /mnt2/yarn directory, and thenset ownership of the directory to the YARN user:
5. Add the /mnt2/yarn directory inside the yarn.nodemanager.local-dirs property of /etc/hadoop/conf/yarn-site.xml. Example:
Raag darbari tv serial youtube telugu. 6. Restart the NodeManager service:
Add more Spark partitions
Depending on how many core and task nodes are in the cluster, consider increasing the number of Spark partitions. Use the following Scala code to add more Spark partitions:
Related Information
How can I troubleshoot stage failures in Spark jobs on Amazon EMR?
Anything we could improve?
Need more help?
No Space Left On Device Error Download
When you try to install a VMware ESXi Update using esxcli, the upgrade fails with the following error message:
'[Errno 28] No space left on device'
The problem is caused by ESXi not having enough free space available to extract the installation packages. This article explains how to solve the issue by enabling swapping to a Datastore.
GUI Method
You do not need a vCenter to enable host swapping. It can be done from the ESXi Host Client.
- Open ESXi UI (https://[ESXi-IP-Address]/ui/)
- Navigate to Host > Manage > System > Swap
- Click Edit settings
- Enable Swapping and select a datastore (ds1 in that example)
- Click Save
- Changes are active immediately, there is no need to reboot the host.
CLI Method
You can also activate Datastore Swapping with the esxcli sched swap system command:
Related posts:
