A user had exfat-utils installed and tried to use GParted to create an
exfat file system. GParted ran this command but it failed:
# mkfs.exfat -L '' '/dev/sdb1'
mkexfatfs 1.3.0
mkfs.exfat: invalid option -- 'L'
Usage: mkfs.exfat [-i volume-id] [-n label] [-p partition-first-sector] [-s sectors-per-cluster] [-V] <device>
The problem is that both exfat-utils and exfatprogs packages provide
mkfs.exfat and fsck.exfat commands but they have incompatible command
line options and GParted is programmed for exfatprogs. So far GParted
just checks the executable exists, hence the mis-identification.
Reported version of exfat-utils commands:
$ mkfs.exfat -V 2> /dev/null
mkexfatfs 1.3.0
Copyright (C) 2011-2018 Andrew Nayenko
$ fsck.exfat -V 2> /dev/null
exfatfsck 1.3.0
Copyright (C) 2011-2018 Andrew Nayenko
Reported versions of exfatprogs commands:
$ mkfs.exfat -V 2> /dev/null
exfatprogs version : 1.0.4
$ fsck.exfat -V 2> /dev/null
exfatprogs version : 1.0.4
Fix this by only enabling exfat support also when the version string of
each command starts "exfatprogs version". Note that this extra checking
is not needed for tune.exfat because only exfatprogs provides that
executable.
Closes#137 - Creating exfat partition with a label fails with error
This method is now only called from one location in the code so put it's
two lines of code there.
Closes#131 - GParted hangs when non-named device is hung
Now we always want to run blkid naming all paths, ensure the FS_Info
cache is explicitly loaded first. Report an error if not done so and
remove the cache loading code from running blkid without naming all
paths. Fewer code paths to consider and reason about.
Closes#131 - GParted hangs when non-named device is hung
Again on Fedora 31 with a slightly different disk layout to the previous
commit. sdb is partitioned with 1 empty partition and sdc remains
completely empty:
# lsblk -o name,maj:min,rm,size,ro,type,fstype,label,mountpoint
NAME MAJ:MIN RM SIZE RO TYPE FSTYPE LABEL MOUNTPOINT
sda 8:0 0 20G 0 disk
|-sda1 8:1 0 1G 0 part ext4 /boot
\-sda2 8:2 0 19G 0 part LVM2_member
|-fedora-root 253:0 0 17G 0 lvm ext4 /
\-fedora-swap 253:1 0 2G 0 lvm swap [SWAP]
sdb 8:16 0 8G 0 disk
\-sdb1 8:17 0 1G 0 part
sdc 8:32 0 8G 0 disk
sr0 11:0 1 1024M 0 rom
# blkid -v
blkid from util-linux 2.34 (libblkid 2.34.0, 14-Jun-2019)
# blkid /dev/sda /dev/sda1 /dev/sda2 /dev/sdb /dev/sdb1 /dev/sdc
/dev/sda: PTUUID="5012fb1f" PTTYPE="dos"
/dev/sda1: UUID="3cd48816-7817-4636-9fec-5f1afe76c1b2" TYPE="ext4" PARTUUID="5012fb1f-01"
/dev/sda2: UUID="PH94ej-C8xU-bnMJ-UIh8-ZimI-4B7f-dHlZxh" TYPE="LVM2_member" PARTUUID="5012fb1f-02"
/dev/sdb: PTUUID="1d120b57" PTTYPE="dos"
/dev/sdb1: PARTUUID="1d120b57-01"
Stracing GParted shows these executions of blkid:
# strace -f -q -bexecve -eexecve ./gpartedbin 2>&1 1> /dev/null | egrep -v 'ENOENT|SIGCHLD'
...
[pid 160040] execve("/usr/sbin/blkid", ["blkid", "/dev/sda", "/dev/sda1", "/dev/sda2", "/dev/sdb", "/dev/sdb1", "/dev/sdc"], 0xa4e1b0 /* 32 vars */ <detached ...>
[pid 160041] execve("/usr/sbin/blkid", ["blkid", "/dev/sdc"], 0xa4e1b0 /* 32 vars */ <detached ...>
...
On Fedora 31 with blkid from util-linux 2.34 it reports information for
sdb (partitioned drive) and sdb1 (empty partition) with only no
information for sdc (empty whole disk drive). Hence no FS_Info cache
entry and re-execution of blkid just for sdc.
On older CentOS 7 with the same disk layout blkid reports this:
# blkid -v
blkid from util-linux 2.23.2 (libblkid 2.23.0, 25-Apr-2013)
# blkid /dev/sda /dev/sda1 /dev/sda2 /dev/sdb /dev/sdb1 /dev/sdc
/dev/sda: PTTYPE="dos"
/dev/sda1: UUID="e7d559e4-3e1d-4fbc-b034-3fdeb498f44d" TYPE="xfs"
/dev/sda2: UUID="B7ODFx-BfTE-hq7N-UlrF-f5ML-CPRe-klSy26" TYPE="LVM2_member"
/dev/sdb: PTTYPE="dos"
And stracing GParted shows these executions of blkid:
# strace -f -q -bexecve -eexecve ./gpartedbin 2>&1 1> /dev/null | egrep -v 'ENOENT|SIGCHLD'
...
[pid 1889] execve("/sbin/blkid", ["blkid", "/dev/sda", "/dev/sda1", "/dev/sda2", "/dev/sdb", "/dev/sdb1", "/dev/sdc"], 0x10b8b10 /* 26 vars */ <detached ...>
[pid 1890] execve("/sbin/blkid", ["blkid", "/dev/sdb1"], 0x10b8b10 /* 26 vars */ <detached ...>
[pid 1891] execve("/sbin/blkid", ["blkid", "/dev/sdc"], 0x10b8b10 /* 26 vars */ <detached ...>
...
This time on CentOS 7 with blkid from util-linux 2.23.2 it reports
information for only sdb (partitioned drive), but not sdb1 (empty
partition) or sdc (empty whole disk drive). Hence no FS_info cache
entries and re-execution of blkid for both sdb1 and sdc.
GParted needs blkid identification of file system images, LVM Logical
Volumes or any other partitions named on the command line which it
wouldn't normally scan [1]. Now every name of interest is passed to
blkid, additional executions of blkid won't get any extra information
and are redundant. Therefore remove this unnecessary code.
Note that these last 2 commits remove creation of "blank" cache entries
(just block special with blank fstype and other attributes) when blkid
reports no information for a particular path. Those entry were needed
to suppress unnecessary additional execution of blkid. However now that
blkid is only executed once (excluding querying the label) this is no
longer necessary. All the getter functions return suitable blank values
when no cache entry is found.
[1] e8f0504b13
Make sure that FS_Info cache is loaded for all named paths (#787181)
Closes#131 - GParted hangs when non-named device is hung
On Fedora 31 with this simple disk layout where both sdb and sdc are
completely empty:
# lsblk -o name,maj:min,rm,size,ro,type,fstype,label,mountpoint
NAME MAJ:MIN RM SIZE RO TYPE FSTYPE LABEL MOUNTPOINT
sda 8:0 0 20G 0 disk
|-sda1 8:1 0 1G 0 part ext4 /boot
\-sda2 8:2 0 19G 0 part LVM2_member
|-fedora-root 253:0 0 17G 0 lvm ext4 /
\-fedora-swap 253:1 0 2G 0 lvm swap [SWAP]
sdb 8:16 0 8G 0 disk
sdc 8:32 0 8G 0 disk
sr0 11:0 1 1024M 0 rom
# blkid /dev/sda /dev/sda1 /dev/sda2 /dev/sdb /dev/sdc
/dev/sda: PTUUID="5012fb1f" PTTYPE="dos"
/dev/sda1: UUID="3cd48816-7817-4636-9fec-5f1afe76c1b2" TYPE="ext4" PARTUUID="5012fb1f-01"
/dev/sda2: UUID="PH94ej-C8xU-bnMJ-UIh8-ZimI-4B7f-dHlZxh" TYPE="LVM2_member" PARTUUID="5012fb1f-02"
Stracing GParted shows extra executions of blkid:
# strace -f -q -bexecve -eexecve ./gpartedbin 2>&1 1> /dev/null | egrep -v 'ENOENT|SIGCHLD'
...
[pid 7659] execve("/usr/sbin/blkid", ["blkid", "/dev/sda", "/dev/sda1", "/dev/sda2", "/dev/sdb", "/dev/sdc"], 0x1d300f0 /* 32 vars */ <detached ...>
[pid 7660] execve("/usr/sbin/blkid", ["blkid", "/dev/sdb"], 0x1d300f0 /* 32 vars */ <detached ...>
[pid 7661] execve("/usr/sbin/blkid", ["blkid", "/dev/sdc"], 0x1d300f0 /* 32 vars */ <detached ...>
...
blkid is only run again for sdb and sdc, not sda, because blkid didn't
report anything for them from the first execution. GParted needs blkid
identification of whole disk devices to ensure that ISO9660 images on
whole disk devices are correctly identified [1]. Now the first run of
blkid passes all the device names, so this additional execution of blkid
won't get any extra information and is redundant. Therefore remove this
unnecessary code.
[1] b2190372d0
Ensure blkid FS_Info cache has entries for all whole disk devices
(#771244)
Closes#131 - GParted hangs when non-named device is hung
A user reported that GParted would hang at "scanning all devices...",
when a fully working disk was named on the command line, but another
device on the machine was hung.
This can be replicated like this:
(on Ubuntu 20.04 LTS for it's NBD support)
1. Export and import NBD:
# truncate -s 1G /tmp/disk-1G.img
# nbd-server -C /dev/null 9000 /tmp/disk-1G.img
# nbd-client localhost 9000 /dev/nbd0
2. Hang the NBD server and therefore /dev/nbd0:
# killall -STOP nbd-server
3. Run GParted:
$ gparted /dev/sda
Tracing GParted shows that execution of blkid never returns.
# strace -f -tt -q -bexecve -eexecve ./gpartedbin 2>&1 1> /dev/null | fgrep -v ENOENT
...
[pid 37823] 13:56:24.814139 execve("/usr/sbin/mkudffs", ["mkudffs", "--help"], 0x55e2a3f2d230 /* 20 vars */ <detached ...>
[pid 37814] 13:56:24.829246 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=37823, si_uid=0, si_status=1, si_utime=0, si_stime=0} ---
[pid 37825] 13:56:25.376796 execve("/usr/sbin/blkid", ["blkid", "-v"], 0x55e2a3f2d230 /* 20 vars */ <detached ...>
[pid 37824] 13:56:25.380824 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=37825, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
[pid 37826] 13:56:25.402512 execve("/usr/sbin/blkid", ["blkid"], 0x55e2a3f2d230 /* 20 vars */ <detached ...>
Tracking of blkid shows that it hangs on either the open of or first
read from /dev/nbd0.
# strace blkid
...
lstat("/dev", {st_mode=S_IFDIR|0755, st_size=4560, ...}) = 0
lstat("/dev/nbd0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x2b, 0), ...}) = 0
stat("/dev/nbd0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x2b, 0), ...}) = 0
lstat("/dev", {st_mode=S_IFDIR|0755, st_size=4560, ...}) = 0
lstat("/dev/nbd0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x2b, 0), ...}) = 0
access("/dev/nbd0", F_OK) = 0
stat("/dev/nbd0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x2b, 0), ...}) = 0
openat(AT_FDCWD, "/sys/dev/block/43:0", O_RDONLY|O_CLOEXEC) = 4
openat(4, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(4) = 0
openat(AT_FDCWD, "/dev/nbd0", O_RDONLY|O_CLOEXEC
Clean up:
1. Resume NBD server:
# killall -CONT nbd-server
2. Delete NBD setup:
# nbd-client -d /dev/nbd0
# killall nbd-server
# rm /tmp/disk-1G.img
Fix this by making GParted specify the whole disk device and partition
names that it is interested in to blkid, rather than letting blkid scan
and report all block devices. Do this both when GParted determines the
devices for itself and when they are named on the command line.
Also update example blkid command output being parsed and cache value
with this change to how blkid is executed.
Closes#131 - GParted hangs when non-named device is hung
GParted already always reads /proc/partitions for whole disk device
names no matter whether it uses whole disk devices named on the command
line, from /proc/partitions or from libparted. As /proc/partitions
lists all the block devices that the kernel knows about, and therefore
all the possible ones blkid could probe, so use it to provide partition
names and device to partition mapping. See code comments for more
details about the assumptions the /proc/partition parsing code makes and
the fact that these are confirmed by examining the Linux kernel source.
This commit just adds debugging to print the existing vector of
validated devices GParted shows in the UI and the vector with all
partitions added, ready for but not yet passed to blkid.
# ./gpartedbin
...
DEBUG: device_paths=["/dev/sda","/dev/sdb"]
DEBUG: device_and_partition_paths=["/dev/sda","/dev/sda1","/dev/sda2","/dev/sdb","/dev/sdb1"]
Also demonstrating that this continues to support named devices,
including file system image files [1].
# truncate -s 256M /tmp/ext4.img
# mkfs.ext4 /tmp/ext4.img
# ./gpartedbin /dev/sda /tmp/ext4.img
...
DEBUG: device_paths=["/dev/sda","/tmp/ext4.img"]
DEBUG: device_and_partition_paths=["/dev/sda","/dev/sda1","/dev/sda2","/tmp/ext4.img"]
[1] e8f0504b13
Make sure that FS_Info cache is loaded for all named paths (#787181)
Closes#131 - GParted hangs when non-named device is hung
Put whole disk device name matching code into a helper function to make
the /proc/partition parsing code easier to understand.
Closes#131 - GParted hangs when non-named device is hung
Now FS_Info::load_cache() and ::load_cache_for_paths() are nearly next
to each other, merge them together to simplify the code a little. This
makes the special case to ensure that file system images named on the
command line were queried by blkid and loaded into the FS_Info cache [1]
become the normal cache loading method. Already passing all discovered
or named devices to load_cache_for_paths() is also a step on the way to
doing it for all devices and partitions of interest.
Just need to ensure that load_cache_for_paths() always loads the cache
as load_cache() did, rather than only when it hadn't already been
loaded. Otherwise GParted will only ever run blkid and load the cache
once at startup and not on each refresh.
[1] e8f0504b13
Make sure that FS_Info cache is loaded for all named paths (#787181)
Closes#131 - GParted hangs when non-named device is hung
PATCHSET OVERVIEW
A user reported that GParted would hang at "scanning all devices...",
when a fully working disk was named on the command line, but another
device on the machine was hung.
This can be replicated like this:
(on Ubuntu 20.04 LTS for it's NBD support)
1. Export and import NBD:
# truncate -s 1G /tmp/disk-1G.img
# nbd-server -C /dev/null 9000 /tmp/disk-1G.img
# nbd-client localhost 9000 /dev/nbd0
2. Hang the NBD server and therefore /dev/nbd0:
# killall -STOP nbd-server
3. Run GParted:
$ gparted /dev/sda
Tracing GParted shows that execution of blkid never returns.
# strace -f -tt -q -bexecve -eexecve /usr/sbin/gpartedbin 2>&1 1> /dev/null | fgrep -v ENOENT
...
[pid 37823] 13:56:24.814139 execve("/usr/sbin/mkudffs", ["mkudffs", "--help"], 0x55e2a3f2d230 /* 20 vars */ <detached ...>
[pid 37814] 13:56:24.829246 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=37823, si_uid=0, si_status=1, si_utime=0, si_stime=0} ---
[pid 37825] 13:56:25.376796 execve("/usr/sbin/blkid", ["blkid", "-v"], 0x55e2a3f2d230 /* 20 vars */ <detached ...>
[pid 37824] 13:56:25.380824 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=37825, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
[pid 37826] 13:56:25.402512 execve("/usr/sbin/blkid", ["blkid"], 0x55e2a3f2d230 /* 20 vars */ <detached ...>
Tracing of blkid shows that it hangs on either the open of or first
read from /dev/nbd0.
# strace blkid
...
lstat("/dev", {st_mode=S_IFDIR|0755, st_size=4560, ...}) = 0
lstat("/dev/nbd0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x2b, 0), ...}) = 0
stat("/dev/nbd0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x2b, 0), ...}) = 0
lstat("/dev", {st_mode=S_IFDIR|0755, st_size=4560, ...}) = 0
lstat("/dev/nbd0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x2b, 0), ...}) = 0
access("/dev/nbd0", F_OK) = 0
stat("/dev/nbd0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x2b, 0), ...}) = 0
openat(AT_FDCWD, "/sys/dev/block/43:0", O_RDONLY|O_CLOEXEC) = 4
openat(4, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(4) = 0
openat(AT_FDCWD, "/dev/nbd0", O_RDONLY|O_CLOEXEC
Clean up:
1. Resume NBD server:
# killall -CONT nbd-server
2. Delete NBD setup:
# nbd-client -d /dev/nbd0
# killall nbd-server
# rm /tmp/disk-1G.img
Going to fix this by making GParted specify the device and partition
names that it is interested in to blkid, rather than letting blkid scan
and report all block devices. Do this both when GParted determines the
devices for itself and when they are named on the command line.
THIS PATCH
Move the loading and initialising of caches used during content
discovery to after device and partition discovery and just before
content discovery. Just makes the code ready for the next change.
Closes#131 - GParted hangs when non-named device is hung
Libparted only allows selection of the partition type indirectly by
specifying the type of the file system it will contain [1] and so far
doesn't know about the exFAT file system. Therefore when GParted is
creating a new exFAT partition, it gets the GParted default of 83
(Linux file system) on MBR partition tables.
Example operation details:
Create Primary Partition #1 (exfat, 512.00 MiB) on /dev/sdb
* create empty partition
* clear old file system signatures in /dev/sdb1
* set partition type on /dev/sdb1
new partition type: ext2
* create new exfat file system
fdisk report:
# fdisk -l /dev/sdb
Disk /dev/sdb: 8 GiB, 8589934592 bytes, 16777216 sectors
Disk model: VBOX HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa2aab629
Device Boot Start End Sectors Size Id Type
/dev/sdb1 2048 1050623 1048576 512M 83 Linux
However the "exFAT file system specification" says:
https://docs.microsoft.com/en-us/windows/win32/fileio/exfat-specification
"10.2 Partition Tables
To ensure interoperability of exFAT volumes in a broad set of usage
scenarios, implementations should use partition type 07h for MBR
partitioned storage and partition GUID
{EBD0A0A2-B9E5-4433-87C0-68B6B72699C7} for GPT partitioned storage.
"
Fix this.
[1] ped_partition_new(..., const PedFileSystemType* fs_type, ...)
https://www.gnu.org/software/parted/api/group__PedPartition.html#g2f94ca75880f9e0c3ce57f7a4b72faf5
ped_partition_set_system(..., const PedFileSystemType* fs_type)
https://www.gnu.org/software/parted/api/group__PedPartition.html#g2f94ca75880f9e0c3ce57f7a4b72faf5Closes!30 - Add exFAT support
With exfatprogs (https://github.com/exfatprogs/exfatprogs) installed the
following operations on exFAT file systems are supported:
- Creation
- Checking
- Labelling
As of the current exfatprogs 1.0.4 the following are not supported:
- Reading usage
- Resizing
- Updating the UUID
Closes!30 - Add exFAT support
To better reflect that they represent the version of mke2fs executable
from the e2fsprogs package, even though the executable is called as
mkfs.ext4.
$ ls -il /sbin/mke2fs /sbin/mkfs.ext*
1978670 -rwxr-xr-x. 4 root root 96384 Aug 9 2019 /sbin/mke2fs
1978670 -rwxr-xr-x. 4 root root 96384 Aug 9 2019 /sbin/mkfs.ext2
1978670 -rwxr-xr-x. 4 root root 96384 Aug 9 2019 /sbin/mkfs.ext3
1978670 -rwxr-xr-x. 4 root root 96384 Aug 9 2019 /sbin/mkfs.ext4
$ mkfs.ext4 -V
mke2fs 1.42.9 (28-Dec-2013)
Using EXT2FS Library version 1.42.9
$ mke2fs -V
mke2fs 1.42.9 (28-Dec-2013)
Using EXT2FS Library version 1.42.9
With removal of support for RHEL / CentOS 5 and it's e4fsprogs package
[1][2] it is no longer necessary to accept:
mke4fs 1.41.12 (17-May-2010)
only:
mke2fs 1.42.9 (28-Dec-2013)
[1] 6c4ab5dc28
Remove checks for e4fsprogs commands (#794253)
[2] de6e70d933
Simplify ext2::get_filesystem_support() with regard ext4 support (#794253)
This earlier commit [1] from 2013 recognised the new names for programs
in dosfstools >= 3.0.18, specifically mkfs.fat and fsck.fat. Now that
the oldest supported distributions use dosfstools >= 3.0.18 it is no
longer necessary to support using the old names of mkdosfs and dosfsck,
so remove that code.
Oldest supported Dosfstools
distributions Version
Debian 8 3.0.27
RHEL / CentOS 7 3.0.20
SLES 12 3.0.26
Ubuntu 14.04 LTS 3.0.26
[1] 1ae03dee95
Recognise new dosfstools program names (#704629)
Closes!57 - Raise minimum support dosfstools to 3.0.18 released
2013-06-06
A user reported that formatting a whole disk device with a file system
failed like this:
Format /dev/sdd as ext4 (ERROR)
+ calibrate /dev/sdd (SUCCESS)
path: /dev/sdd (device)
start: 0
end: 15633407
size: 15633408 (7.45 GiB)
+ clear old file system signatures in /dev/sdd (SUCCESS)
+ write 512.00 KiB of zeros at byte offset 0 (SUCCESS)
+ write 4.00 KiB of zeros at byte offset 67108864 (SUCCESS)
+ write 512.00 KiB of zeros at byte offset 8003780608 (SUCCESS)
+ write 4.00 KiB of zeros at byte offset 8004239360 (SUCCESS)
+ write 8.00 KiB of zeros at byte offset 8004296704 (SUCCESS)
+ flush operating system cache of /dev/sdd (SUCCESS)
+ create new ext4 file system (ERROR)
+ mkfs.ext4 -F -O ^64bit -L '' '/dev/sdd' (ERROR)
mke2fs 1.44.1 (24-Mar-2018)
/dev/sdd is apparently in use by the system; will not make a filesystem here!
Opening the whole disk block device exclusively causes mkfs.ext4 to
report that error like this:
# python
>>> import os
>>> f = os.open('/dev/sdb',os.O_RDONLY|os.O_EXCL)
>>> ^Z
[1]+ Stopped python
# mkfs.ext4 -F -O ^64bit -L '' '/dev/sdb'
mke2fs 1.42.9 (28-Dec-2013)
/dev/sdb is apparently in use by the system; will not make a filesystem here!
# echo $?
1
I have not been able to reproduce this error, but with debugging and
sleeping in GParted, stracing GParted and using 'udevadm monitor' to
watch udev events the following sequence of events is seen:
gparted |format(partition, operationdetail)
gparted | erase_filesystem_signatures(partition, operationdetail)
gparted | get_device(device_path="/dev/sdb", lp_device, flush=false)
gparted | ped_device_get("/dev/sdb")
libparted | open("/dev/sdb", O_RDONLY) = 11
libparted | close(11)
gparted | ped_device_open(lp_device)
libparted | open("/dev/sdb", O_RDWR) = 11
gparted | ped_device_sync(lp_device)
libparted | ioctl(11, BLKFLSBUF)
gparted | ped_device_close()
libparted | close(11)
udev(async)| KERNEL change /devices/.../sdb (block)
udev(async)| UDEV change /devices/.../sdb (block)
gparted | set_partition_type(partition, operationdetail)
gparted | create_filesystem(partition, operationdetail)
gparted | ext2::create(partition, operationdetail)
gparted | FileSystem::execute_command("mkfs.ext4 -F -O ^64bit -L '' '/dev/sdb')
So it is assumed that the processing of the udev change rule after
closing the block device in erase_filesystem_signatures() overlaps with
the execution mkfs.ext4 and causes the seen error. Fix by waiting for
those udev events to complete as was previously done by commits [1][2]
[3].
Also note that this is specific to creating file systems on and
formatting unpartitioned whole disk devices because set_partition_type()
is a no-operation. Where as on a partitioned device
set_partition_type() calls commit() which already waits for udev rules
to complete [3].
[1] 50c8924a8e4d9cc96a2ea45f13291114402affee
Wait for udev to recreate /dev/PTN entries when querying partition
FSs (!46)
[2] 4f6c312e3bc68cafb5e6035fd4a5b5bbbfcea992
Wait for udev change on /dev/DISK when querying whole device FS
(!46)
[3] 2f53876c0f
Wait for the kernel and udev to settle partitions for a second time
(#790418)
Closes#83 - /dev/sdd is apparently in use by the system; will not make
a filesystem here!
Name the structure member to 'fsname' used to store strings like "ext2"
etc. This is equivalent to what was previously done in this commit:
a9f08ddc7d
Rename local variable to fsname in get_filesystem() (#741430)
Closes!52 - Rename members and variables currently named 'filesystem'
Previously made this change:
175d27c55d
Rename enum FILESYSTEM to FSType
Now complete the renaming exercise of members and variables currently
named 'filesystem'.
Closes!52 - Rename members and variables currently named 'filesystem'
In GParted_Core::set_device_partitions() the partition path is being
queried from libparted. However this is done before the switch
statement on the type of the partition, so is called for all libparted
partition objects including PED_PARTITION_FREESPACE and
PED_PARTITION_METADATA ones. As libparted numbers these partition
objects as -1, it returns paths like "/dev/sda-1".
Additionally when using GParted, with it's default DMRaid handling, on a
dmraid started array this results in paths like
"/dev/mapper/isw_ecccdhhiga_MyArray-1" being passed to
is_dmraid_device() and make_path_dmraid_compatible(). Fortunately
make_path_dmraid_compatible() does nothing and returns the same name.
Call chain looks like:
GParted_Core::set_device_partitions()
get_partition_path(lp_partition)
// where:
// lp_partition->disk->dev->path = "/dev/mapper/isw_ecccdhhiga_MyArray"
// lp_partition->type == PED_PARTITION_FREESPACE |
// PED_PARTITION_METADATA
// ->num == -1
ped_partition_get_path(lp_partition)
return "/dev/mapper/isw_ecccdhhiga_MyArray-1"
dmraid.is_dmraid_supported()
dmraid.is_dmraid_device("/dev/mapper/isw_ecccdhhiga_MyArray-1")
return true
dmraid.make_path_dmraid_compatible("/dev/mapper/isw_ecccdhhiga_MyArray-1")
return "/dev/mapper/isw_ecccdhhiga_MyArray-1"
Fix by moving the get_partition_path() call inside the switch statement
so that it is only called for PED_PARTITION_NORMAL,
PED_PARTITION_LOGICAL and PED_PARTITION_EXTENDED partition types.
Relevant commits:
* 53c49349f7
Simplify logic in set_device_partitions method
* 81986c0990
Ensure partition path name is compatible with dmraid (#622217)
This is not strictly necessary as members are already recognised using
blkid since this commit earlier in the sequence "Recognise ATARAID
members (#75)". However it makes sure active members are recognised
even if blkid is not available and matches how file system detection
queries the SWRaid_Info module.
Closes#75 - Errors with GPT on RAID 0 ATARAID array
This matches how the array device is displayed as the mount point for
mdadm started ATARAID members by "Display array device as mount point of
mdadm started ATARAID members (#75)" earlier in this patchset.
Extend the DMRaid module member cache to save the array device name and
use as needed to display as the mount point.
Closes#75 - Errors with GPT on RAID 0 ATARAID array
Again this is to stop GParted allowing overwrite operations being
performed on an ATARAID member while the array is actively using the
member. This time for dmraid started arrays using the kernel DM (Device
Mapper) driver.
The DMRaid module already uses dmraid to report active array names:
# dmraid -sa -c
isw_ecccdhhiga_MyArray
To find active members in this array, (1) use udev to lookup the kernel
device name:
# udevadm info --query=name /dev/mapper/isw_ecccdhhiga_MyArray
dm-0
(2) list the member names exposed by the kernel DM driver through the
/sys file system.
# ls /sys/block/dm-0/slaves
sdc sdd
# ls -l /sys/block/dm-0/slaves
lrwxrwxrwx 1 root root 0 Nov 24 09:52 sdc -> ../../../../pci0000:00/0000:00:0d.0/ata3/host2/target2:0:0/2:0:0:0/block/sdc
lrwxrwxrwx 1 root root 0 Nov 24 09:52 sdc -> ../../../../pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdd
Closes#75 - Errors with GPT on RAID 0 ATARAID array
When an ATARAID member is inactive allow basic supported actions of
copy and move to be performed like with other recognised but only basic
supported types.
Closes#75 - Errors with GPT on RAID 0 ATARAID array
Since earlier commit "Display array device as mount point of mdadm
started ATARAID members (#75)" GParted allows attempting to unmout a
busy ATARAID member as if it was a file system. This is not a valid
thing to do, so disallow it.
Closes#75 - Errors with GPT on RAID 0 ATARAID array
This matches how other non-file systems are handled, by displaying the
access reference in the mount point column. For LVM Physical Volumes
the Volume Group name is displayed [1] and for an active Linux Software
RAID array the array device is displayed [2].
[1] 8083f11d84
Display LVM2 VGNAME as the PV's mount point (#160787)
[2] f6c2f00df7
Populate member mount point with SWRaid array device (#756829)
Closes#75 - Errors with GPT on RAID 0 ATARAID array
This stops GParted allowing overwrite operations (such as create
partition table or format with a whole device file system) being
performed on an ATARAID member while the array is actively using the
member.
Closes#75 - Errors with GPT on RAID 0 ATARAID array
The previous commit, made mdadm recognised IMSM and DDF type ATARAID
members get displayed as "linux-raid" (Linux Software RAID array
member). This was because of query method 1 in detect_filesystems().
Fix this now by exposing and using the fstype of the member from the
SWRaid_Info cache.
Closes#75 - Errors with GPT on RAID 0 ATARAID array
Since mdadm release 3.0 (2009-06-02) [1] it has also supported external
metadata formats IMSM (Intel Matrix Storage Manager) and DDF, previously
only managed by dmraid.
A number of distributions have switched to use mdadm and kernel MD
(Multiple Devices) driver for managing these Firmware / BIOS / ATARAID
arrays. These include: Fedora >= 14 [2], RHEL / CentOS >= 6 [3],
SLES >= 12 [4], Ubuntu >= 16.04 LTS.
Therefore additionally parse members in these ATARAID arrays included in
mdadm output, and when activated using the kernel MD driver, in file
/proc/mdstat. Add fstype to the SWRaid_Info cache records to
distinguish members apart. So far the rest of the GParted code
continues to treat all members as FS_LINUX_SWRAID. This will be
resolved in following commits.
Note that this in no way affects how GParted shows and partitions the
array device itself, even those managed by dmraid and use the GParted
DMRaid module. It only affects how GParted shows the member drives
themselves.
[1] mdadm ANNOUNCE-3.0 file
https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/tree/ANNOUNCE-3.0?h=mdadm-3.0
[2] Fedora 14, Storage Administration Guide, 12.5. Linux RAID Subsystem
https://docs.fedoraproject.org/en-US/Fedora/14/html/Storage_Administration_Guide/raid-subsys.html
"... Fedora 14 uses mdraid with external metadata to access ISW /
IMSM (Intel firmware RAID) sets. mdraid sets are configured and
controlled through the mdadm utility."
[3] RHEL 6, Storage Administration Guide, 17.3. Linux RAID Subsystem
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/storage_administration_guide/raid-subsys
"mdraid also supports other metadata formats, known as external
metadata. Red Hat Enterprise Linux 6 uses mdraid with external
metadata to access ISW / IMSM (Intel firmware RAID) sets. mdraid
sets are configured and controlled through the mdadm utility."
[4] SUSE Linux Enterprise Server 12 Release Notes, 7.2.3 Driver for IMSM
and DDF
https://www.suse.com/releasenotes/x86_64/SUSE-SLES/12/#fate-316007
"For IMSM and DDF RAIDs the mdadm driver is used unconditionally."
Closes#75 - Errors with GPT on RAID 0 ATARAID array
PATCHSET OVERVIEW
A user had a Firmware / BIOS / ATARAID array of 2 devices configured as
a RAID 0 (stripe) set. On top of that was a GPT with the OS partitions.
GParted displays the following errors on initial load and subsequent
refresh:
Libparted Error
(-) Invalid argument during seek for read on /dev/sda
[ Retry ] [ Cancel ] [ Ignore ]
Libparted Error
(-) The backup GPT table is corrupt, but the
primary appears OK, so that will be used.
[ Ok ] [ Cancel ]
This is an Intel Software RAID array which stores metadata at the end of
each member device, and so the first 128 KiB stripe of the set is stored
in the first 128 KiB of the first member device /dev/sda which includes
the GPT for the whole RAID 0 device. Hence when libparted reads member
device /dev/sda it finds a GPT describing a block device twice it's
size and in results the above errors when trying to read the backup GPT.
A more dangerous scenario occurs when using 2 devices configured in an
Intel Software RAID 1 (mirrored) set with GPT on top. On refresh
GParted display this error for both members, /dev/sda and /dev/sdb:
Libparted Warning
/!\ Not all of the space available to /dev/sda appears to be used,
you can fix the GPT to use all of the space (an extra 9554
blocks) or continue with the current setting?
[ Fix ] [ Ignore ]
Selecting [Fix] gets libparted to re-write the backup GPT to the end of
the member device, overwriting the ISW metadata! Do that twice and both
copies of the metadata are gone!
Worked example of this more dangerous mirrored set case. Initial setup:
# dmraid -s
*** Group superset isw_caffbiaegi
--> Subset
name : isw_caffbiaegi_MyMirror
size : 16768000
stride : 128
type : mirror
status : ok
subsets: 0
devs : 2
spares : 0
# dmraid -r
/dev/sda: isw, "isw_caffbiaegi", GROUP, ok, 16777214 sectors, data@ 0
/dev/sdb: isw, "isw_caffbiaegi", GROUP, ok, 16777214 sectors, data@ 0
# wipefs /dev/sda
offset type
---------------------------------------------
0x200 gpt [partition table]
0x1fffffc00 isw_raid_member [raid]
Run GParted and click [Fix] on /dev/sda. Now the first member has gone:
# dmraid -s
*** Group superset isw_caffbiaegi
--> *Inconsistent* Subset
name : isw_caffbiaegi_MyMirror
size : 16768000
stride : 128
type : mirror
status : inconsistent
subsets: 0
devs : 1
spares : 0
# dmraid -r
/dev/sdb: isw, "isw_caffbiaegi", GROUP, ok, 16777214 sectors, data@ 0
# wipefs /dev/sda
offset type
---------------------------------------------
0x200 gpt [partition table]
Click [Fix] on /dev/sdb. Now all members of the array are gone:
# dmraid -s
no raid disks
# dmraid -r
no raid disks
# wipefs /dev/sdb
offset type
---------------------------------------------
0x200 gpt [partition table]
So GParted must not run libparted partition table scanning on the member
devices in ATARAID arrays. Only on the array device itself.
In terms of the UI GParted must show disks which are ATARAID members as
whole disk devices with ATARAID member content and detect array busy
status to avoid allowing active members from being overwritten while in
use.
THIS COMMIT
Recognise ATARAID member devices and display in GParted as whole device
"ataraid" file systems. Because they are recognised as whole device
content ("ataraid" file systems) this alone stops GParted running the
libparted partition table scanning and avoids the above errors.
The list of dmraid supported formats is matched by the signatures
recognised by blkid:
$ dmraid -l
asr : Adaptec HostRAID ASR (0,1,10)
ddf1 : SNIA DDF1 (0,1,4,5,linear)
hpt37x : Highpoint HPT37X (S,0,1,10,01)
hpt45x : Highpoint HPT45X (S,0,1,10)
isw : Intel Software RAID (0,1,5,01)
jmicron : JMicron ATARAID (S,0,1)
lsi : LSI Logic MegaRAID (0,1,10)
nvidia : NVidia RAID (S,0,1,10,5)
pdc : Promise FastTrack (S,0,1,10)
sil : Silicon Image(tm) Medley(tm) (0,1,10)
via : VIA Software RAID (S,0,1,10)
dos : DOS partitions on SW RAIDs
$ fgrep -h _raid_member util-linux/libblkid/src/superblocks/*.c
.name = "adaptec_raid_member",
.name = "ddf_raid_member",
.name = "hpt45x_raid_member",
.name = "hpt37x_raid_member",
.name = "isw_raid_member",
.name = "jmicron_raid_member",
.name = "linux_raid_member",
.name = "lsi_mega_raid_member",
.name = "nvidia_raid_member",
.name = "promise_fasttrack_raid_member",
.name = "silicon_medley_raid_member",
.name = "via_raid_member",
As they are all types of Firmware / BIOS / ATARAID arrays, report all
members as a single "ataraid" file system type. (Except for
"linux_raid_member" in the above blkid source listing which is Linux
Software RAID).
Closes#75 - Errors with GPT on RAID 0 ATARAID array
The HACKING file should be hints for making changes to the code base and
associated processes. A overview of how GParted handled unallocated
space was not that. Also now the size of a JFS is accurately calculated
using JFS as an example of a file system with intrinsic unallocated
space is no longer valid. Therefore removed from the HACKING file.
Instead add the original commit message as an extended comment to method
calc_significant_unallocated_sectors().
Closes!50 - Calculate JFS size accurately
With the same minimum sized 16 MiB JFS used in the previous commit, now
mounted, GParted once again reports 1.20 MiB of unallocated space. This
is because the kernel JFS driver is also just reporting the size of the
Aggregate Disk Map (dmap) as the size of the file system [1].
Fix by reading the on disk JFS superblock to calculate the size of the
file system, but query the free space from the kernel using statvfs().
Need to query mounted JFS free space from the kernel because the on disk
dmap is not updated immediately so doesn't reflect recently used or
freed disk space.
For example, start with the 16 MiB JFS empty and mounted.
# echo -e 'dmap\nx\nquit' | jfs_debugfs /dev/sdb1 | fgrep dn_nfree
[2] dn_nfree: 0x00000000eaa [10] dn_agwidth: 1
# df -k /mnt/1
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb1 15152 136 15016 1% /mnt/1
Write 10 MiB of data to it:
# dd if=/dev/zero bs=1M count=10 of=/mnt/1/file_10M
10+0 records in
10+0 records out
1048760 bytes (10 MB, 10 MiB) copied, 0.0415676 s, 252 MB/s
Query the file system free space from the kernel and by reading the on
disk dmap figure:
# df -k /mnt/1
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb1 15152 10376 4776 69% /mnt/1
# echo -e 'dmap\nx\nquit' | jfs_debugfs /dev/sdb1 | fgrep dn_nfree
[2] dn_nfree: 0x00000000eaa [10] dn_agwidth: 1
# sync
# echo -e 'dmap\nx\nquit' | jfs_debugfs /dev/sdb1 | fgrep dn_nfree
[2] dn_nfree: 0x00000000eaa [10] dn_agwidth: 1
# umount /mnt/1
# echo -e 'dmap\nx\nquit' | jfs_debugfs /dev/sdb1 | fgrep dn_nfree
[2] dn_nfree: 0x000000004aa [10] dn_agwidth: 1
The kernel reports the updated usage straight away, but the on disk dmap
record doesn't get updated even by sync, only after unmounting.
This is the same fix as was previously done for EXT2/3/4 [2].
[1] Linux jfs_statfs() function
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/jfs/super.c?h=v3.10#n142
[2] 3828019030
Read file system size for mounted ext2/3/4 from superblock (#683255)
Closes!50 - Calculate JFS size accurately
Upstream NILFS project calls the package nilfs-utils [1][2]. Arch Linux
/ CentOS / Fedora / OpenSUSE use the upstream name. However Debian /
Ubuntu name it nilfs-tools [3] instead.
Document the needed software as:
nilfs-utils / nilfs-tools
Upstream name first separated by slash from alternative names
distributions use.
[1] NILFS Download page
https://nilfs.sourceforge.io/en/download.html
[2] NILFS Public Git Repositories
https://nilfs.sourceforge.io/en/git_repos.html
[3] Debian package: nilfs-tools
https://packages.debian.org/sid/nilfs-tools
Running JFS read usage test on Fedora 30 fails like this:
$ ./test_SupportedFileSystems --gtest_filter='*ReadUsage/jfs'
...
[ RUN ] My/SupportedFileSystemsTest.CreateAndReadUsage/jfs
unknown file: Failure
C++ exception with description "std::bad_alloc" thrown in the test body.
[ FAILED ] My/SupportedFileSystemsTest.CreateAndReadUsage/jfs, where GetParam() = 17 (41833 ms)
However the same test passes on Fedora 29, Fedora 31 Beta, CentOS 7,
Debian 10 and Ubuntu 18.04 LTS.
Also running GParted on Fedora 30 crashes just the same when reading JFS
usage:
# gparted
GParted 1.0.0
configuration --enable-libparted-dmraid --enable-online-resize
libparted 3.2
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
/usr/bin/gparted: line 202: 19218 Aborted (core dumped) $BASE_CMD
Running jfs_debugfs to query the file system usage the same way GParted
does produces an infinite amount of repeating output:
# echo dm | jfs_debugfs /dev/sdb1
So jfs_debugfs gets stuck in an infinite loop inside the dmap subcommand
when it encounters EOF. GParted and the read JFS usage test read this
output until memory is exhausted and crash. This is exactly what was
happening in closed bug 794947. Even installed jfsutils from Fedora 29
on Fedora 30 and visa versa. jfs_debugfs still produced an infinite
amount of output on Fedora 30 and worked correctly on Fedora 29. So
it's not the build of jfsutils, but something in the OS that is making
the difference!
Anyway fix by providing the instruction to exit from the dmap
subcommand, and quit from jfs_debugfs itself, like this:
# echo -e 'dmap\nx\nquit' | jfs_debugfs /dev/sdb1
Bug 794947 - gparted hangs when sees JFS partition on discovering
partitions
Closes!49 - Add file system interface tests
Mike Fleetwood