Since changes for issue #131 "GParted hangs when non-named device is
hung" FS_Info cache is initialised, cleared and loaded via one call to
load_cache_for_paths(). It runs blkid for named or found disk devices
and associated found partitions from /proc/partitions, rather than
running blkid and letting it report for all block devices.
To avoid the possibility of using blkid on an encryption mapping on a
non-specified and possibly hung block device GParted can't just specify
all encryption mappings. Instead only encryption mappings which belong
to the above identified block devices should be probed. That requires
identifying LUKS encryption data in the block devices first so will
require subsequently loading additional data into the FS_Info cache and
running blkid again.
To accommodate this make the FS_Info cache incrementally loadable,
rather than doing everything in a single call to load_cache_for_paths().
Have a separate clear_cache() call which initialises and clears the
cache and make load_cache_for_paths() just run blkid and insert data for
the named paths.
Closes 148 - Encrypted file systems are no longer recognised
This method is now only called from one location in the code so put it's
two lines of code there.
Closes#131 - GParted hangs when non-named device is hung
Now we always want to run blkid naming all paths, ensure the FS_Info
cache is explicitly loaded first. Report an error if not done so and
remove the cache loading code from running blkid without naming all
paths. Fewer code paths to consider and reason about.
Closes#131 - GParted hangs when non-named device is hung
Again on Fedora 31 with a slightly different disk layout to the previous
commit. sdb is partitioned with 1 empty partition and sdc remains
completely empty:
# lsblk -o name,maj:min,rm,size,ro,type,fstype,label,mountpoint
NAME MAJ:MIN RM SIZE RO TYPE FSTYPE LABEL MOUNTPOINT
sda 8:0 0 20G 0 disk
|-sda1 8:1 0 1G 0 part ext4 /boot
\-sda2 8:2 0 19G 0 part LVM2_member
|-fedora-root 253:0 0 17G 0 lvm ext4 /
\-fedora-swap 253:1 0 2G 0 lvm swap [SWAP]
sdb 8:16 0 8G 0 disk
\-sdb1 8:17 0 1G 0 part
sdc 8:32 0 8G 0 disk
sr0 11:0 1 1024M 0 rom
# blkid -v
blkid from util-linux 2.34 (libblkid 2.34.0, 14-Jun-2019)
# blkid /dev/sda /dev/sda1 /dev/sda2 /dev/sdb /dev/sdb1 /dev/sdc
/dev/sda: PTUUID="5012fb1f" PTTYPE="dos"
/dev/sda1: UUID="3cd48816-7817-4636-9fec-5f1afe76c1b2" TYPE="ext4" PARTUUID="5012fb1f-01"
/dev/sda2: UUID="PH94ej-C8xU-bnMJ-UIh8-ZimI-4B7f-dHlZxh" TYPE="LVM2_member" PARTUUID="5012fb1f-02"
/dev/sdb: PTUUID="1d120b57" PTTYPE="dos"
/dev/sdb1: PARTUUID="1d120b57-01"
Stracing GParted shows these executions of blkid:
# strace -f -q -bexecve -eexecve ./gpartedbin 2>&1 1> /dev/null | egrep -v 'ENOENT|SIGCHLD'
...
[pid 160040] execve("/usr/sbin/blkid", ["blkid", "/dev/sda", "/dev/sda1", "/dev/sda2", "/dev/sdb", "/dev/sdb1", "/dev/sdc"], 0xa4e1b0 /* 32 vars */ <detached ...>
[pid 160041] execve("/usr/sbin/blkid", ["blkid", "/dev/sdc"], 0xa4e1b0 /* 32 vars */ <detached ...>
...
On Fedora 31 with blkid from util-linux 2.34 it reports information for
sdb (partitioned drive) and sdb1 (empty partition) with only no
information for sdc (empty whole disk drive). Hence no FS_Info cache
entry and re-execution of blkid just for sdc.
On older CentOS 7 with the same disk layout blkid reports this:
# blkid -v
blkid from util-linux 2.23.2 (libblkid 2.23.0, 25-Apr-2013)
# blkid /dev/sda /dev/sda1 /dev/sda2 /dev/sdb /dev/sdb1 /dev/sdc
/dev/sda: PTTYPE="dos"
/dev/sda1: UUID="e7d559e4-3e1d-4fbc-b034-3fdeb498f44d" TYPE="xfs"
/dev/sda2: UUID="B7ODFx-BfTE-hq7N-UlrF-f5ML-CPRe-klSy26" TYPE="LVM2_member"
/dev/sdb: PTTYPE="dos"
And stracing GParted shows these executions of blkid:
# strace -f -q -bexecve -eexecve ./gpartedbin 2>&1 1> /dev/null | egrep -v 'ENOENT|SIGCHLD'
...
[pid 1889] execve("/sbin/blkid", ["blkid", "/dev/sda", "/dev/sda1", "/dev/sda2", "/dev/sdb", "/dev/sdb1", "/dev/sdc"], 0x10b8b10 /* 26 vars */ <detached ...>
[pid 1890] execve("/sbin/blkid", ["blkid", "/dev/sdb1"], 0x10b8b10 /* 26 vars */ <detached ...>
[pid 1891] execve("/sbin/blkid", ["blkid", "/dev/sdc"], 0x10b8b10 /* 26 vars */ <detached ...>
...
This time on CentOS 7 with blkid from util-linux 2.23.2 it reports
information for only sdb (partitioned drive), but not sdb1 (empty
partition) or sdc (empty whole disk drive). Hence no FS_info cache
entries and re-execution of blkid for both sdb1 and sdc.
GParted needs blkid identification of file system images, LVM Logical
Volumes or any other partitions named on the command line which it
wouldn't normally scan [1]. Now every name of interest is passed to
blkid, additional executions of blkid won't get any extra information
and are redundant. Therefore remove this unnecessary code.
Note that these last 2 commits remove creation of "blank" cache entries
(just block special with blank fstype and other attributes) when blkid
reports no information for a particular path. Those entry were needed
to suppress unnecessary additional execution of blkid. However now that
blkid is only executed once (excluding querying the label) this is no
longer necessary. All the getter functions return suitable blank values
when no cache entry is found.
[1] e8f0504b13
Make sure that FS_Info cache is loaded for all named paths (#787181)
Closes#131 - GParted hangs when non-named device is hung
A user reported that GParted would hang at "scanning all devices...",
when a fully working disk was named on the command line, but another
device on the machine was hung.
This can be replicated like this:
(on Ubuntu 20.04 LTS for it's NBD support)
1. Export and import NBD:
# truncate -s 1G /tmp/disk-1G.img
# nbd-server -C /dev/null 9000 /tmp/disk-1G.img
# nbd-client localhost 9000 /dev/nbd0
2. Hang the NBD server and therefore /dev/nbd0:
# killall -STOP nbd-server
3. Run GParted:
$ gparted /dev/sda
Tracing GParted shows that execution of blkid never returns.
# strace -f -tt -q -bexecve -eexecve ./gpartedbin 2>&1 1> /dev/null | fgrep -v ENOENT
...
[pid 37823] 13:56:24.814139 execve("/usr/sbin/mkudffs", ["mkudffs", "--help"], 0x55e2a3f2d230 /* 20 vars */ <detached ...>
[pid 37814] 13:56:24.829246 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=37823, si_uid=0, si_status=1, si_utime=0, si_stime=0} ---
[pid 37825] 13:56:25.376796 execve("/usr/sbin/blkid", ["blkid", "-v"], 0x55e2a3f2d230 /* 20 vars */ <detached ...>
[pid 37824] 13:56:25.380824 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=37825, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
[pid 37826] 13:56:25.402512 execve("/usr/sbin/blkid", ["blkid"], 0x55e2a3f2d230 /* 20 vars */ <detached ...>
Tracking of blkid shows that it hangs on either the open of or first
read from /dev/nbd0.
# strace blkid
...
lstat("/dev", {st_mode=S_IFDIR|0755, st_size=4560, ...}) = 0
lstat("/dev/nbd0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x2b, 0), ...}) = 0
stat("/dev/nbd0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x2b, 0), ...}) = 0
lstat("/dev", {st_mode=S_IFDIR|0755, st_size=4560, ...}) = 0
lstat("/dev/nbd0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x2b, 0), ...}) = 0
access("/dev/nbd0", F_OK) = 0
stat("/dev/nbd0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x2b, 0), ...}) = 0
openat(AT_FDCWD, "/sys/dev/block/43:0", O_RDONLY|O_CLOEXEC) = 4
openat(4, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(4) = 0
openat(AT_FDCWD, "/dev/nbd0", O_RDONLY|O_CLOEXEC
Clean up:
1. Resume NBD server:
# killall -CONT nbd-server
2. Delete NBD setup:
# nbd-client -d /dev/nbd0
# killall nbd-server
# rm /tmp/disk-1G.img
Fix this by making GParted specify the whole disk device and partition
names that it is interested in to blkid, rather than letting blkid scan
and report all block devices. Do this both when GParted determines the
devices for itself and when they are named on the command line.
Also update example blkid command output being parsed and cache value
with this change to how blkid is executed.
Closes#131 - GParted hangs when non-named device is hung
Now FS_Info::load_cache() and ::load_cache_for_paths() are nearly next
to each other, merge them together to simplify the code a little. This
makes the special case to ensure that file system images named on the
command line were queried by blkid and loaded into the FS_Info cache [1]
become the normal cache loading method. Already passing all discovered
or named devices to load_cache_for_paths() is also a step on the way to
doing it for all devices and partitions of interest.
Just need to ensure that load_cache_for_paths() always loads the cache
as load_cache() did, rather than only when it hadn't already been
loaded. Otherwise GParted will only ever run blkid and load the cache
once at startup and not on each refresh.
[1] e8f0504b13
Make sure that FS_Info cache is loaded for all named paths (#787181)
Closes#131 - GParted hangs when non-named device is hung
Naming a file system image file on the command line is shown by GParted
as unknown.
$ truncate -s 100M /tmp/fat.img
$ mkfs.vfat /tmp/fat.img
$ sudo ./gpartedbin /tmp/fat.img
Currently the FS_Info cache is loaded for all devices reported by
blkid (plus all whole disk devices identified from /proc/partitions even
if blkid reports nothing). However file system images named on the
command line are not queried so GParted can't identify them.
Fix by ensuring that the FS_Info blkid cache is loaded for all named
devices, including named file system image files.
Note that Mount_Info::load_cache() depends on the contents of the
FS_Info cache to lookup UUID= and LABEL= device names from /etc/fstab.
However only file systems in block devices can be mounted like this, and
never file system image files, so the fact that the cache may be
extended afterwards by FS_Info::load_cache_for_paths() does not matter.
History
Prior to version 0.22.0, when unpartitioned drive support was added,
GParted could recognise some file system image files using loop
partition handling in libparted. However libparted before version 3.2
reported the loop partition name as the whole disk device name appended
with "1" so all the query commands were provided a non-existent name to
use. Therefore no file system usage or the label was displayed.
Bug 787181 - Fix detection of file system images
So far GParted is still loading the default non-reversible encoded
labels from blkid in the initial loading of the FS_Info module cache.
This encoded label is used to match LABEL=<label> when reading
/etc/fstab, via the get_path_by_label() call, so works for ASCII only
labels. This prevents GParted enabling the "mount on >" partition menu
item when non-ASCII labels are used.
To fix this:
1) Stop reading the labels the wrong way.
Via the blkid command used to initially load the FS_Info module cache
and is subject to default non-reversible encoding of non-printable
ASCII bytes.
2) Read all the labels the right way, but only when needed.
Only when /etc/fstab file contains LABEL=<label> and
get_path_by_label() is called, read all the labels from blkid without
encoding them via run_blkid_update_cache_one_label().
3) Return label from the cache.
get_label() returns the cached label, loading it into the cache first
if needed with run_blkid_update_cache_one_label().
In the worst case scenario of having a LABEL=<label> in /etc/fstab blkid
will be run for every partition containing a recognised file system to
read the label. On my desktop with 5 hard drives, 4 SWRaid arrays and
31 recognised file systems running 'blkid -o value -s LABEL ...' 31
times took 0.074 seconds of a total scan time of 9.072 seconds. Less
that 1% of the total scanning time. When LABEL=<label> is not used in
/etc/fstab individual blkid executions are only used to read labels for
file systems where there is no file system specific tool available
reducing the impact further. Blkid itself caches the data in it's
blkid.tab cache file rather than reading all file systems on each
invocation. Also the Linux file system cache will already contain the
blkid executable file, needed libraries files and the blkid.tab cache
file itself. Hence why repeated execution of blkid is so fast.
Further to the updated comment in set_partition_label_and_uuid().
Matching LABEL=<label> from /etc/fstab uses the label obtained from
blkid run in the C locale so this kind of assumes it returns the label
correctly and it does for my limited testing on Unicode enabled
desktops. Just not sure if it would be true for all cases in all
locales compared to the FS specific command run in the users default
locale.
Bug 786502 - Support reading Unicode labels when file system specific
tools aren't available
Move the code which reads the Unicode label from FS_Info::get_label()
into new function run_blkid_update_cache_one_label() which also replaces
the non-reversibly encoded copy loaded during the initial cache load.
This is mainly a bit of code refactoring ready for the following change.
It deliberately keeps the initial loaded labels so that reading
/etc/fstab and decoding LABEL=<label> to block special device names via
FS_Info::get_path_by_label() continues to works, at least for ASCII only
labels.
Bug 786502 - Support reading Unicode labels when file system specific
tools aren't available
More recent versions of blkid don't report an ISO9660 file system on the
whole disk device if partitions can be reports for embedded partitions.
However when querying the whole disk device directly then the expected
ISO9660 file system is reported. For example on CentOS 7 with the
previous ISO images:
# wget http://git.kernel.org/cgit/utils/util-linux/util-linux.git/plain/tests/ts/isosize/sample.iso.gz
# dd if=/dev/zero bs=1M of=/dev/sdc
# zcat sample.iso.gz | dd of=/dev/sdc
# blkid -v
blkid from util-linux 2.23.2 (libblkid 2.23.0, 25-Apr-2013)
# blkid | fgrep /dev/sdc
/dev/sdc1: UUID="2013-01-04-22-05-45-00" LABEL="ARCH_201301" TYPE="iso9660" PTTYPE="dos"
# blkid /dev/sdc
/dev/sdc: UUID="2013-01-04-22-05-45-00" LABEL="ARCH_201301" TYPE="iso9660" PTTYPE="dos"
# wget http://cdimage.debian.org/debian-cd/8.6.0/amd64/iso-cd/debian-8.6.0-amd64-netinst.iso
# dd if=/dev/zero bs=1M of=/dev/sdc
# dd if=debian-8.6.0-amd64-netinst.iso bs=1M of=/dev/sdc
# blkid | fgrep /dev/sdc
/dev/sdc1: UUID="2016-09-17-14-23-48-00" LABEL="Debian 8.6.0 amd64 1" TYPE="iso9660" PTTYPE="dos"
/dev/sdc2: SEC_TYPE="msdos" UUID="17F3-1162" TYPE="vfat"
# blkid /dev/sdc
/dev/sdc: UUID="2016-09-17-14-23-48-00" LABEL="Debian 8.6.0 amd64 1" TYPE="iso9660" PTTYPE="dos"
This behavioural difference with blkid is probably as a result of newer
versions of udev informing the kernel of the partitions embedded within
the ISO9660 image, and not directly as a result of a change in blkid
itself. Older distributions don't have partition entries for the above
ISO images, but CentOS 7 (with udev 219) and later distributions do have
partition entries:
# fgrep sdc /proc/partitions
8 16 8388608 sdc
8 17 252928 sdc1
8 18 416 sdc2
Fix by ensuring that the blkid FS_Info cache has entries for whole disk
devices, even if each entry has all empty attributes because there is a
partition table and not a recognised file system.
Calling blkid on whole disk devices containing partition tables produces
output like this, with newer versions of blkid:
# blkid /dev/sda
/dev/sda: PTTYPE="dos"
# blkid /dev/sdb
/dev/sdb: PTTYPE="gpt"
This will be loaded into the FS_Info cache as a blank entry for the
device by run_blkid_load_cache(). There will be a path name but all the
other attributes will be blank because there are no TYPE, SEC_TYPE, UUID
or LABEL name value pairs. With older versions of blkid no output is
produced at all. In that case load_fs_info_cache_extra_for_path() will
create the same blank entry with just the path name defined.
Bug 771244 - gparted does not recognize the iso9660 file system in
cloned Ubuntu USB boot drives
It made the code look a little messy, is easily resolved in the build
system and made the dependencies more complicated than needed. Each
GParted header was tracked via multiple different names (different
numbers of "../include/" prefixes). For example just looking at how
DialogFeatures.o depends on Utils.h:
$ cd src
$ make DialogFeatures.o
$ egrep ' [^ ]*Utils.h' .deps/DialogFeatures.Po
../include/DialogFeatures.h ../include/../include/Utils.h \
../include/../include/../include/../include/../include/../include/Utils.h \
../include/../include/../include/Utils.h \
After removing "../include/" from the GParted header #includes, just
need to add "-I../include" to the compile command via the AM_CPPFLAGS in
src/Makefile.am. Now the dependencies on GParted header files are
tracked under a single name (with a single "../include/" prefix). Now
DialogFeatures.o only depends on a single name to Utils.h:
$ make DialogFeatures.o
$ egrep ' [^ ]*Utils.h' .deps/DialogFeatures.Po
../include/DialogFeatures.h ../include/Utils.h ../include/i18n.h \
The FS_Info module has a pseudo multi-object interface and used the
constructor to load the cache. However all the data in the class is
static. An FS_Info object doesn't contain any member variables, yet was
needed just to call the member functions.
Make all the member functions static removing the need to use any
FS_Info objects and provide an explicit load_cache() method.
The FS_Info cache is loaded from "blkid" output and compares block
special names. Therefore switch to using BlockSpecial objects so that
comparisons are performed by the major, minor device number instead.
Bug 767842 - File system usage missing when tools report alternate block
device names
FS_Info module caches the output from blkid as a single string and uses
regular expressions to find the line matching the requested block
special file name. This is not compatible with using BlockSpecial
objects to represent block special files, and perform matching by major,
minor device number. Therefore parse the blkid output into a vector of
structures containing the needed fields, ready for switching to
BlockSpecial objects in the following patch.
Interface to the module remains unchanged.
Bug 767842 - File system usage missing when tools report alternate block
device names
Vol_id was removed from udev 142, released 2009-05-13, and udev switched
to using blkid instead [1]. All currently supported distributions use
later versions of udev (or systemd after the udev merge), except for
RedHat / CentOS 5 with udev 095. However RedHat / CentOS 5 does provide
blkid and vol_id is found in udev specific /lib/udev directory not on
the PATH. Therefore effectively vol_id is not available on any
supported distribution and blkid is always available. Therefore remove
use of vol_id from the FS_Info module. Less code to refactor and test
in following changes.
[1] delete vol_id and require util-linux-ng's blkid
http://git.kernel.org/cgit/linux/hotplug/udev.git/commit/?id=f07996885dab45102492d7f16e7e2997e264c725
Bug 767842 - File system usage missing when tools report alternate block
device names
Older versions of blkid don't correctly distinguish between FAT16 and
FAT32 file systems when overwriting one with the other. This effects
GParted too with these file systems on whole disk devices where only
blkid is used to recognise the contents. See previous fix for why only
blkid is used in this case:
Avoid whole disk FAT being detected as MSDOS partition table
(#743181)
Example:
# blkid -v
blkid from util-linux 2.20.1 (liblkid 2.20.0, 19-Oct-2011)
# mkdosfs -F16 -I /dev/md1
# blkid | fgrep md1
/dev/md1: SEC_TYPE="msdos" UUID="7C23-95D9" TYPE="vfat"
# mkdosfs -F32 -I /dev/md1
# blkid | fgrep md1
/dev/md1: SEC_TYPE="msdos" UUID="7F93-98F4" TYPE="vfat"
So blkid recognised the UUID changed but didn't remove the SEC_TYPE for
the FAT32 file system. See FS_Info::get_fs_type() as it uses this to
distinguish between FAT16 and FAT32. This is a caching update bug in
blkid, because telling blkid not to use the cache gets the right
results:
# blkid -c /dev/null | fgrep md1
/dev/md1: UUID="7F93-98F4" TYPE="vfat"
With testing determined that blkid from util-linux 2.23 and later are
not affected and earlier versions are affected. Mostly recently known
affected distribution is Ubuntu 14.04 LTS with util-linux 2.20.1.
The straight forward fix would be to instruct blkid to not use its cache
with 'blkid -c /dev/null'. But using blkid's cache is needed to prevent
blkid hanging for minutes when trying to access a non-existent floppy
drive when the BIOS is set incorrectly. See commit:
18f863151c
Fix long scan problem when BIOS floppy setting incorrect
Instead, when using an older affected version of blkid and when blkid
cache reports a vfat file system, run blkid again bypassing the cache.
The device is known to exist and contain a vfat file system, just not
whether it is a FAT16 or FAT32 file system, so can't be a non-existent
floppy device and won't hang.
Bug 743181 - Add unpartitioned drive read-write support
This is part of parent bug:
Bug #721455 - Obsolete info in license text on multiple modules
and GNOME Goal:
https://wiki.gnome.org/Initiatives/GnomeGoals/Proposals
* verify all source files to make sure they have a license and a
copyright, and that both are up-to-date
Bug #721565 - License text contains obsolete FSF postal address
Include guards need to be unique within GParted code and all included
library header files.
http://en.wikipedia.org/wiki/Include_guard#Difficulties
Use this model for all include guards:
#ifndef GPARTED_FILE_NAME_H
#define GPARTED_FILE_NAME_H
...
#endif /* GPARTED_FILE_NAME_H */
Closes Bug #539297 - Make include guards unique
Normally, the blkid command will detect the UUID for a file system.
In cases where blkid fails to detect the UUID and the vol_id command
is available, then try using the vol_id command.
Mike Fleetwood