The Ubuntu based GitLab CI jobs have recently started being terminated
after the default 1 hour timeout. Installing / updating packages in the
image is updating the tzdata package which is prompting for input which
it will never receive, hence the hang. The end of output from the job
looks like this:
Setting up tzdata (2020a-0ubuntu0.20.04) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
Configuring tzdata
------------------
Please select the geographic area in which you live. Subsequent configuration
questions will narrow this down by presenting a list of cities, representing
the time zones in which they are located.
1. Africa 4. Australia 7. Atlantic 10. Pacific 13. Etc
2. America 5. Arctic 8. Europe 11. SystemV
3. Antarctica 6. Asia 9. Indian 12. US
Geographic area:
...
ERROR: Job failed: execution took longer than 1h0m0s seconds
This is a well known issue [1][2][3]. Probably occurring now because of
a new release of tzdata not included in the base Ubuntu image we are
using. Fix by telling the underlying dpkg tools this installation is
non-interactive.
[1] Avoiding user interaction with tzdata when installing certbot in a
docker container
https://askubuntu.com/questions/909277/avoiding-user-interaction-with-tzdata-when-installing-certbot-in-a-docker-contai
[2] How to install tzdata on a ubuntu docker image?
https://serverfault.com/questions/949991/how-to-install-tzdata-on-a-ubuntu-docker-image
[3] apt-get install tzdata noninteractive
https://stackoverflow.com/questions/44331836/apt-get-install-tzdata-noninteractiveCloses!60 - Fix GitLab CI job failures following Ubuntu docker image
updates
This previous commit [1] excluded unit test
BlockSpecialTest.NamedBlockSpecialObjectBySymlinkMatches because GNOME
GitLab Docker CI images don't have /dev/disk hierarchy and so no
symbolic links to block devices.
Create the /dev/disk/by-id directory and a symlink for this unit test to
use in the new tests/makedev.sh script.
[1] fe2fc33e67
Exclude unit test which fails in Docker CI image (!4)
test_SupportedFileSystems is another unit test that has been failing
since 23-Feb-2020 in GNOME GitLab Continuous Integration test jobs.
Fragments from tests/test-suite.log from a failed test CI job:
FAIL: test_SupportedFileSystems
===============================
...
[ RUN ] My/SupportedFileSystemsTest.Create/lvm2pv
test_SupportedFileSystems.cc:387: Failure
Failed
create_loopdev(): Execute: losetup --find --show 'test_SupportedFileSystems.img'
losetup: test_SupportedFileSystems.img: failed to set up loop device: No such file or directory
create_loopdev(): Losetup failed with exit status 1
create_loopdev(): Failed to create required loop device
Error: Could not stat device - No such file or directory.
test_SupportedFileSystems.cc:446: Failure
Value of: lp_device != NULL
Actual: false
Expected: true
test_SupportedFileSystems.cc:490: Failure
Value of: m_fs_object->create(m_partition, m_operation_detail)
Actual: false
Expected: true
Operation details:
lvm pvcreate -M 2 '' 00:00:00 (ERROR)
WARNING: Failed to connect to lvmetad. Falling back to device scanning.
Device not found.
detach_loopdev(): Execute: losetup --detach ''
losetup: /dev/: detach failed: Inappropriate ioctl for device
detach_loopdev(): Losetup failed with exit status 1
detach_loopdev(): Failed to detach loop device. Test NOT affected
[ FAILED ] My/SupportedFileSystemsTest.Create/lvm2pv, where GetParam() = 20 (64 ms)
...
[ RUN ] My/SupportedFileSystemsTest.CreateAndReadUsage/btrfs
test_SupportedFileSystems.cc:387: Failure
Failed
create_loopdev(): Execute: losetup --find --show 'test_SupportedFileSystems.img'
losetup: test_SupportedFileSystems.img: failed to set up loop device: No such file or directory
create_loopdev(): Losetup failed with exit status 1
create_loopdev(): Failed to create required loop device
Error: Could not stat device - No such file or directory.
test_SupportedFileSystems.cc:446: Failure
Value of: lp_device != NULL
Actual: false
Expected: true
test_SupportedFileSystems.cc:503: Failure
Value of: m_fs_object->create(m_partition, m_operation_detail)
Actual: false
Expected: true
Operation details:
mkfs.btrfs -L '' '' 00:00:00 (ERROR)
btrfs-progs v4.9.1
See http://btrfs.wiki.kernel.org for more information.
ERROR: failed to check size for : No such file or directory
detach_loopdev(): Execute: losetup --detach ''
losetup: /dev/: detach failed: Inappropriate ioctl for device
detach_loopdev(): Losetup failed with exit status 1
detach_loopdev(): Failed to detach loop device. Test NOT affected
[ FAILED ] My/SupportedFileSystemsTest.CreateAndReadUsage/btrfs, where GetParam() = 7 (11 ms)
All the test_SupportedFileSystems unit tests which need a loop device
are failing when running losetup like this:
losetup: <IMAGE_NAME>: failed to setup loop device: No such file or directory
losetup uses /dev/loop-control [1][2] which no longer exists in the
Docker image. However even after creating /dev/loop-control in the
image, losetup continues to fail. Also tried stracing losetup but that
fails like this, presumably because it is not allowed inside the Docker
image:
$ strace losetup --find --show /tmp/disk.img
strace: ptrace(PTRACE_TRACEME, ...): Operation not permitted
+++ exited with 1 +++
For now I have run out of ways to investigate and resolve this, so just
exclude the 12 tests which required loop devices. All the tests still
execute when run locally outside the restricted GNOME GitLab CI Docker
setup.
Closes!59 - Fix GNOME GitLab CI test job failures because of missing
/dev entries
From 23-Feb-2020 onwards, GNOME GitLab Continuous Integration test jobs
have been failing running unit tests which previously succeeded. With
some extra debugging added into test_BlockSpecial to print 'bname' and
'bs' values in the failing tests, here are fragments from
tests/test-suite.log for the the test_BlockSpecial failures in a test CI
job:
FAIL: test_BlockSpecial
=======================
...
[ RUN ] BlockSpecialTest.NamedBlockSpecialObjectBlockDevice
bname="/dev/sr0"
bs=BlockSpecial{"/dev/sr0",0,0}
test_BlockSpecial.cc:218: Failure
Value of: bs.m_major > 0 || bs.m_minor > 0
Actual: false
Expected: true
[ FAILED ] BlockSpecialTest.NamedBlockSpecialObjectBlockDevice (0 ms)
...
[ RUN ] BlockSpecialTest.TwoNamedBlockSpecialObjectBlockDevices
bname1="/dev/sr0"
bname2="/dev/sda"
bs1=BlockSpecial{"/dev/sr0",0,0}
bs2=BlockSpecial{"/dev/sda",0,0}
test_BlockSpecial.cc:250: Failure
Value of: bs1.m_major != bs2.m_major || bs1.m_minor != bs2.m_minor
Actual: false
Expected: true
[ FAILED ] BlockSpecialTest.TwoNamedBlockSpecialObjectBlockDevices (1 ms)
Contents of /proc/partitions inside the Docker image when this test CI
job failed:
$ cat /proc/partitions
major minor #blocks name
11 0 1048575 sr0
8 0 573367448 sda
8 1 573366407 sda1
And the listing of /dev/:
$ ls -l /dev/
total 0
lrwxrwxrwx 1 root root 11 Mar 3 09:00 core -> /proc/kcore
lrwxrwxrwx 1 root root 13 Mar 3 09:00 fd -> /proc/self/fd
crw-rw-rw- 1 root root 1, 7 Mar 3 09:00 full
drwxrwxrwt 2 root root 40 Mar 3 09:00 mqueue
crw-rw-rw- 1 root root 1, 3 Mar 3 09:00 null
lrwxrwxrwx 1 root root 8 Mar 3 09:00 ptmx -> pts/ptmx
drwxr-xr-x 2 root root 0 Mar 3 09:00 pts
crw-rw-rw- 1 root root 1, 8 Mar 3 09:00 random
drwxrwxrwt 2 root root 40 Mar 3 09:00 shm
lrwxrwxrwx 1 root root 15 Mar 3 09:00 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root 15 Mar 3 09:00 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root root 15 Mar 3 09:00 stdout -> /proc/self/fd/1
crw-rw-rw- 1 root root 5, 0 Mar 3 09:00 tty
crw-rw-rw- 1 root root 1, 9 Mar 3 09:00 urandom
crw-rw-rw- 1 root root 1, 5 Mar 3 09:00 zero
See how the test_BlockSpecial fixtures are getting major=0 and minor=0
for the block special devices they are testing with. This is happening
because there aren't any entries in /dev for those disks and partitions
listed in /proc/partitions. Assume that Docker in GNOME GitLab has
changed and that unneeded and unwanted devices in /dev are no longer
being created inside images.
In the test CI jobs execute new script, tests/makedev.sh, to create just
the first two block special devices mentioned in /proc/partitions needed
by test_BlockSpecial.
Closes!59 - Fix GNOME GitLab CI test job failures because of missing
/dev entries
Seems to be referring to how Fill_Label_Device_Info() worked in the
past, but from history before the beginning of the GIT repository.
Remove out of date comment.
Debian user reported a bug [1] that when they had PS_FORMAT environment
variable set it prevented GParted running:
# export PS_FORMAT='ruser,uid,pid,ppid,pri,ni,%cpu,%mem,vsz,rss,stat,tty,start,time,command'
# gparted
The process gpartedbin is already running.
Only one gpartedbin process is permitted.
# echo $?
1
Using ps column 'command' includes the command and all it's arguments,
rather than just the command name as ps displays by default. Thus the
shell wrapper finds the grep command it's using when searching for the
gpartedbin executable.
# ps -e | grep gpartedbin
root 0 26114 14777 19 0 0.0 0.0 112712 940 S+ pts/0 10:42:02 00:00:00 grep --color=auto gpartedbin
Fix by searching for running processes using pidof. pgrep does regular
expression matching where as pidof checks program name is the same [2].
Therefore use of pidof is preferred over pgrep [3].
[1] Debian bug #864932 - gparted fails if PS_FORMAT options are
specified
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864932
[2] Difference between pidof and pgrep?
https://stackoverflow.com/questions/52151698/difference-between-pidof-and-pgrep
[3] [PATCH] gparted.in: Use reliable way of detecting gpartedbin process
existence
https://git.alpinelinux.org/aports/tree/community/gparted/gparted.in-Use-reliable-way-of-detecting-gpartedbin-.patchCloses!54 - Fix gparted not launching when PS_FORMAT environment
variable set
To better reflect that they represent the version of mke2fs executable
from the e2fsprogs package, even though the executable is called as
mkfs.ext4.
$ ls -il /sbin/mke2fs /sbin/mkfs.ext*
1978670 -rwxr-xr-x. 4 root root 96384 Aug 9 2019 /sbin/mke2fs
1978670 -rwxr-xr-x. 4 root root 96384 Aug 9 2019 /sbin/mkfs.ext2
1978670 -rwxr-xr-x. 4 root root 96384 Aug 9 2019 /sbin/mkfs.ext3
1978670 -rwxr-xr-x. 4 root root 96384 Aug 9 2019 /sbin/mkfs.ext4
$ mkfs.ext4 -V
mke2fs 1.42.9 (28-Dec-2013)
Using EXT2FS Library version 1.42.9
$ mke2fs -V
mke2fs 1.42.9 (28-Dec-2013)
Using EXT2FS Library version 1.42.9
With removal of support for RHEL / CentOS 5 and it's e4fsprogs package
[1][2] it is no longer necessary to accept:
mke4fs 1.41.12 (17-May-2010)
only:
mke2fs 1.42.9 (28-Dec-2013)
[1] 6c4ab5dc28
Remove checks for e4fsprogs commands (#794253)
[2] de6e70d933
Simplify ext2::get_filesystem_support() with regard ext4 support (#794253)
This earlier commit [1] from 2013 recognised the new names for programs
in dosfstools >= 3.0.18, specifically mkfs.fat and fsck.fat. Now that
the oldest supported distributions use dosfstools >= 3.0.18 it is no
longer necessary to support using the old names of mkdosfs and dosfsck,
so remove that code.
Oldest supported Dosfstools
distributions Version
Debian 8 3.0.27
RHEL / CentOS 7 3.0.20
SLES 12 3.0.26
Ubuntu 14.04 LTS 3.0.26
[1] 1ae03dee95
Recognise new dosfstools program names (#704629)
Closes!57 - Raise minimum support dosfstools to 3.0.18 released
2013-06-06
A user reported that formatting a whole disk device with a file system
failed like this:
Format /dev/sdd as ext4 (ERROR)
+ calibrate /dev/sdd (SUCCESS)
path: /dev/sdd (device)
start: 0
end: 15633407
size: 15633408 (7.45 GiB)
+ clear old file system signatures in /dev/sdd (SUCCESS)
+ write 512.00 KiB of zeros at byte offset 0 (SUCCESS)
+ write 4.00 KiB of zeros at byte offset 67108864 (SUCCESS)
+ write 512.00 KiB of zeros at byte offset 8003780608 (SUCCESS)
+ write 4.00 KiB of zeros at byte offset 8004239360 (SUCCESS)
+ write 8.00 KiB of zeros at byte offset 8004296704 (SUCCESS)
+ flush operating system cache of /dev/sdd (SUCCESS)
+ create new ext4 file system (ERROR)
+ mkfs.ext4 -F -O ^64bit -L '' '/dev/sdd' (ERROR)
mke2fs 1.44.1 (24-Mar-2018)
/dev/sdd is apparently in use by the system; will not make a filesystem here!
Opening the whole disk block device exclusively causes mkfs.ext4 to
report that error like this:
# python
>>> import os
>>> f = os.open('/dev/sdb',os.O_RDONLY|os.O_EXCL)
>>> ^Z
[1]+ Stopped python
# mkfs.ext4 -F -O ^64bit -L '' '/dev/sdb'
mke2fs 1.42.9 (28-Dec-2013)
/dev/sdb is apparently in use by the system; will not make a filesystem here!
# echo $?
1
I have not been able to reproduce this error, but with debugging and
sleeping in GParted, stracing GParted and using 'udevadm monitor' to
watch udev events the following sequence of events is seen:
gparted |format(partition, operationdetail)
gparted | erase_filesystem_signatures(partition, operationdetail)
gparted | get_device(device_path="/dev/sdb", lp_device, flush=false)
gparted | ped_device_get("/dev/sdb")
libparted | open("/dev/sdb", O_RDONLY) = 11
libparted | close(11)
gparted | ped_device_open(lp_device)
libparted | open("/dev/sdb", O_RDWR) = 11
gparted | ped_device_sync(lp_device)
libparted | ioctl(11, BLKFLSBUF)
gparted | ped_device_close()
libparted | close(11)
udev(async)| KERNEL change /devices/.../sdb (block)
udev(async)| UDEV change /devices/.../sdb (block)
gparted | set_partition_type(partition, operationdetail)
gparted | create_filesystem(partition, operationdetail)
gparted | ext2::create(partition, operationdetail)
gparted | FileSystem::execute_command("mkfs.ext4 -F -O ^64bit -L '' '/dev/sdb')
So it is assumed that the processing of the udev change rule after
closing the block device in erase_filesystem_signatures() overlaps with
the execution mkfs.ext4 and causes the seen error. Fix by waiting for
those udev events to complete as was previously done by commits [1][2]
[3].
Also note that this is specific to creating file systems on and
formatting unpartitioned whole disk devices because set_partition_type()
is a no-operation. Where as on a partitioned device
set_partition_type() calls commit() which already waits for udev rules
to complete [3].
[1] 50c8924a8e4d9cc96a2ea45f13291114402affee
Wait for udev to recreate /dev/PTN entries when querying partition
FSs (!46)
[2] 4f6c312e3bc68cafb5e6035fd4a5b5bbbfcea992
Wait for udev change on /dev/DISK when querying whole device FS
(!46)
[3] 2f53876c0f
Wait for the kernel and udev to settle partitions for a second time
(#790418)
Closes#83 - /dev/sdd is apparently in use by the system; will not make
a filesystem here!
Name the structure member to 'fsname' used to store strings like "ext2"
etc. This is equivalent to what was previously done in this commit:
a9f08ddc7d
Rename local variable to fsname in get_filesystem() (#741430)
Closes!52 - Rename members and variables currently named 'filesystem'
Previously made this change:
175d27c55d
Rename enum FILESYSTEM to FSType
Now complete the renaming exercise of members and variables currently
named 'filesystem'.
Closes!52 - Rename members and variables currently named 'filesystem'
In GParted_Core::set_device_partitions() the partition path is being
queried from libparted. However this is done before the switch
statement on the type of the partition, so is called for all libparted
partition objects including PED_PARTITION_FREESPACE and
PED_PARTITION_METADATA ones. As libparted numbers these partition
objects as -1, it returns paths like "/dev/sda-1".
Additionally when using GParted, with it's default DMRaid handling, on a
dmraid started array this results in paths like
"/dev/mapper/isw_ecccdhhiga_MyArray-1" being passed to
is_dmraid_device() and make_path_dmraid_compatible(). Fortunately
make_path_dmraid_compatible() does nothing and returns the same name.
Call chain looks like:
GParted_Core::set_device_partitions()
get_partition_path(lp_partition)
// where:
// lp_partition->disk->dev->path = "/dev/mapper/isw_ecccdhhiga_MyArray"
// lp_partition->type == PED_PARTITION_FREESPACE |
// PED_PARTITION_METADATA
// ->num == -1
ped_partition_get_path(lp_partition)
return "/dev/mapper/isw_ecccdhhiga_MyArray-1"
dmraid.is_dmraid_supported()
dmraid.is_dmraid_device("/dev/mapper/isw_ecccdhhiga_MyArray-1")
return true
dmraid.make_path_dmraid_compatible("/dev/mapper/isw_ecccdhhiga_MyArray-1")
return "/dev/mapper/isw_ecccdhhiga_MyArray-1"
Fix by moving the get_partition_path() call inside the switch statement
so that it is only called for PED_PARTITION_NORMAL,
PED_PARTITION_LOGICAL and PED_PARTITION_EXTENDED partition types.
Relevant commits:
* 53c49349f7
Simplify logic in set_device_partitions method
* 81986c0990
Ensure partition path name is compatible with dmraid (#622217)
This is not strictly necessary as members are already recognised using
blkid since this commit earlier in the sequence "Recognise ATARAID
members (#75)". However it makes sure active members are recognised
even if blkid is not available and matches how file system detection
queries the SWRaid_Info module.
Closes#75 - Errors with GPT on RAID 0 ATARAID array
This matches how the array device is displayed as the mount point for
mdadm started ATARAID members by "Display array device as mount point of
mdadm started ATARAID members (#75)" earlier in this patchset.
Extend the DMRaid module member cache to save the array device name and
use as needed to display as the mount point.
Closes#75 - Errors with GPT on RAID 0 ATARAID array
Again this is to stop GParted allowing overwrite operations being
performed on an ATARAID member while the array is actively using the
member. This time for dmraid started arrays using the kernel DM (Device
Mapper) driver.
The DMRaid module already uses dmraid to report active array names:
# dmraid -sa -c
isw_ecccdhhiga_MyArray
To find active members in this array, (1) use udev to lookup the kernel
device name:
# udevadm info --query=name /dev/mapper/isw_ecccdhhiga_MyArray
dm-0
(2) list the member names exposed by the kernel DM driver through the
/sys file system.
# ls /sys/block/dm-0/slaves
sdc sdd
# ls -l /sys/block/dm-0/slaves
lrwxrwxrwx 1 root root 0 Nov 24 09:52 sdc -> ../../../../pci0000:00/0000:00:0d.0/ata3/host2/target2:0:0/2:0:0:0/block/sdc
lrwxrwxrwx 1 root root 0 Nov 24 09:52 sdc -> ../../../../pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdd
Closes#75 - Errors with GPT on RAID 0 ATARAID array
When an ATARAID member is inactive allow basic supported actions of
copy and move to be performed like with other recognised but only basic
supported types.
Closes#75 - Errors with GPT on RAID 0 ATARAID array
Since earlier commit "Display array device as mount point of mdadm
started ATARAID members (#75)" GParted allows attempting to unmout a
busy ATARAID member as if it was a file system. This is not a valid
thing to do, so disallow it.
Closes#75 - Errors with GPT on RAID 0 ATARAID array
This matches how other non-file systems are handled, by displaying the
access reference in the mount point column. For LVM Physical Volumes
the Volume Group name is displayed [1] and for an active Linux Software
RAID array the array device is displayed [2].
[1] 8083f11d84
Display LVM2 VGNAME as the PV's mount point (#160787)
[2] f6c2f00df7
Populate member mount point with SWRaid array device (#756829)
Closes#75 - Errors with GPT on RAID 0 ATARAID array
This stops GParted allowing overwrite operations (such as create
partition table or format with a whole device file system) being
performed on an ATARAID member while the array is actively using the
member.
Closes#75 - Errors with GPT on RAID 0 ATARAID array
The previous commit, made mdadm recognised IMSM and DDF type ATARAID
members get displayed as "linux-raid" (Linux Software RAID array
member). This was because of query method 1 in detect_filesystems().
Fix this now by exposing and using the fstype of the member from the
SWRaid_Info cache.
Closes#75 - Errors with GPT on RAID 0 ATARAID array
Since mdadm release 3.0 (2009-06-02) [1] it has also supported external
metadata formats IMSM (Intel Matrix Storage Manager) and DDF, previously
only managed by dmraid.
A number of distributions have switched to use mdadm and kernel MD
(Multiple Devices) driver for managing these Firmware / BIOS / ATARAID
arrays. These include: Fedora >= 14 [2], RHEL / CentOS >= 6 [3],
SLES >= 12 [4], Ubuntu >= 16.04 LTS.
Therefore additionally parse members in these ATARAID arrays included in
mdadm output, and when activated using the kernel MD driver, in file
/proc/mdstat. Add fstype to the SWRaid_Info cache records to
distinguish members apart. So far the rest of the GParted code
continues to treat all members as FS_LINUX_SWRAID. This will be
resolved in following commits.
Note that this in no way affects how GParted shows and partitions the
array device itself, even those managed by dmraid and use the GParted
DMRaid module. It only affects how GParted shows the member drives
themselves.
[1] mdadm ANNOUNCE-3.0 file
https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/tree/ANNOUNCE-3.0?h=mdadm-3.0
[2] Fedora 14, Storage Administration Guide, 12.5. Linux RAID Subsystem
https://docs.fedoraproject.org/en-US/Fedora/14/html/Storage_Administration_Guide/raid-subsys.html
"... Fedora 14 uses mdraid with external metadata to access ISW /
IMSM (Intel firmware RAID) sets. mdraid sets are configured and
controlled through the mdadm utility."
[3] RHEL 6, Storage Administration Guide, 17.3. Linux RAID Subsystem
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/storage_administration_guide/raid-subsys
"mdraid also supports other metadata formats, known as external
metadata. Red Hat Enterprise Linux 6 uses mdraid with external
metadata to access ISW / IMSM (Intel firmware RAID) sets. mdraid
sets are configured and controlled through the mdadm utility."
[4] SUSE Linux Enterprise Server 12 Release Notes, 7.2.3 Driver for IMSM
and DDF
https://www.suse.com/releasenotes/x86_64/SUSE-SLES/12/#fate-316007
"For IMSM and DDF RAIDs the mdadm driver is used unconditionally."
Closes#75 - Errors with GPT on RAID 0 ATARAID array
Mike Fleetwood