E2fsprogs 1.42 adds ext4 64bit feature [1] allowing volume sizes larger
than 16 TiB. However only enable large volumes from e2fsprogs 1.42.9
when a large number of 64bit bugs were fixed [2]. (Also RHEL / CentOS 7
use e2fsprogs 1.42.9 and always enable 64bit feature by default).
[1] Release notes, E2fsprogs 1.42 (November 29, 2011)
http://e2fsprogs.sourceforge.net/e2fsprogs-release.html#1.42
"This release of e2fsprogs has support for file systems > 16TB.
Online resize requires kernel support which will hopefully be in
Linux version 3.2. Offline support is not yet available for > 16TB
file systems, but will be coming".
[2] Release notes, E2fsprogs 1.42.9 (December 28, 2013)
http://e2fsprogs.sourceforge.net/e2fsprogs-release.html#1.42.9
"Fixed a large number of bugs in resize2fs, e2fsck, debugfs, and
libext2fs to correctly handle bigalloc and 64-bit file systems.
There were many corner cases that had not been noticed in previous
uses of these file systems, since they are not as common. Some of
the bugs could cause file system corruption or data loss, so users
of 64-bit or bigalloc file systems are strongly urged to upgrade to
e2fsprogs 1.42.9".
Bug 766910 - Multiple boot loaders don't work on 64bit EXT4 file systems
Calling libparted ped_geometry_new() creates a new PedGeometry object
from malloced memory, however the corresponding ped_geometry_destroy()
is never called to destroy the object and free the memory.
Perform a resize of a FAT file system when running GParted under
valgrind identifies several memory blocks leaked via ped_geometry_new()
from resize_move_filesystem_using_libparted(). One such example:
# valgrind --track-origins=yes --leak-check=full ./gpartedbin
...
==32069== 32 bytes in 1 blocks are definitely lost in loss record 5,419 of 11,542
==32069== at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==32069== by 0x8ECD8C5: ped_malloc (libparted.c:231)
==32069== by 0x8ED23C1: ped_geometry_new (geom.c:79)
==32069== by 0x4764F3: GParted::GParted_Core::resize_move_filesystem_using_libparted(GParted::Partition const&, GParted::Partition const&, GParted::OperationDetail&) (GParted_Core.cc:2666)
==32069== by 0x478007: GParted::GParted_Core::resize_filesystem(GParted::Partition const&, GParted::Partition const&, GParted::OperationDetail&, bool) (GParted_Core.cc:2990)
==32069== by 0x478440: GParted::GParted_Core::maximize_filesystem(GParted::Partition const&, GParted::OperationDetail&) (GParted_Core.cc:3037)
==32069== by 0x4769A0: GParted::GParted_Core::resize(GParted::Partition const&, GParted::Partition const&, GParted::OperationDetail&) (GParted_Core.cc:2746)
==32069== by 0x47582B: GParted::GParted_Core::resize_move(GParted::Partition const&, GParted::Partition&, GParted::OperationDetail&) (GParted_Core.cc:2457)
==32069== by 0x46DDB2: GParted::GParted_Core::apply_operation_to_disk(GParted::Operation*) (GParted_Core.cc:767)
...
There is also a leak of a PedGeometry object from
resize_move_partition(). Fix by calling ped_geometry_destroy() to
delete all the allocated PedGeometry objects and free the memory.
Bug 767009 - PedGeometry objects are memory leaked
When replacing the list of paths for the other partition object involved
in either a Resize/Move or Format operation in apply_operation_to_disk()
should replace the whole list of partitions not just replace with the
first path. Copy the whole path list is the correct action. It makes
no material difference because secondary partition paths are only used
to discover mount points during refresh phase and not at the apply
phase, where only the primary path is used.
Bug 766349 - Resolve code ugliness with partition path getting set to
"copy of /dev/SRC"
Quoting the relevant comments from GParted_Core::calibrate_partition():
Re-add the real partition path from libparted.
When creating a copy operation by pasting into unallocated space the
list of paths for the partition object was set to
["Copy of /dev/SRC"] because the partition didn't yet exist before
the operations were applied. Additional operations on that new
partition also got the list of paths set to ["Copy of /dev/SRC"].
This re-adds the real path to the start of the list making it
["/dev/NEW", "Copy of /dev/SRC"]. This provides the real path for
file system specific tools used during those additional operations
such mkfs for the format operation or fsck and others for the
resize/move operation.
FIXME: Having this work just because "/dev/NEW" happens to sort
before "Copy of /dev/SRC" is ugly! Probably have a separate display
path which can be changed at will without affecting the list of real
paths for the partition.
Having a separate display path is overly complicated and unnecessary.
Just replace the list of paths with the real one reported by libparted
if it contained "Copy of /dev/SRC", determined by checking if the file
exists. Otherwise continue to add the libparted name as an alternate
path. Whole disk devices can never be named "Copy of /dev/SRC" because
they are not partitioned so never created or deleted by GParted, only
ever written to, hence don't need the extra exists test logic.
Bug 766349 - Resolve code ugliness with partition path getting set to
"copy of /dev/SRC"
When composing a copy operation it always named the destination
partition as "copy of /dev/SRC". For the case of pasting into
unallocated space creating a new partition this was the right thing to
do as the partition doesn't yet exist so the path is not yet known.
However for the case of pasting into an existing partition the path is
known and replacing it with "copy of /dev/SRC" is wrong. No other
operation when operating on an existing partition changes it path.
Given a set of existing partitions, sdb1 to sdb4, compose a set of copy
operations as: copy sdb1 to sdb2, copy sdb2 to sdb3 and copy sdb3 to
sdb4. The displayed partitions before being applied become:
/dev/sdb1
copy of /dev/sdb1
copy of copy of /dev/sdb1
copy of copy of copy of /dev/sdb1
And the pending operations are named:
Copy /dev/sdb1 to /dev/sdb2
Copy copy of /dev/sdb1 to /dev/sdb3
Copy copy of copy of /dev/sdb1 to /sev/sdb4
This is perverse. In the case of pasting into an existing partition
keep the real path name. This keeps the partitions being displayed as:
/dev/sdb1
/dev/sdb2
/dev/sdb3
/dev/sdb4
And the pending operations named as:
Copy /dev/sdb1 to /dev/sdb2
Copy /dev/sdb2 to /dev/sdb3
Copy /dev/sdb3 to /dev/sdb4
Much more understandable.
Also switch to an upper case "C" in "Copy of /dev/SRC" as the temporary
path name when pasting into unallocated space. Finally update the
comment in calibrate_partition() to describe the remaining cases when
re-adding the path is still required.
Bug 766349 - Resolve code ugliness with partition path getting set to
"copy of /dev/SRC"
Make the code a little more self documenting by adding the symbolic
constants:
SETTLE_DEVICE_APPLY_MAX_WAIT_SECONDS
SETTLE_DEVICE_PROBE_MAX_WAIT_SECONDS
which highlight that settle_device() is called in two different
contexts, device probe and apply operations, with two different timeout
values.
File system specific commands sometimes fail reporting that the
partition specific /dev entry doesn't exist. Example failing check
operation details:
Check and repair file system (ext4) on dev/sdb4
calibrate /dev/sdb4
path: /dev/sdb4 (partition)
start: 4196352
end: 6293503
size: 2097152 (1.00 GiB)
check file system on /dev/sdb4 for errors and (if possible) fix them
e2fsck -f -y -bv -C 0 /dev/sdb4
e2fsck 1.42.9 (28-Dec-2013)
e2fsck: No such file or directory while trying to open /dev/sdb4
Possibly non-existent device?
This has been reproduced on CentOS 7. Debugging shows that the
libparted calls used to re-read the partition details in
GParted_Core::calibrate_partition() leads to udev removing and re-adding
all the partition /dev entries for the disk.
# udevadm monitor &
# gpartedbin
...
16.480662 +12.618659 calibrate_partition() calling get_device("/dev/sdb", lp_device) ...
16.483644 +0.002982 calibrate_partition() get_device() returned
16.483678 +0.000034 calibrate_partition() calling get_disk(lp_device, lp_disk) ...
16.618113 +0.134435 calibrate_partition() get_disk() returned
KERNEL[19275.707968] remove /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb1 (block)
16.618561 +0.000448 destroy_device_and_disk() calling ped_disk_destroy(lp_disk) ...
16.618584 +0.000023 destroy_device_and_disk() ped_disk_destroy() returned
16.618591 +0.000007 destroy_device_and_disk() calling ped_device_destroy(lp_disk) ...
16.618602 +0.000011 destroy_device_and_disk() ped_device_destroy() returned
16.618687 +0.000085 calibrate_partition() return true
16.618851 +0.000164 execute_command() e2fsck -f -y -v -C 0 /dev/sdb4
KERNEL[19275.708389] remove /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb2 (block)
KERNEL[19275.708500] remove /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb3 (block)
KERNEL[19275.708643] remove /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb4 (block)
KERNEL[19275.768278] change /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb (block)
KERNEL[19275.771171] add /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb1 (block)
KERNEL[19275.771360] add /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb2 (block)
KERNEL[19275.771542] add /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb3 (block)
KERNEL[19275.775858] add /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb4 (block)
UDEV [19275.820153] remove /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb3 (block)
UDEV [19275.823152] remove /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb4 (block)
UDEV [19275.828275] remove /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb1 (block)
16.742735 +0.123884 execute_command() exit status 8
UDEV [19275.841425] remove /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb2 (block)
UDEV [19275.905478] change /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb (block)
UDEV [19276.013580] add /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb3 (block)
UDEV [19276.034728] add /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb4 (block)
UDEV [19276.174840] add /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb1 (block)
UDEV [19276.237105] add /devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/sdb2 (block)
So exactly when GParted is running the external e2fsck command, udev is
in the middle of removing and re-adding all the /dev partition entries
for the disk. Hence the above failure reporting that /dev/sdb4 didn't
exist. This error depends on the timing between GParted running the
external file system specific command and udev removing and re-adding
the entries, so sometimes it works and sometimes it fails.
Further debugging showed that simply opening and closing the whole disk
device read-write triggers the same removing and re-adding of all the
partition /dev entries with udev >= 219. Opening the whole disk device
read-write is what libparted has always done until this post
libparted 3.2 patch to make it open read-only when probing:
http://git.savannah.gnu.org/cgit/parted.git/commit/?id=44d5ae0115c4ecfe3158748309e9912c5aede92d
libparted: Use read only when probing devices on linux (#1245144)
To fix this simply wait for udev devices to settle in
calibrate_partitions(). The longest I have seen udev take to do this is
0.80 seconds in a VM. Wait up to 10 seconds as is done in commit() ->
commit_to_os(), also called when applying operations.
On configurations which don't have this issue execution of udevadm
settle, which will return immediately, adds at most 0.1 seconds to the
time taken for the calibrate step. This won't be noticed in the time
taken of the operation details so there is no point in trying to avoid
executing udevadm settle when not needed.
Bug 762941 - Operations sometimes failing with: No such file or
directory
Minor issues:
1) In the while loop reading from /proc/partitions into variable line,
just after the sscanf() call the variable was re-purposed to hold the
device name making the code unnecessarily hard to follow.
2) Variable c_str was a fixed sized buffer holding the device name read
from /proc/partitions.
3) Variable c_str name provides no meaning as to what data it holds.
4) Return value from all the Utils::regexp_label() calls is converted
from Glib::ustring to std::string to be stored in device variable.
Resolve by using Utils::regexp_label() to extract the device name from
each line in /proc/partitions and store in the variable device, already
used for this purpose and now changed to type Glib::ustring.
realpath(3) manual page says:
BUGS
The POSIX.1-2001 standard version of this function is broken by
design, since it is impossible to determine a suitable size for
the output buffer, resolved_path. According to POSIX.1-2001 a
buffer of size PATH_MAX suffices, but PATH_MAX need not be a
defined constant, and may have to be obtained using pathconf(3).
And asking pathconf(3) does not really help, since, on the one
hand POSIX warns that the result of pathconf(3) may be huge and
unsuitable for mallocing memory, and on the other hand
pathconf(3) may return -1 to signify that PATH_MAX is not
bounded. The resolved_path == NULL feature, not standardized in
POSIX.1-2001, but standardized in POSIX.1-2008, allows this
design problem to be avoided.
The resolved_path == NULL feature of realpath() has existed as a Glibc
extension since realpath() was first added to Glibc 1.90, released in
June 1996. Therefore it can be used unconditionally.
https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=fa0bc87c32d02cd81ec4d0ae00e0d943c683e6e1
Bug 764369 - Use realpath() safely
The previous commit (Fix crash reading NTFS usage when there is no
/dev/PTN entry) identified that the FileSystem member variable "index"
is too small on 64-bit machines. Also this member variable stores no
FileSystem class information and was being used as a local variable.
Replace with local variables of the of the correct type, wide enough to
store the npos not found value.
Bug 764658 - GParted crashes when reading NTFS usage when there is no
/dev/PTN entry
On a 64-bit distribution, with an NTFS file system in a partition
without a /dev entry then GParted will crash when attempting to read
the file system usage. Not having a /dev entry for the partition is
rare and only known to occur for the disk devices used within Fake RAID
(dmraid) arrays, and then only on Ubuntu 12.04 LTS. Other/newer
distributions do create /dev entries for partitions found on disk
devices within Fake RAID arrays.
Create mirror Fake RAID array:
# dmraid -f isw -C MyArray --type 1 --disk /dev/sdc,/dev/sdd
# dmraid -ay
Create NTFS partition on the Fake RAID array. On refresh GParted
crashes:
# ./gpartedbin
(gpartedbin:590): glibmm-ERROR **:
unhandled exception (type std::exception) in signal handler:
what: basic_string::assign
Without a /dev/sdc1 device entry the ntfsresize command reports this:
# ntfsresize --info --force --no-progress-bar /dev/sdc1
ntfsresize v2015.3.14 (libntfs-3g)
ERROR(2): Failed to check '/dev/sdc1' mount state: No such file or directory
Probably /etc/mtab is missing. It's too risky to continue. You might try
an another Linux distro.
The problem code in ntfs::set_used_sectors():
145 index = output.find( "Cluster size" );
146 if ( index == output.npos ||
147 sscanf( output.substr( index ).c_str(), "Cluster size : %Ld", &S ) != 1 )
As "Cluster size" did not exist in the output find() returned the not
found token of string::npos [1], which in a 64-bit environment is
represented by 2^64-1 [2]. However it was saved in the variable index
of type unsigned integer, which is only a 32-bit integer, thus
truncating it to 2^32-1. Therefore the comparison failed and sscanf()
tried to parse the output starting at offset 2^32-1 which resulted in
the crash.
Introduced by commit:
324d99a172
Record file system block size where known (#760709)
Fix by following the same pattern of the other comparisons in
ntfs::set_used_sectors() which checks if index is less than the output
length.
References:
[1] std::string::find
http://www.cplusplus.com/reference/string/string/find/
[2] std::string::npos
http://www.cplusplus.com/reference/string/string/npos/
(Note that Glib::ustring is derived from std::string in the Standard C++
library and provides a compatible interface).
Bug 764658 - GParted crashes when reading NTFS usage when there is no
/dev/PTN entry
As with glibmm [1] the latest versions of libsigc++ also uses ISO C++
2011 features. The NEWS file [2] says:
2.5.1 (unstable):
* Use (and require) C++11
(Kjell Ahlstedt)
* Using C++11 lambda functions to create sigc::slots:
Avoid the need for SIGC_FUNCTORS_DEDUCE_RESULT_TYPE_WITH_DECLTYPE.
(Kjell Ahlstedt)
Without enabling C++11 compiler features, compilation of GParted with
libsigc++ 2.5.1 and later fails with errors such as theses:
/usr/include/sigc++-2.0/sigc++/trackable.h:40:3: warning: identifier 'noexcept' is a keyword in C++11 [-Wc++0x-compat]
trackable_callback(void* data, func_destroy_notify func) noexcept
^
[1] d6d7cb2bbf
Enable C++11 compilation when using glibmm 2.45.40 and later (#756035)
[2] libsigc++ 2.5.1 NEWS file
https://git.gnome.org/browse/libsigcplusplus/tree/NEWS?h=2.5.1
Bug 758545 - gparted-0.24.0 fails to build with gnome 3.18 mm packages
(on Gentoo)
GParted is allowing creation of a FAT32 formatted partition of any size.
However with a 512 byte sector size the maximum volume size of a FAT32
file system is reported to be 2 TiB.
* Wikipedia: File Allocation Table / FAT32
https://en.wikipedia.org/wiki/File_Allocation_Table#FAT32
"The boot sector uses a 32-bit field for the sector count, limiting
the FAT32 volume size to 2 TB for a sector size of 512 bytes and 16 TB
for a sector size of 4,096 bytes."
* Microsoft: Default cluster size for NTFS, FAT, and exFAT / Default
cluster sizes for FAT32
https://support.microsoft.com/en-us/kb/140365
Trying to create a FAT32 file system in a partition larger than 2 TiB
results in unallocated space being left after the file system.
Nuances:
[1] Larger sector sizes allow larger maximum volume sizes up to 16 TiB
with 4096 byte sectors.
[2] mkdosfs/mkfs.fat has an -S SECTOR_SIZE option which allows changing
the "logical" sector size of the file system allowing the maximum
volume to be proportionally increased.
[3] mkfs.fat appears to have an signed overflow bug when the size of the
partition is larger than maximum signed 32-bit integer of logical(?)
sectors. (2 TiB for a sector size of 512 bytes). It reports the
partition size as minus size and creates a 1 TiB file system.
GParted wants a single maximum file system size and the code is not
ready for a differing maximum file system size for different sector
sizes.
In fat16::create() could specify larger "logical" sector sizes to
mkfs.fat when the partition is larger than 2 TiB to allow maximum volume
size to be increased further. However that will take a lot of cross
platform testing to ensure that all sorts of devices support "logical"
sector sizes other than 512 bytes on devices with a hardware sector size
of 512 bytes. This is too much effort.
Therefore implement a single FAT32 maximum volume size of 2 TiB.
Bug 763896 - GParted not restricting creation of FAT32 volume to maximum
size of 2 TiB
As part of the internal block copy operation 5 initial ranges of blocks
are copied using different block sizes to determine the fastest. Then
the remainder is copied using the fastest block size. Each of these
copies reports progress independently, so during the benchmarking phase
the progress bar flashes 5 times as it goes from 0 to 100% in a fraction
of a second, before showing the progress of the remainder.
This looks bad, so report a single progress bar for all the ranges of
blocks copied in a single copy operation.
Already have variables done and length which track progress within each
copied range; and total_done which records amount copied in previous
ranges. Just add total_length to allow overall progress to be reported.
Bug 762367 - Use a single progress bar for the whole of the internal
copy operation
Previously total_done was updated in copy_thread() after copying of the
blocks, but importantly before the last call to set_progress_info() to
update the progress bar. Having total_done varying during the copy of a
single range of blocks, single call to copy_blocks::copy(), is an
impediment to being able to report a single progress bar across the
whole internal copy operation.
Move updating of total_done to copy() immediately after copy_thread()
completes.
Bug 762367 - Use a single progress bar for the whole of the internal
copy operation
Copying of ntfs is performed using ntfsclone, which writes progress
indication to standard output like this:
# ntfsclone -f /dev/sdb2 /dev/sdb1 2> /dev/null
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 21474832384 bytes (21475 MB)
Current device size: 21474836480 bytes (21475 MB)
Scanning volume ...
100.00 percent completed
Accounting clusters ...
Space in use : 1832 MB (8.5%)
Cloning NTFS ...
100.00 percent completed
Syncing ...
Add ntfsclone progress tracker for ntfsclone command. Deliberately
doesn't stop the progress bar. See comment in ntfs::clone_progress()
for the explanation.
Bug 762366 - Add progress bar to NTFS file system specific copy method
Mike Fleetwood