Make especially the Volume Identifier length limit code simpler to
understand and therefore easier to maintain.
Bug 784533 - Add support for UDF file system
UDF label is stored in the Logical Volume Identifier which has space for
either 126 Latin1 or 63 UCS-2 characters. For compatibility reasons
with older versions of blkid, the possibly truncated UDF label is also
stored in the Volume Identifier which only has space for 30 Latin1 or 15
UCS-2 characters.
Because versions of mkudffs prior to 1.1 damage the label if it contains
non-ASCII characters, make sure GParted does not call such versions of
mkudffs with a non-ASCII character label.
Bug 784533 - Add support for UDF file system
Make distcheck is failing like:
<snip>
Make distcheck
Making distclean in tests
make[2]: Entering directory
'/home/gedakc/workspace/gparted/gparted-0.29.0-beta1/_build/sub/tests'
Makefile:658: ../src/.deps/BlockSpecial.Po: No such file or directory
Makefile:659: ../src/.deps/PipeCapture.Po: No such file or directory
make[2]: *** No rule to make target '../src/.deps/PipeCapture.Po'. Stop.
make[2]: Leaving directory
'/home/gedakc/workspace/gparted/gparted-0.29.0-beta1/_build/sub/tests'
Makefile:596: recipe for target 'distclean-recursive' failed
make[1]: *** [distclean-recursive] Error 1
make[1]: Leaving directory
'/home/gedakc/workspace/gparted/gparted-0.29.0-beta1/_build/sub'
Makefile:805: recipe for target 'distcheck' failed
make: *** [distcheck] Error 1
Make distcheck started failing like this with this commit:
f31808989a
Silence subdir-objects warning from automake 1.14 (#781978)
However, also need to be referring to source files from a sibling
directory like this from tests/Makefile.am:
test_BlockSpecial_SOURCES = test_BlockSpecial.cc ../src/BlockSpecial.cc
^^^^^^^
First introduced by this commit:
c37be28148
Add unit tests for BlockSpecial constructors and internal caching (#781978)
This failure, if not exactly common, has been seen a number of times
before. Using this hint from a GNU Automake forum post:
converting to subdir-objects
http://gnu-automake.7480.n7.nabble.com/converting-to-subdir-objects-td21724.html
"I had the same issue and wouldn't care unless Automake 1.15 started
to warn about "source file in a subdirectory". In my case 'bar.o' is
also built from 'foo/Makefile' so I decided to skip using 'bar.c' as
SOURCE and instead link in 'foo/bar.o'.
http://dev.thep.lu.se/svndigest/changeset/1581
Cheers,
Peter
"
Fix by avoiding referring to source files from a sibling directory when
building the unit test programs, instead refer to object files instead.
Note that the automake subdir-objects option is still required to
silence the warning because of the use of source files in subdirectories
by lib/gtest/Makefile.am.
Bug 785308 - make distcheck fails with *.Po: No such file or directory
Add support for detecting UDF file systems and formatting hard disks
with revision 2.01 UDF file systems using udftools. Formatting optical
disks or any other media types is not supported yet. Changing label or
UUID after formatting is not supported as the tools do not yet exist.
Bug 784533 - Add support for UDF file system
Btrfstune changed it's command line help output from being written on
stderr to being written on stdout in the following commit first included
in btrfs-progs 4.7.2 released 05-Sep-2016. This breaks GParted
detection of the change UUID capability. Found on the very latest
distributions: up to date Arch Linux and Fedora 26 Beta. Fix this.
https://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git/commit/?id=57d1cbd867311d99a2ae5e3cdcffd04
btrfs-progs: print help test to stdout
Bug 784467 - No longer detecting btrfs change UUID capability
Configuring GParted on a machine with automake >= 1.14 produces this
warning:
$ ./autogen.sh
...
lib/gtest/Makefile.am:58: warning: source file 'src/gtest-all.cc' is in a subdirectory,
lib/gtest/Makefile.am:58: but option 'subdir-objects' is disabled
automake: warning: possible forward-incompatibility.
automake: At least a source file is in a subdirectory, but the 'subdir-objects'
automake: automake option hasn't been enabled. For now, the corresponding output
automake: object file(s) will be placed in the top-level directory. However,
automake: this behaviour will change in future Automake versions: they will
automake: unconditionally cause object files to be placed in the same subdirectory
automake: of the corresponding sources.
automake: You are advised to start using 'subdir-objects' option throughout your
automake: project, to avoid future incompatibilities.
lib/gtest/Makefile.am:60: warning: source file 'src/gtest_main.cc' is in a subdirectory,
lib/gtest/Makefile.am:60: but option 'subdir-objects' is disabled
lib/gtest/Makefile.am: installing './depcomp'
tests/Makefile.am:22: warning: source file '../src/BlockSpecial.cc' is in a subdirectory,
tests/Makefile.am:22: but option 'subdir-objects' is disabled
tests/Makefile.am:23: warning: source file '../src/PipeCapture.cc' is in a subdirectory,
tests/Makefile.am:23: but option 'subdir-objects' is disabled
...
From the automake 1.14 NEWS file:
http://git.savannah.gnu.org/cgit/automake.git/tree/NEWS?h=v1.14#n125
"The next major Automake version (2.0) will unconditionally activate
the 'subdir-objects' option. In order to smooth out the transition,
we now give a warning (in the category 'unsupported') whenever a
source file is present in a subdirectory but the 'subdir-object' is
not enabled. For example, the following usage will trigger such a
warning:
bin_PROGRAMS = sub/foo
sub_foo_SOURCES = sub/main.c sub/bar.c
"
Set the subdir-objects automake option, silencing the warning.
Bug 781978 - Add Google Test C++ test framework
Internal file system detection is broken for detection of LVM2 PVs
because it reports finding LVM2 PV even when only the first magic is
found. Prepare a partition like this:
# lvm pvcreate /dev/sdb1
# hexdump -C /dev/sdb1 > hexdump-1.txt
Clear the second magic:
# python
f = open("/dev/sdb1","w")
f.seek(0x218)
f.write("\x00"*4)
f.close()
quit()
# hexdump -C /dev/sdb1 > hexdump-2.txt
# diff -u hexdump-[12].txt
00000200 4c 41 42 45 4c 4f 4e 45 01 00 00 00 00 00 00 00 |LABELONE........|
-00000210 58 69 83 e1 20 00 00 00 4c 56 4d 32 20 30 30 31 |Xi.. ...LVM2 001|
+00000210 58 69 83 e1 20 00 00 00 00 00 00 00 20 30 30 31 |Xi.. ....... 001|
^^ ^^ ^^ ^^ ^^^^
00000220 52 4b 31 73 50 77 49 66 6a 72 55 4c 6a 4d 30 58 |RK1sPwIfjrULjM0X|
GParted still detects this as an LVM2 PV even though lvm and blkid do
not.
Correct logic error and only perform memcmp() with the second magic when
the second signature is non-NULL.
Bug 783997 - GParted detecting LVM2 PV regardless whether second magic
matches or not
For large output a lot of time is used copying capturebuf to callerbuf
to provide a Glib::ustring copy of the buffer for the update callback.
However update callbacks are only used when commands are run to apply
operations by FileSystem::execute_command() and their output is
incrementally displayed in the UI. Whereas update callbacks are never
used when commands are used to query information via
Utils::execute_command().
Stop performing interim copying of capturebuf to callerbuf when there
are no update callbacks registered as it is unnecessary.
Time to read portions of the recorded fsck.fat output via
fat16::set_used_sectors() and intermediate copies aren't required:
1 MiB 10 MiB 122 MiB
old code : 0.074 sec 1.41 sec 210 sec [3:30]
new code : 0.063 sec 0.56 sec 6.57 sec
Bug 777973 - Segmentation fault on bad disk
For large output PipeCapture spends all it's time copying the capturebuf
to callerbuf to provide a consistent view for any registered update
callbacks. This overhead is dependant of the size of the ever growing
captured output and the number of times OnReadable() is called.
Therefore increase the maximum read size to exponentially reduce this
overhead.
Time taken to read varying amounts of fsck.fat output with various
read buffer sizes:
1 MiB 10 MiB 122 MiB
512b : 0.60 sec 65 sec [1:05] 17262 sec [4:47:42]
4096b : 0.19 sec 13 sec 2157 sec [ 35:57]
64K : 0.07 sec 1.4 sec 210 sec [ 3:30]
Note that this is only increasing the maximum size that can be read from
the output of the external command. If the command produces it's output
slowly, such as the with progress reporting commands like mkfs.ext4,
then only the available number of bytes is read reporting the next
progress increment. However if the command produces it's output
quickly, such as when testing this bug using a modified fsck.fat
concatenating the 122 MiB of pre-recorded output, then full buffer reads
are performed.
To ensure that a single call to OnReadable() couldn't block the UI too
long, the time taken for OnReadable() to process a full buffer of
various sizes was recorded as:
512b : 0.031 milliseconds
4096b : 0.188 milliseconds
64K : 3.576 milliseconds
Adding this amount of processing time in the UI under normal
circumstances is not a problem.
As the captured output increases, the time taken by OnReadable() becomes
dominated by the time taken to copy the ever increasing capture buffer
to handle it's expansion and to copy it to the caller buffer for the
update callback. At the end of the 122M captured fsck.fat output
OnReadable() takes 350 milliseconds per call. This is not a problem
because this is an extreme case in which GParted is already hung and
increasing the buffer size is reducing the overall hang time from over
4 hours to a few minutes.
Bug 777973 - Segmentation fault on bad disk
If PipeCapture reads a NUL byte in the middle of what is expected to be
a multi-byte UTF-8 character then PipeCapture either returns the
captured characters to the previous update or loops forever depending on
whether the end of the stream is encountered before the read buffer is
full or not. This is equivalent to saying whether the NUL byte occurs
within the last 512 bytes of the output or not.
This is caused by a bug in g_utf8_get_char_validated() reporting that a
partial UTF-8 character has been found when the NUL byte is encountered
in the middle of a multi-byte character even though more bytes are
available in the length specified buffer. g_utf8_get_char_validated()
is always stopping at the NUL byte assuming it is working with a NUL
terminated string.
Workaround this by checking for g_utf8_get_char_validated() claiming a
partial UTF-8 character has been found when in fact there are at least
enough bytes in the read buffer to instead determine that it is really
an invalid UTF-8 character.
Reference:
Bug 780095 - g_utf8_get_char_validated() stopping at nul byte even
for length specified buffers
https://bugzilla.gnome.org/show_bug.cgi?id=780095
Bug 777973 - Segmentation fault on bad disk
Test that binary data that happens to be the start of a multi-byte UTF-8
character but then contains a NUL byte is successfully read. The test
currently detects failure thus:
$ ./test_PipeCapture
...
[ RUN ] PipeCaptureTest.ReadNULByteInMiddleOfMultiByteUTF8Character
test_PipeCapture.cc:346: Failure
Expected: expectedstr
Of length: 7
To be equal to: capturedstr.raw()
Of length: 0
With first binary difference:
< 0x00000000 "._45678" 00 5F 34 35 36 37 38
--
> 0x00000000 ""
[ FAILED ] PipeCaptureTest.ReadNULByteInMiddleOfMultiByteUTF8Character (0 ms)
...
Bug 777973 - Segmentation fault on bad disk
If PipeCapture reads a NUL character, a valid UTF-8 character, it causes
GParted to allocate all available memory and crash. The while loop in
PipeCapture::OnReadable() loops forever reading the same NUL character
from readbuf because g_utf8_find_next_char() doesn't advance past it.
Hence an infinite number of NUL characters are added to the current
line, linevec.
Workaround this by checking for this failure case of
g_utf8_find_next_char() and increment past the NUL character.
This is actually a bug recently fixed in glib 2.49.3 released
2016-07-17. References:
* Bug 547200 - g_utf8_find_next_char() issues
https://bugzilla.gnome.org/show_bug.cgi?id=547200
* https://git.gnome.org/browse/glib/commit/?id=e0e652e4032a181d4f0b0a12aeddf0678b7a3c04
Fix a corner-case in g_utf8_find_next_char
In the case that *p is '\0', we should return p + 1, not p.
This change allows to simplify g_utf8_find_next_char a bit.
Bug 777973 - Segmentation fault on bad disk
Test currently detects failure thus:
$ ./test_PipeCapture
...
[ RUN ] PipeCaptureTest.ReadEmbeddedNULCharacter
unknown file: Failure
C++ exception with description "std::bad_alloc" thrown in the test body.
[ FAILED ] PipeCaptureTest.ReadEmbeddedNULCharacter (31917 ms)
...
Bug 777973 - Segmentation fault on bad disk
PipeCapture::OnReadable() has been almost completely re-written but this
test is still failing thus:
$ ./test_PipeCapture
...
[ RUN ] PipeCaptureTest.MinimalBinaryCrash777973
test_PipeCapture.cc:313: Failure
Expected: inputstr
Of length: 27
To be equal to: capturedstr.raw()
Of length: 26
With first binary difference:
< 0x00000010 "...!......." A9 C2 A0 21 E2 95 9F E2 88 A9 C2
--
> 0x00000010 "...!......" A9 C2 A0 21 E2 95 9F E2 88 A9
[ FAILED ] PipeCaptureTest.MinimalBinaryCrash777973 (0 ms)
...
The OnReadable() code specifically skips invalid bytes which aren't part
of valid UTF-8 characters, because they can't be displayed to the user.
The final C2 byte is the start of a multi-byte UTF-8 character, so on
it's own is invalid. Therefore PipeCapture skips it. Update expected
string accordingly. Now the test passes.
Bug 777973 - Segmentation fault on bad disk
A user had a very corrupted FAT file system such that fsck.fat produced
122 MiB of output. GParted has to read this output to get the file
system usage information. However GParted takes more than 48 hours to
read the 122 MiB of output, while using 100% CPU time and is
unresponsive for the duration.
Modified fsck.fat to output just the first 1 MiB of output and used perf
to capture performance data from GParted reading that output:
# perf -g -F 1999 -- ./gpartedbin
# perf report --stdio
67.84% Glib::ustring::replace [4.23s]
17.67% g_utf8_pointer_to_offset [1.10s]
8.48% g_utf8_offset_to_pointer [0.53s]
[ 6.01% (everything else) ] [0.38s]
[100.00% TOTAL ] [6.24s]
And to read the first 10 MiB of output the performance figures are:
92.95% Glib::ustring::replace [257.44s]
4.35% g_utf8_pointer_to_offset [ 12.05s]
2.13% g_utf8_offset_to_pointer [ 5.90s]
[ 0.58% (everything else) ] [ 1.61s]
[100.00% TOTAL ] [277.00s]
See how the total time is increasing non-linearly, 44 times longer for
only 10 times as much data. This is because of the exponential increase
in time spent in Glib::ustring::replace.
After a lot of experimentation I came to the conclusion that
Glib::ustrings are not appropriate for storing and editing large buffers
of data, sizes megabytes and above. The issues are that iterators are
invalid after the content changes and replacing UTF-8 characters by
index gets exponentially slower as the size of the string increases.
Hence the > 48 hours of 100% CPU time to read and apply the line
discipline to the 122 MiB of fsck.fat output. See code comment for a
more detailed description of the issues found.
Rewrote OnReadable() to use Glib::ustrings as little as possible.
Instead using buffers and vectors of fixed width data types allowing for
fast access using pointers and indexes (converted to pointers by the
compiler with simple arithmetic). Again see code comment for a more
detailed description of the implementation.
Repeating the performance capture with the new code for the first 1 MiB
of fsck.fat output:
63.34% memcpy [0.35s]
[ 36.66% (everything else) ] [0.21s]
[100.00% TOTAL ] [0.56s]
And for the first 10 MiB of fsck.fat output:
96.66% memcpy [63.60s]
[ 3.34% (everything else) ] [ 2.20s]
[100.00% TOTAL ] [65.80s]
Simple timings taken to read portions of the fsck.fat output (when not
using perf):
1 MiB 10 MiB 122 MiB
old code : 6.2 sec 277 sec > 48 hours
(4:37)
new code : 0.6 sec 66 sec 17262 sec
(1:06) (4:47:42)
Performance of the code is still non-linear because of the assignment
of the ever growing capturebuf to callerbuf for every block of input
read. This is required to generate a consistent Glib::ustring copy of
the input for the update callback. However this is much faster than
before, and I have a plan for further improvements.
Bug 777973 - Segmentation fault on bad disk
Initialise capture string with sacrificial text before each test.
Existing tests confirm this is cleared first by checking the captured
string matches the input string.
Bug 777973 - Segmentation fault on bad disk
Seems more logical to initially clear the output capture buffer in a
single location in the PipeCapture class which reads the command output
into said buffer, rather than each calling site before the PipeCapture
objects are constructed.
Bug 777973 - Segmentation fault on bad disk
A user had a very corrupted FAT file system and when GParted was reading
the file system usage it would core dump. The tip of the stack trace
was:
#0 Glib::ustring_Iterator<__gnu_cxx::__normal_iterator<char*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::operator++()
at /usr/include/glibmm-2.4/glibmm/ustring.h line 957
#1 Glib::ustring_Iterator<__gnu_cxx::__normal_iterator<char*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::operator++(int)
at /usr/include/glibmm-2.4/glibmm/ustring.h line 965
#2 GParted::PipeCapture::OnReadable(Glib::IOCondition)
at PipeCapture.cc line 63
#3 GParted::PipeCapture::_OnReadable(_GIOChannel*, GIOCondition, void*)
at PipeCapture.cc line 45
GParted uses 'fsck.fat -n -v /dev/PTN' to get the file system usage.
However because the file system was so corrupted it was reporting every
file name as being damaged, including the file names being reported as
binary data. PipeCapture::OnReadable() reads up to 512 bytes at a time
and then uses a Glib::ustring iterator to loop over the UTF-8
characters, but a Glib::ustring iterator is explicitly not capable of
reading binary data [1]. With invalid UTF-8 bytes the code continued to
read beyond the end of the string until GParted crashed with a
segmentation violation.
Fix by accessing the read string by index instead of by iterator.
[1] Quote from the Glib::ustring_Iterator<T> Class Template Reference:
https://developer.gnome.org/glibmm/stable/classGlib_1_1ustring__Iterator.html
"The Glib::ustring iterated over must contain only valid UTF-8 data.
If it does not, operator++(), operator--() and operator*() may make
accesses outside the bounds of the string."
Bug 777973 - Segmentation fault on bad disk
A user had a very corrupt FAT file system and fsck.fat was reporting
binary data in file names with errors. GParted crashed reading this
output. Boiled down the problematic fsck.fat output to a sample 27
bytes which still triggers a crash and created a test for it. The test
crashes test_PipeCapture program too.
$ ./test_PipeCapture
...
[ RUN ] PipeCaptureTest.MinimalBinaryCrash777973
Segmentation fault (core dumped)
$ echo $?
139
However it still produces a non-zero exit status and the autotools test
runner detects this so 'make check' still reports failure.
$ make check
...
make[2]: Entering directory `/home/centos/programming/c/gparted/tests'
PASS: test_dummy
PASS: test_BlockSpecial
../test-driver: line 95: 16152 Segmentation fault "$@" > $log_file 2>&1
FAIL: test_PipeCapture
...
============================================================================
Testsuite summary for gparted 0.28.1-git
============================================================================
# TOTAL: 3
# PASS: 2
# SKIP: 0
# XFAIL: 0
# FAIL: 1
# XPASS: 0
# ERROR: 0
============================================================================
See tests/test-suite.log
Please report to https://bugzilla.gnome.org/enter_bug.cgi?product=gparted
============================================================================
make[2]: *** [test-suite.log] Error 1
make[2]: Leaving directory `/home/centos/programming/c/gparted/tests'
make[1]: *** [check-TESTS] Error 2
make[1]: Leaving directory `/home/centos/programming/c/gparted/tests'
make: *** [check-am] Error 2
$ echo $?
2
Bug 777973 - Segmentation fault on bad disk
Also ensure that the PipeCapture calls registered update callbacks and
that the data so far captured matches the leading portion of the input.
Bug 777973 - Segmentation fault on bad disk
Google Test string comparison asserts are only designed of C style
strings containing printable text of one or more lines with a
terminating NUL character. GParted is crashing when PipeCapture is
reading the binary file names being reported by fsck.fat from a very
corrupted FAT file system. Therefore need to be able to compare and
report differences of binary data stored in C++ std::string and
Glib::ustrings. Write a specific assertion to handle this.
Now these sample tests:
TEST_F( PipeCaptureTest, BinaryStringFailure )
{
inputstr = "AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCC";
capturedstr = "AAAAAAAAAAAAAAAABBBBBBBBBBbbbb";
EXPECT_BINARYSTRINGEQ( inputstr, capturedstr.raw() );
}
TEST_F( PipeCaptureTest, LeadingBinaryStringFailure )
{
inputstr = "The quick brown fox jumps over the lazy dog";
capturedstr = "The quick brown fox\n";
EXPECT_BINARYSTRINGEQ( inputstr.substr( 0, capturedstr.raw().length() ),
capturedstr.raw() );
}
report failure like this:
$ ./test_PipeCapture
...
[ RUN ] PipeCaptureTest.BinaryStringFailure
test_PipeCapture.cc:270: Failure
Expected: inputstr
Of length: 48
To be equal to: capturedstr.raw()
Of length: 30
With first binary difference:
< 0x00000010 "BBBBBBBBBBBBBBBB" 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
--
> 0x00000010 "BBBBBBBBBBbbbb" 42 42 42 42 42 42 42 42 42 42 62 62 62 62
[ FAILED ] PipeCaptureTest.BinaryStringFailure (1 ms)
[ RUN ] PipeCaptureTest.LeadingBinaryStringFailure
test_PipeCapture.cc:278: Failure
Expected: inputstr.substr( 0, capturedstr.raw().length() )
Of length: 20
To be equal to: capturedstr.raw()
Of length: 20
With first binary difference:
< 0x00000010 "fox " 66 6F 78 20
--
> 0x00000010 "fox." 66 6F 78 0A
[ FAILED ] PipeCaptureTest.LeadingBinaryStringFailure (0 ms)
...
Bug 777973 - Segmentation fault on bad disk
Add test sending 1 MiB of ASCII test into PipeCapture to read. This
requires multiple reads and rewites through the pipe.
This is as much a check of the threading in the PipeCaptureTest class as
it is of PipeCapture itself. This is because a single thread might
block reading from a pipe waiting for itself to write to the pipe.
Deadlock. Hence the requirement to use a separate thread for reading
and writing.
Bug 777973 - Segmentation fault on bad disk
On CentOS 6, with glib/glibmm 2.28, running the tests fails thus:
$ ./test_PipeCapture
Running main() from gtest_main.cc
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from PipeCaptureTest
[ RUN ] PipeCaptureTest.EmptyPipe
GLib-ERROR **: The thread system is not yet initialized.
aborting...
Aborted (core dumped)
For glib before version 2.32, the threading system must be explicitly
initialised with a call to g_thread_init(), or the Glibmm
Glib::thread_init() equivalent to prevent this error.
Deprecated thread API, g_thread_init()
https://developer.gnome.org/glib/stable/glib-Deprecated-Thread-APIs.html#g-thread-init
Do this by providing our own main() which also initialises the Glib
threading system, rather using and linking in the Google Test provided
main().
Bug 777973 - Segmentation fault on bad disk
So far just tests sending 0 bytes and a few ASCII bytes into a pipe and
that they are read correctly by the PipeCapture class.
Bug 777973 - Segmentation fault on bad disk
Kukuh Syafaat