gparted/include/PipeCapture.h

65 lines
2.4 KiB
C
Raw Permalink Normal View History

/* Copyright (C) 2013 Phillip Susi
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
#ifndef GPARTED_PIPECAPTURE_H
#define GPARTED_PIPECAPTURE_H
Improve the performance of PipeCapture for large output (#777973) A user had a very corrupted FAT file system such that fsck.fat produced 122 MiB of output. GParted has to read this output to get the file system usage information. However GParted takes more than 48 hours to read the 122 MiB of output, while using 100% CPU time and is unresponsive for the duration. Modified fsck.fat to output just the first 1 MiB of output and used perf to capture performance data from GParted reading that output: # perf -g -F 1999 -- ./gpartedbin # perf report --stdio 67.84% Glib::ustring::replace [4.23s] 17.67% g_utf8_pointer_to_offset [1.10s] 8.48% g_utf8_offset_to_pointer [0.53s] [ 6.01% (everything else) ] [0.38s] [100.00% TOTAL ] [6.24s] And to read the first 10 MiB of output the performance figures are: 92.95% Glib::ustring::replace [257.44s] 4.35% g_utf8_pointer_to_offset [ 12.05s] 2.13% g_utf8_offset_to_pointer [ 5.90s] [ 0.58% (everything else) ] [ 1.61s] [100.00% TOTAL ] [277.00s] See how the total time is increasing non-linearly, 44 times longer for only 10 times as much data. This is because of the exponential increase in time spent in Glib::ustring::replace. After a lot of experimentation I came to the conclusion that Glib::ustrings are not appropriate for storing and editing large buffers of data, sizes megabytes and above. The issues are that iterators are invalid after the content changes and replacing UTF-8 characters by index gets exponentially slower as the size of the string increases. Hence the > 48 hours of 100% CPU time to read and apply the line discipline to the 122 MiB of fsck.fat output. See code comment for a more detailed description of the issues found. Rewrote OnReadable() to use Glib::ustrings as little as possible. Instead using buffers and vectors of fixed width data types allowing for fast access using pointers and indexes (converted to pointers by the compiler with simple arithmetic). Again see code comment for a more detailed description of the implementation. Repeating the performance capture with the new code for the first 1 MiB of fsck.fat output: 63.34% memcpy [0.35s] [ 36.66% (everything else) ] [0.21s] [100.00% TOTAL ] [0.56s] And for the first 10 MiB of fsck.fat output: 96.66% memcpy [63.60s] [ 3.34% (everything else) ] [ 2.20s] [100.00% TOTAL ] [65.80s] Simple timings taken to read portions of the fsck.fat output (when not using perf): 1 MiB 10 MiB 122 MiB old code : 6.2 sec 277 sec > 48 hours (4:37) new code : 0.6 sec 66 sec 17262 sec (1:06) (4:47:42) Performance of the code is still non-linear because of the assignment of the ever growing capturebuf to callerbuf for every block of input read. This is required to generate a consistent Glib::ustring copy of the input for the update callback. However this is much faster than before, and I have a plan for further improvements. Bug 777973 - Segmentation fault on bad disk
2017-03-12 08:21:20 -06:00
#include <string>
#include <vector>
#include <stddef.h> // typedef size_t
#include <glib.h> // typedef gunichar
#include <glibmm/ustring.h>
Improve the performance of PipeCapture for large output (#777973) A user had a very corrupted FAT file system such that fsck.fat produced 122 MiB of output. GParted has to read this output to get the file system usage information. However GParted takes more than 48 hours to read the 122 MiB of output, while using 100% CPU time and is unresponsive for the duration. Modified fsck.fat to output just the first 1 MiB of output and used perf to capture performance data from GParted reading that output: # perf -g -F 1999 -- ./gpartedbin # perf report --stdio 67.84% Glib::ustring::replace [4.23s] 17.67% g_utf8_pointer_to_offset [1.10s] 8.48% g_utf8_offset_to_pointer [0.53s] [ 6.01% (everything else) ] [0.38s] [100.00% TOTAL ] [6.24s] And to read the first 10 MiB of output the performance figures are: 92.95% Glib::ustring::replace [257.44s] 4.35% g_utf8_pointer_to_offset [ 12.05s] 2.13% g_utf8_offset_to_pointer [ 5.90s] [ 0.58% (everything else) ] [ 1.61s] [100.00% TOTAL ] [277.00s] See how the total time is increasing non-linearly, 44 times longer for only 10 times as much data. This is because of the exponential increase in time spent in Glib::ustring::replace. After a lot of experimentation I came to the conclusion that Glib::ustrings are not appropriate for storing and editing large buffers of data, sizes megabytes and above. The issues are that iterators are invalid after the content changes and replacing UTF-8 characters by index gets exponentially slower as the size of the string increases. Hence the > 48 hours of 100% CPU time to read and apply the line discipline to the 122 MiB of fsck.fat output. See code comment for a more detailed description of the issues found. Rewrote OnReadable() to use Glib::ustrings as little as possible. Instead using buffers and vectors of fixed width data types allowing for fast access using pointers and indexes (converted to pointers by the compiler with simple arithmetic). Again see code comment for a more detailed description of the implementation. Repeating the performance capture with the new code for the first 1 MiB of fsck.fat output: 63.34% memcpy [0.35s] [ 36.66% (everything else) ] [0.21s] [100.00% TOTAL ] [0.56s] And for the first 10 MiB of fsck.fat output: 96.66% memcpy [63.60s] [ 3.34% (everything else) ] [ 2.20s] [100.00% TOTAL ] [65.80s] Simple timings taken to read portions of the fsck.fat output (when not using perf): 1 MiB 10 MiB 122 MiB old code : 6.2 sec 277 sec > 48 hours (4:37) new code : 0.6 sec 66 sec 17262 sec (1:06) (4:47:42) Performance of the code is still non-linear because of the assignment of the ever growing capturebuf to callerbuf for every block of input read. This is required to generate a consistent Glib::ustring copy of the input for the update callback. However this is much faster than before, and I have a plan for further improvements. Bug 777973 - Segmentation fault on bad disk
2017-03-12 08:21:20 -06:00
#include <glibmm/refptr.h>
#include <glibmm/iochannel.h>
namespace GParted {
// captures output pipe of subprocess into a ustring and emits a signal on eof
class PipeCapture
{
public:
PipeCapture( int fd, Glib::ustring &buffer );
~PipeCapture();
Improve the performance of PipeCapture for large output (#777973) A user had a very corrupted FAT file system such that fsck.fat produced 122 MiB of output. GParted has to read this output to get the file system usage information. However GParted takes more than 48 hours to read the 122 MiB of output, while using 100% CPU time and is unresponsive for the duration. Modified fsck.fat to output just the first 1 MiB of output and used perf to capture performance data from GParted reading that output: # perf -g -F 1999 -- ./gpartedbin # perf report --stdio 67.84% Glib::ustring::replace [4.23s] 17.67% g_utf8_pointer_to_offset [1.10s] 8.48% g_utf8_offset_to_pointer [0.53s] [ 6.01% (everything else) ] [0.38s] [100.00% TOTAL ] [6.24s] And to read the first 10 MiB of output the performance figures are: 92.95% Glib::ustring::replace [257.44s] 4.35% g_utf8_pointer_to_offset [ 12.05s] 2.13% g_utf8_offset_to_pointer [ 5.90s] [ 0.58% (everything else) ] [ 1.61s] [100.00% TOTAL ] [277.00s] See how the total time is increasing non-linearly, 44 times longer for only 10 times as much data. This is because of the exponential increase in time spent in Glib::ustring::replace. After a lot of experimentation I came to the conclusion that Glib::ustrings are not appropriate for storing and editing large buffers of data, sizes megabytes and above. The issues are that iterators are invalid after the content changes and replacing UTF-8 characters by index gets exponentially slower as the size of the string increases. Hence the > 48 hours of 100% CPU time to read and apply the line discipline to the 122 MiB of fsck.fat output. See code comment for a more detailed description of the issues found. Rewrote OnReadable() to use Glib::ustrings as little as possible. Instead using buffers and vectors of fixed width data types allowing for fast access using pointers and indexes (converted to pointers by the compiler with simple arithmetic). Again see code comment for a more detailed description of the implementation. Repeating the performance capture with the new code for the first 1 MiB of fsck.fat output: 63.34% memcpy [0.35s] [ 36.66% (everything else) ] [0.21s] [100.00% TOTAL ] [0.56s] And for the first 10 MiB of fsck.fat output: 96.66% memcpy [63.60s] [ 3.34% (everything else) ] [ 2.20s] [100.00% TOTAL ] [65.80s] Simple timings taken to read portions of the fsck.fat output (when not using perf): 1 MiB 10 MiB 122 MiB old code : 6.2 sec 277 sec > 48 hours (4:37) new code : 0.6 sec 66 sec 17262 sec (1:06) (4:47:42) Performance of the code is still non-linear because of the assignment of the ever growing capturebuf to callerbuf for every block of input read. This is required to generate a consistent Glib::ustring copy of the input for the update callback. However this is much faster than before, and I have a plan for further improvements. Bug 777973 - Segmentation fault on bad disk
2017-03-12 08:21:20 -06:00
void connect_signal();
sigc::signal<void> signal_eof;
sigc::signal<void> signal_update;
Improve the performance of PipeCapture for large output (#777973) A user had a very corrupted FAT file system such that fsck.fat produced 122 MiB of output. GParted has to read this output to get the file system usage information. However GParted takes more than 48 hours to read the 122 MiB of output, while using 100% CPU time and is unresponsive for the duration. Modified fsck.fat to output just the first 1 MiB of output and used perf to capture performance data from GParted reading that output: # perf -g -F 1999 -- ./gpartedbin # perf report --stdio 67.84% Glib::ustring::replace [4.23s] 17.67% g_utf8_pointer_to_offset [1.10s] 8.48% g_utf8_offset_to_pointer [0.53s] [ 6.01% (everything else) ] [0.38s] [100.00% TOTAL ] [6.24s] And to read the first 10 MiB of output the performance figures are: 92.95% Glib::ustring::replace [257.44s] 4.35% g_utf8_pointer_to_offset [ 12.05s] 2.13% g_utf8_offset_to_pointer [ 5.90s] [ 0.58% (everything else) ] [ 1.61s] [100.00% TOTAL ] [277.00s] See how the total time is increasing non-linearly, 44 times longer for only 10 times as much data. This is because of the exponential increase in time spent in Glib::ustring::replace. After a lot of experimentation I came to the conclusion that Glib::ustrings are not appropriate for storing and editing large buffers of data, sizes megabytes and above. The issues are that iterators are invalid after the content changes and replacing UTF-8 characters by index gets exponentially slower as the size of the string increases. Hence the > 48 hours of 100% CPU time to read and apply the line discipline to the 122 MiB of fsck.fat output. See code comment for a more detailed description of the issues found. Rewrote OnReadable() to use Glib::ustrings as little as possible. Instead using buffers and vectors of fixed width data types allowing for fast access using pointers and indexes (converted to pointers by the compiler with simple arithmetic). Again see code comment for a more detailed description of the implementation. Repeating the performance capture with the new code for the first 1 MiB of fsck.fat output: 63.34% memcpy [0.35s] [ 36.66% (everything else) ] [0.21s] [100.00% TOTAL ] [0.56s] And for the first 10 MiB of fsck.fat output: 96.66% memcpy [63.60s] [ 3.34% (everything else) ] [ 2.20s] [100.00% TOTAL ] [65.80s] Simple timings taken to read portions of the fsck.fat output (when not using perf): 1 MiB 10 MiB 122 MiB old code : 6.2 sec 277 sec > 48 hours (4:37) new code : 0.6 sec 66 sec 17262 sec (1:06) (4:47:42) Performance of the code is still non-linear because of the assignment of the ever growing capturebuf to callerbuf for every block of input read. This is required to generate a consistent Glib::ustring copy of the input for the update callback. However this is much faster than before, and I have a plan for further improvements. Bug 777973 - Segmentation fault on bad disk
2017-03-12 08:21:20 -06:00
private:
bool OnReadable( Glib::IOCondition condition );
static gboolean _OnReadable( GIOChannel *source,
GIOCondition condition,
gpointer data );
static void append_unichar_vector_to_utf8( std::string & str,
const std::vector<gunichar> & ucvec );
Refactor ::OnReadable() creating get_utf8_char_validated() (#136) Extract call to GLib's g_utf8_get_char_validated() and the associated workaround to also read NUL characters into a separate function to make PipeCapture::OnReadable() a little smaller and simpler, so easier to understand. Add max_len > 0 clause into get_utf8_char_validated() like this: if (uc == UTF8_PARTIAL && max_len > 0) so that the NUL character reading workaround is only applied when max_len specifies the maximum number of bytes to read, rather than when -1 specifies reading a NUL termination string. This makes get_utf8_char_validated() a complete wrapper of g_utf8_get_char_validated() [1], even though GParted always specifies the maximum number of bytes to read. No longer describe the inability to read NUL characters as a bug [2] since the GLib author's said it wasn't [3]. [1] GLib Reference Manual, Unicode Manipulation Functions, g_utf8_get_char_validated () https://developer.gnome.org/glib/stable/glib-Unicode-Manipulation.html#g-utf8-get-char-validated [2] 8dbbb47ce2db0ee733ff909c1ead2f4de9475596 Workaround g_utf8_get_char_validate() bug with embedded NUL bytes (#777973) [3] Bug 780095 - g_utf8_get_char_validated() stopping at nul byte even for length specified buffers https://bugzilla.gnome.org/show_bug.cgi?id=780095#18 "If g_utf8_get_char_validated() encounters a nul byte in the middle of a string of given longer length, it returns -2, indicating a partial gunichar. That is not the obvious behaviour, but since g_utf8_get_char_validated() has been API for a long time, the behaviour cannot be changed. " Closes #136 - 1.2.0: test suite is failing in test_PipeCapture
2021-02-20 16:30:41 -07:00
static gunichar get_utf8_char_validated(const char *p, gssize max_len);
static int utf8_char_length( unsigned char firstbyte );
Improve the performance of PipeCapture for large output (#777973) A user had a very corrupted FAT file system such that fsck.fat produced 122 MiB of output. GParted has to read this output to get the file system usage information. However GParted takes more than 48 hours to read the 122 MiB of output, while using 100% CPU time and is unresponsive for the duration. Modified fsck.fat to output just the first 1 MiB of output and used perf to capture performance data from GParted reading that output: # perf -g -F 1999 -- ./gpartedbin # perf report --stdio 67.84% Glib::ustring::replace [4.23s] 17.67% g_utf8_pointer_to_offset [1.10s] 8.48% g_utf8_offset_to_pointer [0.53s] [ 6.01% (everything else) ] [0.38s] [100.00% TOTAL ] [6.24s] And to read the first 10 MiB of output the performance figures are: 92.95% Glib::ustring::replace [257.44s] 4.35% g_utf8_pointer_to_offset [ 12.05s] 2.13% g_utf8_offset_to_pointer [ 5.90s] [ 0.58% (everything else) ] [ 1.61s] [100.00% TOTAL ] [277.00s] See how the total time is increasing non-linearly, 44 times longer for only 10 times as much data. This is because of the exponential increase in time spent in Glib::ustring::replace. After a lot of experimentation I came to the conclusion that Glib::ustrings are not appropriate for storing and editing large buffers of data, sizes megabytes and above. The issues are that iterators are invalid after the content changes and replacing UTF-8 characters by index gets exponentially slower as the size of the string increases. Hence the > 48 hours of 100% CPU time to read and apply the line discipline to the 122 MiB of fsck.fat output. See code comment for a more detailed description of the issues found. Rewrote OnReadable() to use Glib::ustrings as little as possible. Instead using buffers and vectors of fixed width data types allowing for fast access using pointers and indexes (converted to pointers by the compiler with simple arithmetic). Again see code comment for a more detailed description of the implementation. Repeating the performance capture with the new code for the first 1 MiB of fsck.fat output: 63.34% memcpy [0.35s] [ 36.66% (everything else) ] [0.21s] [100.00% TOTAL ] [0.56s] And for the first 10 MiB of fsck.fat output: 96.66% memcpy [63.60s] [ 3.34% (everything else) ] [ 2.20s] [100.00% TOTAL ] [65.80s] Simple timings taken to read portions of the fsck.fat output (when not using perf): 1 MiB 10 MiB 122 MiB old code : 6.2 sec 277 sec > 48 hours (4:37) new code : 0.6 sec 66 sec 17262 sec (1:06) (4:47:42) Performance of the code is still non-linear because of the assignment of the ever growing capturebuf to callerbuf for every block of input read. This is required to generate a consistent Glib::ustring copy of the input for the update callback. However this is much faster than before, and I have a plan for further improvements. Bug 777973 - Segmentation fault on bad disk
2017-03-12 08:21:20 -06:00
Glib::RefPtr<Glib::IOChannel> channel; // Wrapper around fd
char * readbuf; // Bytes read from IOChannel (fd)
size_t fill_offset; // Filling offset into readbuf
std::vector<gunichar> linevec; // Current line stored as UCS-4 characters
size_t cursor; // Cursor position index into linevec
std::string capturebuf; // Captured output as UTF-8 characters
size_t line_start; // Index into bytebuf where current line starts
Glib::ustring & callerbuf; // Reference to caller supplied buffer
bool callerbuf_uptodate; // Has capturebuf changed since last copied to callerbuf?
};
} // namepace GParted
#endif /* GPARTED_PIPECAPTURE_H */