[Debian GNUstep maintainers] Bug#1040372: edenmath.app: Aborts with stack smashing when calculation result is large enough
Bernhard Übelacker
bernhardu at mailbox.org
Tue Sep 16 15:45:41 BST 2025
On Wed, 05 Jul 2023 08:38:40 +0300 Yavor Doganov <yavor at gnu.org> wrote:
> Package: edenmath.app
> Version: 1.1.1a-8+b5
> Severity: important
>
> Type "40", then press the button "10^x" (second button from right to
> left on the lowest row); EdenMath aborts with:
> *** stack smashing detected *** terminated.
> Cannot be reproduced when built with -fno-stack-protector. I guess the
> culprit is a buffer overflow in -[EMController updateDisplay] where
> buffer size is limited to 32 bytes.
Hello,
attaching a debugger and stopping where the stack canary gets overwritten
confirms, at least I guess, there is an overflow with variable final_string.
Kind regards,
Bernhard
(rr) reverse-stepi
Hardware watchpoint 2: *0x7ffc73ff6138
Old value = 985413424
New value = 985413376
0x000055e811b99098 in -[EMController updateDisplay] (self=0x55e81a3d65a0, _cmd=<optimized out>) at ./EMController.m:212
212 final_string[j] = c_string[j];
1: x/i $pc
=> 0x55e811b99098 <-[EMController updateDisplay]+696>: mov %al,(%rcx,%rdx,1)
(rr) x/1xg 0x7ffc73ff6138
0x7ffc73ff6138: 0x2d9b1b813abc3700
(rr) bt
#0 0x000055e811b99098 in -[EMController updateDisplay] (self=0x55e81a3d65a0, _cmd=<optimized out>) at ./EMController.m:212
#1 0x00007fa069e80072 in -[NSApplication sendAction:to:from:] (self=<optimized out>, _cmd=<optimized out>, aSelector=0x55e819290d10, aTarget=<optimized out>, sender=0x55e81a3c21d0) at ./Source/NSApplication.m:2277
#2 0x00007fa069eb3c95 in -[NSButton sendAction:to:] (self=0x55e81a3c21d0, _cmd=<optimized out>, theAction=0x55e819290d10, theTarget=0x55e81a3d65a0) at ./Source/NSButton.m:588
#3 0x00007fa069ebfcad in -[NSCell trackMouse:inRect:ofView:untilMouseUp:] (self=self at entry=0x55e81a3c97b0, _cmd=_cmd at entry=0x7fa06a1f48c0 <_OBJC_SELECTOR_TABLE+1728>, theEvent=<optimized out>, theEvent at entry=0x55e81a3bf8f0, cellFrame=..., controlView=controlView at entry=0x55e81a3c21d0, flag=0 '\000') at ./Source/NSCell.m:1807
#4 0x00007fa069ee945b in -[NSControl mouseDown:] (self=0x55e81a3c21d0, _cmd=<optimized out>, theEvent=<optimized out>) at ./Source/NSControl.m:932
#5 0x00007fa06a0380f6 in -[NSWindow sendEvent:] (self=0x55e8193d3120, _cmd=<optimized out>, theEvent=0x55e81a3bf8f0) at ./Source/NSWindow.m:4155
#6 0x00007fa069e8625e in -[NSApplication run] (self=0x55e819206c00, _cmd=<optimized out>) at ./Source/NSApplication.m:1588
#7 0x00007fa069e6608d in NSApplicationMain (argc=<optimized out>, argv=<optimized out>) at ./Source/Functions.m:119
(rr) print final_string
$1 = "10000000000000000303786028427003"
-------------- next part --------------
# minimal Trixie/stable amd64 qemu VM 2025-09-16
apt install xserver-xorg slim jwm edenmath.app edenmath.app-dbgsym libgnustep-gui0.32-dbgsym valgrind gdb rr mc
apt build-dep edenmath.app
mkdir /home/benutzer/source/edenmath.app/orig -p
cd /home/benutzer/source/edenmath.app/orig
apt source edenmath.app
benutzer at debian:~$ EdenMath
*** stack smashing detected ***: terminated
Abgebrochen
benutzer at debian:~$
benutzer at debian:~$ rr EdenMath
rr: Saving execution to trace directory `/home/benutzer/.local/share/rr/EdenMath-0'.
*** stack smashing detected ***: terminated
Abgebrochen
benutzer at debian:~$
benutzer at debian:~$ rr replay -a EdenMath-0
*** stack smashing detected ***: terminated
benutzer at debian:~$
rr replay --debugger-option=-q EdenMath-0
set width 0
set pagination off
directory /home/benutzer/source/edenmath.app/orig/edenmath.app-1.1.1a
display/i $pc
cont
bt
benutzer at debian:~$ rr replay --debugger-option=-q EdenMath-0
Reading symbols from /usr/lib/x86_64-linux-gnu/GNUstep/Applications/EdenMath.app/EdenMath...
Reading symbols from /usr/lib/debug/.build-id/25/747f1390c6412da80b4c133560241af139f086.debug...
Remote debugging using 127.0.0.1:1800
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from /usr/lib/debug/.build-id/da/08404553b441511cbee6b34bc78fe29fd5d14c.debug...
warning: BFD: warning: system-supplied DSO at 0x6fffd000 has a section extending past end of file
warning: Discarding section .replay.text which has an invalid size (27) [in module system-supplied DSO at 0x6fffd000]
0x00007fa06a459440 in _start () from /lib64/ld-linux-x86-64.so.2
(rr) set width 0
(rr) set pagination off
(rr) directory /home/benutzer/source/edenmath.app/orig/edenmath.app-1.1.1a
Source directories searched: /home/benutzer/source/edenmath.app/orig/edenmath.app-1.1.1a:$cdir:$cwd
(rr) display/i $pc
1: x/i $pc
=> 0x7fa06a459440 <_start>: mov %rsp,%rdi
(rr) cont
Continuing.
*** stack smashing detected ***: terminated
Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo at entry=6, no_tid=no_tid at entry=0) at ./nptl/pthread_kill.c:44
warning: 44 ./nptl/pthread_kill.c: Datei oder Verzeichnis nicht gefunden
1: x/i $pc
=> 0x7fa06929e95c <__pthread_kill_implementation+268>: mov %eax,%ebx
(rr) bt
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo at entry=6, no_tid=no_tid at entry=0) at ./nptl/pthread_kill.c:44
#1 0x00007fa06929e9ff in __pthread_kill_internal (threadid=<optimized out>, signo=6) at ./nptl/pthread_kill.c:89
#2 0x00007fa069249cc2 in __GI_raise (sig=sig at entry=6) at ../sysdeps/posix/raise.c:26
#3 0x00007fa0692324ac in __GI_abort () at ./stdlib/abort.c:73
#4 0x00007fa069233291 in __libc_message_impl (fmt=fmt at entry=0x7fa0693b51a4 "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:134
#5 0x00007fa069325995 in __GI___fortify_fail (msg=msg at entry=0x7fa0693b51bc "stack smashing detected") at ./debug/fortify_fail.c:24
#6 0x00007fa069326bb0 in __stack_chk_fail () at ./debug/stack_chk_fail.c:24
#7 0x000055e811b991a7 in -[EMController updateDisplay] (self=<optimized out>, _cmd=<optimized out>) at ./EMController.m:227
#8 0x00007fa069e80072 in -[NSApplication sendAction:to:from:] (self=<optimized out>, _cmd=<optimized out>, aSelector=0x55e819290d10, aTarget=<optimized out>, sender=0x55e81a3c21d0) at ./Source/NSApplication.m:2277
#9 0x00007fa069eb3c95 in -[NSButton sendAction:to:] (self=0x55e81a3c21d0, _cmd=<optimized out>, theAction=0x55e819290d10, theTarget=0x55e81a3d65a0) at ./Source/NSButton.m:588
#10 0x00007fa069ebfcad in -[NSCell trackMouse:inRect:ofView:untilMouseUp:] (self=self at entry=0x55e81a3c97b0, _cmd=_cmd at entry=0x7fa06a1f48c0 <_OBJC_SELECTOR_TABLE+1728>, theEvent=<optimized out>, theEvent at entry=0x55e81a3bf8f0, cellFrame=..., controlView=controlView at entry=0x55e81a3c21d0, flag=0 '\000') at ./Source/NSCell.m:1807
#11 0x00007fa069ee945b in -[NSControl mouseDown:] (self=0x55e81a3c21d0, _cmd=<optimized out>, theEvent=<optimized out>) at ./Source/NSControl.m:932
#12 0x00007fa06a0380f6 in -[NSWindow sendEvent:] (self=0x55e8193d3120, _cmd=<optimized out>, theEvent=0x55e81a3bf8f0) at ./Source/NSWindow.m:4155
#13 0x00007fa069e8625e in -[NSApplication run] (self=0x55e819206c00, _cmd=<optimized out>) at ./Source/NSApplication.m:1588
#14 0x00007fa069e6608d in NSApplicationMain (argc=<optimized out>, argv=<optimized out>) at ./Source/Functions.m:119
#15 0x00007fa069233ca8 in __libc_start_call_main (main=main at entry=0x55e811b971e0 <main>, argc=argc at entry=1, argv=argv at entry=0x7ffc73ff65a8) at ../sysdeps/nptl/libc_start_call_main.h:58
#16 0x00007fa069233d65 in __libc_start_main_impl (main=0x55e811b971e0 <main>, argc=1, argv=0x7ffc73ff65a8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc73ff6598) at ../csu/libc-start.c:360
#17 0x000055e811b97221 in _start ()
(rr) b -[EMController updateDisplay]
Breakpoint 1 at 0x55e811b98de0: file ./EMController.m, line 128.
(rr) reverse-cont
Continuing.
Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo at entry=6, no_tid=no_tid at entry=0) at ./nptl/pthread_kill.c:44
44 in ./nptl/pthread_kill.c
1: x/i $pc
=> 0x7fa06929e95c <__pthread_kill_implementation+268>: mov %eax,%ebx
(rr) reverse-cont
Continuing.
Breakpoint 1, -[EMController updateDisplay] (self=0x55e81a3d65a0, _cmd=0x55e811b9f5c0 <_OBJC_SELECTOR_TABLE+1920>) at ./EMController.m:128
128 {
1: x/i $pc
=> 0x55e811b98de0 <-[EMController updateDisplay]>: endbr64
(rr) stepi
0x000055e811b98de4 128 {
1: x/i $pc
=> 0x55e811b98de4 <-[EMController updateDisplay]+4>: push %r15
(rr)
0x000055e811b98de6 128 {
1: x/i $pc
=> 0x55e811b98de6 <-[EMController updateDisplay]+6>: push %r14
(rr)
0x000055e811b98de8 128 {
1: x/i $pc
=> 0x55e811b98de8 <-[EMController updateDisplay]+8>: push %r13
(rr)
0x000055e811b98dea 134 NSString *true_precision = [[NSString alloc] initWithFormat: @"%s%d%s", y, i-1, z];
1: x/i $pc
=> 0x55e811b98dea <-[EMController updateDisplay]+10>: lea 0x225e(%rip),%r13 # 0x55e811b9b04f
(rr)
0x000055e811b98df1 128 {
1: x/i $pc
=> 0x55e811b98df1 <-[EMController updateDisplay]+17>: push %r12
(rr)
0x000055e811b98df3 129 double current_value = [em getCurrentValue];
1: x/i $pc
=> 0x55e811b98df3 <-[EMController updateDisplay]+19>: lea 0x6896(%rip),%r12 # 0x55e811b9f690 <_OBJC_SELECTOR_TABLE+2128>
(rr)
0x000055e811b98dfa 128 {
1: x/i $pc
=> 0x55e811b98dfa <-[EMController updateDisplay]+26>: push %rbp
(rr)
0x000055e811b98dfb 129 double current_value = [em getCurrentValue];
1: x/i $pc
=> 0x55e811b98dfb <-[EMController updateDisplay]+27>: mov %r12,%rsi
(rr)
0x000055e811b98dfe 128 {
1: x/i $pc
=> 0x55e811b98dfe <-[EMController updateDisplay]+30>: push %rbx
(rr)
0x000055e811b98dff 128 {
1: x/i $pc
=> 0x55e811b98dff <-[EMController updateDisplay]+31>: sub $0x68,%rsp
(rr)
0x000055e811b98e03 129 double current_value = [em getCurrentValue];
1: x/i $pc
=> 0x55e811b98e03 <-[EMController updateDisplay]+35>: mov 0x8(%rdi),%rbp
(rr)
0x000055e811b98e07 128 {
1: x/i $pc
=> 0x55e811b98e07 <-[EMController updateDisplay]+39>: mov %fs:0x28,%rbx
(rr)
0x000055e811b98e10 128 {
1: x/i $pc
=> 0x55e811b98e10 <-[EMController updateDisplay]+48>: mov %rbx,0x58(%rsp)
(rr) x/1xg $rsp + 0x58
0x7ffc73ff6138: 0x4044000000000000
(rr) watch *0x7ffc73ff6138
Hardware watchpoint 2: *0x7ffc73ff6138
(rr) stepi
Hardware watchpoint 2: *0x7ffc73ff6138
Old value = 0
New value = 985413376
0x000055e811b98e15 in -[EMController updateDisplay] (self=0x55e81a3d65a0, _cmd=<optimized out>) at ./EMController.m:128
128 {
1: x/i $pc
=> 0x55e811b98e15 <-[EMController updateDisplay]+53>: mov %rdi,%rbx
(rr) x/1xg 0x7ffc73ff6138
0x7ffc73ff6138: 0x2d9b1b813abc3700
(rr) cont
Continuing.
Hardware watchpoint 2: *0x7ffc73ff6138
Old value = 985413376
New value = 985413424
-[EMController updateDisplay] (self=0x55e81a3d65a0, _cmd=<optimized out>) at ./EMController.m:220
220 new_string = [NSString stringWithFormat:@"%s", final_string];
1: x/i $pc
=> 0x55e811b9909b <-[EMController updateDisplay]+699>: mov %r13,%rdi
(rr) x/1xg 0x7ffc73ff6138
0x7ffc73ff6138: 0x2d9b1b813abc3730
(rr) reverse-stepi
Hardware watchpoint 2: *0x7ffc73ff6138
Old value = 985413424
New value = 985413376
0x000055e811b99098 in -[EMController updateDisplay] (self=0x55e81a3d65a0, _cmd=<optimized out>) at ./EMController.m:212
212 final_string[j] = c_string[j];
1: x/i $pc
=> 0x55e811b99098 <-[EMController updateDisplay]+696>: mov %al,(%rcx,%rdx,1)
(rr) x/1xg 0x7ffc73ff6138
0x7ffc73ff6138: 0x2d9b1b813abc3700
(rr) bt
#0 0x000055e811b99098 in -[EMController updateDisplay] (self=0x55e81a3d65a0, _cmd=<optimized out>) at ./EMController.m:212
#1 0x00007fa069e80072 in -[NSApplication sendAction:to:from:] (self=<optimized out>, _cmd=<optimized out>, aSelector=0x55e819290d10, aTarget=<optimized out>, sender=0x55e81a3c21d0) at ./Source/NSApplication.m:2277
#2 0x00007fa069eb3c95 in -[NSButton sendAction:to:] (self=0x55e81a3c21d0, _cmd=<optimized out>, theAction=0x55e819290d10, theTarget=0x55e81a3d65a0) at ./Source/NSButton.m:588
#3 0x00007fa069ebfcad in -[NSCell trackMouse:inRect:ofView:untilMouseUp:] (self=self at entry=0x55e81a3c97b0, _cmd=_cmd at entry=0x7fa06a1f48c0 <_OBJC_SELECTOR_TABLE+1728>, theEvent=<optimized out>, theEvent at entry=0x55e81a3bf8f0, cellFrame=..., controlView=controlView at entry=0x55e81a3c21d0, flag=0 '\000') at ./Source/NSCell.m:1807
#4 0x00007fa069ee945b in -[NSControl mouseDown:] (self=0x55e81a3c21d0, _cmd=<optimized out>, theEvent=<optimized out>) at ./Source/NSControl.m:932
#5 0x00007fa06a0380f6 in -[NSWindow sendEvent:] (self=0x55e8193d3120, _cmd=<optimized out>, theEvent=0x55e81a3bf8f0) at ./Source/NSWindow.m:4155
#6 0x00007fa069e8625e in -[NSApplication run] (self=0x55e819206c00, _cmd=<optimized out>) at ./Source/NSApplication.m:1588
#7 0x00007fa069e6608d in NSApplicationMain (argc=<optimized out>, argv=<optimized out>) at ./Source/Functions.m:119
#8 0x00007fa069233ca8 in __libc_start_call_main (main=main at entry=0x55e811b971e0 <main>, argc=argc at entry=1, argv=argv at entry=0x7ffc73ff65a8) at ../sysdeps/nptl/libc_start_call_main.h:58
#9 0x00007fa069233d65 in __libc_start_main_impl (main=0x55e811b971e0 <main>, argc=1, argv=0x7ffc73ff65a8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc73ff6598) at ../csu/libc-start.c:360
#10 0x000055e811b97221 in _start ()
(rr) set print repeats 0
(rr) print final_string
$1 = "10000000000000000303786028427003"
(rr) print &final_string
$2 = (char (*)[32]) 0x7ffc73ff6110
(rr) print new_string
$3 = <optimized out>
(rr) print/x $rcx
$4 = 0x7ffc73ff6138
(rr) print/x $rdx
$5 = 0x0
(rr) print/x $al
$6 = 0x30
(rr)
(rr) list 140
135
136 NSString *new_string; // = [[NSString alloc] init];
137
138 // variables for the new algorithm to format numbers properly and eliminate unncessary
139 // '0' from the end of a final number.
140 char final_string[32] = "";
141 int cs_len = 0;
142 int j = 0;
143 int decimal_places = 0;
144 BOOL is_decimal = NO; // 0 is false, 1 is true
145 BOOL is_zero = YES; // is true
146 int new_len = 0;
147 int num_zeros = 0;
148
149 NSString *precision = @"%15.10f"; // default precision
150 NSString *string_value = [NSString stringWithFormat:precision, current_value]; // convert to string with certain string format
151
152 if (i != 0) // if there ARE some set trailing digits like 65.2 or 0.001
153 {
154 NSString *other_value = [NSString stringWithFormat:true_precision, current_value];
155 [displayField setStringValue: other_value];
156 }
157 else // no trailing_digits because it is a number like 6 or it is an answer and
158 // trailing_digits was reset to 0
159 {
160 // loop through the string converted version of the current_value, and cut
161 // off any excess 0's at the end of the number, so 63.20 will appear like 63.2
162
163 current_value = [string_value doubleValue];
164 [string_value getCString: c_string];
165
166 // new algorithm for formating numbers properly on output
167
168 // check to see if there is a decimal place, and if so, how many
169 // decimal places exist
170 cs_len = strlen(c_string);
171
172 for (j = 0; j < cs_len; j++)
173 {
174 if (c_string[j] == '.')
175 {
176 is_decimal = YES;
177 while (j < cs_len)
178 {
179 j++;
180 decimal_places++;
181 }
182 }
183 }
184
185 // if a decimal place exists, go through to get rid of unnecessary 0's at
186 // the end of the number so 65.20 will appear to be 65.2
187 if (is_decimal == YES)
188 {
189 for (j = 0; (j < decimal_places) && (is_zero == YES); j++)
190 {
191 new_len = cs_len - (1 + j);
192 // count the number of 0's at the end
193 if (c_string[new_len] == '0')
194 {
195 num_zeros++;
196 }
197 else if (c_string[new_len] == '.')
198 {
199 num_zeros++;
200 is_zero = NO;
201 }
202 else // otherwise, no more excess 0's to be found
203 {
204 is_zero = NO;
205 }
206 }
207
208 // loop through the necessary number of times to get rid of
209 // unneeded 0's
210 for (j = 0; j < (cs_len - num_zeros); j++)
211 {
212 final_string[j] = c_string[j];
213 }
214 }
215 else // otherwise, there is no decimal place
216 {
217 strcpy(final_string, c_string);
218 }
219
220 new_string = [NSString stringWithFormat:@"%s", final_string];
221
222 // When printing out to NSLog, new_string looks odd (\\304\\026\\010\\304),
223 // but when placed as a parameter, it seems to work. Go figure.
224
More information about the pkg-GNUstep-maintainers
mailing list