Question Another crash bug

Caveat: I know just enough about coding to be dangerous, and not enough to diagnose. If this is part of the two currently open threads about crashes, feel free to merge this post into it.

Info: Running 3.68, have not upgraded to the new release yet. Most recent change to code was inserting the updated clans code on January 21. The game has been running with zero crashes up to 1/25 for the past year.

Narrative: Yesterday we experienced two back to back crashes. I was online for them, and I don't have a screen capture for context. Basically, I was in another small screen working on a trigger in another program with eyes on mudlet. I saw a player trying to log in and his password failing several times in a row. He then logged in another character just fine. I said something to him on gossip, he responded, and then we crashed. I went to log back in, the game came up, and as soon as I entered the game itself, we crashed. I tried again, the game came up, and stayed up for about 3 hours. Then there was another crash. During the approx 3 hour time before the third crash the only characters logged into the game were myself and one other staff member. I was afk the entire time, so not doing anything. I believe the other staff member was working on cleaning up triggers.

I reviewed the logs and each time this appears: segmentation fault (core dumped)

The clan code is not actively being used by players, as the game is not open to the public. One character is in one clan but was not online at the time.

There are also some odd characters showing up. This is what I see:

Jan 25 20:37:48 ��� :: Losing descriptor without char.
Jan 25 20:42:48 :: nusage: 2 sockets connected, 2 sockets playing
Jan 25 20:47:48 :: nusage: 2 sockets connected, 2 sockets playing
Jan 25 20:52:48 :: nusage: 2 sockets connected, 2 sockets playing
Jan 25 20:57:48 :: nusage: 2 sockets connected, 2 sockets playing
Jan 25 21:02:48 :: nusage: 2 sockets connected, 2 sockets playing
Jan 25 21:07:48 :: nusage: 2 sockets connected, 2 sockets playing
Jan 25 21:12:48 :: nusage: 2 sockets connected, 2 sockets playing
Jan 25 21:17:48 :: nusage: 2 sockets connected, 2 sockets playing
Jan 25 21:22:48 :: nusage: 2 sockets connected, 2 sockets playing
Jan 25 21:27:48 :: nusage: 2 sockets connected, 2 sockets playing
Jan 25 21:32:48 :: nusage: 2 sockets connected, 2 sockets playing
Segmentation fault (core dumped)

As of right now, the game has been up and running for 10 hours and 11 minutes, with both myself and the same staff member online and working as I type this, with no errors showing up.

Segmentation fault(core dumped) is the codes' way of saying "I tried to read or free memory not allocated to me and now I'll put all my data in this here core file". It's impossible from that message to understand the error.

Please try doing the things described in the "debugging tutorial for gdb" that is pinned in the development forum.

Basically this:
For reference:
www.tbamud.com/forum/4-development/6-debugging-tutorial-for-gdb

When I added the clan patch in, I ended up with a crash at login exactly like what I saw with that scoundrel (me) yesterday logging into AtB.

Small change to add_existing_clan:
That is worth a shot.

Also go through the gdb debugging as thomas has suggested, it is good practise and you will use it a lot.

Thanks, I had pulled that tutorial and read it several times last night - I don't want to waste anyone's time and I'm still learning this stuff. I ran it this morning, this is what I get.

Reading symbols from bin/circle...done.

warning: exec file is newer than core file.
[New LWP 12454]

warning: Error reading shared library list entry at 0x45396600

warning: Error reading shared library list entry at 0x45393e00
Core was generated by `bin/circle -q 4000'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f2db4840c37 in ?? ()
(gdb) Quit
(gdb) bt
#0 0x00007f2db4840c37 in ?? ()
#1 0x00007f2db4844028 in ?? ()
#2 0x0000000000000020 in ?? ()
#3 0x0000000000000000 in ?? ()
(gdb) list
195 #endif
196
197 t->tv_sec = (int) (millisec / 1000);
198 t->tv_usec = (millisec % 1000) * 1000;
199 }
200
201 #endif /* CIRCLE_WINDOWS || CIRCLE_MACINTOSH */
202
203 int main(int argc, char **argv)
204 {
(gdb) info local
No symbol table info available.
(gdb) up
#1 0x00007f2db4844028 in ?? ()
(gdb)
(gdb) bt
#0 0x00007f2db4840c37 in ?? ()
#1 0x00007f2db4844028 in ?? ()
#2 0x0000000000000020 in ?? ()
#3 0x0000000000000000 in ?? ()
(gdb) list
205 int pos = 1;
206 const char *dir;
207
208 #ifdef MEMORY_DEBUG
209 zmalloc_init();
210 #endif
211
212 #if CIRCLE_GNU_LIBC_MEMORY_TRACK
213 mtrace(); /* This must come before any use of malloc(). */
214 #endif
(gdb) info local
No symbol table info available.
(gdb) up
#2 0x0000000000000020 in ?? ()
(gdb)
(gdb) bt
#0 0x00007f2db4840c37 in ?? ()
#1 0x00007f2db4844028 in ?? ()
#2 0x0000000000000020 in ?? ()
#3 0x0000000000000000 in ?? ()
(gdb) list
215
216 #ifdef CIRCLE_MACINTOSH
217 /* ccommand() calls the command line/io redirection dialog box from
218 * Codewarriors's SIOUX library. */
219 argc = ccommand(&argv);
220 /* Initialize the GUSI library calls. */
221 GUSIDefaultSetup();
222 #endif
223
224 /* Load the game configuration. We must load BEFORE we use any of the
(gdb) info local
No symbol table info available.
(gdb)
(gdb) Quit
(gdb) up
#3 0x0000000000000000 in ?? ()
(gdb) bt
#0 0x00007f2db4840c37 in ?? ()
#1 0x00007f2db4844028 in ?? ()
#2 0x0000000000000020 in ?? ()
#3 0x0000000000000000 in ?? ()
(gdb) list
225 * constants stored in constants.c. Otherwise, there will be no variables
226 * set to set the rest of the vars to, which will mean trouble --> Mythran */
227 CONFIG_CONFFILE = NULL;
228 while ((pos < argc) && (*(argv[pos]) == '-')) {
229 if (*(argv[pos] + 1) == 'f') {
230 if (*(argv[pos] + 2))
231 CONFIG_CONFFILE = argv[pos] + 2;
232 else if (++pos < argc)
233 CONFIG_CONFFILE = argv[pos];
234 else {
(gdb) info local
No symbol table info available.
(gdb)
(gdb) Quit
(gdb) up
Initial frame selected; you cannot go up.
(gdb)

Happened again, just now. I had a feeling it was coming because I was making mobs and saw corruption characters as I was walking around the zone:

i1 510H 110M 92V >
load mob 545
You create @H(.

i1 510H 110M 92V >
l
[ 532] A Path Through Wildflowers [ NOBITS ] [ Field ]
An open swath of colorful flowers breaks up the grass of the plains, as if
someone planted a garden that was left untended and grew out of control.
[ Exits: ne sw ]
[825] [T1901] hx9ﾐﾋ[515] A gopher scampers across the land.

i1 510H 110M 92V >
load mob 825
You create a cricket.

i1 510H 110M 92V >
l
[ 532] A Path Through Wildflowers [ NOBITS ] [ Field ]
An open swath of colorful flowers breaks up the grass of the plains, as if
someone planted a garden that was left untended and grew out of control.
[ Exits: ne sw ]
[825] A cricket is sitting here.
[825] [T1901] hx9ﾐﾋ[515] A gopher scampers across the land.

i1 510H 110M 92V >
zpurge
Purged zone #5: Grassy Plains - Caralain Grass.


[ ALERT ] - Socket got disconnected.
Reason: The remote host closed the connection
[ INFO ] - Connection time: 00:32:30.008


Here's the info from using GDB:

Reading symbols from bin/circle...done.

warning: exec file is newer than core file.
[New LWP 12454]

warning: Error reading shared library list entry at 0x45396600

warning: Error reading shared library list entry at 0x45393e00
Core was generated by `bin/circle -q 4000'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f2db4840c37 in ?? ()
(gdb) bt
#0 0x00007f2db4840c37 in ?? ()
#1 0x00007f2db4844028 in ?? ()
#2 0x0000000000000020 in ?? ()
#3 0x0000000000000000 in ?? ()
(gdb) list
195 #endif
196
197 t->tv_sec = (int) (millisec / 1000);
198 t->tv_usec = (millisec % 1000) * 1000;
199 }
200
201 #endif /* CIRCLE_WINDOWS || CIRCLE_MACINTOSH */
202
203 int main(int argc, char **argv)
204 {
(gdb)
(gdb) info local
No symbol table info available.
(gdb) up
#1 0x00007f2db4844028 in ?? ()
(gdb) bt
#0 0x00007f2db4840c37 in ?? ()
#1 0x00007f2db4844028 in ?? ()
#2 0x0000000000000020 in ?? ()
#3 0x0000000000000000 in ?? ()
(gdb) list
205 int pos = 1;
206 const char *dir;
207
208 #ifdef MEMORY_DEBUG
209 zmalloc_init();
210 #endif
211
212 #if CIRCLE_GNU_LIBC_MEMORY_TRACK
213 mtrace(); /* This must come before any use of malloc(). */
214 #endif
(gdb) info local
No symbol table info available.
(gdb) Quit
(gdb) up
#2 0x0000000000000020 in ?? ()
(gdb) bt
#0 0x00007f2db4840c37 in ?? ()
#1 0x00007f2db4844028 in ?? ()
#2 0x0000000000000020 in ?? ()
#3 0x0000000000000000 in ?? ()
(gdb) list
215
216 #ifdef CIRCLE_MACINTOSH
217 /* ccommand() calls the command line/io redirection dialog box from
218 * Codewarriors's SIOUX library. */
219 argc = ccommand(&argv);
220 /* Initialize the GUSI library calls. */
221 GUSIDefaultSetup();
222 #endif
223
224 /* Load the game configuration. We must load BEFORE we use any of the
(gdb) info local
No symbol table info available.
(gdb) Quit
#3 0x0000000000000000 in ?? ()
(gdb) bt
#0 0x00007f2db4840c37 in ?? ()
#1 0x00007f2db4844028 in ?? ()
#2 0x0000000000000020 in ?? ()
#3 0x0000000000000000 in ?? ()
(gdb) list
225 * constants stored in constants.c. Otherwise, there will be no variables
226 * set to set the rest of the vars to, which will mean trouble --> Mythran */
227 CONFIG_CONFFILE = NULL;
228 while ((pos < argc) && (*(argv[pos]) == '-')) {
229 if (*(argv[pos] + 1) == 'f') {
230 if (*(argv[pos] + 2))
231 CONFIG_CONFFILE = argv[pos] + 2;
232 else if (++pos < argc)
233 CONFIG_CONFFILE = argv[pos];
234 else {
(gdb) info local
No symbol table info available.
(gdb) up
Initial frame selected; you cannot go up.
(gdb)


Any idea what's going on, guys? After rebooting the mobs loaded correctly, btw.

Disclaimer: not my strong point, defer to more experienced coders over this

But, what I can see is initially "warning: exec file is newer than core file." may mean you need to re-compile to get more usable information.
Do do a "make clean", then "make".

Secondly the lack of information, eg "#0 0x00007f2db4840c37 in ?? ()", looks like stack corruption.
The stack tells you what function you are in among other things, so if this is overwritten then we are in for an interesting debug session.

A common cause of stack corruption is buffer overflow, which will happen if you write too many characters into a string.
Say you have buf[5] and try strncpy(buf, "toomany") - the extra characters will overwrite memory it shouldn't.

Next steps: make clean and re-run gdb session, try and reproduce the crash reliably, double check the latest bugfixes are in your code (there were a couple found over the last month), good luck!