kernel panic: out of space in kmem_map

From: Andreas Strodl <andreas(at)strodl.org>
Date: Fri Nov 15 2002 - 03:14:27 EST

Hi,

I'm running latest OPENBSD_3_0 kernel and patched all my applications (http://www.openbsd.org/errata.html). But after some days uptime I always get the same kernel panic:

panic: malloc: out of space in kmem_map
Stopped at _Debugger+0x4: leave

trace output:

_Debugger(80,8000,0,e048277b,8,e1649000,fa1bd758,e01a311a,e01a2d64,e04dba4c,
80000,0,fb25,0,11,0,fa1bd760,fa1bd760,fa1bd768,e048283e,fa1ba7c0,db08f829,fa1bd768, 8,e0482eff,1fff339f,8000,0,e000cc60,e0591d18,e2577398,20,e029d717,13,e0581248, fa1bd828,e0223e99,80000,4b,0,a801a850,fb25, 0,11,0,0,0,0,0,e2564804,e000cc60, 278,0,0,0,0,1fff331d,e000cce2,200,fa1bd7f8,e0240d61,e2508ac0) at _Debugger+0x4
_panic(e01a2d64,e04dba4c,80000,0,fb25) at _panic+0x81
_malloc(80000,4b,0,a801a850,fb25) at _malloc+0x29a
_tdb_rehash(1ff339f,e258dc00,100,1fff339f,e258da00) at _tdb_rehash+0x31
_puttdb(e258dc00,e2458dd08,10,e0223581,fa1bdb50) at _puttdb+0x295
_reserve_spi(100,ffffffff,fa1bdb6c,fa1bdb50,32)at _reserve_spi+0x197
_pfkeyv2_send(e19f1500,e257b180,50,e02c675c,e17ea200) at _pfkeyv2_send+0x2c4
_pkfkey_sendup(e17ea200,e19f1500,fa1bdd98,e01d45a7,c0000000) at pfkey_Sendup+0x153
_raw_usrreq(e19fa500,9,e17ea200,0,0) at _raw_usrreq+0x15e
_pfkey_sendup(e19f1500,9,e17ea200,0,0) at _pfkey_sendup+0x398
_sosend(e19f1500,0,fa1bded8,e17ea200,0) at _sosend+0x673
_soo_write(fa032aa4,fa032ac0,fa1bded8,e1997580,fa1bdf88) at _soo_write+0x1f
_dofilewritev(fa077adc,3fa032aa4,362400,4) at _dofilewritev+0x12d
_sys_writev(fa077adc,fa1bdf88,fa1bdf80,52d4,0) at _sys_writev+0x48
_syscall() at _syscall+0x242

--- syscall (number 121)---
0xdb307:
ddb>

ps outupt:

---

PID	PPID	PGRP	UID	S	FKAGS	WAIT	CMD
18824	7921	18824	0	3	0x4084	poll	stunnel
24661	23489	23489	0	3	0x4184	select	local
28945	23489	23489	1001	3	0x4184	select	cleanup
26038	23489	23489	1001	3	0x4184	select	smtpd
2284	23489	23489	0	3	0x4184	select	pickup
24121	23489	23489	1001	3	0x4184	select	cleanup
27789	23489	23489	1001	3	0x4184	nanosleep	smtpd
13345	23489	23489	1001	3	0x4184	select	trivial-rewrite
17878	23489	23489	1001	3	0x4184	nanosleep	smtpd
23744	31269	23744	1020	3	0x4086	pause	screen-3.9.10
31269	32286	31269	1020	3	0x4086	wait	bash
32286	2748	2748	1020	3	0x184	select	sshd
2748	11344	2748	0	3	0x184	netio	sshd
15762	6535	6535	1039	3	0x4184	netio	imapd
6535	7921	6535	0	3	0x4084	poll	stunnel
23097	13481	13481	32767	3	0x184	lockf	httpd
15861	13481	13481	32767	3	0x184	select	httpd
27506	13481	13481	32767	3	0x184	lockf	httpd
7002	13481	13481	32767	3	0x184	lockf	httpd
20492	13481	13481	32767	3	0x184	lockf	httpd
31530	13481	13481	32767	3	0x184	lockf	httpd
18098	13481	13481	32767	3	0x184	lockf	httpd
28999	13481	13481	32767	3	0x184	lockf	httpd
17710	13481	13481	32767	3	0x184	lockf	httpd
17422	13481	13481	32767	3	0x184	lockf	httpd
13481	1	13481	0	3	0x84	select	httpd
25936	1	25936	70	3	0x184	select	named
16425	1	16425	70	3	0x184	select	named
21435	1	18280	1012	3	0x86	select	psybnc
2196	1	2196	1012	3	0x86	select	eggdrop
21820	32259	32259	1020	3	0x4084	kqread	tail
32259	15516	32259	1020	3	0x4084	pause	sh
*18081	1	18081	0	2	0x4		isakmpd
25068	1	25068	0	3	0x84	select	isakmpd
13659	1715	13658	1020	3	0x4086	ttyin	bash
25527	1	25527	1022	3	0x84	select 	ftpd
15516	17509	15516	1020	3	0x84	select	BitchX-1.0c18
17509	1715	17509	1020	3	0x4086	wait	bash
1715	1	1715	1020	3	0x84	select	screen-3.9.10
25901	1	25901	1034	3	0x84	select	eggdrop-1.6.10
9821	851	9821	1003	3	0x4087	poll	irssi
Do you need help?
In addition to our free technical library, Pantek provides professional IT services to companies who utilize Open Source software:
Learn more about Pantek's Technical Support offerings
Place an order for 24/7/365 Technical Support here
Learn more about Pantek, who we are, and what we do
Chat with a Live Operator
Related openbsd.org Links
advocacy
announce
bugs
ipv6
misc
ports
ports-changes
security-announce
smp
source-changes
tech
Request more information about Pantek
851	8872	851	1003	3	0x4086	wait	bash
8872	1	8872	1003	3	0x84	select	screen-3.9.10
17744	23480	27489	1001	3	0x4184	select	qmgr
23489	1	23489	0	3	0x418c	select	master
20967	1	20967	32767	3	0x184	poll	stunnel
10870	1	10870	73	3	0x186	poll	ircd
14522	22693	1985	1000	3	0x4187	select	mysqld
22693	1	1985	0	3	0x4086	pause	sh
16533	7284	16533	1017	3	0x4084	piperd	unlinkd
24564	1	24564	0	3	0x4086	ttyin	getty
5818	1	5818	0	3	0x4086	ttyin	getty
7544	1	7544	0	3	0x4086	ttyin	getty
31605	1	31605	0	3	0x4086	ttyin	getty
28659	1	28659	0	3	0x4086	ttyin	getty
21220	1	21220	0	3	0x84	nanosleep	cron
7284	29797	29797	0	3	0x4184	poll	squid
29797	1	29797	0	3	0x84	wait	squid
11344	1	11344	0	3	0x84	select 	sshd
7921	1	7921	0	3	0x184	select	inetd
29585	1	29585	0	3	0x84	bpf	pflogd
16894	1	16984	0	2	0x84		syslogd
9	0	0	0	3	0x100204	usbevt	usb1
8	0	0	0	3	0x100204	usbevt	usb0
7	0	0	0	3	0x100204	apmev	apm0
6	0	0	0	3	0x100204	crypot_wa	crypto
5	0	0	0	3	0x100204	syncer	update
4	0	0	0	3	0x100204	cleaner	cleaner
3	0	0	0	3	0x100204	reaper	reaper
2	0	0	0	3	0x100204	daemon_sl	pagedaemon
1	0	1	0	3	0x408	wait	init
17575	21220	21220	0	5	0x2004		cron
22127	21220	21220	0	5	0x2004		cron	

dmesg output:
OpenBSD 3.0-stable (MOON) #1: Wed Oct 16 12:25:20 CEST 2002

    andi@moon:/usr/src/sys/arch/i386/compile/MOON cpu0: AMD Duron ("AuthenticAMD" 686-class) 900 MHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SYS,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR real mem = 1073262592 (1048108K)
avail mem = 990257152 (967048K)
using 5689 buffers containing 53764096 bytes (52504K) of memory mainbus0 (root)
bios0 at mainbus0: AT/286+(38) BIOS, date 08/03/01, BIOS32 rev. 0 @ 0xfb130 apm0 at bios0: Power Management spec V1.2 apm0: AC on, battery charge unknown
pcibios0 at bios0: rev. 2.1 @ 0xf0000/0xb5b0

pcibios0: PCI IRQ Routing Table rev. 1.0 @ 0xfdaf0/176 (9 entries)
pcibios0: PCI Exclusive IRQs: 5 10 11 12
pcibios0: PCI Interrupt Router at 000:07:0 ("VIA VT82C596A PCI-ISA" rev 0x00)
pcibios0: PCI bus #1 is the last bus

bios0: ROM list: 0xc0000/0x8000 0xc8000/0x800 pci0 at mainbus0 bus 0: configuration mode 1 (no bios) pchb0 at pci0 dev 0 function 0 "VIA VT8363 Host" rev 0x03 ppb0 at pci0 dev 1 function 0 "VIA VT8363 PCI-AGP" rev 0x00 pci1 at ppb0 bus 1
pcib0 at pci0 dev 7 function 0 "VIA VT82C686 PCI-ISA" rev 0x40 pciide0 at pci0 dev 7 function 1 "VIA VT82C571 IDE" rev 0x06: ATA100, channel 0 configured to compatibility, channel 1 configured to compatibility wd0 at pciide0 channel 0 drive 0: <IC35L040AVER07-0> wd0: 16-sector PIO, LBA, 39266MB, 16383 cyl, 16 head, 63 sec, 80418240 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5 wd1 at pciide0 channel 1 drive 0: <ST340823A> wd1: 16-sector PIO, LBA, 38166MB, 16383 cyl, 16 head, 63 sec, 78165360 sectors atapiscsi0 at pciide0 channel 1 drive 1
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <TEAC, CD-56E, 1.0C> SCSI0 5/cdrom removable wd1(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 5 cd0(pciide0:1:1): using PIO mode 3, DMA mode 1 uhci0 at pci0 dev 7 function 2 "VIA VT83C572 USB" rev 0x1a: irq 5 usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: vendor 0x0000 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1 at pci0 dev 7 function 3 "VIA VT83C572 USB" rev 0x1a: irq 5 usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: vendor 0x0000 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered pchb1 at pci0 dev 7 function 4 "VIA VT82C686 SMBus" rev 0x40 auvia0 at pci0 dev 7 function 5 "VIA VT82C686 AC-97 Audio" rev 0x50: irq 12 ac97: codec id 0x49434511
ac97: codec features headphone, 18 bit DAC, 18 bit ADC, Unknown 3D audio0 at auvia0
xl0 at pci0 dev 8 function 0 "3Com 3c905C 100Base-TX" rev 0x74: irq 11 address 00:04:76:0b:1a:95 exphy0 at xl0 phy 24: Broadcom 3C905C internal PHY, rev. 6 rl0 at pci0 dev 9 function 0 "Realtek 8139" rev 0x10: irq 10 address 00:50:fc:3f:3b:78 rlphy0 at rl0 phy 0: RTL internal phy
vga1 at pci0 dev 10 function 0 "Avance Logic AVL2301" rev 0x00 wsdisplay0 at vga1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
sysbeep0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
npx0 at isa0 port 0xf0/16: using exception 16 pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec biomask c060 netmask cc60 ttymask cce2
pctr: user-level cycle counter enabled
mtrr: Pentium Pro MTRR support
dkcsum: wd0 matched BIOS disk 80
dkcsum: wd1 matched BIOS disk 81
root on wd0a
rootdev=0x0 rrootdev=0x300 rawdev=0x302
WARNING: / was not properly unmounted

I'm connected to three servers with ipsec (2 linux freeswan, 1 openbsd isakmpd) and one dialupuser (openbsd isakmpd). I think the reason for the crash are malformed esp packets from the machines that i'm connected to (over ipsec) because of the trace and ps output.

As you can see I run two named processes, apache, postfix, mysql and ircd and some other user applications.

First I thought the reason for the crash are my RAM chips but I have already bought 2 x 512 MB and the crash still occours.

Any ideas?

greets
Andi Received on Fri Nov 15 03:18:46 2002