Hosting Provided By

System freeze on i386 (Mail from sendbug was rejected)

From: Manfred Wierich <wierich(at)trustcenter.de>
Date: Mon Jun 30 2003 - 13:54:06 EDT

To: gnats@openbsd.org
Subject:
From: manfred.wierich@gmx.de
Cc:
Reply-To: manfred.wierich@gmx.de
X-sendbug-version: 3.97

>Submitter-Id: net
>Originator: Manfred Wierich
>Organization: TC TrustCenter, Hamburg, Germany
net
>Synopsis: System freezes under high load
>Severity: serious
>Priority:
>Category: i386
>Class: sw-bug
>Release: 3.3 Release
>Environment:

        System      : OpenBSD 3.3
        Architecture: OpenBSD.i386
        Machine     : i386

>Description:

        I installed a mailsserver running sendmail with raidconf compiled

several weeks ago under OpenBSD. When I had sent and fetched several hundreds of thousands of mails I took it into production at our company. Embarrasingly enough, it froze the next morning at 6 a.m. Then we switched back to our old mailserver.
After some time I got the idea to stress-test the failed server and got a program called "stress" from Freshmeat.net: http://weather.ou.edu/[..]apw/projects/stress/stress-0.17.2.tar.gz When I ran "stress -v --cpu 4 --hdd 4 --io 4 --vm 4" on the box, it crashed after maybe 10 minutes. I thought that this was maybe due to raidconf wich is not in GENERIC. I tried the same command on two other machines which are the same hardware as the mailserver: A Suse Linux 7.3, and an OpenBSD 3.2. The Linux system of course became very slow, but survived a whole weekend without crashing, the OpenBSD box also became unresponsive after a short time. To be sure, I did a default Install from CD, this time 3.3, on another hardware. It had the same problem. Also, if you run "stress --cpu 4 --hdd 4" as a normal user you can use up all system resources easily. "Top" won't work anymore, "stress" can't be interrupted by pressing ^c, ssh-connections are dropped, root can't log in on another console. If I wait for some 5 Minutes, the system crashes:

_Debugger(0.d08db000.d094e06c.d029b1bc,3f006ed5) at _Debugger+0x4
_panic(d029a2b8.595c98.2000.d02ad0e0.0) at _panic+0x81
_ffs_reallocblks(db38ad1c.a80b.2000.ffffffff.d094e068) at 
_ffs_reallocblks+0xc3
_VOP_REALLOCBLKS(db2d4cac,094e040.db38ad88.d01f31d0.db2d4cac.db2d592c.d38f9388.14)

at _VOP_REALLOCBLKS+0x2e

_cluster_write(d38f9388.db2d592c.1501a000.0.db2d4cac) at _cluster_write+0x130
_ffs_write(db38ae44,3000042,db383280,d095c420,fffff) at _ffs_write+0x3df
_VOP_WRITE(db2d4cac,db38aed8,da8db0f0,db38af88) at _VOP_WRITE+0x3b
_vn_write(db2a4960,db2a497c,db38aed8,da8db0f0,db38af88) at _vn_write+0x94
_dofilewrite(db383280,db38af88,db2a4960,cfafd894,fffff) at _dofilewrite+0x81
_sys_write(db383280,db38af88,db38af7c,d03028db,0) at _sys_write+0x47
_syscall() at _syscall+0x26d

--- syscall (number 4) ---
0x400a39f3
ddb>ps

PID	PPID	PGRP	UID	S	FLAGS	WAIT	COMMAND
31063	17612	19612	1000	3	0x6	getnewbuf	stress
31063	17612	19612	1000	3	0x6	getnewbuf	stress
31063	17612	19612	1000	3	0x6	getnewbuf	stress
31063	17612	19612	1000	3	0x6	getnewbuf	stress
31063	17612	19612	1000	3	0x6	getnewbuf	stress
31063	17612	19612	1000	3	0x6	getnewbuf	stress
31063	17612	19612	1000	3	0x6	getnewbuf	stress
31063	17612	19612	1000	3	0x6	getnewbuf	stress

As I was playing around a bit further

dmesg of this box (running Version 3.3):

OpenBSD 3.3 (GENERIC) #44: Sat Mar 29 13:22:05 MST 2003

deraadt@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC cpu0: Intel Pentium III (Coppermine) ("GenuineIntel" 686-class) 665 MHz cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SYS,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SIMD real mem = 132423680 (129320K)
avail mem = 117112832 (114368K)
using 1642 buffers containing 6725632 bytes (6568K) of memory mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 01/28/00, BIOS32 rev. 0 @ 0xffe90 apm0 at bios0: Power Management spec V1.2 apm0: AC on, battery charge unknown
pcibios0 at bios0: rev. 2.1 @ 0xf0000/0x10000 pcibios0: PCI IRQ Routing Table rev. 1.0 @ 0xfbc40/176 (9 entries) pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82371AB PIIX4 ISA" rev 0x00)
pcibios0: PCI bus #1 is the last bus
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000 0xc9000/0x7000 pci0 at mainbus0 bus 0: configuration mode 1 (no bios) pchb0 at pci0 dev 0 function 0 "Intel 82810E" rev 0x03: rng active, 8Kb/sec vga1 at pci0 dev 1 function 0 "Intel 82810E Graphics" rev 0x03: aperture at 0xf4000000, size 0x4000000
wsdisplay0 at vga1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) ppb0 at pci0 dev 30 function 0 "Intel 82801AA Hub-to-PCI" rev 0x02 pci1 at ppb0 bus 1
fxp0 at pci1 dev 10 function 0 "Intel 82557" rev 0x08: irq 11, address 00:90:27:e4:93:81
inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4 xl0 at pci1 dev 12 function 0 "3Com 3c905C 100Base-TX" rev 0x78: irq 5 address 00:b0:d0:64:16:93
exphy0 at xl0 phy 24: Broadcom 3C905C internal PHY, rev. 7 pcib0 at pci0 dev 31 function 0 "Intel 82801AA LPC" rev 0x02 pciide0 at pci0 dev 31 function 1 "Intel 82801AA IDE" rev 0x02: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility wd0 at pciide0 channel 0 drive 0: <WDC WD102BB> wd0: 16-sector PIO, LBA, 9779MB, 16383 cyl, 16 head, 63 sec, 20028960 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 4 atapiscsi0 at pciide0 channel 1 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <HP, CD-Writer+ 9100, 1.0c> SCSI0 5/cdrom removable
cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2 uhci0 at pci0 dev 31 function 2 "Intel 82801AA USB" rev 0x02: irq 11 usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: vendor 0x0000 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered "Intel 82801AA SMBus" rev 0x02 at pci0 dev 31 function 3 not configured isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0 pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
sysbeep0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
npx0 at isa0 port 0xf0/16: using exception 16 pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec biomask c840 netmask c860 ttymask d8e2
pctr: 686-class user-level performance counters enabled mtrr: Pentium Pro MTRR support
dkcsum: wd0 matched BIOS disk 80
root on wd0a
rootdev=0x0 rrootdev=0x300 rawdev=0x302

dmesg for the other boxens, the mailserver, the 3.2 system and the SuSe Linux system (they're all the same hardware, maybe except NICs):

OpenBSD 3.2-current (GENERIC) #1: Fri Oct 11 12:51:22 CEST 2002

Do you need help?

root@warthog:/usr/src/sys/arch/i386/compile/GENERIC cpu0: Intel Pentium III (Coppermine) ("GenuineIntel" 686-class) 930 MHz cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SYS,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SIMD real mem = 267759616 (261484K)
avail mem = 242499584 (236816K)
using 3294 buffers containing 13492224 bytes (13176K) of memory mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 02/21/01, BIOS32 rev. 0 @ 0xfda74 apm0 at bios0: Power Management spec V1.2 apm0: AC on, battery charge unknown
pcibios0 at bios0: rev. 2.1 @ 0xf0000/0x10000

pcibios0: PCI IRQ Routing Table rev. 1.0 @ 0xf2c40/224 (12 entries)
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82371FB PCI-ISA" rev 0x00)
pcibios0: PCI bus #2 is the last bus

bios0: ROM list: 0xc0000/0x8000 0xc8000/0x800 0xc8800/0x800 pci0 at mainbus0 bus 0: configuration mode 1 (no bios) pchb0 at pci0 dev 0 function 0 "Intel 82815 Hub" rev 0x02 ppb0 at pci0 dev 1 function 0 "Intel 82815 AGP" rev 0x02 pci1 at ppb0 bus 1
vga1 at pci1 dev 0 function 0 "ATI Rage Fury" rev 0x00 wsdisplay0 at vga1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) ppb1 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0x02 pci2 at ppb1 bus 2
xl0 at pci2 dev 9 function 0 "3Com 3c905C 100Base-TX" rev 0x78: irq 3xl0: command never completed!
xl0: command never completed!
xl0: command never completed!
address 00:04:76:35:f4:73
exphy0 at xl0 phy 24: 3Com internal media interface

xl0: command never completed!
xl0: command never completed!
xl0: command never completed!

xl1 at pci2 dev 11 function 0 "3Com 3c905C 100Base-TX" rev 0x78: irq 11xl1: command never completed!
xl1: command never completed!
xl1: command never completed!
address 00:04:76:36:29:d1
exphy1 at xl1 phy 24: 3Com internal media interface

xl1: command never completed!
xl1: command never completed!
xl1: command never completed!

pcib0 at pci0 dev 31 function 0 "Intel 82801BA LPC" rev 0x02 pciide0 at pci0 dev 31 function 1 "Intel 82801BA IDE" rev 0x02: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility wd0 at pciide0 channel 0 drive 0: <Maxtor 5T020H2> wd0: 16-sector PIO, LBA, 19073MB, 16383 cyl, 16 head, 63 sec, 39062500 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5 atapiscsi0 at pciide0 channel 1 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <Lite-On, LTN486 48x Max, YD01> SCSI0 5/cdrom removable
cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2 uhci0 at pci0 dev 31 function 2 "Intel 82801BA USB" rev 0x02: irq 10 usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: vendor 0x0000 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered "Intel 82801BA SMBus" rev 0x02 at pci0 dev 31 function 3 not configured isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
sysbeep0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
npx0 at isa0 port 0xf0/16: using exception 16 pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec biomask c440 netmask cc48 ttymask ccca
pctr: 686-class user-level performance counters enabled mtrr: Pentium Pro MTRR support
dkcsum: wd0 matched BIOS disk 80
root on wd0a
rootdev=0x0 rrootdev=0x300 rawdev=0x302
WARNING: / was not properly unmounted
free inode //3607730 had 163840 blocks
dev = 0x0, block = 2664, fs = /
panic: ffs_blkfree: freeing free block
OpenBSD 3.2-current (GENERIC) #1: Fri Oct 11 12:51:22 CEST 2002

pcibios0: PCI IRQ Routing Table rev. 1.0 @ 0xf2c40/224 (12 entries)
Do you need more help? 
In addition to our free technical library, Pantek provides professional IT services to companies who utilize Open Source software:Learn more about Pantek's Technical Support offerings
Place an order for 24/7/365 Technical Support here
Learn more about Pantek, who we are, and what we do
Chat with a Live Operator
Related openbsd.org Links
advocacy
announce
bugs
ipv6
misc
ports
ports-changes
security-announce
smp
source-changes
tech
Request more information about Pantek
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82371FB PCI-ISA" rev 0x00)
pcibios0: PCI bus #2 is the last bus

xl0: command never completed!
xl0: command never completed!
xl0: command never completed!

xl1: command never completed!
xl1: command never completed!
xl1: command never completed!

This message: [ Message body ]
Next message: Rahul Patel: "Found 2 bugs in OpenBSD 3.3"
Previous message: Daniel Hartmeier: "Re: system/3332"

Contemporary messages sorted: [ By Date ] [ By Thread ] [ By Subject ] [ By Author ] [ By messages with attachments ]

This archive was generated by hypermail 2.1.8 : Wed Aug 23 2006 - 13:29:59 EDT