Open Bug 504342 (CVE-2009-2479) Opened 15 years ago Updated 2 years ago

Investigate milw0rm 9158 "unicode stack overflow"

Categories

(Core :: Layout: Text and Fonts, defect)

Product:
Core
Component:
Layout: Text and Fonts
Version:
1.9.1 Branch
Type:
defect

Tracking

()

People

(Reporter: arzhel, Unassigned)

References

(
URL
)

Details

(Keywords: crash, testcase, Whiteboard: Win:new throws on oom; Mac: crash in Apple routines (like Safari))

Attachments

(4 files)

sample from fx3.0.11
19.99 KB, text/plain
Details
testcase (milw0rm 9158)
591 bytes, text/html
Details
testcase, automatic reload added
646 bytes, text/html
Details
WinDbg log
8.12 KB, application/octet-stream
Details 
I noticed this on milw0rm today
http://www.milw0rm.com/exploits/9158

I'm setting this to security sensitive for now, but the exploit report is public so the bug might not need to be closed.

Thanks Mike Beltzner for helping me to file my first security bug :)
Whiteboard: [sg:investigate]
And the duplicate there is also open, so I'm pretty sure we should just open this up. I'll let someone in sg who's more familiar with process there decide.

Sam and Reed think that someone might be trawling Bugzilla in order to develop exploits. Not sure what to do about that.
I would prefer if we have a closed bug to investigate such reports and post non-public information. If you don't want to close the bug itself, maybe we can have a bug for the sg-investigate part that we open up later (once the fix is in). I am not comfortable talking about what goes wrong and why since that might reveal other, potentially even more severe problems in some cases.
#0  0x92936c92 in TASCIIEncoder::EncodeWithHints ()
#1  0x9292598e in TASCIIEncoder::Encode ()
#2  0x92924e62 in TGlyphEncoder::EncodeChars ()
#3  0x9292490e in TTypesetterAttrString::Initialize ()
#4  0x9292467f in CTLineCreateWithAttributedString ()
#5  0x119a5c82 in gfxCoreTextFontGroup::InitTextRun (this=0x26893570, aTextRun=0xcf054000, aString=0x8781c008, aTotalLength=42000006, aLayoutStart=2, aLayoutLength=42000001) at ../../../../gfx/thebes/src/gfxCoreTextFonts.cpp:911
#6  0x119a5ecc in gfxCoreTextFontGroup::MakeTextRun (this=0x26893570, aString=0x82800008, aLength=42000001, aParams=0xbfff68c0, aFlags=16779265) at ../../../../gfx/thebes/src/gfxCoreTextFonts.cpp:742
#7  0x11994268 in TextRunWordCache::MakeTextRun (this=0x12bb6e80, aText=0x78de5008, aLength=42000000, aFontGroup=0x26893570, aParams=0xbfff7cdc, aFlags=16779264) at ../../../../gfx/thebes/src/gfxTextRunWordCache.cpp:683
#8  0x11994385 in gfxTextRunWordCache::MakeTextRun (aText=0x78de5008, aLength=42000000, aFontGroup=0x26893570, aParams=0xbfff7cdc, aFlags=16779264) at ../../../../gfx/thebes/src/gfxTextRunWordCache.cpp:992
#9  0x18ab7fc6 in MakeTextRun (aText=0x78de5008, aLength=42000000, aFontGroup=0x26893570, aParams=0xbfff7cdc, aFlags=16779264) at ../../../layout/generic/nsTextFrameThebes.cpp:436
#10 0x18abb1a0 in BuildTextRunsScanner::BuildTextRunForFrames (this=0xbfff8ea0, aTextBuffer=0x7de00d08) at ../../../layout/generic/nsTextFrameThebes.cpp:1809
#11 0x18abb601 in BuildTextRunsScanner::FlushFrames (this=0xbfff8ea0, aFlushLineBreaks=1, aSuppressTrailingBreak=0) at ../../../layout/generic/nsTextFrameThebes.cpp:1217
#12 0x18abc282 in BuildTextRuns (aContext=0x26ace480, aForFrame=0x2669386c, aLineContainer=0x28955d38, aForFrameLine=0xbfff9a4c) at ../../../layout/generic/nsTextFrameThebes.cpp:1148
#13 0x18abc3bb in nsTextFrame::EnsureTextRun (this=0x2669386c, aReferenceContext=0x26ace480, aLineContainer=0x28955d38, aLine=0xbfff9a4c, aFlowEndInTextRun=0xbfff9514) at ../../../layout/generic/nsTextFrameThebes.cpp:2002
#14 0x18ac0163 in nsTextFrame::Reflow (this=0x2669386c, aPresContext=0x264fa200, aMetrics=@0xbfff9758, aReflowState=@0xbfff96b0, aStatus=@0xbfff9894) at ../../../layout/generic/nsTextFrameThebes.cpp:6133
#15 0x18a82bdf in nsLineLayout::ReflowFrame (this=0xbfff9a30, aFrame=0x2669386c, aReflowStatus=@0xbfff9894, aMetrics=0x0, aPushedFrame=@0xbfff9890) at ../../../layout/generic/nsLineLayout.cpp:844
#16 0x18a1239f in nsBlockFrame::ReflowInlineFrame (this=0x28955d38, aState=@0xbfffa02c, aLineLayout=@0xbfff9a30, aLine={mCurrent = 0x266938b0, mListLink = 0x28955d7c}, aFrame=0x2669386c, aLineReflowStatus=0xbfff9950) at ../../../layout/generic/nsBlockFrame.cpp:3728
#17 0x18a18e5e in nsBlockFrame::DoReflowInlineFrames (this=0x28955d38, aState=@0xbfffa02c, aLineLayout=@0xbfff9a30, aLine={mCurrent = 0x266938b0, mListLink = 0x28955d7c}, aFloatAvailableSpace=@0xbfff9aec, aAvailableSpaceHeight=@0xbfff9ae4, aFloatStateBeforeLine=0xbfff9ad0, aKeepReflowGoing=0xbfff9d2c, aLineReflowStatus=0xbfff9ae8, aAllowPullUp=1) at ../../../layout/generic/nsBlockFrame.cpp:3544
#18 0x18a19822 in nsBlockFrame::ReflowInlineFrames (this=0x28955d38, aState=@0xbfffa02c, aLine={mCurrent = 0x266938b0, mListLink = 0x28955d7c}, aKeepReflowGoing=0xbfff9d2c) at ../../../layout/generic/nsBlockFrame.cpp:3394
#19 0x18a19de8 in nsBlockFrame::ReflowLine (this=0x28955d38, aState=@0xbfffa02c, aLine={mCurrent = 0x266938b0, mListLink = 0x28955d7c}, aKeepReflowGoing=0xbfff9d2c) at ../../../layout/generic/nsBlockFrame.cpp:2438
#20 0x18a1a708 in nsBlockFrame::ReflowDirtyLines (this=0x28955d38, aState=@0xbfffa02c) at ../../../layout/generic/nsBlockFrame.cpp:1935
#21 0x18a1db7a in nsBlockFrame::Reflow (this=0x28955d38, aPresContext=0x264fa200, aMetrics=@0xbfffa464, aReflowState=@0xbfffa398, aStatus=@0xbfffa4e4) at ../../../layout/generic/nsBlockFrame.cpp:970
#22 0x18a218dd in nsBlockReflowContext::ReflowBlock (this=0xbfffa440, aSpace=@0xbfffa4f4, aApplyTopMargin=1, aPrevMargin=@0xbfffabcc, aClearance=0, aIsAdjacentWithTop=1, aLine=0x28955d94, aFrameRS=@0xbfffa398, aFrameReflowStatus=@0xbfffa4e4, aState=@0xbfffab4c) at ../../../layout/generic/nsBlockReflowContext.cpp:310
#23 0x18a176c2 in nsBlockFrame::ReflowBlockFrame (this=0x28955b10, aState=@0xbfffab4c, aLine={mCurrent = 0x28955d94, mListLink = 0x28955b54}, aKeepReflowGoing=0xbfffa84c) at ../../../layout/generic/nsBlockFrame.cpp:3111
#24 0x18a19b18 in nsBlockFrame::ReflowLine (this=0x28955b10, aState=@0xbfffab4c, aLine={mCurrent = 0x28955d94, mListLink = 0x28955b54}, aKeepReflowGoing=0xbfffa84c) at ../../../layout/generic/nsBlockFrame.cpp:2383
#25 0x18a1a708 in nsBlockFrame::ReflowDirtyLines (this=0x28955b10, aState=@0xbfffab4c) at ../../../layout/generic/nsBlockFrame.cpp:1935
#26 0x18a1db7a in nsBlockFrame::Reflow (this=0x28955b10, aPresContext=0x264fa200, aMetrics=@0xbfffaea4, aReflowState=@0xbfffadfc, aStatus=@0xbfffb118) at ../../../layout/generic/nsBlockFrame.cpp:970
#27 0x18a2cf33 in nsContainerFrame::ReflowChild (this=0x26440c94, aKidFrame=0x28955b10, aPresContext=0x264fa200, aDesiredSize=@0xbfffaea4, aReflowState=@0xbfffadfc, aX=0, aY=0, aFlags=0, aStatus=@0xbfffb118, aTracker=0x0) at ../../../layout/generic/nsContainerFrame.cpp:825
#28 0x18a66986 in CanvasFrame::Reflow (this=0x26440c94, aPresContext=0x264fa200, aDesiredSize=@0xbfffb194, aReflowState=@0xbfffb060, aStatus=@0xbfffb118) at ../../../layout/generic/nsHTMLFrame.cpp:652
#29 0x18a2cf33 in nsContainerFrame::ReflowChild (this=0x24694a34, aKidFrame=0x26440c94, aPresContext=0x264fa200, aDesiredSize=@0xbfffb194, aReflowState=@0xbfffb060, aX=0, aY=0, aFlags=3, aStatus=@0xbfffb118, aTracker=0x0) at ../../../layout/generic/nsContainerFrame.cpp:825
#30 0x18a5972a in nsHTMLScrollFrame::ReflowScrolledFrame (this=0x24694a34, aState=0xbfffb26c, aAssumeHScroll=0, aAssumeVScroll=1, aMetrics=0xbfffb194, aFirstPass=1) at ../../../layout/generic/nsGfxScrollFrame.cpp:528
#31 0x18a5f1a9 in nsHTMLScrollFrame::ReflowContents (this=0x24694a34, aState=0xbfffb26c, aDesiredSize=@0xbfffb564) at ../../../layout/generic/nsGfxScrollFrame.cpp:622
#32 0x18a5f985 in nsHTMLScrollFrame::Reflow (this=0x24694a34, aPresContext=0x264fa200, aDesiredSize=@0xbfffb564, aReflowState=@0xbfffb414, aStatus=@0xbfffb73c) at ../../../layout/generic/nsGfxScrollFrame.cpp:823
#33 0x18a2cf33 in nsContainerFrame::ReflowChild (this=0x26440bb0, aKidFrame=0x24694a34, aPresContext=0x264fa200, aDesiredSize=@0xbfffb564, aReflowState=@0xbfffb414, aX=0, aY=0, aFlags=0, aStatus=@0xbfffb73c, aTracker=0x0) at ../../../layout/generic/nsContainerFrame.cpp:825
#34 0x18acdc51 in ViewportFrame::Reflow (this=0x26440bb0, aPresContext=0x264fa200, aDesiredSize=@0xbfffb700, aReflowState=@0xbfffb658, aStatus=@0xbfffb73c) at ../../../layout/generic/nsViewportFrame.cpp:281
#35 0x189ef5d9 in PresShell::DoReflow (this=0x243ee400, target=0x26440bb0, aInterruptible=1) at ../../../layout/base/nsPresShell.cpp:7168
#36 0x189f6e04 in PresShell::ProcessReflowCommands (this=0x243ee400, aInterruptible=1) at ../../../layout/base/nsPresShell.cpp:7297
#37 0x189f7206 in PresShell::FlushPendingNotifications (this=0x243ee400, aType=Flush_InterruptibleLayout) at ../../../layout/base/nsPresShell.cpp:4873
#38 0x189eaf1b in PresShell::ReflowEvent::Run (this=0x26fa3240) at ../../../layout/base/nsPresShell.cpp:6983
#39 0x0056f3be in nsThread::ProcessNextEvent (this=0x714fc0, mayWait=1, result=0xbfffb9ac) at ../../../xpcom/threads/nsThread.cpp:527
#40 0x004f450c in NS_ProcessNextEvent_P (thread=0x714fc0, mayWait=1) at nsThreadUtils.cpp:230
#41 0x12899082 in nsXULWindow::ShowModal (this=0x275847b0) at ../../../../xpfe/appshell/src/nsXULWindow.cpp:414
#42 0x1288fd14 in nsContentTreeOwner::ShowAsModal (this=0x250703d0) at ../../../../xpfe/appshell/src/nsContentTreeOwner.cpp:528
#43 0x1278e717 in nsWindowWatcher::OpenWindowJSInternal (this=0x75d4d0, aParent=0x24c3d160, aUrl=0x127d4260 "chrome://global/content/commonDialog.xul", aName=0x127d40b4 "_blank", aFeatures=0x127d44d8 "centerscreen,chrome,modal,titlebar", aDialog=1, argv=0x274498b0, aCalledFromJS=0, _retval=0xbfffbf78) at ../../../../../embedding/components/windowwatcher/src/nsWindowWatcher.cpp:991
#44 0x1278edb8 in nsWindowWatcher::OpenWindow (this=0x75d4d0, aParent=0x24c3d160, aUrl=0x127d4260 "chrome://global/content/commonDialog.xul", aName=0x127d40b4 "_blank", aFeatures=0x127d44d8 "centerscreen,chrome,modal,titlebar", aArguments=0x248dfd60, _retval=0xbfffbf78) at ../../../../../embedding/components/windowwatcher/src/nsWindowWatcher.cpp:423
#45 0x1279b0bd in nsPromptService::DoDialog (this=0x1c7edda0, aParent=0x24c3d160, aParamBlock=0x248dfd60, aChromeURL=0x127d4260 "chrome://global/content/commonDialog.xul") at ../../../../../embedding/components/windowwatcher/src/nsPromptService.cpp:795
#46 0x1279a13c in nsPromptService::ConfirmEx (this=0x1c7edda0, parent=0x24c3d160, dialogTitle=0x27240a30, text=0x28e29c08, buttonFlags=0, button0Title=0x27271c80, button1Title=0x26fac660, button2Title=0x26faad60, checkMsg=0x24ea6ff0, checkValue=0xbfffc2bc, buttonPressed=0xbfffc2c0) at ../../../../../embedding/components/windowwatcher/src/nsPromptService.cpp:402
#47 0x1278247f in nsPrompt::ConfirmEx (this=0x273a11b0, dialogTitle=0x27240a30, text=0x28e29c08, buttonFlags=8355711, button0Title=0x27271c80, button1Title=0x26fac660, button2Title=0x26faad60, checkMsg=0x24ea6ff0, checkValue=0xbfffc2bc, buttonPressed=0xbfffc2c0) at ../../../../../embedding/components/windowwatcher/src/nsPrompt.cpp:243
#48 0x18f2f858 in nsJSContext::DOMOperationCallback (cx=0x26717a00) at ../../../dom/base/nsJSEnvironment.cpp:1119
#49 0x0029e89f in js_InvokeOperationCallback (cx=0x26717a00) at ../../../js/src/jscntxt.cpp:1766
#50 0x002db204 in js_Interpret (cx=0x26717a00) at ../../../js/src/jsinterp.cpp:3204
#51 0x0030763e in js_Execute (cx=0x26717a00, chain=0x162d7100, script=0x26fa2e40, down=0x0, flags=0, result=0x0) at jsinterp.cpp:1635
#52 0x002867e1 in JS_EvaluateUCScriptForPrincipals (cx=0x26717a00, obj=0x162d7100, principals=0x26ce0dc4, chars=0x245f6408, length=380, filename=0x27304958 "file:///Users/gal/Desktop/x.html", lineno=3, rval=0x0) at ../../../js/src/jsapi.cpp:5158
#53 0x18f2a263 in nsJSContext::EvaluateString (this=0x26ad78f0, aScript=@0xbfffceb4, aScopeObject=0x162d7100, aPrincipal=0x26ce0dc0, aURL=0x27304958 "file:///Users/gal/Desktop/x.html", aLineNo=3, aVersion=0, aRetValue=0x0, aIsUndefined=0xbfffce34) at ../../../dom/base/nsJSEnvironment.cpp:1682
#54 0x18cfd9c2 in nsScriptLoader::EvaluateScript (this=0x26ceae00, aRequest=0x26826a60, aScript=@0xbfffceb4) at ../../../../content/base/src/nsScriptLoader.cpp:686
#55 0x18cfdc68 in nsScriptLoader::ProcessRequest (this=0x26ceae00, aRequest=0x26826a60) at ../../../../content/base/src/nsScriptLoader.cpp:600
#56 0x18cfeece in nsScriptLoader::ProcessScriptElement (this=0x26ceae00, aElement=0x26f89bf4) at ../../../../content/base/src/nsScriptLoader.cpp:554
#57 0x18cfa4da in nsScriptElement::MaybeProcessScript (this=0x26f89bf4) at ../../../../content/base/src/nsScriptElement.cpp:193
#58 0x18dcdfc1 in nsHTMLScriptElement::MaybeProcessScript (this=0x26f89bd0) at ../../../../../content/html/content/src/nsHTMLScriptElement.cpp:547
#59 0x18dcd275 in nsHTMLScriptElement::DoneAddingChildren (this=0x26f89bd0, aHaveNotified=1) at ../../../../../content/html/content/src/nsHTMLScriptElement.cpp:484
#60 0x18dfe171 in HTMLContentSink::ProcessSCRIPTEndTag (this=0x2676e400, content=0x26f89bd0, aMalformed=0) at ../../../../../content/html/document/src/nsHTMLContentSink.cpp:3096
#61 0x18dff11f in SinkContext::CloseContainer (this=0x26d0b3d0, aTag=eHTMLTag_script, aMalformed=0) at ../../../../../content/html/document/src/nsHTMLContentSink.cpp:1014
#62 0x18e0185d in HTMLContentSink::CloseContainer (this=0x2676e400, aTag=eHTMLTag_script) at ../../../../../content/html/document/src/nsHTMLContentSink.cpp:2376
#63 0x1bf27728 in CNavDTD::CloseContainer (this=0x24f74a40, aTag=eHTMLTag_script, aMalformed=0) at ../../../../parser/htmlparser/src/CNavDTD.cpp:2762
#64 0x1bf2b137 in CNavDTD::HandleEndToken (this=0x24f74a40, aToken=0x26011420) at ../../../../parser/htmlparser/src/CNavDTD.cpp:1641
#65 0x1bf29abc in CNavDTD::HandleToken (this=0x24f74a40, aToken=0x26011420) at ../../../../parser/htmlparser/src/CNavDTD.cpp:721
#66 0x1bf2b858 in CNavDTD::BuildModel (this=0x24f74a40, aTokenizer=0x24f74b00, aCanInterrupt=1, aCountLines=1) at ../../../../parser/htmlparser/src/CNavDTD.cpp:304
#67 0x1bf3654f in nsParser::BuildModel (this=0x26dfa090) at ../../../../parser/htmlparser/src/nsParser.cpp:2452
#68 0x1bf3be33 in nsParser::ResumeParse (this=0x26dfa090, allowIteration=1, aIsFinalChunk=0, aCanInterrupt=1) at ../../../../parser/htmlparser/src/nsParser.cpp:2333
#69 0x1bf3b680 in nsParser::OnDataAvailable (this=0x26dfa090, request=0x26fb9f50, aContext=0x0, pIStream=0x26fba17c, sourceOffset=0, aLength=489) at ../../../../parser/htmlparser/src/nsParser.cpp:2981
#70 0x187742eb in nsDocumentOpenInfo::OnDataAvailable (this=0x24a8d0a0, request=0x26fb9f50, aCtxt=0x0, inStr=0x26fba17c, sourceOffset=0, count=489) at ../../../uriloader/base/nsURILoader.cpp:306
#71 0x11419f18 in nsBaseChannel::OnDataAvailable (this=0x26fb9f20, request=0x26fba110, ctxt=0x0, stream=0x26fba17c, offset=0, count=489) at ../../../../netwerk/base/src/nsBaseChannel.cpp:708
#72 0x1142d83d in nsInputStreamPump::OnStateTransfer (this=0x26fba110) at ../../../../netwerk/base/src/nsInputStreamPump.cpp:508
#73 0x1142dd8e in nsInputStreamPump::OnInputStreamReady (this=0x26fba110, stream=0x26fba17c) at ../../../../netwerk/base/src/nsInputStreamPump.cpp:398
#74 0x0053ba26 in nsInputStreamReadyEvent::Run (this=0x24fdaf30) at ../../../xpcom/io/nsStreamUtils.cpp:111
#75 0x0056f3be in nsThread::ProcessNextEvent (this=0x714fc0, mayWait=0, result=0xbfffdb94) at ../../../xpcom/threads/nsThread.cpp:527
#76 0x004f4642 in NS_ProcessPendingEvents_P (thread=0x714fc0, timeout=20) at nsThreadUtils.cpp:180
#77 0x1187e267 in nsBaseAppShell::NativeEventCallback (this=0x743180) at ../../../../widget/src/xpwidgets/nsBaseAppShell.cpp:121
#78 0x11834546 in nsAppShell::ProcessGeckoEvents (aInfo=0x743180) at ../../../../widget/src/cocoa/nsAppShell.mm:413
#79 0x92da8595 in CFRunLoopRunSpecific ()
#80 0x92da8c78 in CFRunLoopRunInMode ()
#81 0x94fdc28c in RunCurrentEventLoopInMode ()
#82 0x94fdbfde in ReceiveNextEventCommon ()
#83 0x94fdbf19 in BlockUntilNextEventMatchingListInMode ()
#84 0x9340ad0d in _DPSNextEvent ()
#85 0x9340a5c0 in -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] ()
#86 0x934035fb in -[NSApplication run] ()
#87 0x11832570 in nsAppShell::Run (this=0x743180) at ../../../../widget/src/cocoa/nsAppShell.mm:766
#88 0x1250eda6 in nsAppStartup::Run (this=0x75cd80) at ../../../../../toolkit/components/startup/src/nsAppStartup.cpp:193
#89 0x000ace2f in XRE_main (argc=1, argv=0xbffff204, aAppData=0x70e630) at ../../../toolkit/xre/nsAppRunner.cpp:3369
#90 0x000027db in main (argc=1, argv=0xbffff204) at ../../../browser/app/nsBrowserApp.cpp:156
Crash occurs deep in graphics code, not JS.
Looks like we get a special prize from Core Text? adding roc and mr kew
Assignee: general → nobody
Component: JavaScript Engine → Layout: Text
QA Contact: general → layout.fonts-and-text
Looks like CTLineCreateWithAttributedString loses its little mind on MacOSX/Carbon (Apple's code, not ours) and somewhat safely crashes with an unchecked malloc. The symptoms on Windows likely differ. Any volunteers?
Keywords: crash, testcase
Does this affect anything older than 1.9.1?
The reported version cannot be accurate, as gfxCoreTextFonts.cpp doesn't even exist on 1.9.1; see http://mxr.mozilla.org/mozilla1.9.1/source/gfx/thebes/src/.

It also cannot relate to our trunk nightlies, as our builds do not have the Core Text backend yet. So this could only happen in a custom-configured trunk build. (Nevertheless, it's obviously something we need to look at, as this will eventually replace our ATSUI text path.)

Could someone cc me on the duplicate bug 504343 to see if there's any different info there?
We should reproduce on Windows as well. There might be separate/different bug in the win32 font stack that is triggered by the same overlong input.
(In reply to comment #9)
> Could someone cc me on the duplicate bug 504343 to see if there's any different
> info there?

That bug is just somebody noticing the thing on milw0rm and filing a bug. No new information.
Fwiw, I tried to reproduce the crash on windows, but I couldn't. I only got a slow script dialog warning after it was using >600MB, I decided to stop the process.
(In reply to comment #11)
> That bug is just somebody noticing the thing on milw0rm and filing a bug. No
> new information.

Except that bug 504343 refers to Windows, assuming the report is accurate; we certainly need to test it there as well. Pity they didn't provide more detail.
I tried this on Firefox 3.0.10 running on Windows XP, and (after a period of unresponsiveness) I got a crash there too. Submitted via the Mozilla Crash Reporter. I'm not sure if there's a way for me to search for it among those reports (can I?), but it should show the page address as being a local file called "milw0rm9158.html".
(In reply to comment #14)
> I'm not sure if there's a way for me to search for it among those
> reports (can I?)

Visit about:crashes in your browser.
So in the 3.0.10 case on XP, the crash occurred because operator new threw an exception. I expect there are a bunch of places that can happen, if a Javascript program consumes all available memory, or creates such huge documents that the layout/rendering process runs out of memory building frames, textruns, etc.
Attached file sample from fx3.0.11 — 
In 3.0.11 on Mac, the testcase hangs for quite a while and noticeably slows down my machine but never crashes. It recovers from the hang, then proceeds to hang again after that, and so on, until you either navigate away from the page or quit the browser. Here's a sample from the hang.
Firefox 3.0.11 on Windows crashes with this stack (taken from bp-f9f601c0-4b8d-469e-a275-1f45c2090715): 

Frame  	Module  	Signature  	Source
0 	kernel32.dll 	kernel32.dll@0x12aeb 	
1 	mozcrt19.dll 	_CxxThrowException 	throw.cpp:159
2 	mozcrt19.dll 	operator new(unsigned int) 	new.cpp:57
3 	xul.dll 	gfxTextRun::AllocateDetailedGlyphs(unsigned int,unsigned int) 	mozilla/gfx/thebes/src/gfxFont.cpp:1982
4 	xul.dll 	gfxTextRun::SetMissingGlyph(unsigned int,unsigned int) 	mozilla/gfx/thebes/src/gfxFont.cpp:2019
5 	xul.dll 	xul.dll@0x2841f1
The published exploit claims a "stack overflow". I don't think anything we are seeing so far points in that direction.
copy of testcase with correct MIME type for convenience
Is anyone seeing anything other than new throwing on OOM? Does not look exploitable.
Summary: unicode stack overflow (milw0rm 9158) → new throws on oom ("unicode stack overflow" from milw0rm 9158)
Whiteboard: [sg:investigate] → [sg:dos] new throws on oom
The Windows crash (OOM exception) doesn't seem exploitable, though I worry that with the right conditions the allocation failure might happen in a different place and be handled less well (e.g., an unchecked malloc failure).

The crash on OS X (CoreText-enabled trunk build, not shipping product ATM) is a bit different; do we know whether this is definitely an unchecked malloc with a "somewhat safe" crash (comment 7)? The stack in comment 4 doesn't seem to indicate this, AFAICT.
#22: I will catch it in gdb again and check.
Slightly different stack this time, but same ballpark. Again a failed malloc followed by a dereferencing of the NULL pointer.

Btw, maybe we want to consider hooking into the malloc_error_break function in product builds? We could selectively catch such malloc failures inside Apple's code and crash safely instead of hoping that the libraries do the right thing.

Talking about Apple, maybe we should report this to them before opening up the bug. Chances are more applications use the same buggy API (Safari? Chrome?).

firefox-bin(49275,0xa04b0720) malloc: *** mmap(size=336003072) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00000000
0x929471b1 in TRun::CacheGlyphPositions ()
(gdb) bt
#0  0x929471b1 in TRun::CacheGlyphPositions ()
#1  0x9294cbe0 in TLine::CachePositions ()
#2  0x9294d2be in TLine::GetPositionsForRun ()
#3  0x9294705e in TRun::GetPositions ()
#4  0x119a4cda in gfxCoreTextFontGroup::SetGlyphsFromRun (this=0x1ed4bdc0, aTextRun=0x66870000, aCTRun=0x1ed54880, aUnmatched=0x0, aLayoutStart=2, aLayoutLength=42000001) at ../../../../gfx/thebes/src/gfxCoreTextFonts.cpp:993
#5  0x119a5d13 in gfxCoreTextFontGroup::InitTextRun (this=0x1ed4bdc0, aTextRun=0x66870000, aString=0x708a8008, aTotalLength=42000006, aLayoutStart=2, aLayoutLength=42000001) at ../../../../gfx/thebes/src/gfxCoreTextFonts.cpp:926
#6  0x119a5ecc in gfxCoreTextFontGroup::MakeTextRun (this=0x1ed4bdc0, aString=0x61854008, aLength=42000001, aParams=0xbfff4060, aFlags=16779265) at ../../../../gfx/thebes/src/gfxCoreTextFonts.cpp:742
#7  0x11994268 in TextRunWordCache::MakeTextRun (this=0x12b97320, aText=0x4830a008, aLength=42000000, aFontGroup=0x1ed4bdc0, aParams=0xbfff547c, aFlags=16779264) at ../../../../gfx/thebes/src/gfxTextRunWordCache.cpp:683
#8  0x11994385 in gfxTextRunWordCache::MakeTextRun (aText=0x4830a008, aLength=42000000, aFontGroup=0x1ed4bdc0, aParams=0xbfff547c, aFlags=16779264) at ../../../../gfx/thebes/src/gfxTextRunWordCache.cpp:992
#9  0x18ab7fc6 in MakeTextRun (aText=0x4830a008, aLength=42000000, aFontGroup=0x1ed4bdc0, aParams=0xbfff547c, aFlags=16779264) at ../../../layout/generic/nsTextFrameThebes.cpp:436
#10 0x18abb1a0 in BuildTextRunsScanner::BuildTextRunForFrames (this=0xbfff6640, aTextBuffer=0x4d325d08) at ../../../layout/generic/nsTextFrameThebes.cpp:1809
#11 0x18abb601 in BuildTextRunsScanner::FlushFrames (this=0xbfff6640, aFlushLineBreaks=1, aSuppressTrailingBreak=0) at ../../../layout/generic/nsTextFrameThebes.cpp:1217
#12 0x18abc282 in BuildTextRuns (aContext=0x1ed4e670, aForFrame=0xb5e26c, aLineContainer=0x1f2f5738, aForFrameLine=0xbfff71ec) at ../../../layout/generic/nsTextFrameThebes.cpp:1148
#13 0x18abc3bb in nsTextFrame::EnsureTextRun (this=0xb5e26c, aReferenceContext=0x1ed4e670, aLineContainer=0x1f2f5738, aLine=0xbfff71ec, aFlowEndInTextRun=0xbfff6cb4) at ../../../layout/generic/nsTextFrameThebes.cpp:2002
#14 0x18ac0163 in nsTextFrame::Reflow (this=0xb5e26c, aPresContext=0x1f560400, aMetrics=@0xbfff6ef8, aReflowState=@0xbfff6e50, aStatus=@0xbfff7034) at ../../../layout/generic/nsTextFrameThebes.cpp:6133
#15 0x18a82bdf in nsLineLayout::ReflowFrame (this=0xbfff71d0, aFrame=0xb5e26c, aReflowStatus=@0xbfff7034, aMetrics=0x0, aPushedFrame=@0xbfff7030) at ../../../layout/generic/nsLineLayout.cpp:844
#16 0x18a1239f in nsBlockFrame::ReflowInlineFrame (this=0x1f2f5738, aState=@0xbfff77cc, aLineLayout=@0xbfff71d0, aLine={mCurrent = 0xb5e2b0, mListLink = 0x1f2f577c}, aFrame=0xb5e26c, aLineReflowStatus=0xbfff70f0) at ../../../layout/generic/nsBlockFrame.cpp:3728
#17 0x18a18e5e in nsBlockFrame::DoReflowInlineFrames (this=0x1f2f5738, aState=@0xbfff77cc, aLineLayout=@0xbfff71d0, aLine={mCurrent = 0xb5e2b0, mListLink = 0x1f2f577c}, aFloatAvailableSpace=@0xbfff728c, aAvailableSpaceHeight=@0xbfff7284, aFloatStateBeforeLine=0xbfff7270, aKeepReflowGoing=0xbfff74cc, aLineReflowStatus=0xbfff7288, aAllowPullUp=1) at ../../../layout/generic/nsBlockFrame.cpp:3544
#18 0x18a19822 in nsBlockFrame::ReflowInlineFrames (this=0x1f2f5738, aState=@0xbfff77cc, aLine={mCurrent = 0xb5e2b0, mListLink = 0x1f2f577c}, aKeepReflowGoing=0xbfff74cc) at ../../../layout/generic/nsBlockFrame.cpp:3394
#19 0x18a19de8 in nsBlockFrame::ReflowLine (this=0x1f2f5738, aState=@0xbfff77cc, aLine={mCurrent = 0xb5e2b0, mListLink = 0x1f2f577c}, aKeepReflowGoing=0xbfff74cc) at ../../../layout/generic/nsBlockFrame.cpp:2438
#20 0x18a1a708 in nsBlockFrame::ReflowDirtyLines (this=0x1f2f5738, aState=@0xbfff77cc) at ../../../layout/generic/nsBlockFrame.cpp:1935
#21 0x18a1db7a in nsBlockFrame::Reflow (this=0x1f2f5738, aPresContext=0x1f560400, aMetrics=@0xbfff7c04, aReflowState=@0xbfff7b38, aStatus=@0xbfff7c84) at ../../../layout/generic/nsBlockFrame.cpp:970
#22 0x18a218dd in nsBlockReflowContext::ReflowBlock (this=0xbfff7be0, aSpace=@0xbfff7c94, aApplyTopMargin=1, aPrevMargin=@0xbfff836c, aClearance=0, aIsAdjacentWithTop=1, aLine=0x1f2f5794, aFrameRS=@0xbfff7b38, aFrameReflowStatus=@0xbfff7c84, aState=@0xbfff82ec) at ../../../layout/generic/nsBlockReflowContext.cpp:310
#23 0x18a176c2 in nsBlockFrame::ReflowBlockFrame (this=0x1f2f5510, aState=@0xbfff82ec, aLine={mCurrent = 0x1f2f5794, mListLink = 0x1f2f5554}, aKeepReflowGoing=0xbfff7fec) at ../../../layout/generic/nsBlockFrame.cpp:3111
#24 0x18a19b18 in nsBlockFrame::ReflowLine (this=0x1f2f5510, aState=@0xbfff82ec, aLine={mCurrent = 0x1f2f5794, mListLink = 0x1f2f5554}, aKeepReflowGoing=0xbfff7fec) at ../../../layout/generic/nsBlockFrame.cpp:2383
#25 0x18a1a708 in nsBlockFrame::ReflowDirtyLines (this=0x1f2f5510, aState=@0xbfff82ec) at ../../../layout/generic/nsBlockFrame.cpp:1935
#26 0x18a1db7a in nsBlockFrame::Reflow (this=0x1f2f5510, aPresContext=0x1f560400, aMetrics=@0xbfff8644, aReflowState=@0xbfff859c, aStatus=@0xbfff88b8) at ../../../layout/generic/nsBlockFrame.cpp:970
#27 0x18a2cf33 in nsContainerFrame::ReflowChild (this=0x1f527e8c, aKidFrame=0x1f2f5510, aPresContext=0x1f560400, aDesiredSize=@0xbfff8644, aReflowState=@0xbfff859c, aX=0, aY=0, aFlags=0, aStatus=@0xbfff88b8, aTracker=0x0) at ../../../layout/generic/nsContainerFrame.cpp:825
#28 0x18a66986 in CanvasFrame::Reflow (this=0x1f527e8c, aPresContext=0x1f560400, aDesiredSize=@0xbfff8934, aReflowState=@0xbfff8800, aStatus=@0xbfff88b8) at ../../../layout/generic/nsHTMLFrame.cpp:652
#29 0x18a2cf33 in nsContainerFrame::ReflowChild (this=0x1f309410, aKidFrame=0x1f527e8c, aPresContext=0x1f560400, aDesiredSize=@0xbfff8934, aReflowState=@0xbfff8800, aX=0, aY=0, aFlags=3, aStatus=@0xbfff88b8, aTracker=0x0) at ../../../layout/generic/nsContainerFrame.cpp:825
#30 0x18a5972a in nsHTMLScrollFrame::ReflowScrolledFrame (this=0x1f309410, aState=0xbfff8a0c, aAssumeHScroll=0, aAssumeVScroll=1, aMetrics=0xbfff8934, aFirstPass=1) at ../../../layout/generic/nsGfxScrollFrame.cpp:528
#31 0x18a5f1a9 in nsHTMLScrollFrame::ReflowContents (this=0x1f309410, aState=0xbfff8a0c, aDesiredSize=@0xbfff8d04) at ../../../layout/generic/nsGfxScrollFrame.cpp:622
#32 0x18a5f985 in nsHTMLScrollFrame::Reflow (this=0x1f309410, aPresContext=0x1f560400, aDesiredSize=@0xbfff8d04, aReflowState=@0xbfff8bb4, aStatus=@0xbfff8edc) at ../../../layout/generic/nsGfxScrollFrame.cpp:823
#33 0x18a2cf33 in nsContainerFrame::ReflowChild (this=0x1f527da8, aKidFrame=0x1f309410, aPresContext=0x1f560400, aDesiredSize=@0xbfff8d04, aReflowState=@0xbfff8bb4, aX=0, aY=0, aFlags=0, aStatus=@0xbfff8edc, aTracker=0x0) at ../../../layout/generic/nsContainerFrame.cpp:825
#34 0x18acdc51 in ViewportFrame::Reflow (this=0x1f527da8, aPresContext=0x1f560400, aDesiredSize=@0xbfff8ea0, aReflowState=@0xbfff8df8, aStatus=@0xbfff8edc) at ../../../layout/generic/nsViewportFrame.cpp:281
#35 0x189ef5d9 in PresShell::DoReflow (this=0x1f2b9200, target=0x1f527da8, aInterruptible=1) at ../../../layout/base/nsPresShell.cpp:7168
#36 0x189f6e04 in PresShell::ProcessReflowCommands (this=0x1f2b9200, aInterruptible=1) at ../../../layout/base/nsPresShell.cpp:7297
#37 0x189f7206 in PresShell::FlushPendingNotifications (this=0x1f2b9200, aType=Flush_InterruptibleLayout) at ../../../layout/base/nsPresShell.cpp:4873
#38 0x189eaf1b in PresShell::ReflowEvent::Run (this=0x16c8e760) at ../../../layout/base/nsPresShell.cpp:6983
#39 0x0056f3be in nsThread::ProcessNextEvent (this=0x7153d0, mayWait=1, result=0xbfff914c) at ../../../xpcom/threads/nsThread.cpp:527
#40 0x004f450c in NS_ProcessNextEvent_P (thread=0x7153d0, mayWait=1) at nsThreadUtils.cpp:230
#41 0x12899082 in nsXULWindow::ShowModal (this=0x1650b7e0) at ../../../../xpfe/appshell/src/nsXULWindow.cpp:414
#42 0x1288fd14 in nsContentTreeOwner::ShowAsModal (this=0x1650ce80) at ../../../../xpfe/appshell/src/nsContentTreeOwner.cpp:528
#43 0x1278e717 in nsWindowWatcher::OpenWindowJSInternal (this=0x75d8f0, aParent=0x1d34a180, aUrl=0x127d4260 "chrome://global/content/commonDialog.xul", aName=0x127d40b4 "_blank", aFeatures=0x127d44d8 "centerscreen,chrome,modal,titlebar", aDialog=1, argv=0x1650b480, aCalledFromJS=0, _retval=0xbfff9718) at ../../../../../embedding/components/windowwatcher/src/nsWindowWatcher.cpp:991
#44 0x1278edb8 in nsWindowWatcher::OpenWindow (this=0x75d8f0, aParent=0x1d34a180, aUrl=0x127d4260 "chrome://global/content/commonDialog.xul", aName=0x127d40b4 "_blank", aFeatures=0x127d44d8 "centerscreen,chrome,modal,titlebar", aArguments=0x1650b150, _retval=0xbfff9718) at ../../../../../embedding/components/windowwatcher/src/nsWindowWatcher.cpp:423
#45 0x1279b0bd in nsPromptService::DoDialog (this=0x1c7e3d10, aParent=0x1d34a180, aParamBlock=0x1650b150, aChromeURL=0x127d4260 "chrome://global/content/commonDialog.xul") at ../../../../../embedding/components/windowwatcher/src/nsPromptService.cpp:795
#46 0x1279a13c in nsPromptService::ConfirmEx (this=0x1c7e3d10, parent=0x1d34a180, dialogTitle=0x1650ae30, text=0x1f270608, buttonFlags=0, button0Title=0x16c857b0, button1Title=0x16c85810, button2Title=0x16c8eb90, checkMsg=0x1650ae70, checkValue=0xbfff9a5c, buttonPressed=0xbfff9a60) at ../../../../../embedding/components/windowwatcher/src/nsPromptService.cpp:402
#47 0x1278247f in nsPrompt::ConfirmEx (this=0x16cfc060, dialogTitle=0x1650ae30, text=0x1f270608, buttonFlags=8355711, button0Title=0x16c857b0, button1Title=0x16c85810, button2Title=0x16c8eb90, checkMsg=0x1650ae70, checkValue=0xbfff9a5c, buttonPressed=0xbfff9a60) at ../../../../../embedding/components/windowwatcher/src/nsPrompt.cpp:243
#48 0x18f2f858 in nsJSContext::DOMOperationCallback (cx=0xd54600) at ../../../dom/base/nsJSEnvironment.cpp:1119
#49 0x0029e89f in js_InvokeOperationCallback (cx=0xd54600) at ../../../js/src/jscntxt.cpp:1766
#50 0x002db204 in js_Interpret (cx=0xd54600) at ../../../js/src/jsinterp.cpp:3204
#51 0x0030763e in js_Execute (cx=0xd54600, chain=0x12ad85e0, script=0x16c84d20, down=0x0, flags=0, result=0x0) at jsinterp.cpp:1635
#52 0x002867e1 in JS_EvaluateUCScriptForPrincipals (cx=0xd54600, obj=0x12ad85e0, principals=0x16c78b54, chars=0x1f2c5e08, length=456, filename=0x16c72538 "https://bug504342.bugzilla.mozilla.org/attachment.cgi?id=388892&t=ci5IP38aHK", lineno=3, rval=0x0) at ../../../js/src/jsapi.cpp:5158
#53 0x18f2a263 in nsJSContext::EvaluateString (this=0x1ca66380, aScript=@0xbfffa654, aScopeObject=0x12ad85e0, aPrincipal=0x16c78b50, aURL=0x16c72538 "https://bug504342.bugzilla.mozilla.org/attachment.cgi?id=388892&t=ci5IP38aHK", aLineNo=3, aVersion=0, aRetValue=0x0, aIsUndefined=0xbfffa5d4) at ../../../dom/base/nsJSEnvironment.cpp:1682
#54 0x18cfd9c2 in nsScriptLoader::EvaluateScript (this=0x16c78a70, aRequest=0x16c84b10, aScript=@0xbfffa654) at ../../../../content/base/src/nsScriptLoader.cpp:686
#55 0x18cfdc68 in nsScriptLoader::ProcessRequest (this=0x16c78a70, aRequest=0x16c84b10) at ../../../../content/base/src/nsScriptLoader.cpp:600
#56 0x18cfeece in nsScriptLoader::ProcessScriptElement (this=0x16c78a70, aElement=0x16c84884) at ../../../../content/base/src/nsScriptLoader.cpp:554
#57 0x18cfa4da in nsScriptElement::MaybeProcessScript (this=0x16c84884) at ../../../../content/base/src/nsScriptElement.cpp:193
#58 0x18dcdfc1 in nsHTMLScriptElement::MaybeProcessScript (this=0x16c84860) at ../../../../../content/html/content/src/nsHTMLScriptElement.cpp:547
#59 0x18dcd275 in nsHTMLScriptElement::DoneAddingChildren (this=0x16c84860, aHaveNotified=1) at ../../../../../content/html/content/src/nsHTMLScriptElement.cpp:484
#60 0x18dfe171 in HTMLContentSink::ProcessSCRIPTEndTag (this=0x1f4fbc00, content=0x16c84860, aMalformed=0) at ../../../../../content/html/document/src/nsHTMLContentSink.cpp:3096
#61 0x18dff11f in SinkContext::CloseContainer (this=0x16c84750, aTag=eHTMLTag_script, aMalformed=0) at ../../../../../content/html/document/src/nsHTMLContentSink.cpp:1014
#62 0x18e0185d in HTMLContentSink::CloseContainer (this=0x1f4fbc00, aTag=eHTMLTag_script) at ../../../../../content/html/document/src/nsHTMLContentSink.cpp:2376
#63 0x1bf2e728 in CNavDTD::CloseContainer (this=0x16c84030, aTag=eHTMLTag_script, aMalformed=0) at ../../../../parser/htmlparser/src/CNavDTD.cpp:2762
#64 0x1bf32137 in CNavDTD::HandleEndToken (this=0x16c84030, aToken=0x1f356c20) at ../../../../parser/htmlparser/src/CNavDTD.cpp:1641
#65 0x1bf30abc in CNavDTD::HandleToken (this=0x16c84030, aToken=0x1f356c20) at ../../../../parser/htmlparser/src/CNavDTD.cpp:721
#66 0x1bf32858 in CNavDTD::BuildModel (this=0x16c84030, aTokenizer=0x16c83f00, aCanInterrupt=1, aCountLines=1) at ../../../../parser/htmlparser/src/CNavDTD.cpp:304
#67 0x1bf3d54f in nsParser::BuildModel (this=0x16c78eb0) at ../../../../parser/htmlparser/src/nsParser.cpp:2452
#68 0x1bf42e33 in nsParser::ResumeParse (this=0x16c78eb0, allowIteration=1, aIsFinalChunk=0, aCanInterrupt=1) at ../../../../parser/htmlparser/src/nsParser.cpp:2333
#69 0x1bf42680 in nsParser::OnDataAvailable (this=0x16c78eb0, request=0x16c71d50, aContext=0x0, pIStream=0x16c83940, sourceOffset=0, aLength=591) at ../../../../parser/htmlparser/src/nsParser.cpp:2981
#70 0x187742eb in nsDocumentOpenInfo::OnDataAvailable (this=0x16c50110, request=0x16c71d50, aCtxt=0x0, inStr=0x16c83940, sourceOffset=0, count=591) at ../../../uriloader/base/nsURILoader.cpp:306
#71 0x11518d4d in nsStreamListenerWrapper::OnDataAvailable (this=0x16c352d0, aRequest=0x16c71d50, aContext=0x0, aInputStream=0x16c83940, aOffset=0, aCount=591) at ../../../../../netwerk/protocol/http/src/nsHttpChannel.cpp:5681
#72 0x0058c64b in NS_InvokeByIndex_P (that=0x16c352d0, methodIndex=5, paramCount=5, params=0xbfffb304) at ../../../../../../../xpcom/reflect/xptcall/src/md/unix/xptcinvoke_unixish_x86.cpp:179
#73 0x112280f8 in XPCWrappedNative::CallMethod (ccx=@0xbfffb550, mode=XPCWrappedNative::CALL_METHOD) at ../../../../../js/src/xpconnect/src/xpcwrappednative.cpp:2692
#74 0x112341f8 in XPC_WN_CallMethod (cx=0x868e00, obj=0x12ad9500, argc=5, argv=0x16c58490, vp=0xbfffb670) at ../../../../../js/src/xpconnect/src/xpcwrappednativejsops.cpp:1732
#75 0x00308c9e in js_Invoke (cx=0x868e00, argc=5, vp=0x16c58488, flags=2) at jsinterp.cpp:1362
#76 0x002f2e9f in js_Interpret (cx=0x868e00) at ../../../js/src/jsinterp.cpp:5215
#77 0x00308ce7 in js_Invoke (cx=0x868e00, argc=5, vp=0x16c58464, flags=0) at jsinterp.cpp:1370
#78 0x1122149d in nsXPCWrappedJSClass::CallMethod (this=0x1ca2a5a0, wrapper=0x16c72350, methodIndex=5, info=0x8a5148, nativeParams=0xbfffc354) at ../../../../../js/src/xpconnect/src/xpcwrappedjsclass.cpp:1647
#79 0x11218519 in nsXPCWrappedJS::CallMethod (this=0x16c72350, methodIndex=5, info=0x8a5148, params=0xbfffc354) at ../../../../../js/src/xpconnect/src/xpcwrappedjs.cpp:570
#80 0x0058c92e in PrepareAndDispatch (self=0x16c72eb0, methodIndex=5, args=0xbfffc474) at ../../../../../../../xpcom/reflect/xptcall/src/md/unix/xptcstubs_unixish_x86.cpp:93
#81 0x0058ca31 in nsXPTCStubBase::Stub5 (this=0x16c72eb0) at xptcstubsdef.inc:3
#82 0x11518d4d in nsStreamListenerWrapper::OnDataAvailable (this=0x16c77100, aRequest=0x16c71d50, aContext=0x0, aInputStream=0x16c83940, aOffset=0, aCount=591) at ../../../../../netwerk/protocol/http/src/nsHttpChannel.cpp:5681
#83 0x0058c64b in NS_InvokeByIndex_P (that=0x16c77100, methodIndex=5, paramCount=5, params=0xbfffc684) at ../../../../../../../xpcom/reflect/xptcall/src/md/unix/xptcinvoke_unixish_x86.cpp:179
#84 0x112280f8 in XPCWrappedNative::CallMethod (ccx=@0xbfffc8d0, mode=XPCWrappedNative::CALL_METHOD) at ../../../../../js/src/xpconnect/src/xpcwrappednative.cpp:2692
#85 0x112341f8 in XPC_WN_CallMethod (cx=0x868e00, obj=0x12ad8040, argc=5, argv=0x16c5843c, vp=0xbfffc9f0) at ../../../../../js/src/xpconnect/src/xpcwrappednativejsops.cpp:1732
#86 0x00308c9e in js_Invoke (cx=0x868e00, argc=5, vp=0x16c58434, flags=2) at jsinterp.cpp:1362
#87 0x002f2e9f in js_Interpret (cx=0x868e00) at ../../../js/src/jsinterp.cpp:5215
#88 0x00308ce7 in js_Invoke (cx=0x868e00, argc=5, vp=0x16c58410, flags=0) at jsinterp.cpp:1370
#89 0x1122149d in nsXPCWrappedJSClass::CallMethod (this=0x1ca2a5a0, wrapper=0x16c773e0, methodIndex=5, info=0x8a5148, nativeParams=0xbfffd6d4) at ../../../../../js/src/xpconnect/src/xpcwrappedjsclass.cpp:1647
#90 0x11218519 in nsXPCWrappedJS::CallMethod (this=0x16c773e0, methodIndex=5, info=0x8a5148, params=0xbfffd6d4) at ../../../../../js/src/xpconnect/src/xpcwrappedjs.cpp:570
#91 0x0058c92e in PrepareAndDispatch (self=0x16c77370, methodIndex=5, args=0xbfffd7f4) at ../../../../../../../xpcom/reflect/xptcall/src/md/unix/xptcstubs_unixish_x86.cpp:93
#92 0x0058ca31 in nsXPTCStubBase::Stub5 (this=0x16c77370) at xptcstubsdef.inc:3
#93 0x11487acf in nsHTTPCompressConv::do_OnDataAvailable (this=0x16c82de0, request=0x16c71d50, context=0x0, offset=0, buffer=0x1f282200 "<html>\n<head>\n<script language=\"JavaScript\" type=\"Text/Javascript\">\n        var str = unescape(\"%u4141%u4141\");\n        var str2 = unescape(\"%u0000%u0000\");\n        var finalstr2 = mul8(str2, 49000000"..., count=591) at ../../../../netwerk/streamconv/converters/nsHTTPCompressConv.cpp:375
#94 0x114881a9 in nsHTTPCompressConv::OnDataAvailable (this=0x16c82de0, request=0x16c71d50, aContext=0x0, iStr=0x16c82a10, aSourceOffset=0, aCount=308) at ../../../../netwerk/streamconv/converters/nsHTTPCompressConv.cpp:306
#95 0x114627dc in nsStreamListenerTee::OnDataAvailable (this=0x16c82e70, request=0x16c71d50, context=0x0, input=0x16c7366c, offset=0, count=308) at ../../../../netwerk/base/src/nsStreamListenerTee.cpp:97
#96 0x11508696 in nsHttpChannel::OnDataAvailable (this=0x16c71d20, request=0x16c74200, ctxt=0x0, input=0x16c7366c, offset=0, count=308) at ../../../../../netwerk/protocol/http/src/nsHttpChannel.cpp:5110
#97 0x1142d83d in nsInputStreamPump::OnStateTransfer (this=0x16c74200) at ../../../../netwerk/base/src/nsInputStreamPump.cpp:508
#98 0x1142dd8e in nsInputStreamPump::OnInputStreamReady (this=0x16c74200, stream=0x16c7366c) at ../../../../netwerk/base/src/nsInputStreamPump.cpp:398
#99 0x0053ba26 in nsInputStreamReadyEvent::Run (this=0x16c732a0) at ../../../xpcom/io/nsStreamUtils.cpp:111
#100 0x0056f3be in nsThread::ProcessNextEvent (this=0x7153d0, mayWait=0, result=0xbfffdb94) at ../../../xpcom/threads/nsThread.cpp:527
#101 0x004f4642 in NS_ProcessPendingEvents_P (thread=0x7153d0, timeout=20) at nsThreadUtils.cpp:180
#102 0x1187e267 in nsBaseAppShell::NativeEventCallback (this=0x7435b0) at ../../../../widget/src/xpwidgets/nsBaseAppShell.cpp:121
#103 0x11834546 in nsAppShell::ProcessGeckoEvents (aInfo=0x7435b0) at ../../../../widget/src/cocoa/nsAppShell.mm:413
#104 0x92da8595 in CFRunLoopRunSpecific ()
#105 0x92da8c78 in CFRunLoopRunInMode ()
#106 0x94fdc28c in RunCurrentEventLoopInMode ()
#107 0x94fdc0a5 in ReceiveNextEventCommon ()
#108 0x94fdbf19 in BlockUntilNextEventMatchingListInMode ()
#109 0x9340ad0d in _DPSNextEvent ()
#110 0x9340a5c0 in -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] ()
#111 0x934035fb in -[NSApplication run] ()
#112 0x11832570 in nsAppShell::Run (this=0x7435b0) at ../../../../widget/src/cocoa/nsAppShell.mm:766
#113 0x1250eda6 in nsAppStartup::Run (this=0x75d1a0) at ../../../../../toolkit/components/startup/src/nsAppStartup.cpp:193
#114 0x000ace2f in XRE_main (argc=1, argv=0xbffff204, aAppData=0x70ead0) at ../../../toolkit/xre/nsAppRunner.cpp:3369
#115 0x000027db in main (argc=1, argv=0xbffff204) at ../../../browser/app/nsBrowserApp.cpp:156
(gdb)
Aaron: while investigating this latest milw0rm "Firefox" bug we're pretty sure the Windows symptoms are safe OOM issues, but less sure about Mac when CoreText is enabled. Safari also crashes in a similar way.
Summary: new throws on oom ("unicode stack overflow" from milw0rm 9158) → Investigate milw0rm 9158 "unicode stack overflow"
Whiteboard: [sg:dos] new throws on oom → [sg:investigate] Win:new throws on oom; Mac: crash in Apple routines with CoreText enabled
(In reply to comment #24)
> Talking about Apple, maybe we should report this to them before opening up the
> bug. Chances are more applications use the same buggy API (Safari? Chrome?).
> 
> firefox-bin(49275,0xa04b0720) malloc: *** mmap(size=336003072) failed (error
> code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> 
> Program received signal EXC_BAD_ACCESS, Could not access memory.
> Reason: KERN_PROTECTION_FAILURE at address: 0x00000000
> 0x929471b1 in TRun::CacheGlyphPositions ()
> (gdb) bt
> #0  0x929471b1 in TRun::CacheGlyphPositions ()
> #1  0x9294cbe0 in TLine::CachePositions ()
> #2  0x9294d2be in TLine::GetPositionsForRun ()
> #3  0x9294705e in TRun::GetPositions ()

We've reported this and similar bugs in the past to Apple; they have so far had no interest in fixing such bugs in their font rendering subsystems, especially if they're in ATSUI and not CoreText.

Cc'ing jdaggett to confirm that.
Whiteboard: [sg:investigate] Win:new throws on oom; Mac: crash in Apple routines with CoreText enabled → [sg:investigate] Win:new throws on oom; Mac: crash in Apple routines (like Safari)
A Mozilla Security Blog on this matter (http://blog.mozilla.com/security/2009/07/19/milw0rm-9158-stack-overflow-crash-not-exploitable-cve-2009-2479/) states "In Firefox 3.5.x on Windows, the allocations are more robustly checked and no crash will result."

I don't see a basis for this statement after a quick glance at this thread, but I have definitely seen 3.5.1 crash on Vista SP2 using the Securityfocus POC. Here are the details:

Add-ons: {CAFEEFAC-0016-0000-0010-ABCDEFFEDCBA}:6.0.10,{CAFEEFAC-0016-0000-0013-ABCDEFFEDCBA}:6.0.13,jsdeobfuscator@adblockplus.org:1.5.4,longurlplease@darragh.curran:0.4.1,{20a82645-c095-46ed-80e3-08825760534b}:1.1,{972ce4c6-7e08-4474-a285-3208198ce6fd}:3.5.1
BuildID: 20090715094852
CrashTime: 1248048544
Email: larry@larryseltzer.com
InstallTime: 1247790894
ProductName: Firefox
SecondsSinceLastCrash: 1195
StartupTime: 1248047596
Theme: abstractPCNightly
Throttleable: 1
URL: http://mlb.mlb.com/index.jsp
Vendor: Mozilla
Version: 3.5.1

This report also contains technical information about the state of the application when it crashed.
Hi, Larry.  That statement was based on our testing, and analysis of the code path that crashes under FF3.  Could you share the crash report ID for that case with us? (You can find it via about:crashes.)  Does it crash in safe mode for you?  Lastly, are you running on 32-bit or 64-bit Vista?

Thanks for the note; it sounds like we might need to update that post if we find that the PoC can trigger a crash in some configurations.
I think all of these are crashes from this POC:

a543ab19-18e5-4632-986e-9826f2090719	7/19/2009	8:09 PM
af6091f0-9e10-43dc-bd0b-8ebed2090719	7/19/2009	7:50 PM
bdb48189-3e5c-44e1-b8d0-e39cd2090719	7/19/2009	10:26 AM
c457c533-4542-4ea1-acf1-a14bd2090718	7/18/2009	12:24 PM

All 32-bit. I haven't tried safe mode, but I will now.
Yes, it crashes in safe mode too:

048e91bf-ba57-4373-8db0-ba7902090719	7/19/2009	8:34 PM
Sticking "bp-" at the start of those IDs makes Bugzilla turn them into useful links:
bp-a543ab19-18e5-4632-986e-9826f2090719
bp-af6091f0-9e10-43dc-bd0b-8ebed2090719
bp-bdb48189-3e5c-44e1-b8d0-e39cd2090719
bp-c457c533-4542-4ea1-acf1-a14bd2090718
bp-048e91bf-ba57-4373-8db0-ba7902090719

The ones I looked at so far (one is still processing) all have the same thing at the top:
0  	kernel32.dll  	kernel32.dll@0x3fbae  	
1 	mozcrt19.dll 	_CxxThrowException 	throw.cpp:159
2 	mozcrt19.dll 	operator new 	obj-firefox/memory/jemalloc/src/new.cpp:57
OK, some different behaviour than we were seeing with our Vista testing, but no doubt that's the same operator-new-throw-on-alloc-failure we saw in FF3.  I'll update the post now, thanks for the information.
fwiw, using windows 7, browser crashes without crash id. i am getting a dialog box saying:

"Microsft Visual C++ Runtime Library"

Runtime Error!

Program: C:\Program Files\Minefile\firefox.exe
Same testcase with automatic reload.

FWIW, with both 1.9.1 and trunk builds on the Mac (i.e. *not* CoreText-enabled), this seems like a simple denial-of-service bug, the browser crashes via SIGABRT when it runs out of memory.  I'm not sure how this is exploitable on the Mac with non-CT builds.
Attached file WinDbg log — 
(In reply to comment #36)

i dont know if i done this right.
Crash-IDs with 3.5.1 on XP SP3:

bp-b024d98a-b929-4dcd-adc0-0bcf42090720
bp-4b4975e9-b4f6-45bf-988b-bbcc52090722

testcase loaded locally
Those are both "ignorable" OOM exceptions indicating a DOS, not any exploit.
Here is a crash report from the latest Trunk nightly:

bp-3b188a80-458b-4e9a-8156-705942090722

Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2a1pre) Gecko/20090722 ID:20090722042136
crash reports where the second frame are _CxxThrowException are ok. Please don't paste any more of them in this bug report.
Whiteboard: [sg:investigate] Win:new throws on oom; Mac: crash in Apple routines (like Safari) → Win:new throws on oom; Mac: crash in Apple routines (like Safari)
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: