Fixed used with anonymous delegates is tricky

I'm back (5 minutes later) to continue trying to understand a mystery with fixed. At work our problem didn't just involve fixed(), but also the usage of fixed pointers inside an anonymous delegate defined and called within the scope of the fixed block.

Andreas came up with a nice test that surprised us both when we saw the disassembly. Here is the code:

private delegate void Action();
private unsafe void FixedIsColdComfortToDelegates(int runNumber)
{
    byte[] array = new byte[1];
    fixed(byte *p = array)
    {
        Action action = () => { *p = 22; };
        Thread.Sleep(10);
        action();
    }
    if(22 != array[0])
    {
        Console.WriteLine("Experienced failure, run {0}!", runNumber);
    }
}

The disassembly was interesting, basically the fixed() statement was ignored! The question is why. Ultimately I found out the bug:

You can cause a fixed statement to be ignored by the JIT compiler if you simply include an anonymous delegate in the fixed block which references the fixed pointer. There is no need to run the delegate. Any additional usages of the fixed pointer in the fixed block will not prevent the fixed block from being ignored.

Here is my test program:

using System;
using System.Threading;

namespace FixedTest
{
    class FixedProblem
    {    
        private unsafe void Changer(byte *p)
        {
            // Do some allocations
            byte[] bigarray = new byte[1024*50];
            bigarray[3] = 34;
            GC.Collect();
            *p = bigarray[3];            
        }

        private unsafe void NormalFixedStatement(int runNumber)
        {            
            byte[] array = new byte[100];            
            fixed(byte *p = array)
            {
                Changer(p + 1); // will change array[1] to 34
                Thread.Sleep(10);
                *p = 10;            
            }

            if(array[1] != 34)
            {
                Console.WriteLine("Experienced failure, run {0}", runNumber);
            }
        }

        private unsafe void NullifiedFixedStatement(int runNumber)
        {            
            byte[] array = new byte[100];
            fixed (byte* p = array)
            {                                
                Action action = () => { *p = 10; }; // This destroys the fixed statement
                Changer(p + 1); // will change array[1] to 34
                Thread.Sleep(10);
                *p = 10; // but p isn't fixed!
            }

            if (array[1] != 34)
            {
                Console.WriteLine("Experienced failure, run {0}", runNumber);
            }
        }

        static void Main(string[] args)
        {
            FixedProblem p = new FixedProblem();
            
            int count = 500;

            Console.WriteLine("First run a function with a fixed statement and no anonymous delegate that uses the fixed pointer");
            for (int i = 0; i < count; i++)
            {
                p.NormalFixedStatement(i);
            }

            Console.WriteLine("Now run a function with a fixed statement and an anonymous delegate that uses the fixed pointer");
            for (int i = 0; i < count; i++)
            {
                p.NullifiedFixedStatement(i);
            }
        }
    }
}

Here is the output:

C:\FixedTest\bin\Release>fixedtest.exe
First run a function with a fixed statement and no anonymous delegate 
that uses the fixed pointer.


Now run a function with a fixed statement and an anonymous delegate that 
uses the fixed pointer.

Experienced failure, run 4
Experienced failure, run 8
Experienced failure, run 12
Experienced failure, run 16
Experienced failure, run 20
...

I wonder if the CLR team knows about this bug?

Update: My friend Josef points out that if you introduce a temporary variable and pass that in to the anonymous delegate then fixed protection is restored. Something like:

...
byte *pTemp = p;
Action action = () => { *pTemp = 10; }
...

Fixed statement not what you expect (or is it?)

Recently we ran into the problem at work that a fixed statement doesn't have the lifetime you expect. Consider:

...
fixed(byte *p = byteArray)
{
    UnmanagedCode(p);
    OtherStuff();
    Console.WriteLine("finished");
}

You might expect that p is held fixed until after the Console.WriteLine statement, but it's only held fixed to the last usage of p, so the fixed protection terminates at the start of the OtherStuff() line. This is something I used to know as a CLR dev, but since forgot. Do you want to see proof? Let's break out SOS.

Here is a little test function:

private unsafe void FixedToLastUsage() {
    DebugBreak();

    int size = 100;
    byte[] byteArray = new byte[size];
    byteArray[0] = 1;

    fixed(byte *p = byteArray) {
        Console.WriteLine("p = " + *p);
        byte* p2 = p + 1;

        // Now we won't reference p anymore but we'll do some other stuff.
        // A look at the gc encoding should prove that p isn't protected beyond this point.
        int sum = 0;
        for(int i=0;i<10;i++) {
            sum += i;
        }
        Console.WriteLine("sum = " + sum.ToString());
    }
}

The DebugBreak() is a pinvoke call to the Windows DebugBreak() function. This is just to make it easier for me to disassemble the function and use SOS in the Windows Debugger.

Here is the disassembly of the method:

0:000> .loadby mscorwks sos
0:000> !clrstack
OS Thread Id: 0xfe8 (0)
ESP       EIP     
0012f410 7c90120e [NDirectMethodFrameStandalone: 0012f410] FixedTest.Program.DebugBreak()
0012f420 00d3014d FixedTest.Program.FixedToLastUsage()
0012f474 00d300b4 FixedTest.Program.Main(System.String[])
0012f69c 79e71b4c [GCFrame: 0012f69c] 
0:000> !u d3014d
Normal JIT generated code
FixedTest.Program.FixedToLastUsage()
Begin 00d30108, size 159
00d30108 55              push    ebp
00d30109 8bec            mov     ebp,esp
00d3010b 83ec4c          sub     esp,4Ch
00d3010e 33c0            xor     eax,eax
00d30110 8945f4          mov     dword ptr [ebp-0Ch],eax
00d30113 8945ec          mov     dword ptr [ebp-14h],eax
00d30116 894dfc          mov     dword ptr [ebp-4],ecx
00d30119 833de430930000  cmp     dword ptr ds:[9330E4h],0
00d30120 7405            je      00d30127
00d30122 e89aa53979      call    mscorwks!JIT_DbgIsJustMyCode (7a0ca6c1)
00d30127 33d2            xor     edx,edx
00d30129 8955d4          mov     dword ptr [ebp-2Ch],edx
00d3012c 33d2            xor     edx,edx
00d3012e 8955e8          mov     dword ptr [ebp-18h],edx
00d30131 33d2            xor     edx,edx
00d30133 8955d8          mov     dword ptr [ebp-28h],edx
00d30136 33d2            xor     edx,edx
00d30138 8955f8          mov     dword ptr [ebp-8],edx
00d3013b c745e400000000  mov     dword ptr [ebp-1Ch],0
00d30142 33d2            xor     edx,edx
00d30144 8955f0          mov     dword ptr [ebp-10h],edx
00d30147 90              nop
00d30148 e8ffbec0ff      call    0093c04c (FixedTest.Program.DebugBreak(), mdToken: 06000001)
>>> 00d3014d 90              nop
00d3014e c745f864000000  mov     dword ptr [ebp-8],64h
00d30155 8b55f8          mov     edx,dword ptr [ebp-8]
00d30158 b902410c79      mov     ecx,offset mscorlib_ni+0x4102 (790c4102)
00d3015d e83620bfff      call    00922198 (JitHelp: CORINFO_HELP_NEWARR_1_VC)
00d30162 8945d0          mov     dword ptr [ebp-30h],eax
00d30165 8b45d0          mov     eax,dword ptr [ebp-30h]
00d30168 8945d8          mov     dword ptr [ebp-28h],eax
00d3016b 8b45d8          mov     eax,dword ptr [ebp-28h]
00d3016e 83780400        cmp     dword ptr [eax+4],0
00d30172 7705            ja      00d30179
00d30174 e883c13979      call    mscorwks!JIT_RngChkFail (7a0cc2fc)
00d30179 c6400801        mov     byte ptr [eax+8],1
00d3017d 8b45d8          mov     eax,dword ptr [ebp-28h]
00d30180 8945d4          mov     dword ptr [ebp-2Ch],eax
00d30183 837dd400        cmp     dword ptr [ebp-2Ch],0
00d30187 7409            je      00d30192
00d30189 8b45d4          mov     eax,dword ptr [ebp-2Ch]
00d3018c 83780400        cmp     dword ptr [eax+4],0
00d30190 7508            jne     00d3019a
00d30192 33d2            xor     edx,edx
00d30194 8955f4          mov     dword ptr [ebp-0Ch],edx
00d30197 90              nop
00d30198 eb14            jmp     00d301ae
00d3019a 8b45d4          mov     eax,dword ptr [ebp-2Ch]
00d3019d 83780400        cmp     dword ptr [eax+4],0
00d301a1 7705            ja      00d301a8
00d301a3 e854c13979      call    mscorwks!JIT_RngChkFail (7a0cc2fc)
00d301a8 8d4008          lea     eax,[eax+8]
00d301ab 8945f4          mov     dword ptr [ebp-0Ch],eax
00d301ae 90              nop
00d301af b920353379      mov     ecx,offset mscorlib_ni+0x273520 (79333520) (MT: System.Byte)
00d301b4 e8631ebfff      call    0092201c (JitHelp: CORINFO_HELP_NEWSFAST)
00d301b9 8945cc          mov     dword ptr [ebp-34h],eax
00d301bc 8b0530202a02    mov     eax,dword ptr ds:[22A2030h] ("p = ")
00d301c2 8945b8          mov     dword ptr [ebp-48h],eax
00d301c5 8b45cc          mov     eax,dword ptr [ebp-34h]
00d301c8 8b55f4          mov     edx,dword ptr [ebp-0Ch]
00d301cb 8955e0          mov     dword ptr [ebp-20h],edx
00d301ce 8b55e0          mov     edx,dword ptr [ebp-20h]
00d301d1 8a12            mov     dl,byte ptr [edx]
00d301d3 885004          mov     byte ptr [eax+4],dl
00d301d6 8b45cc          mov     eax,dword ptr [ebp-34h]
00d301d9 8945b4          mov     dword ptr [ebp-4Ch],eax
00d301dc 8b4db8          mov     ecx,dword ptr [ebp-48h]
00d301df 8b55b4          mov     edx,dword ptr [ebp-4Ch]
00d301e2 e869ca5978      call    mscorlib_ni+0x20cc50 (792ccc50) (System.String.Concat(System.Object, System.Object), mdToken: 060001c5)
00d301e7 8945c8          mov     dword ptr [ebp-38h],eax
00d301ea 8b4dc8          mov     ecx,dword ptr [ebp-38h]
00d301ed e82637a678      call    mscorlib_ni+0x6d3918 (79793918) (System.Console.WriteLine(System.String), mdToken: 060007c8)
00d301f2 90              nop
00d301f3 8b45f4          mov     eax,dword ptr [ebp-0Ch]
00d301f6 8945dc          mov     dword ptr [ebp-24h],eax
00d301f9 8b45dc          mov     eax,dword ptr [ebp-24h]
00d301fc 40              inc     eax
00d301fd 8945f0          mov     dword ptr [ebp-10h],eax
00d30200 33d2            xor     edx,edx
00d30202 8955ec          mov     dword ptr [ebp-14h],edx
00d30205 33d2            xor     edx,edx
00d30207 8955e8          mov     dword ptr [ebp-18h],edx
00d3020a 90              nop
00d3020b eb0b            jmp     00d30218
00d3020d 90              nop
00d3020e 8b45e8          mov     eax,dword ptr [ebp-18h]
00d30211 0145ec          add     dword ptr [ebp-14h],eax
00d30214 90              nop
00d30215 ff45e8          inc     dword ptr [ebp-18h]
00d30218 837de80a        cmp     dword ptr [ebp-18h],0Ah
00d3021c 0f9cc0          setl    al
00d3021f 0fb6c0          movzx   eax,al
00d30222 8945e4          mov     dword ptr [ebp-1Ch],eax
00d30225 837de400        cmp     dword ptr [ebp-1Ch],0
00d30229 75e2            jne     00d3020d
00d3022b 8b0534202a02    mov     eax,dword ptr ds:[22A2034h] ("sum = ")
00d30231 8945c4          mov     dword ptr [ebp-3Ch],eax
00d30234 8d4dec          lea     ecx,[ebp-14h]
00d30237 e8d40a5b78      call    mscorlib_ni+0x220d10 (792e0d10) (System.Int32.ToString(), mdToken: 06000b22)
00d3023c 8945c0          mov     dword ptr [ebp-40h],eax
00d3023f 8b55c0          mov     edx,dword ptr [ebp-40h]
00d30242 8b4dc4          mov     ecx,dword ptr [ebp-3Ch]
00d30245 e806ea5478      call    mscorlib_ni+0x1bec50 (7927ec50) (System.String.Concat(System.String, System.String), mdToken: 060001c9)
00d3024a 8945bc          mov     dword ptr [ebp-44h],eax
00d3024d 8b4dbc          mov     ecx,dword ptr [ebp-44h]
00d30250 e8c336a678      call    mscorlib_ni+0x6d3918 (79793918) (System.Console.WriteLine(System.String), mdToken: 060007c8)
00d30255 90              nop
00d30256 90              nop
00d30257 33d2            xor     edx,edx
00d30259 8955f4          mov     dword ptr [ebp-0Ch],edx
00d3025c 90              nop
00d3025d 8be5            mov     esp,ebp
00d3025f 5d              pop     ebp
00d30260 c3              ret

And here is the gc encoding for the method. What is gc encoding? I think the SOS help explains it best. (Depending on your feelings about SOS help you can thank or curse me...I wrote it over the Christmas holidays in 2002 or 2003, I forgot)

0:000> !help GCInfo
-------------------------------------------------------------------------------
!GCInfo (methoddesc address="" | code address="")

!GCInfo is especially useful for CLR Devs who are trying to determine if there 
is a bug in the JIT Compiler. It parses the GCEncoding for a method, which is a
compressed stream of data indicating when registers or stack locations contain 
managed objects. It is important to keep track of this information, because if 
a garbage collection occurs, the collector needs to know where roots are so it 
can update them with new object pointer values.

You should copy this gc info into a notepad on a second monitor and read it as you read the disassembly. It becomes clear pretty quick that EBP-0CH is our pointer p.

0:000> !GCInfo d3014d
entry point 00d30108
Normal JIT generated code
GC info 00931934
Method info block:
    method      size   = 0159
    prolog      size   = 17 
    epilog      size   =  4 
    epilog     count   =  1 
    epilog      end    = yes  
    callee-saved regs  = EBP 
    ebp frame          = yes  
    fully interruptible= yes  
    double align       = no  
    arguments size     =  0 DWORDs
    stack frame size   = 19 DWORDs
    untracked count    =  2 
    var ptr tab count  = 10 
    epilog        at   0155
    argTabOffset = 22  
82 59 D2 81 D3 | 
B9 93 F1 40 0A | 
22             | 

Pointer table:
04             |             [EBP-04H] an untracked  local
05             |             [EBP-0CH] an untracked pinned byref local
2C 24 82 35    | 0024..0159  [EBP-2CH] a  pointer
28 0A 82 2B    | 002E..0159  [EBP-28H] a  pointer
30 2F 0F       | 005D..006C  [EBP-30H] a  pointer
34 57 26       | 00B4..00DA  [EBP-34H] a  pointer
48 09 1D       | 00BD..00DA  [EBP-48H] a  pointer
4C 17 06       | 00D4..00DA  [EBP-4CH] a  pointer
38 0E 03       | 00E2..00E5  [EBP-38H] a  pointer
3C 4A 11       | 012C..013D  [EBP-3CH] a  pointer
40 0B 06       | 0137..013D  [EBP-40H] a  pointer
44 0E 03       | 0145..0148  [EBP-44H] a  pointer
B8 5A 40       | 005A        reg EAX becoming live
F1 07          | 0071        reg EAX becoming dead
40             | 0071        reg EAX becoming live
F2 01          | 008A        reg EAX becoming dead
F0 43          | 0095        reg EAX becoming live
F0 03          | 00A0        reg EAX becoming dead
40             | 00A0        reg EAX becoming live
03             | 00A3        reg EAX becoming dead
BF 40          | 00A3        reg EAX becoming live (iptr)
03             | 00A6        reg EAX becoming dead
F0 43          | 00B1        reg EAX becoming live
F1 BF 52       | 00C3        reg EDX becoming live (iptr)
16             | 00C9        reg EDX becoming dead
F0 4E          | 00D7        reg ECX becoming live
53             | 00DA        reg EDX becoming live
0D             | 00DF        reg ECX becoming dead
10             | 00DF        reg EDX becoming dead
4E             | 00E5        reg ECX becoming live
05             | 00EA        reg EAX becoming dead
08             | 00EA        reg ECX becoming dead
BF 44          | 00EE        reg EAX becoming live (iptr)
06             | 00F4        reg EAX becoming dead
F5 45          | 0129        reg EAX becoming live
BF 4E          | 012F        reg ECX becoming live (iptr)
0D             | 0134        reg ECX becoming dead
56             | 013A        reg EDX becoming live
4B             | 013D        reg ECX becoming live
0D             | 0142        reg ECX becoming dead
10             | 0142        reg EDX becoming dead
4E             | 0148        reg ECX becoming live
05             | 014D        reg EAX becoming dead
08             | 014D        reg ECX becoming dead
FF             | 

If you look for usages of EBP-0CH in the code, you'll find first the initialization to 0:

00d3010e 33c0            xor     eax,eax
00d30110 8945f4          mov     dword ptr [ebp-0Ch],eax

then it's population with the address of the first element in byteArray (itself stored as an object reference at location EBP-2CH).

00d3019a 8b45d4          mov     eax,dword ptr [ebp-2Ch]
00d3019d 83780400        cmp     dword ptr [eax+4],0
00d301a1 7705            ja      00d301a8
00d301a3 e854c13979      call    mscorwks!JIT_RngChkFail (7a0cc2fc)
00d301a8 8d4008          lea     eax,[eax+8] (here is the fixed(byte *p = byteArray) statement)
00d301ab 8945f4          mov     dword ptr [ebp-0Ch],eax

The last reference is interesting, and seems to disprove my assertion that the pointer is only protected to the last reference. At around the place where the closing curly brace for the fixed statement is, we have:

00d30257 33d2            xor     edx,edx
00d30259 8955f4          mov     dword ptr [ebp-0Ch],edx
00d3025c 90              nop

Wow. I am surprised and shocked! It looks like indeed the pointer is held live until the end of the scope. This raises more questions...

Managed C++ IS good :-)

After my ill-advised rant about managed C++ I learned something interesting. Managed C++ is a great bridge between a C# model and unmanaged C++ code, especially if your method signatures are kind of complex. Managed C++ does a better (quicker) job of marshaling between it's module and the unmanaged C++ module than PINVOKE does.

Also, you don't have to learn those fancy PInvoke attributes and duplicate structures in C# and C++.

Therefore, managed C++ is the most elegant solution to bridge between C# and unmanaged C++.