Advanced .NET Debugging – Book Review

This review is for: Advanced .NET Debugging (Paperback) by Mario Hewardt.

If you program for the .NET framework you need this book. It helps to take your debugging skills (as well as understanding of the platform) to the next level. How many times in the past have we tried to nail down a particularly tricky bug, and after hours (or days) of pulling at your hair, given up in despair and doubted our worth as a good programmer ? Well, if you have trodden down that path, then I am sure this book will help. I wish to warn you that this is not a book for a beginner nor for the faint-hearted. It will show you the tools, it will show you the way, but you will have to walk that path which begins by buying this book and reading it.

If you already have Advanced Windows Debugging by the same author and have read it, then you should be in good shape to tackle this one. But if you haven’t, fear not, you can still make it.

The book is soft-bound, neatly printed in about 500 pages and contains 10 chapters divided into 3 parts. It doesn’t weight much and can be easily carried around.

Part 1 consists of 3 chapters. In the first chapter the reader is an introduction to the tools. The 2nd chapter – CLR fundamentals – contrary to its name, is not a high level overview for the newbie. Instead it is a wealth of information for all .NET programmers. No matter how senior a programmer you are, I will bet that you will still learn something (probably a lot) from this chapter. The 3rd chapter – Basic Debugging Tasks – is a bit lengthy (I don’t mean that in a bad way) at about 100 pages, helps you get acquainted with the tools and commands.

Part 2 consists of 4 chapters – Assembly Loader, Managed Heap and Garbage Collection, Synchronization and Interoperability. As you might guess from the names, it is pretty advanced. It is hard, but you will emerge with a much better and clearer understanding of the platform. The chapter on interoperability might not be useful for everybody, but for those who have felt the pain of COM interop or PInvoke this chapter pays for the price of the book many times over.

Part 3 consists of the advanced-advanced topics. There is a chapter named Postmortem debugging which includes debugging problems when you have no access to the live machine and you cannot reproduce the problem locally. It consists of taking a dump file and analyzing it off-site. Not an everyday topic for most programmers, but you will surely be thankful for this chapter if you ever come across it. It also explains how the Windows error reporting works. The second last chapter is called PowerTools which includes discussion of PowerDbg which allows you to control native debuggers using powershell (how cool is that !). There is also information on Visual Studio integration with SOS and on CLR Profiler. The last chapter, a small one at about 15 pages, is on .NET 4.0 (based on Beta 1 release though).

The writing style is very clear and precise. There are plenty of samples and some good diagrams to help your understand the concepts better (i loved them, a picture is worth a thousand words). In short, this is book worth purchasing and worth reading and worth reading again (which is what I am going to do).

There is a support website for the book here.

Ladies and Gentlemen, this is *the* F# book !

Programming F# Front Cover

It has been a long time since I got excited about a programming book as much as I did with Programming F#. I have all the books on F# available in the market today. But I should say that this is the best book for learning F# if you have no experience with functional programming. I am not sure how this book would be for a person who is totally new to programming (chances are they would find this ahead of their curve). But for a person who has good experience with imperative programming this book would get you up and running with F# in specific and functional programming in general.

The book by Don Syme Expert F# (Expert’s Voice in .Net) (who invented F#) is also good but not as good as Programming F# for new functional programmers. But once you have read this book, Don Syme’s book might be a good follow up.

The strongest point of this book is the clarity with which concepts are explained and the choice of good examples to explain a concept. They are concise and to the point. It also made the book surprisingly small (at a little less than 400 pages). But no worries, everything that you need to get a firm footing in F# is in here.

This is truly a 5 star book. Highly recommended.

The table of contents:
Part I. Multiparadigm Programming

Chapter 1. Introduction to F#
Chapter 2. Fundamentals
Chapter 3. Functional Programming
Chapter 4. Imperative Programming
Chapter 5. Object-Oriented Programming
Chapter 6. .NET Programming
Chapter 7. Applied Functional Programming
Chapter 8. Applied Object-Oriented Programming

Part II. Programming F#

Chapter 9. Scripting
Chapter 10. Computation Expressions
Chapter 11. Asynchronous and Parallel Programming
Chapter 12. Reflection
Chapter 13. Quotations
Appendix A. Overview of .NET Libraries
Appendix B. F# Interop

Functions in F# Part 2

I had written a post on Functions in F# a couple of days ago and Binil posted a comment which prompted me to post a follow up to add some information that I left out in the original post.

F# Syntax

F# has two syntax forms available – a lightweight syntax and a verbose syntax. The lightweight syntax uses indentation to mark the beginning and end of code blocks. This naturally results in more concise code. The verbose syntax uses additional keywords to demarcate code blocks making the code lengthier than it needs to be but it makes the code less sensitive to indentation (imagine copy-pasting and sending code in email etc).

The lightweight syntax is on by default. You can explicitly set it to on by using

#light

The verbose syntax is always on which means you can still use the verbose keywords even if the lightweight syntax is on.  We can enforce the verbose syntax by switching off the lightweight syntax as follows

#light “off”

The lightweight syntax is in more prominent use than the verbose syntax and it seems that going forward the verbose syntax might be obsolete.

Only spaces can be used for indentation in F#, TABs are not accepted. Visual Studio by default converts TABs to spaces. This can be set in Tools –> Options dialog under Text Editor –> F# –> Tabs.

image

 

Tail Recursion

I had mentioned about recursive functions in my previous post on Functions in F# but I did not mention anything about tail recursion. Given the importance of recursion in F#, this is a very important topic and I want to thank Binil for bringing it to my notice.

First, let me provide some background info on what is bad about recursive functions and why tail recursion is the preferred way to write recursive functions.

When is function needs to be executed, a block of memory is allocated on the system stack for the function and is called a stack frame. When the function completes execution, this memory is reclaimed and is called unwinding the stack.

Recursive functions call themselves repeatedly until an exit condition is met. This means that a new stack frame has to be created each time the recursive call is made. Given enough nested function calls there wouldn’t be any more space on the system stack and would cause an undesirable condition called stack overflow. The program execution is terminated forcefully by the runtime.

If the function calls are not recursive, for example calling a function 1000 times in a for loop, the memory block for a function can be reclaimed when the function execution is over. So what we need to try to achieve is to write the function in such a way that the stack frame of the calling function need not be maintained while execution proceeds to the next function call. The way to do this is to make the recursive call as the last statement in the function. When we do that the compiler produces different code which allows the stack frame of the calling function to be unwound and the memory reclaimed, in effect making it similar to calling the function in a loop. This technique is called tail recursion.

Let us look at some code to make this more clear. Here is the recursive implementation of factorial that I wrote in my previous blog post.

let rec factorial x=
    if x <= 1 then
        1
    else
        x * factorial (x-1)

Here it might look as if the recursive call is the last statement in the function, but it is not. It would become clear if we wrote it this way:

let rec factorial x=
    if x <= 1 then
        1
    else
        let product = x * factorial (x-1)
        product

Now it becomes clear that the last statement in the function is returning the value of product. We also see that the function has to maintain the value of x in its stack frame and multiply it with the value returned by the recursive call. So if the function is written as shown above, there is no way to reclaim the stack memory until the chain of recursive calls return. So let us re-write the function in using tail recursion.

let factorial x =
    let rec tailRecFactorial x prod =
        if x <= 1 then
            prod
        else
            tailRecFactorial (x-1) (prod * x)
    tailRecFactorial x 1

 

Here we have a nested function which takes the product up to the current stage as a parameter which means nothing needs to be stored on the stack of the calling function. The stack frame can be safely reclaimed. The compiler is able to spot the tail recursive calls and generates code as if the function was called in a loop. Execution returns the original calling point when the exit condition is met.

The method I showed above to achieve tail recursion is called accumulator pattern. Another way is to use Continuations.

Trivia

When I was writing the non tail recursive factorial function, by mistake I wrote it this way:

let rec factorial x =
    if x <= 1 then
        1
    else
    x * (factorial x-1)

 

Can you spot the problem ?

Hint: It causes a StackOverflowException when executed.

Plans for 2010 Part 2

As I have posted here, I have made some resolutions for the new year. If nothing, just for the fun. I have to add two more things to the list.

  • Learning F# and
  • Reducing my weight.

I have been trying to learn F# for over a year now (maybe even a little more). I have all the F# books in print now as well as the PDF for the upcoming Manning MEAP – Functional Programming for the Real World. But they been collecting dust on my shelf. Functional programming requires a new mindset and hence requires a little more effort for me than studying a new imperative language. The first book that I bought is Expert F# which is definitely the wrong book for a beginner. It is written by Don Syme et.al. who is the primary designer of the language and is a little dense. It is a good book with a lot of information but it is just not good for somebody new to functional programming. Once you have grasped the main ideas behind functional programming, it is a good book to have. The second edition of this book is coming up soon.

The game changer for me was Programming F# by Chris Smith which, IMHO, is the best book out there today to learn F#. It is a surprisingly small book and is written with such clarity that even somebody with absolutely no exposure to functional programming languages would be able to learn from it. It makes the ideas accessible and has some brilliant examples to showcase some of the features of F#. It kind of feels like an easy-going friend, which is exactly what you need when exploring unfamiliar grounds. I was so impressed with the book that I wrote a review of the book on Amazon giving it 5 stars.

I am not entirely comfortable with F# yet. But I have my feet wet and I plan to make the plunge in 2010.

Another thing that I want to tackle in 2010 is my weight (and fat). I am 190 lbs now. I used to be something like 175 lbs 3 years ago when I came to USA. 15 lbs in 3 years ! I am definitely worried about this. So is my wife. I had taken diets a couple of times previously and had always managed to reduce weight. But it always came back with a vengeance. So I am not sure whether another diet is the solution. Being on a diet is one of the most worst experiences that I have been through. Food is always on your mind and you can’t eat it. You are constantly fighting the sugar craving (I was on a low-carb diet). It is hell.

Although I haven’t made any specific plans, I think I would concentrate more on exercise and less on diet this time. Let us see how it goes.

Happy Holidays !

Functions in F#

Wikipedia says that

…functional programming is a programming paradigm that treats computation as the evaluation of mathematical functions and avoids state and mutable data. It emphasizes the application of functions…

The point I wish to make here is that functions are the  most important “things” in a functional programming language. You can say that functions are “first class citizens” in a functional programming language. This means that functions can be passed around as parameters and return values of other functions, just like any other data. Functions which can take other functions as parameters and/or return values are called Higher-Order Functions.

Functions and Type Inference

Here is an example of a simple function in F#.

let triple x = 3 * x

Here triple is the name of the function and x is the name of the parameter passed to the function. Contrary to C# naming convention for function, F# uses camel casing. Even though we have not specified any type information, the compiler is able to infer that this function takes an integer as parameter and returns an integer. This is called Type Inference and makes your F# program more concise.

How did I know that the compiler inferred the parameter and return value as int? I typed in the above functional declaration in F# interactive and terminated it with ;; and hit Enter. F# showed me:

val triple : int –> int

This means that triple is a function which takes an integer as parameter and returns an integer. If we try to call the function with a float as parameter

triple 3.0

it will spit out an error message as follows.

stdin(4,8): error FS0001: This expression was expected to have type
    int   
but here has type
    float   

We know that * (multiplication operator) works with floats as well, so why this error ? The reason for this is that in the absence of any additional type information the compiler simply defaulted the type to integer. So triple is a function which can take an integer as parameter and return an integer which is equal to three times its input. It is equivalent to the following C# function.

static int Triple(int x)
{
      return 3 * x;
}

Since the function expects integer, a parameter of type float will produce an error.

So what do we do when we need a datatype other than the default type that the compiler is able to infer? We give a hint to the compiler via type annotation.

let triple (x: float) = 3.0f * x

Note: In the above example type annotation is not strictly required because the compiler would infer x as float because 3.0f is float. But it does show you how to annotate a function parameter with its type.

 

Higher Order Functions

The functions mentioned above are not a higher-order functions because they do not take a function as parameter, neither do they return one. List.map and List.filter are examples of higher order functions. List.map takes a function and applies it to each item in the list (also passed as parameter to it) and returns a new list with the output values. List.filter takes a function and list as parameters and returns a new list. The output list contains those values from the input list for which the input function returned true. An example is shown below which makes use of both these higher order functions.

let triple x =
    3 * x

let isEven x =
    x % 2 = 0

let actOnEven action numbers=
    List.map action (List.filter isEven numbers)

[<EntryPoint>]
let main(args:string[]) =
    let results = (actOnEven triple [1 .. 10])
    Console.WriteLine (results)
    0

Here actOnEven is itself a higher order function.

Recursive Functions

Recursive functions are functions that call themselves. An important tenet of functional programming is the use of recursion instead of looping. We can create recursive functions using the following syntax.

let rec factorial num =
    if( num <= 1) then
        1
    else
        num * factorial (num - 1)

The rec keyword is a helping hand to the compiler for type inference. Personally, I don’t like it but I think they couldn’t find a way around it.

Function Currying

Currying (named after Haskell Brooks Curry) is an interesting technique in which you pass one of the arguments to a function and it returns another function which takes the rest of the arguments. For eg., consider the function

let add x y = x + y

If you run this in F# interactive, you will see the following message:

val add : int -> int –> int

This means that add is a function which takes an integer and returns a function which takes another integer as parameter and returns an integer. So if we call add with just one parameter, it will return another function which takes the second parameter.

let addToTwo = (add 2)

To be clear, addToTwo is a function that takes one integer parameter and returns an integer.

For eg,

addToTwo 8

will return 10.

Here addToTwo is the curried function and this technique is called function currying. Although this may look like a strange and useless language feature, it has interesting and powerful applications in functional programming (like the pipe forward operator). I will write more about this in future blog posts.

Nested Functions

In F#, a function can contain another function. Here is a simple example.

let printList numbers=
    let print num=
        printfn "%i" num
    List.map print numbers

Here print is a function nested in printList.

Some of the feature that I have mentioned here would be new to C# programmers and might feel a little strange. But have hope, for all will become more clear with time. We need to see some solid examples to appreciate the power of some of them. I have mostly presented useless examples but I hope a reader would have gleaned atleast a little information about functions and its usage in F# from this post. There is more to write but it is getting late (3 am) and tomorrow is a working day, so I gotta go. Will be back with more on F#.

References:

Programming F#

Expert F#

Duck Typing using C# 4.0

using System;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            QuackTheQuacker(new Toy());
            QuackTheQuacker(new Duck());
        }

        static void QuackTheQuacker(dynamic quacker)
        {
            quacker.Quack();
        }
    }

    class Toy
    {
        public void Quack()
        {
            Console.WriteLine("A toy that quacks");
        }
    }

    class Duck
    {
        public void Quack()
        {
            Console.WriteLine("A duck that quacks");
        }
    }
}

Why we should be testing the release build and not the debug build

If we are deploying the release build of an application, you should be testing the release build of the application. While it doesn’t sound like a big deal, subtle errors arising from the compiler optimizations can bite you in production. Here is an example of it.

As I mentioned in a previous post, one of solutions for the access denied issue was to pre-load the assemblies in the Global.asax Application_Start event handler. For preloading a specific assembly (lets call it foo.dll) I did something like this:

MyNamespace.MyClass.Equals(null, null);

Things worked fine in the debug build, but in the release build we were getting Access denied for the foo.dll assembly. It was a little weird because all the other assemblies were getting loaded as network service and they all succeeded. Foo.dll was getting loaded under the identity of the user and was failing. I decompiled the foo.dll using Reflector (can’t say how many times this nifty little tool saved the day) and found that the code was optimized by the compiler to this:

object.Equals(null, null);

It does make sense. In hindsight.

We tested the solution in debug build where there was no optimization and everything worked as expected.

So my advice is – if you are deploying the release build of an application, you should be testing the release build of the application.

P.S. – There was an interesting discussion in the MSDN forums a couple of years ago which displayed a similar problem. I will blog about it later, for a little while it got me real frightened about my understanding of the .net platform.

Getting Access Denied Errors in your WCF service or ASP.NET application ?

We are using impersonation in our WCF Service so that credentials of the user can flow to the backend layer (where authorization is done). Strangely we were getting Access Denied errors when the client tries to connect to the service. The error message was like:

Could not load file or assembly ‘MyAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null’ or one of its dependencies. Access is denied.

This was very frustrating to say the least. The first thing to do was search the internet with the error message and sure enough, several other people had faced the same issue but nobody had a good solution for the problem. The problem was that the code was running under the identity of the user (because we were using impersonation) and when the assemblies were shadow copied to the “Temporary ASP.NET Files” directory (C:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files), the user did not have write permissions to that directory and hence the Access denied error.

Note: I should say that the Fusion Log Viewer was extremely helpful in identifying the problem and helping to make sure that the problem was resolved.

The obvious solution is to give users write permission to that folder. The testing team decided that this could a potential security issue (even though this was an internal application and permissions were given only to authenticated users). So we began hunting for other solutions and came up with more solutions.

  • Run aspnet_regiis –ga “authenticated users”.

This seemed to fix the problem, but it turned out that it was just a variation of granting authenticated users permission to the Temporary ASP.NET Files folder. What this command does is add the user/user-group to the IIS_USRS groups which is already set to have modify permissions on the Temporary ASP.NET Files folder. It works, but we are back to square one.

  • Disable Shadow copying of assemblies

As I mentioned earlier, the problem was occurring while shadow copying the assemblies to the temp location. The primary purpose of shadow copying is to do dynamic compilation and hot-swapping files. Dynamic compilation does not apply to WCF as much as for ASP.NET. Also we were not very concerned about the hot-swapping of files, so this was a credible option for us. But what concerned us was that we couldn’t find any info about any other side effects of disabling shadow copying. To disable shadow copy, add the following in your web.config inside the configuration section.

<system.web>
    <hostingEnvironment shadowCopyBinAssemblies="false"/>
</system.web>

The good thing about this option is that it does not require a code change and is easy to revert and also there doesn’t appear to be any security issues.

  • Move impersonation to the end of the call stack

Since the problem occurs because the assembly loading happens in the identity of the user, we could do away with declarative impersonation and do an imperative style impersonation where impersonation is done only when making calls to the backend system. To make it more clear, we were doing impersonation like:

[OperationBehavior(Impersonation = ImpersonationOption.Required)]
public Response DoSomething(Request request)
{
        return request.Execute();
}
Instead we could do impersonation only while calling the backend code:
WindowsIdentity callerWindowsIdentity = ServiceSecurityContext.Current.WindowsIdentity;
if (callerWindowsIdentity == null)
{
    throw new InvalidOperationException("The caller cannot be mapped to a WindowsIdentity");
}

using (callerWindowsIdentity.Impersonate())
{
    // Make backend calls
} 

This is definitely a good solution, but has atleast two drawbacks. The first problem (not a big one) is that it requires a code change. The second problem is a performance one and happens only if the backend calls are webservice calls. The reason for this is that the webservice calls usually require a XmlSerializers dll which is generated by Visual Studio and usually deployed along with the application. If this assembly is not present, the runtime will create one dynamically. This can cause a perf hit. Since we do impersonation before making the backend call, the XmlSerializer assembly is loaded in the identity of the user and will fail. But the CLR, failing to bind the assembly, will synthesize one for you thus causing a perf hit.

If your backend calls are not webservice calls, then this is a reasonable workaround for you.

  • Pre-load the assemblies

What we understood from the Fusion Logs was that the assembly loading was happening under the identity of the end user who does not have adequate permissions. So one thing we can do is to pre-load the assemblies when the service is running as the network service account. To do this, we created a Global.asax file and in the Application_Start event handler, we refer to types in all the assemblies of the service, thus forcing them to be loaded even before the first request is processed. The code looks something like this:

namespace MyService
{
    public class Global : System.Web.HttpApplication
    {

        protected void Application_Start(object sender, EventArgs e)
        {
            //Make sure all the dlls are loaded
            Action action = new Action(null);
            EntityOperations.EntityOperationsFactory.CreateOperation();
            DataAccess.DaoFactory.CreateDao(); 
            WebServiceLibrary.WebServiceGateway.Initialize(null);
        }
     }
}

Using the Fusion Logs we were able to see that the assemblies were getting loaded under the identity of the network service account. We could also see that the XmlSerializers were also getting loaded correctly.

This solution might feel a little icky because of its unconventional nature but I think it is a brilliant solution and it seems to work correctly. We have decided to adopt this solution for our project.

  • Change the shadow copy directory to a directory inside the website directory

Frankly I never tried this one out and don’t know if it works, but it is worth some investigation if none of the solutions outlined above works for you.

I will make a follow up post or edit this one, if I find problems with the solutions mentioned here or if I find better solutions.

My first F# program

open System

let display message =
    printfn "%s" message 

[<EntryPoint>]
let main(args) =
    display "Hello, F#"
    0