The memory model is a fascinating topic – it touches on hardware, concurrency, compiler optimizations, and even math.

The memory model defines what state a thread may see when it reads a memory location modified by other threads. For example, if one thread updates a regular non-volatile field, it is possible that another thread reading the field will never observe the new value. This program never terminates (in a release build):

class Test
{
    private bool _loop = true;

    public static void Main()
    {
        Test test1 = new Test();

        // Set _loop to false on another thread
        new Thread(() => { test1._loop = false;}).Start();

        // Poll the _loop field until it is set to false
        while (test1._loop == true) ;

        // The loop above will never terminate!
    }
}

There are two possible ways to get the while loop to terminate:

1. Use a lock to protect all accesses (reads and writes) to the _loop field
2. Mark the _loop field as volatile

There are two reasons why a read of a non-volatile field may observe a stale value: compiler optimizations and processor optimizations.

Compiler optimizations

The first reason why a non-volatile read may return a stale value has to do with compiler optimizations. In the infinite loop example, the JIT compiler optimizes the while loop from this:

while (test1._loop == true) ;

To this:

if (test1._loop) { while (true); }

This is an entirely reasonable transformation if only one thread accesses the _loop field. But, if another thread changes the value of the field, this optimization can prevent the reading thread from noticing the updated value.

If you mark the _loop field as volatile, the compiler will not hoist the read out of the loop. The compiler will know that other threads may be modifying the field, and so it will be careful to avoid optimizations that would result in a read of a stale value.

00000068  test        eax,eax
0000006a  jne         00000068

00000064  cmp         byte ptr [eax+4],0
00000068  jne         00000064

Processor optimizations

On some processors, not only must the compiler avoid certain optimizations on volatile reads and writes, it also has to use special instructions. On a multi-core machine, different cores have different caches. The processors may not bother to keep those caches coherent by default, and special instructions may be needed to flush and refresh the caches.

The mainstream x86 and x64 processors implement a strong memory model where memory access is effectively volatile. So, a volatile field forces the compiler to avoid some high-level optimizations like hoisting a read out of a loop, but otherwise results in the same assembly code as a non-volatile read.

The Itanium processor implements a weaker memory model. To target Itanium, the JIT compiler has to use special instructions for volatile memory accesses: LD.ACQ and ST.REL, instead of LD and ST. Instruction LD.ACQ effectively says, “refresh my cache and then read a value” and ST.REL says, “write a value to my cache and then flush the cache to main memory”. LD and ST on the other hand may just access the processor’s cache, which is not visible to other processors.

Volatile accesses in more depth

To understand how volatile and non-volatile memory accesses work, you can imagine each thread as having its own cache. Consider a simple example with a non-volatile memory location (i.e. a field) u, and a volatile memory location v.

A non-volatile write could just update the value in the thread’s cache, and not the value in main memory:

However, in C# all writes are volatile (unlike say in Java), regardless of whether you write to a volatile or a non-volatile field. So, the above situation actually never happens in C#.

A volatile write updates the thread’s cache, and then flushes the entire cache to main memory. If we were to now set the volatile field v to 11, both values u and v would get flushed to main memory:

Since all C# writes are volatile, you can think of all writes as going straight to main memory.

A regular, non-volatile read can read the value from the thread’s cache, rather than from main memory. Despite the fact that thread 1 set u to 11, when thread 2 reads u, it will still see value 10:

When you read a non-volatile field in C#, a non-volatile read occurs, and you may see a stale value from the thread’s cache. Or, you may see the updated value. Whether you see the old or the new value depends on your compiler and your processor.

Finally, let’s take a look at an example of a volatile read. Thread 2 will read the volatile field v:

Before the volatile read, thread 2 refreshes its entire cache, and then reads the updated value of v: 11. So, it will observe the value  that is really in main memory, and also refresh its cache as a bonus.

Note that the thread caches that I described are imaginary – there really is no such thing as a thread cache. Threads only appear to have these caches as an artifact of compiler and processor optimizations.

volatile string[] _args = null;

public void Write() {
    string[] a = new string[2];
    a[0] = "arg1";
    a[1] = "arg2";
    _args = a;
    ...
}

public void Read() {
    if (_args != null) {
        // Under weak volatile semantics, this assert could fail!
        Debug.Assert(_args[0] != null);
    }
}

Memory model and .NET operations

Here is a table of how various .NET operations interact with the imaginary thread cache:

For each operation, the table shows two things:

• Is the entire imaginary thread cache refreshed from main memory before the operation?
• Is the entire imaginary thread cache flushed to main memory after the operation?

Disclaimer and limitations of the model

This blog post reflects my personal understanding of the .NET memory model, and is based purely on publicly available information.

I find the explanation based on imaginary thread caches more intuitive than the more commonly used explanation based on operation reordering. The thread cache model is also accurate for most intents and purposes.

To be even more accurate, you should assume that the thread caches can form an arbitrary large hierarchy, and so you cannot assume that a read is served only from two possible places – main memory or the thread’s cache. I think that you would have to construct a somewhat of a clever case in order for the cache hierarchy to make a difference, though. If anyone is aware of a case where the hierarchical thread cache model makes a prediction different from the reordering-based model, I would love to hear about it.

If you are interested in the .NET memory model, I encourage you to read Understand the Impact of Low-Lock Techniques in Multithreaded Apps in the MSDN Magazine, and the Memory model blog post from Chris Brumme.

For decades, computer science students have been taught that so-called NP-hard problems do not have known efficient solutions. These problems include the infamous Travelling salesman problem, subset sum, 3SAT, and many more.

But – as is often the case – where theoretical Computer Science failed, sound software engineering practices will succeed. By using loosely-coupled OOP, agile methodologies and the model-view-controller architectural pattern, I developed a solution that someone trapped in the world of formulas and big Ohs would never dream of.

Enough with the background, and let’s take a deep dive into the intriguing design.

Introducing the choose expression

As most other elegant designs, this one is very simple. My proposal calls for a choose expression with this syntax:

    choose { boolean_expression1, boolean_expression2 }

Choose expression is basically the || operator, only with a slight twist. The semantics of the choose expression are similarly simple:

1. If boolean_expression1 or boolean_expression2 will evaluate to true, the runtime will evaluate the true expression, but not the other expression. The return value of the choose expression will be true in this case.
2. If both expressions will evaluate to false, the runtime will evaluate neither expression. The return value of the choose expression is false in this case.

Let’s look at a few simple usage examples:

    bool a = choose {
        1 == 2,
        1 < 2
    };

Variable a will be set to true, because the condition 1 < 2 is true.

Here is another example:

    bool a = choose {
        ((Func<bool>)(() => { Console.WriteLine("Hello"); return false; }))(),
        1 < 2
    };

There is no point executing the first function, because it would return false anyways. So, this code sample does not print anything to screen. Instead, the choose expression will execute the second function. The second expression returns true, so variable a will be set to true.

And another simple one:

    bool a = choose {
        ((Func<bool>)(() => { Console.WriteLine("Hello1"); return false; })(),
        ((Func<bool>)(() => { Console.WriteLine("Hello2"); return false; })(),
    };

This code sample will not be print anything to screen either. It is obvious; why evaluate either of the two functions if they are going to return false anyways? This code simply assigns false to variable a.

Now, let’s cut to the chase, and use choose expressions to give an efficient implementation of an NP-hard problem. Let’s look at subset sum:

    bool SubsetSum(int[] arr)
    {
        return SubsetSumHelper(arr, 0, 0);
    }

    bool SubsetSumHelper(int[] arr, int index, int sumSoFar)
    {
        if (index == arr.Length)
        {
            return sumSoFar == 0;
        }

        return choose {
            () => SubsetSumHelper(arr, index + 1, sumSoFar + arr[index]),
            () => SubsetSumHelper(arr, index + 1, sumSoFar)
        };
    }

Yes, that’s right! An O(N) implementation of the subset sum problem. There you have it, computer scientists. You said it was impossible. If anyone at the University of British Columbia needs my mailing address to send me a refund check for my education, you can find my contact information in the margin.

Under the hood of the choose expression

After a couple hours of coding, I was able to develop a simple prototype. It works perfectly, but since it is only a prototype, I simplified my life a little bit by allowing choose to execute both functions. After all, I don’t have to slave through all the nitty-gritty details in the initial prototype, right? The performance of my implementation is not that great either, but I haven’t had the time to fire up the profiler so far. Perhaps I need to unroll a loop somewhere, or ensure that method calls are getting inlined optimally.

To further prove the feasibility of my design, I developed a non-deterministic Turing machine construction that evaluates choose expressions extremely efficiently.

Non-deterministic Turing machines are known to be a good realistic abstraction of computing hardware; there was a study that proved that. To be exact, the study was only a moderate success. The researchers built a mechanical non-deterministic Turing machine that solved a Travelling Salesperson problem with 5 cities without a hitch. On the 6-city version of the problem, the experiment had to be abruptly interrupted after sprawling machine replicas filled up the room, and the head of one of the researchers got caught in a loop of tape.

So, it is clear that this design is sound. There may be performance issues in the first release, but they will improve as the technology matures. And once CPU manufacturers include a non-deterministic branching instruction in the instruction set, the cost of evaluating the choose expression will drop down to a couple of instructions.

Summary

I don’t know the detailed plans surrounding the C# language, but if there is a C# 4.1., I would like to see the choose expression included.

And the larger lesson of this post is simple: computer science is largely obsolete in today’s world of technology. Computer science says that this is impossible, that is impossible… As you just saw, anything is possible, so long as you have enough paper to print out all the UML diagrams.

   using System.Collections.Generic;
   class Program
   {
       public static void Main()
       {
           int[] arr = new int[10];
           IEnumerator<int> e = arr.GetEnumerator();
       }
   }

If you don’t see it, don’t worry.  I was surprised by this C# behavior as well. Just come back in a couple days to see the solution. Or try to compile the program in Visual Studio.

UPDATE: Finally, I am back with a solution.

To understand what’s going on, let’s take a look at a fixed version of the code:

using System.Collections.Generic;
class Program
{
    public static void Main()
    {
        int[] arr = new int[10];
        IEnumerator<int> e = ((IEnumerable<int>)arr).GetEnumerator();
    }
}

The problem of the original code sample is that the GetEnumerator() method on arrays returns a non-generic IEnumerator. That’s because .NET arrays had GetEnumerator() even before there were generics in .NET.

When generics got introduced in .NET 2.0, arrays got another GetEnumerator() method, one that returns a generic IEnumerator<>. In order to have two GetEnumerator() methods with different return types, one of them had to be a part of an explicit interface implementation.

As a result, arr.GetEnumerator() binds to a method that returns a non-generic IEnumerator, and IEnumerable<int>)arr).GetEnumerator() binds to a method that returns a generic IEnumerator<int>. Somewhat surprising, but understandable.

Disclaimer: this explanation is based on public information and my guesses, and should not be considered official, authoritative, or anything like that.

Of course, SelectMany is not only incredibly useful, but also surprisingly powerful. In fact, a variety of LINQ operators are really just constrained versions SelectMany. Select, Concat, Where, Take, Skip, TakeWhile, SkipWhile and Distinct can all be easily rewritten using a single SelectMany.

To get started, let’s compare SelectMany with Select. SelectMany projects each element into some number of elements. In comparison, Select projects each element into exactly one element. Since SelectMany is more general than Select, it can be used to implement Select:

public static IEnumerable<U> Select<T, U>(
    this IEnumerable<T> source,
    Func<T, U> func)
{
    return source.SelectMany(x => Enumerable.Repeat(func(x), 1));
}

But, Select is not the only operator less general SelectMany. Similarly, you can think of the Where operator as producing zero or one element for each element in the sequence. So, Where can also be easily implemented using SelectMany:

public static IEnumerable<T> Where<T>(
    this IEnumerable<T> source,
    Func<T, bool> filter)
{
    return source.SelectMany(x => Enumerable.Repeat(x, filter(x) ? 1 : 0));
}

Another operator that can be easily implemented with SelectMany is Concat:

public static IEnumerable<T> Concat<T>(
    this IEnumerable<T> source1,
    IEnumerable<T> source2)
{
    return new int[] { 0, 1 }
        .SelectMany(
            x => x == 0 ? source1 : source2);
}

Take and Skip can be implemented using the SelectMany variant that passes indices into the user delegate:

public static IEnumerable<T> Take<T>(
    this IEnumerable<T> source,
    int toTake)
{
    return source.SelectMany((x, i) => Enumerable.Repeat(x, i < toTake ? 1 : 0));
}

public static IEnumerable<T> Skip<T>(
    this IEnumerable<T> source,
    int toSkip)
{
    return source.SelectMany((x, i) => Enumerable.Repeat(x, i >= toSkip ? 1 : 0));
}

And finally, with a few closure tricks, we can even implement TakeWhile, SkipWhile and Distinct:

public static IEnumerable<T> TakeWhile<T>(
    this IEnumerable<T> source,
    Func<T, bool> func)
{
    bool stopped = false;
    return source.SelectMany((x, i) => Enumerable.Repeat(x, (!stopped && (stopped = !func(x))) ? 1 : 0));
}

public static IEnumerable<T> SkipWhile<T>(
    this IEnumerable<T> source,
    Func<T, bool> func)
{
    bool started = false;
    return source.SelectMany((x, i) => Enumerable.Repeat(x, (started || (started = func(x))) ? 1 : 0));
}

public static IEnumerable<T> Distinct<T>(
    this IEnumerable<T> source)
{
    var dict = new Dictionary<T, int>();
    return source.SelectMany((x) => {
        if (!dict.ContainsKey(x))
        {
            dict.Add(x, 0);
            return new T[] { x };
        }
        return new T[] { };
    });
}

If you enjoyed this article, check out these LINQ puzzles. Also, read my article on the 7 tricks to simplify your programs with LINQ.

Why does the last line throw StackOverflowException?

IEnumerable<int> q = new int[] { 1, 2 };
q = from x in new int[] { 1, 2 }
    from y in q
    select x + y;
q.ToArray();

And, how come the code sample runs just fine if you switch the order of the from clauses?

IEnumerable<int> empty = Enumerable.Empty<int>();
for (int i = 0; i < 40; i++)
{
    empty = empty.Concat(empty);
}
int[] emptyArray = empty.ToArray();

Answer in the comments section.

For a slightly harder challenge, check out the next puzzle.

I realized that there is a very clean way to express a multi-clause if statement by composing ternary conditional operators like this:

var result =
    condition1 ? result1
    : condition2 ? result2
    : condition3 ? result4
          ...
    : conditionN ? resultN
    : default;

Traditionally, this would be written in a much more verbose way:

MyType result;
if (condition1) result = result1;
else if (condition2) result = result2;
else if (condition3) result = result3;
   ...
else if (conditionN) result = resultN;
else result = default;

Here is a simple real-world application of this trick:

string commentCount =
    n == 0 ? "no comments"
    : n == 1 ? "1 comment"
    : n < 100 ? n + " comments"
    : "100+ comments";

I really like this pattern because the code is very concise and clean. I am surprised that I have never seen it used anywhere.

string commentCount =
    string.Format("{0} comment%s",
        (n == 0 ? "no"
        : n < 100 ? n.ToString()
        : "100+"),
        (n == 1 ? "" : "s"));

Gotchas

Suprisingly, I don’t believe there are any major ones. The conditional operator has a very low operator precedence in C#, Java and C++. In C# and Java, only the assignment operators (=, +=, <<=, etc) have a lower precedence than the conditional. In C++, you also have to be cautious around the comma operator, but you should be using that construct rarely anyways.

If you really want to mix the switch expression with assignment operators, other conditionals, or even the C++ comma operator, use brackets to ensure that the conditional operators which are part of the switch expression will be applied last.

In all other cases, the pattern should behave as you’d expect.

Comments and Conclusion

It is great to find a neat trick in the good old C-based languages. Not only functional languages are cool.

Any thoughts? Has anyone seen this pattern before? Let me know in the comments.

switch (true)
{
    case n == 0:
        // do something
        break;
    case n > 2:
        // do something else
        break;
    default:
        return;
}

My goal was to include operators that are simple to use, but applicable to a broad range of problems. I  left out operators that I thought were either too complicated to use, or too specific to a particular problem domain.

You can download the full source code of the library here (rename the file to ExtendedEnumerable.cs). Read on to find out what it contains.

ReadLinesFrom, WriteLinesTo – I/O in LINQ queries

LINQ is a great programming model for simple file-processing tasks. Treating a file as an enumerable of lines, we can filter, transform and analyze it using various LINQ operators. To support this use case, my library includes several operators to convert between streams and line enumerables. Two most general overloads are ReadLinesFrom and WriteLinesTo, which have the following signatures:

public static IEnumerable<string> ReadLinesFrom(TextReader reader)

public static void WriteLinesTo(    this IEnumerable<string> lines, TextWriter writer)

However, in most cases you will want to use one of the more specific overloads, ReadLinesFromConsole, ReadLinesFromFile, WriteLinesToConsole and WriteLinesToFile. For example, the Grep method below reads a file, keeps only lines that contain a particular substring, and writes out the results into another file:

static void Grep(string inputFile, string outputFile, string substring)
{
    ExtendedEnumerable.ReadLinesFromFile(inputFile)
        .Where(line => line.Contains(substring))
        .WriteLinesToFile(outputFile);
}

Isn’t that neat?

Generate – generate a sequence from a user delegate

In C# 2, generating arbitrary sequences became much more convenient than it used to be in C# 1. Instead of implementing two classes, the IEnumerable<T> and the IEnumerator<T>, you can implement a single method that yields items using the iterator block syntax (i.e. the yield statements).

However, I still try to avoid creating a method just to generate a simple sequence, particularly if I use that sequence only in one place in my program. The Generate operator below accepts a delegate which generates the sequence element by element. To signal the end of the sequence, the generator returns null.

Since value types cannot be null, we need one overload for reference types, and another overload that uses a nullable wrapper to handle value types:

public static IEnumerable<T> Generate<T>(Func<T> generator)    where T : class

public static IEnumerable<T> Generate<T>(Func<Nullable<T>> generator)    where T : struct

To give a usage example, the ReadLinesFromConsole operator I mentioned above could be implemented as follows:

public static IEnumerable<string> ReadLinesFromConsole()
{
    return ExtendedEnumerable.Generate(() => Console.ReadLine());
}

As another example, this code sample generates an infinite sequence of random integers:

Random rand = new Random();var randomSeq = ExtendedEnumerable.Generate(() => (int?)rand.Next());

This Generate operator has two disadvantages. First, it cannot be used to generate sequences that contain null values, because null is the terminator of the sequence. Second, it is a bit annoying to have to use the cast in the value-type overload (see the cast to int? in the random-sequence example). These are minor disadvantages, though, and I much prefer using the Generate operator over implementing a new method each time I need to generate a simple sequence.

As a side note, apparently Jon Skeet also looked at the problem of generating a sequence from a user’s delegate, and came up with a similar but slightly different solution, which you can find here.

ForEach – execute an action for each element in the sequence

As has been suggested by Magnus Martensson in a comment to my previous posting, as well as by others elsewhere, it is often neat to be able to specify an action at the end of the query using a ForEach operator, rather than having to iterate over the query in a foreach statement.

So, instead of this:

foreach (int x in Enumerable.Range(0,10).Where(i => (i % 2 == 0)).Take(5))
{
    Console.WriteLine(x);
}

You can write this:

Enumerable.Range(0,10).Where(i => (i % 2 == 0)).Take(5)
.ForEach(i => Console.WriteLine(i));

Do – execute side effects in the middle of the query

Sometimes it is useful to add side-effects in the middle of query, rather than to the end. For example, we can log which elements have been processed at a particular stage of the query. The Do operator provides this functionality:

Enumerable.Range(0,10)
    .Do((e) => Console.WriteLine("Processing {0}", e))
    .Select(x => x*2).ToArray();

Combine – combine two sequences

The Combine operator exists in various functional languages including F#, sometimes under the name Zip or ZipWith. It accepts two sequences as inputs, and combines their elements into a single sequence. So, the first element in sequence 1 and the first element in sequence 2 will be combined to produce the first element in the output sequence, and so forth. The function which combines an element from one sequence with an element from the other sequence is provided by the user. If one of the sequences is longer, the remaining elements in the longer sequence will be ignored.

To compute the pairwise sum between elements in seq1 and seq2, use the Combine operator like this:

IEnumerable<int> sumSeq = seq1.Combine(seq2, (a, b) => a + b);

As another example, to check whether a sequence of integers seq is increasing, use this query:

bool isIncreasing = seq.Combine(seq.Skip(1), (a, b) => a < b).All(x => x);

ToStringPretty – convert a sequence to a delimited string

Converting a sequence to a nicely-formatted string is a bit of a pain. The String.Join method definitely helps, but unfortunately it accepts an array of strings, so it does not compose with LINQ very nicely.

My library includes several overloads of the ToStringPretty operator that hides the uninteresting code. Here is an example of use:

Console.WriteLine(Enumerable.Range(0, 10).ToStringPretty("From 0 to 9: [", ",", "]"));

The output of this program is:

From 0 to 9: [0,1,2,3,4,5,6,7,8,9]

FromEnumerator – convert an enumerator to an enumerable

Several times I got into a situation where I have an enumerator, but really need an enumerable instead. There does not seem to be a simple way to do the conversion in .Net. Hence, my library of operators includes FromEnumerator which accepts an enumerator and returns an enumerable.

This sample converts enumerator e1 into an enumerable and then iterates over it in a foreach statement:

foreach (int x inExtendedEnumerable.FromEnumerator(e1)) { ... }

And this sample converts enumerator e2 into an enumerable to use it as a data source in a LINQ query:

var query = from x in ExtendedEnumerable.FromEnumerator(e2)
            where x % 2 == 0
            select x;

Single – convert an item to an enumerable

As I mentioned in my previous posting, I have found converting a single item to an enumerable to be a fairly frequent operation. So, my library includes an operator for the conversion:

IEnumerable<int> e = ExtendedEnumerable.Single(5);

Shuffle – randomly shuffle a sequence

I find myself regularly re-implementing the Shuffle operator when I am testing my code. Shuffle operator accepts a sequence and returns the same sequence, randomly rearranged.

This example prints digits 0..9 in a random order:

Enumerable.Range(0, 10).Shuffle().WriteLinesToConsole();

Comments and Conclusion

Again, the source code is available for download here. If there operators that I haven’t included, but you think they are useful, let me know in the comments!

Related:

• 7 tricks to simplify your programs with LINQ [igoro.com]
• Never write a for loop again [mikehadlow.blogspot.com]
• The missing operator – ForEach [bartdesmet.net]
• LINQ in Action [Book by Fabrice Marguerie]
• C# in Depth: What you need to master C# 2 and 3 [Book by Jon Skeet]

This posting summarizes some of the tricks that I came across. I will show you how to use LINQ to:

• Initialize an array
• Iterate over multiple arrays in a single loop
• Generate a random sequence
• Generate a string
• Convert sequences or collections
• Convert a value to a sequence of length 1
• Iterate over all subsets of a sequence

If you have your own bag of LINQ tricks, please share them in the comments! Also, if you like this article, you may like my next article, Extended LINQ: additional operators for LINQ to objects.

1. Initialize an array 

Often, you need to initialize elements of an array to either the same value, or to an increasing sequence values, or possibly to a sequence increasing or decreasing by a step different from one. With LINQ, you can do all of this within the array initializer – no for loops necessary!

In the following code sample, the first line initializes a to an array of length 10 with all elements set to -1, the second line initializes b to (0,1,..9), and the third line initializes c to (100,110,…,190):

int[] a = Enumerable.Repeat(-1, 10).ToArray();
int[] b = Enumerable.Range(0, 10).ToArray();
int[] c = Enumerable.Range(0, 10).Select(i => 100 + 10 * i).ToArray();

A word of caution: if you are initializing large arrays, you may want to forego the elegance and use the old-fashioned for loop instead. The LINQ solution will grow the array dynamically, so garbage arrays will need to be collected by the runtime. That said, I use this trick all the time when initializing small arrays, or in testing/debugging code.

2. Iterate over multiple arrays in a single loop 

A friend asked me a C# question: is there a way to iterate over multiple collections with the same loop? His code looked something like this:

foreach (var x in array1) {
    DoSomething(x);
}

foreach (var x in array2) {
    DoSomething(x);
}

In his case, the loop body was larger, and he did not like the duplicated code. But, he also did not want to allocate a new array to hold elements from both array1 and array2.

LINQ provides an elegant solution to this problem: the Concat operator. You can rewrite the above two loops with a single loop as follows:

foreach (var x in array1.Concat(array2)) {
    DoSomething(x);
}

Note that since LINQ operates at the enumerator level, it will not allocate a new array to hold elements of array1 and array2. So, on top of being rather elegant, this solution is also space-efficient.

3. Generate a random sequence 

This is a simple trick to generate a random sequence of length N:

Random rand = new Random();
var randomSeq = Enumerable.Repeat(0, N).Select(i => rand.Next());

Thanks to the lazy nature of LINQ, the sequence is not pre-computed and stored in an array, but instead random numbers are generated on-demand, as you iterate over randomSeq.

4. Generate a string 

LINQ is also a nice tool to generate various kinds of strings. I found this quite useful to generate strings for testing and debugging purposes.

Let’s say that you want to generate a string with the repeating pattern "ABCABCABC…" of length N. Using LINQ, the solution is quite elegant:

string str = new string(
    Enumerable.Range(0, N)
    .Select(i => (char)('A' + i % 3))
    .ToArray());

[EDIT] Petar Petrov suggested another interesting way to generate strings with LINQ. His approach applies to different scenarios than my solution above:

string values = string.Join(string.Empty, Enumerable.Repeat(pattern, N).ToArray());

5. Convert sequences or collections 

One thing you cannot do in C# or VB is to cast a sequence of type T to a sequence of type U, even if T us a derived class from U. So, you cannot just simply cast List<string> to List<object>. (For an explanation why, see Bick Byers’ posting).

But, if you are trying to convert IEnumerable<T> to IEnumerable<U>, LINQ has a simple and efficient solution for you:

IEnumerable<string> strEnumerable = ...;
IEnumerable<object> objEnumerable = strEnumerable.Cast<object>();

If you need to convert List<T> to List<U>, there is also a simple LINQ solution, but it involves copying the list:

List<string> strList = ...;
List<object> objList = new List<object>(strList.Cast<object>());

[EDIT] Chris Cavanagh suggested an alternate solution:

var objList = strList.Cast<object>().ToList();

6. Convert a value to a sequence of length 1 

When you need to convert a single value to a sequence of length 1, what do you do? You could construct an array of length 1, but I prefer the LINQ Repeat operator:

IEnumerable<int> seq = Enumerable.Repeat(myValue, 1);

7. Iterate over all subsets of a sequence 

Sometimes it is useful to iterate over all subsets of an array. This situation arises quite frequently in brute-force solutions to hard problems. For small inputs, subset sum, boolean satisfiability and the knapsack problem can all be solved easily by iterating over all subsets of some sequence.

In LINQ, we can generate all subsets of array arr as follows:

T[] arr = ...;
var subsets = from m in Enumerable.Range(0, 1 << arr.Length)
              select
                  from i in Enumerable.Range(0, arr.Length)
                  where (m & (1 << i)) != 0
                  select arr[i];

Note that if the number of subsets overflows an int, the above code will not work. So, only use it if you know that the length of arr is at most 30. If the length of arr is greater than 30, chances are that you don’t want to iterate over all of its subsets anyway because it is going to take minutes or more.

Comments and Conclusion

I hope you find these tricks useful and applicable to your programs.

The code samples in this posting are all implemented in C#, but they can be easily adapted to just about any other .Net language. However, LINQ is most conveniently used from .Net languages that support extension methods, lambda expressions and type inference, such as C# and Visual Basic.

To the best of my knowledge, each code sample in this posting works, but – as is common on the web – I don’t make any guarantees. As always, double check any code before using it.

Related:

• Extended LINQ: additional operators for LINQ to objects [igoro.com]
• LINQ in Action [Book by Fabrice Marguerie]
• C# in Depth: What you need to master C# 2 and 3 [Book by Jon Skeet]

I realize that the description is a bit abstract; let’s look at an example right away! It may look like a bit of code, but the classes are very simple and do just what you’d expect:

// Abstract node in a linked list
public abstract class Node {
	private Node m_next;
	public Node Next { get { return m_next; } }
	public Node(Node next) { m_next = next; }
}    

// Node in a linked list. Different nodes in the same list may carry data
// of different types.
public class Node<T> : Node {
	private T m_data;
	public T Data { get { return m_data; 	} }
	public Node<T>(T data, Node next) : base(next) {
		m_data = data;
	}
}    

// Abstract pair.
public abstract class Pair { }    

// Pair containing two values, possibly of different types.
public class Pair<TFirst, TSecond> : Pair {
	private TFirst m_first;
	private TSecond m_second;
	public Pair(TFirst first, TSecond second) {
		m_first = first;
		m_second = second;
	}
}

Here, Node is a node in a linked list where each node may contain a different type of data. Pair is just a pair of two values.

Now, suppose that we want to implement this method:

// Returns a pair containing the first two values from a linked list
public static Pair FirstTwoValues(Node node) {
	if (node == null) throw new ArgumentException();
	if (node.Next == null) throw new ArgumentException();    

	// Compilation error:
	return new Pair<U,V>((Node<U>)node, (Node<V>)node.Next);
}

Unfortunately, what I wrote on the last line is not valid C# because we cannot introduce a type variable in a cast. It is clear what we need to do: we have two Node objects which we assume to be instances of Node<U> and Node<V> respectively, and we would like to invoke some generic method – in this case a Pair<U,V> constructor – using both U and V as generic type parameters for the method. So how can we implement this?

Solution 1
The easiest solution is to use big nested if statements:

public static Pair FirstTwoValues(Node node) {
	// [...]
	if (node is Node<int>) {
		if (node.Next is Node<int>) return new Pair<int,int>(
			(Node<int>)node, (Node<int>)node.Next);
		}
		else if (node.Next is Node<string>) ...
		...
	}
	else if (node is Node<string>) ...
}

Obviously, this solution only works when the set of possible node data is constrained and very small, and even then the code is quite ugly.

Solution 2

Another possible solution is to use reflection. Reflection is very powerful, but it should be considered a last-resort technique because of its huge negative impact on the performance and general ugliness.

Solution 3

There is a third solution, that is in many cases preferable over the two I already mentioned. One way to “get access” to the type parameter T of the Node<T> is through a virtual method call. Node implements an abstract method, which Node<T> overrides. Calling the method on a Node reference results in that method executing in the context of Node<T> and we finally get access to that elusive type parameter T!

But, what if we need to get access to multiple generic type parameters coming from multiple objects, just like when trying to call a Pair<U,V> constructor given Node<U> and Node<V>? We can have multiple chained virtual methods, where each one fixes one of the generic parameters:

public abstract class Node {
	private Node m_next;
	public Node Next { get { return m_next; } }
	public Node(Node next) { m_next = next; }
	public abstract Pair ConstructPair(Node other);
	public abstract Pair ConstructPair<TFirst>(Node<TFirst> other);
}    

public class Node<T> : Node {
	private T m_data;
	public T Data { get { return m_data; } }
	public Node<T>(T data, Node next) : base(next) {
		m_data = data;
	}    

	public override Pair ConstructPair(Node secondNode) {
		return other.ConstructPair<T>(this);
	}    

	public override Pair ConstructPair<TFirst>(Node<TFirst> firstNode) {
		return new Pair<TFirst,T>(firstNode, this);
	}
}

Unfortunately, in order for this solution to be applicable, we need to have control over all types that could possibly be specified as the generic parameter, and they also need to share a common base class or implement a common interface that we can modify. Another disadvantage is that the type parameter class (in our case, Node) needs to know how to perform a possibly unrelated action (in our case, to construct a Pair). This may or may not be ideal from the design perspective.

Despite the disadvantages, I personally found this little trick quite useful. The problems I applied it to were related to abstract syntax tree manipulations, but I would expect similar issues to arise in other problem domains like data structures, data binding, etc. If you run into a similar design problem with generics, let me know whether you were able to apply this trick or not.