NET Progress

This blog -- taking a break

noreply@blogger.com (John Rusk) — Tue, 27 Apr 2010 08:13:00 +0000

Prompted, in part, by my annoyance at Google dropping the FTP support this blog relies on, but mostly by a lack of time, I won't be doing any work on this blog for the foreseeable future.

Commenting on all posts will be disabled.

However I do have plans to write new material on my other "non-technical" blog, especially about "people skills for geeks". (A topic on which I recently posted this video).

Thanks to everyone who's read and commented on this blog over the past 4 years.

-- John

Memory Leaks in Managed Code

noreply@blogger.com (John Rusk) — Tue, 27 Apr 2010 08:00:00 +0000

(This is a repost of an old article I wrote in early 2006, before I started this blog)

When writing a Window Forms application, it's useful to display the current memory usage in the "About" box. It comes in handy when trouble shooting. It usually proves that memory usage it not the problem, but sometimes it proves the opposite: the application is chewing up more and more RAM.

How can that be? How can a managed application leak memory?

Types of Managed Memory Leaks

There are two main types of "leak" in managed applications

1. Unintended references keeping managed objects alive

You're not using an object any more, so you expect that the garbage collector will clean it up. But the garbage collector doesn't. Why? There is only one possible reason: something, somewhere, still has a reference to the object. Perhaps you put the object in some global (i.e. static) list or perhaps it is referred to be some other object which you are still using.

So this isn't a "real" leak at all, it just looks like one. You've forgotten about a reference to your object, but the garbage collector hasn't. It sees the reference and keeps the object alive.

Of course, a reference only counts if it can be (recursively) traced back to a "root" object which is still in use - i.e. a static field or a local variable/parameter in a currently executing method. References from other objects which are, themselves, due for garbage collection will not keep an object alive. In fact, the garbage collector never even sees them. (It just starts from the roots and recursively visits reachable objects.)

Event handlers are a common cause of this problem. You create objects A and B then run some code like this:

A.SomeEvent += new EventHandler(B.SomeMethod)

Later, you finish using B, but you're still using A. Your on-going use of A keeps B alive, since A refers to B by way of the event handler. If you want B's lifetime to be shorter than A's, you have to unhook the event handler with a line like this:

A.SomeEvent -= new EventHandler(B.SomeMethod) //note the '-='

Of course, event handlers aren't the only cause of this problem, any reference between objects will do. Consequently, these problems can be difficult to track down. You can't think what the remaining reference is, but there must be one there, somewhere. To help track it down try a tool like SciTech .NET Memory Profiler. Or, if you prefer free development tools, try Microsoft's SOS debugger extension. But be warned, it can be a bit complicated.

2. Managed objects holding unmanaged resources

There are two variants of this problem:

(a) Badly-written managed objects, which don't clean-up after themselves. Of course, you don't write any of these, so let's move straight on to (b)…

(You don't write bad objects because you know that a good managed object should always implement IDisposable if it uses unmanaged resources. A good managed programmer will use that interface to clean the object up when he or she has finished with it. IF the programmer forgets, the good managed object will clean up the unmanaged resources anyway when the garbage collector collects the object. )

(b) Small managed objects, which use significant amounts of unmanaged memory. The .NET 1.1 garbage collector cannot see the unmanaged memory. It just sees the small managed part, so it thinks there's no need for a collection. (If it did do a collection, the well-written managed object would indeed free the unmanaged memory, but the garbage collector doesn't realise a collection is required.).

Bitmaps are a classic example of this problem, since they have a small managed component and a large unmanaged part. Under .NET 2.0, there is a new method which such objects can call to inform the garbage collector of their true size. It will solve this problem (as long as it's used correctly).

Other Types of Managed Memory Problems

Two other leak-like problems are possible (although relatively unlikely, in practice, I'd say). The first relates to "pinned" objects preventing the garbage collector from moving objects around. This is an internal .NET implementation issue, and has been significantly improved in .NET 2. The second relates to declaring lots of large objects and never letting them leave scope. In the unlikely even that you find yourself doing that - and suffering unacceptable memory usage - you should stop doing it :-)

This page itemizes all the kinds of memory leaks that I've listed here (although it fails to mention managed objects holding unmanaged resources).

What to Do

Always examine memory usage when you test your application. Don't assume everything will be OK simply because you're using managed code. Test it.
If you're writing a GUI application, consider displaying both managed and unmanaged memory usage in the About box, using GC.GetTotalMemory(true) and Process.GetCurrentProcess().PrivateMemorySize respectively. (See this page, and this one, for an explanation of exactly what PrivateMemorySize means.) Open the About box from time to time during your testing, to check that the figures look reasonable.
Call Dispose on IDisposable objects when you have finished with them. If an object implements IDisposable it is saying "please clean me up when you've finished with me". The object may hold unmanaged memory, or other resources such as files or GDI handles. Call Dispose() or get the using statement to do it for you.
Be careful with global (static) members. Don't overuse them. Why? If you have a lot of statics, you may find it harder to understand which of them are keeping objects alive. (This recommendation is for your benefit, not the garbage collector's.)
Get to know your garbage collector. Here's a good starting point and some more advanced stuff too.

What NOT to Do

In general, do not call GC.Collect(). Leave it to the garbage collector to decide when a collection is required. Having said that, I like to force a collection when my About box opens, to ensure the figures are accurate. (Here's how to force a complete collection - but like I said, you should almost never do so.)
Don't null-out local variables. In general, there is no need to set local variables to null when you have finished using them. In Release builds (unlike Debug builds), a local variable becomes eligible for garbage collection immediately after the last line that uses it. If you add a line after that, to set the variable to null, the new line does no good. It will either (very slightly) extend the lifetime of the object or it will have no effect at all (if the JIT compiler realizes the line is useless and optimizes it away). So keep your code clean and readable by avoiding pointless assignments to null. (When should you set things to null? I can think of a couple of situations: (1) When a small object with long lifetime refers to a large object with a short lifetime. Say you have two objects, A and B, and A contains a field that refers to B. If B is a very large object and you want its lifetime to be significantly shorter than that of A, then consider setting the field to null when you've finished with B. But don't go overboard - you probably won't need to do this very often. (2) Another situation, relating specifically to servers with threads waiting on external resources,is described here. Make sure you read the comments at the end.)
Don't get paranoid about garbage collector performance. It is easy to underestimate just how good the .NET garbage collector is. For instance, how many developers realise that a .NET app can create and destory around 50 million small, short-lived objects per second! (Admittedly, small-shortlived objects is the garbage collector's best-case scenario - but it also happens to be the most common scenaio in real world apps.)

Happy coding!

Debugging IEnumerable

noreply@blogger.com (John Rusk) — Fri, 15 Jan 2010 22:08:00 +0000

I often find enumerables difficult to inspect in the debugger. Here's a trick I just discovered.

Given:

IEnumerable foo = ...

you can inspect it more easily by first writing this in the Immediate Window:

foo = foo.ToList()

Press Enter in the immediate window, and foo changes from a (lazily) evaluated list (which is hard to inspect) into a real List which you can easily inspect by just hovering over it's Non-public members -> items.

Exporting High-Res Graphics from Excel

noreply@blogger.com (John Rusk) — Mon, 26 Oct 2009 00:57:00 +0000

I've recently been battling with technology, trying to save some Excel charts as high-res (900 dpi) files.
Here's the technique I finally came up with. (It also works for PowerPoint slides).:
1. Save the chart as a PDF from Office (The PDF output is vector-based, so is very high quality)
2. Use a command line line, this, to use Ghostscript to convert the PDF to a suitable tiff file. (In my case, I want the final images to be 4 inches wide, at 900 dpi, but here I have specified an output size of 5 inches (72 * 5 = 360), then I'll crop the images back to 4in in the next step.)
gswin32c -dSAFER -dBATCH -dNOPAUSE -sFONTPATH="E:\windows\fonts" -sDEVICE=tiffgray -r900 -dDEVICEWIDTHPOINTS=360 -dDEVICEHEIGHTPOINTS=221 -dPDFFitPage -sOutputFile=%~n1.tiff %1

Key parameters are -dPDFFitPage, to cause resizing when using PDFs ans input, the DEVICExPOINTS to set the size of the output in points (1/72 ths of an inch), and -r to set the DPI.
3. Finally, open the tiff in Photoshop (or similar) to crop and make any other final adjustments, then save from Photoshop.

Fix the Cross Functional Button on Toshiba Portege M200

noreply@blogger.com (John Rusk) — Mon, 31 Aug 2009 09:12:00 +0000

I recently bought a second-hand Toshiba M200 tablet PC. It's great!

I just had one problem, the little 4-way button, which looks like a mini joystick and is technically known as the "Cross Functional Button", basically didn't work. It was more of a non-functional button, really.

It is supposed to allow you to scroll in all 4 directions, which is a very important thing when in tablet mode, because you don't have access to the keyboard. But, it just output numbers instead of scrolling.

I searched and searched and found nothing particularly helpful. Eventually I went to the Toshiba site, found the "Toshiba Tablet PC Buttons Driver", installed it, and my problem was solved!
I have no idea why, out of all the pages I searched, no-one mentioned this. I found one where someone had the exact same problem, but resorted to a much more complicated solution. So, this post is here for the next person who has the same problem. Hopefully, you'll find this post and save yourself the hours of frustration that I had. Just install the driver ;-)
PS while you're there, install the accelerometer driver too, if you have this problem.

Things I'd like to see in the Entity Framework

noreply@blogger.com (John Rusk) — Sat, 31 Jan 2009 20:30:00 +0000

With the announcement that the Entity Framework (aka EF or LINQ to Entities) will become the favoured Microsoft ORM, many people have blogged about how important it will be for EF to achieve better ease-of-use in .NET 4.0.
I'd like to add my 2c worth on what I think EF should include. (By the way, I've never worked with EF, and won't until at least the .NET 4 version, so for all I know some of these things might be present already. Basically, this list is my attempt to capture some of the lesser-known lessons learned from using LINQ to SQL.)
1. Metamodel
An acccessible metamodel that is at least as capable as that in LINQ to SQL (i.e. MetaModel and friends). On my current project the metamodel has saved us over and over again - allowing us to do things that we needed but couldn't do with just the basic generated entities. I think it's extremely important to have this kind of "back door" that lets programmers step outside of the POCO model and access mapping-related state and metadata.

2. Lifecyle information in constructor
A way to tell the difference between explicit construction of an object in code, verus construction by EF as the object is loaded out of the database. This is a gap that we've seen in LINQ to SQL. There are partial methods for various parts of the object lifecyle (such as creation and pre-save validation) but there is no way to tell whether the creation event is happening due to a load from the database, or the "new"-ing of an instance by application code. It seems wrong to have a full set of "hooks" into the object lifecycle except for this.
My suggestion would be to have the generated entity class include two constructors. One would be a parameterised constructor. Its parameter would be an enum that indicates whether creation is due to loading or some other reason. E.g.
public Person(CreationContext context)
{.. }
where context may be something CreationContext.NewInstance or CreationContext.Loading
The other generated constructor would simply look like this
public Person():this(CreationContext.NewInstance)
{...}
So, when we write "new Person()" in code, the parameterless constructor passes the NewInstance parameter through to the "real" constructor. When the framework loads an object from the database, it should call the parameterized constructor directly, passing CreationContext.Loading. Then, at construction time, the object can always tell the purpose for which it is being constructed: does it represent a brand-new entity, or is it, conceptually, an existing entity which is being "deserialized" from the database?
The most obvious use for this is objects which, when created, should always have particular child objects. E.g. an order that always has at least one order line. If the order doesn't know why it's constructor has been called, then it cannot go ahead and make a default child instance - one might already exist in the database in the CreationContext.Loading scenario.
3. Don't presume or require a big impedance mismatch
There are some advantages to LINQ to SQL's very direct mapping approach. In particular, because the database is virtually identical to the object model the team only has to learn, understand and remember one model. The means that Ubiquitous Language extends all the way down to the database. (Particularly important if SQL, rather than objects, will be the basis for report generation; but still useful even if you're not doing SQL reporting, IMHO.)
As soon as you get the complex mappings, which are touted as an advantage of EF, the team has to understand two models, and the relationship between them.
In some applications, such mapping is indeed necessary. Perhaps an existing database doesn't match good object design; or the object design is complex enough to require non-trival mappings.
However, in projects where the same team is building the application and database from scratch, and where the object model is not too complex, it can be a good thing to have a simple mapping that's virtually one-to-one. EF should allow this, and the EF documentation should present it as a valid option.
4. Hand-writable entities
Property getters and setters, on domain entities, should be simple enough to write by hand. This can be done by some kind of AOP (to inject "smarts" into ordinary-looking properties) or by ActiveSharp. LightSpeed is a good example of an ORM in which all the "smarts" of property setting and getting reside behind simple hand-writable properties.
[Disclaimer: I wrote ActiveSharp and Lighspeed (optionally) uses it. My point here is not to promote my own code, but to promote the idea that entity properties should be concise. Concise properties, whether hand-written or codegen'd, make the code much easier to work with.]
5. Object relationships should be able to span diagrams (and assemblies)
An annoying limitation of LINQ to SQL (and LINQ to Entities) is that, in practice, all your entities must be in one designer diagram. I say that because, if they are not in one diagram, then you can't have meaningful relationships (and lazy loading) between objects in different diagrams.
This relates to point 4, above. If properties are simple enough to write by hand, then you can use several different diagrams and then "stitch them together" using hand-coded relationship properties.
E.g. you might have one diagram focusing on enties related to sales, and another on entities related to production. For the (hopefully few) relationships that go from entities in the Sales diagram to entities in the Production diagram, you can create them by hand as long as hand-authoring is a viable option.
I'm planning to do this on my current LINQ to SQL project, possibly using some of this code.
Finally, such a solution should not just allow entities to be defined on different diagrams, but also for those diagrams (and their generated entities) to be in different assemblies/projects.
Well, that's the end of my 2c worth.... for now ;-)

What's up with P&P?

noreply@blogger.com (John Rusk) — Tue, 02 Dec 2008 08:19:00 +0000

What's up with the interface between Microsoft's Patterns and Practices group (P&P) and the wider community?

P&P are writing a new version of their guidance for architecture on the Microsoft platform, and they're asking for community feedback. But by and large the community isn't giving any! And, when feedback is offered, P&P aren't necessarily replying.

For instance:

After the P&P Knowledge base project had been up for about 2 months, I counted exactly three meaningul comments on the substance of what MS had written. Of those three comments, two were from me(!), and to this day those two remain unanswered!

Confusingly, there are now three different CodePlex projects in which P&P are seeking community input. There's the App Arch Guide knowledge base, the App Arch Guide Book, and the App Arch Community Contribution project (which, as if to prove my point, is completely empty)!

So, community, what's up! Has no-one got anything to say about architecture!!!!

And Microsoft, what's up with you? What are you doing to make this work?

Update 1 Feb 08: Microsoft recently contacted me to follow up on my questions. Thanks :-) As much as I appreciate that contact (and I do) the overall lack of engagement seems to remain. I still don't see much meaningful involvement from/with the community. The Community Contribution project is still empty, except for a brief statement of it's purpose, which has only been read 38 times.

Hosting a window from one process inside another

noreply@blogger.com (John Rusk) — Wed, 19 Nov 2008 06:48:00 +0000

Every time this topic comes up, I seem to have lost my bookmarks on it. So, here's a blog entry so I won't lose them again...

Under Windows it is possible to visually "dock" the main window of one process inside a Window belonging to another process. You get the visual effect of one program, but there are still two completely separate exes involved.

I once used this to "host" an EXE inside Internet Explorer. We wrote a little tiny ActiveX (this was in the dark ages before managed code). All the ActiveX did was start our EXE, and then "dock" the EXE's main window inside the client area of the ActiveX. It looked like our EXE was the ActiveX - but our EXE had no idea that any of this was going on. It just doing its thing, running as an independent process.

The secret is the Windows API function SetParent. You can use it to set a window from process A as the parent of the main window of process B.

I haven't done this for a while. As I recall you need to make another call to make the hosted window look like a borderless child window (SetWindowLong IIRC) and I think I also had to detect resising of the "host" and programatically resize the child.

Here are some links on the subject, following a rather brief Google:

http://geekswithblogs.net/gyoung/archive/2006/04/26/76521.aspx

http://www.codeguru.com/forum/showthread.php?threadid=234862

http://www.codeproject.com/KB/miscctrl/winwordcontrol.aspx

I'm fairly sure I learnt this technique from something on Microsoft's site, in about 2000. But this post says they are no longer recommending it (at least, not for hosting Office apps) so perhaps that explains why I can't find the original MS post.

LINQ to SQL Presentation

noreply@blogger.com (John Rusk) — Tue, 04 Nov 2008 07:51:00 +0000

Here is a copy of the presentation I have at the 2008 Christchurch Code Camp. (zipped ppt)

One of the key points of the presentation was to cover what LINQ to SQL includes, both out-of-the-box, and with the additions that we have been able to build on top of it at Optimation. I put together the list that follows from two sources: Ayende's list of 25 things your OR Mapper must do; plus other things that we found useful. Ayende's points are in normal font; my additions to his is are shown in italics.

Out of the box, you get:

CRUD and querying
Transactions
(Bi-directional) associations
Lazy loading
Polymorphic queries (single-table inheritance only)
"Dirty Tracking" with ability to return full change set
Loading properties without loading whole object
Identity Map
Concurrency Control
Acceptable debuggability
Safe multi-threading (1 datacontext = 1 user on 1 thread is the general rule of thumb)
Well-defined exception policy
Lifecycle events (create/update etc)
Composite primary keys
Automatic dependency ordering when saving changes
Paging support (via Skip/Take)
Aggregation support (group/max/min etc)
Original value tracking
Property change notifications
Runtime SQL logging
Ability to generate database from object model
Persistence by reachabilty
Enhanced "reflection" via LINQ to SQL MetaModel

With our add-ons to LINQ to SQL you (or at least we ;-) also get

Undo (both to "as fetched" and "current DB state")
Flexible eager load
Unit testability
Test: is this object new?
Check mapping against DB
In-memory savepoints (like DB savepoints, but for in-memory entities which haven't been saved yet)
Proper serialization of entities (none of the usual LINQ to SQL serialization restrictions)
Clone trees of related entities (clone parent with children)
Delete-any-object (even if is a new unsaved one; automatically sever relationships to other objects)
Get data context from object (based on this with only very minor changes)
Get reachable objects (approximates "find me all objects in the datacontext)

You get only limited support for:

Caching at unit-of-work (datacontext) level
Custom field types (limited to built-in types, enums, xml, .Parse()-able strings and binary ISerializable. Here is the valuable and hard-to-find reference page)

You get no support at all for:

Caching at application-wide level (other than that which is done for you by SQL Server itself)
Cache invalidation policies
Update batching
Cascading update/delete in object model (must rely on DB to do this instead, if you want it)

All in all, its a fairly strong list. Stronger, I suspect, than many people expect.

As for the rest of the presentation, some of it will make sense from written slides, and some won't ;-) A couple of notes here might help:

The solution to comments on entity properties, and refreshing the designer, can be found here. It looks real good, although I haven't got round to trying it yet myself.
Here are the hyperlinks to the change set and association bugs. NOTE: Microsoft have just announced that they will fix the latter in .NET 4.0, which goes to prove that they really are going to include LINQ to SQL fixes in that release.
Using the "mapping checker" code. These three points will help if you want to use it: LinqUtil.GetModel is our wrapper for MappingSource.GetModel() - see later slides for details; LinqUtil.IsMappedType() is our wrapper for calling IsEntity on a metaType returned from the MetaModel; and Uow.All(type) is our wrapper for DataContext.GetTable(type). Simply copy the code, replacing our wrappers as noted above.

That's all for now. As noted in the presentation, please do comment below or email me.

Future of LINQ to SQL

noreply@blogger.com (John Rusk) — Sun, 02 Nov 2008 08:21:00 +0000

Yesterday, I did a presentation on LINQ to SQL. At about the time I was talking about how great it was, Microsoft was announcing that LINQ to SQL will not be their recommended solution in .NET 4! They will be recommending LINQ to Entities instead.
Here, in no particular order, are some intial thoughts:

Sorry I wasn't well-informed enough to mention this at the presentation! It seems like the news hit the net after I left for the Code Camp. I did hear there'd been speculation from outside Microsoft, but I mistakenly dismissed it as an over-reaction to the recent emphasis on Entity Framework.
The news is not as bad as it sounds. The best post on the topic seems to be this one: http://damieng.com/blog/2008/10/31/linq-to-sql-next-steps. Read that post now, if you have not done so already. Basically, LINQ to SQL will still be available and there will even be improvements and bug fixes. In particular "we [Microsoft] are going to make sure LINQ to SQL continues to operate as it should. This doesn’t just mean making sure what we had works in .NET 4.0 but also fixing a number of issues that have arisen as people pick it up for more advanced projects and put it into production environments".
To successfully execute this strategy, Microsoft will have to bring the best of LINQ to SQL to LINQ to Entities - in particular, they will have to bring lightness, speed and simplicity to cases where the mapping is simple and relatively direct. I hope the recent addition of Andrew Peters to the team will help in that regard, due to his experience on Mindscapes' LightSpeed ORM.
I think it will be very important for LINQ to Entities to expose the equivalent of LINQ's MetaModel - complete with the ability to use it to implement workarounds, as I demonstrated in my presentation. (We use the MetaAccessors to do all kinds of things). We've been able to use it plug virtually all the "holes" we've found in LINQ to SQL, and a similar capability is a must in LINQ to Entities if it is to become the favoured product.
Microsoft: to make this kind of announcement, about a product that has been out for less than 12 months, is not a good look! (LINQ to SQL's production release was roughtly 11 months ago). This is particularly true in the area of data access, where there were already plenty of jokes about a long history of technical changes - ODBC, RDO, DAO, ADO, OLEDB, now ADO.NET.
For projects you start today, with .NET 3.5, I believe the advice I gave in the presentation is still accurate. For simple mapping consider products like LINQ to SQL and LightSpeed; for complex mapping consider products like LINQ to Entities and nHibernate. If Microsoft achieves the goals mentioned above, my advice might change for new projects that are started after .NET 4 is released.

Presenting at Christchurch Code Camp 2008

noreply@blogger.com (John Rusk) — Tue, 21 Oct 2008 08:13:00 +0000

I'm presenting at the Christchurch Code Camp on 1 November.

I'm presenting on LINQ-to-SQL and also on my open source project, ActiveSharp. If you were coming to a presentation on either of those topics, what would you like to hear? (This is your chance to shape the presentations, because I haven't finished writing them yet ;-)

At the moment, the abstracts for the two presentations look like this:

Title: Extending LINQ to SQL

Abstract: How to build on LINQ to SQL to add extra capabilities and work around the product’s limitations. The presentation will share techniques we’ve used at Optimation to "push the envelope" when using LINQ to SQL. I’ll outline the base techniques, how we’ve applied them, and the minor limitations that remain unresolved. For those new to LINQ to SQL, the presentation will begin with a whirlwind introduction to the product. It will conclude with some thoughts on how LINQ fits into the "big picture" of .NET ORM tools.

and

Title: Low-Level .NET – a look at ActiveSharp

Abstract: How to get started on working with MSIL, the low-level "assembly" language for .NET. The presentation is based on my experiences writing ActiveSharp, a library which parses MSIL at runtime and also emits its own runtime-generated methods. I’ll introduce the key concepts, cover some techniques to help you "cheat" while you learn, and include examples from the ActiveSharp codebase.

What do you think? Am I focusing on the right points? Are there other aspects of these subjects which you'd prefer me to cover instead? All suggestions welcome...

How eBoostr Works

noreply@blogger.com (John Rusk) — Sat, 04 Oct 2008 22:44:00 +0000

I'm using eBoostr to speed up compile times in Visual Studio.

My only complaint with the product was the lack of documentation on what it caches and when. You can view the list of all the files that it is caching, but I couldn't find any documentation on how it actually chooses which files to cache. So, I emailed eBoostr technical support, and they sent back a very helpful and informative reply - which they kindly allowed me to quote in this post.

eBoostr works as follows:

It monitors all read/write requests to the hard disk, and gathers statistics on the most frequently-read files
By default it will "update cache contents each hour during computer idle time automatically". You can also manually request a cache rebuild (if your computer has no idle time(!), or you just want the cache to reflect your most recent usage patterns).
It doesn't mess with write operations. Each write goes straight to the real disk. If the file was cached, the cached copy is invalidated. The cached copy will not become valid again until the next cache rebuild.
If you are using both RAM and USB caching, RAM "has the highest priority" (so gets the most frequently used files, I presume). Files in RAM are not cached on USB.

At first, the algorithm struck me as a little crude. But on reflection I realised that it makes sense. By rebuilding the cache in batch mode (either automatically or manually) eBooster can make sure that the cache reflects the most commonly-used files. Contrast that with the other approach, which would be to move a file into the cache as soon as it is read - possibly meaning that an infrequently-read file, which maybe you are only going to read once, will bump a more frequently-used file out of the cache.

Subjectively, eBoostr seems to be working well for me. And as I've mentioned previously, it seems to be getting very high cache hit ratios in my typical daily usage.

Visual Studio Performance with Slow Laptop Drives

noreply@blogger.com (John Rusk) — Mon, 15 Sep 2008 08:15:00 +0000

As Scott Guthrie wrote, hard drive speed makes a big difference to the performance of Visual Studio.

So what can you do if, like me, you are using a laptop and so can't upgrade to a faster hard-drive? I've found two things that have helped a lot. In one test case they reduced my build time from 28 seconds down to 3 seconds(*).

Step 1: Avoid slow virus scanners

This made the biggest difference of all. As Jeff Atwood reports, some real-time anti-virus scanners can slow your machine down drastically. My test build dropped from 28 seconds to 7 seconds when I turned off my virus scanner! That seems entirely consistent with the results that Jeff has summarised on his page. In fact, judging by his post, it appears that some products may be even worse.

For the record, I was using TrendMicro OfficeScan. I don't know why it had such a large adverse impact. Googling the vendor's website failed to unearth anything relevant, but I've heard anecdotal evidence of it reading all bytes in a file when Visual Studio merely wants to check the timestamp. On the other hand, my colleague on the next desk uses the same product (on faster disk hardware) and says it's OK.

[Update: I've switched to AVG and it's looking good - much, much faster. I'm also trialing NOD32 on another machine, because I've read good things about it.]

Step 2: Use disk caching (with lots of RAM)

I've upgraded to a total of 4GB RAM in my laptop. Since I'm running Windows XP, the operating system only sees 3-and-a-bit, but that still gives me enough to run an in-memory disk cache.

I'm using eBoostr to run the cache. It seems to be working very well. It reduced my test build from 7 seconds down to 3 seconds, and subjectively it seems to be having a noticable positive influence*.

If you go to the eBoostr site, you'll see that it is advertised as being the XP equivalent of two Vista technologies, ReadyBoost and SuperFetch. Those technologies are about reducing system boot and application start times, but it also turns out to be good at reducing compilation times. That's not suprising really, since it works by caching the files that are most frequently used. Since I'm interested in compile times rather than boot times, I have configured it to do all it's caching in RAM. I don't even plug in a USB device.

In a typical day's coding its built-in monitoring tool reports average cache hit ratios of around 80 or 90%, which means it's allowing the machine to skip a lot of disk access.

I definetely recommend giving eBoostr a try. The cost is very competitive and it has very generous evaluation terms - you can use the eval copy for ever, with the only limitation being that it turns itself off 4 hours after every reboot. This means you can take your time evaluating it, as long as you don't mind rebooting daily.

My only criticism would be that the documentation doesn't really explain how it decides which files to cache. I emailed the vendor about that point, and they sent a very informative reply, which I'll summarise in a future post [here].

One word of advice: if you run Visual Studio and a local copy of SQL Server (as I do), don't bother with RAM caching if you only have 2GB of RAM. Get more RAM first, then start caching. I'm using a cache size of about 800MB, which seems large enough to be useful, and small enough to leave enough RAM for all my dev tools.

Finally...

(*) Your mileage may vary - and in fact my mileage does too, depending on what I'm doing and (presumably) on what is in the cache at the time. In some of my tests, the virus scanner seems to account for all the difference; with little difference coming from the cache. In other tests, the cache makes a noticable difference.

In any case, I'm enjoying TDD again. The 30-second builds used to break my chain of thought, but now the edit-compile-test cycle is short enough that it doesn't interrupt my flow. All in all, my 5400 RPM drive no longer feels like a serious problem.

Syntactic Noise sample in C#

noreply@blogger.com (John Rusk) — Sat, 30 Aug 2008 21:01:00 +0000

Martin Fowler wrote an interesting post on syntactic noise in internal DSLs.

I thought it would be interesting to compare the Java versions (there are two in his post) with a concise C# version. Here is the result, the most concise C# version that I could come up with:
static Event
    DoorClosed = "D1CL",
    DrawOpened = "D2OP",
    LightOn = "L1ON";

static Command
    LockPanel = "PNLK",
    UnlockDoor = "D1UL";

static State
    Idle = new State
    {
        UnlockDoor,
        LockPanel,
        DoorClosed.TransitionTo(Active)
    },
    Active = new State
    {
        DrawOpened.TransitionTo(WaitingForLight),
        LightOn.TransitionTo(WaitingForDraw)
    };

It's reasonably concise (slightly more so that the fluent Java one I think), however it did require me to make everything static - which kind of counts as cheating ;-) [Update 9 Sept:] So here is a more realistic version:

public Event
    DoorClosed = "D1CL",
    DrawOpened = "D2OP",
    LightOn = "L1ON";

public Command
    LockPanel = "PNLK",
    UnlockDoor = "D1UL";

public State
    Idle,
    Active;

protected void Setup()
{
    Idle = new State
    {
        UnlockDoor,
        LockPanel,
        DoorClosed.TransitionTo(Active)
    };
    Active = new State
    {
        DrawOpened.TransitionTo(WaitingForLight),
        LightOn.TransitionTo(WaitingForDraw)
    };
}

The key differences from the Java version are that it uses implicit conversions to convert strings directly to Events and Commands, and that it uses collection initializers to set up the contents of the States. (Note that a collection initializer need not contain things that are all of the same type. Instead, it will accept all types for which the target collection has an Add method. In our case, that is Commands and Transitions (created by TransitionTo)).

Full source code is attached.

PS as I've noted before, I think that C# offers more options for in-language DSL syntax than Java.

A Practical Step in the Stored Proc Debate

noreply@blogger.com (John Rusk) — Tue, 26 Aug 2008 08:14:00 +0000

I've taken a practical step in the long-running debate for and against stored procs.

I've logged a formal suggestion with Microsoft asking them to correct their erroneous advice on the subject. :-)

This Patterns and Practices page says:

Stored procedures generally result in improved performance because the database can optimize the data access plan used by the procedure and cache it for subsequent reuse.

That's been factually incorrect for the last decade! In terms of query plan caching, Parameterized SQL and Stored Procs are handled in the same way - and they have been since SQL Server 7.

So, if you too are disappointed by the level of erroneous information in this long running debate, head over to the feedback page and support my request to fix the P&P page. You can support it via commenting on the page or by using the Validation link at the top.

Let's hope we can get the page corrected, and achieve a small step for rationality in an often irrational debate :-)

(Apologies to Microsoft if my tone sounds hash. I know you've already got the correct information up on other parts of your site.)

Update Oct 08: the P&P guidance is being updated, so one presumes this will be fixed as part of that work.

Software Development Meme

noreply@blogger.com (John Rusk) — Fri, 18 Jul 2008 09:11:00 +0000

I've been tagged by Andrew, in a software development meme that started way back here.

How old were you when you started programming?

I was exactly 13. My parents bought me my first computer for my 13th birthday. It was a second-hand Ohio Scientific Superboard II.

By the way, the Superboard shipped in 1978 as a single circuit board. It had keys mounted on the front and chips mounted on the back. That’s it. No case, no nothing – just one big circuit board with stuff on it. By the time it reached me a few years later, mine had been fitted into a hand-built wooden case, and its memory had been expanded to 16k. (I still have the machine. If anyone knows how to fix a Superboard II that won’t respond to the Break key after power-up, please contact me :-)

How did you get started in programming?

Before my SuperBoard, I’d only seen two other computers. I guess they must have made a big impression on me. I got Mum and Dad to buy me a book on programming in BASIC, and the purchase of the actual computer followed sometime after. I don’t recall how much nagging was involved ;-)

What was your first language?

The Superboard’s version of BASIC, which was apparently "Microsoft 6502 BASIC version 1.0 revision 3.2". In hindsight, I love the fact that it was simple enough for a 13 year-old to learn the whole language. If I recall correctly, there was IF .. THEN, but no ELSE; and GOTO but no GOSUB. Definitely a minimal language. It made me appreciate "real" languages when I finally got to use them.

What was the first real program you wrote?

It was a clone of the first computer program I’d ever seen – a game where you had to steer a tank through a minefield. My version had the randomly-generated minefield, and the tank that was stuck on full-throttle while you steered your way though, but it lacked the system-generated comments that poked fun at your crashes.

What languages have you used since you started programming?

I’ve done serious development in Pascal, Excel 4.0 Macro language, VBA, Ada, PowerBuilder, PL-SQL, Delphi and C#. I’ve dabbled in various others including C++, JavaScript and even VB.NET (although it makes my eyes hurt when I read it ;-)

What was your first professional programming gig?

I did a few small jobs in BASIC and Excel when I was a student, but my first really serious job was in PowerBuilder for an IT consultancy. PowerBuilder’s (in)famous DataWindow really was a fast way to get things done. It set the bar for productivity, and it's only now in 2008 that I feel like we've replicated that speed in .NET - thanks to a good ORM and our in-house framework. (And we get a proper domain model with layering, which we never had in PB!)

If you knew then what you know now, would you have started programming?

Yes.

If there is one thing you learned along the way that you would tell new developers, what would it be?

I might tell them that it’s not about technology, it’s really about people – but I don’t think they’d believe me.

So instead I’d tell them to read How to be a Programmer.

What’s the most fun you’ve ever had... programming?

I’ve particularly enjoyed my old Superboard; writing (most of) a compiler at university; doing some funky stuff with COM in Delphi; and releasing a couple of open source projects.

But the thing I look back on most fondly is a .NET project at Optimation. The highlight wasn’t the technology, which was interesting but not amazing; the highlight was the project as a whole. We had great collaboration with the customer, very much "one team" with great dialogue and innovation, and together we exceeded expectations instead of merely meeting requirements.

Over to you

Bevan
Eduard
and
Kirill

POCO LINQ Update

noreply@blogger.com (John Rusk) — Tue, 15 Jul 2008 10:08:00 +0000

Here is my long-promised near-POCO solution for LINQ to SQL.

As planned, it is deliberately not 100% pure POCO, but it does implement these key goals:

Entity classes are simple enough to be hand-written (no need to generate them with SQL Metal or VS 2008)
Mapping is easy (no need to manually map every property via attributes or external XML)
Lazy-loading works
Bi-directional relationships work (update one end, and the other is synchronized automatically)

Another reason why architects should write code

noreply@blogger.com (John Rusk) — Sat, 07 Jun 2008 21:20:00 +0000

Johanna Rothman explains why architects should write code. I completely agree.

Not only does hands-on involvement reduce risks on the project, it also reduces embarassment when talking to your kids ;-) Here's an actual conversation with my daughter:

What did you do today Dad?
I drew some diagrams. Do you know what a diagram is?
No
It's a picture that shows how something works. I drew some diagrams, then I talked with some people about them.
So you didn't do any work then?

Only 4 New Features in C# vNext?

noreply@blogger.com (John Rusk) — Wed, 30 Apr 2008 08:51:00 +0000

I've always admired the effort that Microsoft put into the design of C#. It's difficult to find the right balance between power and flexibility, and on the whole I think they've done an excellent job.

At the same time, it's been great to see the language grow, particularly with the addition of LINQ. As Jon Skeet has written recently, C# today is noticably more complicated than it was at version 1.0. Personally, I don't think that's a bad thing. The C# community has grown up along with the langauage, learning the new features as they have been released.

I can see this process continuing for some time, with the language continuing to grow. So I was suprised to see this comment from Microsoft's Mads Torgersen, in response to some suggestions that I posted at Microsoft Connect:

"We have to do some harsh prioritization, both because of our implementation and testing resources, but also because we need to keep the number of new langauge features at a manageable level - depending on how you count, we are adding only four language features to C# this time around".

We more-or-less know that one will be some form of dynamic lookup. That just leaves three others. They might be big "killer" features (like generics and lambda expressions in previous versions) but there's still only three of them.

I don't know what to think. Should we be trilled that Microsoft are continuing their excellent track record of keeping the language simple, or should we be dissapointed that there will be so few improvements?

Summary of C# vNext Ideas in this Blog

noreply@blogger.com (John Rusk) — Sun, 24 Feb 2008 05:30:00 +0000

Here's a summary of the ideas I've suggested in this blog, for future versions of C#:

Fix the versioning issues with extension methods
Consider "extension interfaces" to provide mixin-like behaviour
Also, consider improving the resolving of extension methods
Retain some means of serializing iterator state (even if the underlying iterator implementation is changed) (See comments here)
Provide an infoof operator, or similar
Better handling of @-quoted strings that contain double quotes
Enhance automatic properties to retain their concise syntax but to offer greater flexibility to use programmer-defined code. Particularly helpful for automatic INotifyPropertyChanged implementation

Enhanced Automatic Properties

noreply@blogger.com (John Rusk) — Sun, 24 Feb 2008 04:41:00 +0000

Recently there's been a lot of blog posts discussing things people would like to see in the next version of C#. Here's my addition to the wish-list:

"Automatic properties that let me supply a method body." It sounds like a contradiction in terms - but it doesn't have to be. Instead, it would be a compact syntax for defining properties that use standardised method bodies (where you define your own "standard" bodies).

Some places where it would be useful:

Cases where many properties follow a standard format. For instance, all the properties generated by the LINQ designer could be single-line automatic properties. (Much less verbose than the current approach).
ORM's and similar products that require properties to be written according to certain conventions. (E.g. Mindscape LightSpeed)

How would it look?

Right now, we can write automatic properties like this:

public int Foo { get; set; }

My suggestion is that the words "get" and "set" could be replaced by the names of methods. For instance:

public int AdvancedFoo { GetValue; SetValue; }

Where the methods "GetValue" and "SetValue" are generic instance methods with these signatures:

/// <summary>
/// When passed the field, return the property value
/// </summary>
T GetValue<T>(ref T field)
{
// ... your own code goes here...
}

/// <summary>
/// When passed the field, by ref, and the new property value, set the field
// </summary>
void SetValue<T>(ref T field, T value)
{
//...your own code goes here...
}

Given the above property, "AdvancedFoo", the compiler would compile it as if I had written this:

public int AdvancedFoo
{
get { return GetValue(ref _advancedFoo); }
set { SetValue( ref _advancedFoo, value); }
}

private int _advancedFoo; // compiler-generated backing field

Just as in today's automatic properties, the compiler would generate the backing field for me. The difference is that it reads and writes that backing field via methods which I define.

This goes a long way towards giving the best of both worlds: the compact syntax of automatic properties, and the flexibility of my own code (plus standardisation - all me properties that are supposed to work the same way do work the same way, because they all use the same code).

Questions:

Why does the "GetValue" method take the field "by ref"? SetValue needs it by ref, but why GetValue? GetValue takes it by ref simply so that you can use this trick to find out which property is being read - just in case you need to know. This is something you are more likely to need to know when setting, e.g. to raise change notificiations, but maybe some people want to identify the property in GetValue too. Using the ref param makes that possible.

What type should the complier use for the backing field? Simply put, it should use whatever field type is implied by the field parameters of the "GetValue" and "SetValue" methods. E.g. an automatic property for a LINQ EntityRef might look like this:

T GetEntity(ref EntityRef<T> field) where T:class
{}

void SetEntity(ref EntityRef<T> field, T value) where T:class
{}

allowing me to write this:

public Customer Bar{ GetEntity; SetEntity; }

and have the compiler compile it as if I had written this:

public Customer Bar
{
get { return GetEntity(ref _bar);}
set { SetEntity(ref _bar, value);}
}
private EntitySet<Customer> _bar;

Note that the compiler is able to infer that the backing field should be EntitySet, not Customer, based on the signatures of Get/SetEntity.

(And how would Get/SetEntity actually work? That's a question for another day ;-) I do have some (working) prototype code...)

What do you think of this idea for enhanced automatic properties?

Update 2 March: posted on MS Connect here (in the comments)

Update 12 March: And here here as an MS Connect item in its own right.

Extension Methods - Feedback to Microsoft

noreply@blogger.com (John Rusk) — Thu, 14 Feb 2008 08:43:00 +0000

If you've been following the disussions about extension methods in this blog, you might be interested to know that I've submitted the issue to Microsoft's feedback site. Jump over there and vote for it, if you'd like to see something done.

Better Property Change Notification

noreply@blogger.com (John Rusk) — Fri, 08 Feb 2008 22:58:00 +0000

aka Automatic INotifyPropertyChanged

Some time ago I developed a new way to do property change notification. It consists of code which can automatically figure out which property has changed, without you having to tell it.

Details are here.

I've now put together an updated version, and posted it on CodePlex here.

Here's how to use the new version:

Option 1:

Here's the easiest option...

Add this code to your class or base class:

PropertyChangeHelper _propertyChangeHelper = new PropertyChangeHelper();

public event PropertyChangedEventHandler PropertyChanged
{
add { _propertyChangeHelper.Add(value); }
remove { _propertyChangeHelper.Remove(value); }
}

protected void SetValue<T>(ref T field, T value)
{
_propertyChangeHelper.SetValue
(this, ref field, value);
}

Then write your property setter methods like this:

public int Foo
{
get { return _foo; }
set { SetValue(ref _foo, value); }
}

Option 2:

If you want more control, you can find which property changed like this (and then do whatever you like):

protected void SetValue<T>(ref T field, T value)
{
...
field = value;   //Actually assign the new value
PropertyInfo changedProperty = PropertyMap.GetProperty(this, ref field);
// do whatever you like, now that you know which prop was changed
...
}

Key differences in the new version are:

No "trial sets". The old version used to call all your property setters, to figure out which fields they corresponded to. That could be annoying, because you ended up recieving calls to the set methods when your objects were not necessarily ready for them. The new version doesn't need to do this. Instead, it inspects the MSIL of the set methods, to figure out what it needs to know. (The inspection is a one-off runtime operation, so it won't be a performance problem.)
Fewer contraints on how classes are coded. The old version required that you nominated a special field as the "reference field", to which all others were compared. The new version does not require that. Also, the new version lets you use your own base class (with your own SetValue method, named whatever you want). Fianlly, you can put your own code in the property setter methods (along with the call to your "SetValue" method), something which could cause problems in the old version
Helper class for easy implemetation of property change notifications (as above)
Granualar implementation, so that you can call some of the implmentaiton classes, but still write the outer "layer" of the API yourself if you need to
Lock-free. The new version is threadsafe without explict locks (and without and [ThreadStatic]s). Update 10 Feb: This used to rely on a little trick with generics and static fields; but that doesn't play well with inheriance. It now uses a class I wrote called LazyDictionary, which is basically a dictionary with double-checked locking to ensure that locks are only requested during initialization, and not during the on-going running of the application. (Didn't want a lock request on every property set call!)
No separate C++ assembly. The old version used managed C++ to do stuff that C# cannot. The new version uses reflection emit, to produce the corresponding routine on the fly, so that the separate assembly is not required.

Finally, I have resurrected the name "ActiveSharp" (which I used for a previous, abandonned project) for this new version of the code. I did that for two reasons, firstly the new version borrows code from the old ActiveSharp (particulaly the MSIL inspection); and secondly I liked the old name and couldn't think of a better one ;-) Just like the previous "ActiveSharp", this new version is a product of my microISV, Tasman Logic Ltd. (The microISV is not exactly a money-making venture, given that its only product is open-source ;-)

Resolving Extension Methods

noreply@blogger.com (John Rusk) — Tue, 15 Jan 2008 07:11:00 +0000

Paul Stovell (author of SyncLINQ) has pointed out another issue with Extension Methods. As he writes:

The problem is the methods are *resolved* at compile time.

Suppose you had this:

void DoStuff(this IFoo foo);

class FooFactory
IFoo Create()

If your factory returns an object that provides its own instance implementation of DoStuff, it WILL NOT be invoked - the extension method will instead. This breaks many of the rules around polymorphism and good OO in general...

In other words, the compiler favours instance methods over extension methods, but only if it can "see" the instance method at compile time.

The reason is simple: currently, resolving an extension method is just a compiler trick, so compile time is the only time when it CAN be done.

But Paul's comment got me thinking. It occured to me that, if my idea about using interfaces was implemented, then it would be possible to properly solve the problem he described. It would work like this:

First, to recap: I suggest that there should be a mapping between extension methods and interfaces. Think of a set of extension methods as an "extension contract". You can make calls on the methods in that contact either by calling intstance methods (on objects which implement the interface) or by extension methods (for objects which do not implement the interface). This is to solve the versioning problem, but it can also be used to solve the problem Paul described.

How? I'll try to illustrate with an example. Imagine that our "extension contract" is called IExtensionContract. It contains two methods, Foo and Bar. We can call Foo and Bar on objects which don't support IExtensionContract, simply by calling them as extesion methods. But, when we call Foo and Bar on objects which DO support IExtensionContract, we want to call the version that is on the object even if we can't tell that the object supports them at compile time. (That's the heart of the issue, right Paul?)

That could be done if the complier worked like this:

Given this code to compile

ISomething s = ...
s.Foo();

The compiler should compile it as if the programmer had written

ISomething = ...
IExtensionContract c = s as IExtensionContract;
if(c == null)
StaticExtender.Foo(s);  // invoke the extension method
else
c.Foo();                // invoke the instance method

This would be possible, if the interface-based versioning solution was adopted, because that solution introduces just enough runtime information about the extension methods - namely the "contract" to which the methods belong. Contrast that to the current situation, where there is no runtime information about extension methods.

Here is the user-authored (i.e. not compiler-generated) code for the example above:

public static class StaticExtender : IExtensionContract 
// "inheriting" from interface here is my suggested addition to the language
{
public static Foo(this ISomething s){...}

public static Bar(this ISomething s){...}
}

public interface IExtensionContract
{
void Foo();
void Bar();
}

Finally, all the suggested logic I have blogged about previously still applies. The compiler still needs to decide, at compile time, whether the call is to a method in the extension contract or to a like-named (but otherwise unrelated) instance method. All that logic still applies as I described it here.

Readability in Validation

noreply@blogger.com (John Rusk) — Sun, 13 Jan 2008 00:59:00 +0000

My favourite pattern for validation code is to have one method per validation rule. I think it makes it easier to read and manage the rules.

For instance, I like to be able to write something like this:

[Validate]
protected bool EndDateMustBeAfterStartDate()
{
return EndDate > StartDate;
}

I like to refine the pattern a little, in order to:

(a) Construct error messages based on the method names. That makes the code more concise and encourages good method names. E.g. the failure message from the rule above will be "End date must be after start date".
(b) Allow you to, alternatively, specify dynamically-generated failure messges from within the method body (see below)
(c) Easily specify if the rule is not applicable in a particular situation (again see below).

I've attached some sample code (with tests).

By the way, I once wrote some code to fire rules automatically, whenever a property used by the rule was changed. But that's not included here. In this sample, you have to ask the system explictly to run the rules at a particular time.

Here are some example rules, and the code to call them:

public class Animal
{
...
public string AnimalName
{
get { return _animalName; }
set { _animalName = value; }
}

public int NumberOfLegs
{
get { return _numberOfLegs; }
set { _numberOfLegs = value; }
}

[Validate]
protected Result NumberOfLegsMustBeEven()  
{
return NumberOfLegs % 2 == 0;
}

[Validate]
protected Result MythicalSpeciesAreNotAllowed()
{
if (AnimalName == "Unicorn" || AnimalName == "Dragon")
return Validation.Error("'" + AnimalName + "' is not a real species");
else
return Validation.NotApplicable;
}  

// Note the use of the OR operator here to give a compact syntax for returning a custom error
[Validate]
protected Result NumberOfLegsMustBePlausible()
{
return NumberOfLegs <= 10 || Validation.Error("Your '" + AnimalName +  "' has too many legs");
} 

...
// execute like this
foreach(Result r in Validator<Animal>.ExecuteRulesOn(myAnimal))
if(!r.Pass)

// or, like this, if you don't want to use the generic Validator class
Validator v = Validator.GetInstanceFor(typeof(Animal));
foreach (Result r in v.ExecuteRulesOn(myAnimal))
...