Py's Sty

Quick tip, Respecting your own contract.

2012-01-19T21:42:00.001+10:00

I wish C# was smart enough to figure this out by itself, perhaps it is but I haven't found out how to get it to cooperate. I like using interfaces as a contract for how code should behave. I also default to using explicit interface implementation to enforce this contract. Where this gets a little annoying is when adopting a DNRY attitude and trying to reference an interface implemented method/property from within the same class.

Ultimately there are two approaches:
#1. Interface implementations call private methods. This is pure enough but I really don't like duplicating method signatures, one for the interface, and a second private method that does the work.
----
public class MyClass : IMyInterface
{
IEnumerable IMyInterface.LookupMyNames()
{
return LookupMyNames();
}
private IEnumerable LookupMyNames()
{
// Do stuff....
}
string IMyInterface.GetCombinedName()
{
var names = LookupMyNames();
// Do stuff...
}
----
#2. Access the implementation by casting "this" to the interface. This avoids the extra method declarations but frankly looks hideous.
----

public class MyClass : IMyInterface
{
IEnumerable IMyInterface.LookupMyNames()
{
// Do stuff....
}
string IMyInterface.GetCombinedName()
{
var names = ((IMyInterface)this).LookupMyNames();
// Do stuff...
}
----
Another option that's a bit easier on the eyes in my mind is to use a private property to expose the class to itself through its interface.
----
public class MyClass : IMyInterface
{
private IMyInterface This
{
get{ return (IMyInterface)this;}
}
IEnumerable IMyInterface.LookupMyNames()
{
// Do stuff....
}
string IMyInterface.GetCombinedName()
{
var names = This.LookupMyNames();
// Do stuff...
}
----

Obviously this only works well in cases where a class is primarily interested in implementing a single interface, however when using contracts in this way this is almost exclusively the case. You can of course expose a property for each interface using some form of naming convention. I used "This" since it has meaning, but perhaps "AsIMyInterface" is your taste....

Refreshing Settings and dll.config Files

2011-10-12T21:33:00.002+10:00

Having worked on plugin-based architecture and projects involving a number of assemblies there has always been a bit of a bad smell to .Net's implementation for configuration settings files; In particular, the use of Settings.Default. I really like how Settings.Default wraps configuration settings presenting them as strongly typed values. What I hate about it is how it uses the DefaultSettingValueAttribute to specify the default value in the event that a setting cannot be loaded. Why is Settings.Default a problem?

1) Because it hides problems with configuration. Essentially it means if you don't have a configuration setting specified, it will use the attributed value by default. This can lead to spitting venom at the computer as you try to spot a type-o in a the configuration XML six months after the fact when that mail server name changes and you realize it's been using the attributed value and absolutely refuses to accept your configuration setting.

2) Because at runtime you may want to have a service running and be able to change configuration settings and have the service pick up those changes. Unfortunately you're out of luck because while user settings can be reloaded, application settings cannot. The service needs to be stopped and restarted to accept the new configuration settings.

3) Because in cases where you have support assemblies in your project, the pain comes in that while it might be nice and sensible to have each assembly's settings located in their .dll.config file (which Visual Studio produces for you) by default, Settings.Default will only look at settings in the calling application's .exe.config file, even if you place the .dll.config file in the runtime folder. This means if you have 30 configuration settings across six assemblies, you're manually copying across 6 configuration sections and 30 settings into the app's .exe.config file. Now you may be perfectly able to build an NAnt task to do this, but it leaves you with a painfully large and convoluted configuration mess in the .exe.config file.

Now all of these problems can be avoided if you choose to load and parse configuration settings yourself using ConfigurationManager and the like. However, you lose that nice encapsulation that Settings does give you. There may also be solutions on the web for one or more of these issues, but I never was able to find one.

So I set about to change that.

Source Code

SettingsExtension.cs is an extension method you can utilize to refresh any Settings instance from it's assembly's .config file. This means if you have a DLL containing a Settings & associated app.config section and deploy the .dll.config file, you can call the Refresh() method to retrieve the values at any time from the .dll.config file. It works for .exe.config files as well, however calling it on a DLL's Settings will not refresh from .exe.config settings if you've modified settings there. The DLL only discovers its own configuration file.

What this means is that if you have a project with an EXE and X number of supporting DLLs, and some of those DLLs want to use configuration settings, you can copy the .dll.config files into your deployment and as long as they have a call to Settings.Default.Refresh() (I.e. from a static Constructor) you can ensure that that DLL will always use the values from its .dll.config file.

Additionally if you want to get creative, you can now set up a File Watcher on a .exe.config file (or .dll.config files) and when it detects a change, call the .Refresh() on the Settings.Default instance to reload those changes.

Looking through the code it should be pretty easy to see what it's doing. It looks for the Setting's assembly's config file, applicationSettings section group, and section for settings, then uses reflection to go through all known settings properties and update them with whatever is found in the file.

An added feature is support for an optional Refresh Listener which the refresh method can communicate back statuses about what it is processing and if it encounters any errors. Exceptions are non-fatal, the tool will report them back and continue. The reporting includes a message and a TraceLevel as an indication of what went wrong. Simply register a method using SettingsExtensions.InitializeListener() method in order to receive messages.

I've tested the implementation with most Settings-supported data types but there might be a few that need some special parsing.

Something's fishy with Google...

2011-10-05T20:37:00.000+10:00

It might be time to start lining my room with aluminium foil but something very strange has been happening relating to when I try to install Google software on my machine. This is a fairly new machine running Windows 7 Pro. I'm anything but a MS Fanboi but being a MS technology stack developer has paid the bills quite nicely over the years so who am I to start bitching. One thing I do not like is IE, so as soon as the system was up, I installed Firefox and uninstalled IE. This led to some rather annoying but laughable problems like when applications cannot seem to figure out that I don't have IE, they try launching HTML files in MS Word.

A while back I had a play with Chrome on the office laptop and decided to try it at home. All went well until I started noticing that my Firefox browser would hang, and then Windows 7 would completely hang just out of the blue when browsing. (Not using Chrome at the time) This started to get really annoying because task manager etc. didn't even respond. I started digging around and noticed a few other people having trouble with Chrome and Firefox playing nice together on Windows 7 / Vista. I uninstalled Chrome and the problem vanished. For the record I do not use any Google apps such as toolbar etc. either. I passed it off for some weird glitch between the two systems and something with Win 7 falling over after not figuring out how to resolve their differences. Disappointing to say the least, but I prefer Firefox as a browser to Chrome.

However, recently I decided to dust off my old Java books, install the latest Eclipse, and have a peek at the Android SDK. All was fine until I launched the Android SDK Manager then decided to do a bit of browsing while it was downloading my selected SDK versions and such. Within 10 min Firefox had hung again and the system went completely dead!

So really now, Google, What the Farq are you installing with your software that is getting completely shirty with Firefox and/or Windows 7?! My guess is that the SDK downloader is using netcode shared with Chrome that is in some way incompatible with whatever Firefox is using. Or is it something more sinister? While I have found some references of other people having similar problems I haven't seen anything that remotely looks like a solution or acknowledgement that there is a problem.

The question is, if I sacrifice my (relatively minor) preference for Firefox in favour of Chrome, will I be able to browse and utilize the Android SDK without having my machine lock up? And more importantly, will I wake up the next morning without finding a completely new preference shift from the soda I drink to the car I drive due to some form of wifi-induced subliminal suggestion? :)

What is my client's/employer's IP?

2011-09-17T14:05:00.003+10:00

This topic has boiled up recently on Ayende's blog here and here relating to providing samples from pet projects. Several people have commented on how some contracts are laid out to state that any work an employee does, whether at work or at home is automatically the IP of the company.

Disclaimer: I am not a lawyer. This subject has intrigued me over the years and I have studied what information I could find applicable to Canada and Australia on the subject.

The answer to this question depends entirely on the country and/or state/province the contract was formed. Generally, any unresolvable IP dispute between employer and employee/ex-employee will need to be brought up in court because there are a large number of factors to consider beyond terms in the original contract. Generally, most edge-cases would rarely ever see a courtroom unless there was a clear breach of proprietary code (such as making similar code available to competitors or a developer choosing to become a competitor) or other breaches of contract. (Such as employees that try to keep the cream of their work to themselves, and deliver crap to their employer or try to extort employers for more money for "better" solutions they claim to have developed on their own time.)

Things that a court will factor in: The history of the development of the IP. Specific terms within the contract relating to IP. The seniority or responsibilities of the employee. How the IP relates to the core business of the employer.

A sticky point between employees and employers is often the contract. I've often seen IP ownership clauses that state something as broad as "all IP created by the employee during the course of employment is unconditionally transferred to the employer." This is a very broad statement that can be interpreted a number of ways. (Hence the need to involve lawyers & case history) "during the course of employment" could be considered to mean "during work hours", or it could be considered to mean "from the moment you sign the contract until the contract ends." The best course of action an employee can take is to *not* sign such a contract until the terms are clarified and the clause is amended. In my experience I have seen broad clauses like this in most contracts I have been presented. As someone that works on a number of side projects this is the first section I look at when presented with a new contract. Usually the client has no problems clarifying the contract as the clause is a common part of the contract boilerplate that their lawyer provides. Own its own, a clause like this is generally unenforceable as a court will likely decide that the role of the employee, and the compensation awarded to the employee, form a restraint of trade on the employee.

Two hypothetical scenarios to consider these factors:
A junior software employee signs on with a company, and gets delegated minor tasks such as commenting code, testing, and some trivial work in a .Net shop. He have an interest in learning technologies such as ASP.Net MVC and developing websites on their own and while tinkering on their own time and development license, come up with an interesting website/service that he hosts himself. If his contract had such a general IP ownership and their employer attempted to seize the copyright / IP for that website it would likely be defeated in court. The employee's level of responsibility and the distinct difference in function between what the employee contributes in the course of employment would show that the employer would have no grounds to claim the IP. An employer cannot claim copyright over a fiction book you write on your own time, or that wicked recipe for Chili Con Carne. They cannot claim IP you work on that is unrelated to the type of work and technologies they employ you for.

A senior software employee signs on to work with a company building a workflow engine and designer interface for the company's line of products. After a couple years of working on the system he decides that he can build a better workflow engine and/or designer himself and sets out to write one on from scratch on his own time and resources. Under a similar IP agreement an employer challenge to the IP would likely be upheld in court. The employee has applied intellectual property in the form of design and implementation that was created or learned in the course of their employment into a product that directly falls in line with their employer's line of work. Another similar example which would likely fall under the same sort of principles would be if an employer chose to scrap a project and an employee decided it still had merit and began developing it on their own.

But the contract alone is not the only consideration. The history of the IP is considered, as in who came up with the idea and when. How much of the idea was developed at work or using company resources. (Technically this can include discussing the IP with other employees.) Generally it is a good idea to keep a log of when you are working on IP and avoid involving any company equipment / software licenses, or discussing the project features with other employees during work hours. Using company equipment or doing work on the IP during paid company hours is a strong determining factor into IP settlements in the company's favour. However, in cases where the contract does not state the company is entitled to IP of employees, this often doesn't automatically grant IP to the employer.

Cases where a contract is quite explicit that creations outside of work hours are considered the IP of the company would be enforceable in court. However, the other mitigating considerations would still apply, and these would generally be reserved for very high-level roles such as application architects. Companies attempting to use conditions such as these on relatively low responsibility and low paying jobs can find these contract clauses unenforceable.

What about scenarios where your contract allows you to keep IP on stuff you do on your own time, and you come up with something that would be of use to the company? This would be a scenario where you need to tread carefully. If the project is something not directly related to the work you are doing (such as a timesheet tracker, or maybe a simple queuing system for the integration environment, etc.) then the best thing to do before utilizing it at work is requesting permission and granting a license to the company to use your product. (whether you grant it for free, or they agree to a price.) This way it is clearly defined that they have no entitlement to the IP. You're better off not attempting to build something related to the business on your own time without prior explicit written permission from the company that you can spend your own time on a given complimentary application. (I.e. an add-on, diagnostic tool, etc.)

Regarding specifically to some of the comments on Ayende's blog around people that don't have time or energy outside of work, but do build plenty of spikes and such while at work. These generally always fall under company IP and the availability of using this code for your own benefit will depend entirely on your employer. Many employers will grant you permission to use spike code for your own work or to submit on something like CodeProject if you ask, as long as it's not business-specific. If you do intend to use such code as samples for securing a new contract or role with another company, then it would be a good idea to think ahead and obtain permission for interesting bits as they come up, not the week you're expecting to be going in for interviews. :) Resources like CodeProject are fantastic for this sort of thing because it gives you a valid, innocent reason to ask to use the code, and you can point potential employers at your CodeProject profile for examples of your work. (Which is kudos if you can get some complimentary comments associated with it.)

Windows 8, Vista All Over Again.

2011-09-16T22:33:00.002+10:00

Around the time Vista was released, I had a dilema. I had started looking at Vista and frankly I hated it. I found it slow, bulky, loathed the UAC. It was just Windows me but more annoying and clumsy. I was determined never to install that OS, and I didn't want to do software development on that OS so I stayed with XP and Visual Studio until I was either forced to adopt it or Microsoft saw their blunder and corrected it for their next Windows release. I resolved that if the next version of Windows wasn't significantly better I'd be switching camp to Linux and Java/C++ development or maybe something like Ruby. Fortunately Windows 7 proved to be much better, and while I'll admit I still prefer XP, it is responsive enough to be bearable, I just wish it was a bit more stable. In the 5 months I've been running it, 7 has crashed 12 times. (and I mean frozen, no BSOD, but needing a hard boot) In the nearly 10 years I've been running XP I've seen a BSOD perhaps a handful of times and have never needed a hard boot.

Well I got a play with the developer release for Windows 8 today, and while I acknowledge it's probably quite rough around the edges, what I did see of it has me 10x more worried than Vista ever did.

To be blunt and completely honest, it is a steaming pile of shit. Maybe if you want to run it on a mobile or tablet and you're used to Win7 Mobile or iOS it looks perfectly normal, but for a desktop it is a complete and utter disgrace. The Metro desktop looks like a VB6 graphical menu I built for an application over 10 years ago. I found no obvious way to actually figure out what applications were installed via the metro interface, aside from blindly typing words into the Search feature. IE10 metro-mode was a complete joke. Just selecting the address bar blanked the whole flipping screen to show me suggestions, and I couldn't get back to my previous page unless I hit the refresh button. No hotkey support of any kind was present (I.e. Esc, Win-M or Alt-F4) other than Alt-Tab (thank goodness) I farted around in the UI and couldn't find any way to actually CLOSE an application aside from going back to the classic mode and bringing up Task Manager.

And what the hell were they thinking with the term "Metro" anyways? Are they expecting the name is going to appeal to the Metrosexuals of the world enough to convince those yuppy toy-fondling wankers to drop their iPhones, iPads, and Mac Airs?

At first I felt a little more at home when I fired up the classic mode, but that joy was very short-lived when the Windows button dragged me right back to that cursed Metro home screen. I tried tinkering around with settings to see if I could have SOME form of control over how this piece of crap was behaving. What few options actually came up in Metro appeared in some form of side-bar which didn't have any form of "Ok" or commit button and only closed by launching something via the Metro screen. Even then it didn't remember my settings because opening it again just showed the original values. Selecting the Control panel dumped me back to the classic mode screen with a familiar looking control panel, but I didn't see any options for the Metro behaviour.

Frankly by this point I seriously gave up on it. I'll take a look when a more formal pre-release is available but at this point it's looking far-far worse than Vista did. As a Silverlight developer I'm absolutely steaming that support has been dropped from what they consider their default OS mode. IE's not even coordinated enough to detect plugins already installed in the classic mode. It dilligently downloads the Silverlight installer then switches to classic mode to install it... after I've already installed it in classic mode. It used to amuse me when I'd hear people talk about how more and more PC functions would be taken over by tablets and mobile devices, but there's only so much these devices can accomplish beyond viewing web pages & PDFs, writing 1-liner e-mails, and flinging various birds at pigs in wooden houses. It doesn't surprise me that Microsoft would be looking to compete in this trendy-driven arena, but that they'd castrate virtually all forms of usability on desktops to do it?!

I hope that this was just a really rough cast of what a small part of Windows 8 will be. I wish I'd never even seen it because if it's not, all I can hope is that I'm going to wake up and this was all just a really bad dream... *pinch!*..... damn.

A recent example of the value and purpose of unit tests.

2011-09-11T20:55:00.001+10:00

Just last week I was approached by our BA(-ish roled individual) who was tasked with documenting our testing process, and in particular our unit testing strategy. He wanted to know what our unit tests did, and how he could express their value to the client. I had just finished identifying an interesting issue that was identified as I wrote unit tests to exercise a recent feature, and it made the perfect example of what the unit test suite did, and how it added value to the project.

A bit of background into the issue:
The business requirements were that a service would be contacted to retrieve updated events for an order. All events that weren't already associated with the order would be associated with the order. The application would report on any new events associated with the order.
*events have no distinct Key from the service, they are matched up by date/time and event code.

The code essentially did the following. *returning IEnumerable

var recentEvents = EventService.GetEventsForOrder(order.Key);

var newEvents = recentEvents.Except(order.Events);

foreach (var newEvent in newEvents)

{

    order.Events.Add(newEvent);

}

return newEvents;

Based on the requirements I scoped out several tests. Ensure that:
Given the event service returns no events, no new events are added or returned.
Given the event service returns new events and the order contains no events, all events are added and returned.
Given the event service returns new events and the order contains one with matching code and date/time, only the new unmatched events are added to the order and returned.
Given the event service returns an event with matching code but different date time, the new event is added to the order without replacing the existing event and returned.

Adding the 2nd test I was suprised to find it failed. The code is de-coupled and the EventService is mocked out to return two events. There were 2 main asserts, one to assert the order contained the new events, and the second that the return value contained the new events. The order assert showed it contained the two new orders, however the returned set had a count of 0. I'm actually pleased when a test does fail, but this was a momentary "WTF?" followed shortly by a "well Duh!". I was returning the IEnumerable from the Except() expression and I had added the items to the list driving the Except; any further iteration of the IEnumerable would see no items since matches were now in the list.

The issue was easy enough to fix with a ToList() call, and I felt it warranted a simple comment to explain why the fixed code was done that way in case someone in there later went and tried re-factoring it out to just use the enumerable.

This gave me a perfect situation to demonstrate to the BA exactly how the unit tests reflected requirements within the application, and how they served to guard those requirements from unexpected future changes. I had integrated the two tests with working code to show how the CI environment reported that all tests pass, then re-introduced the buggy assumption to show how the tests pick up the error.

The other interesting thing to note was that I also showed the BA our NCover results and was a bit surprised to see that with just the first two unit tests that block of logic was reporting back a test coverage of 100%. However I showed that the "touch" count was showing a lot of 1's indicating that a single test was touching many parts of the code only 1 time. This is a warning flag to me that the code really isn't being exercised . If I was solely concerned with test coverage percentage I could have left the tests at that, but I knew the other two scenarios were not represented so I demonstrated how the stats changed by adding them. The test coverage remained 100%, however the touch counts increased to show a minimum of 3, and an average touch count of 6 to 9. It's not a perfectly reliable indication that the coverage really is complete, but by thinking through the scenarios to exercise the code, I'm quite confident that the code is reasonably guarded from the unintentional introduction of bugs.

Unit Tests Aren't Expensive

2011-09-07T20:55:00.002+10:00

A short time after developers start adopting unit tests, by this I'm meaning committing to unit tests and looking at good test coverage, grumblings will start. Someone will want to make a design change, and by doing so they break a large number of existing tests. All of the sudden it sinks in that now that there are unit tests in addition to existing production code, changes like this look twice as big or bigger. This can be a crisis point for a project. Having unit tests appears to mean that code is less flexible and change is significantly more expensive. This attitude couldn't be farther from the truth. Unit tests aren't expensive. Change is expensive.

The first thing we need to look at is why the unit tests are breaking. Often when developers start writing tests the tests can be overly brittle merely due to inexperience. In these cases it is a learning experience and hopefully the fixed up tests will be a little more robust than they were before. Still, even with well sculpted unit tests guarding the code's behaviour, developers will find reasons to justify changing that behaviour. The presence of unit tests, now broken unit tests, doesn't make that change expensive, it just makes the true cost of that change visible. Making behaviour changes to an existing code base (that has already committed to some form of acceptance testing, and likely is already out in production) is a very costly exercise. Breaking unit tests are demonstrating what the cost of that change is up-front. Without the unit tests, the deposit you pay up front for the change is smaller, but the rest of the cost is charged with interest in the forms of risk and technical debt. Unit tests show you exactly what is dependent on the behaviour as it was when you first committed to it. You have a clear map not only to fix the tests, but ensure that you can fix the code documented by the tests to suit the desired new behaviour. Without that map, you're driving blind and only regression tests are going to find the spots you miss if you're careful, or it will be your customers that find them if you're not.

So, how do you keep the cost of change under control? An obvious answer would be to identify all change before coding starts, and allowing no change until coding finishes. This would be a motto for BDUF (Big Design Up Front) advocates, but in the realm of typical business software this really doesn't work. By the time coding starts and the customer actually sees implementation they are already thinking about changes. The Agile solution is to implement only the minimum functionality you need to meet requirements as you go. Often the biggest contributor to these kinds of large behaviour changes are due to development teams trying to save time in the future by implementing code that isn't necessary now. Classic reasons include over-engineering solutions, scope creep, or committing to a particular technology/framework on the basis that it may solve problems you might encounter, just not right now. Other ways to avoid large change costs is by embracing constant re-factoring. Get in the habbit of including small improvements into your code as you implement new features. Introduce base classes for common code, delete stale code with a vengeance. Any improvement to reduce the overall size of the code base should be a very high priority. Don't leave things for later, because that is debt that starts accumulating interest as new functionality comes to depend on it, or reproduces it for the sake of consistency.

On a side note:
Writing software can be like a game of chess. You don't walk up to a match with a memorized set of moves that will guarantee victory. This isn't tic-tac-toe. You've got to be prepared to adapt to the opponent that is your customer, your manager, your budget; and you've got to be prepared to sacrifice pieces in the hopes of winning the game. A developer with over-engineered code is like a player that values each piece on the board too highly. A lot of time and sweat has been spent in the code and you cannot justify sacrificing it when the demands of the game deviate from the original plan. The result is putting even more sweat and time into the solution and the resulting choice of moves doesn't win you the game, they lose it and the customer walks away.

How XP can help your solo projects too.

2011-08-28T22:04:00.000+10:00

Something has irked me for a while. Solo projects have a rather unique set of problems, at least from my perspective, compared with client projects:
Time- Solo projects are developed in the gaps that you can scrape between the hours you work for a client, and personal life distractions. When you get married and have children, this becomes quite difficult, though I had found plenty of other distractions before that happened. :)

Scope- On client projects it's easy to fight against scope creep, and easy to spot & discuss "kitchen sink" type features or architecture choices being considered today for some mythical benefit tomorrow. Generally I can be quite "lazy" in that I don't want to do more work than I have to, and it saves me headaches in the future. In solo projects I've found it quite a bit harder to fight scope creep as there were no hard targets for time or feature list.
Requirements- In client projects, you either don't have enough requirements and easily have someone to moan to/about getting more detail from, or you have people making attempts to provide too much in the way of requirements up front. With solo projects I am responsible for the requirements, and deciding what will be v1.0 and sticking with that (or at least challenging my wayward heart) is quite difficult. I also HATE writing down requirements. I can't get through more than a few features before I catch myself opening Visual Studio!

One thing that sold me on XP was measures it took in quality. Not only in terms of software quality with unit testing and pair programming, but in terms of systems quality with user stories, planning games, and continuous integration. It made dealing with customers much easier. It let the production of features begin much sooner, increased value in what was being developed, and made cost transparent to the customer. They can see the velocity of the project and value added for extra time invested in getting features just right. These processes have been really effective in dealing with cases where the customer was really "willy-nilly" with their requirements. Then I realized, *I'm* that willy-nilly customer! Why can't I apply XP principles to my own projects?

So the first thing I did is stop worrying about requirements. For some reason I was trying to capture more detail on paper for my own stuff then I try and initially capture from clients for their features. I switched instead to point-form lists, then expanded the most important one into a user story and tasks as I went. I am the customer, or at least the B.A. so I'm a perfect XP customer because I'm accessible 24/7. I also try to distinctly switch hats from developer to customer. (If I don't end up bi-polar by the end of this, both of me will be surprised.) As a customer I let myself loose with the "wouldn't this be cool" but NEVER with the computer running. That stuff goes down on paper. As the developer I try to be as lazy as I can. The main change from the process was devoting more page area for notes as I progressed. I'm working 1-2 hours at a time if I'm lucky, and maybe 2-3 days in a row. I try to get blocks of work done, while noting down what I had in mind for the next blocks.

TDD I'm already a very strong supporter in, whether Test-First or Test-Second. My rules for solo project is that the code for a new task does not get written until the previous task is unit tested. As I'll be working on these projects for some time, and hopefully plan to get other developers on-board with them in the future, unit tests are crucial. The code must always build, run, and the test suite pass before I finish for a day. (That one can irk the wife! :)

It's still early days for applying XP to my current project stack but it has been quite successful so far. If anything for keeping me more focussed on getting chunks of value-added work done. Hopefully if I can keep this up for a month I can work out a system to keep the momentum going without getting caught up. At that point I can bring in at least one other developer to contribute to the projects without wasting their time.

Pair Programming.

2011-08-21T15:48:00.000+10:00

I am an XP (Extreme Programming) advocate. As a whole I've used in on one successful project, and I try to bring elements of it to any client I work for. Pair programming has to be one of the toughest elements to sell, though admittedly I don't pitch it as the most valuable one. I'm sure many XP advocates would cry foul, as pair programming a cornerstone, if not the foundation behind XP. Every other element could easily be discarded by a developer working alone, pair programming helps reinforce that the other elements are followed. I certainly do not disagree.

However, pair programming is the hardest element to get in place. Most clients have existing development teams that have never heard of pair programming and development environments that aren't set up for it. In the project I started with XP this second point was quite an obstacle. We had comfortable cubicle environments but not shaped to fit two chairs with people working side-by-side. Fortunately it was a big office and we confiscated a large meeting room, arranged the tables in a large rectangle so that 4 pairs of developers could sit side-by-side. This went on for about 3 months, but the company wanted their meeting room back, and we were also getting other teams eyeing our original cubicles.

The solution we came up with was Pair Analysis + Task Swap. Each developer was paired with another and both selected a set of stories for the iteration. The pair would sit together to scope out the tasks for all of their combined stories and discuss the approach. Then the actual development was done individually. If during implementation a developer thought a deviation was needed, then they discussed it with their pair. As a task on a story was completed, it was handed over to the other developer. Each developer would not only review the other's work, but see if there were ways to improve it, or look for other possible scenarios that hadn't been thought of, discussing together as necessary before they committed the work. This was a little painful back when working with a repository that did not support branches. Essentially developers had shares to their development folder that their pair would open to review work. Done again today with a branching repository, each task would be developed against a story branch.

From a technical perspective, how this works:

Developers A & B select one story each. (typically in an iteration they would choose 1~3 each) The development would be done on branches A & B respectively. When developer A completes his work for a task, he pings developer B for a review. Hopefully B will be finishing up a task pretty soon as well, but while waiting for something to review, developer A can continue with tasks in an unrelated section of code. Developer B reviews A's changes on Branch A, while A review's B's changes on Branch B. When both are satisfied of the changes, developer A merges the changes from Branch B and performs an integration confirmation then developer B merges from Branch A and performs the integration confirmation. After the integrations, each developer resumes working in their original branch until their story is complete. A branch only lives as long as a story. If developer B finishes all tasks of their story, developer A will have merged all changes into Trunk, so developer B will confirm the end of the story and terminate the branch. A new branch will be taken off $trunk to start the next story.

Physically during reviews there is a lot of chatter between the two developers and they'll often be at each other's desks when going over code. Sometimes one developer will find an optimization that the other hadn't thought of. There are two options available, either he can pass the task back to the original developer, or take the task on himself, and the other developer can start working on a task on the alternate story that he just finished reviewing. (Swapping stories/branches.) You can even consider encoraging developers swap branches/stories every other task or so.

This approach has trade-offs with pure pair programming. In situations where a lot of re-factoring is found, such as in cases where you're pairing up experienced developers with less experienced developers, pair programming would be more efficient. A pair working together will spot these optimizations as they're going, where this situations leads to work getting re-factored during review. However, the efficiency hit should reduce greatly as the lesser experienced developer experience and exposure to reviewing the other developer's work increases. This swapping is done at the task frequency, not story frequency, so the impact of these re-factorring should be kept quite small. The advantage of this approach is that it can easily be applied in physical environments that make pairing difficult, while maintaining most of the benefit of pair programming. (knowledge sharing, near real-time code review, and emphasis to adhere to the rest of the principles.) It's also a good lead-in to pair programming with developers that find the concept a bit alien. Hopefully it fosters an increasing amount of communication between team members, so much so that they find that the time apart is the wasteful part of the job and end up pushing their desks together themselves. :)

Business Analysts in the mist.

2011-08-21T14:51:00.000+10:00

One thing I've noticed since moving to Australia is the lack of business analysts within organizations I've worked at, or worked for. Occasionally a client will have someone who's title is BA, but the normal responses I get when asking if they have a business analyst is either "No" or "Not now, but we plan to hire one." Now in Canada, this would be sending little red flags flying, but it seems to be par for the course at least here in Brisbane. What really sends the alarm bells clanging is when I ask "who defines the requirements then?" If the answer isn't a B.A. or a client then prepare for pain. Usually the answer is either "The Developers/Lead Developer" or "Sales."

Developers generally makes for very poor analysts. Developers are technical, they don't grok business process, only software process. A developer can analyse how something should be done, but not what should be done. Salespeople are often even worse. Salespeople only worry about signing on new customers or upgrading existing ones. They have an excellent perspective on the extreme high-level of what should be done, but they don't understand either from a business perspective or a technical perspective how it could be done.

The best person to define requirements is the client. Now if you're fortunate enough to be using a methodology like Extreme Programming and the client has someone valuable embedded in your team, then there's no real need for a BA. However, the next best thing is a dedicated BA. Now this gets back to businesses that have someone who's title is BA, but isn't a BA. An excellent example of this was a large government organization client. When I asked if they had a business analyst, their response was that they had a whole team of business analysts! What they said was true, it was actually a small department of about 8 BAs that were under the same director, but not actually part of the software development department. Their idea of a BA's role was to go and meet with the client, understand their business processes, write it up in a document, and hand it off to the software developers to build, washing their hands of it. This meant that the developers had a document, some months after a project has started, then if there are issues or clarifications needed, well too bad, lodge a request with the BA department to get the documentation adjusted. Often the BA that did the original work was tied up in a new project so you get a new BA that has no idea about the project. Of course, this model made perfect sense to them. They needed to have a billable block of time to charge back to the client. Once that was done, they needed to be charging other clients. This was not only frustrating from the software development side of things, it drove the clients nuts. (Having to explain the same thing to two or more BA's, and being expected to *pay* for the new BA to get up to speed with the project.)

Most often what businesses call a BA is essentially nothing more than a clerk. Write down requirements so that we effectively have a contract that we can associate a dollar figure to and get a sign-off on. But what a BA is should be so much more than that. In Canadian teams where I've worked with properly embedded BAs, the BA was effectively a conduit to the client. Even within an Extreme Programming project where the client was in another province, the BA proxied for the client when the client couldn't send someone to our office. If we weren't sure about something, we asked the BA. If they weren't sure, it was their job to get in touch with the client and sort it out. They had the business knowledge, and were abreast of the technical details of how the software was being implemented. They were instrumental in giving initial feedback for UAT phases. In short, they started the project, and they finished the project. In XP the BA was the tracker and the client.

So if anyone around Brisbane works for a client/company that has a BA similar to what I describe above, count yourself as lucky and let me know. I'd like to get my picture taken with them because I'm thinking they must be rarer here than dropbears. :)

Business software should advise

2011-08-17T16:19:00.000+10:00

Business software should be an advisor to users, not a dictator above them. Business software boils down to one thing: Business Rules. How business rules are implemented in software is just as important as the rules are themselves. Some applications seek to "enforce" business rules by restricting behaviour until certain information is provided, or sequence of steps have been followed. There are definitely cases where this must certainly be expected to be the case, such as enforcing authentication and authorization to features. However, when applied to business logic this kind of enforcement leads to inflexible and, in some cases, very costly issues.

People generally like dictators at first. Life is pain, they need someone to stimulate the economy, get the trains running on time. When a software system is first designed, the idea of enforcing rules to save time and minimize mistakes is certainly attractive. Unfortunately, software has to evolve as business requirements evolve, and before you know it, your wonderful software application has divided Poland with Microsoft Office, and invaded Czechoslovakia. All you wanted was a system that would bring the efficiency back into your organization, but pretty soon you have a behemoth costing you hours of time, stacks of bug reports, and your business is failing to serve its customer base which is costing you customers.

Enforcing business rules restricts flexibility. Rather than designing software to be an enforcer, design it solely to be a time saver. Ensure that the only mandatory fields are things that ARE mandatory, and if the system can default a bunch of other optional fields, then fine. Someone can always change the values later. Also accept that a certain amount of business logic is best left in people's heads and hearts. Sure, you could define rules, even strive to make them configurable, but in the end keeping the most flexible business rules outside of software is sometimes the best option.

A perfect example of this was brought up today when one developer was querying another about a legacy system for manufacturing. The question was that when orders are received, different products take different amounts of time to manufacture. How does the factory worker know when they need to manufacture each product to get it done by the dispatch date? The answer was, "The floor supervisor decides what to manufacture to ensure everything is done on time." This is based on a report with the various orders and their respective dispatch due dates. It took a few rounds of questions to let this fact sink in. The software system didn't tell them when to manufacture the product, it simply told them what needed to be manufactured and by when, and a person would need to determine what should be done first.

There are an assortment of rules that govern when products get manufactured, and a *lot* of environmental variables involved. I'm sure the first thoughts of this developer would have been along the lines that the rules could be codified so that the software system could calculate and dispatch out work to the factory floor so that products are manufactured by their dispatch date. This would be more efficient and could probably mean they could accept more work or work with fewer staff. But the problem is that you cannot hope to codify *all* variables that are accounted for in the decisions to get work done. Staff being sick or on vacation, whups, need a rostering component. Machine break-downs or services, ah, incident tracking. Last minute order changes, cancellations, or changes in priorities. Stock shortages or quality issues. Issues that can crop up that haven't even happened before. Machines follow very clear an concise rules very effectively, but they cannot adapt to unknown variables like a human being can.

The result of leaving a good chunk of the business rules for producing the product in a person's head means that the actual mechanical work of getting product produced is completely dynamic. The machine simply advises what needs to be done, can make suggestions based on information it can compile, and records the results of the production. If the machine goes down, the data can still be queried, the product produced. The machine is not relied upon, it can be updated once it sorts itself out.

How to hire a "Senior Software Developer".

2011-08-15T21:16:00.000+10:00

It is hiring time again. The recruitment agents start instinctively calling, smelling the scent of a commission, and pretty soon the resume's start flowing in. The local I.T. market is pretty restricted right now. Initially the client wanted to hire two permanents, (Expectedly to phase me, the final contractor, out) but has had to settle on 1 perm and 1 new contractor because we have a load of work coming down the pipe. Since we're looking to hire people that are expected to hit the ground running in the project these roles were for highly experienced developers that would be familiar with working with things like TDD, IOC/DI, and be able to write loosely coupled, "clean" code. One of my tasks was to put together a coding quiz of sorts to give to successful candidates. Nothing really difficult, but something to give them a taste of what we're looking for and something to give us an idea of how they understand requirements and complete a task. The quiz consisted of a set of requirements with some general instructions on what the sample project already contained, and what they would need to complete. The project they were given consisted of a small number of interfaces for dependencies that would provide much of the additional functionality that they would utilize to meet the requirements. (I.e. data retrieval, sending e-mails etc.) The quiz was expected to take about 2-3 hours to complete including unit tests. (Tested with one of our junior developers.)

Part of the interesting element of this quiz was identified when our developer gave it a test run. There were a few holes in the quiz. For instance a requirement would reference an expiry date on an existing domain object which did not exist. The test implies that elements that their specific requirements were being provided by other team members, so issues like this were left in to see how candidates responded to such problems. (Ignored, noted with ToDos, or brought up when they submitted their response.)

The reason I chose to create a template project with the quiz was to give candidates hints into the way we develop code, and the kinds of things we are looking for. Do they understand how to do dependency injection for their service class based on the interfaces we've provided, or do they use the IOC Container like a global Factory/Registry? Do they try to follow our naming convention and style? Even with the requirements I dropped hints about TDD/BDD and mocking.

The results of the quiz were even better than we expected. Out of a total of 8 candidates, (2 perm applicants and 6 contractors) all with impressive looking resumes, the quiz made it very clear who knew their shiz from those that just continued to write poor quality, unmaintainable code with modern tools and libraries. The quiz probably worked a bit too well because of those 8 candidates, only one really stood out, and he was the one with the least experience. However, he was the only one that understood the concept of dependency injection, and while he had no experience directly with mocking frameworks, he picked up on it from the requirements, researched one, and tried applying it (reasonably well) within a sample unit test. His main failing was the amount of work for the time he spent, but the fact was he hit all of the requirements with clear markers where details still needed to be filled in.

What was really surprising was how bad some of the contractor submissions were. Only one actually met all of the requirements, but while his resume toted .Net 3.5 & 4.0, the code sample he wrote was effectively .Net 2.0. He had provided unit tests, but they were merely auto-generated and code-coverage results for his service was only 70% with much of it single-hit results. Most of the other contractors (these are guys in the same region as me with > 14 years of experience) either missed basic behaviour requirements, or had fairly severe logic errors present in their samples. One big one I looked for was a requirement that each operation in the service had an authorization requirement. Most of the submissions went and did the authorization check in the construction of the service rather than on each call. A risky assumption in any situation with a service, and in cases where this service was Singleton scoped in an IOC container (as something like this would commonly be) this would easily lead to embarrassing bugs. One candidate's project didn't even compile, while another attempted to write unit tests, which all failed when run.

So in the end we'll probably settle for 1 decent intermediate that hopefully has the enthusiasm to step up to the plate, and one contractor who delivered a working sample, but we hope was writing rather dated code out of habbit. (He's done a lot of work for government and their pace of adopting new technology I am well aware of.) The moral of the story is when looking to hire people, don't put too much credit in their resume, or whether they can recite namespaces for commonly used libraries. Build up a code sample project in the style you like with a set of reasonable requirements, and get them to write some code. Seperate the shiny nuggets from the vast slurry that call themselves "Senior Software Developers".

Pet Peeve: Misuse of KeyValuePair

2011-07-08T13:44:00.000+10:00

This is one that irks me when I come across it.

public IList<KeyValuePair<bool, string>> ProcessRecords(IList<MyRecord> records)
{
var results = new List<KeyValuePair<bool, string>>();
foreach( var record in records)
{
// Do some processing...
if (success)
results.Add(new KeyValuePair<bool, string>(true, "Message indicating record was processed successfully.");
else
results.Add(new KeyValuePair<bool, string>(false, "Message indicating record was not processed successfully.");
}
return results;
}

The above is just a pseudo-example similar to some situations I've come across and even some examples on the web on how you can use KeyValuePair, and even one that gave an example of a KeyValuePair of KeyValuePairs to return a bastardization of triplets. *shudder* The alternative to KeyValuePair would be to create a new class which by all rights would be identical to KeyValuePair. So why not just use KeyValuePair?

Because it is misleading, and it's no different to writing a class to represent a tax invoice and naming it "Order", or "Thing" for that matter. KeyValuePair is meant to store a Key, as in a unique value, against a value. If your method is designed to return a unique list of keys with respective values then by all means use KeyValuePair. But if you're using it to return arbitrary pairs of values then for clairity just create a Pair class instead.

The problem with returning KeyValuePairs is that looking at that return type you would expect that the data would be suited to being placed in a Dictionary. However if you're returning arbitrary pairs of values then you are misleading other developers all for the sake of being too lazy to define a simple generic class.

Getting .Net Property Names without Magic Strings

2011-04-22T14:58:00.009+10:00

This was something that had been perking my interest every so often ever since I started truly adopting Agile development practices and re-factoring code with abandon. This meant that properties and methods could be added, removed, and renamed at any point within the life of a project. There are cases where in debug messages, log entries, reflection lookups, or Argument-related exceptions I want to extract a property name. This resulted in a magic string appearing in the code.

A classic example of needing property names is with WPF binding and PropertyChanged events. Your viewmodels may be listening for property changes on bound domain objects in order to perform actions or update calculated values. Take for example:

The problem here is that if properties within InterestRate (Delta, Rate, and EffectiveDate) are renamed, the above code will stop working as expected. Now effective unit tests should help guard against behaviour changes but it would be nice if we could avoid having a hard-coded string for the property name.

Enter the PropertyName method: I had come across a solution a while ago on Clinton's Blog around using a static method to extract property names. It worked well enough but it was still a bit clumsy. What I ended up with was:

This works well enough but I didn't really like having to explicitly declare the Type (In the above example: InterestRate x) in the parameter expression. Lately I got thinking why this functionality couldn't be adapted into an extension method...

Now the calling code looks like:

This is a compliment to the GeneralToolbox static method in that the extension method will only work against an instance of a class where the static method can work against the type. (In situations where an instance isn't present.)

- Edit: Code & unit tests are now available here.

Around 6+ hours I'd love to get back.

2011-04-11T21:24:00.001+10:00

WPF is a beautiful thing most days, but every so often it rears up and slaps you in the face when all you think you have left is a fairly trivial bit of UI functionality left. Then you are burning HOURS making sure you haven't done something completely stupid in your bindings, and Dr. Googling for anyone else who's run into the same problem. The burn isn't that it's a particularly complex thing to do, it's that there end up being so many variations of things to try, most of which won't work for one reason or another, and many other suggestions simply never worked or were even tried. This burns hours upon hours. Even if I say to myself "This isn't that important, set it aside and come back to it later" it's still smouldering in my mind and within a couple minutes I'm back trying something else that comes to mind, and burning more time on it.

In this case all I had left were two little unrelated UI interactivity features that I wanted to polish off before continuing with the next set of requirements.

#1. I present details in a list that is sorted by date. In editing an item within the list I can edit the date. The logical behaviour is that the list should be re-sorted.

#2. Presented in the list rather than using separate views for viewing and editing the details I wanted to swap out a data template (or user control) inside the list item content. (Click a button to expand for edit/review, and another to save/restore to summary mode.)

WPF has Collection View Source objects that sounded like they should have fit my needs for item #1. (instead of binding directly to observable collections) WPF also has DataTemplateSelectors which looked like they should service my needs for item #2. All set! Not quite....

CollectionViewSource allows you to sort sure enough, but editing the collection items doesn't cause the view source to refresh the sorting/grouping. I spent HOURS of digging and experimenting with different options to tackle this including extending ObservableCollection to provide the sorting, (see here) refreshing via Move operations, to trying to hook into the CollectionViewSource.View.Refresh() with limited success. Finally I hit paydirt with someone that got fed up with exactly the same problem. (see here)

After finally tackling #1 I had renewed energy to tackle #2. (which ironically I had shelved in order to tackle issue #1) I had arm-wrestled with the data template selectors earlier and quickly found while they were good at picking a template, they did not listen for changes to anything they were bound to in order to make that selection so they were only good for a one-off choice. This time the inspiration for the solution came from a little gem of an idea I glanced upon from Stack Overflow (see here), specifically "You could even make your data template a ContentControl, and use a trigger to change the ContentTemplate." I had used DataTriggers before and knew I could swap out individual controls between view and edit variations but I was looking to swap out an entire template in one go. Using a data template containing a content control, and the data trigger to swap out the content template was bloody BRILLIANT!

Finally these WPF UI thorns in my side have been removed and I can resume work without these damn things flaring up to burn up even MORE time. I find it very strange that implementing such functionality was such a chore within WPF, but in case Randall Doser ever comes across this blog... #2, definitely #2.
:)

3 years, $1M

2011-02-26T20:09:00.000+10:00

Most people that know me in the industry know my automatic response when asked to estimate on vague requirements for enhancements or a new small-to-medium-sized system.
"7 years, $1M."
This often was turned around to "1 year, $7M if you're in a hurry."

I've since revised this to a more reasonable "3 years, $1M"

This usually gets a laugh out of people but I'm actually quite sincere about it as a general estimate. If something has been well thought out and requirements have been neatly separated into units of work that can be estimated and built then I can give a detailed estimate for the exact amount of work needed. When all someone gives me is a rough idea of what they want, then my response is that I'm reasonably confident that I can deliver exactly what they want for $1M (preferably up-front) and in 3 years.

This covers the time to do proper requirements gathering, prototyping, iterative development & re-factoring, plus testing to ensure the end product is spit-shined. They will have something available in production before the three year mark, but what they had in their mind (and reasonable additional stuff they're surely to think of or require along the way) will be complete within 3 years. The simple truth is the endless cycles of negotiation, re-prioritization, and up-scaling to try and meet unreasonable time or scope expectations wastes far, far, more money.

Ouch, 3-day bug!

2010-12-09T21:45:00.000+10:00

Earlier I had posted on how to write tests for the Quantumbits ObservableList class by enabling a DispatcherCycler to essentially perform a "DoEvents" to hook up appropriate events.

This was all working just fine until one day after adding a more complete suite of tests for an implementation of an ObservableList I had a failing test. I had run the test successfully before so I took a quick look and re-run the test. It passed. I ran the suite again and it failed. I ran it several times as a suite and individually to see if it might be an intermittent problem but alone it always succeeded, as part of the suite it always failed. And it failed for a completely wrong reason.
        [Test]
        public void EnsureChangingRelativeDeltaUpdatesRemainingCorrectly()
        {
            var mortgageInterestList = new MortgageInterestList(new MortgageInterest.Initial(0.05m, DateTime.Today));
            var second = new MortgageInterest.Adjustment("+.5%", DateTime.Today.AddDays(2));
            var third = new MortgageInterest.Adjustment("+.2%", DateTime.Today.AddDays(3));
            mortgageInterestList.Add(second);
            mortgageInterestList.Add(third);
            new DispatcherCycler(mortgageInterestList.CurrentDispatcher).Cycle(); // * Failed here

            second.UpdateByAbsoluteOrRelativeValue("+.3%");
            new DispatcherCycler(mortgageInterestList.CurrentDispatcher).Cycle();
            Assert.AreEqual(0.05m, mortgageInterestList.SortedList.First().InterestRate, "First interest rate should not have changed.");
            Assert.AreEqual(0.053m, second.InterestRate, "Second interest rate was not updated.");
            Assert.AreEqual(0.055m, third.InterestRate, "Third interest rate was not updated.");
        }

The guts of the list in question contained a set of data that would be sorted in date-order. It contained an initial instance (which could have it's date adjusted) and had logic that basically asserted that "adjustment" instances could not be inserted before the initial item, or moved (date changed) prior to the initial item. The error behind the failing test indicated that the test was trying to move an item prior to the initial item, which it obviously wasn't. The test merely changes the value of one of the items which should update the later item. Debugging that failure it almost looked like the adjustments were being added to the collection before the initial item, which was completely impossible.

After staring this down for two evenings looking for bugs with my Dispatcher implementations within the Observable List and cycler, Dispatcher behaviour and multithreading, TDD.Net behaviour, etc. I decided to start looking for tests that were supposed to be replicating this behaviour. Sure enough I did have tests for this behaviour:
        [Test]
        [ExpectedException(typeof(ArgumentException))]
        public void EnsureAdjustmentInterestRateCannotBeMovedBeforeInitialRate()
        {
            var mortgageInterestList = new MortgageInterestList(new MortgageInterest.Initial(1.25m, DateTime.Today.AddDays(1)));
            mortgageInterestList.Add(new MortgageInterest.Adjustment("+5%", DateTime.Today.AddDays(2)));
            new DispatcherCycler(mortgageInterestList.CurrentDispatcher).Cycle();
            mortgageInterestList.SortedList[1].EffectiveDate = DateTime.Today;
        }

I popped an [Ignore] attribute onto this test and re-ran the suite... All remaining tests pass! I re-enable the test and the other test fails once more. Now I'm seriously doing a WTF? but making progress.

Side Note: Notice now my Dispatcher cycler is accepting a dispatcher in it's constructer, rather than being a static helper using Dispatcher.CurrentDispatcher. I was chasing all kinds of ideas around whether Dispatcher actually was thread-safe, and whether I needed to ensure the Cycle call would use the same dispatcher as the list. I went through a LOT of trial and error to try and track this behaviour down...

Then I read up on Dispatcher and how it can capture unhandled exceptions. Since this test was waiting for an exception that would have been raised on the Dispatcher (UI) thread based on the nature of ObservableList I decided to see what happens if I handled the exception at the Dispatcher level. I added a text fixture setup implementation to handle unhandled exceptions:

            Dispatcher.CurrentDispatcher.UnhandledException += (sender, e) =>
                {
                    e.Handled = true;
                };
Re-running the tests I had different failures. Essentially the classes that were expecting the exception weren't receiving it... But the originally failing test was passing.

Finally I removed the fixture setup and tried the following:

        [Test]
       [ExpectedException(typeof(ArgumentException))]
       public void EnsureAdjustmentInterestRateCannotBeMovedBeforeInitialRate()
       {
           var mortgageInterestList = new MortgageInterestList(new MortgageInterest.Initial(1.25m, DateTime.Today.AddDays(1)));
           mortgageInterestList.Add(new MortgageInterest.Adjustment("+5%", DateTime.Today.AddDays(2)));
           new DispatcherCycler(mortgageInterestList.CurrentDispatcher).Cycle();
           DispatcherUnhandledExceptionEventHandler handler = (sender, e) =>
           {
               e.Handled = true;
           };
           mortgageInterestList.CurrentDispatcher.UnhandledException += handler;
           mortgageInterestList.SortedList[1].EffectiveDate = DateTime.Today;
           mortgageInterestList.CurrentDispatcher.UnhandledException -= handler;
       }

Sure enough all tests PASS! I was actually expecting the above test to fail since the dispatcher would not raise the ArgumentException that would be raised when the effective date was changed, such as was the case when I had this essentially in the Fixture setup. However, the ArgumentEception does get raised and handled.

I explicitly add and remove the delegate handler since this would be attaching a handler to a CurrentDispatcher on the executing thread. I don't want it to interfere beyond the scope of this test.

Actually, as I write this I want to be certain that other tests that might trigger an exception (as a test failure) or test the exception behaviour don't trip up other tests.

The code can be substituted so that the handler is assigned on test start up, and removed on test cleanup. Fortunately I have a base class wrapper for my nUnit implementation so this is a snap!

        private DispatcherUnhandledExceptionEventHandler _handler = (sender, e) =>
        {
            e.Handled = true;
        };

        public override void TestSetup()
        {
            base.TestSetup();
            Dispatcher.CurrentDispatcher.UnhandledException += _handler;
        }

        public override void TestTearDown()
        {
            base.TestTearDown();
            Dispatcher.CurrentDispatcher.UnhandledException -= _handler;
        }

And now all tests are covered to behave nicely with one another if the Dispatcher encounters an error. No messy hooks in individual test cases needed.

Damn that felt good to squish!

Contract vs. Permanent

2010-11-24T16:01:00.000+10:00

This is an odd one that I come across from time to time, and again just recently.
I work as a contractor for a client and they've expressed an interest in ensuring stability within their development team, so they had asked if I would be interested in taking a permanent position. I politely declined since the benefits of a permanent position don't coincide with my own objectives. I can definitely understand the desire to have some stability within a development team, and since I don't have any serious problem with how things are arranged, they have my assurances that I will be available essentially for as long as they need me.

The odd thing is that in talking with permanent staff, the impression of a contractor is that they're someone that can just vanish tomorrow; that being a "permanent employee" somehow indicates that the company has an investment in you, and you have an investment in them. However, when you look at the facts this doesn't hold any water, especially in the I.T. industry. How does a company invest in you? "Permanent" implies that you will always have your job through hell and high water. However every permanent employee has the exact same termination clause in their contract as I have in mine. You can be "let go" or decide to leave with 4 weeks notice. You could argue that a permanent employee would have some additional protection in the form of unfair dismissal, but no more-so than I would if I were given a six month contract which the client tried to end after just two. In either case it will be an ugly business to dispute. Companies will often offer training to permanent employees. However, in the over 15 years I have been in software development in 10 organizations/clients (7 of which as a permanent employee) I have received training twice. (1x 1-day course on advanced VB, and a 2hr course on time management.) I have only rarely seen other permanent employees actually get sent off for training or seminars. Depending on your client, a contractor can be treated to other benefits that are given to permanent staff such as inclusion in company dinners, and gifts of appreciation.

Contracting also has a bit of a myth associated with it that often rubs permanent staff the wrong way: Contractors get paid more. False, with a grain of truth. Being a contractor, especially on long-term contracts (>6 mo.) means you are exchanging most of your entitled benefits for cash. Annual leave, sick leave, paid holidays, etc. This means if you work for a full year, not taking any sick days off, as a permanent employee you are entitled to an extra 20 days pay. If you don't use it, those days are paid out when you leave and taxed at the maximum tax bracket. You won't get that extra tax back until the end of the tax year. (So in Australia, be sure to quit that permanent position in June.) In the I.T. industry and especially in smaller organizations, paid leave is rarely "convenient" to take. You're always busy, and you know that any leave you take is going to be a significant burden on the company, your co-workers, and extra urgent work to worry about when you get back. Some permanents I've worked with try to plan their vacations after the next major release, only to find that the schedule slips and that leave they've booked for months in advance now falls smack in the middle of the production roll-out. Permanent positions are kind of like an insurance policy. If you're sick or want to take some time off, your income flow stays stable. But if you don't use that, well, you might be better off with the money up front. Short term contracts will result in marginally higher rates simply because a contractor must factor in the time they need to take in-between short-term contracts in order to secure more work.

The reason I contract as a sole trader is to maximize my income to minimize my debts. There are a number of factors that I considered to reach this:
1) I have a sizable mortgage with an offset account. Offset interest savings are 100% tax free, and are valued directly against the current interest rate. (Where any other investment earnings contribute to and are taxed at your overall tax rate and Super contributions are taxed at 30% with super earnings taxed at 15%)
2) As a sole trader I invoice my client directly and weekly, collect GST and and pay it back quarterly, pay income tax annually, and do not contribute to Superannuation. (And soon I will be contributing to it again annually.)
This maximizes the amount of money I keep in my offset account working to reduce the interest my mortgage is adding up on a daily basis.
3) I'm rarely if ever sick, so sick leave isn't any value to me. (Since it's not paid out if unused, and I'm generally too ethical to "chuck a sickie".)

Because of this, contracting under this arrangement will help pay off my mortgage many years faster than if I was a permanent employee.

Confused the compiler/runtime or what?

2010-11-11T15:57:00.002+10:00

This was an innocent enough of a typo but it had me questioning whether the framework's string.Split() method was working with the remove empty entries option.

Line in question:
var parameterPairs = parameterValue.Split(new char['&'], StringSplitOptions.RemoveEmptyEntries).ToList();

The parameterValue being parsed was "&Token=8765"
I expected to get a result back of "Token=8765", but instead got back a value of "&Token=8765".

I was thinking was there a problem with RemoveEmptyEntries when the separator was the first character? Was there a problem with '&' as a separator? Both checked out. Then I noticed the typo. It should have been:

var parameterPairs = parameterValue.Split(new char[] {'&'}, StringSplitOptions.RemoveEmptyEntries).ToList();

And presto, the test passed. But that got me wondering, what was char['&'] resolving to? Why did the compiler allow it to compile, and why did string.Split() seem to merely ignore it at runtime?

I put in a precursor condition:

var test = new char['&'];

along with a breakpoint.

Curious...

Beware! The clipboard may lead you to question your sanity.

2010-11-02T15:10:00.000+10:00

This one had me scratching my head for a good while. I was able to fix it easily enough, but getting to the bottom of just what the hell happened took a bit of digging.

The problem started because I wanted to switch a test against a certificate located on the disk to using a certificate located in a certificate store. For this test I decided to pull the certificate by serial number so I snatched the serial number from the certificate properties easily enough.

I tried this as-is and it wasn't working. So I tried removing the spaces, still no joy. So I grabbed the entire list of certificates on the machine and found the certificate to verify the serial #. The results showed the serial number in upper case, so I quickly popped a .ToUpper() on my key. NO-JOY. Ok, WTF. So I copy the serial number out of the watch results in the IDE and try again, it works.

But I couldn't leave it at that. What was wrong with my original value? I restored the changes and created a second set of checks (See first screen) and started cursing my monitor in disbelief. I lined them up, reversed the order they were called in the Find() statement, and cleaned and rebuilt the test application several times. String.Compare returns a match, .Equals returns not equals. The call to the store to Find returns a result for only the one value, regardless of which order they are executed.

The only possibility: copy and pasting the original value from the certificate dialog resulted in some unprinted Unicode character being embedded in the original string. (serial2) String.Compare ignores it, but .Equals does not. (The test1/test2/result check was just me testing my sanity that .Equals on string wasn't doing a reference equals.) To quote Sir Arthur Conan Doyle: "Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth."

Keeping up is fun.

2010-11-01T21:26:00.003+10:00

Software development is constantly evolving. Years ago there would be years between product releases, and while there were libraries popping up every so often, there were typically a lot fewer to follow for any given project you might be working on. The pace of change was generally slower.

Today, change is constant and occurring at a significantly faster pace.On top of that, the open-source movement has meant a lot more releases of a lot more useful tools to incorporate within a project. It's easy to get left behind, or running after new versions without a chance to stop and smell the roses.

A perfect example. nUnit is a brilliant unit test framework port for .Net. I've used it for many years, but I've never really given each new version more than a passing glance. Unitl I caught a mention of a new attribute called "[TestCase]". (As opposed to the "[Test]" which denotes a test case.)

Take a case where you want to test a range of outputs such as a minimum, maximum, and assortment of values to ensure a method is working properly. This might be 4 tests or more, consisting of setup, call, and assert. For example I had 3 tests that verified formatting behaviour for a currency converter. Positive value, negative value, and zero. Scenarios like this can now be replaced by something like this:

        [TestCase(1000d, Result = "$1,000.00")]
        [TestCase(-1000d, Result = "-$1,000.00")]
        [TestCase(0d, Result = "$0.00")]
        public string EnsureConverterFormatsCurrencyValues(decimal value)
        {
            IValueConverter testConverter = new DecimalToCurrencyConverter();
            return (string)testConverter.Convert((decimal)value, typeof(decimal), null, CultureInfo.CurrentUICulture);
        }
*Sigh*... Those roses sure smell nice when you finally get a chance. Not only is it one test case (executed and reported 3 times) but it doesn't even need the Assert.Equals line. It would have been nice to have spotted this a year ago when they added it. :)

*Note: The above declaration is actual code, not a typo. The parameters on the attribute are provided as doubles (d) not decimal (m) even though the parameter type is decimal. Decimals cannot be provided via attributes (a limitation in .Net) so TestCase facilitates this translation.

Amber, Green, Re-factor

2010-09-23T21:32:00.004+10:00

Red, Green, Re-factor (Red-Green for simplicity in this entry) is the mantra of Test-driven development. (TDD) The goal being to satisfy a requirement you write a test that fails, then write the functionality to satisfy solely that test, then re-factor as you implement further functionality. There's definitely nothing wrong with this approach, but I don't think it's for everybody. An alternative to TDD is Test-soon development. (TSD) In this model you write tests soon after you're satisfied that target requirements have been met. However, TSD is still Test-driven development, it just isn't Red-Green. TDD is test-driven because design decisions place an emphasism on meeting requirements with testable and, most importantly, tested code. This is obvious when you write the unit tests first, but is no-less true when writing unit tests second. The key is that the code is written with tests in mind. When you choose to write them is irrelevant, so long as you test it right.

TSD is less disciplined than TDD in that it is easier to defer testing as you continue to implement related functionality. This accrues up technical debt, leaving a larger pile of unit testing required. Tools like NCover are excellent at tracking your technical debt bill, so if you're using TSD, NCover is an absolute must. Another negative point against TSD to the TDD purist is that unit tests against working code would be written to pass, so they would be less reliable since they may not actually fail when they need to. This is a significant concern, but I think it applies just as much to Red-Green TDD as it does to TSD. In Red-Green you might only write tests that satisfy the "chosen path" but not exercise the code properly when paths cross. Either way you need to really think about behaviour to test, not just satisfy the obvious aspect of a requirement.

This second concern of writing tests to fail really got me motivated to ensure tests were exercised fully along-side the code they intended to test. My motto of choice to this effect is Amber, Green, Re-factor or Amber-Green. Essentially I write code to satisfy a requirement, then I create tests, and excercise test & code (Amber) until I'm satisfied they pass (Green) then the code is ready to be re-factored as necessary for future functionality. The reason I prefer this to TDD is that I've invested the time into satisfying architectural decisions that often inter-twine with satisfying requirements. I'm satisfied not "that it works", but rather "that it's going to work". Now the focus shifts to developing the tests to ensure that it truly works. My goal with writing the unit tests is to break the code. I know for a fact that I didn't think of everything so it's time to find what I missed by probing the code with tests. When I write a test that passes first run I alter the code and make sure the test fails as it's supposed to. When I write tests that don't fail then I become concerned that I'm not exercising it fully. Even if I think of something I missed, I write the test, watch it fail, then fix it.

Ultimately the approach isn't much different to Red-Green. With Red-Green it pays to think up-front while fleshing out boundaries and behaviours for code before you write it. Personally I find it's easier to flesh this out once I know what I'm planning to take to production, and switching my mentality to "break it!" helps to really excercise the code.

So the next time the TDD purist on your team scorns you for not writing tests first, rest assured you can still be test-driven with Amber-Green; Just keep a close eye on your technical debt and break that code good.

Unit Testing for the QuantumBits ObservableList

2010-09-17T17:08:00.004+10:00

In using the ObservableList example from QuantumBits I came across a bit of a problem. This list ensures all event notifications happen on the UI thread by using the dispatcher. One change I added was to expose virtual methods in the ObservableList to "hook in" custom actions such as a flow-on effect for calculations as items in the list are changed. (OnItemAdded, OnItemRemoved, OnPropertyChanged etc. )I have extended functionality that interacts with these events to run calculations and I wanted to unit test this behaviour.

An example test:

[Test]
public void EnsureThatRelativeRateIsUpdatedWhenEarlierInterestRateChanges()
{
  var mortgageInterestList = new MortgageInterestList();
  var first = new MortgageInterest.Absolute(0.05m, DateTime.Today);
  var second = new MortgageInterest.Relative("1%", DateTime.Today.AddDays(2));
  mortgageInterestList.Add(first);
  mortgageInterestList.Add(second);

  first.UpdateInterestRate("+0.5%");
  Assert.AreEqual(0.055m, first.InterestRate, "Interest rate was not updated.");
  Assert.AreEqual(0.01m, second.InterestDelta, "Second's delta should not have been updated.");
  Assert.AreEqual(0.065m, second.InterestRate, "Second's interest rate should have been updated.");
}

An absolute interest rate is such as when a user says the interest rate was 5% as-of a specified date. A relative interest rate is one where the user specifies a +/- delta from any previous interest rate. The idea being that if an interest rate changes right before a relative rate, the relative's delta should be re-applied to adjust it's rate.

The problem is that while the MortgageInterestList (extending ObservableList) is wired up with the event to perform the calculation, running in a unit test the dispatcher doesn't actually run to set up the event hooks so that the edit triggers the flow-on effect.

The problem stems from this in the ObservableList:

public void Add(T item)
{
  this._list.Add(item);
  this._dispatcher.BeginInvoke(DispatcherPriority.Send,
    new AddCallback(AddFromDispatcherThread),
    item);
  OnItemAdded(item);
}

This method is perfectly correct, and in an application utilizing this functionality it works just great. However, unit tests don't like this kind of thing. The callback is added, but even with a Priority of "Send" (highest) the hookup doesn't actually happen during the test execution. The event doesn't fire, the rate isn't updated, the test fails.

After a lot of head scratching and digging around on the web I came across the solution on StackOverflow. Ironically while someone was having a similar problem to me with testing around Dispatcher-related tasks, this wasn't the accepted solution, but it worked a treat.
2nd Answer

public static class DispatcherUtil
{
  [SecurityPermissionAttribute(SecurityAction.Demand, Flags = SecurityPermissionFlag.UnmanagedCode)]
  public static void DoEvents()
  {
    DispatcherFrame frame = new DispatcherFrame();
    Dispatcher.CurrentDispatcher.BeginInvoke(DispatcherPriority.Background,
      new DispatcherOperationCallback(ExitFrame), frame);
    Dispatcher.PushFrame(frame);
  }

  private static object ExitFrame(object frame)
  {
    ((DispatcherFrame)frame).Continue = false;
    return null;
  }
}

Basically it's a mechanism that you can run inside a test to signal the dispatcher to go ahead and process anything currently pending. This way the events will be wired up before the test resumes. Damn simple and it works.

The test in this case changes to:

[Test]
public void EnsureThatRelativeRateIsUpdatedWhenEarlierInterestRateChanges()
{
   var mortgageInterestList = new MortgageInterestList();
   var first = new MortgageInterest.Absolute(0.05m, DateTime.Today);
   var second = new MortgageInterest.Relative("1%", DateTime.Today.AddDays(2));
   mortgageInterestList.Add(first);
   mortgageInterestList.Add(second);
   DispatcherUtil.DoEvents();

   first.UpdateInterestRate("+0.5%");
   Assert.AreEqual(0.055m, first.InterestRate, "Interest rate delta change was not based on the previous interest rate.");
   Assert.AreEqual(0.01m, second.InterestDelta, "Second's delta should not have been updated.");
   Assert.AreEqual(0.065m, second.InterestRate, "Second's interest rate should have been updated.");
}

SO if "jbe" should ever read this, thanks a bundle for the solution!

I've never been a fan of DoEvents coding in VB which this is pretty closely mirroring so this little gem will be located in my Unit Testing library where I have my base classes for uniform test fixture behaviour.

*Edit* I didn't much like the name of the class when I incorporated it into my test library. The updated signature is:

public static class DispatcherCycler
{
  [SecurityPermissionAttribute(SecurityAction.Demand, Flags = SecurityPermissionFlag.UnmanagedCode)]
  public static void Cycle()
  {
    DispatcherFrame frame = new DispatcherFrame();
    Dispatcher.CurrentDispatcher.BeginInvoke(DispatcherPriority.Background,
    new DispatcherOperationCallback((f) =>
      {
        ((DispatcherFrame)f).Continue = false;
        return null;
      })
      , frame);
    Dispatcher.PushFrame(frame);
  }
}

Linq2SQL Cache & Connectivity

2010-09-15T15:30:00.009+10:00

One "gotcha" you will need to consider when using Linq2SQL is how to handle situations where a connection is lost at a moment when a SubmitChanges call is made. (Basically the period of time between when the last object reference is retrieved from the cache, and SubmitChanges is called.)

The SubmitChanges call will fail, expressing what is wrong, however the cached copies of the data will hold onto their changes until they are committed, or refreshed. This can have some truly confusing side-effects that can be a lot of fun to track down.

Case in point: I recently had to track down one of these issues in a simple barcode stock application. It basically consisted of orders and boxes. As boxes of stock come in, they are scanned with a barcode reader, looked up in the DB using Linq2SQL, have their statuses validated, and updated. As each box is scanned in, the order is also loaded to inspect whether or not all of its boxes have been scanned into stock, and when they all are, the order is updated to "complete". The bug was that every so often (once every several weeks or more) we got a situation where an order was found to be complete while one of its boxes was not recorded in the DB as "in stock". We eliminated every possibility, scoured to code for status change bugs, but there was nothing. Scanning exceptions were already brought up to the screen, but operators typically dealt with several scan issues (re-scanning boxes after being interrupted, etc.) so they didn't notice anything odd. After extending the logging support to see what was going in I found that the box was scanned, an exception occurred, but when re-scanned it said it was in-stock, and after all remaining boxes on the order were scanned, the order found all boxes to be in-stock, and diligently closed itself off.

The issue was Linq2SQL's caching behaviour. The exception was occurring when the box's change was submitted to the database. Any problem with the connection to the server (This was a client running in Victoria hitting a DB in Queensland) and the SubmitChanges would fail, but the updated objects remained in the cache. Since the same data context was used for all boxes scanned in during a receive stock session, further requests for that box retrieved the cached copy.

This is a bad scenario to be in. Your options are either to retry the submit changes until it either goes through, or you try to revert the changes until they revert, or you force the application to shut down if neither of those work. I was reporting back to the user that there was an error, and IMO if there is a problem then the user should retry what they were doing, I.e. re-scan the box, so I wanted to revert the changes. If the data source still wasn't available then the user should shut down the application and contact support.

The first solution that came to mind was to catch the exception from SubmitChanges and refresh the affected object graph. Since the box had child entities that were also updated, plus an order line item, this entire graph would need to be refreshed. Easy enough, as objects are queried and updated, they got added to a revertList should the SubmitChanges fail:

try
{
    DataContext.SubmitChanges();
}
catch
{ 
    DataContext.Refresh(RefreshMode.OverwriteCurrentValues, revertList);
    throw;
}

Unfortunately this doesn't work as the Refresh method will fail with:
System.InvalidOperationException: The operation cannot be performed during a call to SubmitChanges.
at System.Data.Linq.DataContext.CheckNotInSubmitChanges()
at System.Data.Linq.DataContext.Refresh(RefreshMode mode, IEnumerable entities) ...

So I needed to trigger the Refresh call outside of the SubmitChanges scope. The first thing to note is that while the system is processing, the scanner is turned off until the processing is completed. For minor errors such as scanning a box twice, etc. the scanner beeps out, displays a message, and reactivates. For serious application errors the scanner remains off until the application determines the exception can be corrected. The user knows that if the scanner beeps out and turns off, check the screen, and try restarting the application if it doesn't turn on within a minute or so.

In any case, the solution was to use an asynchronous operation to handle the scanner error and data refresh. This meant worker thread and locking off the scanner access until the refresh was successful, or timed itself out trying.

try
{
  DataContext.SubmitChanges();
}
catch
{ 
  revertDomainObjects(scanner, revertList);
  throw;
}

/// <summary>
/// This method will attempt to revert changes to objects to reflect the data
/// state. This will be called in situations where the data connectivity has 
/// been interrupted, where data state does not reflect cached state. This will
/// attempt to refresh cached copies of objects to match data state.
/// </summary>
/// <param name="revertList">List of domain objects to revert back to data state.</param>
private void revertDomainObjects(IScannerHandler scanner, List<object> revertList)
{
  ThreadPool.QueueUserWorkItem((e) => 
  {
    lock (_dataContext)
    { 
      var inscopeScanner = ((List<object>)e)[0] as IScannerHandler;
      var inscopeRevertlist = ((List<object>)e)[1] as List<object>;
      int attemptCounter = 0;
      bool isSuccessful = false;
      while (!isSuccessful && attemptCounter <= 4)
      {
        try
        {
          _dataContext.Refresh(RefreshMode.OverwriteCurrentValues, inscopeRevertlist);
          isSuccessful = true;
          inscopeScanner.RecoveredFromCriticalException();
          scanner.Status = Objects.ScannerStatus.Idle;
          scanner.StatusMessage = "Scanner has recovered from the previous error. Please retry operation and contact Licensys I.T. if the issue continues.";
        }
        catch
        { // If the rollback fails, sleep for half a second and retry up to 5 times.
          Thread.Sleep(500);
          if (attemptCounter++ > 4)
          throw;
        }
      }
    }
  }, 
  new List<object>() {scanner, revertList});
}

Probably a better solution would be to use a Task<> implementation if it's available in .Net 3.5, but this does the job and I don't need to be able to interrupt it.

Now in the case of an exception during SubmitChanges, the scanner will alert the user and be de-activated (triggered when the exception bubbles)and the worker process will kick off to try and refresh the affected data. if that is successful the scanner will be re-activated and the user notified that they can continue.

Caveats: This did require adding some thread-safety to an otherwise single-threaded application. I added an accessor property that attempted to lock the data context before returning it. Not ideal but in this case the means of triggering calls to the data context (scanner) was disabled until the operation was completed. This will be reviewed and refactored shortly as we move to support multiple simultaneous scanners.

Rolling in the mud.

2010-07-30T10:37:00.000+10:00

Every idea starts its life as a shiny new marble. Unfortunately the world around us is a pretty muddy place, and when that idea gets pushed around in the real world, it soon starts collecting mud.

Software development has had countless shiny marbles since I started in the industry, and it's pretty interesting watching as they all slowly succumb to the mud. Development languages, best practices, design patterns, frameworks, etc. Each virtually glows with merits and virtues, flinging mud on its predecessors. Everyone wants a turn at pushing around the new shiny marble, but pretty soon it too is coated in mud.

A most recent example: MVVM (Model View View-Model) This is a relatively new design pattern stemming from the MVC (Model View Controller) pattern ancestry. Essentially you have a history that we'll start off at MVC which is a great pattern for earlier, relatively state-expensive terminal applicationa, and is very well suited to stateless web applications. However, for environments where maintaining state was cheap, and users wanted GUI-rich applications, a new shiny variation was born, MVP. (Model View Presenter) It's essentially the same thing, just with some responsibilities shifted between layers. A good explanation can be found here.

Microsoft chased MVP for quite a while with their WinForms and later, WebForms implementations. WebForms was a bit of a disaster because of the cost of maintaining state over the wire.

Martin Fowler coined a new design pattern called Presenter Model which was essentially a MVP pattern where the Presenter was converted into a representation of the model for the view. A subtle variation of this was later given the title of MVVM, and the next shiny marble began rolling around. Microsoft re-defined the way they represent views in both applications and web applications to utilize XAML, pushing aside the older WinForm and WebForm constructs in favour of WPF and Silverlight. These were designed to gel with this new MVVM pattern.

The part I find odd is the seemingly outright vilification of the code-behind aspect still present in XAML, similar to the construct used in WinForms and WebForms. Back in the day these were shiny features, now they're shunned, and to be replaced at all cost with new shiny features.

But it's kind of funny, because in flinging the mud on code-behind, it seems that MVVM is starting to roll deeper into the mud itself. MVP would rely somewhat on code-behind. In a clean example, the code-behind would essentially implement events from the view definition (interface) and upon receiving an event from a UI element such as a button, raise an appropriate view event to the presenter. You might also have some simple method declarations that the presenter can call into the view to perform actions. (Such as clearing out all fields, etc.) The view is essentially anemic. But, there is code-behind.. POO-POO!

In MVVM-land, your view is bound to a View-Model, and your View-Model can define Commands that can bound to things like buttons, so that clicking a button will automatically execute a command. Ok, I can see the merit in this, but wait, it's not that simple, it's got to be shinier. The Command accepts delegates so you can define what the command does.... back in the View-Model.... um, ok... This is starting to smell. Great, so we can put command logic in the View-Model, that is delegated through another object, just so we can avoid declaring an event in code-behind.

Granted, there is some merit in this approach. This does mean that a "designer" can tweak behaviour in an application merely by editing XAML, they don't need to see C#/VB code-behind files. But, they still need to know what these commands are so they can wire them up in a view. In the end, the code is more complicated, with Func<> declarations scattered around and Lambdas, etc. all to remove a couple of simple events.
Now I find this whole argument of defining more stuff in XAML a pretty weak one. How many companies actually advertise for and hire "designers"? More importantly, is it such a huge ask to have a developer work with one of these designers to wire up UI changes, in the event that a designer with a likely web background that has to be able to breathe Javascript, couldn't figure out how to wire an event in C#?

MVVM is starting to get ripe with issues where those wonderful savings that it offered over MVP and code-behind. Setting focus after validation, and dealing with message boxes are just a couple that are popping up. It's not that these issues don't have solutions, but these solutions are typically either hacks around limitations, or fairly complex solutions for what should be simple issues that have been done to death in earlier patterns.

MVVM is getting muddy. I wonder when the next shiny marble will appear.

SQL Count by criteria.

2010-06-04T15:06:00.000+10:00

This one had me scratching my head for a while. I had a tiered query grouped by a number of fields in the table structure. I wanted to get a count of all applicable items matching a set of given statuses.

For example: Toys
Type
Manufacturer # Sold # Returned

Cars
--Vrroom 14 0
--Bigga 16 2
Dolls
--Munchkin 22 4
--Slimer 7 1

etc.

Originally I had a query along the lines of:
SELECT i.InventoryType,
p.ProductName,
m.ManufacturerName,
CASE t.TransactionType
WHEN 1
THEN 'Sold'
WHEN 2
THEN 'Returned'
ELSE 'N/A'
END AS TransactionStatus,
COUNT(t.TransactionID) AS TransactionCount
FROM Inventory i
INNER JOIN Product p ON i.InventoryID = p.InventoryID
INNER JOIN Manufacturer m ON p.ManufacturerID = m.ManufacturerID
INNER JOIN Transaction t ON p.ProductID = t.ProductID
WHERE .... -- Conditions...

(Now this is only a rough approximation of the issue I was facing)
I then grouped within the report and used a conditional field on the transaction status to count applicable items. It sort of worked, but produced 2 lines for the items that had both sales and returns:

Cars
--Vrroom 14 0
--Bigga 16 0
-- 0 2
Dolls
--Munchkin 22 0
-- 0 4
--Slimer 7 0
-- 0 1

At first this looked like an issue within the SSRS report where it was grouping by Details. I removed that grouping which at first looked like it was working (1 row per product manufacturer) but that just "knocked off" one of the values. (I.e. for "Bigga" I'd end up with either 16/0 or 0/2, essentially depending on which status was found first.)

Round 2: I went back to the query determined to do the grouping properly in SQL.

...
m.ManufacturerName,
CASE t.TransactionType
WHEN 1
COUNT(t.TransactionID)
ELSE
0
END AS SoldCount,
CASE t.TransactionType
WHEN 2
COUNT(t.TransactionID)
ELSE
0
END AS ReturnedCount
...

This required t.TransactionType to be in the GroupBy clause which resulted in query results pretty much identical to what I was seeing in the report.

Round 3:
...
m.ManufacturerName,
SUM(CASE t.TransactionType
WHEN 1
THEN 1
ELSE 0
END) AS SoldCount,
SUM(CASE t.TransactionType
WHEN 2
THEN 1
ELSE 0
END) AS ReturnedCount
...

And Bingo!

Cars
--Vrroom 14 0
--Bigga 16 2

Dolls
--Munchkin 22 4
--Slimer 7 1

(Also, if a transaction had a quantity you wanted to count by then it'd be summing on t.Quantity rather than "1" (transaction count))

Thou shall not transmit localized DateTime values.

2010-06-04T13:49:00.000+10:00

Date and time formats come in a variety of flavours, but one combination causes countless bugs, crashes, and potentially numerous lawsuits: dd/mm/yyyy vs. mm/dd/yyyy. If you ever come across "It worked yesterday, but it's crashing today" pretty much anywhere other than North America you've met this little gem.

The important thing to remember about DateTime values is that they are floating point numbers. 06/01/2010 is NOT a date, it is the localization of a date. The computer you give that localization to has to interpret what the actual date is. Is it January 6th, or June 1st? ANY, and EVERY time you need to transmit or transfer a DateTime from one source to another, you MUST take this into account. Generally this means every time the datetime transfers from one machine to another, such as to a Database server, a reporting server, or a web service. Today that may be two machines that have the same regional settings, but it just takes one server or client PC to be set to a different regional date format to totally hose your system. One thing you should NEVER see in code is a ".ToString()" (empty parameters) applied to a DateTime variable.

Now you don't need to go and start passing DateTime values around as floating point numbers to avoid this issue. (though that is one option.) There is one other practical option when transmitting DateTime values is to use ISO formatting. International Standard formatting for DateTime values is: yyyy-MM-dd HH:mm:ss. That is, 2010-01-06 19:04:30. In .Net this is commonly known as "Sortable". The main reason why this is advantageous is that regardless of their local date formats, any application worth its weight in salt *will* accept an ISO DateTime with its 24hr time format. ISO DateTimes also have the advantage that they are fully sortable so you can sort down to the month, day, hour, etc. or generate sequential numbers for things like filenames.

So there's no need to abandon the DateTime field type whether in .Net, SSRS, Crystal Reports, or SQL Server / Oracle or start farting around with regional settings; Just adopt ISO date and time formats when passing around dates.

Linq2SQL is NOT an ORM

2010-05-01T17:16:00.000+10:00

I think this needs to be stressed as much as possible. Linq2SQL, and other deviations that generate an object model based on a relational database are NOT ORMs.

ORM: Object-Relational Mapper

"Mapper" is the key missing ingredient with solutions like Linq2SQL. The key driving concept of an ORM is to allow you to develop an object model in a manner that meets the business needs of an application, or suite of applications, completely independently of the data source or sources serving that business. This doesn't mean that a product tied to a particular RDBMS has to be any less of an ORM than one that can serve a number of RDBMS; that isn't the point of the mapping. The point of the mapping element is that the views in your application can be served by domain objects designed specifically for the purpose of serving those views, irregardless of the data structure behind the data of that view. If it helps, think of earlier implementations of Views within an RDBMS, or stored procedures that were designed to flatten and translate highly relational data into a data form designed to serve a specific purpose. An ORM takes over for that role.

Linq2SQL in no way does any of this. A more accurate acronym for Linq2SQL and the like would be a ROG, or Relational Object Generator. Linq2SQL generates objects in accordance to the relational model within the database for the application to consume. Not that this approach does not have its merits for certain situations, but it is a completely different barrel of fish.

Any "ORM" that purportedly generates an object model from a database is flat-out not an ORM. There is no mapping when the relationships are 1-1 between objects and data tables; It's a generator.

When configuration goes pear-shaped.

2010-04-09T10:45:00.001+10:00

This one was a rather amusing recent development in the system I am currently assigned to maintain.

The system is highly configurable. Several of the driving queries used within the application are themselves stored as SQL within the database. Even the menu structure is built based on a query stored in the DB. The premise behind this was that if we need to tweak how these queries pull data we can do so without re-deploying the application. (A Silverlight + Web Services business app.) Wonderful!

So an issue comes up where some data is coming back in the wrong sort order. It turns out to be an issue between #null vs. empty string values. A relatively simple change to the stored SQL statement should be all that's necessary so I test and script the change, deploy it into the test environment, then close and re-open my SL client to be sure and....
The old query results are still coming back.

Those beautiful dynamic SQL statements are cached in the application pool which means after changing a script you got to cycle the server. *sigh*.

The drive for configuration is mostly valid because we want to avoid making changes to the Silverlight client as much as possible. (A whopping 3.5MB D/L whenever this changes, [recently shrunk to 2.7MB] which with several hundred clients and more on the way will eat dearly into our ISP upload thresholds.) But queries like this are executed server-side anyways so it's easy enough to update server-side code without re-building the SL client.

Regardless it's not that bad of a scenario, but just goes to show that often the best laid plans still develop sizeable potholes.

Using consistency's name in vain.

2010-03-24T13:09:00.000+10:00

This is a bit of a pet peeve, and something I've heard repeatedly to justify bad code and/or bad design. "It needs to stay consistent."

Question: How can code or design improve if it must remain consistent?

Code and design should be free to evolve within the lifespan of a project, to freely address concerns that come up with the initial approach and constantly challenge whether or not the original assumptions were really the best decision for the business. Going back through all existing functionality and re-factoring based on new technologies, patterns, or preferences of the customer can be prohibitively expensive, but that should not be an excuse not to adopt it going forward... <caveat> *If* it is what the customers wants. </caveat>

A consistent approach to an application is important, but it's a bad idea to take it so far as to make it extremely rigid. A common pitfall is with UI. For instance, stressing a design that says that all forms will have a default set of action buttons for wizard-like behaviour (Next, Prev, Cancel, Skip) is just asking for trouble when the customer wants to do something different. Pretty soon the "Cancel" button gets re-labelled and on screen X, performs action Y. Designing all forms to be dynamic, based based on some construct of a framework leading to slow, buggy, and not-so-user-friendly experiences is another example.

It's not that designs like this are inherently bad. They aren't, but they aren't guaranteed to be suitable to 100% of cases out there.

Explicit Interfaces

2010-03-13T17:16:00.000+10:00

The explicit use of interfaces is something I really try to follow and encourage others to start using.

Implement Interface Explicity
The second option for implementing interfaces implements the interface member effectively as a guarded public method, accessible solely through an interface reference instead of a public member. I don't think I've come across a development team that has used this option, but after experimenting with it, I think it's something that should be adopted a bit more by default.

My goal when writing code is to define contracts for the interaction between distinct units of work. This gives me the flexibility to swap out classes easily and quickly. This means I'm predominantly working through interfaces. The most significant benefit of implementing interfaces explicitly is that it means any change to the interface is immediately picked up in the implementing member(s) by the compiler. This means if I re-factor code and remove methods from an interface, the compiler notifies me on the next build that I now have dead code in implementing members.

By designing concrete classes with explicit interfaces, the public API for those classes are kept tidy, and it helps enforce usage through the contract interfaces. I hate seeing code "twisted" by having concrete references constructed and passed around. It defeats the point of defining an interface in the first place.

Regex is dead! Long live the Lambda!

2010-03-12T14:20:00.000+10:00

Lambda expressions are cool. Linq is cool. But Lambdas and Linq can be a quazi-pain-in-the-ass when it comes to inspecting and debugging the process flow of your application.

Remember regular expressions?

"Some people, when confronted with a problem, think 'I know, I'll use regular expressions.' Now they have two problems."

A famous quote from Jamie Zawinski. Regexes are cool. You can do a lot with them, but the trouble is that you can do a lot that you really, really shouldn't with them.

Linq and Lamba's fall exactly into this category and I fear in a few years of misuse and abuse, people will be shivering at the sight of Lambda the same as they slink away from regular expressions. I've already seen several examples of Lambda expressions that make my skin crawl.

One common Lambda misuse is the same damn misuse you see with looping structures. "x", cursed "x". And when "x" is used up, you start to see "y", "t", "i", and other letters popping around inside complex nested lambda expressions. I even started doing it myself and then I decided that I deserved a punch to the head if I continued this nonsense.
"x" kills readability.

Lambda is also tough to debug, and unfortunately even with VS2010 the debug windows (such as Watch) still cannot compile them, and they probably never will. The issue with Lambda as I see it is that it is a very powerful and useful tool, but just as prone to misuse as regular expressions. A word of prophetic warning to developers out there: "Just because you can do something with Lambda, doesn't mean you should do it with Lambda."

"I've got string input... After it's been massaged, squeezed, filtered, and transformed by my uber Regular Expression I... I'm not getting the result I'm expecting... WTF?"

"I've got data... After it's been massaged, squeezed, filtered, and transformed by my uber Lambda expressions, I... I'm not getting back the results I'm expecting... WTF?"

I can't tell the difference, can you?

Let me be the first to coin the phrase:
"Some people, when confronted with a problem, think 'I know, I'll use Lambda expressions and LINQ.' Now they have two problems."

Divorcing the Framework

2010-03-01T16:00:00.000+10:00

If the goal is to reuse code between projects, and adopt a consistent approach to building applications, it is possible to accomplish this by convention rather than spending a lot of time trying to invent a framework.

As a starting point I would highly recommend reading "Clean Code" by Robert C. Martin (Uncle Bob) as a freshening view of how to write better code. My own realization was to compare software development with writing or painting. Many of the best writers out there still start their works with pen (or pencil) and paper, not on the computer. The reason for this is because they are applying a time-honoured technique of drafting down their ideas before reviewing and refining them. Painters don't just crack out the oils and produce a finished portrait. They draft in lines, paint over what they aren't satisfied with, and gradually build a portrait through refinement.

This is where software fails. Too often, a developer will write a block of code to satisfy a requirement, verify that it "works", and then move on to the next requirement. There is little to no refinement involved, either from performance or stability (engineering) or how well the intent behind the requirement is satisfied. Only later, when the product is put through its paces, do issues around performance or behaviour start coming to the surface. But by then the code is draft built upon draft, built upon draft.
Frameworks only compound this failure by trying to restrict options in how to accomplish a task, and significantly increase the investment required to refine any specific scenario. The failure to draft ideas and then re-factor not only the code, but the design & requirement is the paramount failing of a project.

The second step to building better projects is to gain an appreciation and understanding for Object-Oriented software development, S.O.L.I.D. principles, and design patterns. Technologies change all of the time, but the fundamentals don't. Learn to write self contained units of work that have clear boundaries. Develop automated unit tests around these units so that these boundaries are monitored. Getting to this point is 90% of the battle. Each unit of work can now be utilized in any number of projects, allowing a consistent approach to a specific domain problem. If it doesn't suit one given scenario it can be substituted or extended without corrupting the original. There is no framework, no configuration, only simple components representing units of distinct work. Tying these together is basic, fundamental OOP inheritance and composition via interfaces. They should not care how they are persisted, how they communicate with one another, or even what concrete implementation they are communicating with. Too often, these details (persistence, and communication) become the driving force behind building a framework. They start introducing limitations on the desired behaviour of the application, and the instant this happens, the framework is working against you. Clients don't pay developers to tell them what they cannot have, or to come up with ways to make what they want more expensive.

Most frameworks that I've worked on have been designed with a lot of good coding principles and patterns in mind, but also with a significant amount of investment in the intangible. Teams have invested many hours into "framework code" that does not directly meet functional requirements. This forms a type of debt that they expect to get back once the framework is mature enough and features can be quickly bolted in and configured. Unfortunately the vision at the start is incomplete and a lot of extra time is needed to adjust or work around limitations from earlier assumptions in the framework.

The benefit of breaking things down into units of work is that it opens the door to update popular components to new technologies while still allowing them to interoperate with older components. There are challenges as components are upgraded, but these are only in the ties between components. By following good patterns and practices these challenges are easily dealt with. A good example is when dealing with collections of objects. In .Net 1.1 the common approach was using ArrayLists while .Net 2.0 and onward adopted IList<> generics. If we re-factor units of work to take advantage of technologies such as Linq but still want to utilize these components in existing products, we need to manage the translation between ArrayLists and Lists via adapters.

The overall goal is to keep things as simple to implement as possible with as little excess up-front investment as possible. By breaking up applications into shareable units of work, implemented in simple and consistent ways, developers can accomplish their task while remaining flexible to future change. This is something a framework will never give them without first taking its pound of flesh.

Descriptive Code

2010-02-28T15:21:00.001+10:00

I'm not a big fan of the term "self-documenting code" mainly because it seems to light the fires under both extremes of whether it's nobler to comment everything, or wiser to insist all code expresses what it does perfectly well. As with everything, moderation is the key.

I've participated in project teams that believe that every method in every class should be commented, and in others where I was told in a code review to remove all comments. At the time I was a bit confused why I would have been told to remove comments, but over time it did begin to make sense.

The objective of self-documenting code is that the code should describe it's intent without the need for additional external documentation.

My original attitude about commenting code was that while anyone reading code should be able to understand "what" the code does, there should be comments to reflect "why" the code does what it does. However, the trouble with commenting is that as you re-factor code, the intent of the code can change from what was originally commented. "Comments Lie."

On the other end of the spectrum, commenting everything is a complete waste of time and just leads to copy & paste errors, and more lies. Why do you need a comment like this on something like a Customer Presenter?

/// <summary>
/// Saves the provided Customer.
/// </summary>
/// <param name="customer">Customer to save.</param>
public void Save(Customer customer)
{
//...
}

Ok, if your intent is to be able to produce documentation about an API, by all means comment public interface methods and comment them well; but for general code? This is just a waste of time. Most of the time when I see this in practice there are plenty of examples where the comment headers aren't accurate anyway; referencing parameters that were removed, or missing parameters that were added.

I still stand beside my original belief. Code by itself is misleading, it only tells half of the story. I can see what it does, but not why it does it. The best advice I can give regarding self-documenting code:

1. Code should be organized into distinct units of work. It should do one thing, and one thing only, even if that is just to coordinate other distinct actions. The name of that method should perfectly describe what that code is supposed to do.
For example: private void updateOrderStatusToDispatched(Order order)
This method name perfectly describes what you intend its code to perform. When this method is called it is crystal clear what you are intending to do.

2. Reduce, or better eliminate the use of conditional logic in your code. Nesting ifs/else ifs or a switch statement is a good indication that you can break code down into smaller units of meaningful work. 90% of the "if" statements I use in code are for fail-fast assert behaviour. If the method should do nothing based on the parameters it's given, return now. If a parameter isn't valid, throw an exception.

Where you do have conditional bits for a reason, be sure to use a comment to describe "why" that condition exists. Too many times I've seen code break when some conditional term was added for one particular case which was not documented, it caused a bug in another case, so the bug fix of removing that condition fixed the one case, but broke the original. You might remember why you added an "if" condition around a particular block after 1 month, or maybe 3 months, how about 6 months? two years? And what about the other poor souls that are working on the code base with you?

3. Write unit tests around the behaviour of the functionality, not specific code implementation. Something adopted from BDD (Behaviour-Driven-Design/Development) is that a unit test can represent the original functional requirement for the application. In this way, the code being tested is documented with the "why" it is implemented the way that it is.
For example:
[Test]
public void EnsureThatWhenACustomerConfirmsAnOrderTheOrderIsDispatched()

This test method would not test a method like updateOrderStatusToDispatched() directly. The update method is a private member, but it does test the behavior of when a customer confirms an order and asserts that the expected behaviour of that action is that the order should be dispatched.

Methods such as updateOrderStatusToDispatched() serve self-documentation by describing precisely what their intended behaviour. By incorporating unit tests that represent the various requirements of our application, our code is now documenting itself in ways that are virtually impossible with separate requirements and design documents. If I change the behaviour of the code unexpectedly, my unit tests break telling me I changed behaviour. If I need to change or add behaviour, the changes to my unit tests assert those changes as I re-design the code implementation. Keeping a separate "Word" document up to date for a software design is frustrating by comparison.

It's official. I HATE Reporting Services.

2010-02-24T16:45:00.000+10:00

I started working with Reporting Services about 2 months ago. Coming from a Crystal Reports background, and all of the trials and tribulations that go along with that, I was skeptical but optimistic about working with an alternative.

I put Microsoft's earlier attempt to provide a reporting solution other than Crystal with Visual Studio behind me (which ended up little more than a glorified Print Form) and dived in.

At first I was liking what I saw. Shared data sources was a little bit twitchy, but overall it worked well, and deployment to the report server was a breeze, either through Visual Studio or the web interface. Bringing up a report in a pop up window was a snap. It seemed a bit "wrong" that users would need to install something in order to print those reports. I don't see where/if/how/why HTML rendering wasn't provided. But in all, getting some simple reports out to local users was quite pleasant. Then the trolls started lurking out from under their bridge.

The first point of pain was when I had to generate a rather complex report similar to a cross-tab. Getting the data into the report wasn't a problem, but keeping the report confined to a single page proved to be a challenge. The issue was due to some form of moronic layout engine Microsoft adopted for the report designer. I had a set of text and calculated fields up at the top of the report (my report header) and a tablix control well below them. The tablix control would dynamically create columns based on the report data so it was sized quite small. For some reason the controls starting right of the tablix would shift to the right to always be printed after the end of the tablix control. I could understand this if the controls somehow overlapped, but these were up at the top of the screen. The result was having header controls pushed off onto a second page

No issue, I should be able to just create a section for the report header to stop confusing the layout with the tablix control.... There is no concept of sections in the designer.
My only option was to place these controls in the page header, which in this case is not a major issue but it doesn't give me an option to use a report header.

The real kick to the teeth with Reporting Services is the draconian lock-down making it virtually impossible to expose a report in an extranet capacity. I can understand arguments for not wanting to publish reports on the Internet, and by all means, educate developers to avoid such configuration, or how to do it securely, but to just lock the option out completely? Come on! I have an external site that *isn't* part of our network that needs to get access to the reports. No, we don't use VPN tunnelling or want to set them up as part of our network in any capacity. We just want them to access a website (which is locked down by IP filtering) and be able to click on a link to bring up a report. But it's effectively impossible with Reporting Server 2008.

IMO A reporting service should just GENERATE REPORTS. Let me take care of who has access and how they access it; Reporting services, just produce the bleeping report.

Frameworks are like marrying your Sister

2010-02-20T14:03:00.001+10:00

I came across a rather interesting article of an opinion of ORMs & DBMSs here which is a bit of a push away from the old mentality, and considering new options when it comes to persisting data.

If using a DBMS is like kissing your sister, designing & building "frameworks" should be like marrying her. It might be a bit too strong (as one should never marry one's sister) but 99.5% of the time development teams come up with the idea that "we should write a framework" they really, really should think again. I'm of course talking about development teams who's job is to write "application" software, *not* development teams who's primary goal is to write a framework. (Ala DNN, etc.)

Frameworks are a good thing when your goal is to apply a consistent approach to a problem domain. When you invest in a framework, the reward is to utilize it in as many ways, and as many times as to make the investment worth-while. It isn't worthwhile to create a framework that is used once, twice, five times, or even ten times. If the framework can be used in 100 different projects or more, *then* it is worth while. Time is the key factor in why application development teams should *never* waste time trying to write frameworks. How long will it be before that framework would be used in even 5 projects let alone 100?

Time is every framework's enemy. Technologies are constantly changing and it takes quite an effort to keep any piece of software up to date to take advantage of the current mainstream technology. If your development team will be expected to produce 10 applications (or distinct major releases of applications) over the next five years, can you afford or justify keeping all previous versions of the applications updated while you're also supposed to be implementing new functionality on new products? If you let projects rot on the vine with previous builds of the framework, you must realize that clients are going to expect to modify and enhance those applications for years to come.

There are a number of arguments I've heard in favor of building frameworks such as "code re-use". You do not need a framework to re-use code. Libraries and components are code re-use. "You can re-configure a framework or plug into it to introduce future functionality". Sure, but that requires investing today what you may not, and likely will not need tomorrow. And how much extra effort is going to be spent when it turns out what you guessed yesterday was wrong, and need to re-work or work around the framework to implement today's functionality?

Sometimes, it's just that development teams have stars in their eyes; a dream that they can write the ultimate system today that tomorrow and for years to come they can sit back smugly as new functional requests come in where they can just tweak a couple settings, and write a simple module. (They dream of writing themselves out of a job. :) But there's the kicker, "years". It will take years, many years before any framework constructed by an application team returns it's initial investment of tens to hundreds of thousands of lines of code, specific technologies and back-ends all perfectly aligned to meet today's vision. Ask any developer today whether they want to be working with .Net 1.1? Five years from now, are they going to be "happy" maintaining a .Net 3.5 framework? Or were they planning on upgrading their framework, and every application on it to date to .Net 4, .Net 5, and .Net 6 over the next five to ten years?

A framework is like building a closet organizer. If you build it for only what you need today, it would likely be a waste of space, and expensive/messy to tear out and replace when your needs change. So you plan for the future and build it to be flexible, to accommodate every thinkable scenario... Until you realize you didn't think of a place for your ties, so you're left stapling them to the wall.

A framework built in .Net 1.1 such as DNN justifies it's existence by enticing 1000 projects to use it. It evolves into .Net 2.0 and beyond on the merits and feedback from previous versions, hopefully to be used on a further 10,000 or more projects. When your goal today is to write a handful of applications, and you get the idea of writing that mondo-cool framework: Think of your sister.

Unit Testing using IoC Principles and Mocking/Stubbing

2010-02-01T10:00:00.003+10:00

Overview

This document is geared towards providing an outline to producing effective unit tests for functionality using inversion of control principles and mocking frameworks.

There are a few different, and sometimes conflicting definitions of what is a mock vs. what is a stub. Generally a mock performs validation about if/how a mocked method is called, where a stub is merely a placeholder to return a canned result to a message. For the sake of simplification, this document will use the term mock to represent both mocks and stubs. A mocking framework can easily accommodate both mocks and stubs.

Inversion of Control and Testing
One of the key benefits of adopting Inversion of Control (IoC) is the enablement of testing classes in isolation. Take for example a scenario where we have a class responsible for providing functionality around domain objects. There is an additional class that provides validation of business rules such as permissions, and another class that manages interactions with persistence.

We could write this scenario similar to below:

It’s pretty straight forward where there is a good separation of concerns between objects. But how do we test the LocationProvider or LocationValidator class? LocationProvider is dependant on LocationValidator and DataContainer. LocationValidator is dependant on DataContainer. Assuming that data container has some method of being configured with a connection string, all we can hope to do is set up a test case to ensure that DataContainer points at a test database and the test setup should probably be inserting records into that database to ensure that the expected data is present.

Another important consideration from above is that any test we would write for LocationProvider would now be dependant on the behaviour of LocationValidator. This problem is compounded as the number and depth of dependencies increases. Our test case for LocationProvider would need to set up data in order to ensure that the Validator passes, or fails as expected. What happens when the Validator rules (behaviour) change? Our LocationProvider unit tests, and any tests that are dependant on the LocationValidator or the LocationProvider will also fail because they would need to be updated to set up data according to the new rules.

This scenario may seem extreme, but it is the primary cause behind the frustration and eventual abandoning of unit testing. Tests become too complex to set-up, and far too brittle to be written early in the development cycle. It may not be as extreme as this where data is the central contention point for unit tests. Developers experience and then fear situations where changes to one class start breaking unit tests all over the project. This creates “noise” in the test suite that takes time to understand and fix all of the affected tests.

It certainly doesn’t need to be like this.

Using IoC to De-couple Dependencies
Inversion of control provides a mechanism to de-couple dependencies from one another, allowing each dependency to be tested independently. IoC can be implemented a number of ways, such as providing dependencies in the constructor, setting dependencies via properties, or passing necessary dependencies as parameters within the method calls themselves. You could define dependencies as concrete instances, but for the interest of test-ability the dependencies will be defined as interfaces so that they can be substituted. The following example changes the scenario to provide dependencies via the constructor.

The code around the functionality looks pretty much the same as the original example. The key difference is that the instances of the dependencies are now provided when each class is constructed. Now we have something we can test.

Testing in Isolation with Mocks
Lets first look at writing a test for our Validation class using Moq. (2.6) *Note: for newer versions of Moq (3.0+) substitute the “Expect” method name with “Setup”.

In this example, we’ve created a stub object for the location (1), and another for the DataContainer dependency.(2) The data container is configured to expect a meth call to “CountObjects” and is instructed to return a non-zero value. (3) We create the validator passing it in our stubbed data container. (4). The test then asserts than the validator returns back “False” based on the results.

By definition, the data container substitute that we’ve created is a stub, not a mock. The reason is that this object doesn’t validate that it was actually called. This would be better served by a true mock instance, which can be done by modifying the test as follows:

Converting this to a true mock by adding the verification is important because it ensures that the CanDelete method actually does call the data container. (The business logic says it should.) The mocking framework would automatically detect if a different method was called, or different values for parameters, but it doesn’t report back whether a call actually occurred unless we ask it.

A note regarding the naming convention for the test: This example unit test is named to reflect the behaviour, or business requirement of the code being tested. This is in the spirit of Behaviour Driven Development. (BDD) By testing around the behaviour, the unit test itself becomes part of the design for the given system. It defines a required behaviour, and as an integrated part of the project, anyone seeking to change to the requirement (test) or implemented behaviour (code) will have to ensure that the two reflect one another.

The next obvious test scenario is that CanDelete returns True if user locations are not found.

More on Mocking Frameworks

Mocks and stubs can be hand-rolled by creating test classes that implement the various interfaces. Mocking frameworks like Moq are invaluable because they hide the messy extras of implementing interfaces where you’d need to include signatures for every method in the interface. With mocking frameworks you only implement the interface methods you want to use. The IDataContainer may eventually have 20 or 30 methods in it but with the mocking framework your tests can safely be written around only the methods you care about. If you hand-roll a stub class, as you extend the interface you’ll need to continuously go back to the stubs to add place-holder methods to match the interface.

There are a number of mocking frameworks available for .Net including Rhino Mocks, Moq, NMock, and EasyMock.Net. Rhino Mocks is arguably one of the oldest and feature-rich. Moq is a “back to basics” mocking framework written around .Net 3.0 offering an elegant, simple tool for defining stubs and mocks. My argument for going with the basic and elegant option is that unit testing should strive to be as flexible and non-intrusive as possible. If tests adopt complex mocking behaviour you end up spending a lot of time trying to build tests to assert what really should be basic concepts. If you find you need to mock out complex scenarios then you either need to take another look at how you’re structuring your dependencies, or you’re testing far too deeply into dependency layers.

Testing Layered Dependencies
The real power of using IoC coupled with mocking presents itself in scenarios where dependencies are layered. Lets look at the case of testing the LocationProvider class. This class had a method to perform a delete that would ask the validator then instruct the data provider to perform an action if the validator gave the go-ahead. We already have unit test coverage for the validator to ensure that it does the right thing with inputs and outputs, so as far as tests for the location provider go, the validator can be mocked.

We create our stub object again (1) and a mock validator. (2) We instruct the validator to expect a call to CanDelete with our stubbed location, and to return True giving the go-ahead to delete. (3) We instruct the data container mock to expect a call to DeleteObject, provided with our stubbed location. (4) Note that these calls will fail if either call is made with anything other than the reference to our stubbed location. We create an instance of our provider class providing the two mock dependencies, and call the Delete method with the stubbed location. (5) The final step is to verify that our mock methods were actually called.

Notice that we don’t need to add any complexity around trying to ensure what the validator does, where the data container would be expecting count requests and such. That behaviour is already tested. All we care about is that a delete goes ahead when the validator is satisfied.

The next scenario to test is that a delete *doesn’t* occur if the validator reports back a no-go condition.

The data provider won’t be called if the validator returns false so that obviously won’t pass if we try to verify the call, so we removed that line and the verify. But wait, now this test doesn’t actually test anything other than that the call to the validator occurs. This is ineffective. What we really want to do is verify that the data provider *doesn’t* get called in this scenario:

To accomplish this, we put the .Expect call back in. But instead of doing a verify against it, we instruct it to throw an exception. This will cause the test to fail should the data container be instructed to delete an object even if the validator said no-go.

This type of enforcement with mocks is invaluable to help detect and prevent aggravating bugs from working their way into the system undetected until late in the development cycle, or worse, when the product makes it into production.

Lets look at a simple example of how this kind of thing could happen.

Cowboy Coding
Meet Cowboy Cody. Cody’s a great coder, quick as a whip. Cody starts off running with his new enhancement, to add a condition to change some rules around deleting locations. He looks at the existing code and gets as far as the Location Provider spotting the method he wants to change. “What’s this validator thing? Aw, hell, that kind of stuff isn’t necessary, all I want to do is add this one little condition. I don’t want to go through the trouble of adding a parameter so it’s going to be as easy as adding a module-level variable. Haha, 5 minutes and I’m done!”

Lets see what damage Cody’s done…

At a glance it looks harmless enough. Cody was at least good enough to go into the application and try out some of the conditions around his new 5-minute flag. He’s checked his code in.

Prudent Pat, the team lead, sees that changes have been made to the project and he immediately runs the test suite before starting his own enhancements. Red Light! “Ensure Delete Doesn’t Occur If Validation Rules Are Not Satisfied” has failed, DeleteObject has been called!

It turned out that the default for _someOtherCondition was “true”, and Cody has now introduced a behaviour change bug where his added condition can override the other validation rules. Pat consults Cody and Cody admits “Whups, that should have been an ‘&&’”

The next morning, Cody’s assigned the task of “fixing” his change properly, in the right place, and correctly ensuring the unit tests reflect the behaviour.

Conclusion
Hopefully this is a useful guide for how to effectively apply IoC principles in combination with a mocking framework to write unit tests. Writing unit tests can often be a bit of a daunting task but gradually you will find the quality of the tests you write goes up, as the time spent writing them and managing breaking scenarios goes down.

One Final Mention
Unit testing is nothing that should be dramatically increasing the amount of time you spend writing code. Every developer factors in time to start up a project as they’re working and try out their changes as they are working on them. Adopting unit tests is taking the majority of that time investment and using it to create tests that are repeatable. The initial investment is larger than the time you might spend trying out a change, but hopefully you aren’t in the habit of only trying things out when you think you’re “done”. The cost of running existing unit tests to ensure you haven’t accidentally broken anything is a lot less than manually running through existing functionality as you introduce new functionality.

Append-Only Data Models

2010-01-29T23:05:00.002+10:00

An append-only data model follows that when a data entity changes, each change is represented as an insert against the relevant table. There are no updates or deletes recorded against the data. The domain model and any adopted ORM strategy needs to reflect this behaviour difference. Soft-delete is a term used when records are marked inactive rather than permanently deleted from the database. This is an integral part of append-only models, but can be easily employed separately from append-only models.

Advantages of Append-OnlyAppend-only models offer several advantages to applications. The most significant is a built-in audit trail for reviewing all changes to records and a snapshot of a record at any point in time. This audit trail can be used for reporting, undo-chains, and review/approval behaviour.

Challenges of Append-OnlyAppend-only models automatically increase the amount of storage space needed to represent data historically. This proves to raise additional challenges when dealing with ORMs to ensure references correctly point to the latest version when applicable, and avoiding new associations with stale references.

Structuring Data for Append-OnlyThere are 3 conventional approaches for adopting an append-only data model. Each addresses the fact that for any given data entity there can be one or more record in the database to represent it over time.

Key + Version Number
The first option is to introduce a version number and form a concatenated key between the record ID and Version. New records receive the incremented version number. An optional table can be introduced to help indicate the most current version number for a data entity rather than performing index scans against the table.

Advantages

ID is preserved between versions. (Suitable for meaningful IDs)
FKs by ID are preserved. (Though not unique by themselves so caution is needed when querying.)
Ancestor records are “pure”. (Unmodified when new descendants are created.)
Old records can be archived/deleted easily.

Disadvantages

Concatenated PK.
Not intuitive to map relationships that are versioned.

Timestamps can be added in place of version numbers, or complimenting version numbers to provide functionality to assess a record’s state a particular point in time.

Ancestor Reference + Version Number
The second option is to use a version number in combination with a reference (FK) to the record’s ancestor. This approach utilizes a meaningless PK for the record so each new version has a unique ID plus a reference to the previous version. (The ancestor.) The version number is tracked as well as a reference to identify the most current version of the record.

At some point we may want to archive or discard older records. This can be a bit of a problem with the FK relationship between record and ancestor when the ancestor is removed from the production data.

Setting a revision’s ancestor reference to null is fine if the ancestor is deleted, however when archiving, special care will be needed to re-attach the reference when the revision itself is moved to the archive.

Advantages

Unique IDs suitable for FK relationship. (No concatenation needed.)
Ancestor records are “pure”. (Unmodified when new descendants are created.)
Maps easier to ORM solutions, though care is needed to ensure stale references are avoided.

Disadvantages

External FK relationships must be updated to the new version.
May require index scans like above solution to keep references fresh.
Old records cannot be easily removed due to FK relationships.

Descendant Reference
The third option is to use a reference (FK) to the record’s descendant. This is similar to the second approach however instead of the current object holding a reference to its elder, where each elder is modified to contain a reference to its descendant. The advantage in this model is that the version number is not required, as the current instance will always have a null descendant ID. This makes it simple to detect when a reference has been made stale.

It also facilitates deleting and archiving better than the ancestor model since records can be deleted without invalidating FKs. The archived records, not remaining production records, merely require a placeholder proxy for the cut off point descendant.

Advantages

Unique IDs suitable for FK relationship. (No concatenation needed.)
Simpler detection for current instance and stale references.
Old records can easily be removed or archived.

Disadvantages

External FK relationships must be updated to the new version.
Ancestor records are not “pure”. Ancestor records must be updated with their descendant reference.

Snapshots
A snapshot refers to a model where a snapshot of the data is taken and recorded before each and every data modification. This is not append-only by definition because it serves only to provide a picture of historical data, but it is not geared towards making versioned data accessible to references. Snapshots can be recorded in the same table, or in separate history tables. Snapshots can also be configured to be performed automatically by the database. This can be beneficial in situations where data can be manipulated by more than one application or process.

What Version Do I Point To?
Regardless of the approach taken, it is important that the domain model for the application is designed with append-only behaviour in mind. The most significant challenge in adopting this model is that you need to be explicit when dealing with the relationships between data. Certain design and behaviour decisions are needed within the application. If a revision is made to an object that is referenced by other objects, should those references be automatically updated to the latest revision, or is it valid that they can remain referenced to the version that was current when the association was made?

For example, if we make a change to a Doctor entity to update contact details or other relevant information, it would probably be beneficial to ensure that any appointments associated to that Dr., past, present, or future, should be updated to the current revision of the Doctor’s record. Alternatively, if we were to adjust a billing rate applicable for a service, we’d likely want to ensure that past, and possibly present appointments refer to the previous version of the record, while new and future appointments reflect the updated rate. (Possibly prompting users to ask whether or not present and future appointments should be updated or not.)

This is an important consideration when designing domain and data models around this kind of behaviour. For instance, having a service such as an Interpreter in the system, we’d want to be sure that information such as contact details would be kept separate from figures such as billing charges. Some data we may want to ensure is always pointing to the current revision, while others may be more selective.

In the case of selective association, this is one scenario where option 1 outlined above becomes a clumsy choice. Any foreign key association where the revision can be selective requires both the ID and version number. In the case of an appointment to service provider billing information, the appointment will likely have a revision number, as well as the ID to the billing information and the version of the billing information.

Dependency Injection: One step back.

2010-01-25T11:19:00.002+10:00

Ayende had some interesting feedback here from an blog from Uncle Bob about why developers shouldn't use IOC containers.

Most modern IOC containers can be configured to auto-inject dependencies via constructor injection. The benefit of using this approach is that it's essentially configure & forget. Provided that you don't mess up the configuration, the container should continue to detect and provide dependencies. There are two things that I don't really like about this though:

#1. From a test-ability perspective, you need to provide mocks or stubs for all dependencies when testing code.

#2. When things get a bit out of sync you're chasing Null Reference Exceptions, and I *loathe* NullReferenceException.

An alternative that I've been exploring is going back to using the IOC container as more of a service locator/provider via Property-based dependencies. This is kept nice and tidy using a partial class for the dependency definitions.

The benefit of this approach is that my unit tests only need to provide dependencies that are actually used for the functionality being tested. Dependencies can also be effectively "lazy-loaded" as necessary since they will be located and provided only the first time they are referenced. The second part to this is to be sure that the IOC container is covered by a facade. This keeps out the tight coupling to a single IOC provider, and allows you to be sure that test fixtures configure a stub container that throws exceptions to expose unexpected new dependencies.

I find this approach also blends into legacy code a lot easier than constructor injection. Existing legacy code can construct instances of re-factored classes as per normal. The only re-factoring you need to do is to strip out the "newing" and substitute it with the IOC provided properties.

The cost of TDD / Test-soon-development.

2010-01-24T20:55:00.003+10:00

A common argument against writing automated unit tests is that no one has time to write them. Ultimately this is not true, in that writing code with unit tests takes no longer than writing code without them.

Without unit tests devs still need to assert that their code works. This typically means writing the code, then launching the application and ensuring the code does what it’s supposed to do. This is then repeated as you continue to implement functionality. Unit testing is written using the time they would normally spend running the application to assert that the code works. The real difference appears when you start looking at continuous revision. As you revise code, are you always going back over the complete functionality to ensure your new code hasn’t broken any previously verified behaviour? Honestly, you should be, but think how much time that would take.

As you begin to invest pieces of time writing unit tests instead of manually firing up applications to one-time test, you are still asserting your code will do what you expect it to do, and you gain that regression assessment for free.

For the record, my acronym for testing style would be BDTSD. (Behaviour Driven Test Soon Development:)

Starting small on the Blog scene.

2010-01-23T11:03:00.000+10:00

Well, after reading what several others have to say in the software development circles I've decided to start a small blog about aspects of software development that matter to me. This will primarily be in the realm of software design, Agile software development, software usability, and testability.

Py's Sty

Quick tip, Respecting your own contract.

Refreshing Settings and dll.config Files

Something's fishy with Google...

What is my client's/employer's IP?

Windows 8, Vista All Over Again.

A recent example of the value and purpose of unit tests.

Unit Tests Aren't Expensive

How XP can help your solo projects too.

Pair Programming.

Business Analysts in the mist.

Business software should advise

How to hire a "Senior Software Developer".

Pet Peeve: Misuse of KeyValuePair

Getting .Net Property Names without Magic Strings

Around 6+ hours I'd love to get back.

3 years, $1M

Ouch, 3-day bug!

Contract vs. Permanent

Confused the compiler/runtime or what?

Beware! The clipboard may lead you to question your sanity.

Keeping up is fun.

Amber, Green, Re-factor

Unit Testing for the QuantumBits ObservableList

Linq2SQL Cache & Connectivity

Rolling in the mud.

SQL Count by criteria.

Thou shall not transmit localized DateTime values.

Linq2SQL is *NOT* an ORM

When configuration goes pear-shaped.

Using consistency's name in vain.

Explicit Interfaces

Regex is dead! Long live the Lambda!

Divorcing the Framework

Descriptive Code

It's official. I HATE Reporting Services.

Frameworks are like marrying your Sister

Unit Testing using IoC Principles and Mocking/Stubbing

Append-Only Data Models

Dependency Injection: One step back.

The cost of TDD / Test-soon-development.

Starting small on the Blog scene.

Linq2SQL is NOT an ORM