Contract vs. Permanent

This is an odd one that I come across from time to time, and again just recently.
I work as a contractor for a client and they've expressed an interest in ensuring stability within their development team, so they had asked if I would be interested in taking a permanent position. I politely declined since the benefits of a permanent position don't coincide with my own objectives. I can definitely understand the desire to have some stability within a development team, and since I don't have any serious problem with how things are arranged, they have my assurances that I will be available essentially for as long as they need me.

The odd thing is that in talking with permanent staff, the impression of a contractor is that they're someone that can just vanish tomorrow; that being a "permanent employee" somehow indicates that the company has an investment in you, and you have an investment in them. However, when you look at the facts this doesn't hold any water, especially in the I.T. industry. How does a company invest in you? "Permanent" implies that you will always have your job through hell and high water. However every permanent employee has the exact same termination clause in their contract as I have in mine. You can be "let go" or decide to leave with 4 weeks notice. You could argue that a permanent employee would have some additional protection in the form of unfair dismissal, but no more-so than I would if I were given a six month contract which the client tried to end after just two. In either case it will be an ugly business to dispute. Companies will often offer training to permanent employees. However, in the over 15 years I have been in software development in 10 organizations/clients (7 of which as a permanent employee) I have received training twice. (1x 1-day course on advanced VB, and a 2hr course on time management.) I have only rarely seen other permanent employees actually get sent off for training or seminars. Depending on your client, a contractor can be treated to other benefits that are given to permanent staff such as inclusion in company dinners, and gifts of appreciation.

Contracting also has a bit of a myth associated with it that often rubs permanent staff the wrong way: Contractors get paid more. False, with a grain of truth. Being a contractor, especially on long-term contracts (>6 mo.) means you are exchanging most of your entitled benefits for cash. Annual leave, sick leave, paid holidays, etc. This means if you work for a full year, not taking any sick days off, as a permanent employee you are entitled to an extra 20 days pay. If you don't use it, those days are paid out when you leave and taxed at the maximum tax bracket. You won't get that extra tax back until the end of the tax year. (So in Australia, be sure to quit that permanent position in June.) In the I.T. industry and especially in smaller organizations, paid leave is rarely "convenient" to take. You're always busy, and you know that any leave you take is going to be a significant burden on the company, your co-workers, and extra urgent work to worry about when you get back. Some permanents I've worked with try to plan their vacations after the next major release, only to find that the schedule slips and that leave they've booked for months in advance now falls smack in the middle of the production roll-out. Permanent positions are kind of like an insurance policy. If you're sick or want to take some time off, your income flow stays stable. But if you don't use that, well, you might be better off with the money up front. Short term contracts will result in marginally higher rates simply because a contractor must factor in the time they need to take in-between short-term contracts in order to secure more work.

The reason I contract as a sole trader is to maximize my income to minimize my debts. There are a number of factors that I considered to reach this:
1) I have a sizable mortgage with an offset account. Offset interest savings are 100% tax free, and are valued directly against the current interest rate. (Where any other investment earnings contribute to and are taxed at your overall tax rate and Super contributions are taxed at 30% with super earnings taxed at 15%)
2) As a sole trader I invoice my client directly and weekly, collect GST and and pay it back quarterly, pay income tax annually, and do not contribute to Superannuation. (And soon I will be contributing to it again annually.)
This maximizes the amount of money I keep in my offset account working to reduce the interest my mortgage is adding up on a daily basis.
3) I'm rarely if ever sick, so sick leave isn't any value to me. (Since it's not paid out if unused, and I'm generally too ethical to "chuck a sickie".)

Because of this, contracting under this arrangement will help pay off my mortgage many years faster than if I was a permanent employee.

Confused the compiler/runtime or what?

This was an innocent enough of a typo but it had me questioning whether the framework's string.Split() method was working with the remove empty entries option.

Line in question:
var parameterPairs = parameterValue.Split(new char['&'], StringSplitOptions.RemoveEmptyEntries).ToList();

The parameterValue being parsed was "&Token=8765"
I expected to get a result back of "Token=8765", but instead got back a value of "&Token=8765".

I was thinking was there a problem with RemoveEmptyEntries when the separator was the first character? Was there a problem with '&' as a separator? Both checked out. Then I noticed the typo. It should have been:

var parameterPairs = parameterValue.Split(new char[] {'&'}, StringSplitOptions.RemoveEmptyEntries).ToList();

And presto, the test passed. But that got me wondering, what was char['&'] resolving to? Why did the compiler allow it to compile, and why did string.Split() seem to merely ignore it at runtime?

I put in a precursor condition:

var test = new char['&'];

along with a breakpoint.

Curious...

Beware! The clipboard may lead you to question your sanity.

This one had me scratching my head for a good while. I was able to fix it easily enough, but getting to the bottom of just what the hell happened took a bit of digging.

The problem started because I wanted to switch a test against a certificate located on the disk to using a certificate located in a certificate store. For this test I decided to pull the certificate by serial number so I snatched the serial number from the certificate properties easily enough.

I tried this as-is and it wasn't working. So I tried removing the spaces, still no joy. So I grabbed the entire list of certificates on the machine and found the certificate to verify the serial #. The results showed the serial number in upper case, so I quickly popped a .ToUpper() on my key. NO-JOY. Ok, WTF. So I copy the serial number out of the watch results in the IDE and try again, it works.

But I couldn't leave it at that. What was wrong with my original value? I restored the changes and created a second set of checks (See first screen) and started cursing my monitor in disbelief. I lined them up, reversed the order they were called in the Find() statement, and cleaned and rebuilt the test application several times. String.Compare returns a match, .Equals returns not equals. The call to the store to Find returns a result for only the one value, regardless of which order they are executed.

The only possibility: copy and pasting the original value from the certificate dialog resulted in some unprinted Unicode character being embedded in the original string. (serial2) String.Compare ignores it, but .Equals does not. (The test1/test2/result check was just me testing my sanity that .Equals on string wasn't doing a reference equals.) To quote Sir Arthur Conan Doyle: "Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth."

Keeping up is fun.

Software development is constantly evolving. Years ago there would be years between product releases, and while there were libraries popping up every so often, there were typically a lot fewer to follow for any given project you might be working on. The pace of change was generally slower.

Today, change is constant and occurring at a significantly faster pace.On top of that, the open-source movement has meant a lot more releases of a lot more useful tools to incorporate within a project. It's easy to get left behind, or running after new versions without a chance to stop and smell the roses.

A perfect example. nUnit is a brilliant unit test framework port for .Net. I've used it for many years, but I've never really given each new version more than a passing glance. Unitl I caught a mention of a new attribute called "[TestCase]". (As opposed to the "[Test]" which denotes a test case.)

Take a case where you want to test a range of outputs such as a minimum, maximum, and assortment of values to ensure a method is working properly. This might be 4 tests or more, consisting of setup, call, and assert. For example I had 3 tests that verified formatting behaviour for a currency converter. Positive value, negative value, and zero. Scenarios like this can now be replaced by something like this:

        [TestCase(1000d, Result = "$1,000.00")]
        [TestCase(-1000d, Result = "-$1,000.00")]
        [TestCase(0d, Result = "$0.00")]
        public string EnsureConverterFormatsCurrencyValues(decimal value)
        {
            IValueConverter testConverter = new DecimalToCurrencyConverter();
            return (string)testConverter.Convert((decimal)value, typeof(decimal), null, CultureInfo.CurrentUICulture);
        }
*Sigh*... Those roses sure smell nice when you finally get a chance. Not only is it one test case (executed and reported 3 times) but it doesn't even need the Assert.Equals line. It would have been nice to have spotted this a year ago when they added it. :)

*Note: The above declaration is actual code, not a typo. The parameters on the attribute are provided as doubles (d) not decimal (m) even though the parameter type is decimal. Decimals cannot be provided via attributes (a limitation in .Net) so TestCase facilitates this translation.

Amber, Green, Re-factor

Red, Green, Re-factor (Red-Green for simplicity in this entry) is the mantra of Test-driven development. (TDD) The goal being to satisfy a requirement you write a test that fails, then write the functionality to satisfy solely that test, then re-factor as you implement further functionality. There's definitely nothing wrong with this approach, but I don't think it's for everybody. An alternative to TDD is Test-soon development. (TSD) In this model you write tests soon after you're satisfied that target requirements have been met. However, TSD is still Test-driven development, it just isn't Red-Green. TDD is test-driven because design decisions place an emphasism on meeting requirements with testable and, most importantly, tested code. This is obvious when you write the unit tests first, but is no-less true when writing unit tests second. The key is that the code is written with tests in mind. When you choose to write them is irrelevant, so long as you test it right. 
 
TSD is less disciplined than TDD in that it is easier to defer testing as you continue to implement related functionality. This accrues up technical debt, leaving a larger pile of unit testing required. Tools like NCover are excellent at tracking your technical debt bill, so if you're using TSD, NCover is an absolute must. Another negative point against TSD to the TDD purist is that unit tests against working code would be written to pass, so they would be less reliable since they may not actually fail when they need to. This is a significant concern, but I think it applies just as much to Red-Green TDD as it does to TSD. In Red-Green you might only write tests that satisfy the "chosen path" but not exercise the code properly when paths cross. Either way you need to really think about behaviour to test, not just satisfy the obvious aspect of a requirement.

This second concern of writing tests to fail really got me motivated to ensure tests were exercised fully along-side the code they intended to test. My motto of choice to this effect is Amber, Green, Re-factor or Amber-Green. Essentially I write code to satisfy a requirement, then I create tests, and excercise test & code (Amber) until I'm satisfied they pass (Green) then the code is ready to be re-factored as necessary for future functionality. The reason I prefer this to TDD is that I've invested the time into satisfying architectural decisions that often inter-twine with satisfying requirements. I'm satisfied not "that it works", but rather "that it's going to work". Now the focus shifts to developing the tests to ensure that it truly works. My goal with writing the unit tests is to break the code. I know for a fact that I didn't think of everything so it's time to find what I missed by probing the code with tests. When I write a test that passes first run I alter the code and make sure the test fails as it's supposed to. When I write tests that don't fail then I become concerned that I'm not exercising it fully. Even if I think of something I missed, I write the test, watch it fail, then fix it.

Ultimately the approach isn't much different to Red-Green. With Red-Green it pays to think up-front while fleshing out boundaries and behaviours for code before you write it. Personally I find it's easier to flesh this out once I know what I'm planning to take to production, and switching my mentality to "break it!" helps to really excercise the code.

So the next time the TDD purist on your team scorns you for not writing tests first, rest assured you can still be test-driven with Amber-Green; Just keep a close eye on your technical debt and break that code good.

Unit Testing for the QuantumBits ObservableList

In using the ObservableList example from QuantumBits I came across a bit of a problem. This list ensures all event notifications happen on the UI thread by using the dispatcher. One change I added was to expose virtual methods in the ObservableList to "hook in" custom actions such as a flow-on effect for calculations as items in the list are changed. (OnItemAdded, OnItemRemoved, OnPropertyChanged etc. )I have extended functionality that interacts with these events to run calculations and I wanted to unit test this behaviour.

An example test:

[Test]
public void EnsureThatRelativeRateIsUpdatedWhenEarlierInterestRateChanges()
{
  var mortgageInterestList = new MortgageInterestList();
  var first = new MortgageInterest.Absolute(0.05m, DateTime.Today);
  var second = new MortgageInterest.Relative("1%", DateTime.Today.AddDays(2));
  mortgageInterestList.Add(first);
  mortgageInterestList.Add(second);

  first.UpdateInterestRate("+0.5%");
  Assert.AreEqual(0.055m, first.InterestRate, "Interest rate was not updated.");
  Assert.AreEqual(0.01m, second.InterestDelta, "Second's delta should not have been updated.");
  Assert.AreEqual(0.065m, second.InterestRate, "Second's interest rate should have been updated.");
}

An absolute interest rate is such as when a user says the interest rate was 5% as-of a specified date. A relative interest rate is one where the user specifies a +/- delta from any previous interest rate. The idea being that if an interest rate changes right before a relative rate, the relative's delta should be re-applied to adjust it's rate.

The problem is that while the MortgageInterestList (extending ObservableList) is wired up with the event to perform the calculation, running in a unit test the dispatcher doesn't actually run to set up the event hooks so that the edit triggers the flow-on effect.

The problem stems from this in the ObservableList:

public void Add(T item)
{
  this._list.Add(item);
  this._dispatcher.BeginInvoke(DispatcherPriority.Send,
    new AddCallback(AddFromDispatcherThread),
    item);
  OnItemAdded(item);
}

This method is perfectly correct, and in an application utilizing this functionality it works just great. However, unit tests don't like this kind of thing. The callback is added, but even with a Priority of "Send" (highest) the hookup doesn't actually happen during the test execution. The event doesn't fire, the rate isn't updated, the test fails.

After a lot of head scratching and digging around on the web I came across the solution on StackOverflow. Ironically while someone was having a similar problem to me with testing around Dispatcher-related tasks, this wasn't the accepted solution, but it worked a treat.
2nd Answer

public static class DispatcherUtil
{
  [SecurityPermissionAttribute(SecurityAction.Demand, Flags = SecurityPermissionFlag.UnmanagedCode)]
  public static void DoEvents()
  {
    DispatcherFrame frame = new DispatcherFrame();
    Dispatcher.CurrentDispatcher.BeginInvoke(DispatcherPriority.Background,
      new DispatcherOperationCallback(ExitFrame), frame);
    Dispatcher.PushFrame(frame);
  }

  private static object ExitFrame(object frame)
  {
    ((DispatcherFrame)frame).Continue = false;
    return null;
  }
}

Basically it's a mechanism that you can run inside a test to signal the dispatcher to go ahead and process anything currently pending. This way the events will be wired up before the test resumes. Damn simple and it works.

The test in this case changes to:

[Test]
public void EnsureThatRelativeRateIsUpdatedWhenEarlierInterestRateChanges()
{
   var mortgageInterestList = new MortgageInterestList();
   var first = new MortgageInterest.Absolute(0.05m, DateTime.Today);
   var second = new MortgageInterest.Relative("1%", DateTime.Today.AddDays(2));
   mortgageInterestList.Add(first);
   mortgageInterestList.Add(second);
   DispatcherUtil.DoEvents();

   first.UpdateInterestRate("+0.5%");
   Assert.AreEqual(0.055m, first.InterestRate, "Interest rate delta change was not based on the previous interest rate.");
   Assert.AreEqual(0.01m, second.InterestDelta, "Second's delta should not have been updated.");
   Assert.AreEqual(0.065m, second.InterestRate, "Second's interest rate should have been updated.");
}

SO if "jbe" should ever read this, thanks a bundle for the solution!

I've never been a fan of DoEvents coding in VB which this is pretty closely mirroring so this little gem will be located in my Unit Testing library where I have my base classes for uniform test fixture behaviour.

*Edit* I didn't much like the name of the class when I incorporated it into my test library. The updated signature is:

public static class DispatcherCycler
{
  [SecurityPermissionAttribute(SecurityAction.Demand, Flags = SecurityPermissionFlag.UnmanagedCode)]
  public static void Cycle()
  {
    DispatcherFrame frame = new DispatcherFrame();
    Dispatcher.CurrentDispatcher.BeginInvoke(DispatcherPriority.Background,
    new DispatcherOperationCallback((f) =>
      {
        ((DispatcherFrame)f).Continue = false;
        return null;
      })
      , frame);
    Dispatcher.PushFrame(frame);
  }
}

Linq2SQL Cache & Connectivity

One "gotcha" you will need to consider when using Linq2SQL is how to handle situations where a connection is lost at a moment when a SubmitChanges call is made. (Basically the period of time between when the last object reference is retrieved from the cache, and SubmitChanges is called.)

The SubmitChanges call will fail, expressing what is wrong, however the cached copies of the data will hold onto their changes until they are committed, or refreshed. This can have some truly confusing side-effects that can be a lot of fun to track down.

Case in point: I recently had to track down one of these issues in a simple barcode stock application. It basically consisted of orders and boxes. As boxes of stock come in, they are scanned with a barcode reader, looked up in the DB using Linq2SQL, have their statuses validated, and updated. As each box is scanned in, the order is also loaded to inspect whether or not all of its boxes have been scanned into stock, and when they all are, the order is updated to "complete". The bug was that every so often (once every several weeks or more) we got a situation where an order was found to be complete while one of its boxes was not recorded in the DB as "in stock". We eliminated every possibility, scoured to code for status change bugs, but there was nothing. Scanning exceptions were already brought up to the screen, but operators typically dealt with several scan issues (re-scanning boxes after being interrupted, etc.) so they didn't notice anything odd. After extending the logging support to see what was going in I found that the box was scanned, an exception occurred, but when re-scanned it said it was in-stock, and after all remaining boxes on the order were scanned, the order found all boxes to be in-stock, and diligently closed itself off.

The issue was Linq2SQL's caching behaviour. The exception was occurring when the box's change was submitted to the database. Any problem with the connection to the server (This was a client running in Victoria hitting a DB in Queensland) and the SubmitChanges would fail, but the updated objects remained in the cache. Since the same data context was used for all boxes scanned in during a receive stock session, further requests for that box retrieved the cached copy.

This is a bad scenario to be in. Your options are either to retry the submit changes until it either goes through, or you try to revert the changes until they revert, or you force the application to shut down if neither of those work. I was reporting back to the user that there was an error, and IMO if there is a problem then the user should retry what they were doing, I.e. re-scan the box, so I wanted to revert the changes. If the data source still wasn't available then the user should shut down the application and contact support.

The first solution that came to mind was to catch the exception from SubmitChanges and refresh the affected object graph. Since the box had child entities that were also updated, plus an order line item, this entire graph would need to be refreshed. Easy enough, as objects are queried and updated, they got added to a revertList should the SubmitChanges fail:

try
{
    DataContext.SubmitChanges();
}
catch
{ 
    DataContext.Refresh(RefreshMode.OverwriteCurrentValues, revertList);
    throw;
}

Unfortunately this doesn't work as the Refresh method will fail with:
System.InvalidOperationException: The operation cannot be performed during a call to SubmitChanges.
at System.Data.Linq.DataContext.CheckNotInSubmitChanges()
at System.Data.Linq.DataContext.Refresh(RefreshMode mode, IEnumerable entities) ...

So I needed to trigger the Refresh call outside of the SubmitChanges scope. The first thing to note is that while the system is processing, the scanner is turned off until the processing is completed. For minor errors such as scanning a box twice, etc. the scanner beeps out, displays a message, and reactivates. For serious application errors the scanner remains off until the application determines the exception can be corrected. The user knows that if the scanner beeps out and turns off, check the screen, and try restarting the application if it doesn't turn on within a minute or so.

In any case, the solution was to use an asynchronous operation to handle the scanner error and data refresh. This meant worker thread and locking off the scanner access until the refresh was successful, or timed itself out trying.

try
{
  DataContext.SubmitChanges();
}
catch
{ 
  revertDomainObjects(scanner, revertList);
  throw;
}

/// <summary>
/// This method will attempt to revert changes to objects to reflect the data
/// state. This will be called in situations where the data connectivity has 
/// been interrupted, where data state does not reflect cached state. This will
/// attempt to refresh cached copies of objects to match data state.
/// </summary>
/// <param name="revertList">List of domain objects to revert back to data state.</param>
private void revertDomainObjects(IScannerHandler scanner, List<object> revertList)
{
  ThreadPool.QueueUserWorkItem((e) => 
  {
    lock (_dataContext)
    { 
      var inscopeScanner = ((List<object>)e)[0] as IScannerHandler;
      var inscopeRevertlist = ((List<object>)e)[1] as List<object>;
      int attemptCounter = 0;
      bool isSuccessful = false;
      while (!isSuccessful && attemptCounter <= 4)
      {
        try
        {
          _dataContext.Refresh(RefreshMode.OverwriteCurrentValues, inscopeRevertlist);
          isSuccessful = true;
          inscopeScanner.RecoveredFromCriticalException();
          scanner.Status = Objects.ScannerStatus.Idle;
          scanner.StatusMessage = "Scanner has recovered from the previous error. Please retry operation and contact Licensys I.T. if the issue continues.";
        }
        catch
        { // If the rollback fails, sleep for half a second and retry up to 5 times.
          Thread.Sleep(500);
          if (attemptCounter++ > 4)
          throw;
        }
      }
    }
  }, 
  new List<object>() {scanner, revertList});
}

Probably a better solution would be to use a Task<> implementation if it's available in .Net 3.5, but this does the job and I don't need to be able to interrupt it.

Now in the case of an exception during SubmitChanges, the scanner will alert the user and be de-activated (triggered when the exception bubbles)and the worker process will kick off to try and refresh the affected data. if that is successful the scanner will be re-activated and the user notified that they can continue.

Caveats: This did require adding some thread-safety to an otherwise single-threaded application. I added an accessor property that attempted to lock the data context before returning it. Not ideal but in this case the means of triggering calls to the data context (scanner) was disabled until the operation was completed. This will be reviewed and refactored shortly as we move to support multiple simultaneous scanners.