Shrinking JSON for Data Storage

 Lately I have been working on a pet project for helping visualize how an offset account impacts one's mortgage. This has mainly been a foray into building apps in Blazor. The goal is to have a version of the app that can run in a fully offline mode leveraging local storage, which I have used Blazored.LocalStorage to assist with.  Behind the scenes it is serializing objects to JSON to be stored in the local storage which works just fine.

However, this did raise a bit of a concern in that the data being stored would be entered over years worth of entries to track transaction amounts to compute the offset impact. Each record is reasonably small, but there will potentially be a fair number of them, and I'd like the option to include things like descriptive names. JSON itself adds additional storage concerns as the objects and all values need to be represented as strings. With databases in modern systems, space is cheap for storage and indexing. With local storage we are limited to 5MB for storage. This means we need to be a bit more cautious with our data.  This turned into a rather interesting problem to find a balance between ease of use and storage space.

The first thing I looked at was ensuring that I was only storing info that was absolutely needed. With a relational database a Transaction record might look something like:

TransactionId, TransactionDate, AccountId, Amount, Description

With this offline store, it resembles more like a document store where the transactions will fall under an account document. We still want an Id for the transaction within the document, but we can get rid of the FK.  Now when it comes to the description, many rows will be null, but for ones that people fill in, there is a good chance that many will repeat. For this I decided to normalize by adding a TransactionDescription table. This replaces Description with TransactionDescriptionId, being an integer rather than the string in each row. As users start typing we can look up common descriptions and offer to link to an existing one or create a new row for new values.

When it comes to dates I decided to treat these as raw values with an unmapped transalation to a C# date. So dates would be stored in a short-form ISO format of "yyyyMMdd".

For dollar amounts I elected to store these in cents, letting the DTO expose a floating currency value.

So a typical record would start to look like:

{ "TransactionId":1, "TransactionDate": "20251012", "Amount": 1545, "DescriptionId": 32 }

... where if the transaction had no description:

{ "TransactionId":1, "TransactionDate": "20251012", "Amount": 1545, "DescriptionId": null }
91 characters.

This is already a bit more compact than if I took a more default C#, denormalized object serialized to JSON. Though we can do better. For a start, when serializing to JSON there is an option on how #null values are handled, either included or excluded. For data contracts it makes sense to include them so consumers know they are a value in the destination object. For data storage though we can ignore them. In the second case with no description this becomes:

{ "TransactionId":1, "TransactionDate": "20251012", "Amount": 1545 }
68 characters.

Next, we can use the [DataContract] and [DataMethod] attributes. Rather than the default opt-out serialization behaviour where we would need [JsonIgnore] to exclude properties we don't want serialized, if we use [DataContract] we need to opt-in properties to include with [DataMethod] but we can also specify a value name. For instance with my above simple transaction, the DTO looks something like:

    [DataContract]
    public class Transaction : IDto
    {
        [DataMember(Name ="id")]
        public int TransactionId { get; init; }

        [DataMember(Name="dt")]
        public int TransactionDateRaw { get; init; }
        public DateTime TransactionDate => DateTime.ParseExact(TransactionDateRaw, "yyyyMMdd", null);

        [DataMember(Name ="am")]
        public long Amount { get; init; }

        [DataMember(Name ="td")]
        public int? TransactionDescriptionId { get; init; }
    }

The JSON now looks like:

{ "id":1, "dt": "20251012", "am": 1545 }
40 characters.

We have gone from 91 characters to less than half at 40 characters. 

The next change I made was around the date. I was going to want to provide daily figures for things like the balance, while supporting that multiple transactions on a given day. When it comes to loans with an offset account, the general recommendation is to use the interest free period on credit cards for daily transactions and pay the card in full each month. This reduces reporting on each transaction but we'll still have things like bill payments and other transactions falling on the same day. The structure I ended up with pulled the date value to an AccountDay class containing the starting and final balance with a collection of transactions. This pulled the repeated date out of the transaction rows.

The offset account manages a SortedList<int, AccountDay> collection of days indexed by the date (yyyyMMdd format)  The resulting JSON looks like:

{"na":"Test Account","dz":{"20251001":{"ib":5000,"tx":[],"ba":5000},"20251002":{"ib":5000,"tx":[{"id":1,"am":44400},{"id":2,"am":29153},{"id":3,"am":17166}],"ba":95719},"20251003":{"ib":95719,"tx":[{"id":1,"am":39177},{"id":2,"am":39521},{"id":3,"am":27103},{"id":4,"am":41982}],"ba":243502},"20251004":{"ib":243502,"tx":[{"id":1,"am":43598}],"ba":287100},"20251005":{"ib":287100,"tx":[],"ba":287100},"20251006":{"ib":287100,"tx":[{"id":1,"am":12655},{"id":2,"am":9537},{"id":3,"am":38402}],"ba":347694},"20251007":{"ib":347694,"tx":[{"id":1,"am":46109},{"id":2,"am":43684}],"ba":437487},"20251008":{"ib":437487,"tx":[{"id":1,"am":11688},{"id":2,"am":2481}],"ba":451656},"20251009":{"ib":451656,"tx":[{"id":1,"am":40970}],"ba":492626},"20251010":{"ib":492626,"tx":[{"id":1,"am":32517},{"id":2,"am":20656},{"id":3,"am":17146}],"ba":562945},"20251011":{"ib":562945,"tx":[{"id":1,"am":1350},{"id":2,"am":21341}],"ba":585636},"20251012":{"ib":585636,"tx":[],"ba":585636},"20251013":{"ib":585636,"tx":[],"ba":585636},"20251014":{"ib":585636,"tx":[{"id":1,"am":41260},{"id":2,"am":31053}],"ba":657949},"20251015":{"ib":657949,"tx":[{"id":1,"am":47606}],"ba":705555}}}

It isn't very readable by any stretch, but it is compact and deserializes back into the object model where all of the work & formatting is done.

My thoughts are that I'll load and save account data in parcels by month or by year, depending on the final data size and trying to ensure this tool remains device friendly. For instance if data is automatically parcelled and serialized by "yyyyMM" then the "days" key shrinks from "yyyyMMdd" to just "dd", saving even more space. 

I could have left it at this and let Blazored.LocalStorage serialize this structure, and I'd likely have enough local storage for the decades of data for the typical user, but Blazored.LocalStorage can also work with storing string values which got me thinking: What if I serialized and compressed the data? Compression should significantly shrink the JSON, but then I'd need to Base64 the string, adding about 1/3rd to the compressed size. From here I wanted to see what a typical use case might look like so creating around 15 years worth of transactions. The test generator built anywhere from around 35,000-50,000 records. With the original JSON objects, this amounted to a size of around 1.3MB. The shortened JSON properties dropped that to around 545kB. The last step was to see what compression might do.

    public static string? Serialize(this IDto dto)

    {

        try

        {

            string json = JsonConvert.SerializeObject(dto, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore });

            var buffer = Encoding.UTF8.GetBytes(json);

            using var fromStream = new MemoryStream(buffer);

            using var toStream = new MemoryStream();

            using var zipStream = new DeflateStream(toStream, CompressionLevel.Optimal);

            fromStream.CopyTo(zipStream);

            zipStream.Flush();

            string base64 = Convert.ToBase64String(toStream.ToArray());

            return base64;

        }

        catch (Exception ex)

        {

            return null;

        }

    }

    public static T? Deserialize<T>(string dataBase64) where T : IDto

    {

        if (string.IsNullOrEmpty(dataBase64)) return default;

        try

        {

            using var toStream = new MemoryStream();

            var data = Convert.FromBase64String(dataBase64);

            using var fromStream = new MemoryStream(data);

            using var zipStream = new DeflateStream(fromStream, CompressionMode.Decompress);

            zipStream.CopyTo(toStream);

            zipStream.Flush();

            string json = UTF8Encoding.UTF8.GetString(toStream.ToArray());

            return JsonConvert.DeserializeObject<T>(json);

        }

        catch (Exception ex)

        {

            return default;

        }

    }

With this I was able to get the data size down to 98kB using DEFLATE/GZip including the Base64 conversion. This also means I can easily package up and offer to save a backup of data from the local storage on request, and re-import into local storage.  

The next question was total time needed to serialize/deserialize with and without compression. Times did vary between runs but the general impact was that compression was going to add around 20% to the read and write times. This was while running on a PC so I'm far more cautious about this impact when running on a device once I have a suitable build ready for testing. However, this was reading and writing 15 years worth of data in one hit. Even in that situation it was adding around 100ms to a 500ms operation. The goal is to parcel data by month or by year worst case where the complete data set doesn't need to be loaded.

In any case this provided a lot of food for thought when thinking about JSON serialized data for data communication and storage as opposed to a more contractual or otherwise visible medium. When we want to package up data tightly for storage or transport then we can leverage options in the JSON serialization as well as compression when we have compute available on the consumer. The trade-offs to balance besides space/size and performance also include the complexities of working with the data and potential for bugs. (For instance if I move the "yyyyMM" aspect out of the AccountDay and rely on the loaded month "packet" of data.

When will AI's "Monsanto Moment" come?

 I cannot help but see parallels between the introduction of AI into software development, and the rise of gene modification in crops. Crops have had their genes modified through domestication for centuries, but the rise of corporations like Monsanto popularizing lab-based manipulation led to the patent system being used to protect that investment.  Back in the early 2000's there was a big push to try and scare consumers about supposed risks of GMO crops, and a lot of that fear carries over today with consumers demanding non-GMO produce on the assumption that it is healthier. The reality behind that push wasn't around health and safety, it was around money and the use of these patents to force farmers to change their practices. Prior to GMO crops, farmers would buy seed stock, and if they had a good harvest they could opt to save some of it as seed for the next year. When they bought GMO seed stock, stuff that was resistant to insects or to herbicides, etc. they sign a contract that bars them from being able to keep seed stock. Each year they would need to buy the full allotment of GMO seed. The company would even go so far as to sue adjacent farmers that might benefit from cross-pollination from GMO fields.  This rather heavy-handed treatment of farmers wasn't likely to garner much sympathy from the general public who only stood to benefit from better supply from the increased yields. Still, to attack back at these corporations, supporters attacked GMO any way they could.

Today, AI tools are becoming increasingly available, at low cost or even free. People have started asking the questions around ownership, both in terms of the licensing/copyright of the code these AI tools have been trained with, and the code they in turn generate. For now, companies like Microsoft will claim if you use an AI tool and in turn get challenged by a copyright holder for a violation, they've got your back, while at the same time what code the tool generates is your IP.. But for how long? Monsanto didn't start from scratch with the gene makeup of the crops it improved. Generations of botanists, farmers, etc. experience and cross-breeding had supplied the current base. Much of that was done for little more than recognition for their work, out in the public domain, expecting that future efforts for improvement would remain freely available. That is until big industry finds a way to patent it's work, commercialize it, and take ownership of it. When it comes to business, it is a lot like fishing. When you first sense a fish is biting at your bait, you need to resist the urge to yank the line or you pull the hook out of its mouth. Fish can be clever and grab the edge of a bait and waiting for a free meal. No, instead you wait patiently until the fish is committed, then pull and set the hook. Once they could convince the patent office and courts, the hook was firmly set.

Today, companies like Microsoft have invested a good deal of money into developing and marketing these AI tools. They are putting their lines out in the water with tasty bait offering to help companies and development teams produce better quality code and products faster than ever. They are being patient, not to spook the fish. Today you own what the tool generates, but soon, I'd wager, companies like Microsoft will set that hook, and like Monsanto, demand their share of the value of the yield you produce directly, or indirectly from their seed; By force. How that exactly shapes up, we'll have to see. Perhaps terms that once you start development with AI assistance you cannot "opt out"? Or will they demand part ownership on the basis that their tools generated a share of the IP in the end product?

Why I don't Grok AI

 I'm a bit of a dinosaur when it comes to software development. I've been on the rollercoaster chasing the highs from working with a new language or new toolset. I've ridden through the lows when a technology I happen to really enjoy working with ends up getting abandoned. (Silverlight, don't get started, for a website? Never, for Intranet web apps? Chef's kiss)

I'm honestly not that worried about AI tools in software development. As an extension to existing development tools, if it makes your life simpler, all the more power to you. Personally I don't see myself ever using it for a few reasons. One reason is the same as why I don't use tools like Resharper. You know, the 50,000 different hotkey combinations that can insert templated code, etc. etc. The reason I don't use tools like that is because for me, coding is about 75% thinking and 25% actual writing. I don't like to, nor want to write code faster because I don't need to. Often in thinking about code I realize better ways to do it, or in some cases, that I don't actually need that code at all. Having moar code fast can be overwhelming. Sure, AI tools are trained (hopefully) on best practices and should theoretically produce better code the first time around, not needing as much re-factoring, but the time to think and tweak is valuable to me. It's a bit like the tortoise and the hare. Someone with AI assistance will probably produce a solution far faster than someone without one, but at the end of the day, what good is speed if you're zipping along producing the wrong solution? Call me selfish but I also think any developer should see the writing on the wall that if a tool saves them 50% of their time, employer expectations are going to be pushing for 100% more work out of them in a day. 

The second main reason I don't see myself using AI is when it comes to stuff I don't know, or need to brush back up on, I want to be sure I fully understand the code I am responsible for, not just requesting something from an LLM. Issues like "impostor syndrome" are already a problem in many professions. I don't see the situation getting anything but worse when a growing portion of what you consider "employment" is feeding and changing the diapers on a bot. I have the experience behind me to be able to look at the code an LLM generates and determine whether it's fit for purpose, or the model's been puffing green dragon. What somewhat scares me is the idea of "vibe coding" where people that don't really understand coding use LLMs in a form of trial and error to get a solution done. Building a prototype? Great idea. Something you're going to convince people or businesses to actually use with sensitive data or decisions with consequences? Bad, bad idea.

Personally I see the value in LLM-based code generation plateauing rather quickly in terms of usefulness. It will get better, to a point, as it continues to learn from samples and corrections written and reviewed by experienced software developers. However, as Github starts to get filled by AI-generated code, and sites like StackOverflow die off with the new generation of developer consulting LLMs for guidance and "get this working for me" rather than "explain why this doesn't work", the overall quality of generated code will start to slip. With luck it's noticeable before major employers dive all-in, giving up on training new developers to understand code & problem solve, and all of us dinosaurs retire.

Until then I look forward to lucrative contracts sorting out messes that greenhorns powered by ChatGPT get themselves into. ;)

Autofac and Lazy Dependency Injection: 2025 edition

 Thank you C#!  I couldn't believe it's been two years since I last posted about my lazy dependency implementation. Since that time there have been a few updates to the C# language, in particular around auto-properties that have greatly simplified the use of lazy dependencies and their property overrides to simplify unit testing. I also make use of primary constructors, which are ideally suited to the pattern since unlike regular constructor injection, the assertions happen in the property accessors, not a constructor.

The primary goal of this pattern is still to leverage lazy dependency injection while making it easier to swap in testing mocks. Classes like controllers can have a number of dependencies that they use, but depending on the action and state passed in, many of those dependencies don't actually get used in all situations. Lazy loading dependencies sees dependencies initialized/provided only if they are needed, however this adds a layer of abstraction to access the dependency when it's needed, and complicates mocking the dependency out for unit tests a tad more ugly.

The solution which I call lazy dependencies +property mitigates these two issues. The property accessor handles the unwrapping of the lazy proxy to expose the dependency for the class to consume. It also allows for a proxy to be injected. Each lazy dependency in the constructor is optional. If the IoC Container doesn't provide a new dependency or a test does not mock a referenced dependency, the property accessor throws a for-purpose DependencyMissingException to note when a dependency was not provided.

Updated pattern:

 public class SomeClass(Lazy<ISomeDependency>? _lazySomeDependency = null)
 {
     [field: MaybeNull]
     public ISomeDependency SomeDependency
     {
         protected get => field ??= _lazySomeDependency?.Value ?? throw new DependencyMissingException(nameof(SomeDependency));
         init;
     }
 }

This is considerably simpler than the original implementation. We can use the primary constructor syntax since we do not need to assert whether a dependency was injected or not. Under normal circumstances all lazy dependencies will be injected by the container, but asserting them falls on the property accessor. No code, save the accessor property, should attempt to access dependencies through the lazy dependency. The auto property syntax, new to C# gives us access to the "field" keyword. We also leverage a public init setter so that our tests can inject mocks for any dependencies they will use, while the getter remains protected (or private) for accessing the dependency within the class. The dependency property will look for an initialized instance, then check the lazy injected source, before raising an exception if the dependency has not been provided.

Unit tests provide a mock through the init setter rather than trying to mock a lazy dependency:

Mock<ISomeDependency> mockDependency = new();
mockDependency.Setup(x => /* set up mocked scenario */);

var classUnderTest = new SomeClass
{
    SomeDependency = mockDependency.Object
};

// ... Test behaviour, assert mocks.

In this simple example it will not look particularly effective, but in controllers that have several, maybe a dozen dependencies, this can significantly simplify test initialization. If a test scenario is expected to touch 3 out of 10 dependencies then you only need to provide mocks for the 3 dependencies rather than always mocking all 10 for every test.  If internal code is updated to touch a 4th dependency then the test(s) will break until they are updated with suitable mocks for the extra dependency. This allows you to mock only what you need to mock, and avoid silent or confusing failure scenarios when catch-all defaulted mocks try responding to scenarios they weren't intended to be called upon.

Autofac and Lazy Dependency Injection

 Autofac's support for lazy dependencies is, in a word, awesome. One issue I've always had with constructor injection has been with writing unit tests. If a class has more than a couple of dependencies to inject, you quickly run into situations where every single unit test needs to be providing Mocks for every dependency even though a given test scenario only needs to configure one or two dependencies to satisfy the testing.

A pattern I had been using in the past with introducing Autofac into legacy applications that weren't using dependency injection was using Autofac's lifetime scope to serve as a Service Locator, so that the only breaking change was injecting the Autofac container in the constructors, then using properties that would resolve themselves on first access. The Service Locator could even be served by a static class if injecting was problematic with teams, but I generally advise against that to avoid the Service Locator being used willy-nilly everywhere which becomes largely un-test-able. What I did like about this pattern is that unit tests would always pass a mocked Service Locator that would throw on a request for a dependency, and would be responsible for setting the dependencies via their Internally visible property-based dependencies. In this way a class might have six dependencies with a constructor that just includes the Service Locator. When writing a test that is expected to need two of those dependencies, it just mocks the required dependencies and sets the properties rather than injecting those two mocks and four extra unused mocks.

An example of the Service Locator design:

private ISomeDependency? _someDependency = null;
public ISomeDependecy SomeDependency
{
    get => _someDependency ??= _scope.Resolve<ISomeDependency>()
        ?? throw new ArgumentException("The SomeDependency dependency could not be resolved.");
    set => _someDependency = value;
}

public SomeClass(IContainer container)
{
    if (container == null) throw new ArgumentNullException(nameof(container));
    _scope = container.BeginLifetimeScope();
}

public void Dispose()
{
    _container.Dispose();
}

This pattern is a trade-off of reducing the impact of re-factoring code that doesn't use dependencies to introduce IoC design to make the code test-able. The issue to watch out for is the use of the _scope LifetimeScope. The only place the _scope reference should ever be used is in the dependency properties, never in methods etc.  If I have several dependencies and write a test that will use SomeDependency, all tests construct the class under test with a common mock Container that will provide a mocked LifetimeScope that will throw on Resolve, then use the property setters to initialize the dependencies used with mocks. If code under test evolves to use an extra dependency, the associated tests will fail as the Service Locator highlights requests for an unexpected dependency.

The other advantage this gives you is performance. For instance with MVC controllers and such handling a request. A controller might have a number of actions and such that have touch-points on several dependencies over the course of user requests. However, an individual request might only "touch" one or a few of these dependencies. Since many dependencies are scoped to a Request, you can optimize server load by having the container only resolve dependencies when and if they are needed. Lazy dependencies.

Now I have always been interested in reproducing that ease of testing flexibility and ensuring that dependencies are only resolved if and when they are needed when implementing a proper DI implementation. Enter lazy dependencies, and a new example:

private readonly Lazy<ISomeDependency>? _lazySomeDependency = null;
private ISomeDependency? _someDependency = null;
public ISomeDependency SomeDependency
{
    get => _someDependency ??= _lazySomeDependency?.Value
        ?? throw new ArgumentException("SomeDependency dependency was not provided.");
    set => _someDependency = value;
}

public SomeClass(Lazy<ISomeDependency>? someDependency = null)
{
     _lazySomeDependency = someDependency;
}

This will look similar to the above Service Locator example except this would be operating with a full Autofac DI integration. Autofac provides dependencies using Lazy references which we default to #null allowing tests to construct our classes under test without dependencies. Test suites don't have a DI provider running so they will rely on using the setters for initializing mocks for dependencies to suit the provided test scenario. There is no need to set up Autofac to provide mocks or be configured at all such as with the Service Locator pattern.

In any case, this is something I've found to be extremely useful when setting up projects so I thought I'd throw up an example if anyone is searching for options with Autofac and lazy initialization.

JavaScript: Crack for coders.

I've been cutting code for a long time. I was educated in Assembler, Pascal, and C/C++. I couldn't find local work doing that so I taught myself Access, Visual Basic, T-SQL, PL-SQL, Delphi, Java, Visual C++, and C# (WinForms, WebForms, MVC, Web API, WPF, and Silverlight). It wasn't until I really started to dive into JavaScript that programming started to feel "different".

With JavaScript you get this giggly-type high when you're sitting there in your editor with a dev server running on the other screen tinkering away and the code "just works". You can incrementally build up your application with "happy little accidents" just like a Bob Ross painting. WebPack and Babel automatically wrangled the vast conflicting standards for modules, ES versions, and such into something that actually runs on most browsers. React's virtual DOM makes screen updates snappy, MobX is managing your shared state without stringing together a range of call-backs and Promises. And there is so much out there to play with and experiment. Tens of thousands of popular packages on npm waiting to be discovered.

But those highs are separated by often lingering sessions testing your Google-fu trying to find current, relevant clues on just why the hell your particular code isn't working, or how to get that one library you'd really like to use to import properly into your ES6 code without requiring you to eject your Create-React-App application. You get to grit your teeth with irritations like when someone managing moment.js decided that add(number, period) made more sense than add(period, number) while all of the examples you'd been using had it still the other way around. Individually, these problems seem trivial, but they really add up quick when you're just trying to get over that last little hurdle between here and your next high.


JavaScript development is quite literally Crack for coders.

YAGNI, KISS, and DRY. An uneasy relationship.

As a software developer, I am totally sold on S.O.L.I.D. principles when it comes to writing the best software. Developers can get pedantic about one or more of these principles, but in general following them is simple and it's pretty easy to justify their value in a code base. Where things get a bit more awkward is when discussing three other principles that like to hang out in coding circles:

YAGNI - You Ain't Gonna Need It
KISS - Keep It Stupidly Simple
DRY - Don't Repeat Yourself

YAGNI, he's the Inquisitor. In every project there are the requirements that the stakeholders want, and then there are the assumptions that developers and non-stakeholders come up with in anticipation of what they think stakeholders will want.YAGNI sees through these assumptions and expects every feature to justify it's existence.

KISS is the lazy one in the relationship. KISS doesn't like things complicated, and prefers to do the simplest thing because it's less work, and when it has to come back to an area of a project later on, it wants to be sure that it's able to easily understand and pick the code back up.  KISS gets along smashingly with YAGNI because the simplest, easiest thing is not having to write anything at all.

DRY on the other hand is like the smart-assed, opinionated one in a relationship. DRY wants to make sure everything is as optimal as possible. DRY likes YAGNI, and doesn't get along with KISS and often creates friction in the relationship to push KISS away. The friction comes when KISS advocates doing the "simplest" thing and that results in duplication. DRY sees duplication and starts screaming bloody murder.

Now how do you deal with these three in a project? People have tried taking sides, asking which one trumps the other. The truth is that all three are equally important, but I feel that timing is the problem when it comes to dealing with KISS and DRY.  When teams introduce DRY too early in the project you get into fights with KISS, and can end up repeating your effort even if your intention wasn't to repeat code.

YAGNI needs to be involved in every stage of a project. Simply do not ever commit to building anything that doesn't need to be built. That's an easy one.

KISS needs to be involved in the early stages of a project or set of features. Let KISS work freely with YAGNI and aim to make code "work".  KISS helps you vet out ideas in the simplest way and ensure that the logic is easy to understand and easy to later optimize.

DRY should be introduced into the project no sooner that when features are proven and you're ready to optimize the code. DRY is an excellent voice for optimization, but I don't believe it is a good voice to listen to early within a project because it leads to premature optimization.

Within a code base, the more code you have, the more work there is to maintain the code and the more places there are for bugs to hide. With this in mind, DRY would seem to be a pretty important element to listen to when writing code. While this is true there is one important distinction when it comes to code duplication. Code should be consolidated only where the behavior of that code is identical. Not similar, but identical.  This is where DRY trips things up too early in a project. In the early stages of a project, code is highly fluid as developers work out how best to meet requirements that are often still being fleshed out. There is often a lot of similar code that can be identified, or code that you "expect" to be commonly used so DRY is whispering to centralize it, don't repeat yourself!

The problem is that when you listen to DRY too early you can violate YAGNI unintentionally by composing structure you don't need yet, and make code more complex than it needs to be. If you optimize "similar" code too early you either end up with conditional code to handle the cases where the behaviour is similar but not identical, or you work to try and break down the functionality so atomically that you can separate out the identical parts. (Think functional programming) This can often make code a lot more complex than it needs to be too early in the development leading to a lot more effort as new functionality is introduced. You either struggle to fit with the pre-defined pattern and restrictions, or end up working around it entirely, violating DRY.

Another risk of listening to DRY too early is when it steers development heavily towards inheritance rather than composition. Often I've seen projects that focus too heavily and too early on DRY start to architect systems around multiple levels of inheritance and generics to try and produce minimalist code. The code invariably proves to be complex and difficult to work with. Features become quite expensive to work with because they cause use cases that don't "fit" with previous preconceptions of how related code was written to be re-used. This leads to complex re-write efforts or work-arounds.

Some people argue that we should listen to DRY over KISS because DRY enforces the Single Responsibility Principle from S.O.L.I.D.  I disagree with this in many situations because DRY can violate SRP where consolidated code can now have two or more reasons to change.  Take generics for example. A generic class should represent identical functionality across multiple types. That is fine, however you can commonly find a situation where a new type could benefit from the generic except...   And that "except" now poses a big problem. A generic class/method violates SRP because it's reason for change is governed by every class that leverages it. So long as the needs of those classes is identical we can ignore the potential implications against SRP, but as soon as those needs diverge we either violate SRP, violate DRY with nearly duplicate code or further complicate the code to try and keep every principle happy. Functional code conforms to SRP because it does one thing, and does it well. However, when you aren't working in a purely functional domain, consolidated code serving a part of multiple business cases now potentially has more than one reason to change. Clinging to DRY can, and will make your development more costly as you struggle to ensure new requirements conform.

Personally, I listen to YAGNI and KISS in the early stages of developing features within a project, then start listening to DRY once the features are reasonably established. DRY and KISS are always going to butt heads, and I tend to take KISS's side in any argument if I feel like that area of the code may continue to get attention or that DRY will risk making the code too complex to conveniently improve or change down the track.  I'll choose simple code that serves one purpose and may be similar to other code serving another purpose over complex code that satisfies an urge not to see duplication, but serves multiple interests that can often diverge.

The stale danish

One Monday, while walking to the office you pass a little bakery with a window display. A beautiful looking danish catches your eye. The crust looks so light and fluffy, a generous dollop of jam in the middle, and the drizzle of icing sugar. So tempting, but you promised yourself to cut back on the snacks, and continue your way to the office.

Each morning, you walk past that same bakery and see a danish sitting there, just beckoning you... Finally, Friday morning comes around and you see that gorgeous danish waiting there. "I can't take it any more!" you scream to yourself and you give in, walk into the bakery and buy the danish. After the guilt melts away holding that beautiful danish, you take a bite... It's long since gone stale, having been sitting there since Monday.

The stale danish is how I view the Internet, and the marvelous source of information we, as software developers have come to rely on it for. When technology is new, the Internet provides a flury of information and experiences that the global software development community collectively accumulates. However, much of the time, that information isn't versioned, and often sources like blog posts & walk-throughs aren't even dated. As the technology matures, finding the information you need to solve a particular problem becomes a treasure hunt to find a fresh danish in a sea of stale ones. Working with JavaScript frameworks like React and it's supporting libraries like Redux in particular has proven extremely challenging time and time again. Every undated bit of information needs to be taken with a grain of salt, and it's compounded by the fact that with JavaScript, there are always a number of variations to do what should be relatively standard stuff, all conforming to different interpretations of "best practices".

Even normal consumers of information are getting tricked by the stale danishes floating around the web. Technologies like Google street view in particular are marvelous, but they require constant updates because as the world around us changes, the maps and camera images quickly become out of date. Using these services to assist with landmarks can be fraught with danger as you're looking for an intersection after a car dealership or restaurant that relocated six months ago. At least Google had the common sense to put dates on their imagery.

The stale danish is the one key reason that I would much, much rather work in the back end with technologies like .Net, Entity Framework, and SQL, than working in the front end with JavaScript libraries. I love working in the front end, solving usability issues and presenting users with useful views to data and features they need. I enjoy MVC, WPF, and had enjoyed Silverlight largely because the pace of change was a lot slower, and there were a handful of sources to find directly relevant information about the technologies and how to effectively use them.

JavaScript, and open source in general is far more volatile. While you may argue this makes it more responsive so that problems get solve more quickly, the real trouble is that everyone's perception of a problem is different, so there are a lot of "solutions" out there that are screaming on the web for validation. As a software developer, projects need to have a completion date so you do need to choose from those available options at the time, and commit to them. Having relevant and accurate information at hand through the life of the project is essential for when challenges do crop up. Ripping out X to replace it with Y isn't an option, and often the breaking changes from X v0.8 to X v0.9 prohibit the idea of upgrading, so when searching for possible answers and work-arounds to problems, it would be a wonderful if more articles and posts were versioned, or at least had a "Baked On" date listed.

2 very good reasons why you should never pass entities to the client.

So you are building a web application with Entity Framework on the back end. You've got the code to load the entities now you need to get that data to the client. Then the question enters your head, "I've already loaded this data, why not simply return the entity?" Why bother creating a view model when it's just going to contain many of the fields you've already got here? Well, here's 2 very good reasons why you shouldn't, and if you have, you should seriously consider changing it.

1. Security

If you are passing an entity to the client, and critically, if you are accepting that entity back from the client, attaching it to a context, and saving it, you have opened a massive, gaping security hole in your application.

As an example lets say I have a system that a user logs into and displays a list of debts that user currently has, and allows them to do something innocent like updating a status or record a new note. The debt entity might look something like:

We have a model which contains a list of debts and a view that lets a user tick off a debt and mark it as accepted. We might only display the debt amount and maybe a couple extra details on the screen, however when you look at the debugger where the model might be exposed such as our action to accept the debt:

Since we passed the entity, and our serializer diligently serialized all properties and children we expose all of that data to the client. Hence I might have notes that aren't meant to be seen by the client, yet here they are.

The action handler looks something like:

Now this is a very basic example. In reality we could have a view that allows for a more meaningful edit against an entity.  When we click the Accept button the record gets updated as accepted, all seems good. Except...

I don't want to accept a debt of $5000. I want to make that $1.

Using the debug tools, with a break-point before that call to UpdateDebt, I can edit the debt payload to change the Amount from "5000" to "1" then resume the execution.

Suddenly my data shows that the debt was accepted, but now has an amount of $1?! 

Heck, I don't even want to pay $1. Let's make it someone else's problem.  Using the same break-point we can see that it's assigned to a Customer with ID = 1.  Let's change those Customer ID references to 2, maybe update that customer name to "Fred" and anything else that might identify me.   Presto, that debt is now someone else's problem. What happened?

Let's have a look at the UpdateDebt method:

Part of the attraction I fear developers have with Entity Framework and reattaching entities is that it will save an extra read to the database. Just attach it and save. It's also only 3 lines of code! Win. There are libraries such as GraphDiff that even help dive through related entities to attach them and mark them as modified. Doing this violates the cardinal rule of client-server software development. "Trust nothing that the client sends you!"

Now you might argue that it doesn't make sense to have a client update a single field like that. That an "accept" action would simply take a debt ID and do that behind the scenes. True, but any update action in a system that takes an entity and attaches it is vulnerable to this tampering, and importantly, any entity associated with that entity via it's references.

Even using a view model is not immune to this form of tampering. If you use a tool like Automapper to map entities to view models, but then use Automapper to map view model's back to entities you can expose yourself to the same unintended changes, though to a lesser degree provided those view models limit the data visible. 

In any case, *never* trust the data coming back from the client. Even if it's an action that just accepts a debt ID, verify that the debt ID actually belongs to the currently authenticated user. *always*. If it isn't, log the user out immediately, and log the attempt to notify administrators. You either have a bug that should be fixed, or someone actively trying to circumvent or compromise the system.

2. Performance

Now point #1 should be enough, but if you've chosen to pass entities anyways, and are validating everything coming back from the client, the second reason you should reconsider using entities is that sending entities incur a performance cost, and are easily susceptible to serious performance issues. One issue that I see commonly tripping up users around entity framework is lazy loading where they pass entities to the client. When MVC /  Web API go to send entities back to the client, the serializer touches each property on the entity. Where that property is a lazy-loadable proxy method, that touch triggers off a lazy-load call. Each of those loaded entities then have each of their properties touched to be serialized, triggering more lazy loads. What is worse is that where you might have loaded 20 rows for search results, the lazy loads for each of those 20 entities and their children will be performed 1 by 1, not using the criteria you used to load the 20 parent entities as if would have been the case if you used .Include() to eager load them. The simple work-around is to eager load everything, but that means eager loading absolutely everything associated with that entity, whether you intend to display it or not.

Even in this case we are now faced with a situation where we are loading dozens of columns of data where we might only want to display a handful. That is extra work and memory for your database to provide those unnecessary columns, little to no optimization the database can do with indexes, extra data across the wire to your app server, then extra data across the wire to the client, and extra memory needed on both the app server and client. How much faster would it be responding if it just queried and returned that handful of columns? It's noticeable.

Addressing these issues

The simple answer to avoid these issues? Use view-models. A view model is a POCO (Plain 'ol C# Object) that contains just the data you want your view to see, and the IDs you will need to perform any action from that view. Automapper now provides support for the IQueryable interface and Linq integration via the .ProjectTo() method which can pass through the Select criteria that the database will need to pass back just what is needed for the view-model. The client doesn't see anything it shouldn't, and the data is optimal for performance by not sending anything that isn't needed.

When it comes to getting those view models back: Trust nothing. Every operation and update absolutely must be authenticated and have all applicable IDs resolved back to the user. Any deviation should be logged and generate an alert to administrators.

Conclusion

Hopefully this helps convince developers to reconsider ideas to pass entities between their controllers and views / API consumers.  If you are working on a system that does pass entities around, at least at a minimum be sure to test your client with the debugger tools thoroughly to see exactly what data is being exposed, and ensure that every screen and action is properly guarding against JSON data tampering.

Maximizing Performance with Entity Framework & MVC / Web API

You've just started on a contract and the manager or lead developer comes up to you and says: "Ok, we were using Entity Framework for our data layer but it's too slow so we need to replace it with {Dapper/NHibernate/EF-Core/NoSQL/ADO+Sprocs/etc.}"

When I hear this (and I have heard it more than a couple times) it sends the warning bells ringing, and presents an opportunity to save clients a whole lot of money. 99.5% of the time the issue isn't that EF is too slow; The issue is that they just don't know how to use it, and even switching to another technology stack they will likely encounter the same or worse performance issue.

Whenever I face the task of improving performance with EF, there are always issues and smells to investigate and eliminate. In this article I'll be outlining some of the major ones, and what can be done about them. Some of these items can involve a fair amount of re-factoring to implement, but the performance impact of these changes will be evident quite clearly to justify the changes.

Culprit #1 - Code-First

I'll get it out there right now. I *hate* EF code-first. I mean I really hate code-first. It is in the list of technologies that I wish were never invented, right up there with "remember password" and "hide file extensions". The road to Hell was paved in good intentions, and code-first accounts for more than a few bricks on that path. Code-first and Migrations make up now the majority of EF related issues on Stack Overflow. They are great tools for whipping up Alpha 0.1 and getting something up and running fast, but beyond that they should simply be turned off. Code-first databases *can* be set up efficiently, but I'd say that every instance of a code-first database implementation that I've seen in a production system makes my inner-DBA shrivel and die a little. I am not a DBA, but I have worked with enough to have picked up on what makes for efficient and effective database design. Out of the box, code-first offers none of this. My recommendation is that if you're using it and not experiencing performance issues, then you shouldn't be reading this article. If you are reading this article and using code-first, seriously consider turning it off unless you know this beast well enough to have addressed these following smells:

a) Inefficient or missing indexes. Indexes are something that in EF systems should be added and maintained as systems mature. Looking at real-world usage and performance you should be constantly evaluating and implementing indexes. This isn't something that should be set up "first" (premature optimization), and it shouldn't be forgotten/ignored.
b) Poor schema design. Simple things like using GUIDs for PKs with NewId() instead of NewSequentialId(), or using meaningful string PKs/FKs, not to mention probably a dozen sins that a DBA will point out about generated schemas.
c) Time wasted attempting to resolve issues with code first generation and migrations, and the inevitable work-around/hacks to keep it happy. How many columns are left as inefficient data types, confusing names, or otherwise polluting your schema, indexes, and entities because you couldn't get EF migrations to clean them up properly without failures or side-effects?
d) Generation is no substitute for a good DBA. More than a couple teams that I have joined have foregone a DBA thinking that developers using code-first negate the need for a DBA. (Frankly, if I was a DBA faced with a team planning to use code first, I'd expect I'd either get frustrated and quit, or have been forced out.) If you're using code-first without access to a DBA to tune it you are asking for trouble. If you do still have a in the team DBA, give them a hug for sticking around and putting up with you!

No, I don't like code-first at all, and it's responsible in part or whole for at least a few of the issues below. I will always recommend using a database first approach with EF and manage the schema manually, preferably with access to a DBA.

Culprit #2 - Schema related issues.

Even if code first wasn't responsible for the schema, there are a few schema related bugbears to investigate and potentially improve. These often aren't the primary performance bottleneck, but they are one of the quickest wins to squeeze a bit of performance out without code changes, and these steps will help identify bigger wins in re-factoring for further culprit items.

The first thing that every EF developer needs to be familiar with is a profiler. This can be an EF profiler, or any SQL profiler for your database of choice will work equally well. A common one I use for SQL Server is ExpressProfiler (https://github.com/OleksiiKovalov/expressprofiler) which is a quick and simple profiling tool. Using a profiler against an isolated database that you can test against will help reveal slow, expensive queries that you can copy and run against the database with an execution plan. This can highlight missing indexes for example.

The key things to check in the schema:
a) Correctly typed keys with efficient identity/defaults. For example, if you use UniqueIdentifier PKs, are you using NewSequentialId() instead of NewId()? Are you using composite PKs where a dedicated meaningless PK could be used? Are you using meaningful string PKs instead of meaningless numeric ones?
b) Are there missing indexes, or indexes that can be improved?
c) Do you have an index maintenance schedule?
d) Do you have Transaction Log maintenance scheduled as part of your backup strategy? (SQL Server)
e) In general, has a DBA gone over your database and maintenance scheduling? If no, get someone to help look at it.

Culprit #3 - Exposing entities instead of View Models / DTOs.

In terms of performance impacts on a system, this culprit is far an above the most significant issue to track down and eliminate. By now you should have a profiler installed, and the profiler is instrumental in demonstrating the pain that passing entities around can cause. The first problem that you can encounter with returning entities, especially in the case of MVC controllers is unexpected lazy load calls. These show up in a profiler when you trigger a call against the controller then see a whole stream of database hits go through. If you put a breakpoint at the end of the call, then run to that point, and then resume to exit the method, these extra DB calls will appear after exiting the method.  What is happening is that ASP.Net is attempting to serialize the entities to return to the view. The serializer iterates over the properties and by doing so, trips the lazy loaded property proxy methods, triggering a load from the database. 

What is worse is that when you're faced with returning a collection of entities such as search results, each set of lazy loaded dependencies will be queried against the database *one.. by.. one*. Ouch! So lets say I query a page of order results. Orders have a customer reference and a delivery address reference. Customers have an address and contact details reference. If you returned a page of 20 orders, the lazy load hits will include 20x "SELECT FROM tblCustomer WHERE CustomerID = x" plus 40x "SELECT FROM tblAddress WHERE AddressID = x" (20 for the order delivery address, and 20 for the customer address) plus 20x "SELECT FROM tblContact WHERE CustomerID = x", etc. etc. for every reference of Order, and every reference of that reference and so-on.

These lazy loads can be mitigated by eager-loading the data using .Include() statements, however this still presents performance issues as the database server needs to retrieve all columns of all related tables for the applicable rows. This data then needs to be transmitted and allocated memory on the application server, before being serialized and sent across the wire to the client. That is almost always amounting to a lot more data than is needed, and doesn't present opportunities to take full advantage of indexing.

Sending entities also most importantly exposes systems to revealing more information than users should be able to access. Even if you don't add controls to your view for that data, a simple F12 and inspection in the debugging tools can expose that information to clever viewers. Passing entities back to the server and being "clever" by attaching them to a context and saving is a cardinal sin. Those entities expose all FKs etc. and could have been modified in ways that your client doesn't support, but a clever debugger could modify.

Lastly, dealing with detached entities adds complexity and potential errors within your server code. You need to test whether entities aren't already associated to a DbContext before reattaching them, and ensuring that references are updated before being saved.

Using POCO view models you can improve performance considerably by letting EF query just the columns and rows you need from the database, and pass that smaller payload across the wire to the client. Tools like Automapper now support IQueryable extensions via .ProjectTo which means you can relatively easily integrate mapping an entity query and related data into a view model that is optimized for the particular view.

I believe the biggest argument for passing entities (besides being lazy about defining a view model which looks similar to an entity) is to magically avoid having to re-read the data when performing the update. I call B.S. on that "optimization" on the grounds that systems generally do a lot more reads than writes, and updates typically involve fetching a relatively small amount of data by PK which EF performs ridiculously fast.  Time also passes from when that data is read and when it is written so systems still need to handle the fact that the server data could have been updated, so the code still needs to be prepared to re-load the entity anyways. Trading off the security and performance for avoiding defining an extra set of simple POCO classes and trying to avoid a very fast read is a bad deal.

Culprit #4 - Async, async everywhere.

When developing large, complex systems, you need to know and understand async/await and how it works with EF. However, many developers are a bit mistaken about what it does, and where it should or should not be used. Generally, I find that once teams decide to use it, they decide to use it *everywhere* for consistency. This is a mistake.

Async does not make queries faster. It makes them slower. What async does do is make servers more responsive in that you can kick off a fairly expensive query for one request, then free up that request thread to start processing another request. When that first query finishes, it will resume to finish off that first request on a new worker thread.  When you implement async code you add an overhead to set up the code to effectively suspend and set up a continuation point for resumption later. That overhead is both in terms of server resources, and thread resource synchronization. If you use it for big, expensive requests, those requests will not hold up server processing waiting for those queries. However, if you use it for *all* queries, every operation, including the short & sweet ones will be participating in that park & resume scheduling game.  My recommendation when it comes to async is to start with everything running synchronously, then as you identify heavy queries, update these to use async operations. Typical candidates include:

• Searches
• Retrieving entity graphs for updates
• Retrieving large graphs for reports or detail views.

Removing unnecessary async code is a relatively easy re-factor and something I suggest when I see systems where async/await have been lathered throughout a system.

Culprit #5 - Large DbContexts.

As systems grow larger, the number of tables and entities can get quite large. The larger a DbContext gets, the longer it takes to spin one up. IMO this is a major drawback of most Code-First implementations. Using bounded contexts (Contexts geared towards specific areas of an application with just those entities and their direct dependencies) can noticeably improve the performance of EF operations. Entities used as a read-only reference in one context can be greatly simplified so that they can be loaded and associated more efficiently. For example a Customer may have 40+ columns, many non-nullable, and the act of creating an Order needs a reference to a customer but rather than mapping an entire Customer table entity for an OrderContext, you can simplify it to just the relevant details you would need for a customer as far as an order is concerned. Smaller entity definition + fewer entity definitions = faster context spin-up and simpler read/write code. There is probably a way to get bounded contexts cooperating against a single DB schema /w migrations, but frankly I find code-first is more hassle than it's worth.

Introducing bounded contexts with a DB-first configuration is a relatively simple task to approach on a screen by screen basis. Often this is an approach I use to introduce EF into legacy systems, and a similar approach can be employed to separate a set of pages from an existing global context into using a new purpose-built context. An absolute must for this step is to ensure that Culprit #3 isn't rearing it's head. To have this level of flexibility you cannot be passing entities around.

Culprit #6 - Entity Framework 4

This item deserves special mention. In more than a few instances I've seen criticisms of EF performance from teams that are still using Entity Framework 4. I've used EF since EF 4 was first introduced (technically EF v2) and unless I *had* to use it, I didn't. I'd previously been an NHibernate advocate. I'd tinkered with EF 5, but I didn't take EF seriously until EF 6 was released. From a performance perspective, EF 4 is a dog's breakfast, and any project still using it should immediately be looking at upgrading to EF 6. The migration path from 4 to 6 is pretty simple. Unfortunately code written around EF 4 will not be initially taking advantage of many of the improvements introduced with 6 such as deferred execution, but those can be re-factored in over time to net significant performance improvements. Simply replacing EF 4 with EF 6 would net 10~25% performance improvements without any other changes.

Conclusion

Hopefully these six culprits can help you with identifying performance problems with EF-backed systems and some ideas on how to improve them rather than giving into believing EF needs to be replaced for something else. If you have any comments or questions about coaxing some performance out of EF flick me a comment or post your question on StackOverflow tagged under "entity-framework" as I tend to follow that tag.

On Agile Project Management

I've often seen a bit of confusion around how to manage projects that are using Agile approaches for software development. On the one hand businesses are very receptive to continuous releases, however on the other hand they struggle with trying to fit the project budget and timeline into more traditional project management molds.

The trouble stems from Agile projects being measured on a velocity basis with the highest priority features being tackled first. From a project management perspective you have to keep an eye on what needs to be done, what has been done, how fast things are progressing, and that they are heading the right direction. Unfortunately the instruction given to them is to bolt this somehow into something like a PRINCE2 where the two are pretty distinctly incompatible.

How I visualize managing Agile projects:

You have a large furnace. This furnace burns money (logs) and produces business value. (steam) Developers, testers, and business analysts take the form of valves for directing the steam to practical goals. Your projects are funded by budgets which form different stacks of logs. Adding up all of the valves tells you how fast the furnace burns logs, that is a constant unless you add valves or increase their size.

Now as a project manager you have control over what logs get put in the furnace and where the steam is directed. The furnace burns a constant rate based on the attached valves. Something is always fed into the furnace, so you nominate the pile of logs to pull from, but if you don't specify, the furnace loader will just start throwing in anything flammable which could include the furniture.

A common problem that I'm faced with is when a project manager is trying to directly tie the output of one or more valves to the burning of a specific log. Between sprints, or sometimes even within a sprint they are tempted to fiddle with the valves to switch between different outputs while a log is burning. (This work needs to be billed against billing code X, while that work needs to be billed against Y.)

Some key problems with this:
Agile teams work most efficiently when they are not context switching between tasks. Each time you fiddle with a valve, steam is lost. Developers have to drop what they're doing, start something, put it down, and go back to what they were working on.
Logs burn more smoothly than shards. Project managers stop feeding logs, instead they try and budget for pieces of work and hand-feed just enough money from a particular pile to produce a specific amount of steam. Agile projects estimate on *difficulty* not cost. The measurement is how fast features are implemented, and when they're deemed good enough, not predicting how long something will take to complete. It might be tempting to pre-cut budgets to work in micro-iterations, however the reality is that staff and contractors are paid 9-5. You'll end up with 5 hours of "budget" spent to cover 8+ real hours of cost, and 3+ hours coming from "somewhere".

If you must context switch for budgeting purposes then the best approach I can recommend is to set it up as a completely separate furnace and dedicate valves to it. Avoid attempting to move valves back and forth frequently.

Does this mean that Agile teams cannot switch between priorities? Certainly not! They're ideally set up for dealing with shifting priorities, but what a project manager must tackle is how the logs are accounted for, and managing the valve changes as efficiently as possible.

1) Feed in the logs, and balance out the piles at the end of the sprint. At the beginning of a sprint set the valves. For these two weeks these developers will be working on these stories, while the rest of the team continues with Y. At the end of the sprint you look at what was delivered in both projects, account for which piles the logs for that sprint were coming from, and what the valve settings will need to be for the next sprint. This is a different frame of reference, at the beginning of the sprint you aren't allocating 3 logs from budget A, and 7 from budget B, you are merely setting the valves and putting 10 logs in the furnace. Based on what you get out at the end of the sprint will determine what piles the logs were pulled from.

2) Look at the quality and type of valves available. Some valves leak more than others, especially when fiddled with. Valves can represent individual members of the team, or groups within the team. Fewer large valves will direct the steam more efficiently than lots of small ones. Overall, the less you fiddle with valves, the more efficiently the steam is materialized into product. Getting into the habit of grouping developers into larger valves is also beneficial when you look to grow a team to increase the burn rate for the furnace. Adding 10 individual valves to 10 existing individual valves will mean you have 10 new, fairly leaky valves. However if you had developers grouped into teams, new developers can be merged into new teams with some or all of the developers from the existing teams to tackle new challenges, or added to existing teams. Developers reinforce each others efforts which helps mitigate leaks. (or at least brings issues to light early to be addressed.)

How does this fit with efforts to get budget approvals, set deliverable feature sets and delivery dates? I can't answer that, but I hope my perspective above gives some food for thought about how to better fit it into more traditional frames of reference, or convince upper echelons to to better fuel Agile projects and continue to see the benefits to the end deliverable.

On Deadlines

It's been pretty busy with one of my current clients. The deadline looms for a release, issues are being uncovered, worked on, resolved... The development manager's facial expressions are changing faster than Melbourne's weather, and the project manager is scurrying around like a Corgi that found it's owner's Ecstasy stash. Just the other day the PM came up to me and asked "Steve, how can you be so calm?" Everything in the project is tracking well, the users were happy with the previews, it was just pre-release jitters. I didn't really have an answer for him, so it was just a bit of humorous banter to lighten the mood. It's not something I've really thought about much as it just seems natural by now, but after thinking about it I thought it might help others deal with stress, deadlines, and potential confrontations in the workplace.

My secret to staying calm is that I don't over-care. Over-caring is when you try to take the world onto your shoulders, give 110% and then some, do whatever it takes! Rather, I invest in the projects I work on. I allocate a reasonable amount of my energy into the project, and can choose to invest a bit extra to push through difficult spots. If I didn't believe in a project then I wouldn't be working on it. In the end, there is always the possibility that the project won't succeed, but if it fails, I can remain confident that it won't be due to something within my control. Also, by being conscious of what I invest in the project (and where) I don't have to get overly distracted by "taking ownership" for a project, which leads to conflicts, time wasting meetings, and frustration.

This is in-line with my general philosophy of life. I live by a policy of Truth, and with a focus of doing what I feel is the right thing, even if it isn't always the most popular thing. This allows one to be supported by their convictions. When you don't waste energy trying to mislead people or mislead yourself it is much easier to view things around you in a more objective fashion. You see opportunities you would otherwise miss if your thoughts are clouded by wondering where you'll get the extra energy to maintain any deception during an inevitable crisis.

Some key points people will notice while I work:
1. I am paid for what I do, not how long I sit at my workstation. Does sitting over your keyboard staring at misbehaving code help you figure out what's going wrong? I doubt it does. I'll browse for ideas, catch up on news, go to the bathroom, knock some balls around on the pool table, see if there's someone else's problem I can help out with around the office or on StackOverflow. When I worked for a company that wrote software to manage a postal print house I'd often spend a few minutes from time to time reloading printers and helping out with the manual mail inserting. It kept me in-touch with the guys that would be using my software plus frees up my thought processes. The context shift lets me spot a new idea when it sneaks up. In the end, all my client needs to decide is whether the amount of value I deliver to the project matches the fee I charge them.

2. I focus on memorizing as little as possible. Preconception is a productivity killer. Convincing yourself that you "know" things leads to arrogance and making assumptions that can easily waste your time plus lead to confrontation within a team. There are people that can just soak in information and retrieve it on demand, and provided that they do it accurately without falling into arrogance it can certainly be an asset. "Googling" or checking out something on StackOverflow should never be viewed as an admission that you're somehow a sub-par software developer. Knowing how to find useful information, absorb, and apply it is a very useful skill.

3. I judge things by what I see people do, not what they say. Before I engage anyone in a discussion about an approach I prepare or select examples outlining my thoughts. I expect the same before I'm convinced of an alternative. There is no build up or stand-off beforehand turning it into a big debate of principles or best practices, just show me code and I'll show you mine. Often by comparing two ideas we come up with a completely new approach that captures the best of both. Other times I'm convinced or doing the convincing and both of us come away knowing more than when we went in.

In general this avoids conflict and confrontation in the workplace, keeps my mind free to spot opportunities, and avoids stress build-up. When I maintain myself in a relatively relaxed state of mind it is very easy to switch into a high-gear to clearly see a problem for what it is and solve it. If I allowed myself to be burning high-octane by default there would be nothing left for the inevitable crisis, and running hot would most certainly lead to mistakes and fuel for those crisis'.


Finally some sci-fi quotes to live by: (What kind of programming geek would I be without them!)

"Understanding is a three edged sword: your side, their side, and the truth." - J. Michael Straczynski

"I've been around long enough to know how ignorant I am. I don't assume the universe obeys my preconceptions. Hah! But I know a frelling fact when it hits me in the face." - Christopher Wheeler (Spoken by Rygel from "I Shrink Therefore I Am")

The cost of TDD: Numbers

This is a question/discussion that crops up again and again. Looking back at my earlier post I thought that perhaps it would be more clear with some numbers.  By "TDD" I refer literally to "test driven development" in the sense that unit tests, whether written before, or after behaviour, are a driving consideration for development.

The trouble with pitching unit testing to developers almost inevitably comes down to "it takes too much time."

Lets say you spend one hour developing a nice, clean, atomic feature. How do you know it works? You fire it up and run through a scenario or two. How long does that take? 5 minutes? Maybe you find a problem so you spend a total of 15 minutes debugging and running through a healthy set of scenarios to ensure your code meets all of the requirements. This is assuming you're a competent software developer that's focused on doing the right thing. You're checking to ensure that the code does what you intended it to do.

Now how long would it take to write unit tests in addition to that manual check? 20 minutes extra? Sounds like a lot, 25 minutes per hour vs. 5 minutes, best case. But let's work with it.

How many hours-worth of features are you going to write in a given two week period? 80? Surely not, I generally get around 60 productive hours in a good iteration. Spending 5 minutes per hour ensuring your code works would give you 55 productive feature hours. Spending 25 minutes per hour  would give you 35 productive feature hours.

But hang on a moment. Let's look back at our first scenario. We spend 5 minutes ensuring that our code does what each requirement is supposed to do. What happens when we change the code to meet the next requirement? Are you sure your previous requirements are still met? Granted every feature is not dependent on every other feature, but lets look at the worst case scenario for the heck of it. Those bugs do love to creep in right where they should be inconceivable. By all means, the first feature requires 5 minutes to test, the second will require 5 minutes plus 5 minutes to regression test the first feature. The third feature will require 15 minutes, the 4th, 20 minutes. By the fifth feature we've matched the per-feature cost of unit testing. We're still ahead, but there's a long way to go and it's only getting more expensive to be sure. By 8 productive hours we're spending 3 hours testing. By 16 hours, we've spent over 11 hours testing. By 28 productive hours we've run out of time. Over half of our time is spent regression testing.

And this is just the first iteration. It only continues uphill from there.  Now obviously not EVERY feature needs to be fully regression tested and require 5 minutes to test. You make a judgement call to determine risk vs. cost.  There will be times it takes more than 5 minutes to test something, and less than 5 minutes to regression test it. But then there is the time needed to record just what needs to be regression tested.  Unit tests will often cost MORE than 25 minutes to write, but in the end it doesn't really matter how long they take because sooner, rather than later, that regression cost and risk is going to catch up. Are you really that confident that after six months of development with 4+ developers touching the common code that every requirement you've written still works the way the client intends to see it when you release?  Oh yeah, there's a regression test before release (we hope) but how many issues do you think they're going to find when that happens, and how much time are you going to spend tracking each and every deviation down? Testing has a nasty way of starting before the developer's pencils are down leaving the door truly open to bugs shipping if developers aren't taking responsibility for asserting their changes don't frak up something else.

But now what about the cost of change? Unit tests get in the way. When the client changes their mind and you try changing stuff, tests break and it's more work for you.  However, unit tests aren't what are expensive, it's change that is expensive. All unit tests have done is helped show you to cost of that change up-front. Those 100 unit tests that broke when you changed that behaviour: Those are the 100 requirements that function based on the original behaviour, and you've just invalidated them. What were you intending to do about those features? Assume they would "just work"? Hope that a regression test picked up those issues? (But won't that still cost you a lot of time and head scratching to track them all down?) Didn't you expect those 100 areas might depend on that change you're about to be making?  Or, perhaps, shouldn't you be glad that *you* see the impact of that change before your client does?

On SharePoint

You’d think embedding content in a SharePoint page, even their bastardized “wiki”, would be a relatively simple, refined feature… I mean it’s all about content management… they have picture libraries, right?!  Sure enough, the steps involve adding an image to a picture library, no problem there.  But then go to embed an image in the page and you get this beauty:

Seriously.. WTF?! Is this 1990 and MS-Access? Was this “feature” just introduced in SharePoint 2007? I gotta wonder because that dialog box is the kind of thing you build when your boss says “Can we have a feature where we can embed an image?” and you stub something in to verify it will work. Then you’re *supposed* to replace it with a more polished, and functional feature! I.e. options to select an image from an image library, or upload an image. (auto-creating a default image library) Something…. ANYTHING…  This is, as Chris Pebble is credited with the coining the phrase, “Protoduction”. (@CodingHorror)

So now all I need is the URL to the image I added in the image library… Now where do I find that… Click on my image in the library to view the item… name, preview, title…. No URL. Right-click on the preview and bring up properties: http://server/teams/IS/WikiImages/_w/Announcements_png.jpg ?? On the previous screen it is http://server/teams/IS/WikiImages/_t/Announcements_png.jpg. Ok, so just how many JPEGs does it need to render of my original PNG? 

Finally, clicking on the preview image itself brings up a browser window containing actual PNG file to get the actual URL… It would have never crossed my mind that a “copy URL” button or slapping it in text field might remotely have been useful especially when faced like idiotic dialog boxes asking me to type in the URL.

So now I finally have an image link in my wiki pointing to a PNG in an image library that has at least two mug-shot JPEGs taken of it. 

SharePoint: Content mismanagement at its finest.

On "clever".

Recently I came across some odd behaviour in a web system around date and time parsing. There was a validation step responsible for ensuring one date/time value was greater than the other. But it seemed to trip up in certain scenarios, such as around 08:00 in the morning. (Anyone used to working with Javascript probably knows exactly what the problem is already.:)

Fortunately I'd come across the cause of this bug a short time ago when enhancing a Javascript-based date parser.

Someone responsible for Javascript's parseInt() method thought it would be clever to try and determine what kind of numeric value was being passed in by inspecting the string and choosing an appropriate conversion rule. (I've read this may be a carry over from C, but regardless it is a stupid assumption that should never have been propagated.) If you pass in "0x..." it can be assumed you want to do a Hex conversion. Fair enough. But then they assumed that if you passed in a value with just a leading 0(zero) you'd want to do an Octal conversion.

I'm sorry, but this has to be one of the dumbest assumptions I've ever seen and surely has led to, and will continue to lead to countless bugs throughout the history of web applications. It's an absolutely stupid assumption because in either Octal or Decimal, 00-07 will result in: 0 - 7. The fun bit is that if you pass parseInt "08" or "09", you get back: wait for it, #null. Pass it "10" and the logic switches to assume you mean Decimal, so it passes you back 10.

So yes, according to parseInt, when you want to check a month number, your calendar must be:
January, Febuary, March, April, May, June, July, #null, #null, October, November, December.

So, if you're encountering unexplained bugs first thing in the morning, or with data for August or September be sure to inspect any and all parseInt calls.

parseInt("08", 10);

Someone owes an apology to the countless developers handed a #null for "08" and "09", and the possible *single* developer wondering where his 9 went after being led to believe parseInt defaulted to Octal.

Moq - Avoiding optimized return results.

Recently I was putting together a unit test for a MVVM+C Controller that accessed a VM factory that I was mocking out. This was to address a bug where multiple calls for the same domain object would result in multiple VM instances rather than references to the same instance. (Something I hadn't handled and spotted with some odd UI behaviour.)

Unfortunately I had to eat my dogfood with this bug because I went and fixed it before I had a unit test to reproduce it. My fix appeared to work, and I wrote a unit test that asserted it, but I wanted to be sure it covered the original bug, so I reverted the fix, but the test still passed! Hmm, I verified that the bug was happening at runtime through the UI, and the test was pretty basic. (Kick off the re-creation of the VMs in a particular way, and check two VM references to see if they're the same or not.) I finally tracked it down to Moq doing something unexpected (though likely by design...)

Here's an example of the problematic statement:


IParticipantViewModelFactory mockParticipantViewModelFactory = new Mock();

mockParticipantViewModelFactory.Setup( pvmf => pvmf.Build( stubDtos[0] ) )
.Returns( new ParticipantFullViewModel( stubDtos[0]) );

It's an innocent enough factory mock, wouldn't one expect that each call would return a new participant VM reference? After eliminating other possible issues with my test setup I knocked in the following sanity check:

var test1 = mockParticipantViewModelFactory.Object.Build(stubDtos[0]);

var test2 = mockParticipantViewModelFactory.Object.Build(stubDtos[0]);

Assert.AreNotSame(test1, test2);

Surprisingly the Assert failed?! Two calls to the Mock containing a .Return(new...) returned the same reference. (Where the real factory would have returned references to two distinct objects.)

The solution was to be a little less lazy with the mock definition:

mockParticipantViewModelFactory.Setup( pvmf => pvmf.Build( stubDtos[0] ) )
.Returns( (IFullDto dto) => new ParticipantFullViewModel( dto ) );

Now the above test, even passing in the exact same DTO returns references to two distinct View Models. 

It would appear that the Moq Mock optimized the initial "static" return into a single reference for all .Return calls, where by specifying that it should use the value from the Setup (even though it's always exactly the same) it actually builds a return value for each call.

It was an amusing behaviour to track down. I guess you could ask why did I simplify that mock return like that in the first place? it doesn't look like a very effective mock. The answer was because in this test case I don't care about the "guts" of how the view model that was being set up, I already have unit tests that assert that the VM Factory composes valid VMs, and that the controller composes those VMs through the factory correctly. This test case was for a specific bug where the factory the application was using returned 2 VMs for the same domain object, and that the application controller should handle using an existing reference if it has one.

*Edit: And in deciding to write this up to the Moq team it dawns on me why this did what it did.... I told it to return an object on that call by declaring:
.Returns( new ParticipantFullViewModel( stubDtos[0]) )
when in fact I should have written it as:
.Returns(() => new ParticipantFullViewModel( stubDtos[0]) )

*sigh!

reCAPTCHA, Stretcha

I hate sifting through spam, and I can sympathise with anyone that doesn't want their blog/site strewn with comments about how their life would be so much happier with V1AGR4. But on the other hand, measures to combat bots such as reCAPTCHA are starting to edge me to question whether or not I actually am human any more... Case in point:

Ok, I thought this service was supposed to use real words. But that last one had me guessing... "ltursth"? "lturyth"? So I ask for a refresh and I get...

*groan*.