Monday, July 30, 2007

Serialization Engine Wish List

Cereal On several occasions I have need to serialize an object graph in memory out to a stream and also needed to read XML data back into objects. Each time I'm confronted by such a task I find myself hunting through the various MSDN serialization topics and becoming increasingly depressed.

Neither the System.Runtime.Serialization namespace nor the System.Xml.Serialization namespace seem to fit my requirements without needing to write large chunks of custom serialization code. It's getting to the point that I'm tempted to write my own serialization engine.

I know that if I do write my own engine, it will consume most of my time, so for now I'm simply going to document what I'd like to see in said engine.

  • Classes should not require a parameter-less constructor to be deserialized. As a simple alternative a class could define a static method which takes a collection of key value pairs from the engine, creates an instance via an appropriate constructor and returns it back to the engine. Ideally the engine would support declarative attributes on an existing constructor defining how to call it.
  • Classes should not need to override their entire serialization process because the engine does not understand one of it's field types. If my class has four integer fields, two string fields, and a complex unserializable Foo field, the engine should handle the integers and strings itself then hand the final stage to my custom serialization function to handle only the Foo field.
  • Classes should be able to override part of the serialization process then return control to the engine to finish the task. Extending my Foo example, I might have custom code to step in and serialize the Foo data but if Foo has a simple serializable member of type Bar, I want to hand serialization of that member back to the engine.
  • Classes should not need public modifiers on members to enable serialization. The engine should make efficient use of reflection to access private and protected members. Ideally the CLR would have a serialization-access level to allow an engine to work with private members in a low trust environment. I think serialization is such a fundamental function that it warrants some cooperation and special treatment by the runtime environment.
  • Classes shouldn't care what format the engine is serializing to or from. If it's binary, SOAP, XML, JSON, or any other new format, the attributes on my class and my custom serialization code should not need to change.

I'm sure there are some other requirements that I've forgotten and I'm sure this is somewhat too late for Visual Studio 2008 but hopefully I'm not the only one that feels this could be improved. What challenges have you faced with the .NET serialization system?

 Saturday, July 28, 2007

Visual Studio versus MSBuild

Hammer I have recently setup a TFS Team Build server for our work projects after realising that the secondary reference issue is not as problematic as first thought. We haven't been using Visual Studio's Remove Unused Reference feature as much as expected and we need to prepare our projects for Visual Studio 2008.

With a build server configured to perform automatic builds every night we are on our way to full continuous integration with builds on check-in. However, there are several other differences between Visual Studio and MSBuild that are creating slight issues for our TFS build agent.

For what I understand of the situation, Visual Studio utilises an in-process variation of the MSBuild engine to support specific features like background compilation but has failed to maintain full compatibility.

For example, installation of SQL Server 2005 on your development machine will offer four new Business Intelligence project types: Analysis Services, Integration Services, Report Server, and Report Model. None of these project types will build from the command line in MSBuild or therefore Team Build. We have had to move these project into separate solutions that we exclude from the build server's automated processing.

Also, strongly-typed DataSets and other designer-based project items are usually based on a metadata file (like an XSD) and use a custom tool to generate an appropriate VB or C# code file before the project is compiled. The project files contain the information about which generator tool to use to update the code files from the metadata but again MSBuild does not support it. Any changes to the metadata that is checked-in will not be reflected in the Team Build output.

The idea to have an IDE and a build engine based on the same technology and file formats is brilliant. The need to only manage build settings in one place is the biggest reason for not moving to NAnt or another offering where I'd need to maintain both the Visual Studio projects and the third party scripts manually.

Unfortunately I feel that Microsoft made two major mistakes with MSBuild and Visual Studio:

  • APIs were provided to allow Visual Studio to be extended to support new project items and project types but the API doesn't force the extension to be callable by MSBuild.
  • Microsoft then utilised this API to extend Visual Studio for SQL Server projects and others too and also ignored MSBuild compatibility themselves.
 Wednesday, July 25, 2007

Code Camp SA: Darren Neimke

Remote location On Sunday July 8th, Darren of Readify, and most importantly of Adelaide, closed the inaugural Code Camp SA with a quality presentation. After a couple of false starts it was eventually revealed that Darren would be discussing remote development.

Remote development doesn't necessarily mean sitting atop the Alps with a laptop and some pretty amazing Wi-Fi Internet. It can be a simple as working apart from your fellow development team members such that you can't simply step over to the neighbouring office and discuss your task face to face.

While I generally share an office with the other developers where I work, we do have two offices about thirty minutes apart and I have been working from home on an increasingly regular basis. With this trend likely to continue, the ability to effectively communicate with the team when separated is a growing concern.

In the office I can ask a coworker to look over my shoulder to give a second opinion on some code or we can step over to the whiteboard and represent the problem visually and discuss a solution. In my study at home I have email, I have instant messaging, and I have my mobile phone. These are all excellent systems for communicating verbally or textually but that's where it ends.

I've tried the whiteboard-style collaboration software but drawing class diagrams with a mouse or notebook touchpad just isn't helpful. The other benefit you lose on your own is other team members overhearing your coding pains and butting in with a solution that you probably would have struggled with for much longer before finally asking for help.

Darren demonstrated that having a Tablet PC (or a similar device) can be great for shared whiteboard sessions and new tools like SharedView can help with peer code review or even Pair Programming. I'm not sure what to do about "overhearing" your coworkers apart from a persistent VOIP conference call running in the background and that could be embarrassing when you forget it's on.

Overall, Darren's presentation gave me some good solutions for working remotely and gave me some other ideas to ponder on. And, Darren, if you are reading this, do you have any recommendations for coffee shops with good affordable Wi-Fi Internet in Adelaide?

Touched By PowerShell

Today I was asked how to "touch" all files in a folder in Windows to set the Modified timestamp to now. If I was asked about the Created timestamp I would have suggested to copy the files to a new folder and use the copies which will have a new Created timestamp. Unfortunately Windows doesn't seem to have an easy way to edit the Modified timestamp.

Thankfully, everyone who has Windows also has PowerShell, or should have, or should upgrade their OS. With PowerShell, touching files is easy:

Get-ChildItem * | % { $_.LastWriteTime = [DateTime]::Now }

 Saturday, July 14, 2007

Dirty Pictures

Example roadworks map Frustrated by the lack of up-to-date data for my TomTom, I started poking around the web for various sorts of local information. I discovered that the Department for Transport has a page listing current roadwork sites. Unfortunately, it is only available in HTML form.

Upon inspecting the source however, I discovered patterns in the data suggesting it is automatically driven from by internal database. I asked myself, what is the quickest, easiest, most hackish way I can suck this information into a map? The answer was Google Mapplets.

The result, is an XML Google Gadget that is hosted on my website but runs in the context of the Google Maps site and screen scrapes the DTEI's HTML using JavaScript. Worse Than Failure would be proud.

If you'd like to try it yourself, the gadget xml file is here. You should be able to open Google Maps, click the My Maps tab and choose Add Content where you will be taken to the directory. Choose the Add By URL and paste the link to my xml file.

I might tidy up the abomination that is my JavaScript code if anyone cares enough and submit the gadget to the public Google directory. I am also considering implementing the gadget in Popfly but I am waiting on being able to sign-in to that site.

 Wednesday, July 11, 2007

TomTom One XL

TomTom One XL Last week, after coveting my friend's GPS Pocket PC for some time, I purchased the new TomTom One XL car navigation unit. I had used my friend's GPS Pocket PC a few times and liked the Tom Tom software it had and the large screen of the One XL closed the deal.

I was very pleased to find that the One XL is self-installable with an elegant windscreen mount and separate car 12v power adaptor. The TomTom Home PC software installed and functioned surprisingly trouble free on my non-admin Vista x64 PC and connected to the One XL over USB just as easily.

Over the past week of use, I have only managed to crash the device twice (pretty good from my experience with iPods and mobile phones) and there is very little about the TomTom hardware and software combination that I can take issue with. It even connected to my "unlisted" Samsung mobile over Bluetooth to download updates.

My biggest problem with the One XL, which I presume will extend to most GPS devices sold in Australia, is the poor data. TomTom offer subscription services on top the initial purchase of the GPS device. Services like live traffic jam and road work updates, safety (ie speed/red light) camera locations, new Points Of Interest, and TomTom Buddies (a system for locating your TomTom using friends on the map). None of these "PLUS" services are offered in Australia.

Either the local organisations responsible for geographic data in Australia are useless at providing updated information or they try to resell this information at such an expensive price that TomTom can't justify the costs to the smaller Australia customer base. Neither situation would surprise me.

The Bakewell bridge and Glover Avenue have been out of service since October 2006 but are still valid routes in the One XL. Portrush Road, one of few Adelaide roads to have speed limits in the TomTom, still has 25km/h zones that have not existed since the major road upgrades finished over two years ago. The McDonald's and Pizza Hut near my home, that have existed for much longer than two years aren't in the standard POI set while other McDonalds's and Pizza Hut outlets are.

There are the minor afore mentioned issues with the Tom Tom itself too. Automatic time synchronisation assumes Eastern Standard Time instead of Adelaide's Central Standard Time (you'd think a GPS would know I was in South Australia). A planned route recently suggested I perform a U-Turn in the middle of an intersection: a maneuver legal in Queensland but not in South Australia.

Also, as part of the live traffic updates, the TomTom Home software allows the user to enter their own problem traffic areas they want to avoid, however just because Australia can't receive feeds from TomTom, the software won't allow the user to add their own either.

Ultimately though, while there are many issues to consider, the unit really helps with finding parking or getting somewhere new at night when street names are hard to read. I feel much more confident taking long drives to the middle of nowhere too knowing I can easily find my way back or give someone my longitude and latitude coordinates. I have no regrets about the purchase but I am reminded once again that online/data services in Australia still suck.

Code Camp SA: Mitch Denny

Mitch's presentation about "TFS Virtualization and Advanced Build Techniques" was my highlight for day one of Code Camp SA.

Mitch suggested that one of the major benefits of virtualizing Team Foundation Server was the ability to rollback the whole server at any point. While that is certainly helpful if the service packs don't install right the first time, the benefit I've found personally is being able to pick up the whole VHD and drop it on another Virtual Server machine without any reconfiguration. Great for hardware failures.

Mitch also suggested getting a *big* server to host the virtual TFS and use the slack space to host additional virtual build servers. We toyed with a build server once and based on performance I figured a physical machine would be best but Mitch demonstrated a great idea combining multiple virtual build servers with a false TFS build agent to auto-redirect to a free, pre-cleaned build server.

In fact, the Readify team have some great TFS tools available to get the most from the system:

Mitch showed how, with these tools and some PowerShell glue, you can have a complete, automated process to go from a check-in, automatic build, to update dependencies, rebuild those, deploy for QA, then deploy to production. Nice.

With a lot of VB in our production code we've been held back from a CI system by the MSBuild secondary reference issue but a quick chat with Mitch after the presentation cleared that up:

Me: "Doctor, it hurts when I do this."

Mitch: "Well, don't do that."

 Sunday, July 08, 2007

Code Camp SA: Dave Glover

Dave Glover, an always delightful presenter, spoke on two topics this weekend. Saturday morning he spoke about Silverlight and the related Popfly, and Sunday morning he repeated his Windows Mobile 6.0 presentations from Code Camp Oz for those who missed it.

I have been reading many things about Silverlight over the past month or so but have been too overwhelmed to really look into it. Dave's coverage was just what I was looking for. Ultimately, Silverlight is Microsoft's answer to flash, and in version 1.0 with the JavaScript backend it is especially so. However, version 1.1 will be the real winner, combining the existing XAML/WPF based visual layer with .NET code behind.

The range of Microsoft Expression tools make drawing and animations look easy, even with Dave wrestling with a notebook touchpad. Microsoft are tackling portability of the Silverlight engine properly too with plugins for both IE and Firefox on Windows and even an implementation for Mac OS. The Linux community have stepped up to the plate too with their own complete implementation called Moonlight, developed in a stunning 21 days.

Popfly is a neat system for combining multiple "Web 2.0" applications into a clever mashup. Examples, like plotting Twitter messages onto Virtual Earth or building bar charts from custom web services are just the beginning.

Code Camp SA: Michael Baker

Michael Baker presented a session entitled "Design By Contract" on Saturday morning. Design by contract is a broad idea that can be approached in several ways. Michael focused on the idea of using pre- and post-condition macros in all your functions and combining that with extensive testing to ensure the conditions are met at runtime on all code paths. He also focused on implementing the macros as assertions rather than error return codes or exceptions.

I think that design by contract should not be treated so specifically but more as a combination of both discipline and an effective implementation for early problem detection.

You may establish a rule to document all methods with a comment header describing the intentions and expectations or you may insist (like FxCop) on throwing exceptions for invalid parameters. The important point is that it is done consistently.

As for effective implementation, comments are great for readability and guard clauses or unit tests are great for automated checking. However, the best outcome will be achieved with a system that can verify at compile time that all code paths pass the conditions and perhaps autogenerate human readable documentation too.

Defining the type of method parameters and return values in C# and VB is already a basic form of compile time verification of very basic conditions. Microsoft Research offers a variant of C#, known as Spec#. It extends on the C# language using new keywords and a variety of custom attributes to enable the developer to specify, declaratively, what values are valid as inputs and outputs from methods. The Spec# compiler is then able to use basic static analysis technique during build to highlight problem errors of code that could pass unacceptable values.

However, this all falls down when there are environmental conditions that cannot be verified at compile time. The absence of an expected file could throw an exceptions at runtime and the availability of network resources is a common problem also. Both Java and Spec# help here with checked exceptions and more advanced static analysis tools can track where possible exceptions can be thrown and verify that the application handles them appropriately.

Code Camp SA: Che Metcalfe

I turned up a few minutes late to this presentation after wrestling with directions to the venue. Based on the title "Kukan Studio and mEgaSA" I had no idea what to expect. The presentation was based on the idea of starting a mobile network, Podmo, where developers can easily publish their mobile games and applications and get a fair deal compared to other mobile content publishers.

The biggest highlight for me was the confirmation of something that I had always felt but never investigated: mobile phone data rates are disgustingly and prohibitively expensive. According to data Che had pulled from the Telstra website a few months ago, a user with a casual data plan can expect to pay approximately $15.00 per MB. On the Telstra Next G Network (which I use) the minimum expected speed is 550kbps which equates to about $45.00 of data downloaded in a minute at the afore mentioned rate.

Even better, in true geek style, Podmo plans to utilise Bluetooth hot spots at cafes, homes, and possibly eventually a scatternet across the CBD to enable free data for mobile users looking for Podmo contents. Way to stick it to the telcos!

 Saturday, July 07, 2007

Code Camp SA vs Code Camp Oz

Peter Griffith introducing the next presentation Today I attended the inaugural Code Camp SA, a two day event with presentations about .NET development inspired by the successful Code Camp Oz held in Wagga Wagga over recent years. Not everyone in Adelaide can afford the time and travel costs to spend a weekend in Wagga so it is great that ADNUG, UniSA, and the ACS have sponsored this event.

I was privileged to attend Code Camp Oz (CCOZ) this year and can't help but compare Code Camp SA (CCSA) to it. CCSA is held in a well-equipped lecture theatre at the local university and an excellent array of accomplished speakers have been gathered to present on relevant topics and upcoming technologies.

The sponsors have also arranged for morning and afternoon tea to be provided to attendees and a tasty BBQ lunch too. All this comes at no cost to the attendees. Unfortunately, CCSA missed a few issues that the CCOZ organisers have learned to expect.

Firstly, the University of South Australia's City West campus is not an area I'm familiar with, posted signage would have been very helpful for locating the venue.

Secondly, attendees like to bring notebook PCs to the presentations and most notebook batteries don't last the seven hours of the event. CCOZ supplied a box of power extension cables to allow attendees to switch their notebooks to AC.

Lastly, for an event dedicated to software developers, the wireless Internet connectivity was poorly organised. At a time when my work colleagues, my friends, myself, and even my parents all have high-speed wireless broadband connections at work and home, I expect the university to offer the same.

 Friday, July 06, 2007

Still Recovering

A new HP Compaq 6710b business notebook arrived at work the other day. It was very shiny, had plenty of grunt, and was preloaded with Vista... apparently. Somehow, during the excitement, the notebook was turned on for the first time and then the lid was closed again before Windows had started. We didn't want to go through the initial setup just then, we had more important things to do.

Later, the time finally came to setup the new notebook and install and configure all the usual goodies. Unfortunately, just after the BIOS POST and before Vista started loading, we were greeted with an Vista boot loader error message: "\windows\system32\winload.exe could not be loaded because the application is missing or corrupt". All other options in Windows Boot Manager lead to the same result and no amount rebooting was going to help.

The answer to this problem, as described on the Boot Loader error screen, is to boot from the Windows installation disc. However HP like many OEMs today, do not ship any CDs with the hardware. To get the recovery disc you would normally burn them after setting up Windows for the first time (Catch-22) or order them from HP (takes about three business days).

Next idea: to be able burn the recovery discs after setting up Windows, HP must, like other brand notebooks, have the recovery images on the hard drive in a hidden partition. I downloaded the user guide from the HP website to find out how to use the recovery images on the hard drive. Option 1, run the HDD based recovery tool from the Start Menu (uh...) or Option 2, press F11 while booting to start the recovery tool without Windows. Excellent.

F11, F11, F11, F11,... doesn't work. POST screen lists others but not F11. Googling suggests that the HP boot-time recovery tool is not a BIOS feature but a boot loader on the recovery partition and the active partition is set wrong if F11 doesn't work. Ok, what is the easiest way to change the active partition on a non-booting PC?

We booted from a Windows XP install disc to use the Windows Recovery Console. XP install couldn't detect a hard drive in the notebook and refused to go any further. We created a bootable floppy with a USB floppy drive on an XP machine and copied diskpart.exe to it. Diskpart requires the full Win32 environment to work. We created a bootable Windows ME boot floppy but FDISK is unusable on drives larger than 137GB.

Lastly, we waited for the Ultimate Boot CD to download from a very slow mirror and we got Ranish Partition Manager running on the notebook. Ranish showed the notebook's hard drive correctly as a 160GB drive. It also showed that there was a single 40GB active partition formatted as NTFS and the rest of the drive was unpartitioned. Bugger.

We are now waiting for the recovery discs to arrive in the post from HP. I'm guessing that the guys at HP who setup the initial Vista image for this notebook range forgot to leave the recovery partition intact. Just goes to show though, shut down your PC properly.