ack and pv

The latest additions to my shell toolbox are ack and pv. Very cool indeed!

/shell | 03 Feb 2009 | #

Real-world use for MD5 Collisions

Practical implications of weak cryptography - Creating a rogue CA certificate. What does this mean? If this approach is generalised, you can't trust secure websites at all -> every single secure website may not be who you think they are, even if your browser happily presents the "little green tick". So who uses secure websites? ecommerce (Amazon, eBay, your bank), the tax office, many government departments, social networking portals, software update sites, software download sites. Scary, yes, but highlights the need to build seemlessly upgradable cryptographic software solutions - and we need to do this now before public confidence in our virtual world is eroded.

| 31 Dec 2008 | #

Python 3.0 released!

Python 3.0 is released into the wild. Download now...

/Python | 04 Dec 2008 | #

Mind Rot

Charles Petzold on Does Visual Studio Rot the Mind?. For some reason I hadn't seen this before (Thanks Kevin for pointing it out). It raises some very important questions about code automation, programming productivity, maintainability, data driven software and the importance of naming things right. It's very interesting as an opinion piece circa-2005 for comparison against where we now are at 3 years later.

/C-Sharp | 27 Oct 2008 | #

Python 2.6 released!

Awsome news! Python 2.6 has been released! Here's a list of what's new.

/Python | 02 Oct 2008 | #

Perl's CGI is broken

  use strict;
  use warnings;
  use CGI;
  use Test::More qw( no_plan );

  my $cgi = new CGI;

  $cgi->param('foobar', 6);
  ok( $cgi->param("foobar") eq '6', "6 is a magical number" );

  $cgi->param('foobar', 4);
  ok( $cgi->param("foobar") eq '4', "And 4 is a favourite" );

  my %hash = ( "random_key" => '13' );

  $cgi->param('foobar', $hash{'randomKey'} );
  ok( $cgi->param("foobar") eq '13',
      "but a simple typo in a hash key shouldn't be so hard to find" ); # Fail
  ok( $cgi->param("foobar") eq '4',
      "so it hasn't changed but there's no complaint" );

  $cgi->param('foobar', $hash{'random_key'} );
  ok( $cgi->param("foobar") eq '13', "The key is to get the key right :-)" );

What sort of deranged API silently ignores calling a setter with an undef? Surely a warning or exception could have been thrown? Or perhaps setting the value to undef, or to the empty string, or even "You messed up, Idiot!". But quietly swallowing the error and leaving the value unchanged is really bad form.

This sort of thing doesn't rate well on Rusty's API design advice: How Do I Make This Hard to Misuse?. Grrr.

/Perl | 20 Jun 2008 | #

Biometric Insecurity

A very nice succinct blog about why we should be cautious about biometric authentication schemes. It's a useful read.

I've been discussing this with Ken for the last couple of years - security systems are only as secure as the weakest link in the system. If your biometric reader (fingerprint, retinal scan, DNA fingerprint, whatever) just dumps raw data over some data bus, who says you can't do a man-in-the-middle data capture and replay attack? How about your storage of the data? How about the security of that database? And the operating system on which that database sits? How about protection from unauthorised modification? How about that data falling into the hands of unauthorised users? We've already seen databases with the records of millions of citizens "disappear". Given that our judiciary holds biometric data in very high esteem, who will guarantee its integrity?

I'm all for improving security, but the illusion of security only harms the innocent. It seems to me that many biometric system proposals are susceptible to the Bribing the Doorkeeper on the Great Wall of China-kind of attack.

| 05 Dec 2007 | #

pyglet enters public beta

Ever since Richard Jones introduced pyglet at OSDC 2006, I've been waiting expectingly for it to get to beta. Well, that day has come!. As the webpage says, pyglet is a cross-platform python windowing and multimedia library for Python - something cool to play with. Congratulations to all involved!

/Python | 09 Nov 2007 | #

Shell redirection

The standard idiom of redirecting stdout and stderr to /dev/null is of course:

          frobnicate 1>/dev/null 2>&1

What is lesser known is that this can be short-cutted to:

          frobnicate &>/dev/null

/shell | 09 Oct 2007 | #

Maven Trick #257

The project I'm looking at right now uses Maven to do dependency management - and trust me, the project has so many dependencies it _needs_ Maven :-)

So the question begs, when integrating the large working source tree with external-to-maven tools, how do you get the CLASSPATH out of Maven for use elsewhere.

Many googles and reading bits of Maven: The Definitive Guide later the solution is a semi-obvious:

mvn dependency:build-classpath

/Java | 04 Oct 2007 | #

Playing with certificates

One frustrating thing is working with digital certificates that someone else has created and deployed - and you have to pick up where they left off. I mean, with a directory of crt, ket and csr files, how do I verify which ones belong to which? and exactly what was that openssl command-line to check? It's not something I do everyday. That's what I use this blog for - to cover the overflow when my brain is full

Reverse encoding the ASCII-armored text is relatively simple:

• openssl x509 -noout -text -in webserver.crt
• openssl rsa -noout -text -in webserver.key
• openssl req -noout -text -in webserver.csr

Of course checking the modulus and public exponent sections manually is error-prone. So make it easy on yourself and check the shorter hash instead:

• openssl x509 -noout -modulus -in webserver.crt | openssl md5
• openssl rsa -noout -modulus -in webserver.key | openssl md5

And once you realise that the private key has been lost, or that the certificate has expired, you'll need to do one or both of the following to regenerate thus:

• openssl genrsa -out webserver.key 1024 to generate the key,
• openssl req -new -key webserver.key -out webserver.csr to create the certificate signing request file to ship to the certificate authority.

| 12 Sep 2007 | #

Moonlight: Silverlight on Linux

Miguel and his team have been busy - 21 days after Microsoft revealed their cross-platform browser-hosted CLR efforts, the free software world have reimplemented a good junk of it.

• Implementing Silverlight in 21 Days (aka Moonlight)
• Another report of the 21-day hackathon at Novell
• Moonlight-based Desklets
• A Mono Developers Guide to writing XAML by hand. (ie. the language of Silverlight)
• Silverlight Surface demo

Besides the cool tech and the flashy demos, what is very impressive by itself is what can be achieved by a small focused team in a small amount of time. Slow viscosity/inertia/velocity (or whatever you want to call it) is the enemy of innovation. Slow development progress begats further slow progress, whereas quick results sky-rockets the motivation of the team. Seriously, the more projects I see, the more I'm convinced that if you can get the obstacles and interruptions out of the way then you have a great chance to get a coherent, innovative, quality software product developed quickly. It's less about process, it's less about documentation, and it's more about focus.

The hackfest/sprint is certainly under-utilised as a software development lifecycle today.

/C-Sharp | 29 Jun 2007 | #

Dynamic Support for Scripting Languages

Microsoft today announced 1st class support for Dynamic Scripting Languages for the .NET Framework. That is, they're adding first-class support for dynamic languages on top of the Common Language Runtime (CLR) - they've coined it the Dynamic Language Runtime (DLR).

Currently the target languages are Python, JavaScript (EcmaScript 3.0), Visual Basic and Ruby. I'm not sure if this is big P Python, or just improving the integration of IronPython.

This is important, even though it's just a Microsoft announcement for their Windows platform. The Mono project will no doubt be inspired by Silverlight to raise the bar again. Looks like we're going to get multi-language, cross-platform, client-side immersive web applications - this is AJAX++.

Links:

• Jim Huginin's announcement
• Miguel de Icaza's prophetic post
• Microsoft's Silverlight
• IronPython

| 01 May 2007 | #

Large Behomeths, Small Startups, and Open Source

So Paul Graham has come out and said that Microsoft is dead. That's a pretty big claim on a company about to get several million sheep to pay several hundred dollars each to upgrade to the latest offerings of Vista and Office. But it might not be fatal - if they have the will and are willing to use the tens of billions of dollars they have in the bank, it would be possible to change corporate culture - but it would be a monumental change. There are good signs already (and some negative ones), but it's really all small stuff.

The issue is that lumbering behomeths have trouble being agile. Innovative software doesn't get written by large companies with long-standing culture practices and heavy-weight processes; but rather by small nimble startups, where hours worked are long, and everything can be challenged. Demotivation is limited because there are no boundaries.

One particular instance where there are no boundaries (read this as disruptive technology) is of course an Open Source development model. The keys here are software freedom, distributed development, collaboration, international 24x7x365 involvement with full internationalisation.

While I have my own personal biases (blind-spots?), I think the Open-Source Software snowball is rolling down the hill, gaining momentum, and can't be stopped. Jump on board, or be overtaken.

| 11 Apr 2007 | #

Step 3 - check!

After considerable negotiating, arguing, pain (by me) and hard work (by James and Michael) my day-job project finally gets to tick off the Step 3 of the Joel Test - Do you make daily builds?.

If our platform had a GNU build system this would have been easy - make, xunit, cron, mail - but this is on Windows, so we have a solution held together by duct-tape and chewing gum (with more and more python getting added :-) But it is holding together.

The good news is that I now get an automated command-line daily build on a controlled build-box, straight out of source control, followed by xunit tests run automagically, with the result emailed to me every day (including logs so that failures can be analysed without returning to the build box).

Yay!

| 28 Mar 2007 | #

Trusted Code 2.0

So back in .NET 1.1 days we had a problem running C# code off a remote fileserver - Windows trusts, by default, code on the local machine only. Fortunately there is a work around, which we dubbed The Dilbert Zone.

Moving our product across to .NET 3.0 means changing the security trust on the local machine for this new environment in a similar fashion.

Start the appropriate configurator found at Start|Settings|Control Panel|Administrative Tools|.NET Framework 2.0 Configuration

Navigate to Console Root|.NET Framework 2.0 Configuration|My Computer|Runtime security Policy|Machine|Code groups|All_Code|LocalIntranet_Zone and select Add a Child Code Group.

Create The Dogbert Zone, with condition type = Zone, Zone = Local Intranet, and Permission set = FullTrust. Select Ok a few times and exit out of this user-unfriendly administrative console.

Restart Visual Studio 2005, and voilà - it just works as expected.

/C-Sharp | 16 Feb 2007 | #

SHA-1 not considered safe

So I don't know how I missed this:

Federal agencies should stop using SHA-1 for digital signatures, digital time stamping and other applications that require collision resistance as soon as practical, and must use the SHA-2 family of hash functions for these applications after 2010

Both MD5 and SHA-1 aren't safe as previously mentioned - but having an advisory indicating that you should stay away from SHA-1-series algorithms is biting.

Here's a paper on the implications on S/MIME, TLS and IPSEC. This is something that really should be considered soon.

| 21 Jan 2007 | #

About AES

Russell blogs about choosing encryption algorithms. He comments:

The US government (which incidentally employs some of the best cryptologists in the world) recommends encryption methods for data that is important to US interests (US military and banking operations for starters). Why wouldn't you want to follow those recommendations? Do you think that they are putting back-doors in their own systems?

This is true, but for an additional reason not mentioned. The current block cipher standard is AES. While it was chosen by the US government as a standard, it wasn't develoed by the US government - making the chance of back-doors even less likely. This is not the case for DES - which was an in-house US Government development effort - meaning that it's open to claims of back-door introducing.

There are some questions asked about whether there are inherit weaknesses in AES, but these centre around the "newness" of the mathematics used - the strength of any crypto system is based on solving "hard" mathematical problems, so if someone finds an easy way the "tower of cards" all falls down - and all data is open. But the same could be said for ECC. That's why some people encrypt their encrypted data with a different encryption algorithm - even if one crypto system falls, there's another barrier protecting your data.

So, as a recommendation, you need to ask a few things - how important is the data you are wanting to protect? What timeframe does it need to stay secure? If it needs to be protected forever - like who killed JFK :-) - then you need a really strong crypto system. If it's a personal deep dark secret, probably just your lifetime. If it's your travel planes, only a few weeks. If it's your online banking session - it's only minutes. The other thing is identifying your adversary - if it's the government, then nothing really is secure enough - be assured that if it's in their national interest they will break open the data. If it's some local phisher/cracher - then a much weaker system is all that's needed to protect you as they don't have the mathematicians, nor the computing power, to cause you grief in the timeframe that the data is sensitive.

The easiest solution though is have nothing to hide :-)

Addendum: There's nothing new here - just go read Schneier for more info.

| 16 Dec 2006 | #

Java to be Open Sourced

As gman says, "Hell has frozen over."

Sun will be making an announcement here very soon that Java is about to be released under GPLv2.

This is great news, albeit 5 years too late. I hope Java gets some momentum again now as a result. Well done Sun! Long live Java!

/Java | 13 Nov 2006 | #

Microsoft .NET Framework 3.0 released

.NET 3.0 has finally been released. What's is it?

Basically it's .NET 2.0 with extra APIs such as Windows Presentation Foundation WPF (was "Avalon"), Windows Communication Foundation (was "Indigo"), workflow stuff etc. What will it let me do? Hopefully do high-level programming for rich clients. Hopefully not having to avoid bugs in the 2.0 release.

The most interesting question is how this relates to Mono. Will these new APIs be supported?

/C-Sharp | 08 Nov 2006 | #

SHA-1 partial chosen plaintext attacks successful

So back in February, we found out that SHA-1 was gone - researchers could generate 2 plaintexts that generated the same hash. But at least the plaintexts were gibberish, meaning that while SHA-1 was broken, the break was of limited use.

Now comes a more serious blow - in a similar vein to the previously reported MD5 attacks it's now possible to choose part of the plaintext and still get the same hash. Yikes.

Quoting the article:

         Using the new method, it is possible, for example, to produce two HTML
         documents with a long nonsense part after the closing  tag, which, 
         despite slight differences in the HTML part, thanks to the adapted appendage
         have the same hash value.

Now what if I could add some nasty javascript to a web page and retain the original hash? Validating the web page with a MD5 or SHA-1 hash won't tell you of the maliciousness. Combine that with DNS redirection and you have something a bit scary. Can you say phishing attack?

We need a new hashing function, openly and publicly selected, just like AES. Moving to SHA-256 or SHA-512 are just stop-gap measures.

| 28 Aug 2006 | #

A small update on work

In 7 months we've made 13 formal releases of our software - this morning we sat down and looked back at this busy year to date. That's 3 product lines out of the same source tree - #ifdef'd and managed by ClearCase config specs, each product release interleaved as customer deadlines dictated. All of the code "fagan" inspected, and manually tested (don't get me started on manual testing). Soon you'll be able to buy the results of our blood, sweat and tears - boxed and shrunk-wrapped.

So sitting here on a Friday night, thinking about what we've achieved, I have to say that I'm pretty happy with the team. We've achieved a lot. We've pushed the boundaries, and now we're looking to push them even more - hopefully a new product line coming out the successes of what has been achieved (*sigh* more ClearCase branching :-) using different hardware, adding new cool functionality to the product line, and being able to support more networks.

Besides happy, I'm also exhausted. Holidays aren't far away, and I'm looking forward to both re'ing and laxing.

| 14 Jul 2006 | #

eXtreme Programming Overview

While trying to find an online version of the famous Kent Beck quote for a team member:

       "Make it run, make it run, make it fast, make it small"

I found the following XP Overview quite handy.

| 07 Mar 2006 | #

Music to code by

So, someone showed this to me 12 months ago, but I only just realised that it's good to code to - badger badger badger.

| 06 Feb 2006 | #

Software Requirements Management

This is the sad truth of commercial software requirements management.

| 31 Jan 2006 | #

WebClient class omission

In the .Net framework there is a nice helper class WebClient which makes it easy to [up|down]load files off a network. Unfortunately it ignores the GlobalProxySelection setting - meaning that there is no way to use WebClient through an http proxy.

A small note is found on this page which suggests that WebClient has a proxy property in .Net 2.0. This is another indication of the immaturity of the .Net Framework library - so many simple things have been overlooked. sigh.

Note: Haven't checked the implementation at Mono yet.

Update: Ok, so it is possible. WebClient will honour the global proxy settings, just not with the default credentials of the user invoking it. For reference...

     System.Net.WebProxy proxyObject = new System.Net.WebProxy(proxy);
     proxyObject.Credentials = new System.Net.NetworkCredential(username, pass);
     GlobalProxySelection.Select = proxyObject;

     WebClient client = new WebClient();
     client.DownloadFile(this.uri, localFilename);

Thanks James...

/C-Sharp | 19 Jan 2006 | #

HP Laserjet fun

Having fun with C# and HP Laserjets

/C-Sharp | 07 Dec 2005 | #

Generics go mainstream...

James, you mean like Ada83 had 22 years ago? and Java 5 also has now?

It's taken just a little while for this wonderful language feature to make it out of the trusted computing / defence / aerospace market... :-)

And before you say it, C++ templates are just macros, so they don't really count :-)

| 14 Nov 2005 | #

Parallel Peer Programming

While looking at Planet+ I found Scott James Remnant's excellent piece on Parallel Peer Programming.

It's not very surprising to see that the fantastic efforts of Canonical are due, at least in part, to using some XP-techniques.

| 04 Nov 2005 | #

IronPython now free

Previously I ranted about how IronPython made a huge mistake by going non-free - meaning that Boo was now the pythonesque language of choice for cross-platform development.

So this morning I was pleasantly surprised to see that IronPython is now usable - it's changed licence to the CPL. This doesn't take away from Boo, but gives us even more choice for rapid application prototyping.

Well done Jim Hugunin!

/C-Sharp | 31 Oct 2005 | #

Converting ArrayList to string[]

For the third time today someone has asked how to convert an ArrayList into a string[].

    ArrayList myArrayList = new ArrayList();
    ...
    string[] myStringArray = (string[])myArrayList.ToArray(typeof(string));

That's how.

/C-Sharp | 19 Oct 2005 | #

IP Address parsing weirdness

Hmmm, I wouldn't have expected this:

    string str = "2.10";
    if (null != System.Net.IPAddress.Parse(str))
    {
        System.Console.WriteLine("This is an IP address");
    }
    else
    {
        System.Console.WriteLine("This is NOT an IP address");
    }

Can you guess what is printed? I didn't know you could abbreviate IPv4 addresses. Can anyone point me to an RFC that justifies this weirdness?

Update: Well, I'll be a suck egg mule. See your local copy of inet(3N).

/C-Sharp | 07 Sep 2005 | #

Code Comments Aren't Evil

Stewart writes, "Comments Are Evil", most specifically, "Remember kids, comments in code are evil. They are wrong - or misleading at best".

Unfortunately, this is wrong advice.

Code comments can be misleading - they can do more harm than good. But I've seen both ends of the spectrum, the Ada-comment-every-line-describing-what-the-loc-does and the *nix kernel hacking idea of the-code-is-your-comments idea. Both are extremes and both are wrong, even though "the definitive record is the code itself".

Code comments should:

• state the functional intention of what the block of code is trying to achieve, not how it is achieving it;
• highlight any tricky gotchas for future maintainers; and
• identify any use of common algorithms or design patterns or similar so that someone trying to understand it may get a leg up.

Getting the level of commenting right is hard, especially if you haven't written much code, or if you are unfamiliar with the domain or the implementation language. But commenting done right can greatly assist yourself and others when you revisit that chunk of code - whether that be to find that heisenbug, or to add new functionality, or even just understand what you were trying to achieve back 3 weeks ago.

| 11 Aug 2005 | #

More good junkcode

Ian Wienard's published junkcode collection is pretty cool. Sometimes the solutions being solved are a little obscure, but real-world and not contrived.

Good stuff.

| 09 Aug 2005 | #

Standards are important for documents too

Standardisation of code is a no-brainer. We need to follow applicable standards so we can interroperate with other pieces of software.

But we also need to follow standards when writing documentation. Example: I'm currently spending today reformatting a (binary) document by hand for changes in document styles. It's going to take 3-4 hours and it's very handrolic. There is no benefit besides consistency to be gained, but since this document is part of a greater set, the changes need to be made.

The problem is that a standard was originally agreed upon, but someone decided to start tweaking the format for some trivial benefit. Now since we had inconsistency, the format was reconsidered and changed substantially. Now I don't have a problem with that, but since the document format is un-machine-editable binary, the changes have to be made by hand. If the document source was XML it would be just an XSLT transformation away.

Laziness is a wonderful attribute of the OSS community. Spend more time upfront to save time later - the downstream task time drops dramatically, as well as gaining many benefits. See Tridge's keynote on auto-generated code in Samba. This is sadly missing in other worlds - the focus is on peep-hole optimisations and not optimising the whole process.

| 27 Apr 2005 | #

Gtk# or Windows.Forms

Mikal asks whether developing cross-platform .NET you should use Windows.Forms or Gtk# as your widget set.

Good question. Mono now uses a non-Wine Windows.Forms implementation by default, which is a plus going that way. I think Gtk# looks nicer, so right now it might be looking at what your primary audience is - if you are targeting Windows and some Linux, go Windows.Forms. If you are targeting Linux, with some Windows, go Gtk#.

I think from a compatibility and stability front, Gtk# is probably a better choice today.

/C-Sharp | 08 Apr 2005 | #

IronPython makes a release, but Boo is fun

So Jim Hugunin made a new IronPython release here after a rather long break, and despite repeated calls to kick start developemnt again over the past 6 months.

In the mean time, I moved to Boo instead.

I was hoping that I'd still have time to track and play with IronPython, but now that I have some investment in Boo, that might just not happen. IronPython is *real* python, which is a major bonus, but Boo is *almost* python and gives me static typing, my biggest complaint against python. It's a bit of a toss up which one is better technically. But, IronPython has just moved to a Microsoft Shared Source licence, so the final answer is now clear - it's one of right and wrong. Boo, licenced under BSD/MIT, wins hands down. Sorry IronPython, you are destined to be history now.

Both projects have sparse documentation, so DumpAssembly has been quite valuable.

| 25 Mar 2005 | #

Hash attacks

Schneier comments on more hash attacks. MD5 collisions in 8 hours. /etc/passwd soon to be in trouble.

| 11 Mar 2005 | #

DumpAssembly

After stuffing around trying to work out what name a .Net assembly used in an interface to an existing C library, I wrote DumpAssembly. At least now I can find class and method signatures, even if there is no documentation for that API.

I note Mikal did something similar once before for different reasons, and I know others have probably done similar, but I wanted to play with reflection anyway :-)

/C-Sharp | 07 Mar 2005 | #

SHA-1 gone!

Following on from the MD5 collision, it has been reported by Schneier that SHA-1 has been broken.

No collisions have been made public or even claimed, but the claim is that collisions are possible in 2^69 hash operations instead of 2^80. No immediate threat, but the writing is on the wall. We need better hashing algorithms soon.

| 17 Feb 2005 | #