Could the same technique be used to filter your Twitter timeline? This could be either to detect spam, or, as I wanted to use it, to filter out things that I’m just not interested in, e.g. the current discussion of American politics. (Not that this is not worthy of discussion, it’s just that I have very little interest in it, and live in England.)

The idea is that upon reading a tweet, you decide whether it is interesting or not, and mark it accordingly. The filter learns that this particular combination of words is spam or ham, and uses this to adjust its assessment of the spamminess of other tweets. Tweets are then displayed in different colours, depending on the probability being high, low or medium. The record of your choices is stored on disk, and reloaded on startup; the effect being that over time, the program learns your interests.

One problem is the 140-character limit of tweets. This does not provide much with which to train a filter! However, I have had some initial success, not yet to the extent that I could simply not display low priority tweets; the filter still needs tuning.

The filter is an extension to Cameron Kaiser’s excellent TTYtter command-line Twitter client, which is written in Perl 5. As TTYtter does not currently provide a mechanism for an extension to indicate the ‘priority’ of tweets, and hence, colour them differently, I have patched TTYtter 1.2.5. In addition to TTYtter’s requirements, you will also need to install Gea-Suan Lin’s Algorithm::Bayesian module from the CPAN. I also recommend installing Term-ReadLine-TTYtter. The filter memory is persisted using Data::Dumper.

To use, invoke TTYtter as:

./ttytter.pl -readline -exts=bayes.pl -ansi

To mark tweets as spam, use the command "/spam xx yy .. zz" where xx, yy, zz are one or more tweet menu IDs, as shown in the incoming timeline. To mark as ham, use "/ham xx yy .. zz". Alternatively, use "/- xx yy .. zz" for spam and "/+ xx yy .. zz" for ham.

Tweets are then coloured according to the filter. Ham or interesting tweets are bold white, spam is yellow, undecided is plain white.

The weighting of the filter’s probability could possibly be better: I’m splitting the probability range [0..1] into three equal parts, [0..1/3), [1/3..2/3) and [2/3..1]. This was a guess

The patched TTYtter, and the module, can be found on the DevZendo Miscellaneous Mercurial repository . Changes to TTYtter are under the same licence as TTYtter itself; the Bayesian filter extension is under the Apache License v2.0.

I’d considered ScribeFire, I used it previously, but was not impressed. I’d hoped that it might have improved recently, but a recent rewrite received something of a panning in reviews, hence, the search for something better. Qumana seems to do what I need.

It syncs up well with WordPress; I now have some of my earlier posts on the laptop.

It doesn’t seem to like the apostrophe in my blog title, showing it as a HTML Entity – bloggers do use apostrophes, you know; some of us, correctly. It does the same with categories, one has an ampersand which it shows as &. Hmmmm.

It gets the funky foreign letters in BoøkWürm correct; and even inserted that as a hypertext link to that category. As I mouse over the link in the editor, its address doesn’t show anywhere. I’d have to switch to HTML Source View.

I can’t seem to add categories from it. That’s a pain. It does allow me to add tags though, which link through to Technorati

It’s written in Java. This has pros and cons. Portability is good. There are small resize and toolbar layout problems; it has that certain Javaesque je ne sais quoi. I know from my own Open Source Mac OS X endeavours that making Java programs look beautiful is hard, and on the Mac, even harder. (I use Quaqua; Qumana uses JGoodies Looks, which I use on Windows and Linux)

The editor is very nice – offering WYSIWYG and HTML Source View. It has basic formatting options: fonts, underline/bold/italic/strikethrough, alignment, lists, blockquotes, indent/outdent, images and links. Just enough; I don’t want anything fancy. Word count would be useful. The spell checker is useful, but I can’t change the language from American English to British English.

I won’t be making use of its Ad feature. The Blog Manager allows multiple blogs. Could be useful

In all, a nice application – and it seems reliable so far.

As I expect to be programming and using computers until I can’t physically do it any more, I need to look after my ability to type effectively. Qwerty isn’t optimal.

A couple of times in the last few years, I’ve tried to remedy this by learning the Dvorak layout. I’ve gone through lessons with a tutor program twice, but then as soon as I started using it for the day job, the difficulty of adjusting my muscle memory’s knowledge of Vim and Eclipse rears up, and I stop using it. Having to enter the usual panoply of symbols whilst programming is also more difficult.

If I were only writing straight English, this wouldn’t be so much of a problem. So, write more English… like this blog, which has become woefully neglected. There’s also the possibility of resurrecting writing short stories. (Thomas Pynchon need not start getting worried.) I also like the idea of 750words.com, where a writer puts down 750 words of any stuff that comes into your head, first thing, every day.

The Colemak layout is supposed to be better for programming with than Dvorak, so for my third and final attempt at speeding up, I’m using aTypeTrainer4Mac with this layout. It’s going well so far; I’m just moving off the home row.

So the plan is to train up, then when I’ve got all the keys in muscle memory, write English daily with it. Then and only then start looking at adapting it to use for programming.

This is in preparation for another large-scale project, to replace my hodgepodge of clunky shell/perl backup scripts and Windows XP backup with something a little more… industrial.

I need a decent backup system. Time Machine looks great for the Mac, and I use it… but don’t entirely trust it. I use FileVault, and having to log out just to start a back up of any bands that have changed is just retarded.

I could just go out and buy Retrospect, but that’s not my style

It has to back up in the background, notifying the user of activity unobtrusively. I’m thinking of a continuous incremental system, from the usual trio of operating systems, managing retention periods and disk space. I want to go back to any point in time and see/restore that data. It has to store all file metadata, so, extended attributes, modes/ACLs, multiple streams, sparse files, long filenames, symlinks, long symlinks to long filenames, ownership, etc. I want to be able to translate backup data into tar streams for verification and off-site backups. i.e. give me a tar stream of the whole data from last Wednesday, 18:34 onto this disk. It has to have a sparse, Zen-like UI – I’ve seen too many backup programs that were obviously written for developers!

I don’t ask much, really…..

The first seeds have been sown; I have the beginnings of a cross-platform library for filesystem access and access to all the above enhanced metadata that’s not available in Java. I know Java 7 has some of this; I can’t wait for Java 7. I wrote quite a bit of this for UNIX some time ago, using Ant and Eclipse, but didn’t do it using TDD. I’m revisiting this, and starting from first principles using TDD.

I also need to reuse some common code from MiniMiser. Mercurial makes transplanting code between repositories quite easy.

I think with this project that I’ll open it as soon as I’ve cut something that builds in my CI server. Shouldn’t be long now.

Actually, there has been progress on the homebrew Z80 system!

No actual hardware yet, but I now have a working simulator, with the beginnings of a Z80SIO serial I/O simulator. This correctly intercepts all the I/O port access that the chip understands, and displays the state of all the registers on screen, with an interpretation: rather than just dumping the raw register contents, it describes what each bit or bit group is intended to do, i.e. 7-bit data, 2 stop bits, receive interrupt enabled.

I’ve got a prototype perl script communicating back-to-back with minicom via a pseudo-terminal; now to reimplement this in C, and wire it into the SIO simulator.

Getting the pty code controlled by the register access is going to be tricky.

Then it’s time to write a quick ‘local echo’ test program in Z80, and off we go…..

Been there – although in the interests of reducing RF noise into an Amateur Radio environment, no way would I ever use a perspex case. Also, no Windows 7, and the clock speeds are decidedly vintage 2004.

This is all quite reasonable. I’ve been building Intel-based computers like this since probably 1992 (didn’t keep that one) or 1996. For the most part, this has been one continual upgrade of one part or another, since 1996. I’m still using the same floppy drive I had back then – everything else has changed. Several times. It’s totally boring. Something essential has been lost over the years. Modern hobbyist computing has become somewhat jaded; innovation stifled under a smothering of bland conformity.

This was most definitely not the case back in the golden days of the microcomputer revolution in the UK.

(By the way, one reason for this post is so I can recollect all the stuff that follows. I’m 41 now, and it’s been hard trying to sequence all of it. If any friends remember any additional things, or can correct me, please let me know!)

Thanks to Jim Walwyn and Pete Wood, maths and computer studies teachers at Sir Thomas Boughey High School, I first used a computer at around age 11, way back in 1981.

This was a teletype connected via acoustic coupler to – I believe – the Manchester Computing Centre. I remember being fascinated by the game of Nim it played (I always lost; these days I understand it’s all about Sprague-Grundy theory….). The school also had an Apple ][, which sparked a life-long appreciation of Apple hardware; at one stage, I was collecting, renovating and selling second-hand Apple ][ and /// hardware - possibly a topic for another blog post; there's a motherlode in there.

The school soon took charge of several other computers of the day: ZX81s. My neighbour had a ZX80, so I had an idea of what to expect.

It may have been Christmas 1981 or 1982, that I pursuaded my parents that a ZX81 would be the perfect Christmas present. Of course, it would be useful for computer studies and maths at school, so can I have one, please, please please? (And it would also be jolly good for games.) My parents relented, and although expensive, they bought me one.

I learned ZX81 BASIC of course, and quickly went on to Z80 assembler, typing in listings (as hex dumps) from the computer magazines of the day, and writing my own games. I really did use it for educational purposes as well, honestly! I very much doubt I would be doing what I do today if I hadn’t had a home computer – and had one at that time. There was a critical mass of children getting computers and mastering them at that time. It would be very interesting to research the mastery level of children exposed to computers at that age, over the years. My cynical hypothesis is that as the interface to computers has made them less intimidating, and has hidden their programmability, mastery will have decreased. People are tool users these days, not tool makers. Or game players, not game writers, if you will.

Anyway, I digress.

After the ZX81, I upgraded to a Vic 20 (later with 8k ‘STACK’ expansion pack’), then ZX Spectrum, Atari 800XL and 1050 disk drive. I think it was while I was going through my Atari phase that I started work. I later upgraded to an Amiga 500, Amiga 2000. Around this time, I left home, had a brief dalliance with a 386, but it was fairly boring. Upgrade to an Amiga 1200. Around this time I was at university, towards the end of which, I bought my first ‘real’ PC, installed Beta-Tamu Linux, then Yggdrasil Linux on it, and the rest is history…. I’m going to leave the PC trail here, because as I stated above, they’re incredibly boring.

All the above computers came with BASIC available as their main programming language, although I usually preferred their native assembly languages, Z80, 6502, 80×86 and 680×0. If I wasn’t coding in assembler, I’d be writing in FORTH. I started using FORTH on the Spectrum with an implementation called White Lightning by Oasis Software. I never bought a Jupiter ACE. I should have. FORTH is a very simple, yet powerful language. Chuck Moore, its creator, is clearly a genius. There are many good articles on FORTH: Mike Vanier’s post is particularly good. FORTH is having something of a renaissance, or at least, it has inspired new languages like the Factor programming language.

Throughout this time, I was also interested in electronics and amateur radio – I still am, although it’s taken a distinct back seat since university.

So to actually get to the point, although I’ve assembled PCs, I’ve never actually built my own computer; although I’ve used FORTH implementations written by others on all the above computers, I’ve never finished an implemention of it myself (see Psion Forth for the furthest I got, the last time I tried).

For the past 30 years (crikey!) I’ve been a tool user, not a tool maker.

So, to remedy this, and revitalise those parts of my memory where the Z80 opcodes live, I plan to build a small single-board Z80-based microcomputer, and implement FORTH on it. Nothing earth-shattering, just the CPU, ROM, RAM, serial I/O to a terminal (will have to get an RS232-to-USB serial converter for the Mac!), and possibly a CompactFlash disk.

I am fortunate enough to have a box of Z80 and Z80-SIO chips; I have EPROMs, and a programmer (not sure if it still works), but no UV eraser. No RAM yet, but there’s always eBay. No MAX232 line drivers, but RS still sell them.

I have books: Rodnay Zaks’ Programming the Z80, and the masterpiece of FORTH wisdom that is R. G. Loeliger’s Threaded Interpretive Languages (a copy of which was always tempting me, on the bookshelves at Keele University; now I have my own copy).

Don’t need any fancy PCB design, layout, etching kit – it’s possible to build something with just matrix board: take a look at this: Homebrew Z80 Computer. It’s a work of greatness.

I’m doing it in three parts.

First, get a Z80 + hardware simulator running on the Mac, with a simple serial bootloader. I started that part in August 2006, but put it down – for four years.

Second, implement FORTH for it.

Third, build the hardware.

Watch this space…

Cool, uses Blowfish… by Bruce Schnieier… must be secure.

And it’s Open Source too! This’ll fox those pesky NSA spooks.

At least, this was an assumption – proved wrong when I came to transfer my encrypted data over to the iPod. This is a two stage process:

1. Extract the encrypted data from the synced CryptoPad database, and decrypt it on the PC. (How hard can this be?!!)
2. Import it into some new, as yet undiscovered program on the PC, that syncs with the iPod, allowing me to encrypt it.

The first part is going amazingly well – and has proven that even a good algorithm can be useless (or reduced in security) by an implementation flaw.

The problem is that the password you encrypt your memo with is encrypted with itself, and the ciphertext of this stored in the memo. When attempting to open a memo, you enter a password. The password-stored-as-ciphertext is then decrypted with the entered password, and if this yields the password you entered, then you have the correct password, and the ciphertext of the memo is decrypted with it.

The length of the password is also stored in the memo record. It’s trivial to mount a brute force attack against all possible n-character passwords, especially since you could restrict the keyspace to that allowed by the Grafitti alphabet.

I don’t actually have to do the brute force attack, as I know all the passwords I use, so can get the plaintext of all my memos easily.

But this serves as a warning to check the source code of any crypto system you’re intending to use. Unfortunately, crypto systems are often complex. Sometimes – as in this case – this complexity is deployed with a chink in the armour that can be detected even by those without strong crypto-fu.

The next stage of this project is to find an iPod app that allows import of memo data on the PC. I’m thinking of mSecure, but if there’s no source code, I’m not buying.

Call me cynical, but I doubt many companies actually follow through on this. Perhaps they record the enraged customers, and play them back at parties for a good laugh. I just don’t hear any improvements in customer support from many companies.

So, in keeping with my penchant for V’s “People should not fear their governments; governments should fear their people”, here’s perhaps one way that consumers can really make a difference in the companies they deal with.

“Your call may be recorded for quality and training purposes. … Hello, thank you for calling MegaCorp inc., you’re through to Sarah, how may I help you?”

“Hello Sarah, my name is Matt Gumbley, and before discussing my problem, I heard that you’ll be recording this for quality and training purposes – I’d like to state I will also be blogging a summary of my experiences with MegaCorp, inc. in this matter. I hope this won’t be a problem for a company with the reputation of MegaCorp, one who takes such a pro-active stance on excellent customer relations? You see, my blog has a wide readership, and I’m often asked for recommendations of suppliers and products; my readers want to know which companies to trust, and which would give them shoddy treatment(*). I won’t be mentioning your name personally, of course, but as a MegaCorp’s customer service department, you represent your company.”

I am curious to see whether this strategy would work.

* I should wish!

See the project home page for more details!

One of the tenets of Open Source development is “Release early, release often” (see The Cathedral and the Bazaar); another is “If you build it, they will come” (not sure where this came from – it’s in Field of Dreams…).

I work from slightly different versions of these: Release quality first, then release it often. If you build the code, then you have to build the community, then they come.

I could have created a public repository somewhere (SourceForge? BitBucket? GitHub?) and committed the very first lines of code, allowing developers to keep up with the early progress of the application, but I chose not to. Instead, I’ve spend a year and a half developing on my own, not making the code available. It still isn’t available, nor will it be, until it’s ready. Until it has reached some level of maturity and works as I intend; until it’s sufficiently complete, and until it has ‘quality’ (whatever that is; I’m trying to define it in another post). I’ve seen way too many Open Source projects falter after releasing too early: I want to ensure that the project is viable for myself before I expect others to judge it or use it.

An analogy might help me defend this position: I play guitar and keyboards (occasionally), but I don’t like others listening to me practice. I’d rather perfect my technique and give a performance when I’m good enough. Some players don’t mind others listening to their duff notes and bad timing; I’d rather not inflict that on others.

Likewise, with the code, I think it’s better to let me refactor it into the cleanest, highest-quality, most understandable, tested and testable design that does what it was intended to do first, before letting others loose on it.

Sure, I miss out on early feedback, and now, there’s a lot of code for people to work with and give feedback on (which could be grossly negative, however, I think I’ve got a good design, so that this won’t be the case). The codebase is currently standing at 47,464 lines of Java code in total, that’s 25,207 LOC main code, 22,257 LOC unit/integration test code – 580 tests in all.

However, I’d rather spend the time completing the architecture; I’ve got ideas to express that I need to get down in tests and code – then, I’ll take it all public. I definitely don’t want design by committee at this early stage. I might release a few “technology preview” versions though, if only to get the release mechanism in place – Paul Graham said it best:

In the desktop software business, doing a release is a huge trauma, in which the whole company sweats and strains to push out a single, giant piece of code. Obvious comparisons suggest themselves, both to the process and the resulting product.

— “Hackers and Painters, p62”

He’s talking about the difference between installed software and web-based, server-deployed software, but his point about releases of desktop software is valid: it takes a lot of infrastructure buildup, testing and polish.

One working definition of quality for this project is that the code should be free from internal contradictions (Christopher Alexander, quoted in Richard Gabriel, Patterns of Software, p35, – and having one developer define the architecture helps in this – there’s a single vision; I don’t have to work against several different aspects of the system at once, or mentor others who may be working in different areas (perhaps this is a trust / delegation issue?) thereby allowing me the time to get things to the state I want them in. Fred Brooks’ suggests, for the size of the team and the creative force:

Ten people, seven of them professionals, are at work on the problem, but the system is the product of one mind – or at most two, acting uno animo.

(The Mythical Man-Month, 20th Anniversary Ed, p35)

A view against my stance was recently posted by chromatic in his excellent “Modern Perl Books” blog:

I’ve written before that rapid, feedback-aware iteration is an effective way to understand a problem and find working solutions. Feedback from real users with real uses is invaluable (see also why Pugs and, later, Rakudo multiplied the velocity of Perl 6). You won’t understand the problem fully until you’ve had to balance multiple, sometimes contradictory, constraints. Just as no battle plan survives contact with the enemy, no software design survives contact with users unscathed. You may find that you’re painting yourself into a corner.
There’s a balance: you don’t want to cause users unnecessary pain by rapid, arbitrary change, but legacy code and features can hamstring a project.

– from Designing for the Unanticipated

So I could be “painting myself into a corner” by not releasing until it’s ready. However, perhaps even with the amount of code I have already, it’s still to early to release – it doesn’t do what it’s intended to do, yet.

Once I cut that first release, then I’ll release often. I prefer “Release quality, release often”. Then I’ll see whether they’ll come after something usable’s built.