Parallel Processing & Distributed Computing


Just had a short break away from the constant pace of previous work – 16 months’ worth of near-daily updates – in which I caught up on the stack of magazines, articles, books I’d been meaning to get through. So away from tech for a while hence no blog post last month.

The book stack never goes down – see Michael Simmons’ article at https://medium.com/accelerated-intelligence/the-5-hour-rule-if-youre-not-spending-5-hours-per-week-learning-you-re-being-irresponsible-791c3f18f5e6 – if I could take a yearly two-week reading vacation, I would… but back in the real world..

Now I’m ready to start on Parachute 0.0.2, the rework of the node server protocol to be iserver compatible. This should mean that the emulator could work with other emulators’ iservers, and vice-versa. However, the link emulation mechanism would need additional variants, to use the mechanism used by other emulators – e.g. http://lcm-proj.github.io/ as used by Gavin Crate’s emulator (see https://sites.google.com/site/transputeremulator/Home/multiprocessor-jserver-support).

During this work, I’ll update the Hello World assembly program, and start upgrading the C++ code to C++11 as needed.

Advertisements

16 months after picking up this old project, I’m happy to release the first cross-platform version of my T800ish Transputer Emulator!

See the Parachute 0.0.1 release notice for more details, download links etc.

As the previous blog post here described, there have been several frustrating aspects to the development- and these have only continued since then. However, I now have a good base, with mostly fully automated build/test and release systems- so subsequent releases should be easier.

One aspect of the build has been to use Maven & Jenkins for everything- despite several components of Parachute not being pure Java. Between these two, and a moderate collection of plugins, I have a multi-platform C++ build, with deployment to Maven Central.

I have a long-standing dislike of language ecosystems building their own component build/dependency/deployment systems. They re-invent Maven – sometimes poorly, usually because of a mistaken idea that it is Java-only. I’d much rather build on the existing base of Maven than see my language experiments as in some way “special”, requiring me to unnecessarily re-invent it. Maven is not perfect, but I think it’s a good fit for Parachute. I’ll still be providing build tools as command line tools and libraries, for easy use in non-Maven systems.

In the next phase of Parachute I’ll be converting the protocol between the emulator and host I/O server to be compatible with the Inmos iServer, then getting eForth working.

But first, a rest. Happy Summer Solstice!

TL;DR: Frustration, but the end is in sight.

Parachute is composed of several separate projects, with independent versions, held in separate repositories:

  • the Transputer Emulator itself, written in C++, built using Maven/CMake/Make, which requires building and packaging on macOS, CentOS 7, Ubuntu 1604 and 1804, Raspbian Stretch, and Windows 10.
  • the Transputer Macro assembler, written in Scala, built using Maven, which requires building and packaging on macOS, Linux (one cross-platform build for all the above Linux variants), and Windows 10.
  • and eventually there will be the eForth build for Transputer, other languages, documentation, etc.

Getting all this to build has been quite the journey!

I use Maven as an overall build tool, since it gives me sane version management, build capability for all the languages I use via plugins, packaging, signing, deployment to a central repository (I’m serving all build artefacts via Maven Central).

Each project’s build runs across a set of Jenkins instances, with the master on macOS, and nodes on virtual machines, and a physical Raspberry Pi.

Each project deploys a single artefact per target OS, into Maven Central’s staging repository. So there are six build jobs, one on each node, that can sign and deploy on request.

The effect of this is that a single commit can trigger six build jobs for the C++ code, and three for the JVM-based code (since all Linux systems package the same scripts). Deployment is manually chosen at convenient points, with manual closing of the staging repository in Sonatype’s OSSRH service.

The manual deployment choices may be removed once all this is running smoothly. Since I cannot produce all platform-specific artefacts from a single Maven build, I cannot use the Maven Release Plugin.

Once the emulator and assembler are deployed for all their variants, there is a final build job that composes the Parachute distribution archives, signs them and deploys them to Maven Central via Sonatype OSSRH.

There have been several ‘gotchas’ along the way..

… the GPG signing plugin does not like being run on Jenkins nodes. It gets the config from the master (notably, the GPG home, from which it builds its paths to the various key files). So that had to be parameterised per-node.

… getting the latest build environments for C++ on each of the nodes. I’m not using a single version of a single compiler on everything. A variety of clangs (from 3.5.0 to 8.0.0) and Microsoft Visual C++ Build Tools.

… Windows. It’s just a world of pain. Everything has to be different.

So this long ‘phase one’ is almost at an end, and I hope to ship the first build very soon.

It would be ‘fun’ to see if I can replicate all the above with a cloud-based build system instead of Jenkins + VMs. However, Windows, macOS and Raspberry Pi will be problematic. Travis CI does not have CentOS or Raspberry Pi hosts; Circle CI does not have Windows, CentOS or Raspberry Pi hosts (Windows is on their roadmap).

Since Feb/Mar 2018, I’ve been working on a new phase of one of my old projects: Parachute, a modern toolchain for programming the Transputer, and a Transputer Emulator – cross-platform for Mac OSX, Windows 10 and Linux.

The Transputer architecture is interesting since it was one of the first microprocessors to support multitasking in silicon without needing an operating system to handle task switching. Its high level language, occam, was designed to facilitate safe concurrent programming. Conventional languages do not easily represent highly concurrent programs as their design assumes sequential execution. Java has a memory model and some facilities (monitors, locks, etc.) to make parallel programming possible, but is not inherently concurrent, and reasoning about concurrent code in Java is hard. occam was the first language to be designed to explicitly support concurrent (in addition to sequential) execution, automatically providing communication and synchronisation between concurrent processes. If you’ve used go, you’ll find occam familiar: it’s based on the same foundation.

My first goal is to get a version of eForth running on my emulator (as I’ve long wanted to understand Forth’s internals). The eForth that exists is a) untested by its author and b) only buildable on MASM 6, which is hard to obtain (legally). I’m trying to make this project as open and cross-platform as possible, so first I had to write a MASM-like macro assembler for the Transputer instruction set This is mostly done now, written in Scala, and just requires a little packaging work to run it on Mac OS X, Linux and Windows.

I’ve written up the history of this project at Parachute History, so won’t repeat myself here..

I’m not yet ready to release this, since it doesn’t build on Windows or Linux yet, and there are a few major elements missing. Getting it running on Windows will require a bit of porting; Linux should be a cinch.

Once I have a cross-platform build of the emulator, I intend to rewrite my host interface to be compatible with the standard iServer (what I have now is a homebrew experimental ‘getting started’ server).

There are quite a few instructions missing from my emulator – mostly the floating point subset, which will be a major undertaking.

The emulator handles all the instructions needed by eForth. eForth itself will need its I/O code modifying to work with an iServer.

Once eForth is running, I have plans for higher-level languages targetting the Transputer…

… but what I have now is:

… to be continued!

Ah, the optimism of the 1st January!

As I reflected on 2018, it became apparent that ‘starting, not finishing’ is a big problem, chez M0CUV. My muse bestows plenty of interesting ideas, but some of them are a bit ambitious. I start, then things grind to a halt. This, coupled with chronic procrastination means a lot of churn, and a feeling of dissatisfaction, angst, and despair at not being able to find the time to do all this – or to prioritise better. A look back through the log shows a big gap of radio silence from June to October, a page of digital mode contacts, and not a single CW QSO throughout the whole year. On the software side, I hung up my own P2P stack after a baffling test failure wore me down. I do want to return to that.. However, I spent most of my hobby development time working on the Parachute project, which despite being really tricky, is going well. I never thought writing an assembler would be this hard. The devil is in the details.

So, after giving myself a good slap for a lacklustre radio year, 2019’s going to be goal-driven. Hopefully these goals are SMART (Specific/Stretching, Measurable/Meaningful, Agreed-upon/Achievable, Realistic/Rewarding and Time-based/Trackable). There are quite a few, and only the tech/radio-related ones are blogged about here. I’ve been using the Getting Things Done method for a while, and it stresses the importance of defining the Next Action on your activities..

So…

  • 1 blog post/month, at least. Progress updates on projects, etc.
  • 1 CW QSO/month, or 12 in the whole year. This’ll probably be really tough.
  • 1 QSLable QSO/month, or 12 in the whole year, any mode/band. FT8 makes this much easier!
  • Try to contact each month’s special callsign for the Bulgarian Saints award from Bulgarian Club Blagovestnik. I’ve already bagged January’s, LZ1354PM via SSB on 1st Jan 🙂
  • Take the next step on the magnetic loop project: build the frame, house the capacitor. I bought some wood and a food container for this yesterday.
  • Box up the 30m QCX transceiver properly – then use it.
  • Keep up with magazines as they arrive rather than building a pile of them.
  • Fix the current bizzare bug in the Transputer assembler – then ship the first release on Windows, macos and Ubuntu/Raspbian
  • Convert the Parachute Node server to use the IServer protocol – then write the IO code for eForth.
  • Build a web application with elm. I’m thinking of a web-front-end to WSJT-X, to allow me to operate remotely.

Let’s see how I get on…!