image showing the logs of a running toolbelt server

I’m fortunate to have both the time and energy to work on technical side projects in my spare time. I feel joy when I automate something tedious or make something new possible for my single customer (me!). Life is good, but I’m hooked and want more side project goodness.

With all that’s been going on with Heroku, I took stock of my personal technical side project debt. I had Heroku Dynos, Zapier Zaps, GitHub Actions workflows and random bits of AWS and GCP up to my neck. I was drowning in side projects built with the wrong priorities: speed and free tier utilisation. I’d built config loading, database initialization/migration, web servers, cron jobs & object storage wrappers many times over and in various ways. It was all a bit of a mess.

I’ve worked on a infrastructure consulting team in the past, so it didn’t take long to realise what I needed was a platform, a personal side project platform!

Why build a personal side project ‘platform’?

I chose to look for a better solution the following reasons:

What do we mean by platform?

I’m using the term loosely. We’re not talking about a GitOps system for deploying Kubernetes clusters for side projects. My side project platform is smaller scale and higher trust and your typical enterprise infra platform.

However, there are similarities - platforms are about tenants sharing functionality. In an enterprise infrastructure platform this might be monitoring, logging, networking, ingress, backups and so on. For my platform, I offer tenants (tools) secret config management, a database schema, publicly accessible web endpoints and cron jobs.

My shared functionality is higher level and less isolated, but it’s what I needed to get the job done.

What’s a tool? What’s a toolbelt?

A toolbelt is a collection of likely unrelated tools grouped together. The toolbelt wraps up the shared functionality and exposes it to tools. The idea is that a small side-project comprises a single tool - though it may use services exposed by others.

More precisely, a tool is a go module with a publicly importable tool type which implements the Tool interface below. A Toolbelt can have instances of Tools registered on it - sharing its functionality. Once tools have been registered, the toolbelt web server and job runner can be started.

type Tool interface {
	// Name returns the name of the tool for display purposes
	Name() string

	// FeatureSet returns the features that the tool uses and indicates
	// to the tool belt how the tool should be configured
	FeatureSet() FeatureSet

	// SetConfig sets the configuration for the tool
	SetConfig(config map[string]interface{}) error

	// DatabaseMigrate runs the database migrations for the tool
	DatabaseMigrations() (*embed.FS, string, error)
	// DatabaseSet sets the database connection for the tool
	DatabaseSet(db *sql.DB)

	// HTTPPath returns the base path to use for the subrouter
	HTTPPath() string
	// HTTPAttach configures the tool's subrouter
	HTTPAttach(router *mux.Router) error

	// Jobs returns a list of jobs that the tool defines
	// and needs to have run
	Jobs() ([]Job, error)
}

A toolbelt is intended to be co-located a collection of side projects (tools) running together.

Okay, but what do your tools do?

Edit: please see the toolbelt README for an updated list of tools - there are some new ones!

I currently have one ‘platform tool’ which is used by other tools. My idea is that some tools will be used by other tools as services to further accelerate my side project efforts with the platform. webhook-rss exposes endpoints to accept new entries for RSS feeds, new feeds can be created on the fly to make it flexible.

Based on webhook-rss I have built the following tools to ‘RSSify’ things. These tools are:

I have also migrated another few side projects to my toolbelt format. Previously these tools used a mess of implementations, now they are all standardised and sharing in the platform’s functionality.

Which features of the toolbelt are used by different tools?

The table below is intended to outline how different tools are using the platform’s functionality. All tools need config loading, but other functionality is only used by some tools.

Tool NameConfigDatabaseHTTPCronDepends On
webhook-rss
tool-twitter-rsswebhook-rss
tool-activities-rsswebhook-rss
tool-airtable-contactswebhook-rss
tool-inoreader-github-actions-trigger
tool-json-status
tool-subpub

How does one run a toolbelt?

The toolbelt can be used however you need. Sometimes it makes sense to have a single instance running jobs and running a webserver. Other times it might be more suitable to run cron jobs on another instance. My jobs are all lightweight and mostly IO bound, so I run them on the same instance as the web server - to save some cash and make deployment extra simple.

It looks a little like this:

// create a new toolbelt to register tools to
tb := tool.NewBelt()
// set the config containing config values for all tools,
// these are handed down as tools are registered
tb.SetConfig(cfg)
// set the toolbelt's database, this can be shared by
// tools which need it
tb.SetDatabase(db)

// add tool to the toolbelt
err = tb.AddTool(&example.Example{})
if err != nil {
    log.Fatalf("failed to add tool: %v", err)
}
// ... add more tools
err = tb.AddTool(&example.Example{})
if err != nil {
    log.Fatalf("failed to add tool: %v", err)
}

// start a job worker
go tb.RunJobs(ctx)

// start a web server
tb.RunServer(ctx, host, port)

I have a private project in my GitHub account where I define how my toolbelt looks. When I commit to that repo, an image is built and deployed to Northflank. I’ve replaced Heroku with Northflank now and am very happy with the platform, the service and the pricing.

In the toolbelt repo, there’s an example here you might find useful if you’d like to run your own toolbelt.

What’s next?

I have a few ideas for future tools and functionalities.

I’d like to be able to expose a Go Cloud Bucket to tools. This would allow tools to store and retrieve blob data in some standard way with a prefixed path.

I think that it’d also be nice to offer a means for tools to spawn async jobs in an ad-hoc way. This would allow me to run more CPU intense jobs on a separate instance when required. It’d plan to use the Northflank API to enable this functionality.

Finally, I think it’d also be nice for tools to be able to register for particular hostnames. This would mean that tools would be able to serve content on the “/” path making it useful for user-facing sites rather than arbitrary webhooks and RSS feeds like I have at the moment.

Other than features, I’m interested to present this project internally to get feedback on the idea. I think it’s a good model for me, and I’m interested to learn what might stop others from using it for their projects.

Further Reading

You can find the toolbelt repo here. There are some example tools in that repo, however you might want to check out the link in the post above for more fully-featured examples.

Feel free to tweet or email me (see homepage) if you have any ideas, questions or feedback.