I've explored this problem several times. It's not just Fastmail, many services that offer Address Book or Contacts services don't provide them as 'client' services to other CardDAV servers. It seems they all want to be your authoritative contacts service.
I've managed to achieve inter-service contacts syncing by syncing a CardDAV 'client' (such as a mobile phone) to multiple CardDAV servers, but the best solution I've found is to use Memotoo (
memotoo.com) as an intermediary.
Memotoo is an old service that is not being kept up to date and the interface is not pretty or modern, but the service is incredibly comprehensive (it will sync CardDAV and CalDAV with Activesync/Exchange, for example) and it works in the background, syncing contacts (and calendars, bookmarks, files, etc); there is no need to use the website once it's set up.
My only hesitation in recommending it is that it seems to be neglected and I worry that it might close down or fail one day. Unfortunately I haven't found an alternative that is as comprehensive and reliable.