We have been using SFDX and CircleCI for quite some time now to automate our Salesforce deployments. Over time, our bash scripts to manage packages, verify Scratch Orgs, and scale test execution grew more and more in complexity. Now is the time to put everything into an orb and make it available to the community. In this post I would like to introduce the jsc/salesforce orb: It is the only SFDX orb on CircleCI that truly supports package-based development.
Why Yet Another CircleCI Orb For Salesforce?
The standard orbs (circleci/salesforce-sfdx and circleci/salesforce-apex) are very limited in scope and functionality. At the time of writing, they bring commands to install the CLI, handle Scratch Orgs, deploy source and execute tests. Unfortunately, they lack the functionality to build and deploy second-generation packages. As a result, they barely support the new Salesforce Developer Experience.
The jsc/salesforce orb tries to fill that gap. It is packed with commands and jobs to professionally deploy your second-generation packages:
- Authenticate with multiple persitent orgs (Production and Sandboxes) for multi-stage deployment processes.
- Build, install, promote and rollback packages versions (release candidates and beta packages) for package-based development and deployment.
- Scale test execution horizontally and vertically for more consistent results on larger orgs.
I tried to abstract away some of the complexity of the internal CLI commands. As a result, the orb commands are slightly opinionated on how to create and install package versions and how to execute apex tests. The orb works best if it is used in approval-based workflows (as opposed to tag-based workflows).
How The Orb Uses SFDX
For technical documentation of the orb’s commands and jobs, reference the official listing on the CircleCI registry or the Github repository. This focuses on the fundamental concepts and trade-offs.
Install and Authenticate the SFDX CLI
The commands to install and authenticate the CLI are straightforward and almost identical to the official ones. However, I do not support the node-based installation (
npm install -g sfdx-cli) and always install the latest CLI (the official salesforce orb uses an obsolete installer that defaults to
Managing 2nd Generation Packages
The most effort was put into commands to support second-generation packages. This encompasses commands to
install new package versions. Since packaging makes rollbacks incredibly easy, the
install command can also be used to roll back to previously installed versions. One of the biggest challenges is the asynchronous nature of Package Version creation. To solve this, all commands support two modes:
- Export package versions to environment variables (and access them to install a specific version).
- Dynamically retrieve the „latest package version“ of a certain package (to install or promote it).
Commands that „produce“ package versions (package-build and package-get-installed) support a
packageVersionExport parameter. You can specify the name of an environment variable and the command will export the package version Id there. A command that „uses“ a package version (package-install and package-promote) can then optionally read from this environment variable. You can pass in the environment variable’s name with the
packageVersion parameter. However, the default behavior is to dynamically query the latest package version.
I tried to keep the default values compatible. For example, the „build“ command exports to
$SUBSCRIBER_PACKAGE_VERSION_ID and „install“ reads from the same environment variable, if it is not configured to query the latest version.
Scaling Test Execution
The other area the orb tries to improve is test execution. Salesforce lacks functionality to reliably scale test execution, especially in larger Orgs with several thousand Apex tests. The
apex:test:run command has a maximum wait time of 60 minutes and CircleCI times out after 10 minutes by default. Simply increasing the timeout doesn’t help. Eventually, you will hit the limit again. As a sustainable solution, the orb uses test suites to execute all tests in batches.
run-test-suites-command supports two modes:
- Run a specific set of test suites (you can group test suites that are executed together using commas and separate them in groups to execute in sequence by space).
- Run all test suites on the target org and execute them one-by-one. This loops through all test suites and executes each suite as a single command.
These concepts enable horizontal scaling or sandbox sharding. In other words: You can run your tests on multiple sandboxes in parallel. Dynamically querying existing test suites is a little bit slower than a full test run but more reliable and scalable. The command can potentially run for hours without timing out. If something goes wrong, the command reliably returns partial results.
The orb does not directly support scratch org commands. That includes commands to create and delete Scratch Orgs, resolve and install upstream dependencies, import data import, and assign permissions. Instead, I advise using your own script to set up development environments. Such a script handles the installation of upstream dependencies, pushing the package source, and assigning permissions.
Why? We want to automate the provisioning of development environments as far as possible. And we want our CI pipeline to verify, that a repository is in a „working state“ at any given point in time. So we write a script once and use it for local setup and execute it in our CI environment.
The orb ships with a
scratch_org_test-job that executes a setup script, runs all apex tests on the Scratch Org, and uploads the test results. Running the job is as easy as that:
workflows: package_build: jobs: - jsc-sfdx/scratch_org_test: devhubUsername: << pipeline.parameters.devhubUsername >> devhubInstanceUrl: << pipeline.parameters.devhubInstanceUrl >> setupScript: scripts/setup/macOS.sh jwtKey: DEVHUB_JWT_KEY consumerKey: DEVHUB_CONSUMER_KEY context: - salesforce additionalSteps: - node/install: install-npm: true
In our case, this expands to a complete scratch org verification that looks like this
I believe, this orb is the only orb you will ever need to automate your SFDX pipelines with CircleCI. We use the orb in all our projects, so expect many more jobs and commands to be implemented.