Today I want to dive deeper into package-based deployment processes with SFDX. Interestingly enough, there are very little resources out there that explain how to set up a continuous integration (CI) pipelines to deploy a second-generation (2nd-gen) unlocked package (with SFDX). To close that gap, I created the jsc/salesforce orb for CircleCI, that strives to make this straight-forward and easy.
All of the following examples are written in CircleCI, but the concepts are vendor agnostic. You should be able to implement them in any major CI provider.
Anatomy Of A Package-Based CI Pipeline
2nd-gen packaging is still new, and it lacks vital features. Unfortunately, we do not have an equivalent of the
--checkonly parameter that is available on the Metadata API. We have to install the package on the target org, execute regression tests, and mplement rollback functionality by ourselves. However, thanks to acceptable dependency management and test execution during package build, we can perform much more validation and regression testing bevor putting our code on a sandbox.
From a bird’s perspective, every pipeline should consist of three stages: Scratch Org, Sandbox(es) and Production. You want to include the package build and the package install in the same pipeline.
Scratch Org Stage
This stage is used for project or repository validation. This is where you ensure that anyone can check out the latest commit, spin up a scratch org and start developing. On larger orgs with a decent functional package architecture, this step should be considerably faster than replicating a whole org.
We do not build new package versions yet. Instead, we enforce formatting and linting, run LWC tests, spin up a scratch org, install all the upstream dependencies, push the source and execute apex tests on it. Additionally, we can execute API E2E tests (since we do not need care about cleaning up afterwards). This is useful, because we have full control over existing records such as configuration records or other data that our tests depend on.
The orb supports an out-of-the-box scratch org job that does all that.
This is the first time we build a package. It is also the first time our code sees the rest of our org with all its unpackaged metadata and it’s other packages. Depending on the size of the org (and the quality of your apex tests), regression testing can take minutes to hours.
The sandbox stage is pretty straight-forward and not very complex. We build a beta-version of the package (for performance reasons), install it, and run all regression tests on that org. The jsc/salesforce orb provides a convenience job that does all that: beta_package_deploy. You can easily chain multiple of these jobs, if you plan to deploy to multiple sandboxes.
In our case, we always deploy to our Dev sandbox without manual approval. After a manual approval, we deploy to our QA sandbox. The jobs list in your CircleCI workflow section can look like this:
workflows: package_build: jobs: [...] - jsc-sfdx/beta_package_deploy: name: "dev_sandbox_deploy" devhubUsername: << pipeline.parameters.devhubUsername >> targetOrgUsername: << pipeline.parameters.devSandboxUsername >> devhubInstanceUrl: << pipeline.parameters.devhubInstanceUrl >> devhubJwtKey: SFDX_JWT_KEY targetOrgJwtKey: SFDX_JWT_KEY devhubConsumerKey: SFDX_CONSUMER_KEY targetOrgConsumerKey: SFDX_CONSUMER_KEY_DEV package: PACKAGE_ID requires: - jsc-sfdx/scratch_org_test context: - salesforce
In our case, this expands to the following pipeline job:
The production stage itself consists of a package build of the release candidate, an installation of this build to all sandboxes, the promotion and the final installation on your production system. For performance reasons, it makes sense to run the validated package build asynchronously, in a separate job. The install_production job then simply queries the latest validated package build and proceeds with it.
The jsc/salesforce orb provides a convenience job that builds a release candidate: jobs-build_release_candidate_version. Simply add it to your workflow section and run it in parallel together with the scratch org setup. Even though we only need it after all sandboxes stages, it may give valuable feedback early on (implementing the fail-fast strategy).
In the workflow section of your config, add the job configuration, and let the orb do the rest.
workflows: package_build: jobs: [...] - jsc-sfdx/build_release_candidate_version: devhubUsername: << pipeline.parameters.devhubUsername >> devhubInstanceUrl: << pipeline.parameters.devhubInstanceUrl >> jwtKey: SFDX_JWT_KEY consumerKey: SFDX_CONSUMER_KEY package: PACKAGE_ID context: - salesforce
The jsc/salesforce orb does not yet support a convenient job to install the latest release candidate to production. Therefore, we need to configure the job in our config explicitly.
jobs: install_production: docker: - image: cimg/base:stable steps: - checkout - jsc-sfdx/install - jsc-sfdx/auth: username: << pipeline.parameters.devhubUsername >> instanceUrl: << pipeline.parameters.devhubInstanceUrl >> jwtKey: SFDX_JWT_KEY consumerKey: SFDX_CONSUMER_KEY setDefault: true - jsc-sfdx/auth: instanceUrl: https://test.salesforce.com username: << pipeline.parameters.devSandboxUsername >> jwtKey: SFDX_JWT_KEY consumerKey: SFDX_CONSUMER_KEY_DEV - jsc-sfdx/auth: instanceUrl: https://test.salesforce.com username: << pipeline.parameters.qaSandboxUsername >> jwtKey: SFDX_JWT_KEY consumerKey: SFDX_CONSUMER_KEY_STAGING - jsc-sfdx/package-get-installed: targetOrg: << pipeline.parameters.devhubUsername >> devhubUsername: << pipeline.parameters.devhubUsername >> - jsc-sfdx/package-get-latest-build: releaseCandidateOnly: true packageVersionExport: LATEST_RELEASE_CANDIDATE_BUILD devhubUsername: << pipeline.parameters.devhubUsername >> - jsc-sfdx/package-install: installLatestBuild: false packageVersion: LATEST_RELEASE_CANDIDATE_BUILD targetOrg: << pipeline.parameters.devSandboxUsername >> - jsc-sfdx/package-install: installLatestBuild: false packageVersion: LATEST_RELEASE_CANDIDATE_BUILD targetOrg: << pipeline.parameters.qaSandboxUsername >> - jsc-sfdx/package-promote: promoteLatestBuild: false packageVersionId: LATEST_RELEASE_CANDIDATE_BUILD devhubUsername: << pipeline.parameters.devhubUsername >> - jsc-sfdx/package-install: installLatestBuild: false packageVersion: LATEST_RELEASE_CANDIDATE_BUILD targetOrg: << pipeline.parameters.devhubUsername >> - jsc-sfdx/run-test-suites: targetOrg: << pipeline.parameters.devhubUsername >> outputDir: test-results/apex - jsc-sfdx/package-rollback: packageVersion: INSTALLED_PACKAGE_VERSION_ID targetOrg: << pipeline.parameters.devhubUsername >> when: on_fail - store_test_results: path: test-results workflows: package_build: jobs: [...] - approve_production: type: approval requires: - qa_sandbox_deploy - jsc-sfdx/build_release_candidate_version filters: branches: only: - /^version/.*/ - install_production: context: - salesforce requires: - approve_production filters: branches: only: - /^version/.*/
Putting It Together
And that’s it. Easy as that you can leverage SFDX and deploy your package with CI. In my team it’s not uncommon that juniors deploy to production on their first or second day. This is what our current deployment pipeline looks like, when it is executed on a master commit:
I highly recommend approval-based workflows. The reason being the dreaded „This test is already in the execution queue“-error. If your pipeline automatically deploys to a Sandbox, other pipelines to the same Sandbox could fail.
Such a pipeline ensures reproducibility, high reliability and high stability. We put so much trust in it, that roughly 90% of our production releases are completely blind. If the 1300+ unit tests are green, we do not feel compelled to verify manually.