Today I want to dive deeper into package-based deployment processes with SFDX. Interestingly enough, there are very little resources out there that explain how to set up a continuous integration (CI) pipelines to deploy a second-generation (2nd-gen) unlocked package (with SFDX). To close that gap, I created the jsc/salesforce orb for CircleCI, that strives to make this straight-forward and easy.

All of the following examples are written in CircleCI, but the concepts are vendor agnostic. You should be able to implement them in any major CI provider.

Anatomy Of A Package-Based CI Pipeline

2nd-gen packaging is still new, and it lacks vital features. Unfortunately, we do not have an equivalent of the --checkonly parameter that is available on the Metadata API. We have to install the package on the target org, execute regression tests, and mplement rollback functionality by ourselves. However, thanks to acceptable dependency management and test execution during package build, we can perform much more validation and regression testing bevor putting our code on a sandbox.

From a bird’s perspective, every pipeline should consist of three stages: Scratch Org, Sandbox(es) and Production. You want to include the package build and the package install in the same pipeline.

Scratch Org Stage

This stage is used for project or repository validation. This is where you ensure that anyone can check out the latest commit, spin up a scratch org and start developing. On larger orgs with a decent functional package architecture, this step should be considerably faster than replicating a whole org.

We do not build new package versions yet. Instead, we enforce formatting and linting, run LWC tests, spin up a scratch org, install all the upstream dependencies, push the source and execute apex tests on it. Additionally, we can execute API E2E tests (since we do not need care about cleaning up afterwards). This is useful, because we have full control over existing records such as configuration records or other data that our tests depend on.

The orb supports an out-of-the-box scratch org job that does all that.

Sandbox Stage

This is the first time we build a package. It is also the first time our code sees the rest of our org with all its unpackaged metadata and it’s other packages. Depending on the size of the org (and the quality of your apex tests), regression testing can take minutes to hours.

The sandbox stage is pretty straight-forward and not very complex. We build a beta-version of the package (for performance reasons), install it, and run all regression tests on that org. The jsc/salesforce orb provides a convenience job that does all that: beta_package_deploy. You can easily chain multiple of these jobs, if you plan to deploy to multiple sandboxes.

In our case, we always deploy to our Dev sandbox without manual approval. After a manual approval, we deploy to our QA sandbox. The jobs list in your CircleCI workflow section can look like this:

workflows:
  package_build:
    jobs:
      [...]
      - jsc-sfdx/beta_package_deploy:
          name: "dev_sandbox_deploy"
          devhubUsername: << pipeline.parameters.devhubUsername >>
          targetOrgUsername: << pipeline.parameters.devSandboxUsername >>
          devhubInstanceUrl: << pipeline.parameters.devhubInstanceUrl >>
          devhubJwtKey: SFDX_JWT_KEY
          targetOrgJwtKey: SFDX_JWT_KEY
          devhubConsumerKey: SFDX_CONSUMER_KEY
          targetOrgConsumerKey: SFDX_CONSUMER_KEY_DEV
          package: PACKAGE_ID
          requires:
            - jsc-sfdx/scratch_org_test
          context:
            - salesforce

In our case, this expands to the following pipeline job:

Dev integration pipeline job

Production Stage

The production stage itself consists of a package build of the release candidate, an installation of this build to all sandboxes, the promotion and the final installation on your production system. For performance reasons, it makes sense to run the validated package build asynchronously, in a separate job. The install_production job then simply queries the latest validated package build and proceeds with it.

The jsc/salesforce orb provides a convenience job that builds a release candidate: jobs-build_release_candidate_version. Simply add it to your workflow section and run it in parallel together with the scratch org setup. Even though we only need it after all sandboxes stages, it may give valuable feedback early on (implementing the fail-fast strategy).

In the workflow section of your config, add the job configuration, and let the orb do the rest.

workflows:
  package_build:
    jobs:
      [...]
      - jsc-sfdx/build_release_candidate_version:
          devhubUsername: << pipeline.parameters.devhubUsername >>
          devhubInstanceUrl: << pipeline.parameters.devhubInstanceUrl >>
          jwtKey: SFDX_JWT_KEY
          consumerKey: SFDX_CONSUMER_KEY
          package: PACKAGE_ID
          context:
            - salesforce

The jsc/salesforce orb does not yet support a convenient job to install the latest release candidate to production. Therefore, we need to configure the job in our config explicitly.

jobs:
  install_production:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - jsc-sfdx/install
      - jsc-sfdx/auth:
          username: << pipeline.parameters.devhubUsername >>
          instanceUrl: << pipeline.parameters.devhubInstanceUrl >>
          jwtKey: SFDX_JWT_KEY
          consumerKey: SFDX_CONSUMER_KEY
          setDefault: true
      - jsc-sfdx/auth:
          instanceUrl: https://test.salesforce.com
          username: << pipeline.parameters.devSandboxUsername >>
          jwtKey: SFDX_JWT_KEY
          consumerKey: SFDX_CONSUMER_KEY_DEV
      - jsc-sfdx/auth:
          instanceUrl: https://test.salesforce.com
          username: << pipeline.parameters.qaSandboxUsername >>
          jwtKey: SFDX_JWT_KEY
          consumerKey: SFDX_CONSUMER_KEY_STAGING
      - jsc-sfdx/package-get-installed:
          targetOrg: << pipeline.parameters.devhubUsername >>
          devhubUsername: << pipeline.parameters.devhubUsername >>
      - jsc-sfdx/package-get-latest-build:
          releaseCandidateOnly: true
          packageVersionExport: LATEST_RELEASE_CANDIDATE_BUILD
          devhubUsername: << pipeline.parameters.devhubUsername >>
      - jsc-sfdx/package-install:
          installLatestBuild: false
          packageVersion: LATEST_RELEASE_CANDIDATE_BUILD
          targetOrg: << pipeline.parameters.devSandboxUsername >>
      - jsc-sfdx/package-install:
          installLatestBuild: false
          packageVersion: LATEST_RELEASE_CANDIDATE_BUILD
          targetOrg: << pipeline.parameters.qaSandboxUsername >>
      - jsc-sfdx/package-promote:
          promoteLatestBuild: false
          packageVersionId: LATEST_RELEASE_CANDIDATE_BUILD
          devhubUsername: << pipeline.parameters.devhubUsername >>
      - jsc-sfdx/package-install:
          installLatestBuild: false
          packageVersion: LATEST_RELEASE_CANDIDATE_BUILD
          targetOrg: << pipeline.parameters.devhubUsername >>
      - jsc-sfdx/run-test-suites:
          targetOrg: << pipeline.parameters.devhubUsername >>
          outputDir: test-results/apex
      - jsc-sfdx/package-rollback:
          packageVersion: INSTALLED_PACKAGE_VERSION_ID
          targetOrg: << pipeline.parameters.devhubUsername >>
          when: on_fail
      - store_test_results:
          path: test-results

workflows:
  package_build:
    jobs:
     [...]
      - approve_production:
          type: approval
          requires:
            - qa_sandbox_deploy
            - jsc-sfdx/build_release_candidate_version
          filters:
            branches:
              only:
                - /^version/.*/
      - install_production:
          context:
            - salesforce
          requires:
            - approve_production
          filters:
            branches:
              only:
                - /^version/.*/

Putting It Together

And that’s it. Easy as that you can leverage SFDX and deploy your package with CI. In my team it’s not uncommon that juniors deploy to production on their first or second day. This is what our current deployment pipeline looks like, when it is executed on a master commit:

A full package ci pipeline with scratch org, dev sandbox, qa sandbox and production install.

I highly recommend approval-based workflows. The reason being the dreaded „This test is already in the execution queue“-error. If your pipeline automatically deploys to a Sandbox, other pipelines to the same Sandbox could fail.

Such a pipeline ensures reproducibility, high reliability and high stability. We put so much trust in it, that roughly 90% of our production releases are completely blind. If the 1300+ unit tests are green, we do not feel compelled to verify manually.