Skip to main content

Clone multiple code repos in one pipeline

In addition to the pipeline's default codebase, you can use Git Clone, Run, and Plugin steps to clone additional code repos into the pipeline's workspace. For example, you can use this to:

  • Build multiple artifacts in the same pipeline. For example, suppose you use Packer and Ansible to build artifacts automatically, and you have separate repos for Packer, Ansible, and code. You can clone all three repos into the pipeline's workspace.
  • Pull code from separate code and build repos. For example, if your code files are in a repo managed by the Engineering team and your Dockerfiles are in a different repo managed by the Security team, you can clone both repos into the pipeline's workspace.
  • Clone codebases without using the built-in clone codebase function.
Large file storage

If you need to clone LFS-enabled repositories or run git lfs commands (such as git lfs clone), go to Git Large File Storage.

Configure the default codebase​

When you add a Build stage to a CI pipeline, you specify the Git account or repository where your code is stored. The codebase declared in a pipeline's first stage becomes the pipeline's default codebase, and this repo is cloned into the workspace automatically when the pipeline runs.

For more information about creating pipelines, configuring the Build stage, and specifying the default codebase, go to CI pipeline creation overview, Configure codebase, and CI Build stage settings.

Add a Git Clone or Run step​

You can use a Git Clone or Run step to clone an additional repo into a pipeline's workspace.

For example, assume the default codebase is a repo that contains app code files, and the Dockerfile necessary to build the app image is in a different repo. You can use a Git Clone or Run step to clone the second repo into the workspace. Then, you can use a Build and Push step to build and push an image using files from both repos.

Add a Git Clone step to clone a second repo into the pipeline's workspace.

              - step:
                  type: GitClone
                  name: clone second repo
                  identifier: clone_second_repo
                  spec:
                    connectorRef: account.git2
                    build:
                      type: branch
                      spec:
                        branch: main

The Git Clone step has the following settings. Depending on the stage's build infrastructure, some settings might be unavailable.

Name, Id, and Description​

Enter a Name summarizing the step's purpose. Harness automatically assigns an Id (Entity Identifier Reference) based on the Name. You can change the Id.

The Description is an optional text string.

Connector​

Select a connector for the source control provider hosting the code repo that you want the step to clone.

The following topics provide more information about creating code repo connectors:

Repository Name​

If the connector's URL Type is Repository, then Repository Name is automatically populated based on the repository defined in the connector's configuration.

If the connector's URL Type is Account, then you must specify the name of the code repo that you want to clone into the pipeline workspace.

Build Type, Branch Name, and Tag Name​

For Build Type, select Git Branch if you want the step to clone code from a specific branch within the repo, or select Git Tag if you want the step to clone code from a specific commit tag. Based on your selection, specify a Branch Name or Tag Name.

tip

You can use fixed values, runtime input, or variable expressions for the branch and tag names. For example, you can enter <+input> for the branch or tag name to supply a branch or tag name at runtime. You could also use expressions to match the pipeline's codebase branch or tag so that, for example, the pipeline and the Git Clone step both pull code from the same environment, such as production when a production build runs or development when a development build runs.

This setting applies only to the repo specified in this Git Clone step. It is separate from the codebase object for the pipeline's Build stage. If you want this Git Clone step's repo to use the same branch or commit as the primary codebase, specify either <+codebase.branch> or <+codebase.tag> for Branch Name or Tag Name. These expressions pull runtime input from the pipeline; for example, if the pipeline's primary codebase uses the development branch, then the Git Clone step clones the development branch from its repo. For more information, go to the CI codebase variables reference.

Clone Directory​

An optional target path in the pipeline workspace where you want to clone the repo.

You can't specify /harness/ as a target directory for a Git Clone step because this folder is reserved for the Build stage's codebase. You can specify Shared Paths in your CI Build stage settings to share data across steps in your Build stage.

Depth​

The number of commits to fetch when the step clones the repo.

The default depth varies by build and trigger type:

  • For manually-triggered branch and tag builds, the default depth is 50. This means each git clone operation fetches the 50 most recent commits.
  • For manually-triggered PR builds and all auto-triggered builds (such as webhook triggers), the default depth is 0. This means each git clone operation fetches all commits from the relevant branch.

For more information, go to the git clone documentation.

SSL Verify​

If True, which is the default value, the pipeline verifies your Git SSL certificates. The build fails if the certificate check fails. Set this to False only if you have a known issue with the certificate and you are willing to run your builds anyway.

If you want to use self-signed certificates in a Kubernetes Cluster build infrastructure, go to Configure a Kubernetes Build Farm to use Self-Signed Certificates

Run as User​

This setting is available for Kubernetes cluster build infrastructures only.

All Git clone steps, including the default clone codebase step and any additional Git Clone steps, use user 1000 by default for Kubernetes.

If necessary, you can specify, in Run as User, a user ID to use to run all processes in the pod if running in containers. For more information, go to Set the security context for a pod.

Specifying Run as User at the step level overrides Run as User in the build infrastructure settings, if you had also specified it there.

Set Container Resources​

Set maximum resource limits for the resources used by the container at runtime:

  • Limit Memory: The maximum memory that the container can use. You can express memory as a plain integer or as a fixed-point number using the suffixes G or M. You can also use the power-of-two equivalents Gi and Mi. The default is 500Mi.
  • Limit CPU: The maximum number of cores that the container can use. CPU limits are measured in CPU units. Fractional requests are allowed; for example, you can specify one hundred millicpu as 0.1 or 100m. The default is 400m. For more information, go to Resource units in Kubernetes.

Timeout​

Set the timeout limit for the step. Once the timeout limit is reached, the step fails and pipeline execution continues. To set skip conditions or failure handling for steps, go to:

SSH-keyscan timeout​

If your connector uses SSH authentication, you can add a PLUGIN_SSH_KEYSCAN_TIMEOUT stage variable to override the ssh-keyscan command's timeout limit (the default is 5s).

Stage variables are configured in stage settings, not step settings.

    - stage:
        ...
        variables:
          - name: PLUGIN_SSH_KEYSCAN_TIMEOUT
            type: String
            description: ""
            value: 90

Add this variable to all stages where you need to override the SSH-keyscan timeout limit.

tip

When cloning additional codebases that use the same credentials as your default codebase, you can use your default codebase connector's credentials in your Run step.

Build an artifact from both code repos​

Now that the files from both repos will be cloned into a common workspace, you can add a step to build an image using code from both repos, such as a Build and Push to Docker step.

Pay attention to settings like the Dockerfile setting that assume files are located at the codebase's root directory if not otherwise specified. This is because the pipeline's default codebase files are cloned in the root folder (/harness), while other codebase files are cloned into subfolders.

Depending on the default codebase, you might need to specify a non-root path for build files. You can also use commands, such as cp, in Run steps to move cloned files around the workspace before building the image.

YAML examples​

The following YAML example describes a pipeline that clones two code repos, one as the default codebase (cloneCodebase: true) and the second in the GitClone step.

pipeline:
  name: Clone two repos
  identifier: Clone_two_repos
  projectIdentifier: my_project
  orgIdentifier: default
  tags: {}
  properties:
    ci:
      codebase:
        connectorRef: account.git1
        repoName: my-code-repo
        build: <+input>
  stages:
    - stage:
        name: build from two repos
        identifier: build_from_two_repos
        description: ""
        type: CI
        spec:
          cloneCodebase: true
          platform:
            os: Linux
            arch: Amd64
          runtime:
            type: Cloud
            spec: {}
          execution:
            steps:
              - step:
                  type: GitClone
                  name: clone second repo
                  identifier: clone_second_repo
                  spec:
                    connectorRef: account.git2
                    build:
                      type: branch
                      spec:
                        branch: main
              - step:
                  type: BuildAndPushDockerRegistry
                  name: BuildAndPushDockerRegistry_1
                  identifier: BuildAndPushDockerRegistry_1
                  spec:
                    connectorRef: mydockerhubconnector
                    repo: username/imagerepo
                    tags:
                      - <+pipeline.executionId>
                    dockerfile: /path/to/dockerfile

Use a Plugin step​

As an alternative to the Git Clone and Run steps, you can clone a codebase by running the Git Drone plugin in a Plugin step.