Python application
You can build and test a Python application using a Linux platform on Harness Cloud or a self-hosted Kubernetes cluster build infrastructure.
This guide assumes you've created a Harness CI pipeline. For more information about creating pipelines, go to:
If you don't have a Harness account yet, you can create one for free at app.harness.io.
Install dependencies
Use Run steps to install dependencies in the build environment.
- Harness Cloud
- Self-hosted
- step:
type: Run
identifier: dependencies
name: Dependencies
spec:
shell: Sh
command: |-
python -m pip install --upgrade pip
pip install -r requirements.txt
envVariables:
PIP_CACHE_DIR: "/root/.cache"
- step:
type: Run
identifier: dependencies
name: Dependencies
spec:
connectorRef: account.harnessImage
image: python:latest
command: |-
python -m pip install --upgrade pip
pip install -r requirements.txt
Background steps can be used to run dependent services that are needed by steps in the same stage.
Cache dependencies
Add caching to your stage.
- Harness Cloud
- Self-hosted
Cache your Python module dependencies with Cache Intelligence.
Add caching to your stage.spec.
- stage:
spec:
caching:
enabled: true
key: cache-{{ checksum "requirements.txt" }}
paths:
- "/root/.cache"
sharedPaths:
- /root/.cache
With self-hosted build infrastructures, you can:
Python cache key and path requirements
Python pipelines typically reference requirements.txt in Save Cache and Restore Cache steps, for example:
spec:
key: cache-{{ checksum "requirements.txt" }}
Additionally, spec.sourcePaths must include the python cache (typically /root/.cache) in the Save Cache step, for example:
spec:
sourcePaths:
- "/root/.cache"
YAML example: Save and restore cache steps
Here's an example of a pipeline with Save Cache to S3 and Restore Cache from S3 steps.
steps:
- step:
type: RestoreCacheS3
name: Restore Cache From S3
identifier: Restore_Cache_From_S3
spec:
connectorRef: AWS_Connector
region: us-east-1
bucket: your-s3-bucket
key: cache-{{ checksum "requirements.txt" }}
archiveFormat: Tar
- step:
type: Run
envVariables:
PIP_CACHE_DIR: "/root/.cache"
...
- step:
type: SaveCacheS3
name: Save Cache to S3
identifier: Save_Cache_to_S3
spec:
connectorRef: AWS_Connector
region: us-east-1
bucket: your-s3-bucket
key: cache-{{ checksum "requirements.txt" }}
sourcePaths:
- "/root/.cache"
archiveFormat: Tar
Run tests
Add Run steps to run tests in Harness CI.
- Harness Cloud
- Self-hosted
- step:
type: Run
name: Test
identifier: test
spec:
shell: Sh
command: |-
pip install pytest
pytest tests.py --junit-xml=report.xml
envVariables:
PIP_CACHE_DIR: /root/.cache
- step:
type: Run
name: Test
identifier: test
spec:
connectorRef: account.harnessImage
image: python:latest
shell: Sh
command: |-
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install pytest
pytest tests.py --junit-xml=report.xml
Visualize test results
If you want to view test results in Harness, make sure your test commands produce reports in JUnit XML format and that your steps include the reports specification.
reports:
type: JUnit
spec:
paths:
- report.xml
Specify version
- Harness Cloud
- Self-hosted
Python is pre-installed on Harness Cloud runners. For details about all available tools and versions, go to Platforms and image specifications.
If your application requires a specific Python version, add a Run step to install it.
Use the setup-python action in a GitHub Actions step to install the required Python version.
You will need a personal access token, stored as a secret, with read-only access for GitHub authentication.
Install one Python version
- step:
type: Action
name: Install python
identifier: installpython
spec:
uses: actions/setup-python@v4
with:
python-version: 3.10.10
token: <+ secrets.getValue("github_token") >
Install multiple Python versions
- Add the matrix looping strategy configuration to your stage.
- stage:
strategy:
matrix:
pythonVersion:
- 3.11.2
- 3.10.10
- Reference the matrix variable in your steps.
- step:
type: Action
name: Install python
identifier: installpython
spec:
uses: actions/setup-python@v4
with:
python-version: <+ stage.matrix.pythonVersion >
token: <+ secrets.getValue("github_token") >
Specify the desired Python Docker image tag in your steps. There is no need for a separate install step when using Docker.
Use a specific Python version
- step:
type: Run
name: Python Version
identifier: pythonversion
spec:
connectorRef: account.harnessImage
image: python:3.10.10
shell: Sh
command: |-
python --version
Use multiple Python versions
- Add the matrix looping strategy configuration to your stage.
- stage:
strategy:
matrix:
pythonVersion:
- 3.11.2
- 3.10.10
- Reference the matrix variable in the
imagefield of your steps.
- step:
type: Run
name: Python Version
identifier: pythonversion
spec:
connectorRef: account.harnessImage
image: python:<+ stage.matrix.pythonVersion >
shell: Sh
command: |-
python --version
Full pipeline examples
The following full pipeline examples are based on the partial examples above.
- Harness Cloud
- Self-hosted
This pipeline uses Harness Cloud build infrastructure and Cache Intelligence.
If you copy this example, replace the placeholder values with appropriate values for your code repo connector and repository name. Depending on your project and organization, you may also need to replace projectIdentifier and orgIdentifier.
Pipeline with default Python version
pipeline:
name: Test a Python app
identifier: Test_a_Python_app
projectIdentifier: default
orgIdentifier: default
stages:
- stage:
name: Test
identifier: test
description: ""
type: CI
spec:
cloneCodebase: true
caching:
enabled: true
key: cache-{{ checksum "requirements.txt" }}
paths:
- "/root/.cache"
execution:
steps:
- step:
type: Run
identifier: dependencies
name: Dependencies
spec:
shell: Sh
command: |-
python -m pip install --upgrade pip
pip install -r requirements.txt
envVariables:
PIP_CACHE_DIR: "/root/.cache"
- step:
type: Run
name: Test
identifier: test
spec:
shell: Sh
command: |-
pip install pytest
pytest tests.py --junit-xml=report.xml
envVariables:
PIP_CACHE_DIR: /root/.cache
reports:
type: JUnit
spec:
paths:
- report.xml
platform:
os: Linux
arch: Amd64
runtime:
type: Cloud
spec: {}
sharedPaths:
- /root/.cache
properties:
ci:
codebase:
connectorRef: YOUR_CODE_REPO_CONNECTOR_ID
repoName: YOUR_REPO_NAME
build: <+input>
Pipeline with multiple Python versions
pipeline:
name: Test a Python app
identifier: Test_a_Python_app
projectIdentifier: default
orgIdentifier: default
stages:
- stage:
name: Test
identifier: test
description: ""
type: CI
strategy:
matrix:
pythonVersion:
- 3.11.2
- 3.10.10
spec:
cloneCodebase: true
caching:
enabled: true
key: cache-{{ checksum "requirements.txt" }}
paths:
- "/root/.cache"
execution:
steps:
- step:
type: Action
name: Install python
identifier: installpython
spec:
uses: actions/setup-python@v4
with:
python-version: <+ stage.matrix.pythonVersion >
token: <+ secrets.getValue("github_token") >
- step:
type: Run
identifier: dependencies
name: Dependencies
spec:
shell: Sh
command: |-
python -m pip install --upgrade pip
pip install -r requirements.txt
envVariables:
PIP_CACHE_DIR: "/root/.cache"
- step:
type: Run
name: Test
identifier: test
spec:
shell: Sh
command: |-
pip install pytest
pytest tests.py --junit-xml=report.xml
envVariables:
PIP_CACHE_DIR: /root/.cache
reports:
type: JUnit
spec:
paths:
- report.xml
platform:
os: Linux
arch: Amd64
runtime:
type: Cloud
spec: {}
sharedPaths:
- /root/.cache
properties:
ci:
codebase:
connectorRef: YOUR_CODE_REPO_CONNECTOR_ID
repoName: YOUR_REPO_NAME
build: <+input>
If you copy this example, replace the placeholder values with appropriate values for your code repo connector, kubernetes cluster connector, kubernetes namespace, and repository name. Depending on your project and organization, you may also need to replace projectIdentifier and orgIdentifier.
Pipeline with one specific Python version
Here is a single-stage pipeline, with steps that use Python 3.10.10.
pipeline:
identifier: Test a Python app
name: Test_a_Python_app
orgIdentifier: default
projectIdentifier: default
stages:
- stage:
identifier: default
name: default
spec:
cloneCodebase: true
execution:
steps:
- step:
type: Run
name: Test
identifier: test
spec:
connectorRef: account.harnessImage
image: python:3.10.10
shell: Sh
command: |-
pip install pytest
pytest tests.py --junit-xml=report.xml
reports:
type: JUnit
spec:
paths:
- "report.xml"
infrastructure:
type: KubernetesDirect
spec:
connectorRef: YOUR_KUBERNETES_CLUSTER_CONNECTOR_ID
namespace: YOUR_KUBERNETES_NAMESPACE
automountServiceAccountToken: true
nodeSelector: {}
os: Linux
type: CI
properties:
ci:
codebase:
connectorRef: YOUR_CODE_REPO_CONNECTOR_ID
repoName: YOUR_REPO_NAME
build: <+input>
Pipeline with multiple Python versions
Here is a single-stage pipeline, with a matrix looping strategy for Python versions 3.11.2 and 3.10.10.
pipeline:
identifier: Test a Python app
name: Test_a_Python_app
orgIdentifier: default
projectIdentifier: default
stages:
- stage:
strategy:
matrix:
pythonVersion:
- 3.11.2
- 3.10.10
identifier: default
name: default
spec:
cloneCodebase: true
execution:
steps:
- step:
type: Run
name: Test
identifier: test
spec:
connectorRef: account.harnessImage
image: python:<+ stage.matrix.pythonVersion >
shell: Sh
command: |-
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install pytest
pytest tests.py --junit-xml=report.xml
reports:
type: JUnit
spec:
paths:
- "report.xml"
infrastructure:
type: KubernetesDirect
spec:
connectorRef: YOUR_KUBERNETES_CLUSTER_CONNECTOR_ID
namespace: YOUR_KUBERNETES_NAMESPACE
automountServiceAccountToken: true
nodeSelector: {}
os: Linux
type: CI
properties:
ci:
codebase:
connectorRef: YOUR_CODE_REPO_CONNECTOR_ID
repoName: YOUR_REPO_NAME
build: <+input>
Next steps
Now that you have created a pipeline that builds and tests a Python app, you could:
- Create triggers to automatically run your pipeline.
- Add steps to build and upload artifacts.
- Add a step to build and push an image to a Docker registry.
- Explore other ways to optimize and enhance CI pipelines.