Latest Answer (Dec 2, 2023, GitLab@^15.12 & >13)
In response to valuable feedback, the original approach of using artifacts for caching in GitLab has been revisited and improved upon. With updated documentation from GitLab regarding the use of cache, including support for multiple cache keys per job (up to 4), a more efficient method for managing node_modules across pipelines has emerged.
The strategy behind this implementation is grounded in a thorough understanding of both GitLab's functionality and the behavior of npm. Here are the key principles:
NPM advises the utilization of npm ci over npm install in CI/CD environments. It requires the presence of package-lock.json to ensure package versions remain consistent across builds (npm i lacks this determinism).
npm ci clears the node_modules directory before reinstalling packages listed in package-lock.json. Therefore, it's best to run npm ci once in GitLab and share the resulting node_modules with subsequent jobs.
NPM maintains its own cache at ~/.npm/ for offline builds and speed optimization. The cache location can be customized using the --cache <dir> option.
GitLab cannot cache directories outside the repository, such as the default cache directory ~/.npm.
GitLab's global cache configuration is applied to every job by default, requiring explicit overrides when necessary. YAML anchors can be used to copy and modify the global cache config.
To execute additional npx or npm run <script> commands without reinstallation, caching the node_modules/ folders across the pipeline is recommended.
GitLab encourages utilizing the cache feature for handling dependencies and restricting the use of artifacts to dynamically generated build results. This revised approach aligns better with GitLab's guidelines, acknowledging artifact size restrictions and storage usage quotas.
The use of needs or dependencies affects automatic download or deletion of artifacts between jobs.
GitLab's cache monitors file hash values, updating only when package-lock.json changes. Using package.json would compromise build determinism.
If dealing with multiple packages in a mono-repo, the cache entry limit during the install job might pose challenges. Combining cache definitions and careful key management can address this limitation effectively.
GitLab's cache list order influences cache matching between jobs. Attention must be given to the order of cache entries to ensure proper functioning across stages.
Optimized Solution
- Execute a single install job as the
.pre stage, leveraging cached downloaded packages across the entire repository.
- Cache all
node_modules/ folders for subsequent jobs within the same pipeline execution, preventing unnecessary uploads and streamlining pipeline runtime.
- Pass the
build/ directory via artifacts only when required by other jobs.
# .gitlab-ci.yml
stages:
- build
- test
- deploy
# Global cache settings for all jobs
cache:
- &global_cache_node_mods
key:
files:
- package-lock.json
paths:
- node_modules/
policy: pull
install:
image: ...
stage: .pre
cache:
- <<: *global_cache_node_mods
policy: pull-push
- key: ${CI_JOB_NAME}
paths:
- .npm/
policy: pull-push
script:
- npm ci --cache .npm --prefer-offline
build:
stage: build
script:
- npm run build
artifacts:
paths:
- dist/
test:
stage: test
needs:
- job: build
artifacts: true
script:
- npm test
deploy:
stage: deploy
when: on_success
cache: {}
needs:
- job: build
artifacts: true
- job: test
artifacts: false
script:
- npm publish
Refer to GitLab's documentation for detailed recommendations on caching npm.
[DEPRECATED] Original Approach (Oct 27, 2021, GitLab<13.12)
Prior attempts at caching with npm & GitLab lacked completeness in achieving effective caching strategies.
To realize comprehensive caching with npm & GitLab, consider the following points:
Refer to point #1 above for context.
When using npm ci, clearing the node_modules directory ensures a clean slate for subsequent installations based on package-lock.json. Caching node_modules between build jobs in GitLab becomes redundant under this scenario.
Explore insights outlined in points #3-#4.
Global cache downloads in multiple stages may not align with desired outcomes.
For running additional npx commands without re-installation, transferring the node_modules/ folder as an artifact to other jobs proves beneficial.
[DEPRECATED] Initial Solution
- Initiate a single install job as the
.pre stage, harnessing cached downloaded packages (tar.gz's) across the entire repository.
- Share
node_modules and the build directory with other jobs only when necessary.
stages:
- build
- test
- deploy
install:
image: ...
stage: .pre
cache:
key: NPM_DOWNLOAD_CACHE
paths:
- .npm/
before_script:
- cp .env.gitlab-testing .env
- composer install --no-progress --no-interaction
- php artisan key:generate
script:
- npm ci --cache .npm --prefer-offline
artifacts:
paths:
- vendor/
- bootstrap/
- node_modules/
- .env
- public/mix-manifest.json
build:
stage: build
needs:
- job: install
artifacts: true
script:
- npm run build
artifacts:
paths:
- dist/
test:
stage: test
needs:
- job: install
artifacts: true
- job: build
artifacts: true
script:
- npm test
deploy:
stage: deploy
needs:
- job: build
artifacts: true
- job: test
artifacts: false
script:
- npm publish