This page displays a summary of all CMS Jenkins projects , their sub projects , upstream projects
and downstream projects. To see the deatil and confgiuration of a project in Jenkins , click on project name.
Kill a running job (by default it is 'ib-run-pr-tests' and jobs started by it). The idea is that if pull request was updated, all the test should be restarted. This job will kill all the running jobs for that PR (with matching parameters). It will ignore the given job ID - the ID of upstream job that started this job.
This job basically triggers the `abort-jenkins-jobs` job for all the jobs related to PR tests.
Aborts and kills a release building process.
This is triggered by github webhook for each cmssw/cmsdist branch merge event. This is just a place holder job to trigger one sub-job per destionation branch for which the forward porting should be done. If this fails then this means that one of its sub-job failed. There is no need to re-try this job. Retry the failed sub-jobs only.
This job forward ports git changes from one branch to another. If it fails due to network of github related issues then just re-try it. Most of the times this job fails due to merge conflicts e.g. cmsdist master branch has different version of root as compare of rootmaster rootnext branch. If merge conflicts are just the version differences, then re-build the job with "STRATEGY=ours" as parameter, but if there are complex conflicts then better to resolve the issue by hand and directly push changes to github
Manually executed job for backporting pull requests given the repo name, the branch for which the PR should be created and the PR to be backported.
This jobs starts an Integration Build (IB). Base on state of CMSDIST/CMSSW git repositories, it builds either a full release or patch release.
Build Full IB if:
This job builds container images and uploads them to dockerhub.
Containers are built when either their base image is updated or if changes were made to dockerfile.
Failures may arise from the changes in the base image. In this case, the best approach is to log into the machine, reproduce the issue
and try to adapt the dokerfile accordingly (e.g., changing dependencies, ordering, etc).
If the job fails due to a glitch in the infrastructure, retry should work. If the container image is changed, we need to retrigger this job from check-docker-container job (so that the correct checksum is passed to build-docker-container job).
Since the built container images are left on the machine, this job can also fail due to disk full.
There is a separate jenkins job (clean-build-docker-machine) cleaning the images left on the device that runs once a week. In any case, one can always log into the machine and run the docker cleanup as follows:
$ docker image prune
$ docker system prune --volumes
If the ./singularity folder is big enough ($ du -hs /build/cmsbld/jenkins/workspace/.singularity), it can be also removed.
This job builds a CMSSW_X_Y_Z_FWLITE release based on an existing CMSSW_X_Y_Z release. CMSSW_X_Y_Z should be already build/uploaded before building its FWLITE version.
This job is responsible for building FWLITE release for each Integration Build (IB). Results of this build can be seen via CMSSDT IB page (old page).
This job actually builds a release.
It is triggered by cms-bot after the Release build Issue is approved by the release manager.
This job checks for HOST certificate in cmssw/cms:rhelX images, and fail if host certificate is going to expire in less than CERT_EXPIRY_DAYS days.
In such case, we need to check if there is already an updated osg-ca-certs package available and rebuild the container.
One can rerun the job with LOCAL_CERTIFICATE=/cvmfs/grid.cern.ch/etc/grid-security/certificates and CHECK_LOCAL_CERTIFICATE=true to see it new certificates are already available.
If the job still fails, this means that new root certificates are still not available.
Note that the job fails some days before the actual expiration (CERT_EXPIRY_DAYS).
This job cleans up disk space on cmsdev machines.
This job connects to the slave and checks if docker service is useable.
Job can fail for three reasons:
1. user is not in docker group
2. docker service is not running on the machine
3. docker daemon reports that there isn't space in the machine ("no space left on device").
If machine is in our control, we should take the appropriate action (e.g. login into the machine and manually run the ssh command), but if machine is in CERN IT control (e.g OpenLab machines), we should open a SNOW ticket.
In case of no space on the device, one can manually run the job clean-check-docker indicating the name of the machine as an input parameter.
This job checks for changes in the parent image. If there are changes, it triggers the `build-docker-container` job so that our based image is updated and uploaded to the registry.
Periodicaly runs https://github.com/cms-sw/cms-bot/blob/master/fix-backport-labels.py to check if the PR on master, which backport has been requested, is merged. If the original PR has been merged, it changes all opened backport PRs of it from backport(blue) to backport-ok(green)
The project checks for a running job and shows the status of process running on a remote machine. This can be useful to see if any process is handing and kill it. To run this job - Copy the url of a running job e.g. - https://cmssdt.cern.ch/jenkins/job/ib-run-relvals/243936/ - https://cmssdt.cern.ch/jenkins/job/ib-run-relvals/243936/console - Start the job with JOB_URL=url-of-a-running-job - Check the outout and see if any process is running for too long and should be killed? - If a process should be killed then return the job with KILL_PID
This job check for unused cmsdist files so that one can do the cleanup.
Connects to the slave and look for runaway process. This job fails when it finds such processes.
In case of failure, we need to check the runaway process list (e.g. see the job log) and make sure that these processes are really runaway (e.g. cmsRun jobs running with parent PID 1).
If all looks good, then one can restart the job with KILL_HANGING_PROCESSES=true.
Note: A runaway process is a process that enters an infinite loop and spawns new processes.
This job prevents running out of disk in the machines used to build container images (e.g., the ones used for running the build-docker-container job).
This job cleans the indicated machine when docker daemon reports errors of the type "no space left on device" in job check-docker.
It is necessay to indicate the machine name as an input parameter.
This job deletes the release build areas after three days.
This job cleanup the cms-sw/cms-sw.github.io repository history. Specially the data and _data directories history.
This cleans up old cms.weekN.PR_* repositories from cmsrep.cern.ch server.
This job cleans up disk space on cmssdt.
This job cleans up old cms.weekN.PR_* repositories from cmsrep.cern.ch server.
This job removes outdated tags from docker repository.
Connect to selected slave and cleans the auto-builds directories
Cleanup IB tags from the official CMSSW repository.
This is cms bot job which is triggered by github webhooks ( https://cmssdt.cern.ch/SDT/cgi-bin/github_webhook ) for every valid comment added to github PRs.
Will kill a scheduled/running job acording to comments.
Q/A
This job builds a cmssw external package ( e.g. crab , SCRAMV1 ) using the master cmsdist/pkgtools branches. It then trigger the cvmfs installation job to install the newly build packages.
This project search for tags.yaml files in cms-sw/cms-docker repostory and create new image tags if needed
This job collects metadata for pull requests since reading github directly when necessary might become time consuming or rejected by github on reached limit hits/hour grounds.
On every CMSSW PR, this job dumps list of files that are going to be modified by open PRs to files_changed_by_prs.json .
This jobs starts an Integration Build (IB). Base on state of CMSDIST/CMSSW git repositories, it builds either a full release or patch release.
Build Full IB if:
Copy unused Os/archs RPMs to EOS
This is cms bot job which is triggered by github webhooks ( https://cmssdt.cern.ch/SDT/cgi-bin/github_webhook ) for every valid comment added to github PRs. Also it runsevery 30mins to make sure any webhooks were not missed.
Generate doxygen documentation for CMSSW project
This is cms bot job which is triggered by github webhooks ( https://cmssdt.cern.ch/SDT/cgi-bin/github_webhook ) for every valid comment added to github PRs. This is to process bot commands for cmsdist comp branches
Comopare results of material budget of two releases using Validation/Geometry/test/TrackerMaterialBudgetComparison.C macro
Download the files of the baseline IB and compare them with the results of the ones of a pull request. Rate of failure of this job is very low and when failed then just re-try it and either delete the failed job or update the build information that it has been re-tried.
This job runs once a day to clean up the /eos/cms/store/user/cmsbot/CRAB_PrivateMC directory e.g. delete every thing older than X days.
In case of failure , just re-run it.
This job checks and updates cuda-compatible-runtime. This is used by CMS pilot jobs to find available gpu on the worker node.
Requestsed by marco.mascheroni@cern.ch and andrea.bocci@cern.ch
This job checks and updates the cmssw git mirror daily and monthly based on
https://github.com/bockjoo/cvmfs-cms-install-scripts/blob/master/cvmfs_update_cmssw_git_mirror_v3.sh
This job checks and updates generators model configurations
This job checks and updates generators model configurations
This job checks and updates gridpacks based on
https://github.com/bockjoo/cvmfs-cms-install-scripts/blob/master/cron_rsync_generator_package_from_eos_individual.sh
This job checks and updates gridpacks based on
https://github.com/bockjoo/cvmfs-cms-install-scripts/blob/master/cron_rsync_generator_package_from_eos_individual.sh
This job checks and updates lhapdf based on
https://github.com/bockjoo/cvmfs-cms-install-scripts/blob/master/cron_download_lhapdf.sh
Job to deploy the contents of https://gitlab.cern.ch/cms-analysis-metadata to /cvmfs/cms-griddata.cern.ch/cat
This job checks and updates premixPUlist
https://its.cern.ch/jira/browse/CMSVOC-616
This job reports status of upstream projects, siteconf and gridpacks.
This job checks and updates siteconf based on
https://github.com/bockjoo/cvmfs-cms-install-scripts/blob/master/cvmfs_check_and_update_siteconf.sh
This job checks/compares EOS directory with cvmfs nd trigger the sync if there are changes
This changes the /cvmfs/cms.cern.ch/common/crab symlink to point to either crab-pre, crab-prod or crab-dev.
Note any cmssw release which installs a different crab version will revert the symlink back to crab-prod
Note that this only changes the /cvmfs/cms.cern.ch/common/crab symlink. For crab python api, one still needs to source /cvmfs/cms.cern.ch/common/crab-setup.sh prod|pre|dev
This deploys a gitlab repository under /cvmfs/cms.cern.ch/rsync/repository/name
- POG scale factors on CVMFS: Requested by XPOG silvio.donato@cern.ch
This job checks and updates cuda-compatible-runtime. This is used by CMS pilot jobs to find available gpu on the worker node.
Requestsed by marco.mascheroni@cern.ch and andrea.bocci@cern.ch
Job to install (test) releases (and IBs) once the upload is completed. The job should be triggered by upstream job upload-release. It's purpose is to only trigger cms-install-package with the same argument it would get from upload-release job, but without using upload-release instance (it's just a trigger job for now)
Trigger the installation of new production architecture for CMS or reinstall cms-common new revision.
- Production architecture if obtained via config.map
This job is triggered by cmsrep-webhook which should pass the cms-common revision and architecture.
Job to install python whenever there is a new version pushed on cmsrep.cern.ch
Install path is /cvmfs/${CVMFS_REPOSITORY} + COMP install dir (boostraped for comp)
This job installs packages using cmspkg tool To be used for - CRAB3, PHEDEX, spacemon-client, python, releases(cmssw), cms-common Triggered by webhook
This job installs packages using cmspkg tool To be used for - CRAB3, PHEDEX, spacemon-client, python, releases(cmssw), cms-common Triggered by webhook
This job is to reseed a already installed architecture. This is needed when we have updated the packages to be seeded from the system installation.
This deploys a gitlab repository under /cvmfs/cms.cern.ch/rsync/repository/name
This job syncs an EOS directory on to CVMFS
This job updates CA/CRL under /cvmfs/cms.cern.ch/grid/etc/grid-security as in the following script:
https://github.com/bockjoo/cvmfs-cms-install-scripts/blob/master/update_ca_crl.sh
Copy baseline results from cmssdt for an IB
This runs CVMFS GC (once a week)
To install PR externals
Copy baseline results from cmssdt for an IB and deploy them on cvmfs
This runs CVMFS GC on the Gateway, make sure that top level path "/" has a lease.
This jobs install an IB on /cvmfs/cms-ib.cern.ch.
Copies root files for a given dataset localy on eos to be used from local cache (eos) if they diasappear from the site where they resided.
Updates cms-bot clone on Jenkins master
This job deploys doxygen scripts to web server.
It is later used to serve CMSSW documentation
Could be automated by github hooks.
This job migrates exisiting images in dockerhub to the github registry.
avalenzu: reporting back on existing issues
avalenzu: testing docker image comparison
avalenzu: testing watching concrete pkgs
Synchronizes Docker Hub Organization's setup (repositories/teams/members/permissions) with docker configuration yaml file.
Index CMSSW documentation using DXR.
This job keeps last 4 weeks of data in Elasticsearch open, and it closes older indexes (archive it). We do not care about older data. By doing it we make Elasticsearch faster.
Insert record into ES linking Issue/PR and error in IB
This job keeps last 4 weeks of data in Elasticsearch open, and it closes older indexes (archive it). We do not care about older data. By doing it we make Elasticsearch faster.
avalenzu(testing): dummy job for builid cmssw images.
Periodicaly runs https://github.com/cms-sw/cms-bot/blob/master/fix-backport-labels.py to check if the PR on master, which backport has been requested, is merged. If the original PR has been merged, it changes all opened backport PRs of it from backport(blue) to backport-ok(green)
Mirror one git repository.
Q/A:
Triggered when new IB results are available and send a Mattermost notification if there are new failures
Create GIT Reference for cms-sw/cmssw repository in /cvmfs/cms-ci.cern.ch/git/cms-sw. This is automatically triggered by "git push" to cmssw repo.
Adds and configures web hooks in github repositories. The web hooks are then used to send some events to a specified url and then a jenkins job is triggered based on the information passed from the web hook , as jenkins parameters.
Adds and configures web hooks in github repositories. The web hooks are then used to send some events to a specified url and then a jenkins job is triggered based on the information passed from the web hook , as jenkins parameters.
PyGitHub API at: https://pygithub.readthedocs.io/en/latest/github.html
Updates cms-bot clone on cmssdt. This job is also triggered via github web hook. Please do not add/remove any parameters of this job otherwise github web hooks will not be able to triiger the job
Refactors the CMSSW repo.
Refactors one module of CMSSW and makes an PR.
This is cms bot job which is triggered by github webhooks ( https://cmssdt.cern.ch/SDT/cgi-bin/github_webhook ) for every valid comment added to github PRs. Also it runsevery 30mins to make sure any webhooks were not missed.
Check if there are jobs queued for condor nodes, and if so kill placeholder job.
shutdowns condor job. TODO: needs to check if manuall deletion is working.
This job is triggered by HTCondor pilot jobs. When we submit a request to get a HTCondor node then first pilot runs and triggers this job at various stages with different "STATUS"
- STATUS=online
- When pilot script first runs on the HTCondor node then it sends an "online" event along with CONDOR_JOB_ID. In this case, this job tried to add the new HTCondor node as jenkins agent.
In case a job with "online" status fails:
1. First check if the new HTCondor node has been successfully added as a Jenkins agent (https://cmssdt.cern.ch/jenkins/computer/). It can happen that the agent is created,
but the connection failed. In this case, just re-starting
the agent should work.
2. If the node has not been successfully added, then please first run https://cmssdt.cern.ch/jenkins/job/grid-check-jobs/ job and see if HTCondor job with
CONDOR_JOB_ID is still running. For example look for messages like:
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE HOLD TOTAL JOB_IDS
cmsbuild ID: CONDOR_JOB_ID DATE TIME _ 1 _ _ 1 CONDOR_JOB_ID"
If it is still in "RUN" state, then just re-try this job. If it is in "IDLE" state then do not do anything and if it is in "DONE" state then better
to run https://cmssdt.cern.ch/jenkins/view/Grid/job/grid-shutdown-node/ to kill it.
- STATUS=offline
- This event is sent when HTCondor pilot has run 90% of its max allocated time. When this event is received then this jobs marks the grid${CONDOR_JOB_ID} jenkins agent as offline
so that no new job can be run on this agent. When grid${CONDOR_JOB_ID} is offline, and it is not running any job then this agent is automatically
deleted by https://cmssdt.cern.ch/jenkins/view/Grid/job/grid-check-nodes/. Depending on the agent "LABELS" (e.g. auto-recreate) , this job can request a new HTCondor node to replace it.
- STATUS=shutdown
- This event is sent when a pilot has reached its max life and going to shutdown. In this case this job tried to delete the agent from the jenkins.
It is build periodically (H/30 * * * *). Runs on cmssdt. Projects to build: update-github-pages
Job to run das client and cache the results in github to be used by IBs.
This jobs look for on demand baseline generation requests (made by ib-run-pr-tests job) and start the baseline generation job ( if not already started).
This check for /cvmfs/REPO/SITECONF contents and triggers ib-install-siteconf (for each repo) if https://github.com/cms-sw/siteconf has updates. Thsi jobs runs every hour
Inject the FastTimerService files to CMSMonit
This jobs install an IB on /cvmfs/cms-ib.cern.ch. As this job runs on the CVMFS Stratum 0, so only one job can run at a time. Sometimes ( specially IBs for non-86-64 archs) are stuck and do nothing. In that case better to kill the job and re-try it. This gives chance to other IBs to get installed and at the end the re-tried job will re-run. There is no automatic re-try setup for this job. It rarely fails but in case it fails then just re-try the failed job and either delete the failed job instance or update the "Build information" and mentioned that it has been re-tired. This allows others to not re-try it.
Update the contents of/cvmfs/REPO/SITECONF from https://github.com/cms-sw/siteconf
This job is triggered by `ib-run-crab` or `ib-run-pr-crab` once a test analysis is submitted to CRAB. It monitors the status of the CRAB job by using curl calls until the job finishes.
Finally, it generates a status file that is visible in the IB page (https://cmssdt.cern.ch/SDT/jenkins-artifacts/ib-run-crab).
Runs addons test on IB. Results are shown as "Other test" in the IB page for the IB.
This job generates the IB baseline. It is two phase job
- First it runs on any suitable node and look for workflows for which the baseline should be generated. It then re-run itself with the list of workflows and build an upload the actual baseline.
- In case of failure, just retry this job (Unless there are network/github issues)
This job gets the cfipython files for a cmssw release and push the changes to cms-sw/cmssw-cfipython repo.
Check for missing headers and parse the log for all errors (clang modules)
Runs edm class version checks for IB
This job integrates CRAB into the CI system.
Note: CRAB is the CMS Computing tool to submit CMSSW users' analysis jobs to distributed computing resources (https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideCrab).
Runs Flawfinder test on IB.
Appends job's time out information into jenkins.log file.
Runs IgProf on IB.
Results are available on IB page.
run-ib-igprof is executed with `pp` flag for `performance profiling`.
Q/a:
Runs iwyu logs parsing for each IB
Runs iwyu logs parsing for each IB
Runs Lizard, a Cyclomatic Complexity Analyzer, on IB. Results are available on IB page.
Runs Validation/Geometry/test/runP_Tracker_cfg.py and MaterialBudget.C for an IB
Build mutiple pull requests. Run cmssw addON (HLT tests)
PR Unit Test for CRAB updates.
Job to run das client and cache the results in github to be used by IBs. Ignore any failed job if a newer job has succeeded.
Build mutiple pull requests. test cmssw externals packages static checks e.g. hardcoded build paths etc.
Build mutiple a pull requests. Same puprose as `ib-any-integration`, just different script is called.
Build mutiple a pull requests. Same puprose as `ib-any-integration`, just different script is called.
This is CMSSw CI Job which runs the build and unit tests part of CI. It first kills any other job whcih is running for same PullRequests/Architecture. and then starts the build process. Once the build process if done then it trigger other jobs to start the deployment of the build artifects on CVMFS and start the tests. at the end it run the unit tests. Unit tests are run here as it needs the CMSSW/tmp directory which is not deployed on CVMFS. Q/A: In case this job fails then in most cases just re-try it. In case it is failed due to network/github issues then it is better to wait for the service to get healthy.
Build mutiple pull requests. Run cmssw GPU unit tests
Same puprose as `ib-any-integration`, just different script is called.
This job waits until the build has been deployed into cvmfs (cms-ci.cern.ch repository).
If the job fails, it could be because of a failure in the previous job (cvmfs-install-pr) that produces and uploads the build. Therefore, the current job will time out.
The way to proceed in case of failure is to look for the PR number present in the build name and trace back what has happened. It could be that the PR test were stuck and the build was never produced or uploaded.
Runs FastTimerService and Igprof on the RECO and PAT steps for high pileup workflow.
Runs NSYS on the RECO and PAT steps for high pileup workflow.
Runs Jemalloc memory profiling on the RECO and PAT steps for high pileup workflow.
Runs FastTimerService and Igprof on the RECO and PAT steps for high pileup workflow.
Runs iwyu logs parsing for each IB
Runs Quality Assurance (QA) test on IB. Results are available at
IB page's Q/A section.
If this job fails, logs can be found under
https://cmssdt.cern.ch/buildlogs//os_arch_compiler/www/week_day/...
Sometimes it can hang during scp copy / ssh. This failure can be found by looking for a gap of several hours in the log file during scp / ssh.
If the job runs for more than several hours, the procedure is to login to build node, check which job is hanging and take the appropiate action, if necessary.
The job runs release validations, as validations are separated on pieces (1of6 2of6 etc).
It can fail due to connection issues (e.g. Remote call on machine-XYZ failed).
Sometimes it can also hang during scp copy / ssh (which is usually the last job).
This failure can be found by looking for a gap of several hours in the log file where the Jenkins job is trying to finish the last test job.
Runs a few tests only for the IB, for comparison with those ran by the pull request.
This job runs valgrind tool for selected IBs when build IB job is complete.
This is a warpper job which runs daily at 11h and 23h (except no IB at 23h Sat & 11h Sun but a special IB at Sun 00h) to triger 'ib-tag-and-schdule' sub-project.
This was created to avoid the issue with Jenkins Dynamic Parameters.
Wrapper project to schedule aditonal test such as Flawfinder, Lizard, IgProf, etc.
Build a pull request
This job tags and schedules all the IBs.
It reads the cms-bot/config.map to find all available CMSSW release cycles and tag them in github.
Once tags is done then it triggers 'build-any-ib' sub-job to actually build IBs for all possible architectures (mentioned in cms-bot/config.map).
For Developement IB, it also triggers the generation of LXR indexes
For Sunday's 00h IBs, it also resets the CMSREP weekly repositories by building a dummy package and uploading it to cms.weekN. It also
then triggers 'ib-install-cvmfs' sub-job to get the new cms.weekN deployed on the /cvmfs/cms-ib.cern.ch
This job re-tries itself on failure. So there is no need to re-try it manually (unless all re-tries failed).
avalenzu: To be removed soon. HLT test can run as part of https://cmssdt.cern.ch/jenkins/job/ib-test-lto-openstack/
This job waits for the recently installed IB on CVMFS to be visible on CERN Stratum 1. Once the IB is available on Stratum 1 then it triggers various validation jobs. If this job fails or timeout then just re-run it. Failure rate is very low but on Wednesday CERN IT runs CVMFS Stratum 1 garbage collection which can add extra few hours delay and may cause this job to timeout.
This is a helper job to copy a Jenkins Project from this Jenkins master to another Jenkins master.
This job is to start those nodes which only has LABEL set to auto-label.
This happens when jenkins is restart and labels are read from nodes/config.xml.
We need to make a initial connection to the slave to get its labels assigned.
This job takes the backup of Jenkins master configuration (which includes projects, jenkins configuration, slaves, secrets etc.) Backups jenkins scripts on https://github.com/cms-sw/jenkins-backup.git
Deletes nodes from jenkins and then deletes it on openstack.
Disable all nodes that started with the provided wildcard string, example parameter cmsbuild* would enable all cmsbuild machines found
Disable all jenkins slaves based on the slave wildcard filter.
This downloads a jenkins RPM from https://pkg.jenkins.io/redhat-stable/ and make it available via http://cmsrep.cern.ch/cmssw/download/jenkins for faster access to CMs internal jenkns servers
This job runs a python script to push useful info from jenkins build logs to elasticsearch
Enable all Jenkins projects based on the wildcard filter.
Enables all nodes that started with the provided wildcard string, example parameter cmsbuild* would enable all cmsbuild machines found
This project is run once a new Jenkins instance is setup or upgrated to a new version. This runs some sub-projects to check if basic functionality of jenkins is working.
This is Jenkins installation tests project. It tests if various Jenkins Command-line interface service are working.
Jenkins installaton test job to test the triggering of a sub-project via Command-line-interface.
This is Jenkins installation test project. This tests the triggering of a sub-project.
Kill a specific running job.
This job runs a groovy script to dump jenkins projects info , which is further read by python to display it in html form. The current page you are viewing is generated and updated by this job.
This job runs a groovy script to dump jenkinsprojects info , which is further read by python to display it in markdown form. It is later imported to CMSSDT wiki.
This jobs looks for jenkins slaves which went offline due to job failures
Test new bootstrap for cmssw
Test new bootstrap for cmssw
Jenkins installation test job 1 to check for the parameters passed from a parent job.
Jenkins tests Project to test jenkins functionality.
Jenkins tests Project to test jenkins functionality.
avalenzu: debug
This job monitors parser retries.
avalenzu: Doing some cleanup (06-12-2024) - Possibly to be deleted.
This job retries a build given the job name and the build number to retry.
It copies the build parameters from the given build and adds a retry counter as an environmental variable to keep track of the number of retries (a maximum of 3 retries per build has been set).
This job is automatically triggered by the job "jenkins-test-parser" when a build fails due to a known issue and a retry action is needed.
This job is expected to fail if the same build has been retried more than 3 times. If it fails due to another issue, we need to take the appropiate action.
This Jenkins project test various CMS Jenkins slaves and makes sure that various communication channels between Jenkins Master/slave and slave and various services are working.
Jenkins project to trigger the jenkins slave test job for each selected slave
Aborts and kills a release building process.
This project checks all selected host for process running longer then set treshold (default 2 days) and kill them.
Q/A:
Kill stuck PR testing job and update PR accordingly
Job to run das client and cache the results in github to be used by IBs.
Update sif image for LUMI HPC
This job checks if https://cmssdt.cern.ch/lxr/ is acessable and if not, starts lxr-run-container job to restart the service.
This job sets modification timestamp of CMSSW source code according to commit history before
indexing it using LXR.
LXR index files based on modification timestamp. `git clone` ,however, sets files timestamps
to command's execution time. Without it, LXR would index every file, increasing jobs execution duration
and database size.
Deletes index of of IB's older then treshold on LXR ( default 14 days). If specified, it also deletes Release index.
Run this if there is any MySQL table to repair e.g. if lxr-remove-idex fails with error
ERROR 144 (HY000) at line 3: Table './lxr/lxr_usages' is marked as crashed and last (automatic?) repair failed
then run it with LXR_TABLE=lxr_usages
This job stops and deletes old docker contaner of LXR service and starts new one.
Runs Validation/Geometry/test/runP_Tracker_cfg.py and MaterialBudget.C for an IB
This job run periodically to monitor the running state of vms in openstack , if stopped , creates an email alert.
This job runs a check that makes sure /afs, /cvmfs, /cvmfs/cms.cern.ch, /cvmfs/cms-ci.cern.ch, /cvmfs/cms-ib.cern.ch, /cvmfs/grid.cern.ch and /cvmfs/unpacked.cern.ch are mounted and that singularity can start a container in the node. It runs on lxplus7, lxplus8 and lxplus9 considering the available hosts at the time of running and both aarch64 and ppc64le machines.
If any of the checks fails, it sends an email notification and writes the node name into a blacklist. For lxplus nodes, this blacklist is checked before connecting any node to make sure Jenkins does not connect to a blacklisted node. For aarch64 and ppc64le machines, it takes the corresponding nodes offline.
The blacklist is cleaned up for lxplus nodes if the tests run again successfully in any of the nodes or if the host is not in the list of available hosts anymore. The aarch64 and ppc64le nodes must be taken online manually after receiving the email notification and fixing the issue.
This job checks the status of the Jenkins nodes every day to verify that none of them has been manually disconnected/or disconnected by CLI and not bring back online.
It also notifies about the blacklisted nodes.
Find all hosts in an hostgroup and add them to jenkins. This job needs to run `ai-*` commands
to be run on aiadm nodes. As due to 2FA we can not do a batch login to aiadm nodes so `ai-*`
commands are run via a acrontab job
~cmsbuild/private/jenkins/scripts/process.sh ~/private/jenkins
Find all hosts in an hostgroup and add then to jenkins
Create Openstack VMs for a selected hostgroup.
Node configuration is obtained from https://github.com/cms-sw/cms-bot/tree/master/openstack/hg/
Job to delete openstack instances providing only the name. Only kills builder nodes (cmsbuild)
Job to delete openstack instances providing only the name
Job to delete openstack instances providing only the name
Job to delete openstack instances providing only the name
This job marks selected realeases depricated in realease map. Grid jobs will not use depricated release and posibly uninstall it. This is useful if a specific release contains and needs to be removed.
This job process resource metrics jsons for externals
This job process partial logs of Relvals and place files accordingly.
There is no need to re-try newer run is successful or running. If it keeps on failing then one need
to check the logs and find out the reason of failure. In that case some manual work is needed to cleanup
cmssdt.cern.ch logs.
This job process partial logs of Relvals and place files accordingly.
There is no need to re-try this is a newer run is successful. If it keeps on failing then one need
to check the logs and find out the reason of failure. In that case some manual work is needed to cleanup
cmssdt.cern.ch logs.
Processes a github issue to check if it is requesting the build of a new release. If the issue is not requesting any release, it ignores it.
Processes a github issue to check if it is requesting the creation of a new data repository. If the issue is not requesting any release, it ignores it.
Job to reindex existing indexes whenever the template schema is changed or for whatever other reason
This job processes data from release-run-reco-profiling and prepares https://cms-reco-profiling.web.cern.ch/cms-reco-profiling/
Posts a message in the github issue that triggered the build. Structure of the message depends on the option used.
Profiling jobs that are submitted manually for each CMSSW release. The performance is summarized on https://cms-reco-profiling.web.cern.ch/. Managed by cms-offline-conveners-reco@cern.ch.
This job tests submission to multiple sites. Some of them are expected to fail.
This run "scram build code-checks" for a cmssw PR to find out if it comply with cmssw code checks. In a CMSSW dev area, it runs git cms-merge-topic -u PR scram build code-checks This job re-tries twice to recover. If it still fails after two retires then try to understand the failure and re-try again.
Profiling jobs that are submitted manually for each CMSSW release. The performance is summarized on https://cms-reco-profiling.web.cern.ch/. Managed by cms-offline-conveners-reco@cern.ch.
Run this job at the start of your shift to receive MM pings from failed builds/tests
A wrapper project that starts a workspace-cleanup-slave and test-docker jobs on selected slaves.
Generates statistics for each IB ( merged pull request since last IB, test result summary, .etc)
as well as structure of release que and stores it in .json files. It then push it to cms-sw.github.io repo as well
as deploys on the web server. It is used to generate IB page.
Q/A
avalenzu: Testing build error in release https://github.com/cms-sw/cmssw/issues/41976 for el8_amd64_gcc12
This job generates property files for triggering os test jobs (e.g., test-os-cs8, test-os-alma8, etc) for any os_arch_compiler combination.
The operating systems tested are rhel8, rocky8, ubi8, el8, cs8, alma8 and lxplus8 os.
This job checks for changes in the parent image. If there are changes, it triggers the `build-docker-container` job so that our based image is updated and uploaded to the registry.
Generates categories.json file and uploads it to this github repo.
The date is later used to generate this page.
Job to run das client and cache the results in github to be used by IBs.
Job could fail due to github issues or disk related issue. Normally it is safe to just retry it once github/disk issues are resolved.
There is no need to retry if it has been rerun successfully after the last failure.
This job creates new release and tag it on cms-data PR merge. It also creates cmsdist PR with the new release tag
This job update contents of the "data" directory in cms-sw.github.io. It downloads our github pages, parses the log files, updates the documentation accordingly, and uploads the pages again. In this case, the job is automatically triggered after some minutes. If it fails, one just needs to check that the next build was successful so that pages are updated correctly.
Job to run das client and cache the results in github to be used by IBs.
This project adds the release information in cms-bot/releases.map file. CVMFS installation is started once a release is available in this file.
Generates CMSSW release notes using Jekyll.
This job uploads a release on cmsrep server once approved by release manager.
This job uploads a release on cmsrep server once approved by release manager.
Job used to transpile cmssdt ib page from ECMAscript6 to regular javascript and push changes to github.
Transpiles Logreader to pure js/html/css and deploys to cmssdt-web.
Connect to selected slave and cleans (deletes) workspace.