Answer a question

I am writing my first Jenkins pipeline using the docker agent. When executing the pipeline, I get the following errors in the log in the Run Docker Image stage (note that Pipeline steps have been removed for clarity):

sh: can't create /data/jenkins/workspace/my-workspace@tmp/durable-01234567/pid: nonexistent directory

Jenkins Log

Started by user Doug R.
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to http://my-git.example.com/scm/im/my-project.git
 > git config remote.origin.url http://my-git.example.com/scm/im/my-project.git # timeout=10
Fetching origin...
Fetching upstream changes from origin
 > git --version # timeout=10
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/*
Seen branch in repository origin/master
Seen 1 remote branch
Obtained Jenkinsfile from 01234567890abcdef01234567890abcdef
Running in Durability level: MAX_SURVIVABILITY
Running on Jenkins in /data/jenkins/workspace/my-workspace
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url http://my-git.example.com/scm/im/my-project.git # timeout=10
Fetching without tags
Fetching upstream changes from http://my-git.example.com/scm/im/my-project.git
 > git --version # timeout=10
 > git fetch --no-tags --progress http://my-git.example.com/scm/im/my-project.git +refs/heads/*:refs/remotes/origin/*
Checking out Revision 01234567890abcdef01234567890abcdef (master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 01234567890abcdef01234567890abcdef
Commit message: "Yet another test"
 > git rev-list --no-walk 01234567890abcdef01234567890abcdef # timeout=10
[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository http://my-git.example.com/scm/im/my-project.git
 > git init /data/jenkins/workspace/my-workspace # timeout=10
Fetching upstream changes from http://my-git.example.com/scm/im/my-project.git
 > git --version # timeout=10
 > git fetch --no-tags --progress http://my-git.example.com/scm/im/my-project.git +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url http://my-git.example.com/scm/im/my-project.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url http://my-git.example.com/scm/im/my-project.git # timeout=10
Fetching without tags
Fetching upstream changes from http://my-git.example.com/scm/im/my-project.git
 > git fetch --no-tags --progress http://my-git.example.com/scm/im/my-project.git +refs/heads/*:refs/remotes/origin/*
Checking out Revision 01234567890abcdef01234567890abcdef (master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 01234567890abcdef01234567890abcdef
Commit message: "Running without reuseNode true"
Running on Jenkins in /data/jenkins/workspace/my-workspace@2
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url http://my-git.example.com/scm/im/my-project.git # timeout=10
Fetching without tags
Fetching upstream changes from http://my-git.example.com/scm/im/my-project.git
 > git --version # timeout=10
 > git fetch --no-tags --progress http://my-git.example.com/scm/im/my-project.git +refs/heads/*:refs/remotes/origin/*
Checking out Revision 01234567890abcdef01234567890abcdef (master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 01234567890abcdef01234567890abcdef
Commit message: "Running without reuseNode true"
Wrote authentication to /root/.docker/config.json
[my-workspace@2] Running shell script
+ docker inspect -f . alpine:3.8
.
Jenkins does not seem to be running inside a container
$ docker run -t -d -u 0:0 -w /data/jenkins/workspace/my-workspace@2 -v /data/jenkins/workspace/my-workspace@2:/data/jenkins/workspace/my-workspace@2:rw,z -v /data/jenkins/workspace/my-workspace@2@tmp:/data/jenkins/workspace/my-workspace@2@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat alpine:3.8
[my-workspace@2] Running shell script
sh: can't create /data/jenkins/workspace/my-workspace@2@tmp/durable-0123457/pid: nonexistent directory
sh: can't create /data/jenkins/workspace/my-workspace@2@tmp/durable-0123457/jenkins-log.txt: nonexistent directory
sh: can't create /data/jenkins/workspace/my-workspace@2@tmp/durable-0123457/jenkins-result.txt: nonexistent directory
$ docker stop --time=1 01234567890abcdef01234567890abcdef01234567890abcdef01234567890abcdef
$ docker rm -f 01234567890abcdef01234567890abcdef01234567890abcdef01234567890abcdef
ERROR: script returned exit code -2
Finished: FAILURE

Jenkinsfile

Based on what I have been able to determine, the Jenkinsfile appears to be written correctly, but I'm certain it's not.

import groovy.json.*
pipeline {

    environment {
        COMPOSE_TLS_VERSION = "TLSv1_2"
        DOCKER_TLS_VERIFY = 1
        DOCKER_CERT_PATH = "/opt/ucp"
        DOCKER_HOST = "tcp://my-docker-host.example.com:443"
    }

    agent any
    stages {
        stage('Prepare Workspace') {
            steps {
                script {
                    step([$class: 'WsCleanup'])
                    checkout scm
                }
            }
        }

        stage('Run Docker Image') {
            agent {
                docker {
                    reuseNode true
                    image 'alpine:3.8'
                    registryUrl 'https://my-docker-registry.example.com'
                    registryCredentialsId 'my-docker-credentials'
                }
            }
            steps {
                sh 'ls -la'
                sh 'printenv'
            }
        }
    }
}

Update

I found this link, which talks about a Docker-in-Docker scenario, but I'm not running Docker-in-Docker.

Answers

My issue has a different root cause, but is the same basic issue as the Docker-in-Docker (see above).

In my situation, Jenkins was running the docker image on a different node than where Jenkins was located, so it didn't have access to my workspace directory. As a result, the solution identified in the article would likely also work:

One way to make it work is to have the agent workspace shared from the host, in your case using /jenkins/workspace as a volume in the agent so the data written by the agent is actually written in the host and then can be shared to the new container. You can also change the workspace location in the agent definition to something else than the default jenkins.

However, I don't have the necessary privileges to do this, so I'm currently using the workaround of building a temporary image, then executing that portion of the build in there. While this works in my case, it's unlikely that it would work in all cases.