The code below is example of Jenkins job that copies files or folders form S3 bucket to EFS.
settings = [
s3_path_is_folder: false
]
pipeline {
agent {
label 'master'
}
parameters {
string(name: 'S3_PATH', defaultValue: '', description: 'S3 file path (s3://codeflex/examples/golang/main.go) or folder path (s3://codeflex/examples)', trim: true)
string(name: 'EFS_PATH', defaultValue: '', description: 'EFS folder path (ex. myfolder) or leave empty for root path', trim: true)
}
environment {
CODEFLEX_AWS_ACCOUNT_ID = '77777777777'
CODEFLEX_REGION = 'eu-west-1'
SECRET_ID = 'codeflex-rds'
}
options {
timestamps()
ansiColor('xterm')
skipDefaultCheckout true
disableConcurrentBuilds()
buildDiscarder(logRotator(numToKeepStr: '25', artifactNumToKeepStr: '25'))
}
stages {
stage('S3 Path Validation') {
steps {
script {
withAWS(role: 'codeflex-jenkins', roleAccount: env.CODEFLEX_AWS_ACCOUNT_ID, region: env.CODEFLEX_REGION) {
if ("${params.S3_PATH}".length() == 0) {
error('S3_PATH must be set!')
}
stdout = sh (script: "aws s3 ls ${params.S3_PATH}", returnStdout: true)
echo stdout
if (stdout.contains(' PRE ')) {
settings.s3_path_is_folder = true
echo 'S3_PATH is a folder'
}
else {
settings.s3_path_is_folder = false
echo 'S3_PATH is a file'
}
}
}
}
}
stage('Mount EFS') {
steps {
script {
withAWS(role: 'codeflex-jenkins', roleAccount: env.CODEFLEX_AWS_ACCOUNT_ID, region: env.CODEFLEX_REGION) {
sh('mkdir -p ~/efs-mount-point')
sh('sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport 172.28.81.41:/ ~/efs-mount-point')
sh('sudo chown -R jenkins:jenkins ~/efs-mount-point')
}
}
}
}
stage('Check Mount') {
steps {
script {
withAWS(role: 'codeflex-jenkins', roleAccount: env.CODEFLEX_AWS_ACCOUNT_ID, region: env.CODEFLEX_REGION) {
echo '**********************************************'
sh('df -T')
echo '**********************************************'
sh('ls -l ~/efs-mount-point')
echo '**********************************************'
}
}
}
}
stage('EFS Path Validation') {
steps {
script {
if ("${params.EFS_PATH}".length() > 0) {
efs_path = '~/efs-mount-point/' + params.EFS_PATH
stdout = sh( script: '''#!/bin/bash
set -ex
if [ -d ''' + efs_path + ''' ]; then
echo "true"
else
echo "false"
fi
''', returnStdout: true)
echo stdout
if (stdout.contains('false')) {
error('No such directory on EFS: ' + efs_path)
}
}
}
}
}
stage('Copy From S3 to EFS') {
steps {
script {
withAWS(role: 'codeflex-jenkins', roleAccount: env.CODEFLEX_AWS_ACCOUNT_ID, region: env.CODEFLEX_REGION) {
final_efs_path = ''
if ("${params.EFS_PATH}".length() > 0) {
final_efs_path = '~/efs-mount-point/' + params.EFS_PATH
}
else {
final_efs_path = '~/efs-mount-point'
}
if (settings.s3_path_is_folder) {
folder_name = (params.S3_PATH).substring((params.S3_PATH).lastIndexOf('/') + 1)
echo 'Folder name: ' + folder_name
final_efs_path = final_efs_path + '/' + folder_name
sh("aws s3 cp ${params.S3_PATH} ${final_efs_path} --recursive")
}
else {
sh("aws s3 cp ${params.S3_PATH} ${final_efs_path}")
}
echo '**********************************************'
sh("ls -l ${final_efs_path}")
echo '**********************************************'
}
}
}
}
}
post {
always {
deleteDir()
script {
sh('sudo umount ~/efs-mount-point')
sh('sudo rm -rf ~/efs-mount-point')
}
}
}
}
In the first stage ‘S3 Path Validation‘ I check that S3_PATH is not empty and then determine that provided path is an object or a folder.
In the second stage I’m creating a new directory and then running mounting command for EFS. Right after that I changing ownership to the Jenkins user. It’s important step, otherwise the copy command might fail due to lack of permissions.
In the next stage I just printing content of the directory where I mounted EFS and then Im validating that provided path on EFS exists.
‘Copy From S3 to EFS’ stage – here I construct the final path and depends on what was provided on S3_PATH I finally running aws s3 cp command which copies S3 objects to the mounted EFS.
In the post stage I removing the workspace by deleteDir() command and un-mounting the EFS.
