Share files between different agent workspace in the same Jenkins Pipeline via stash|unstash

 


Share files between different agent workspace in the same Jenkins Pipeline via stash|unstash
Photo by The Halal Design Studio / Unsplash

I was working on a legacy app recently. The legacy app has angular as its frontend and spring boot app as its backend. Originally the frontend build outputs are committed in the app's git repository and packaged by spring boot by copying them to the ${project.basedir}/target/classes/static/ class path(since content under the folder will be served as static resources by spring boot). And we are using Jenkins pipeline to run the build for this app. Hence the pipeline only use a java jenkins slave to compile and package the spring boot app. But that is not the right way, isn't it?

We should run both build the pipeline for frontend angular app and backend sprint boot app. But an angular app required a node slave to build it out. We don't have a Jenkins slave that has both java and node installed. Then issue comes: How to build java and javascript apps in a single Jenkins pipeline?

Then I found Pipeline allows utilizing multiple agents in the Jenkins environment from within the sameJenkinsfile, which can helpful for more advanced use-cases such as executing builds/tests across multiple platforms.

utilizing multiple agents in Pipeline

Good. We could use node agent to build the angular app and use java agent to compile the spring boot app.

    agent {
        label 'maven' # agent can build java app
    }
    stages {
        stage('Build UI') {
            agent {
                label 'node8' # agent can build javascript app
            }
            steps {
                sh 'npm install'
                sh 'run build'
            }
        }
        stage('Compile') {
            // Build the code to generate the spring-boot jar
            steps {
                sh "mvn -B clean install -f ${POM_FILE}"
            }
        }
    }

The above snippet works well, both frontend and backend got built out. I encountered another challenge: Each agent will run the steps in its own container, which means we are not able to access the dist build out from node agent in the maven agent, since they are using different workspaces.

Share files between agents

While I was searching a proper way to address the above concern, I found stash and unstash directives:

stash: Stash some files to be used later in the build
unstash: Restore files previously stashed

Jenkinsfile

pipeline {
    agent {
        label 'maven'
    }
    environment {
        POM_FILE 'server/pom.xml'
    }
    stages {
        stage('Build UI') {
            agent {
                label 'node8'
            }
            steps {
                sh 'npm --prefix=ui install'
                sh 'npm --prefix=ui run build'
                stash includes: 'ui/dist/**', name: 'uidist'
            }
        }
        stage('Compile') {
            // Build the code to generate the spring-boot jar
            steps {
                sh "mvn -B clean install -f ${POM_FILE}"
            }
        }
        stage('Package') {
            // Build the code to generate the spring-boot jar
            steps {
                unstash "uidist"
                sh "ls -all ui/dist"
                sh "mvn -DskipTests=true -B package -f ${POM_FILE}"
            }
        }
    }
}
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<executions>
<execution>
<id>copy-resources</id>
<phase>validate</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>${project.basedir}/target/classes/static/</outputDirectory>
<resources>
<resource>
<directory>../ui/dist</directory>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>

References:


https://docs.cloudbees.com/docs/admin-resources/latest/automating-with-jenkinsfile/using-multiple-agents

What does a ws() block do in Jenkins?
I’m trying to convert few Jenkinsfiles from Scripted Pipeline to Declarative Pipeline. I have a block like this in a Jenkinsfile: ws(”/path/to/dir”) { // do stuff} I wonder what does it do e...
How to re-use previously created workspace across stages
I am facing an issue where I have two stages defined in my pipeline that both are run on the same node and need to be run in the same workspace. The first of these stages runs on my master node
Aggregating results of downstream parameterised jobs in Jenkins
I have a Jenkins Build job which triggers multiple Test jobs with the test name as a parameter using the Jenkins Parameterized Trigger Plugin. This kicks off a number of test builds on multiple exe...

Swagger annotations for API that allows downloading files as zip

 I have built out an API that allows you downloading log files as zip in a sprint boot app. The app also integrated with Swagger 3, so I also want this API can be hit in the Swagger UI. In this post, I will demonstrate how to use @ApiOperation annotation to tell Swagger UI download the response as a zip file.

use @ApiOperation for your method

The key is set media type to "application/octet-stream" which can be recognized by Swagger UI. By doing so, you are set "accept":"application/octet-stream" in the request header. Put below annotation section on top of your API's controller method.

@Operation(
    		summary = "",
    		description = "",
    		method="GET",
    		responses = {
                    @ApiResponse(responseCode = "200", description = "", content = {
                            @Content(
                                    mediaType = "application/octet-stream"
                            )
                    })
            })

You also need to set 'produces = {"application/octet-stream"}' in the Mapping annotation. This is to set the response's header with "Content-Type":"application/octet-stream".

@GetMapping(value = "/yourpath", produces = {"application/octet-stream"})

set response Header, "content-disposition": "attachment; filename=xxx.zip"

In a regular HTTP response, the Content-Disposition response header is a header indicating if the content is expected to be displayed inline in the browser, that is, as a Web page or as part of a Web page, or as an attachment, that is downloaded and saved locally.

@GetMapping(value = "/yourpath", produces = {"application/octet-stream"})
public void method(HttpServletResponse response) {
            response.setHeader(HttpHeaders.CONTENT_DISPOSITION, String.format("attachment; filename=%s.zip", transactionId));
}

Try it out in Swagger UI

You will be able to see a 'Download file' anchor once the api successfully response you the zip file.
You also be able find in the response headers:
content-disposition: attachment; filename=xxx.zip

How to Extract|Compress zip files using Java

 import java.util.zip.ZipEntry;

import java.util.zip.ZipInputStream;
import java.util.zip.ZipOutputStream;

extract Zip

public class ZipExtract {

    public List<String> extractZipFile(MultipartFile zipFile) {
        try (val zipInput = new ZipInputStream(zipFile.getInputStream())) {
            List<String> logFiles = new ArrayList<>();
            ZipEntry zipEntry;
            log.info("Received Zip File: {}", zipFile.getOriginalFilename());
            while ((zipEntry = zipInput.getNextEntry()) != null) {
                log.info("Extracted Log File: {}", zipEntry.getName());
                logFiles.add(IOUtils.toString(zipInput, StandardCharsets.UTF_8));
            }
            return logFiles;
        } catch (IOException e) {
            log.error("Fail to extract the zip file!{}", zipFile.getOriginalFilename(),  e);
        }
        return Collections.emptyList();
    }

}

comprese Zip

public class ZipCompress {

    public ZipOutputStream compressZip(List<List<String>> messages, OutputStream outputStream) throws IOException {
        val zos = new ZipOutputStream(outputStream);
        for (int i = 0; i < messages.size(); i++) {
            List<String> message = messages.get(i);
            val ze = new ZipEntry(message.get(0) + ".xml");
            zos.putNextEntry(ze);
            zos.write(message.get(1).getBytes(StandardCharsets.UTF_8), 0, message.get(1).length());
            zos.closeEntry();
        }
        return zos;
    }

}

Resolved: Generate XSD or WSDL jaxb classes into multiple packages via maven-jaxb2-plugin

assume your app consumes multiple XSD or WSDL files from different vendors. You might want to generate the jaxb binding classes into separately packages per vendor's schema files. in this post, I will tell you 2 ways to do so.

Use <generateDirectory>${project.build.directory}/generated-sources/xxx</generateDirectory>

add multiple executions with different configuration. for each execution, you have to specify different generateDirectory, or it won't work. the below is a sample in pom.xml.

<plugin>
    <groupId>org.jvnet.jaxb2.maven2</groupId>
    <artifactId>maven-jaxb2-plugin</artifactId>
    <version>0.14.0</version>
    <executions>
        <execution>
            <id>xxx-wsdl-generate</id>
            <goals>
                <goal>generate</goal>
            </goals>
            <configuration>
                <schemaLanguage>AUTODETECT</schemaLanguage>
                <generatePackage>com.xxx</generatePackage>
                <schemaDirectory>${project.basedir}/src/main/resources/xxx</schemaDirectory>
                <noFileHeader>true</noFileHeader>
                <schemaIncludes>
                    <include>*.wsdl</include>
                </schemaIncludes>
                <generateDirectory>${project.build.directory}/generated-sources/xxx</generateDirectory>
            </configuration>
        </execution>
        <execution>
            <id>yyy-wsdl-generate</id>
            <goals>
                <goal>generate</goal>
            </goals>
            <configuration>
                <schemaLanguage>AUTODETECT</schemaLanguage>
                <schemaDirectory>${project.basedir}/src/main/resources/yyy</schemaDirectory>
                <noFileHeader>true</noFileHeader>
                <schemaIncludes>
                    <include>*.wsdl</include>
                </schemaIncludes>
                <generateDirectory>${project.build.directory}/generated-sources/yyy</generateDirectory>
            </configuration>
        </execution>
    </executions>
</plugin>

use xjb binding files

A list of regular expression file search patterns to specify the binding files to be processed. Searching is based from the root of bindingDirectory.
If left undefined, then all *.xjb files in schemaDirectory will be processed. You can add below into <configuration></configuration> section.

    <bindingIncludes>
        <bindingInclude>*.xjb</bindingInclude>
    </bindingIncludes>

Here is a snippet of a xjb file.

<jaxb:bindings xmlns:jaxb="http://java.sun.com/xml/ns/jaxb" xmlns:xjc="http://java.sun.com/xml/ns/jaxb/xjc"
  xmlns:xs="http://www.w3.org/2001/XMLSchema" version="2.0">
  <jaxb:bindings
    schemaLocation="xxx.xsd"
    node="/xs:schema">
    <jaxb:schemaBindings>
      <jaxb:package name="com.xxx"/>
    </jaxb:schemaBindings>
  </jaxb:bindings>
  <jaxb:bindings
    schemaLocation="yyy.xsd"
    node="/xs:schema">
    <jaxb:schemaBindings>
      <jaxb:package name="com.yyy"/>
    </jaxb:schemaBindings>
  </jaxb:bindings>
  <jaxb:bindings
    schemaLocation="zzz.wsdl"
    node="(//xs:schema)[1]">
    <jaxb:schemaBindings>
      <jaxb:package name="com.zzz.first"/>
    </jaxb:schemaBindings>
  </jaxb:bindings>
  <jaxb:bindings
    schemaLocation="zzz.wsdl"
    node="(//xs:schema)[2]">
    <jaxb:schemaBindings>
      <jaxb:package name="com.zzz.second"/>
    </jaxb:schemaBindings>
  </jaxb:bindings>
</jaxb:bindings>

common XJC Errors

you might encounter the below issues while xjc try to generate the jaxb binding classes from your schema files, to name a few in below:

Two declarations cause a collision in the ObjectFactory class

unable to honor class customization

A class/interface with the same name is already in use. Use a class customization to resolve this conflict.

In my opinion, the root cause is package conflict. Please check the error logs and it will report to you in detail which line in which schema caused the errors. Find the line in your schema and figure out the xpath for that node. Then set package to a different one should always fix the issue for your case.

You might noticed, in the above xjb sample, for zzz.wsdl, I set the package name to different values for two different nodes. in the zzz.wsdl, it has two xs:schema nodes, both of them has a same class which cause conflicts.  so I use node="(//xs:schema)[1]" to find the first xs:schema node and set its package  to zzz.first.

  <jaxb:bindings
    schemaLocation="zzz.wsdl"
    node="(//xs:schema)[1]">
    <jaxb:schemaBindings>
      <jaxb:package name="com.zzz.first"/>
    </jaxb:schemaBindings>
  </jaxb:bindings>
  <jaxb:bindings
    schemaLocation="zzz.wsdl"
    node="(//xs:schema)[2]">
    <jaxb:schemaBindings>
      <jaxb:package name="com.zzz.second"/>
    </jaxb:schemaBindings>
  </jaxb:bindings>

ghost-storage-adapter-oracle-ords, A Oracle Cloud Autonomous JSON Databases storage adapter for Ghost 5.x

npm install  ghost-storage-adapter-oracle-ords

https://www.npmjs.com/package/ghost-storage-adapter-oracle-ords

https://github.com/lengerrong/ghost-storage-adapter-oracle-ords#readme


In my last post,  how to custom storage adapters to make your self-hosted Ghost instance filesystem completely external?, I had a plan to implement a storage adapter of Ghost for Oracle Cloud Autonomous JSON Databases to leverage the always free 40Gb space provided by Oracle Cloud.

After researched Oracle ORDS REST API, I implement a orcal-ords-client lib to encapsulate the REST APIs for easy usage. Then I developed this storage adapter based on the orcal-ords-client lib.

Core Idea

The storage adapter is to save the images uploaded to Ghost into external rather than the file system where your Ghost instance deployed. I designed the below JSON document for a image:

{
  "path": "the unique path of the image, just consider the image is saved to local file system",
  "blob": "the json string of a Buffer object which is the content of the image, maximum size is 4.4M after tested",
  "type": "the mimetype of the image, such as png, jpeg and etc, used to set the content-type when the image been requested"
}

With the three interfaces provided by the orcal-ords-client lib,
 putJSONDocument(alias: string, json: JSONValue): Promise<Item>;
 deleteJSONObject(alias: string, id: string): Promise<boolean>;
 queryJSONDocument(alias: string, query: any, payload: any): Promise<Collection>;
We are able to implement the required methods to custom the storage adapter.
delete: first use queryJSONDocument to find the id 
of the image stored by path, and then delete the image via deleteJSONObject
exists: use queryJSONDocument to find the image stored by path
save: construct the JSON object and save into database via putJSONDocument
serve: use queryJSONDocument to find the image stored by path and then
pipe the Buffer content to response stream
read:
use queryJSONDocument to find the image stored by path and
return the Buffer from the blob field.

Ghost Configuration

"storage": {
  "active": "OracleImagesStorage",
  "OracleImagesStorage": {
    "oauthClients": [
        {
            "client_id": "YOUR_ORACLE_AUTONOMOUS_JSON_DATABASE_OAUTH_CLIENT_ID",
            "client_secret": "YOUR_ORACLE_AUTONOMOUS_JSON_DATABASE_OAUTH_CLIENT_SECRET",
            "schema": "YOUR_ORACLE_AUTONOMOUS_JSON_DATABASE_USER",
            "ords_url": "YOUR_ORACLE_AUTONOMOUS_JSON_DATABASE_ORDS_URL",
            "alias": "YOUR_ORACLE_AUTONOMOUS_JSON_DATABASE_COLLECTION_NAME"
        },
    ]
  }
}
You can put multiple Oracle Cloud Autonomous JSON Databases clients in the above.
You can get 40Gb free space to use per account. if you have 10 Oracle Cloud account,
You can use 400Gb free space.:)

Use this adapter in your Ghost

In my another blog(https://errong.win) hosted on free Google Computer Engine,
It is a self hosted Ghost instance. The below photo is served with path starting
with /content/images/, but you can't any images under the /content/images directory
on my GCP vm's local file system.



oracle-ords-client, a javascript lib to persistent your Oracle Cloud Autonomous JSON Databases via Oracle ORDS Database Management REST API


oracle-ords-client

oracle-ords-client is a JavaScript library that defines a simple and consistent abstraction for interacting with a Oracle Cloud Autonomous Databases via Oracle ORDS Database Management REST API.

declare class OracleORDSClient implements ORDSOAuthClient {
    config: ORDSOAuthClientConfig;
    private access_token;
    private expires_at;
    constructor(config: ORDSOAuthClientConfig);
    putJSONDocument(alias: string, json: JSONValue): Promise<Item>;
    deleteJSONObject(alias: string, id: string): Promise<boolean>;
    queryJSONDocument(alias: string, query: any, payload: any): Promise<Collection>;
    private buildUrl;
    private ensureOauthToken;
}

npm install oracle-ords-client

https://www.npmjs.com/package/oracle-ords-client

github home page

https://github.com/lengerrong/oracle-ords-client

Maximum JSONDocument size: 4.4M

Usage Sample:


References:

How To Setup OAuth Clients to Connect Your Oracle Cloud Autonomous Databases via REST

cy.request with FormData and QueryString

correct FormData snippet

please pass form: true and with the payload as body rather than use FormData

cy.request({
    url,
    method: "post",
    headers: {
      Authorization: "Basic xxx",
      Accept: "*/*",
      "Cache-Control": "no-cache",
      "Accept-Encoding": "gzip, deflate, br"
    },
    form: true,
    body: {
      "xxx":"yyy"
    }
  }).then((response) => {})

I tried create a FormData object and past it as body, but didn't work.

correct QueryString snippet

please pass qs with a JSON object rather than use URLSearchParams

cy.request({
      url,
      method: "post",
      headers: {
        Authorization: `Bearer ${access_token}`,
      },
      qs: {
        "parametername": "parametervalue"
      }
    });

I tried create a URLSearchParams  object and past it as qs, but didn't work.

cy.intercept won't work if the request not send by browser or send via cy.request

the below code will not work. the wait for the request alias will timeout with error: expect request of url but never occurred.

const url = "your api"
cy.intercept(url).as("yourapi");
cy.request(url);
cy.wait("@yourapi");

fixed: embedded-redis: Unable to run on macOS Sonoma

Issue you might see below error while trying to run embedded-redis for your testing on your macOS after you upgrade to Sonoma. java.la...