ghost-storage-adapter-oracle-ords, A Oracle Cloud Autonomous JSON Databases storage adapter for Ghost 5.x

In my last post, how to custom storage adapters to make your self-hosted Ghost instance filesystem completely external?, I had a plan to implement a storage adapter of Ghost for Oracle Cloud Autonomous JSON Databases to leverage the always free 40Gb space provided by Oracle Cloud.

After researched Oracle ORDS REST API, I implement a orcal-ords-client lib to encapsulate the REST APIs for easy usage. Then I developed this storage adapter based on the orcal-ords-client lib.

Core Idea

The storage adapter is to save the images uploaded to Ghost into external rather than the file system where your Ghost instance deployed. I designed the below JSON document for a image:

{

"path": "the unique path of the image, just consider the image is saved to local file system",

"blob": "the json string of a Buffer object which is the content of the image, maximum size is 4.4M after tested",

"type": "the mimetype of the image, such as png, jpeg and etc, used to set the content-type when the image been requested"

}

With the three interfaces provided by the orcal-ords-client lib,

 putJSONDocument(alias: string, json: JSONValue): Promise<Item>;
 deleteJSONObject(alias: string, id: string): Promise<boolean>;
 queryJSONDocument(alias: string, query: any, payload: any): Promise<Collection>;

We are able to implement the required methods to custom the storage adapter.

delete: first use queryJSONDocument to find the id

of the image stored by path, and then delete the image via deleteJSONObject

exists: use queryJSONDocument to find the image stored by path

save: construct the JSON object and save into database via putJSONDocument

serve: use queryJSONDocument to find the image stored by path and then

pipe the Buffer content to response stream

read:

use queryJSONDocument to find the image stored by path and

return the Buffer from the blob field.

Ghost Configuration

"storage": {
  "active": "OracleImagesStorage",
  "OracleImagesStorage": {
    "oauthClients": [
        {
            "client_id": "YOUR_ORACLE_AUTONOMOUS_JSON_DATABASE_OAUTH_CLIENT_ID",
            "client_secret": "YOUR_ORACLE_AUTONOMOUS_JSON_DATABASE_OAUTH_CLIENT_SECRET",
            "schema": "YOUR_ORACLE_AUTONOMOUS_JSON_DATABASE_USER",
            "ords_url": "YOUR_ORACLE_AUTONOMOUS_JSON_DATABASE_ORDS_URL",
            "alias": "YOUR_ORACLE_AUTONOMOUS_JSON_DATABASE_COLLECTION_NAME"
        },
    ]
  }
}

You can put multiple Oracle Cloud Autonomous JSON Databases clients in the above.

You can get 40Gb free space to use per account. if you have 10 Oracle Cloud account,

You can use 400Gb free space.:)

Use this adapter in your Ghost

In my another blog(https://errong.win) hosted on free Google Computer Engine,

It is a self hosted Ghost instance. The below photo is served with path starting

with /content/images/, but you can't any images under the /content/images directory

on my GCP vm's local file system.

oracle-ords-client, a javascript lib to persistent your Oracle Cloud Autonomous JSON Databases via Oracle ORDS Database Management REST API

oracle-ords-client

oracle-ords-client is a JavaScript library that defines a simple and consistent abstraction for interacting with a Oracle Cloud Autonomous Databases via Oracle ORDS Database Management REST API.

declare class OracleORDSClient implements ORDSOAuthClient {
    config: ORDSOAuthClientConfig;
    private access_token;
    private expires_at;
    constructor(config: ORDSOAuthClientConfig);
    putJSONDocument(alias: string, json: JSONValue): Promise<Item>;
    deleteJSONObject(alias: string, id: string): Promise<boolean>;
    queryJSONDocument(alias: string, query: any, payload: any): Promise<Collection>;
    private buildUrl;
    private ensureOauthToken;
}

npm install oracle-ords-client

https://www.npmjs.com/package/oracle-ords-client

github home page

https://github.com/lengerrong/oracle-ords-client

Maximum JSONDocument size: 4.4M

Usage Sample:

https://github.com/lengerrong/oracle-ords-client/blob/main/lib/index.spec.ts

References:

How To Setup OAuth Clients to Connect Your Oracle Cloud Autonomous Databases via REST

cy.request with FormData and QueryString

correct FormData snippet

please pass form: true and with the payload as body rather than use FormData

cy.request({
    url,
    method: "post",
    headers: {
      Authorization: "Basic xxx",
      Accept: "*/*",
      "Cache-Control": "no-cache",
      "Accept-Encoding": "gzip, deflate, br"
    },
    form: true,
    body: {
      "xxx":"yyy"
    }
  }).then((response) => {})

I tried create a FormData object and past it as body, but didn't work.

correct QueryString snippet

please pass qs with a JSON object rather than use URLSearchParams

cy.request({
      url,
      method: "post",
      headers: {
        Authorization: `Bearer ${access_token}`,
      },
      qs: {
        "parametername": "parametervalue"
      }
    });

I tried create a URLSearchParams object and past it as qs, but didn't work.

cy.intercept won't work if the request not send by browser or send via cy.request

the below code will not work. the wait for the request alias will timeout with error: expect request of url but never occurred.

const url = "your api"
cy.intercept(url).as("yourapi");
cy.request(url);
cy.wait("@yourapi");

Spring JPA: Dynamic Schema|Table at runtime

named parameter bind only works on the where clause. what if you want different schema at runtime for the table part. such as

'select * from :tablename where ...'

'update :tablename set ..."

You will find the above won't work in your @Query annotation

use EntityManager::createNativeQuery

You can get the single result, the first result, result lists or result stream.

use entityManager.createNativeQuery(sql).executeUpdate()

use executeUpdate to update or delete records in the table

Not allowed to create transaction on shared EntityManager

Caused by: java.lang.IllegalStateException: Not allowed to create transaction on shared EntityManager - use Spring transactions or EJB CMT instead

Please call joinTransaction before execute update sql

Resolved: SSL Connection Reset

A java app encountered a weird SSL Connection Reset issue while upgrading to use java11. it talked to a legacy service, it works very well while running with java8. somehow 40% percent requests will failure in SSLException after java11.

  java.net.SocketException: Connection reset
  	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
  	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
  	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
  	at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
  	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
  	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:110)
  	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1408)
  	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1314)
  	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
  	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)

Cause

The legacy server is running on Oracle WebLogic Server with java8. the server will reset the connection sometimes due to unknown reason.

Analysis

We are using HttpClients custom the default request config.

CloseableHttpClient httpClient = HttpClients.custom().setDefaultRequestConfig(config)      .setSSLSocketFactory(sslCsf).setConnectionManager(cm).build();

I thought it might be cipher suite not match or socket timeout issue. It turns out the not migrated app also will receive the connect reset from the legacy server after enable the logs. The difference from the logs are:

java.net.SocketException: Connection reset // the existed app with java8 runtime
javax.net.ssl.SSLException: Connection reset // the new migrated app with java 11 runtime

I also observed the retry log from the existed app, but not see them in the new one.
org.apache.http.impl.execchain.RetryExec : Retrying request to {s}

Which makes sense, the existed app will retry the requests and got successfully response finally when connection reset happened. but the new app just throw the exception.

Why requested failed with SSLException not retry, but SocketException retried?

The default http request retry handler will not retry for the below 4 exceptions.

Solution

Write a custom http client retry handler and retry the request even encountered javax.net.ssl.SSLException: Connection reset.

You can implement your own HttpRequestRetryHandler class and set it as the retry handler for your http client.

I just copy the same implementation of DefaultHttpRequestRetryHanlder and removed SSLException.class

CloseableHttpClient httpClient = HttpClients.custom().setDefaultRequestConfig(config)
					.setSSLSocketFactory(sslCsf).setConnectionManager(cm)
					.setRetryHandler(new HttpRequestRetryHandler())

Debugging SSL/TLS Connections

use the following command-line option on the java command:

java -Djavax.net.debug=all

The following example shows potential logging settings in application.properties:

logging:
  level:
    org:
      apache:
        http: DEBUG

References:

https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/ReadDebug.html
https://docs.spring.io/spring-boot/docs/2.1.18.RELEASE/reference/html/boot-features-logging.html

2023, New Start! New AdSense Account! New Earnings!

It's been a while after I created adsense account to display ads on this blog. A few years later the payment finally reached my payout threshold, I just found the existed adsense account is bind to China. but I moved to Canada 3 years ago. I tried to change the country of it, but just found

Unfortunately it's not possible to change the country of your payment address in AdSense. If you've moved to a new country or territory, you'll need to cancel your existing AdSense account and create a new AdSense account.

So I created a new adsense account today on 2023-01-12. Though new account is still under reviewing, I believe it will be approved soon.

To reach out payout threshold in this year, I set up the below plan:

write one blog per week day!!!

write one blog per week day!!!

Try ingress more traffic to this blog

How To Setup OAuth Clients to Connect Your Oracle Cloud Autonomous Databases via REST

Oracle Cloud provides you about 40G always free Autonomous JSON Database. That's a lot for your personal learning or blog. Let's find a way to utilize the free space in your web application. This is a step by step to set up a OAuth Client to connect your Oracle Cloud Autonomous Databases.

Create a user account

ADMIN is the default user and it has the administrator permission to your database. You don't want to your application use ADMIN user to connect. It is recommended to create a separate user per web application.

As the ADMIN user, access Database Actions and create a user with the required privileges.