Linting MLE JavaScript Modules in Continuous Integration Pipelines

Continuous Integration (CI) is the process of automatically building and testing your application's code each time a developer pushes a commit to a remote source-code control repository like GitLab or GitHub. The use of CI is an integral part of DevOps and can be used to great effect: lead times (e.g. the time it takes from idea to shipping) can be reduced and thanks to automated unit and integration testing, error rates typically drop as well.

In most cases a CI Server coordinates the execution of tests using a so-called CI Pipeline. CI Pipelines are typically defined as code, and checked into the same Git repository as the application's code. Most CI servers accept YAML syntax to define jobs and associate them to stages such as linting, build, test, deploy.

One of the golden rules in CI is the "keep the pipeline green", in other words should the CI Pipeline encounter any errors work needs to stop and all hands must jointly resolve the problem before further commits can be pushed. It is easy to see that errors should be avoided to prevent "soft outages" - times during which development work is at best slowed down, in the worst case it can come to a halt for a period of time.

Local Tests

Testing code locally before committing/pushing it to the remote Git repository is very important to avoid problems with the CI Pipeline's execution. This article concerns itself with local tests of JavaScript modules used with Oracle Database 23c Free-Developer Release. JavaScript Modules are an interesting new database feature allowing developers to process data where they live using one of the most popular programming languages. With the addition of JavaScript Oracle Database features 3 programming languages (PL/SQL and Java being the other 2). The feature enabling JavaScript support on Linux x86-64 in Oracle Database 23c Free-Developer Release is known as Multilingual Engine (MLE).

Using eslint with its TypeScript plugin it possible to thoroughly check MLE modules locally before submitting them to the database. This approach does a lot more than what eslint would be able to do on its own. On the downside it is a little more involved than writing pure JavaScript code, but using a minimum set of TypeScript features pays off in the end. Not only is it possible to check for syntax errors, using the TypeScript declaration you can even use type checks in the form of "is this parameter I'm passing to a function correct?"

The MLE team has provided the TypeScript declarations used by all MLE built-in modules on NPM, documentation can be found on GitHub.

NOTE: The use of eslint is merely an example, not an endorsement for the tool. There are many more linters available to developers. Always ensure you are comfortable working with a given piece of software, and that you are (license, ...) compliant.

Versions Used

The following versions were used in the creation of this article:

• @typescript-eslint/eslint-plugin@5.60.1
• @typescript-eslint/parser@5.60.1
• eslint@8.44.0
• mle-js@23.2.0
• typescript@5.0.4
• node 18.16.1

Example

For the sake of demonstration let's assume someone created the following MLE JavaScript module in their favourite editor, saved as blogpost.ts (yes, that's a TypeScript file, otherwise it wouldn't be possible to perform compile time checks using the type declarations, please bear with me, it'll get clearer in a minute). Note that errors have been added deliberately.

/**
 * Update the "lastUpdated" field in a purchase order, adding it if it does not 
 * yet exist. Uses the example defined in the JSON Developer's Guide, chapter 4 
 * Examples 4-1 and 4-3
 * @param {object} purchaseOrder - the PO to update 
 * @param {string} lastUpdate - a string representation of a date (YYYY-MM-DDThh:mm:ss)
 * @returns {object} the updated purchaseOrder
 */
function setLastUpdatedDate(purchaseOrder, lastUpdate) {

    if (purchaseOrder === undefined) {
        throw Error("unknown purchase order");
    }

    if (lastUpdate = undefined) {
        lastUpdate = new Date().toISOString();
    }

    console.log(`last update set to ${lastUpdate}`);

    purchaseOrder.lastUpdate = lastUpdate;

    return purchaseOrder;
}

/**
 * Use vanilla JavaScript to validate a PurchaseOrder. This could have been
 * done with JSON schema validation as well but would have been harder to
 * lint using eslint.
 * @param {object} purchaseOrder - the PO to validate
 * @returns {boolean} true if the PO could be successfully validated, false if not
 */
function validatePO(purchaseOrder) {

    // a PO must contain line items
    if (purchaseOrder.LineItems.length <= 0) {
        return false;
    }

    // a PO must contain shipping instructions
    if (purchaseOrder.ShippingInstructions === undefined) {
        return false;
    }

    return true;
}

/**
 * Fetch a PurchaseOrder from the database and process it. Store the last modification
 * timestamp alongside
 * @param {number} poNumber - the PONumber as stored in j_purchaseorder.po_document.PONumber 
 */
export function processPurchaseOrder(poNumber) {

    const result = session.execute(
        `SELECT
            po.po_document as PO
        FROM
            j_purchaseorder po
        WHERE
            po.po_document.ponumber = :1`,
        [ poNumber ],
        "thisIsAnIncorrectParameter"
    );

    // ensure the PO exists
    if (result.rows === undefined) {
        throw Error(`could not find a PO for PO Number ${poNumber}`);
    } else {
        const myPO = result.rows[0].PO;
    }

    // make sure the PO is valid
    if (! validatePO(myPO)) {
        throw Error(`Purchase Order ${poNumber} is not a valid PO`);
    }

    // do some fancy processing with the PO

    // indicate when the last operation happened
    myPO = setLastUpdatedDate(myPO, undefined);

    result = session.execute(
        `UPDATE j_purchaseorder po
        SET
            po.po_document = :myPO
        WHERE
            po.po_document.PONumber = :poNumber`,
        {
            myPO: {
                dir: oracledb.BIND_IN,
                type: oracledb.DB_TYPE_JSON,
                val: myPO
            },
            poNumber: {
                dir: oracledb.BIND_IN,
                type: oracledb.NUMBER,
                val: poNumber
            }
        }
    );

    if ( result.rowsAffected != 1) {
        throw Error(`unable to persist purchase order ${poNumber}`);
    }
}

Before submitting the code to the database for testing the developer should ensure the code doesn't have any errors. The example has been chosen to put emphasis on the fact that it's very hard to ensure code quality by merely eyeballing the text ;)

Installing linter and TypeScript compiler

As per the MLE module documentation the first step is to install the mle-js module from NPM. This module contains the type declarations for all built-in modules used later.

mkdir $HOME/path/to/project && cd $HOME/path/to/project
npm install mle-js --save-dev

In the next step you need to decide which linter to use, this post focuses on eslint with its TypeScript plugin. There was a regression in eslint preventing the use of the latest TypeScript version (5.1.6) at the time of writing. This might no longer apply, you can check typescript-eslint issue 6934 for an update on the progress.

npm install typescript@5.0.4 --save-dev
npm install eslint @typescript-eslint/parser @typescript-eslint/eslint-plugin --save-dev

Creating a basic linter configuration

ESLint needs a configuration file, the following example is a good starting point for your own .eslintrc.cjs:

module.exports = {
    extends: [
        "eslint:recommended",
        "plugin:@typescript-eslint/recommended",
        "plugin:@typescript-eslint/recommended-requiring-type-checking"
    ],
    parser: "@typescript-eslint/parser",
    parserOptions: {
        project: "./tsconfig.json",
        tsconfigRootDir: __dirname
    },
    plugins: [
        "@typescript-eslint"
    ],
    root: true,
    rules: {
        "no-const-assign": 2,
        "no-console": 2,
        "@typescript-eslint/no-unsafe-member-access": "off",
        "@typescript-eslint/no-unsafe-assignment": "off",
        "@typescript-eslint/no-unsafe-return": "off",
        "@typescript-eslint/restrict-template-expressions": "off"
    }
}

The example is taken from the documentation and extended for the use with MLE JavaScript modules. The rules features a number of exceptions that are necessary unless you create your code in "proper" TypeScript.

Verifying your code for correctness

The TypeScript code must be told about the type definitions in mle-js. This is done using a triple-slash directive to be added at the first line of the source code file, blogpost.ts:

/// <reference types="mle-js" />

/**
 * Update the "lastUpdated" field in a purchase order, adding it if it does not

[... more code ...]

With the type declarations made available to the source file you are ready to perform the linting process. As per TypeScript eslint issue 352 eslint does not concern itself with type checking. This is done by TypeScript's tsc command line utility. Therefore 2 passes are required:

1. Syntax checks using eslint
2. Type checks using tsc

Syntax checks

Use eslint to check for syntax errors (there are of course many):

$ npx eslint blogpost.ts

/home/martin/path/to/projectblogpost.ts
  17:9   error    Expected a conditional expression and instead saw an assignment  no-cond-assign
  17:9   error    Unexpected constant condition                                    no-constant-condition
  21:5   error    Unexpected console statement                                     no-console
  72:15  warning  'myPO' is assigned a value but never used                        @typescript-eslint/no-unused-vars
  85:5   error    'result' is constant                                             no-const-assign

✖ 5 problems (4 errors, 1 warning)

This is a good start, but as you can see the error in processPurchaseOrder() was not detected:

    const result = session.execute(
        `SELECT
            po.po_document as PO
        FROM
            j_purchaseorder po
        WHERE
            po.po_document.ponumber = :1`,
        [ poNumber ],
        "thisIsAnIncorrectParameter"
    );

Let's move on to the Typescript part. Not only does this check for errors, it also creates the JavaScript version of the TypeScript file to be used in the database. Remember that MLE JavaScript doesn't support native TypeScript - you have to transpile TypeScript to JavaScript first before you can use it as the source for a MLE module.

Syntax checking

Use the tsc command line utility to check for additional errors, using the TypeScript declarations as defined in mle-js. The --noEmit flag tells TypeScript not to create output. This is an important step, otherwise you may end up with broken code.

$ npx tsc --noEmit
blogpost.ts:65:9 - error TS2559: Type '"thisIsAnIncorrectParameter"' has no properties in common with type 'IExecuteOptions'.

65         "thisIsAnIncorrectParameter"
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

blogpost.ts:76:22 - error TS2304: Cannot find name 'myPO'.

[... more output skipped ...]

As you can see the error was detected - this would have otherwise slipped through the net.

Transpiling TypeScript to JavaScript

The next step in your pipeline concerns the JavaScript code generation. Up until then the CI Pipeline

• validated your TypeScript code for syntactical correctness using eslint
• used the type declarations to perform compile time checks

With both of these steps completed successfully you can upload the transpiled code into the database. My pipeline uses the following snippet to transpile the code

$ npx tsc --project tsconfig.json 

This will create the JavaScript file, blogpost.js, to be used with Oracle Database 23c Free-Developer Release.

Note: You can find the tsconfig.json I used in the appendix.

Summary of the CI Pipeline's linting stage

Using linting and compile type checks offer more confidence in your code deployment than using eslint on its own. It doesn't come without a price though, you need to either write TypeScript code or pretend your JavaScript code is TypeScript by using a different file extension and a few exceptions to the rule set.

As you could see the combination of eslint and tsc detected a bunch of problems. As a result I didn't check the "broken" code into the repository and didn't cause problems with the CI Pipeline. After correcting the code no errors are encountered, and I have a working JavaScript file I can import into the database later. And it didn't require me to use all the bells and whistles of TypeScript although I could have made use of additional features.

Note: I provided the "fixed" version of the MLE module in the appendix.

Loading the module into Oracle Database 23c Free-Developer Release

Once the transpiling stage succeeds, the JavaScript module can be loaded into the database. Remember not to load the TypeScript file (*.ts) - MLE in Oracle Database 23c Free-Developer Release requires JavaScript.

A multitude of ways exist to achieve this goal from your CI Pipeline, from SQL Loader to REST APIs powered by ORDS to read the module as a BLOB. Another option is to use a directory object and the using BFILE() clause. Assuming your source file resides in a directory know to Oracle, for example SRC_CODE_DIR, you can easily create the module as follows:

create or replace mle module blogpost_module
language javascript
version'1.2'
using BFILE(SRC_CODE_DIR, 'blogpost.js');
/

CI Pipeline

With the JavaScript code passing the lint stage it can be submitted to the version control repository after suitable unit tests have been added (not shown here-this topic will be covered in a later blog post). Thanks to the project's setup the linting configuration file is identical in the remote repository, all tests that passed locally should also pass in the pipeline.

The only difference when performing syntax checks in the CI Pipeline compared to the local tests is the scope: instead of a single file, all JavaScript files in the project are subject to linting.

Summary

Using CI is considered an industry best-known-method. Provided a suitable CI Pipeline exists many problems known to hamper the software release cycle can be addressed with great effect. Linting code is an integral part of the CI Pipeline's execution, this article demonstrated how to perform linting of JavaScript modules.

You are not limited to using a linter in your CI Pipeline, many popular code editors feature plugins for eslint and its cousins.

A follow-up to this post describes how to create unit tests for MLE JavaScript code, stay tuned for more details.

Appendix

Corrected version of the buggy blogpost.ts file:

/// <reference types="mle-js" />

/**
 * Update the "lastUpdated" field in a purchase order, adding it if it does not 
 * yet exist. Uses the example defined in the JSON Developer's Guide, chapter 4 
 * Examples 4-1 and 4-3
 * @param {object} purchaseOrder - the PO to update 
 * @param {string} lastUpdate - a string representation of a date (YYYY-MM-DDThh:mm:ss)
 * @returns {object} PO object with its lastUpdate field changed
 */
function setLastUpdatedDate(purchaseOrder, lastUpdate) {

    if (purchaseOrder === undefined) {
        throw Error("unknown purchase order");
    }

    if (lastUpdate === undefined) {
        lastUpdate = new Date().toISOString();
    }

    purchaseOrder.lastUpdate = lastUpdate;

    return purchaseOrder;
}

/**
 * Use vanilla JavaScript to validate a PurchaseOrder. This could have been
 * done with JSON schema validation (inside the database) as well but would 
 * have been harder to lint using eslint.
 * @param {object} purchaseOrder - the PO to validate
 * @returns {boolean} true if the PO could be successfully validated, false if not
 */
function validatePO(purchaseOrder) {

    // a PO must contain line items
    if (purchaseOrder.LineItems.length <= 0) {
        return false;
    }

    // a PO must contain shipping instructions
    if (purchaseOrder.ShippingInstructions === undefined) {
        return false;
    }

    return true;
}

/**
 * Fetch a PurchaseOrder from the database and process it. Store the last modification
 * timestamp alongside
 * @param {number} poNumber - the PONumber as stored in j_purchaseorder.po_document.PONumber 
 */
export function processPurchaseOrder(poNumber) {

    let result = session.execute(
        `SELECT
            po.po_document as PO
        FROM
            j_purchaseorder po
        WHERE
            po.po_document.PONumber = :1`,
        [ poNumber ]
    );

    // ensure the PO exists
    if (result.rows === undefined || result.rows.length === 0) {
        throw Error(`could not find a PO for PO Number ${poNumber}`);
    }

    let myPO = result.rows[0].PO;

    // make sure the PO is valid
    if (! validatePO(myPO)) {
        throw Error(`Purchase Order ${poNumber} is not a valid PO`);
    }

    // do some fancy processing with the PO (not shown here)

    // indicate when the last operation happened
    myPO = setLastUpdatedDate(myPO, undefined);

    result = session.execute(
        `UPDATE j_purchaseorder po
        SET
            po.po_document = :myPO
        WHERE
            po.po_document.PONumber = :poNumber`,
        {
            myPO: {
                dir: oracledb.BIND_IN,
                type: oracledb.DB_TYPE_JSON,
                val: myPO
            },
            poNumber: {
                dir: oracledb.BIND_IN,
                type: oracledb.NUMBER,
                val: poNumber
            }
        }
    );

    if ( result.rowsAffected != 1) {
        throw Error(`unable to persist purchase order ${poNumber}`);
    }
}

Generated tscfonfig.json:

{
  "compilerOptions": {

    /* Projects */

    /* Language and Environment */
    "target": "es2022",                                  /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */
    "lib": ["es2022"],                                   /* Specify a set of bundled library declaration files that describe the target runtime environment. */

    /* Modules */
    "module": "es2022",                                  /* Specify what module code is generated. */
    "rootDir": "./",                                     /* Specify the root folder within your source files. */

    /* JavaScript Support */
    "allowJs": true,                                  /* Allow JavaScript files to be a part of your program. Use the 'checkJS' option to get errors from these files. */
    "checkJs": true,                                  /* Enable error reporting in type-checked JavaScript files. */

    /* Emit */

    /* Interop Constraints */
    "esModuleInterop": true,                             /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables 'allowSyntheticDefaultImports' for type compatibility. */
    "forceConsistentCasingInFileNames": true,            /* Ensure that casing is correct in imports. */

    /* Type Checking */
    "strict": true,                                      /* Enable all strict type-checking options. */
    "noImplicitAny": false,                              /* Enable error reporting for expressions and declarations with an implied 'any' type. */

    /* Completeness */
    "skipLibCheck": true                                 /* Skip type checking all .d.ts files. */
  },
  "exclude": [
    "node_modules"
  ]
}