Continuous Integration (CI) is the process of automatically building and testing your application’s code each time a developer pushes a commit to a remote source-code control repository like GitLab or GitHub. The use of CI is an integral part of DevOps and can be used to great effect: lead times (e.g. the time it takes from idea to shipping) can be reduced and thanks to automated unit and integration testing, error rates typically drop as well.
In most cases a CI server coordinates the execution of tests using a so-called CI Pipeline. CI Pipelines are typically defined as code, and checked into the same Git repository as the application’s code. Most CI servers accept YAML syntax to define jobs and associate them to stages such as linting, build, test, deploy.
One of the golden rules in CI is the “keep the pipeline green”, in other words should the CI Pipeline encounter any errors work needs to stop and all hands must jointly resolve the problem before further commits can be pushed. It is easy to see that errors should be avoided to prevent “soft outages” – times during which development work is at best slowed down, in the worst case it can come to a halt for a period of time.
Local Tests
Testing code locally before committing/pushing it to the remote Git repository is very important to avoid problems with the CI Pipeline’s execution. This article concerns itself with local tests of JavaScript modules used with Oracle Database 23ai on Linux x86-64. JavaScript Modules are an interesting new database feature allowing developers to process data where they live using one of the most popular programming languages. With the addition of JavaScript Oracle Database features 3 programming languages (PL/SQL and Java being the other 2). The feature enabling JavaScript support on Linux x86-64 in Oracle Database 23ai is known as Multilingual Engine (MLE).
Testing your code typically involves the following steps:
- checking for syntactical correctness
- linting
- unit testing
- others
The easiest way to generate good code right from the beginning is to include support for these right when you write the code. Since this is a bit tricky with JavaScript (it doesn’t know about types) the choice fell to TypeScript. This should allow you to detect problems with your code very quickly, however it comes at the expense of a little extra work. This extra work however is almost guaranteed to pay off.
The MLE team has provided the TypeScript declarations used by all MLE built-in modules on NPM, documentation can be found on GitHub.
A previous version of this blog post used typescript-eslint for linting, however due to the deprecation of inflight it might no longer be safe to use. A replacement was found in Biome, which seems to be pretty popular in the React community.
NOTE: The use of biome is merely an example, not an endorsement for the tool. There are many more linters available to developers. Always ensure you are comfortable working with a given piece of software, and that you are (license, …) compliant.
Versions Used
The following versions were used in the creation of this article:
cat package.json
{
"dependencies": {
"@biomejs/biome": "1.8.0",
"mle-js": "^23.4.0",
"typescript": "^5.4.5"
},
"type": "module"
}
$ node --version
v20.14.0
Oracle Database 23ai Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
Version 23.4.0.24.05
With that out of the way let’s look at some code.
Initial JavaScript Example
For the sake of demonstration let’s assume someone created the following MLE JavaScript module in their favourite editor, saved as src/badfile.js. It’s deliberately riddled with issues, don’t use it.
/**
* Update the "lastUpdated" field in a purchase order, adding it if it does not
* yet exist. Uses the example defined in the JSON Developer's Guide, chapter 4
* Examples 4-1 and 4-3
* @param {object} purchaseOrder - the PO to update
* @param {string} lastUpdate - a string representation of a date (YYYY-MM-DDThh:mm:ss)
* @returns {object} the updated purchaseOrder
*/
function setLastUpdatedDate(purchaseOrder, lastUpdate) {
if (purchaseOrder === undefined) {
throw Error("unknown purchase order");
}
if (lastUpdate = undefined) {
lastUpdate = new Date().toISOString();
}
console.log(`last update set to ${lastUpdate}`);
purchaseOrder.lastUpdate = lastUpdate;
return purchaseOrder;
}
/**
* Use vanilla JavaScript to validate a PurchaseOrder. This could have been
* done with JSON schema validation as well
* @param {object} purchaseOrder - the PO to validate
* @returns {boolean} true if the PO could be successfully validated, false if not
*/
function validatePO(purchaseOrder) {
// a PO must contain line items
if (purchaseOrder.LineItems.length <= 0) {
return false;
}
// a PO must contain shipping instructions
if (purchaseOrder.ShippingInstructions === undefined) {
return false;
}
return true;
}
/**
* Fetch a PurchaseOrder from the database and process it. Store the last modification
* timestamp alongside
* @param {number} poNumber - the PONumber as stored in j_purchaseorder.po_document.PONumber
*/
export function processPurchaseOrder(poNumber) {
const result = session.execute(
`SELECT
po.po_document as PO
FROM
j_purchaseorder po
WHERE
po.po_document.ponumber = :1`,
[poNumber],
" thisIsAnIncorrectParameter "
);
// ensure the PO exists
if (result.rows === undefined) {
throw Error(`could not find a PO for PO Number ${poNumber}`);
} else {
const myPO = result.rows[0].PO;
}
// make sure the PO is valid
if (!validatePO(myPO)) {
throw Error(`Purchase Order ${poNumber} is not a valid PO`);
}
// do some fancy processing with the PO
// indicate when the last operation happened
myPO = setLastUpdatedDate(myPO, undefined);
result = session.execute(
`UPDATE j_purchaseorder po
SET
po.po_document = :myPO
WHERE
po.po_document.PONumber = :poNumber`,
{
myPO: {
dir: oracledb.BIND_IN,
type: oracledb.DB_TYPE_JSON,
val: myPO
},
poNumber: {
dir: oracledb.BIND_IN,
type: oracledb.NUMBER,
val: poNumber
}
}
);
if (result.rowsAffected != 1) {
throw Error(`unable to persist purchase order ${poNumber}`);
}
}
Before submitting the code to the database for testing the developer should ensure the code doesn’t have any errors. The example has been chosen to put emphasis on the fact that it’s very hard to ensure code quality by merely eyeballing the text 😉
Improving Code Quality
Converting the above code to TypeScript provides lots of benefits to developers:
- Type declarations available for all MLE modules add a safety net to the code.
- Linting TypeScript adds even more checks to ensure the transpiled code is good to go.
The remainder of this article is concerned with the transition to TypeScript and correction of the errors encountered.
Installing MLE JavaScript Type Declarations
As per the MLE module documentation the first step is to install the mle-js module from NPM. This module contains the type declarations for all built-in modules used later. The following instructions are relative to the project root directory.
$ npm install mle-js --save-dev
Installing the linter
In the next step you need to decide which linter to use, this post focuses on biome as per the introduction.
$ npm install --save-dev --save-exact @biomejs/biome
Creating a basic configuration
The first step working with biome is to create a configuration. The following code snippet shows the configuration used for this post. I created it using npx @biomejs/biome init, it’s a sane default and good starting point. Adjust as needed for your project.
{
"$schema": "https://biomejs.dev/schemas/1.8.0/schema.json",
"organizeImports": {
"enabled": true
},
"linter": {
"enabled": true,
"rules": {
"recommended": true
}
}
}
JavaScript to TypeScript
The first step in the conversion from JavaScript to Typescript is to rename the source file to goodFile.ts. Next you need to create a tsconfig.json file, needed for type-checking. The following is a minimal file instructing TypeScript to transpile JavaScript code with the latest version of JavaScript supported by Oracle Database 23ai (at the time of writing):
{
"compilerOptions": {
"target": "ESNext",
"module": "ESNext",
"rootDir": "src",
"outDir": "dist",
"noEmitOnError": true,
"esModuleInterop": true,
"forceConsistentCasingInFileNames": true,
"strict": true,
"skipLibCheck": false,
"lib": [
"ES6",
"ES2017",
"ES2021"
]
}
}
You also need to install TypeScript in your project, this is done using the now familiar npm i --save-dev typescript command.
TypeScript code must be told about the MLE JavaScript type definitions found in mle-js. This is done using a triple-slash directive to be added at the first line of the source code file, goodFile.ts:
/// <reference types="mle-js" />
/**
* Update the "lastUpdated" field in a purchase order, adding it if it does not
* yet exist. Uses the example defined in the JSON Developer's Guide, chapter 4
* Examples 4-1 and 4-3
* @param {object} purchaseOrder - the PO to update
* @param {string} lastUpdate - a string representation of a date (YYYY-MM-DDThh:mm:ss)
* @returns {object} the updated purchaseOrder
*/
function setLastUpdatedDate(purchaseOrder, lastUpdate) {
You will almost immediately see things light up in red in your IDE. If not, you can use the command line to check for problems. Let’s start with the compilation:
$ npx tsc --pretty false
src/goodFile.ts(11,29): error TS7006: Parameter 'purchaseOrder' implicitly has an 'any' type.
src/goodFile.ts(11,44): error TS7006: Parameter 'lastUpdate' implicitly has an 'any' type.
src/goodFile.ts(35,21): error TS7006: Parameter 'purchaseOrder' implicitly has an 'any' type.
src/goodFile.ts(55,38): error TS7006: Parameter 'poNumber' implicitly has an 'any' type.
src/goodFile.ts(65,9): error TS2559: Type '" thisIsAnIncorrectParameter "' has no properties in common with type 'IExecuteOptions'.
src/goodFile.ts(76,21): error TS2304: Cannot find name 'myPO'.
src/goodFile.ts(83,5): error TS2304: Cannot find name 'myPO'.
src/goodFile.ts(83,31): error TS2304: Cannot find name 'myPO'.
src/goodFile.ts(85,5): error TS2588: Cannot assign to 'result' because it is a constant.
src/goodFile.ts(95,22): error TS2304: Cannot find name 'myPO'.
The first step is to add types to get rid of the any type errors. The next pass looks a lot better, and it now catches quite a few problems:
npx tsc --pretty false
src/goodFile.ts(41,9): error TS2322: Type 'undefined' is not assignable to type 'string'.
src/goodFile.ts(89,9): error TS2559: Type '" thisIsAnIncorrectParameter "' has no properties in common with type 'IExecuteOptions'.
src/goodFile.ts(100,21): error TS2304: Cannot find name 'myPO'.
src/goodFile.ts(107,5): error TS2304: Cannot find name 'myPO'.
src/goodFile.ts(107,31): error TS2304: Cannot find name 'myPO'.
src/goodFile.ts(107,37): error TS2345: Argument of type 'undefined' is not assignable to parameter of type 'string'.
src/goodFile.ts(109,5): error TS2588: Cannot assign to 'result' because it is a constant.
src/goodFile.ts(119,22): error TS2304: Cannot find name 'myPO'.
Quite a few of them are common mistakes, like trying to assign a value to a constant after it has been initialised (linke 109). Other errors include the infamous assignment vs comparison operator (line 41) and so on.
One of the prime benefits of TypeScript – type checking – is clearly demonstrated in line 89. In this line the code tries to pass an unknown option to the execute() statement. With pure JavaScript this wouldn’t be detected until the first execution.
Let’s fix these errors as well. The resulting file is shown here for reference. Please note that the types are merely stubs to drive the point home, a proper implementation would add all the other required fields from the PO as well.
/// <reference types="mle-js" />
/**
* Stub interfaces, for demostration purpose only, should be fleshed out to
* reflect all data as per Example 4-3 in chapter 4 of the JSON Developer's
* Guide
*/
interface ILineItem {
ItemNumber: number,
Part: string,
Quantity: number
}
interface IShippingInstructions {
name: string,
address: string,
phone: string
}
interface IPurchaseOrder {
PONumber: number,
lastUpdate: string,
LineItems: ILineItem[],
ShippingInstructions: IShippingInstructions[]
}
/**
* Update the "lastUpdated" field in a purchase order, adding it if it does not
* yet exist. Uses the example defined in the JSON Developer's Guide, chapter 4
* Examples 4-1 and 4-3
* @param {object} purchaseOrder - the PO to update
* @param {string} lastUpdate - a string representation of a date (YYYY-MM-DDThh:mm:ss)
* @returns {object} the updated purchaseOrder
*/
function setLastUpdatedDate(purchaseOrder: IPurchaseOrder, lastUpdate: string): IPurchaseOrder {
if (purchaseOrder === undefined) {
throw Error("unknown purchase order");
}
if (lastUpdate == undefined) {
lastUpdate = new Date().toISOString();
}
console.log(`last update set to ${lastUpdate}`);
purchaseOrder.lastUpdate = lastUpdate;
return purchaseOrder;
}
/**
* Use vanilla JavaScript to validate a PurchaseOrder. This could have been
* done with JSON schema validation as well
* @param {object} purchaseOrder - the PO to validate
* @returns {boolean} true if the PO could be successfully validated, false if not
*/
function validatePO(purchaseOrder: IPurchaseOrder): boolean {
// a PO must contain line items
if (purchaseOrder.LineItems.length <= 0) {
return false;
}
// a PO must contain shipping instructions
if (purchaseOrder.ShippingInstructions === undefined) {
return false;
}
return true;
}
/**
* Fetch a PurchaseOrder from the database and process it. Store the last modification
* timestamp alongside
* @param {number} poNumber - the PONumber as stored in j_purchaseorder.po_document.PONumber
*/
export function processPurchaseOrder(poNumber: IPurchaseOrder["PONumber"]): void {
let result = session.execute(
`SELECT
po.po_document as PO
FROM
j_purchaseorder po
WHERE
po.po_document.PONumber = :1`,
[ poNumber ]
);
// ensure the PO exists
if (result.rows === undefined || result.rows.length === 0) {
throw new Error(`could not find Purchase Order ${poNumber}`);
}
let myPO = result.rows[0].PO as IPurchaseOrder;
// make sure the PO is valid
if (! validatePO(myPO)) {
throw new Error(`Purchase Order ${poNumber} failed validation`);
}
// do some (imaginary) fancy processing with the PO ...
// ... then: indicate when the last operation happened
myPO = setLastUpdatedDate(myPO, "");
result = session.execute(
`UPDATE j_purchaseorder po
SET
po.po_document = :myPO
WHERE
po.po_document.PONumber = :poNumber`,
{
myPO: {
dir: oracledb.BIND_IN,
type: oracledb.DB_TYPE_JSON,
val: myPO
},
poNumber: {
dir: oracledb.BIND_IN,
type: oracledb.DB_TYPE_NUMBER,
val: poNumber
}
}
);
if (result.rowsAffected != 1) {
throw new Error(`unable to persist purchase order ${poNumber}`);
}
}
At this stage, the TypeScript transpiler stops complaining
$ npx tsc --pretty false && echo "well done!"
well done!
Next up: linting
The above code looks all right on first inspection, doesn’t it? Well, you’re in for a surprise. The linter has uncovered some interesting details:
$ npx @biomejs/biome ci ./src
./src/goodFile.ts:41:20 lint/suspicious/noDoubleEquals FIXABLE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✖ Use === instead of ==
39 │ }
40 │
> 41 │ if (lastUpdate == undefined) {
│ ^^
42 │ lastUpdate = new Date().toISOString();
43 │ }
ℹ == is only allowed when comparing against null
39 │ }
40 │
> 41 │ if (lastUpdate == undefined) {
│ ^^
42 │ lastUpdate = new Date().toISOString();
43 │ }
ℹ Using == may be unsafe if you are relying on type coercion
ℹ Unsafe fix: Use ===
41 │ ····if·(lastUpdate·===·undefined)·{
│ +
./src/goodFile.ts:129:29 lint/suspicious/noDoubleEquals FIXABLE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✖ Use !== instead of !=
127 │ );
128 │
> 129 │ if (result.rowsAffected != 1) {
│ ^^
130 │ throw new Error(`unable to persist purchase order ${poNumber}`);
131 │ }
ℹ != is only allowed when comparing against null
127 │ );
128 │
> 129 │ if (result.rowsAffected != 1) {
│ ^^
130 │ throw new Error(`unable to persist purchase order ${poNumber}`);
131 │ }
ℹ Using != may be unsafe if you are relying on type coercion
ℹ Unsafe fix: Use !==
129 │ ····if·(result.rowsAffected·!==·1)·{
│ +
./src/goodFile.ts:42:9 lint/style/noParameterAssign ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✖ Reassigning a function parameter is confusing.
41 │ if (lastUpdate == undefined) {
> 42 │ lastUpdate = new Date().toISOString();
│ ^^^^^^^^^^
43 │ }
44 │
ℹ The parameter is declared here:
33 │ * @returns {object} the updated purchaseOrder
34 │ */
> 35 │ function setLastUpdatedDate(purchaseOrder: IPurchaseOrder, lastUpdate: string): IPurchaseOrder {
│ ^^^^^^^^^^^^^^^^^^
36 │
37 │ if (purchaseOrder === undefined) {
ℹ Use a local variable instead.
Well, that’s quite a list! Time to fix the code some more. Since it’s a bit tedious switching from command line to editor, many IDEs offer support for linting. Using IDE support for linting and syntax checking is the best way moving forward. If your list of problems is zero, chances are high that your code will pass the CI Pipeline, too.
Don’t forget to add linting and syntax checking to your CI Pipeline
With the hard part completed you can now add your linter and formatter to your toolchain. Git Hooks are pretty good for this, but more importantly, add support for linting, syntax checking and your unit tests into your CI Pipeline.
Happy coding!