X

Break New Ground

Recent Posts

DevOps

Using GitLFS for Git Binary File Management with Visual Builder Studio

As a best practice your Git repository shouldn't include large binary files - since you usually don't actually manage the content of these files in a similar way to source code files. But sometime these files are part of your code base, and you would like to track them along with the rest of your code. GitLFS - an open source project - helps in doing this in an efficient way. The latest version of Oracle Visual Builder Studio added support for using GitLFS. Git Large File Storage (LFS) replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git, while storing the file contents on a remote server. Visual Builder Studio will store the files in your Oracle Cloud Object Storage. In the demo below you can see how easy it is to use GitLFS with your free Git repositories in Visual Builder Studio. After you download and install GitLFS on your client, you can add it to a specific Git repo by issuing a "git lfs install" in the directory into which you cloned your repository. Then indicate which files to track using commands such as "git lfs track *.bin". Add the new .gitattributes file to your git transaction, and that's it. From now on, when you add files of the specific type to your git repo - they will be stored outside of your git repo. Here is a quick demo -  For more info see our doc.  

As a best practice your Git repository shouldn't include large binary files - since you usually don't actually manage the content of these files in a similar way to source code files. But...

DevOps

Analyzing Your Code Dependencies for Vulnerabilities with Visual Builder Studio

Most of the projects developed today rely on open source libraries - it is likely that your project does too. While using these libraries accelerates development by relying on proven solution for common needs, it also introduces a potential risk factor. Over time people discover vulnerabilities in many of these libraries. While most of the issues will get fixed in newer versions of the libraries, it is up to the consumer of the library to make sure they are up-to-date on the latest version. It is a hard task to constantly track the libraries you are using and to keep them up to date. For projects that are already in production, and have not kept their libraries updated, this is an even bigger risk opening organization to potential malicious attacks leveraging these known vulnerabilities. Automating a review process of the libraries used in your code against databases of known vulnerability can help you stay one-step ahead of attackers. This is exactly what the Dependency Vulnerability Analyzer in Visual Builder Studio helps you achieve. In the demo below you can see how simple it is to automate the process of reviewing the libraries your code uses, get a report about any known vulnerability in those libraries, and even how fixing your code can be streamlined. The demo specifically uses a project that is managing the libraries used through NPM - and scans those dependencies from the package.json file against several repositories of known vulnerabilities. You can also conduct similar checks for Java projects that leverage Maven and pom files to manage dependencies. Check out the demo, and start implementing these security check for your project today!    

Most of the projects developed today rely on open source libraries - it is likely that your project does too. While using these libraries accelerates development by relying on proven solution for...

Open Source

Getting Started With The Oracle Cloud Infrastructure Python SDK

In a recent blog post I illustrated how to use the OCI Command Line Interface (CLI) in shell scripts. While the OCI CLI is comprehensive and powerful, it may not be the best solution when you need to handle a lot of data in shell scripts. In such cases using a programming language such as Python and the Oracle Cloud Infrastructure Python SDK makes more sense. Data manipulation is much easier, and the API is —as expected— more complex. In an attempt to demystify the use of the OCI Python SDK, I have re-written and improved the sample oci-provision.sh shell script in Python. This sample project is named oci-compute and is published on GitHub. This blog post highlights the key concepts of the OCI Python SDK, and together with the oci-compute sample code it should help you to get started easily. About oci-compute The oci-compute tool does everything oci-provision.sh does; better, faster and with some additional capabilities: List available Platform, Custom and Marketplace images Create Compute Instances from a Platform, Custom or Marketplace image A cloud-init file can be specified to run custom scripts during instance configuration List, start, stop and terminate Compute Instances Command line syntax and parameters naming are similar to the OCI CLI tool. See the project README for more information on usage and configuration. I am using this tool on a daily basis to easily manage OCI Compute instances from the command line. OCI Python SDK installation At the time of this writing, the SDK supports Python version 3.5 or 3.6 and can be easily installed using pip, preferably in a Python virtual environment. Installation and required dependencies are described in detail in the documentation. oci-compute installation The oci-compute utility is distributed as a Python package. The setup.py file lists the SDK as dependency; installing the tool will automatically pull the SDK if not already installed. See the README file for detailed installation steps, but in short it is as simple as creating a virtual environment and running: $ pip3 install . The package is split in two main parts: cli.py: handles the command line parsing using the Click package. It defines all the commands, sub-commands and their parameters; instantiate the OciCompute class and invoke its methods. oci_compute.py: defines the OciCompute class which interacts with the OCI SDK. This is the most interesting part of this project. OCI SDK Key concepts This section describes the key concepts used by the OCI SDK. Configuration The first step for using the OCI SDK is to create a configuration dictionary (Python dict). While you can build it manually, you will typically use the oci.config.from_file API call to load it from a configuration file. The default configuration file is ~/.oci/config. It is worth noticing that the OCI CLI uses the same configuration file and provides a command to create it: $ oci setup config For oci-compute, the configuration file is loaded during the class initialization: self._config = oci.config.from_file(config_file, profile) API Service Clients The OCI API is organized in Services, and for each Service you will have to instantiate a Service Client. For example, our oci-compute package uses the following Services: Compute Service (part of Core Services): to manage the Compute Services (provision and manage compute hosts). Virtual Network Service (part of Core Services): to manage the Networking Components (virtual cloud network, Subnet, …) Identity Service: to manage users, groups, compartments, and policies. Marketplace Service: to manage applications in Oracle Cloud Infrastructure Marketplace We instantiate the Service Clients in the class initialization: # Instantiate clients self._compute_client = oci.core.ComputeClient(self._config) self._identity_client = oci.identity.IdentityClient(self._config) self._virtual_network_client = oci.core.VirtualNetworkClient(self._config) self._marketplace_client = oci.marketplace.MarketplaceClient(self._config) Models Models allows you to create objects needed by the API calls. Example: to use an image from the Marketplace, we need to subscribe to the Application Catalog. This is done with the ComputeClient create_app_catalog_subscription method. This method needs an CreateAppCatalogSubscriptionDetails object as parameter. We will use the corresponding model to create such object: oci.core.models.CreateAppCatalogSubscriptionDetails. In oci-compute: app_catalog_subscription_detail = oci.core.models.CreateAppCatalogSubscriptionDetails( compartment_id=compartment_id, listing_id=app_catalog_listing_agreements.listing_id, listing_resource_version=app_catalog_listing_agreements.listing_resource_version, oracle_terms_of_use_link=app_catalog_listing_agreements.oracle_terms_of_use_link, eula_link=app_catalog_listing_agreements.eula_link, signature=app_catalog_listing_agreements.signature, time_retrieved=app_catalog_listing_agreements.time_retrieved ) self._compute_client.create_app_catalog_subscription(app_catalog_subscription_detail).data Pagination All list operations are paginated; that is: they will return a single page of data and you will need to call the method again to get additional pages. The pagination module allows you, amongst other, to retrieve all data in a single API call. Example: to list the available images in a compartment we could do: response = self._compute_client.list_images(compartment_id) which will only return the first page of data. To get get all images at once we will do instead: response = oci.pagination.list_call_get_all_results(self._compute_client.list_images, compartment_id) The first parameter to list_call_get_all_results is the paginated list method, subsequent parameters are the ones of the list method itself. Waiters and Composite operations To wait for an operation to complete (e.g.: wait until an instance is started), you can use the wait_until function. Alternatively, there are convenience classes in the SDK which will perform an action on a resource and wait for it to enter a particular state: the CompositeOperation classes. Example: start an instance and wait until it is started. The following code snippet shows how to start an instance and wait until it is up and running: compute_client_composite_operations = oci.core.ComputeClientCompositeOperations(self._compute_client) compute_client_composite_operations.instance_action_and_wait_for_state( instance_id=instance_id, action='START', wait_for_states=[oci.core.models.Instance.LIFECYCLE_STATE_RUNNING]) Error handling A complete list of exceptions raised by the SDK is available in the exception handling section of the documentation. In short, if your API calls are valid (correct parameters, …) the main exception you should care about is the ServiceError one which is raised when a service returns an error response; that is: a non-2xx HTTP status. For the sake of simplicity and clarity in the sample code, oci-compute does not capture most exceptions. Service Errors will result in a Python stack traceback. A simple piece of code where we have to consider the Service Error exception is illustrated here: for vnic_attachment in vnic_attachments: try: vnic = self._virtual_network_client.get_vnic(vnic_attachment.vnic_id).data except oci.exceptions.ServiceError: vnic = None if vnic and vnic.is_primary: break Putting it all together The oci-compute sample code should be self explanatory, but let’s walk through what happens when e.g. oci-compute provision platform --operating-system "Oracle Linux" --operating-system-version 7.8 --display-name ol78 is invoked. First of all, the CLI parser will instantiate an OciCompute object. This is done once at the top level, for any oci-compute command: ctx.obj['oci'] = OciCompute(config_file=config_file, profile=profile, verbose=verbose The OciCompute class initialization will: Load the OCI configuration from file Instantiate the Service Clients The Click package will then invoke provision_platform function which in turn will call the OciCompute.provision_platform method. We use the oci.core.ComputeClient.list_images to retrieve the most recent Platform Image matching the given Operating System and its version: images = self._compute_client.list_images( compartment_id, operating_system=operating_system, operating_system_version=operating_system_version, shape=shape, sort_by='TIMECREATED', sort_order='DESC').data if not images: self._echo_error("No image found") return None image = images[0] We then call OciCompute._provision_image for the actual provisioning. This method uses all of the key concepts explained earlier. Pagination is used to retrieve the Availability Domains using the Identity Client list_availability_domains method: availability_domains = oci.pagination.list_call_get_all_results( self._identity_client.list_availability_domains, compartment_id ).data VCN and subnet are retrieved using the Virtual Network Client (list_vcns and list_subnets methods) Metadata is populated with the SSH public key and a cloud-init file if provided: # Metadata with the ssh keys and the cloud-init file metadata = {} with open(ssh_authorized_keys_file) as ssh_authorized_keys: metadata['ssh_authorized_keys'] = ssh_authorized_keys.read() if cloud_init_file: metadata['user_data'] = oci.util.file_content_as_launch_instance_user_data(cloud_init_file) Models are used to create an instance launch details (oci.core.models.InstanceSourceViaImageDetails, oci.core.models.CreateVnicDetails and oci.core.models.LaunchInstanceDetails methods): instance_source_via_image_details = oci.core.models.InstanceSourceViaImageDetails(image_id=image.id) create_vnic_details = oci.core.models.CreateVnicDetails(subnet_id=subnet.id) launch_instance_details = oci.core.models.LaunchInstanceDetails( display_name=display_name, compartment_id=compartment_id, availability_domain=availability_domain.name, shape=shape, metadata=metadata, source_details=instance_source_via_image_details, create_vnic_details=create_vnic_details) Last step is to use the launch_instance_and_wait_for_state Composite Operation to actually provision the instance and wait until it is available: compute_client_composite_operations = oci.core.ComputeClientCompositeOperations(self._compute_client) response = compute_client_composite_operations.launch_instance_and_wait_for_state( launch_instance_details, wait_for_states=[oci.core.models.Instance.LIFECYCLE_STATE_RUNNING], waiter_kwargs={'wait_callback': self._wait_callback}) We use the optional waiter callback to display a simple progress indicator oci-compute demo Short demo of the oci-compute tool: Conclusion In this post, I’ve shown how to use oci-compute to easily provision and manage your OCI Compute Instances from the command line as well as how to create your own Python scripts using the Oracle Cloud Infrastructure Python SDK.

In a recent blog post I illustrated how to use the OCI Command Line Interface (CLI) in shell scripts. While the OCI CLI is comprehensive and powerful, it may not be the best solution when you need to...