Introduction

System administration is a crucial activity to ensure adherence of customer’s IT environment to security, compliance, technical, business and financial goals of the organization. While system administration consists of many important actions, patching is one of the most critical activity.

An ideal patching setup will consist of guidelines and processes for how patches are identified, tested, scheduled, and rolled out across on-premises, cloud and hybrid environments. In addition to the guidelines and processes, the setup also need to focus on communication plan, change management, tools and automation requirements (and possibilities) related to patching.

Patching process will change over time with the change in technology. To ensure all changes are analyzed, reviewed and implemented continuously, this process need to be of living nature.

Several companies and security patch administrators consider patching process to be a single step that provides secure computing landscape. In practice, patching is multistep approach and is necessary to ensure:

1. Integrity of servers is maintained by applying latest operating system security updates and patches based on recommendations from Risk Management Team (RMT). 

Servers represent access points to sensitive, confidential data and services, ensuring that security updates and patches are distributed and implemented in a timely manner is essential towards mitigating malware, exploitation and other threats.

Patching process is a continuous cycle that must be strictly followed. Each step in the process must be tuned and modified based on previous successes and failures. By spending time up front to create policies and procedures, companies can minimize the time and resource requirements needed to fulfill the patching demands.

Patch management is designed to give an organization control over the software updates it deploys. Any organization planning to patch its operational environment should ensure that the company has:

• Effective operations, including people who understand their roles and responsibilities
• Tools and technologies that are appropriate for effective patch management
• Effective project management processes

2. A baseline and time frame for patching is established, along with associated compliance needs.

A baseline is a set of configurations for a product or system that has been established as the company standard for building and deploying.

The process of updating the baseline can be handled through the standard patch management processes.

An application or software baseline should contain the information required to build or rebuild a system to a desired state. And, more importantly, the baseline should be used to rebuild or deploy a new system to the most current secure state—meaning that the baseline should contain all of the most current vendor-released patches. An environment might require several separate baselines to meet the needs of the organization. For example, the HR department will need a different set of applications installed than the engineering group will require another set. Thus, each department would need a separate baseline. Adhering to your baselines is important because a single unpatched, non-standard computer can make the entire environment vulnerable to attack. If a computer is deployed that falls under your current baseline, you will have to act quickly to bring it into compliance.

• If patching cannot be completed in the timeframe listed in the table above, compensating controls must be put in place within the timeframes above and the exception process must be followed.

• If a patch requires a reboot for installation, the reboot must occur within the timeframes outlined above.

Standard Patching Process

To achieve the objective of Integrity and the baseline are as follows:

1. Each quarter, the patch deployment team should consolidate security patches to be deployed during that quarterly patching window. In addition, if a mandatory patch is released by vendor during the quarter, the quarterly patch set should be updated only on an exception basis for that quarter
2. The patches should be deployed to all Non-Prod (test, stage, dev and demo) parallel, and the patch set should be tested on all the environments 
3. If any issue is found during the testing on customers Non-Prod environments, then the production deployment should be postponed until the issue is resolved
4. Once the patch set is tested on Non-Prod environments, Patch deployment team should seek approval by sending the test report the Internal Security Team (IST). IST will give a go-ahead for deploying the OS security patches on Prod after their evaluation. Then, a notification will be sent by IST to the Change Advisory Board (CAB) to approve the downtime for the Customer Environment.
5. CAB will inform the Technical Assistance Center (TAC) team, which will then send the communication about the approved downtime to the customer
6. Before the approved downtime, the Patch deployment team stages the patch bundle for that quarter and co-ordinates with the Apps team for the service stop/start and server restart. 
7. Once the service stop/start is done, patches should be deployed, and servers needs to be rebooted.
8. After the Pre and Post Health checks are validated, the servers will be released to the customers.
9. Customer will validate the application /services and will update the status in the ticket.
10. If the customer reports any issue on the apps side, we should immediately roll back the patches which was deployed or restore the servers from the previous backup.

Conclusion:  

Procedures for patching attempt to be as detailed as possible, but there is the presumption that those who are using it have intermediate administration skills. This document should not be followed blindly, but should serve as a guide to ensure that those standards and procedures adopted and approved by Customer Global IT are followed.