X

Blogs about Deep Learning, Machine Learning, AI, NLP, Security, Oracle Traffic Director,Oracle iPlanet WebServer

Recent Posts

Meena Vyas

Anomaly Detection

What is Anomaly Detection In data science, anomaly detection is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data. In the following figure anomaly data which is a spike (shown in red color). But the same spike occurs at frequent intervals is not an anomaly. There are 3 types of Machine Learning Techniques Supervised Machine learning Unsupervised Machine Learning Semi- supervised Machine learning Refer https://machinelearningmastery.com/supervised-and-unsupervised-machine-learning-algorithms/ for more details. Unsupervised anomaly detection techniques detect anomalies in an unlabeled test data set under the assumption that the majority of the instances in the data set are normal by looking for instances that seem to fit least to the remainder of the data set. We will need Unsupervised Anomaly detection when we don’t have labelled data. i.e. we don’t have data with label of when anomaly has occurred. Different types of Anomaly detection techniques are described below. A safe bet is to use wisdom of the crowds by using multiple ensemble methods. We can then choose to combine them through majority vote, or union or intersection of the individual algorithms’ verdicts. Isolation Forest and LoF This is nearest neighbour based Anomaly detection sklearn has IsolationForest and LocalOutlierFactor (LoF) If data is too big, there is an implementation of LoF for spark ‘K’ Nearest Neighbour This is a Nearest Neighbour based approach Simply finding z-scores to ‘k’ nearest neighbors and using cutoff of 3 works surprisingly well in practice (though is limited to global anomalies only and can’t figure out local outliers). One class SVM Classification based approach One-class Support Vector Machine (OCSVM), can be used as an unsupervised anomaly detection method. However, to work well, the percentage of anomalies in the dataset needs to be low. CBOF (Cohesiveness Based Outlier Factor It is a clustering based Anomaly detection. Deep Learning LSTM/Auto encoders RNN, LSTM (long short term memory), auto encoders Neural network approach Available in Keras/Tensorflow and other libraries Typically neural networks need a lot of data There are some more methods like probability based multivariate gaussian distribution, PCA,t-SNE. Feel free to walk through my ipython notebook https://github.com/meenavyas/Misc/blob/master/AnomalyDetection.ipynb In this notebook , I have tried IsolationForest amd Lof. As you can see in the plots given below, points which got high scoring from these algorithms are anomalies. When we run anomaly detection automatically on streaming data for that we may need infrastructure like Apache Spark. This blog is also posted in my personal blog here. References https://en.wikipedia.org/wiki/Anomaly_detection More about the dataset used http://odds.cs.stonybrook.edu/http-kddcup99-dataset/ https://github.com/scikit-learn/scikit-learn/blob/master/benchmarks/bench_isolation_forest.py https://github.com/scikit-learn/scikit-learn/blob/master/benchmarks/bench_lof.py https://scikit-learn.org/stable/auto_examples/plot_anomaly_comparison.html#sphx-glr-auto-examples-plot-anomaly-comparison-py

What is Anomaly Detection In data science, anomaly detection is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data. I...

Meena Vyas

Face recognition - can we identify “Boy” from “Alien”?

The question is can we identify “Boy” from “Alien”? Face Recognition addresses "who is this identity" question. This is a 1:K matching problem.  We have a database of K faces we have to identify whose image is the give input image. Facenet is Tensorflow implementation of the face recognizer described in the paper "FaceNet: A Unified Embedding for Face Recognition and Clustering". FaceNet learns a neural network that encodes a face image into a vector of 128 numbers.  By comparing two such vectors, we can then determine if two pictures are of the same identity. FaceNet is trained by minimizing the triplet loss. For more information on triplet loss refer https://machinelearning.wtf/terms/triplet-loss/ Since training requires a lot of data and a lot of computation, I haven’t trained it from scratch here. I have used previously trained model. I have taken the inception networks model implementation and weights from 4th course deeplearning.ai “Convolutional Neural Networks” from Coursera. The network architecture follows the Inception model from [Szegedy *et al.](https://arxiv.org/abs/1409.4842). More details about inception v1 is in this blog https://www.analyticsvidhya.com/blog/2018/10/understanding-inception-network-from-scratch/ This network uses 96x96 dimensional RGB images as its input. It encodes each input face image into a 128-dimensional vector. First, for each image of “Alien” and “Boy” (I have taken 52 images of each), I converted them into encoding and stored into a database. Here is the code that does that What happens when Alien and Boy will pass through our image recognition system? For each of the images of “Alien” and “Boy”, first compute the target encoding of the image from image path. Find the encoding from the database that has smallest distance with the target encoding. If minimum distance (L2 distance between the target "encoding" and the current "encoding" from the database) is greater than 0.7 we assume the face is not in the database.   When Alien tries to pass through our face recognition system   Input Test Image of Alien Result Closest image Alien Alien Boy Alien Alien Alien Note that there is no image in the database like the green eyed image. distance is 0.5105655. So ay  be we can keep a cut off at 0.5 instead of 0.7   When Boy tries to pass through our face recognition system   Input Test Image of Boy Result Closest image Alien Boy   Boy Alien Alien Boy Results look pretty good. Summary We should re-train facenet with Alien and Boy pictures to get better results. Image dimensions were only 96x96 so that could have thrown a lot of information away Model was trained on human faces which has different embeddings than cats I have split database images and final images based on dates on which pictures were taken assuming pictures of the same dates must be similar. On inspection, I found that in the cases were the final images are very different from images we added in database that is they were never seen before, the results are incorrect. This can be fixed by adding more different types of images in the database. This content is also posted in my personal space here.

The question is can we identify “Boy” from “Alien”? Face Recognition addresses "who is this identity" question. This is a 1:K matching problem.  We have a database of K faces we have to identify whose...

Object Detection Using OpenCV YOLO

Object Detection Using OpenCV YOLO You only look once (YOLO) is a state-of-the-art, real-time object detection system. It applies a single neural network to the full image. This network divides the image into regions and predicts bounding boxes and probabilities for each region. These bounding boxes are weighted by the predicted probabilities. It looks at the whole image at test time so its predictions are informed by global context in the image. It is extremely fast, more than 1000x faster than R-CNN and 100x faster than Fast R-CNN.  Non-Maxima Suppression : During prediction time you may have lot's of box predictions around a single object the non maxima supression algorithm will filter out those boxes that overlap between each other and also some threshold. I tried object detection on this video. Download the following files : yolov3.cfg yolov3.weight which contains pre-trained weights using wget command as shown below wget https://pjreddie.com/media/files/yolov3.weights yolov3.txt which contains all the class names this library can detect. Import relevant packages. Add random color to each class which will be used to draw rectangles. For each image, call processImage function which does the following Takes image frame as input read pre-trained model, Gather predictions  If confidence is less than 0.5 ignore the detection Apply non-max suppression Draw boundary boxes Save the output images with boundary boxes Now we try to observe a few of these output images Due to non maxima suppression sometimes if the two cars are in the same area, one gets undetected at times. Refer my ipython notebook https://github.com/meenavyas/Misc/blob/master/ObjectDetectionUsingYolo/ObjectDetectionUsingYolo.ipynb for full source code. References and thanks to https://www.arunponnusamy.com/yolo-object-detection-opencv-python.html https://pjreddie.com/darknet/yolo/ https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/single-shot-detectors/yolo.html This blog is also posted in my personal website here.

Object Detection Using OpenCV YOLO You only look once (YOLO) is a state-of-the-art, real-time object detection system. It applies a single neural network to the full image. This network divides...

Meena Vyas

Cat face detection using OpenCV

In this blog I am going to explain object detection using OpenCV library. OpenCV (Open Source Computer Vision Library: http://opencv.org) is an open-source BSD-licensed library that includes several hundreds of computer vision algorithms. It has modules like Image Processing, Video Analysis, Object Detection. OpenCV was designed for computational efficiency and with a strong focus on real-time applications. A Haar Cascade is a classifier which is used to detect the object for which it has been trained for, from the source. The Haar Cascade is by superimposing the positive image over a set of negative images. The training is generally done on a server and on various stages. Download haar-cascade xml files from  link here. Read license terms before downloading, copying, installing or using. You can create your own haar cascade files by looking at the videos here.  I have downloaded two files 'haarcascade_frontalcatface.xml' and 'haarcascade_frontalcatface_extended.xml' Set tunable parameters like scale factor and minimum neighbors. I am reading 6 images of cats and 1 image dog. First convert the image to gray scale. Use the above two haar cascades to get coordinates of rectangles where cat front face is located (if any). Plot the rectangles. When there are some errors retry with different scale factor and minimum neighbors  parameters. As you can see it plots the rectangles around 6 images of cats properly and doesn’t plot anything around the dog face as expected. Code is in my github repository. You can also look at yolo object detection blog https://www.pyimagesearch.com/2018/11/12/yolo-object-detection-with-opencv/ Thanks to my models : Alien, Princess, Boy and Lucky Reference ·       https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_objdetect/py_face_detection/py_face_detection.html#face-detection ·       https://opencv.org/ ·       https://www.quora.com/What-is-haar-cascade ·        https://docs.opencv.org/trunk/d7/d8b/tutorial_py_face_detection.html ·       https://pythonprogramming.net/haar-cascade-object-detection-python-opencv-tutorial/ ·       https://github.com/opencv/opencv/tree/master/data/haarcascades https://www.pyimagesearch.com/2018/11/12/yolo-object-detection-with-opencv/ This site is also posted in my personal blog here.  

In this blog I am going to explain object detection using OpenCV library. OpenCV (Open Source Computer Vision Library: http://opencv.org) is an open-source BSD-licensed library that includes several...

Code Generation using LSTM (Long Short-term memory) RNN network

A recurrent neural network (RNN) is a class of neural network that performs well when the input/output is a sequence. RNNs can use their internal state/memory to process sequences of inputs. Neural Network models are of various kinds One to one: Image classification where we give an input image and it returns a class to which the image belongs to. One to Many: Image Captioning where input is a picture and output is a sentence describing the picture. Many to One: Sentimental Analysis where input is a tweet and the output is a class like positive or negative. Many to Many: Sequence to sequence model with Encoder – Decoder architecture: Language translation model where input is a sentence (let’s say in English) and output is a sentence in another language (let’s say French). There are two popular variants of RNNs LSTM (Long Short-term memory) GRU We should try both to see which one is performing better for the problem we are trying to solve. In this blog I have tried to generate new source code using LSTM. Here are the steps Import required packages Then set EPOCH and Batch size. These should be tuned properly. In preprocessing stage, I have downloaded Openssl source code from github and concatenated all .c files into a file called “train.txt”. I was getting out of memory so I just took 1/3rd Openssl files. We can improve this code to load the source code in batches. We have to create a vocab list in preprocessing stage and saving it into a file and reading the file. I have used character based model. We can made word based model also. We can use word embedding layer also which will be needed when we have more difficult problem sets. I have used 2 LSTM layers with Dropout of 0.2 each and a Dense in the end with softmax. We can try different models and compare. Visualize the model as shown below Training for 10 epochs. As you can see loss is coming down gradually in every epoch from 2.97 to 1.55. Here is the output it generated. We have given it a random starting point As you can see it has done a very good job. It has returned values from a function based on if condition and start another function. Here is the code in github. Please try it out and see. References https://en.wikipedia.org/wiki/Recurrent_neural_network https://www.tensorflow.org/tutorials/sequences/recurrent https://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/ https://en.wikipedia.org/wiki/Long_short-term_memory https://en.wikipedia.org/wiki/Gated_recurrent_unit THis blog is also posted in my personal website here.

A recurrent neural network (RNN) is a class of neural network that performs well when the input/output is a sequence. RNNs can use their internal state/memory to process sequences of inputs. Neural...

Meena Vyas

Plant Seedlings Classification using Keras

This blog is dedicated to my friends who want to learn AI/ML/deep learning. Explore Plant Seedling Classification dataset in Kaggle at the link https://www.kaggle.com/c/plant-seedlings-classification. It has training set images of 12 plant species seedlings organized by folder. Each image has a filename that is its unique id. The goal of the competition is to create a classifier capable of determining a plant's species from a photo. Test set we need to predict the species of each image. You can download this code from here. Start a new Kernel. First import all the required python modules We can look at the contents of ../input/train directory to see what it contains. Create two functions that converts string classes of plant seedlings into integer and reverse. This is for beautification only. Then we set the parameters of the model like Epoch, Learning rate, Batch size. The more we tune these the better the results will be. In training neural network, one epoch means one pass of the full training set.  Batch size refers to the number of training examples utilized in one iteration.  Here is a blog that explains learning rate.   Then we read the training data images. We resize all images into 128*128. Then we create model we user 3 layers with activation function ReLU and in the last layer add a "softmax" layer. In the context of artificial neural networks, the rectifier is an activation function. It enables better training of deeper networks,compared to the widely used activation functions prior to 2011, i.e., the logistic sigmoid and its more practical counterpart, the hyperbolic tangent. The rectifier is, as of 2018, the most popular activation function for deep neural networks.  A unit employing the rectifier is also called a rectified linear unit (ReLU). The softmax function is often used in the final layer of a neural network-based classifier. Such networks are commonly trained under a log loss (or cross-entropy) regime, giving a non-linear variant of multinomial logistic regression. We have used loss function is categorical cross-entropy function and Adam Optimizer. Then we read training data partition into 75:25 split,  compile the model and save it. We also used image augmentation. We have added Image Data Generator to generate more images by slightly shifting the current images. Next step is to generate matplotlib plots and read test data The output of this is shown below : Next step is to create the CSV file for test data and upload it to the competition. References https://www.pyimagesearch.com/2017/NUM_CLASSES/11/image-classification-with-keras-and-deep-learning/ https://machinelearningmastery.com/grid-search-hyperparameters-deep-learning-models-python-keras/ https://machinelearningmastery.com/save-load-keras-deep-learning-models/ https://en.wikipedia.org/wiki/Softmax_function https://en.wikipedia.org/wiki/Rectifier_(neural_networks) https://machinelearningmastery.com/image-augmentation-deep-learning-keras/ A copy of this blog is posted here

This blog is dedicated to my friends who want to learn AI/ML/deep learning. Explore Plant Seedling Classification dataset in Kaggle at the link https://www.kaggle.com/c/plant-seedlings-classification. I...

Meena Vyas

spaCy - Named Entity and Dependency Parsing Visualizers

I was searching for some pre-trained models that would read text and extract entities out of it like cities, places, time and date etc. automatically as training a model manually is time consuming and needs a lot of data to train if somebody has already done it why not reuse it. Named-entity recognition (NER) (also known as entity identification, entity chunking and entity extraction) is a sub-task of information extraction that seeks to locate and classify named entities in text into predefined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. spaCy is the leading open-source library for advanced NLP. spaCy has excellent pre-trained named-entity recognizers in a number of models. Note that we used "en_core_web_sm" model. I have read that some spaCy models are case-sensitive. I tried converting text of a random news article into Named Entities using this visualization tool "displaCy Named Entity Visualizer". You can look at the results in the link here  Here is the output of the paragraph I had entered in the tool If you look at spaCy documentation, it gives the explanation of these entity types PERSON (People, including fictional): It classified "AI", "CAGR", "Tencent" wrongly as person in our context. NORP (Nationalities or religious or political groups): It classified 'Asian' and "Chinese" correctly as nationality. GPE (Countries, cities, states): It classified country "U.S." correctly but misclassified "Alibaba" and "AI" in our context. ORG (Companies, agencies, institutions etc): It classified "Baidu", "Google", "IBM", and "Microsoft" correctly. CARDINAL (Numerals that do not fall under another type): It classified "one" and "three" correctly. PERCENT(Percentage, including "%"): "45%", "50%" and "65%" were classified correctly. DATE (Absolute or relative dates or periods): "2017" was classified correctly. For the entry types which are not correct , we need to re-train the model with our own contextual data as training set. A Part-Of-Speech Tagger (POS Tagger) is a piece of software that reads text in some language and assigns parts of speech to each word (and other token), such as noun, verb, adjective, etc. A dependency parser analyzes the grammatical structure of a sentence, establishing relationships between "head" words and words which modify those heads. The figure below shows a snapshot of dependency parser of the paragraph above. Full image can be viewed in Dependency Visualizers here. Dependency Parsers can read various forms of plain text input and can output various analysis formats, including part-of-speech tagged text, phrase structure trees, and a grammatical relations (typed dependency) format. Dependency Parsing can be used to solve various complex NLP (Natural Language Processing) problems like Named Entity Recognition, Relation Extraction, translation. For more details on Dependency parsing, watch this Stanford video. Read about Parsey McParseface (and SyntaxNet), open source dependency parser here This blog is posted here.

I was searching for some pre-trained models that would read text and extract entities out of it like cities, places, time and date etc. automatically as training a model manually is time consuming and...

Meena Vyas

Recommendation Systems

A recommender system or a recommendation system seeks to predict the "rating" or "preference" a user would give to an item. The system recommends users certain items that they think the user may be interested in, based on what they know about the user, especially when the catalogue of items is very large. Recommender systems are a useful alternative to search algorithms since they help users discover items they might not have found otherwise. Some of the examples of recommendation engines are: Amazon and Netflix recommend products based on previous behaviour. Mobile application Peapod uses a recommendation engine to allow users to fill their shopping basket based on previous orders. Recommendations can be: Editorial hand curated For example: staff picks, home pages of websites Simple aggregates Depends on aggregated activities of other users not on user. For example: "top", "most popular", "most recent" youtube videos. Tailored to individual users like Amazon, Netflix recommendation systems. Suppose we have 5 users and 3 items/movies which users have rated.  Recommender system predicts the ratings of the empty cells. Steps involve gathering known ratings and extrapolate known ratings from unknown ratings. There are two methods of ratings collection Explicit Ratings: Simply ask users to rate items. In this case data we get is excellent but it doesn’t scale. Problem is matrix is sparse i.e. most people don’t rate. Examples of explicit data collection include the following: Asking a user to rate an item on a sliding scale. Asking a user to search. Asking a user to rank a collection of items from favourite to least favourite. Presenting two items to a user and asking him/her to choose the better one of them. Asking a user to create a list of items that he/she likes. Implicit ratings: Learn ratings from other user actions. For example purchase implies high rating but we can’t learn low ratings. Examples of implicit data collection include the following: Observing the items that a user views in an online store. Analysing item/user viewing times. Keeping a record of the items that a user purchases online. Obtaining a list of items that a user has listened to or watched on his/her computer. Analysing the user's social network and discovering similar likes and dislikes. When building a model from a user's behaviour, a distinction is often made between explicit and implicit forms of data collection. Different Kinds Of Recommendation Systems Content Based filtering (personality-based approach) Content-based filtering approaches utilise a series of discrete characteristics of an item in order to recommend additional items with similar properties. For example, Recommend items to customer “c” similar to previous items rated highly by “c”. Recommend movies with same actor, director, genre. Recommend articles with similar content Recommend people with many common friends Collaborative filtering Collaborative filtering approaches build a model from a user's past behaviour (items previously purchased or selected and/or numerical ratings given to those items) as well as similar decisions made by other users. This model is then used to predict items that the user may have an interest in. Collaborative filtering is based on the assumption that people who agreed in the past will agree in the future, and that they will like similar kinds of items as they liked in the past. The system generates recommendations using only information about rating profiles for different users or items. Collaborative systems locate peer users / items with a rating history similar to the current user or item and generate recommendations using this neighbourhood. Many algorithms have been used in measuring user similarity or item similarity in recommender systems. For example, the k-nearest neighbour (k-NN) approach and the Pearson Correlation (centered cosine similarity). A particular type of collaborative filtering algorithm uses matrix factorization, a low-rank matrix approximation technique. There are two models of collaborative filtering User-to-user similarity model Item-to-item similarity model Item-to-item collaborative filtering (people who buy item “i” also buy item “j”) is an algorithm popularised by Amazon’s recommender system. Can use same similarity metrics and prediction functions as user-user model. Item to item collaborative filtering works much better than user-user Collaborative Filtering. It is easy to explain to others but not as accurate and doesn’t scale well. We must consider the following biases : Normalisation Bias user bias (score of grouchiness or mood of customer that day) item bias (some items have much higher rating) time bias (movies get more popular with time as only people who like those movies will watch and rate them) Demographic Recommendation System A demographic recommender provides recommendations based on a demographic profile of the user. Recommended products can be produced for different demographic niches, by combining the ratings of users in those niches. Knowledge-based Recommendation System A knowledge-based recommender suggests products based on inferences about a user’s needs and preferences. This knowledge will sometimes contain explicit functional knowledge about how certain product features meet user needs. Hybrid Recommender Systems Hybrid approaches can be implemented in several ways: by making content-based and collaborative-based predictions separately and then combining them; by adding content-based capabilities to a collaborative-based approach (and vice versa); or by unifying the approaches into one model. Comparison of Recommendation Systems     PRO CONS Collaborative Filtering Works on any item Cold start problem: New items have no ratings, new users have no history. Need enough users in a system to find a match. These systems often require a large amount of existing data on a user in order to make accurate recommendations. No feature selection is needed for complex systems like images, movies, music etc. It does not rely on machine analyzable content and therefore it is capable of accurately recommending complex items such as movies without requiring an "understanding" of the item itself. Sparsity: If the matrix is sparse, it is hard to find users who rate the same items. For example, the number of items sold on major e-commerce sites is extremely large. The most active users would only have rated a small subset of the overall database. Thus, even the most popular items have very few ratings.   First rater: Can not recommend an unrated item, new items.    Popularity bias: This system tends to recommend popular items   Scalability: In many of the environments in which these systems make recommendations, there are millions of users and products. Thus, a large amount of computation power is often necessary to calculate recommendations. Content Based No need for other user's data. Finding appropriate features is hard in complex systems like images, movies, and music. Able to make recommendations for users with unique tastes. Overspecialisation. Able to recommend new and unpopular items (no first rater problem). Never recommends outside user's content profile. We know why a user is being recommended that item. People may have varied interests.   Unable to exploit quality judgements of other users.   Cold start problem (what to do with new users with few ratings?) for new users as there is no user profile for new users. Knowledge-based   Knowledge engineering bottleneck Hybrid Provides more accurate recommendations than pure approaches.   Used to overcome some of the common problems in recommender systems such as cold start and the sparsity problem.   Evaluation Evaluation is important in assessing the effectiveness of recommendation algorithms. The commonly used metrics are the mean squared error and root mean squared error or use "prediction at top k". The information retrieval metrics such as precision and recall or DCG are useful to assess the quality of a recommendation method. Recently, diversity, novelty, and coverage are also considered as important aspects in evaluation. This blog is also available here. References https://en.wikipedia.org/wiki/Recommender_system https://www.youtube.com/watch?v=gCaOa3W9kM0 https://www.youtube.com/watch?v=1JRrCEgiyHM https://www.youtube.com/watch?v=2uxXPzm-7FY https://www.youtube.com/watch?v=9siFuMMHNIA https://www.youtube.com/watch?v=h9gpufJFF-0 https://www.youtube.com/watch?v=6BTLobS7AU8 https://www.youtube.com/watch?v=VZKMyTaLI00 https://www.youtube.com/watch?v=E8aMcwmqsTg https://www.youtube.com/watch?v=GGWBMg0i9d4

A recommender system or a recommendation system seeks to predict the "rating" or "preference" a user would give to an item. The system recommends users certain items that they think the user may...

Simple Neural Network Model using Keras and Grid Search HyperParametersTuning

In this blog, I have explored using Keras and GridSearch and how we can automatically run different Neural Network models by tuning  hyperparameters (like epoch, batch sizes etc.). I have used Jupyter Notebook for development. Data set is UCI Cerdit Card Dataset which is available in csv format. Download the dataset from Kaggle https://www.kaggle.com/uciml/default-of-credit-card-clients-dataset  Description of fields is in https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients First step is to import relevant packages and load CSV file contents into dataframe. Check the shape of the data frame and inspect the column names. The output shown below Dataset has 30000 records and 25 columns. We know that "ID" column is not relevant for modelling so we can remove it. We have put rest of the columns into an array called "X". The output is in column name "default.payment.next.month" so save it in variable called "Y". We have a function to create a model. For now I have used simple parameters. But we can fine tune it by adding more layers etc.   then we create a model and try to set some parameters like epoch, batch_size in the Grid Search. As we can see from the output window that above various combinations of epoch and batch_sizes were run. For now I have kept epoch very small because it was taking time. We should test higher values also. We try to figure out when we get the best scores As you can see in the output given above the best score we got was when we use epoch 1 and batch size of 5000. We can try different parameters like different values of activation functions, momentum, learning rates, drop out rates, weight constraints, number of neurons, initializers, optimizer functions. Full code is available here Simplest kera code for MNIST dataset is here Coding is very simple and easier if you use keras package. I have posted this blog here as well. References https://github.com/keras-team/keras/blob/master/examples/mnist_cnn.py https://machinelearningmastery.com/grid-search-hyperparameters-deep-learning-models-python-keras/ https://keras.io/ http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html

In this blog, I have explored using Keras and GridSearch and how we can automatically run different Neural Network models by tuning  hyperparameters (like epoch, batch sizes etc.). I have used Jupyter...

Finding similarity between text documents

I have tried using NLTK package in python to find similarity between two or more text documents.  One common use case is to check all the bug reports on a product to see if two bug reports are duplicates. A document is characterised by a vector where the value of each dimension corresponds to the number of times that term appears in the document. Cosine similarity then gives a useful measure of how similar two documents are likely to be in terms of their subject matter. For more details on cosine similarity refer this link. So I downloaded a few bugs from https://bugzilla.mozilla.org/show_bug.cgi?id=bugid First step is to import all the relevant packages. Open a file, read all lines and the words and tokenise them. Convert words into lower case. Use Porter Stemmer to stem the words.  Stemming is the process of reducing inflected words into their word stem or root form. Like "runs", "running" get converted into it's root form "run". Remove stop words like "a", "the".In natural language processing, useless words (data), are referred to as stop words. For more information on stop word removal refer this link. Then count the occurrence of each word in the document. then calculate the cosine similarity between 2 different bug reports. Here is the output which shows that Bug#599831 and Bug#1055525 are more similar than the rest of the pairs. Things to improve This is just 1-Gram analysis not taking into account of group of words. For example "core" and "dump" are read as individual words not as a single phrase "core dump". For more information on N-grams refer this link. Similar words with same meaning (like "core dump" and "crash") have not been taken into account. All the documents (bugs) are downloaded as single text file. Ideally different weights should be given to bug subject and description. There are other methods to find similarity of documents. Can try K means clustering or run linear regression on duplicate bugs Only two document comparison is being done. all word lists could be calculated from all the documents (like all bug reports). For now have considered only english documents I have blogged this in my personal blogs as well. References http://www.ling.helsinki.fi/kit/2008s/clt231/nltk-0.9.5/doc/en/book.html http://www.geeksforgeeks.org/removing-stop-words-nltk-python/ http://www.nltk.org/book/ch03.html http://masongallo.github.io/machine/learning,/python/2016/07/29/cosine-similarity.html https://en.wikipedia.org/wiki/Stemming https://en.wikipedia.org/wiki/Cosine_similarity https://www.knime.com/blog/sentiment-analysis-with-n-grams

I have tried using NLTK package in python to find similarity between two or more text documents.  One common use case is to check all the bug reports on a product to see if two bug reports are...

Analysing Credit Card default Datasets using Apache Spark and Scala

I started experimenting with Kaggle Dataset Default Payments of Credit Card Clients in Taiwan using Apache Spark and Scala. I have tried different techniques like normal Logistic Regression, Logistic Regression with Weight column, Logistic Regression with K fold cross validation, Decision trees, Random forest and Gradient Boosting to see which model is the best. On inspecting the dataset given in the study UCI_Credi_card.csv, found that the dataset is unbalanced, out of 30000 records, count of records with default.payment.next.month value is “0” is 23,364 and the rest have value “1”. If value of default.payment.next.month of 0 mean they have paid the instalment and 1 means they have defaulted. Aim of this model is to predict who will default. Data is clean and has no NA so data cleaning was not required refer this link of UCI machine learning repository for more details about the dataset. The following are categorical variables  SEX : integer : possible values 1 or 2 MARRIAGE : integer : possible values 0,1,2,3 AGE : integer : possible values from 21 to 79. Assuming it is year as per this link of UCI machine learning repository which describes this dataset, X5 is Age (year) EDUCATION : integer : possible values 0 to 6 PAY_0,2,3,4,5,6 : integer : possible values between -2 to 8 Correlation plot using R rattle. This shows that Bill Amounts have high correlation with each other and PAY_* variables have high correlation. We should remove these variables which have high multicollinearity. But for simplicity of this blog I am not going into those details will address it in future blogs.  Unlike multiple linear regression, we do not have to check for normal distribution. Install Apache Spark Download spark-2.2.0-bin-hadoop2.7.tgz from https://spark.apache.org/downloads.html Extract it. $ cd spark-2.2.0-bin-hadoop2.7 In this directory keep the csv file UCI_Credit_Card.csv and the code explained below. $./bin/spark-shell And run code manually line by line or :load code.scala Or $./bin/spark-shell -i code.scala Initial Steps First thing to do is to import all the packages Read the csv file from file UCI_Credi_card.csv. Make sure we read its headers and we will try to infer data types otherwise it will read everything as a string.  Check schema of the data frame The column “ID” is of type “integer” but should be of type “string”. Rest of the types were detected properly. Convert “ID” to string. Also rename column “default.payment.next.month" to "y".  As dataset is unbalanced, out of 30000 records, count of records with y is 0 is 23,364 and the rest are y value is 1. To weight the records,  refer this link. When y value is 1 classWeightCol is set to 0.7789 and 0.2212 when it is 0.    Convert Categorical Variables Convert categorical variables by using OneHotEncoder https://spark.apache.org/docs/latest/ml-features.html#onehotencoder. Create a StringIndexer for each column and then run OneHotEncoder on them. Pipeline both and run to get modified dataset. Check the schema again to observe that columns with “_Vec” in names are created.  Combine useful columns into a column named "features" on which models will be run. Check the schema again to see that the new column “features” is created. Split test and training data, create evaluators Split data into training (80%) and test (20%) or 70-30 ratio. Create one binary classification evaluator which looks at "rawPrediction". Metrics for this Binary Classification Evaluator is “areaUnderROC”. Regression evaluator looks at "prediction". For Regression Evaluator use metrics “rmse”. Try other metrics also like “mse” or “r2” or “mae”. Try different models like Simple Logistic Regression, Logistic Regression with Weight column, Logistic Regression with K fold cross validation, Decision trees, Random forest and Gradient Boosting Run logistic regression and print Area Under ROC and RMSE.  When we used logistic regression with weighted column, we get better Area Under ROC curve than linear regression. Try 10 fold cross validation. We get better Area under ROC than simple linear regression. Create a Parameters Grid with regulariziation parameters of 0.05, 0.1 and 0.2 and maximum iterations 5,10 and 15. Create a Decision Tree Regressor with maximum bins 32 and maximum depth 5.  Try random forest and also create a Gradient Boosting Regressor with maximum iterations 10  K fold cross validation is a very popular resampling technique to train and test model k times on different subsets of training data. For more details on K fold Cross validation this link on 7 Important Model Evaluation Error Metrics Everyone should know Output Here is the output from the program with Area Under ROC curve and RMSE for different models. Comparison of RMSE for different methods  What is RMSE and why it is important factor to decide the model ? Root-mean-square error (RMSE) is a frequently used measure of the differences between values (sample and population values) predicted by a model or an estimator and the values actually observed.  It is the square root of the mean of the square of all of the error.  RMSE <- sqrt(mean((y-yPred)^2)) For more details on Area Under ROC Curve, RMSE and K fold corss validation, refer this link on 7 Important Model Evaluation Error Metrics Everyone should know   Comparing the outputs of all the models tried above  Gradient boosting shows minimum RMSE. Will blog more about this dataset later. I have posted this at my blogging site  also References https://stackoverflow.com/questions/33372838/dealing-with-unbalanced-datasets-in-spark-mllib/38951595 Kaggle Dataset Default Payments of Credit Card Clients in Taiwan UCI dataset https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients https://spark.apache.org/docs/latest/ml-features.html#onehotencoder https://spark.apache.org/downloads.html RMSE https://en.wikipedia.org/wiki/Root-mean-square_deviation 7 Important Model Evaluation Error Metrics Everyone should know https://www.kaggle.com/wiki/RootMeanSquaredError    

I started experimenting with Kaggle Dataset Default Payments of Credit Card Clients in Taiwan using Apache Spark and Scala. I have tried different techniques like normal Logistic Regression, Logistic...

Performance Analysis of Oracle Traffic Director or Web Server

Performance Analysis of Oracle Traffic Director or Oracle iPlanet Web Server using Oracle Sun Solaris Studio Performance Analyzer and DTrace scripts. In this blog I will show how to use the Sun Studio Collector and Performance Analyzer and DTrace script to measure the performance. 1. Using Oracle Solaris Studio 12.2 Performance Analyzer with Oracle Traffic Director or Oracle iPlanet Web Server Thanx to Basant for teaching me collector and analyzer and thanx to Julien for his help on DTrace scripts. Install Sun Studio 12. Lets say is installed in /opt/SUNWSpro. 1.1 setting up ~/.er.rc cat ~/.er.rcdmetrics e.user:i.user:e!wall:i!wall:e.total:i.total:e.system:i.system:e!wait: \i!wait:e!lock:i!lock:e!text:i!text:e!data:i!data:e!owait:i!owait:!size:!address:namedsort e.userscc basic:version:warn:parallel:query:loop:pipe:inline:memops:fe:cgdcc basic:version:warn:parallel:query:loop:pipe:inline:memops:fe:cg:srcsthresh 75dthresh 75name longview usertlmode thread:root:depth=10tldata sample:clock:hwc:heaptrace:synctrace:mpitrace:msgtrace:dataracesetpath $expts:.tabs functions:callers-callees:source:disasm:timeline:header:en_desc on 1.2 Collecting Data Using the collect Command 1.2.1& Run the Collector  using the collect command # collect collect-options program program-arguments For OTD or Web Server we edit start script bin/startserv to have a new --collect option as shown below  Red color lines are the new lines I have added.  Copied all the lines in start option, replaced ${SERVER_BIN} (which is trafficd-wdog) by "trafficd". And added "collect $COLLECT_OPTS" before it. In the section below replace <profiler directory> by the directory where you want the profiler to collect data. case $COMMAND in --start|-start) ${SERVER_BIN} -d "${SERVER_CONFIG_DIR}" \ -r "${OTD_PRODUCT_HOME}" -t "${SERVER_TEMP_DIR}" \ -u "${SERVER_USER}" ${SVC_OPT} $@ STATUS=$? if [ $STATUS -ne 0 ] ; then exit $STATUS fi enable_failover ;; --collect|-collect)PATH=$PATH:/opt/SUNWspro/bin COLLECT_OPTS="-t 180-300 -F all -d <profiler directory>"; collect $COLLECT_OPTS trafficd -d "${SERVER_CONFIG_DIR}" \ -r "${OTD_PRODUCT_HOME}" -t "${SERVER_TEMP_DIR}" \ -u "${SERVER_USER}" ${SVC_OPT} $@ STATUS=$? if [ $STATUS -ne 0 ] ; then exit $STATUS fi ;; for web server instead of trafficd it will be webservd.  1.2.2 start the server using --collect option Since we replaced trafficd-wdog by trafficd, the server will start up without any watchdog process and will run in console, not in background.   bin/startserv --collect 1.2.3 run some stress tests 1.2.4 After 5-6 minutes stop the server This will create a directory called (e.g.) 'test.1.er' which contains the experiment. The default name for a new experiment is test.1.er. The Collector automatically increments n by one in the names of subsequent experiments. 1.3 Open the profile and start the Oracle Sun Studio 12.2 analyzer Set DISPLAY env. $ cd <profiler directory>$ export JAVA_PATH=/opt/SUNWwbsvr/jdk $ /opt/SUNWspro/bin/analyzer test.1.er/ 1.4 The er_print utility prints an ASCII version of the various displays supported by the Performance Analyzer. $ /opt/SUNWspro/bin/er_print -outfile er_print1.out -functions test.1.er In this, functions are sorted by "Exclusive User CPU Time" or $ /opt/SUNWspro/bin/er_print -outfile er_print2.out \-metrics e.user:i.user:e.%user:i.%user:e.total:i.total:e.%total:i.%total:name \-sort i.total -functions test.1.er In this, functions are sorted by "Inclusive Total LWP Time" You can look at these files and figure out which function is taking how much time. These files look like : Functions sorted by metric: Exclusive User CPU Time Excl.     Incl.     Excl.       Incl.       Excl.     Incl.      Name  User CPU  User CPU  Total LWP   Total LWP   Sys. CPU  Sys. CPU            sec.      sec.         sec.        sec.     sec.      sec.     851.666   851.666   105477.573  105477.573  106.525   106.525    <Total> 493.435   493.435      558.110     558.110   26.368    26.368    fn1  56.840   326.308       64.015     368.568    3.042    17.352    fn2  28.280    28.280       34.574      34.574    1.701     1.701    fn3... 2. Using DTrace script to see how much time is spent in which function #!/usr/sbin/dtrace -s#pragma D option bufsize=1g#pragma D option specsize=1g#pragma D option aggsize=1g#pragma D option dynvarsize=1gpid$target:::entry{ self->ts[probefunc] = timestamp;}pid$target:::return/self->ts[probefunc]/{ @time[probefunc, probemod] = sum(timestamp - self->ts[probefunc]); self->ts[probefunc] = 0;} Note I have given 1g sizes you can tune it as per your machine. Run this D script using : #sudo dtrace -s functime.d -p 27910 -o dtrace.log where 27910 is the pid of the process you are examining (in this case webservd or trafficd). This will generate the output as shown below  PListFindValue libns-httpd40.so 3871 getbucketnum libc.so.1 4995 R_SSL_version libnnz11.so 6106 http_format_server libns-httpd40.so 6807 void SimpleHashTemplateBase::_insert(unsigned long,hashEntry*) libns-httpd40.so 7059 long atoi64(const char*) libns-httpd40.so 7288... I wrote this wrapper script to report percentages. To get the last column I used logic of looking for spaces after the string "lib". For LM1`ld.so.1, I temporarily added a hack. #!/usr/bin/perl $logfile = "dtrace.log"; $tmpfile = "temp"; open(IN,"<",$logfile) || die "Can not open $logfile: $!"; $total = 0; $total_rounded = 0; while (<IN>) {     chomp($_);     s/(.*\s*)lib([^\s]*)\s*([0-9]*)/$1lib$2=$3/g;     my ($a, $b) = split('=');     $total += $b; } print "total = $total\n"; close(IN); open(OUT,">", $tmpfile) || die "Can not open $tmpfile: $!"; open(IN,"<",$logfile) || die "Can not open $logfile: $!"; while (<IN>) {     chomp($_);     s/LM1`ld.so.1/libld.so.1/g;     s/(.*\s*)lib([^\s]*)\s*([0-9]*)/$1lib$2=$3/g;     my ($a, $b) = split('=');     $rounded = sprintf("%.10f", ($b*100)/$total);     $a =~ s/libld.so.1/LM1-ld.so.1/g;     print OUT "$b   $rounded%   $a\n";     $total_rounded += $rounded; } close(IN); close(OUT); print "total rounded = $total_rounded\n"; `sort -n -r $tmpfile | tee $logfile.sorted`; `rm $tmpfile` This produced output of the following format : 20318357230394 16.4117840797% poll libc.so.120317702791746 16.4112554688% _pollsys libc.so.120313615393944 16.4079539474% __pollsys libc.so.12684593654698 2.1684317735% int DaemonSession::GetConnection() libns-httpd40.so... 3. Using DTrace Profile Probes  #!/usr/sbin/dtrace -s profile-1000 /pid == $1 && arg1 != NULL/ {     @proc[umod(arg1), ufunc(arg1),ustack()] = count(); } END {     printa(@proc); } run it as : dtrace -x ustackframes=20 -s profiler-probes.d <pid> -o dtrace.log It creates output in the format library name, function name, user stack and the count number of times called, newline   libc.so.1                                           libc.so.1`mutex_lock_impl                      libc.so.1`mutex_lock_impl+0xaf              libc.so.1`mutex_lock+0xb              libumem.so.1`umem_cache_alloc+0x33              libumem.so.1`umem_alloc+0xaf              libumem.so.1`malloc+0x2e              libnnz11.so`sys_malloc+0x23             1068 .... 4. References Oracle Solaris Studio 12.2: Performance Analyzer http://docs.oracle.com/cd/E18659_01/html/821-1379/ Using DTrace with Sun Studio Tools to Understand, Analyze, Debug ... http://docs.oracle.com/cd/E19205-01/820-4221/ Profile Provider https://wikis.oracle.com/display/DTrace/profile+Provider Kernel and user profiling with dtrace - Darryl Gove's Blog https://blogs.oracle.com/d/entry/kernel_and_user_profiling_with Brendan Gregg' blog https://blogs.oracle.com/brendan/entry/dtracing_off_cpu_time

Performance Analysis of Oracle Traffic Director or Oracle iPlanet Web Server using Oracle Sun Solaris Studio Performance Analyzer and DTrace scripts. In this blog I will show how to use the Sun Studio...

Configuring Server Name Indication (SNI) in Oracle Traffic Director 11.1.1.6 and 11.1.1.7

What is SNI ? It is explained very well in http://en.wikipedia.org/wiki/Server_Name_IndicationIf your SSL server needs certificate(s) for different domains, you can choose one of the different options : Use multiple certificates using SNI feature (configure server to return different certificates for different domains) - recommended Use a single certificate with SubjectAltName Extension (one hostname in CN and other hostnames in SubjectAltName extension in the certificate) Use a single certificate with wild card in subject (lets say certificate with "CN=*.*.oracle.com", so it will be valid for different domains) - not preferred Notes Unbound Virtual Server: <virtual-server> doesn't have <http-listener> as a sub element. Bound Virtual Server: <virtual-server> has a <http-listener> sub element, it is said to be bound to that http listener. To figure out which Virtual server is the Default Virtual Server for a listener, look at the Virtual Server name in <default-virtual-server> of <http-listener> in server.xml. How to configure SNI in Oracle Traffic Director In this blog I will cover the following Enable SSL on an HTTP listener and create a certificate for it. Create two Virtual Servers both bound to an HTTP listener. One of the Virtual Server contains a certificate and the other doesn't. Send SNI and non-SNI requests to those two Virtual Servers. Create and add certificate for the default Virtual Server(which could be unbound or bound) and add <host> element value of <host> of our Virtual Server which doesn't have a certificate. Send a SNI request to the virtual server which doesn't have a certificate, it returns certificate from the default virtual server. What we will find out  is If SNI host is NOT sent by the browser in SSL Handshake, then the server sends the certificate from the http listener. --------- 1 else (i.e. if SNI host is sent by the browser in SSLHandshake) If SNI Host sent by browser doesn't match with a <host> element in any of the bound Virtual server  - goto STEP 2 else (i.e. If SNI host sent by browser matches with <host> element of any bound Virtual Server) If that Virtual Server has certificate,  the server sends the certificate from the Virtual Server. ----------- 2 else (that Virtual Server DOES NOT have a certificate) - goto STEP 2 STEP 2: get the default Virtual Server for this http listener : If the default virtual Server DOES NOT have a certificate, then the server sends the cert from the http listener ------- 3 else (i.e. If the default virtual Server has a certificate) then the server sends the cert from this default Virtual Server ------- 4 Exercise for readers : If Virtual Server has certificate of only one Type either ECC or RSA,  but the http listener has two types of certs one each of ECC and RSA (this should not happen in ideal case), then the server will send Virtual Server's cert has OR http listener certificate depending on the cipher requested in SSL Handshake.  Files  Contents sni-abc.req HEAD /index.html HTTP/1.1Host: abcConnection: close  sni-anyhost.req HEAD /index.html HTTP/1.1Host: anyOtherValueConnection: close  sni-nocertvs.req HEAD /index.html HTTP/1.1Host: www.nocertvs.comConnection: close TSTCLNT="tstclnt" is NSS tool to send SSL requests to the server. 1. Install OTD2. Start the Origin Server 3. Start OTD Admin Server 4. Create self signed cert for the http listener with subject name "www.ls.com" (for easy identification) and nickname "Server-Cert" $INSTANCE_HOME/bin/tadm create-selfsigned-cert --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --server-name=www.ls.com --nickname=Server-Cert --key-type=rsa CLI201 Command 'create-selfsigned-cert' ran successfully 5. Enable SSL and set this self signed cert with nickname "Server-Cert" in the http listener $INSTANCE_HOME/bin/tadm set-ssl-prop --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --http-listener=http-listener-1 enabled=true server-cert-nickname=Server-Cert CLI201 Command 'set-ssl-prop' ran successfully 6. Create a Virtual Server VSabc with www.abc.com <host> in server.xml  and bind it to the http listener "http-listener-1" $INSTANCE_HOME/bin/tadm create-virtual-server --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --host-pattern=www.abc.com --http-listener-name=http-listener-1 --origin-server-pool-name=origin-server-pool-1 VSabc CLI201 Command 'create-virtual-server' ran successfully 7. Create self signed cert for the Virtual Server with subject "www.abc.com" and nickname "abc" $INSTANCE_HOME/bin/tadm create-selfsigned-cert --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --server-name=www.abc.com --nickname=abc --key-type=rsa Command 'create-selfsigned-cert' ran successfully 8. Set this certificate with nickname "abc" and subject "www.abc.com" in the Virtual Server "VSabc" $INSTANCE_HOME/bin/tadm set-virtual-server-prop --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --vs=VSabc server-cert-nickname=abc CLI201 Command 'set-virtual-server-prop' ran successfully 9. Create a Virtual Server VSnocertvs with "www.nocertvs.com" <host> in server.xml $INSTANCE_HOME/bin/tadm create-virtual-server --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --host-pattern=www.nocertvs.com --http-listener-name=http-listener-1 --origin-server-pool-name=origin-server-pool-1 VSnocertvs CLI201 Command 'create-virtual-server' ran successfully 10. Set the error log level to "finest" if you wish to see log messages are logged for SNI at all levels $INSTANCE_HOME/bin/tadm set-log-prop --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG log-level=finest CLI201 Command 'set-log-prop' ran successfully 11. Deploy these changes $INSTANCE_HOME/bin/tadm deploy-config --force --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd $CONFIG CLI201 Command 'deploy-config' ran successfully 12. Start the server instance $INSTANCE_HOME/bin/tadm start-instance --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG CLI204 Successfully started the server instance. Testing using tstclnt / Browser 13. Just for testing add www.abc.com  and www.nocertvs.com entries in /etc/hosts. cat /etc/hosts | grep www.abc.comcat /etc/hosts | grep www.nocertvs.comIdeally your DNS server must resolve these hosts to the same IP address we are using in OTD http listener. 14. Send a request via tstclnt with -a "www.abc.com"(sends this host in SSL handshake) and in request headers Host: "www.abc.com" - should get cert from the Virtual Server VSabc with subject  DN "CN=www.abc.com" $TSTCLNT -c y -h $HOST -d $INSTANCE_HOME/https-$CONFIG/config -n Server-Cert -o -p $TD_PORT -2 -a www.abc.com < $DEMO_DIR/sni-abc.req 15. Send a request via tstclnt with -a "www.nocertvs.com"(sends this host in SSL handshake) and in request headers Host: "www.nocertvs.com" - should get cert from the http listener with subject DN "CN=www.ls.com" as Virtual Server VSnocertvs with <host> www.nocerts.com doesn't have any certs. $TSTCLNT -c y -h $HOST -d $INSTANCE_HOME/https-$CONFIG/config -n Server-Cert -o -p $TD_PORT -2 -a www.nocertvs.com < $DEMO_DIR/sni-nocertvs.req 16. Send a NON SNI request via tstclnt i.e. WITHOUT any host in SSL Handshake - should get the cert from the http listener with subject DN "CN=www.ls.com" $TSTCLNT -c y -h $HOST -d $INSTANCE_HOME/https-$CONFIG/config -n Server-Cert -o -p $TD_PORT -2 < $DEMO_DIR/sni-anyhost.req Summary If SNI host is NOT sent by the browser in SSL Handshake, then the cert is returned from http listener. If SNI host is sent by the browser in SSLHandshake and it matches with <host> element in Virtual Server, cert is returned from that Virtual Server. If SNI host is sent by the browser in SSLHandshake and it matches <host> element in Virtual Server which doesn't have any certificates, certificate is returned from that http listener. - This gets a bit more complicated with Default virtual servers, will discuss in the next section. Advanced - Default Virtual Server tests 17. Stop the instance $INSTANCE_HOME/bin/tadm stop-instance --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG CLI205 Successfully stopped the server instance. 18. Create self signed cert with subject "www.defaultvscert.com" for the Default Virtual Server (Virtual Server in <default-virtual-server> of http-listener in server.xml i.e. in our case it is Virtual server with vs name $CONFIG) $INSTANCE_HOME/bin/tadm create-selfsigned-cert --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --server-name=www.defaultvscert.com --nickname=defaultvscert --key-type=rsa CLI201 Command 'create-selfsigned-cert' ran successfully 19. Set this certificate with subject "www.defaultvscert.com" in the Default Virtual Server (vs name $CONFIG) $INSTANCE_HOME/bin/tadm set-virtual-server-prop --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --vs=$CONFIG server-cert-nickname=defaultvscert CLI201 Command 'set-virtual-server-prop' ran successfully 20. Deploy the changes $INSTANCE_HOME/bin/tadm deploy-config --force --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd $CONFIG CLI201 Command 'deploy-config' ran successfully 21. Start the instance $INSTANCE_HOME/bin/tadm start-instance --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG CLI204 Successfully started the server instance. 22. Send a request via tstclnt with -a "www.nocertvs.com"(sends this host in SSL handshake) and in request headers Host: "www.nocertvs.com" - should get cert from default virtual server subject DN: CN=www.defaultvscert.com" $TSTCLNT -c y -h $HOST -d $INSTANCE_HOME/https-$CONFIG/config -n Server-Cert -o -p $TD_PORT -2 -a www.nocertvs.com < $DEMO_DIR/sni-nocertvs.req Summary If SNI host is sent by the browser in SSL Handshake, look for every Virtual Server bound to that http listener if it has <host> element whose value matches with it, if that VS has certs - return cert from this VS. if that VS doesnt have any certs, then get the default Virtual Server(default-virtual-server>) for this http listener(it may be bound or it may be unbound), if default VS has a certificate - return cert from this default VS else  - return the certificates form http listener. FLOW CHART OF SNI

What is SNI ? It is explained very well in http://en.wikipedia.org/wiki/Server_Name_IndicationIf your SSL server needs certificate(s) for different domains, you can choose one of the different options...

Configuring Oracle iPlanet WebServer / Oracle Traffic Director to use crypto accelerators on T4-1 servers

Configuring Oracle iPlanet Web Server / Oracle Traffic Director touse crypto accelerators on T4-1 servers Jyri had written a technical article on ConfiguringSolaris Cryptographic Framework and Sun Java System Web Server 7 onSystems With UltraSPARC T1 Processors. I tried to find out what has changed since then in T4. I have used a T4-1 SPARC system with Solaris 10. Results slightly vary for Solaris 11.  For Solaris 11, the T4 optimization was implemented inlibsoftcrypto.so while it was inpkcs11_softtoken_extra.so for Solaris 10. Overviewof T4 processors is herein this blog. Many thanx to Chi-Chang Lin and Julien for their help. 1. Install Oracle iPlanet Web Server / OracleTraffic Director.  Go to instance/config directory.  # cd/opt/oracle/webserver7/https-hostname.fqdn/config 2. List default PKCS#11 Modules #../../bin/modutil -dbdir . -listListing of PKCS #11Modules-----------------------------------------------------------1.NSS Internal PKCS #11 Moduleslots: 2 slots attachedstatus:loadedslot: NSS Internal Cryptographic Servicestoken: NSSGeneric Crypto Servicesslot: NSS User Private Key andCertificate Servicestoken: NSS Certificate DB2. RootCertslibrary name: libnssckbi.soslots: 1 slotattachedstatus: loadedslot: NSS Builtin Objectstoken:Builtin ObjectToken----------------------------------------------------------- 3. Initialize the soft token data store in the$HOME/.sunw/pkcs11_softtoken/ directory # pktool setpinkeystore=pkcs11Enter token passphrase: olderpasswordCreatenew passphrase: passwordRe-enter new passphrase:passwordPassphrase changed. 4. Offload crypto operations to Solaris CryptoFramework on T4 $../../bin/modutil -dbdir . -nocertdb -add SCF -libfile/usr/lib/libpkcs11.so -mechanisms RSA:AES:SHA1:MD5 Module "SCF"added to database. Note that -nocertdb means modutil won't try to openthe NSS softoken key database. It doesn't even have to be present. PKCS#11 library used is /usr/lib/libpkcs11.so.If the server is running in 64 bit mode, we have to use/usr/lib/64/libpkcs11.so Unlike T1 and T2, in T4 we do not have to disablemechanisms in softtoken provider using cryptoadm. 5. List again to check that a new module SCF isadded #../../bin/modutil -dbdir . -list Listing of PKCS#11Modules-----------------------------------------------------------1.NSS Internal PKCS #11 Moduleslots: 2 slots attachedstatus:loadedslot: NSS Internal Cryptographic Servicestoken: NSSGeneric Crypto Servicesslot: NSS User Private Key andCertificate Servicestoken: NSS Certificate DB2.SCFlibrary name: /usr/lib/libpkcs11.soslots: 2 slotsattachedstatus: loadedslot: Sun Metaslottoken: SunMetaslotslot: n2rng/0 SUNW_N2_Random_Number_Generator token:n2rng/0 SUNW_N2_RNG 3. Root Certs libraryname: libnssckbi.so slots: 1 slot attached status:loaded slot: NSS Builtin Objects token: BuiltinObjectToken----------------------------------------------------------- 6.  Create certificate in “Sun Metaslot” : I have used certutil, but you must useAdmin Server CLI / GUI #../../bin/certutil -S -x -n "Server-Cert" -t "CT,CT,CT"-s "CN=*.fqdn" -d . -h "Sun Metaslot"EnterPassword or Pin for "Sun Metaslot": password 7. Verify that the certificate is createdproperly in “Sun Metslaot” #../../bin/certutil -L -d . -h "Sun Metaslot"CertificateNickname Trust AttributesSSL,S/MIME,JAR/XPIEnter Password orPin for "Sun Metaslot": passwordSunMetaslot:Server-Cert CTu,Cu,Cu# 8. Associate this newly created certificateto http listener using Admin CLI/GUI. After that server.xml shouldhave <http-listener>...    <ssl>        <server-cert-nickname>SunMetaslot:Server-Cert</server-cert-nicknamer>    </ssl> Note the prefix "SunMetaslot" 9. Disable PKCS#11 bypass To use the accelerated AES algorithm,turn off PKCS#11 bypass, and configure modutil to have the AESmechanism go to the Metaslot. After you disable PKCS#11 bypasss using Admin GUI/CLI,  check that server.xml should have <server> ....    <pkcs11>         <enabled>1</enabled>        <allow-bypass>0</allow-bypass>     </pkcs11> With PKCS#11 bypass enabled,Oracle iPlanet Web Server will only use the RSA capability of theT4, provided certificate and key are stored in the T4 slot(Metaslot). Actually, the RSA op is never bypassed in NSS, it'salways done with PKCS#11 calls. So the bypass settings won't affectthe behavior of the probes for RSA at all. The only thing thatmatters if where the RSA key and certificate live, ie. which PKCS#11token, and thus which PKCS#11 module gets called to do the work. Ifyour certificate/key are in the NSS certificate/key db, you will seelibsoftokn3/libfreebl libraries doing the RSA work. Ifthey are in the Sun Metaslot, it should be the Solaris code. 10. Start the server instance #../bin/startserv Oracle iPlanet Web Server 7.0.16 B09/14/201203:33Please enter the PIN for the "Sun Metaslot" token:password...info: HTTP3072:http-listener-1: https://hostname.fqdn:80 ready to acceptrequestsinfo: CORE3274: successful server startup 11. Figure out which process to run this DTrace script on # ps -eaf | grep webservd | grep -vdogwebservd 18224 18223 013:17:25 ? 0:07 webservd -d/opt/oracle/webserver7/https-hostname.fqdn/config -r/opt/root 18225 18224 0 13:17:25 ?0:00 webservd -d/opt/oracle/webserver7/https-hostname.fqdn/config -r /opt/ (For Oracle Traffic Director look for process named "trafficd") We see that thechild process id is “18225” 12. Clients for testing : You can use any browser. I used NSS tool tstclnt for testing $cat > req.txtGET/index.html HTTP/1.0 For checking both RSA and AES, I used cipher “:0035”which is TLS_RSA_WITH_AES_256_CBC_SHA $./tstclnt -hhostname -p 80 -d . -T -f -o -v -c “:0035” < req.txt 13. How do I make sure that crypto accelerator isbeing used 13.1 Create DTrace script The following D script should be able to uncover whetherT4-specific crypto routine are being called or not. It also displaysstats per second. # cat >t4crypto.d#!/usr/sbin/dtrace-spid$target::*rsa*:entry,pid$target::*yf*:entry{    @ops[probemod,probefunc] = count();}tick-1sec{    printa(@ops);    trunc(@ops);} Invoke with './t4crypto.d -p <pid>' 13.2 EXPECTED PROBES FOR Solaris 10 : If offloading to T4 HW are correctly set up, the expected DTraceoutput would have these probes and libraries library Operations PROBES pkcs11_softtoken_extra.so RSA soft_decrypt_rsa_pkcs_decode,soft_encrypt_rsa_pkcs_encode soft_rsa_crypt_init_common soft_rsa_decrypt, soft_rsa_encrypt soft_rsa_decrypt_common, soft_rsa_encrypt_common AES yf_aes_instructions_present yf_aes_expand256, yf_aes256_cbc_decrypt, yf_aes256_cbc_encrypt, yf_aes256_load_keys_for_decrypt, yf_aes256_load_keys_for_encrypt, Notethat these are for 256, same for 128, 192... these are forcbc, same for ecb, ctr, cfb128... DES yf_des_expand, yf_des_instructions_present yf_des_encrypt libmd_psr.so MD5 yf_md5_multiblock, yf_md5_instruction_present SHA1 yf_sha1_instruction_present, yf_sha1_multibloc 13.3SAMPLE OUTPUT FOR CIPHER TLS_RSA_WITH_AES_256_CBC_SHA(0x0035)ON T4 SPARC SOLARIS 10 WITHOUT PKCS#11BYPASS # ./t4crypto.d-p 18225 pkcs11_softtoken_extra.so.1  soft_decrypt_rsa_pkcs_decode    1 pkcs11_softtoken_extra.so.1  soft_rsa_crypt_init_common      1 pkcs11_softtoken_extra.so.1  soft_rsa_decrypt                1 pkcs11_softtoken_extra.so.1  big_mp_mul_yf                   2 pkcs11_softtoken_extra.so.1  mpm_yf_mpmul                    2 pkcs11_softtoken_extra.so.1  mpmul_arr_yf                    2 pkcs11_softtoken_extra.so.1  rijndael_key_setup_enc_yf       2 pkcs11_softtoken_extra.so.1  soft_rsa_decrypt_common         2 pkcs11_softtoken_extra.so.1  yf_aes_expand256                2 pkcs11_softtoken_extra.so.1  yf_aes256_cbc_decrypt           3 pkcs11_softtoken_extra.so.1  yf_aes256_load_keys_for_decrypt 3 pkcs11_softtoken_extra.so.1  big_mont_mul_yf                 6 pkcs11_softtoken_extra.so.1  mm_yf_montmul                   6 pkcs11_softtoken_extra.so.1  yf_des_instructions_present     6 pkcs11_softtoken_extra.so.1  yf_aes256_cbc_encrypt           8 pkcs11_softtoken_extra.so.1  yf_aes256_load_keys_for_encrypt 8 pkcs11_softtoken_extra.so.1  yf_mpmul_present                8 pkcs11_softtoken_extra.so.1  yf_aes_instructions_present    13 pkcs11_softtoken_extra.so.1  yf_des_encrypt                 18 libmd_psr.so.1               yf_md5_multiblock              41 libmd_psr.so.1               yf_md5_instruction_present     72 libmd_psr.so.1               yf_sha1_instruction_present    82 libmd_psr.so.1               yf_sha1_multiblock             82 This indicatesthat both RSA and AES ops are done in Solaris Crypto Framework. 13.4SAMPLE OUTPUT FOR CIPHER TLS_RSA_WITH_AES_256_CBC_SHA(0x0035)ON T4 SPARC SOLARIS 10 WITHPKCS#11 BYPASS #./t4crypto.d -p 18225 pkcs11_softtoken_extra.so.1  soft_decrypt_rsa_pkcs_decode 1 pkcs11_softtoken_extra.so.1  soft_rsa_crypt_init_common   1 pkcs11_softtoken_extra.so.1  soft_rsa_decrypt             1 pkcs11_softtoken_extra.so.1  soft_rsa_decrypt_common      1 pkcs11_softtoken_extra.so.1  big_mp_mul_yf                2 pkcs11_softtoken_extra.so.1  mpm_yf_mpmul                 2 pkcs11_softtoken_extra.so.1  mpmul_arr_yf                 2 pkcs11_softtoken_extra.so.1  big_mont_mul_yf              6 pkcs11_softtoken_extra.so.1  mm_yf_montmul                6 pkcs11_softtoken_extra.so.1  yf_mpmul_present             8 For this cipher,when I enable PKCS#11 bypass, Only RSA probes are being hit AESprobes are not being hit. 13.5ustack() for RSA operations /probefunc == "soft_rsa_decrypt" / Shows thatlibnss3.so is calling C_*functions of libpkcs11.sowhich is calling functions of pkcs11_softtoken_extra.sofor both cases with and withoutbypass. When PKCS#11 bypass is disabled(allow-bypass is 0) pkcs11_softtoken_extra.so.1`soft_rsa_decrypt pkcs11_softtoken_extra.so.1`soft_rsa_decrypt_common+0x94 pkcs11_softtoken_extra.so.1`soft_unwrapkey+0x258 pkcs11_softtoken_extra.so.1`C_UnwrapKey+0x1ec libpkcs11.so.1`meta_unwrap_key+0x17c libpkcs11.so.1`meta_UnwrapKey+0xc4 libpkcs11.so.1`C_UnwrapKey+0xfc libnss3.so`pk11_AnyUnwrapKey+0x6b8 libnss3.so`PK11_PubUnwrapSymKey+0x8c libssl3.so`ssl3_HandleRSAClientKeyExchange+0x1a0 libssl3.so`ssl3_HandleClientKeyExchange+0x154 libssl3.so`ssl3_HandleHandshakeMessage+0x440 libssl3.so`ssl3_HandleHandshake+0x11c libssl3.so`ssl3_HandleRecord+0x5e8 libssl3.so`ssl3_GatherCompleteHandshake+0x5c libssl3.so`ssl_GatherRecord1stHandshake+0x30 libssl3.so`ssl_Do1stHandshake+0xec libssl3.so`ssl_SecureRecv+0x1c8 libssl3.so`ssl_Recv+0x9c libns-httpd40.so`__1cNDaemonSessionDrun6M_v_+0x2dc When PKCS#11 bypass is enabled(allow-bypass is 1) pkcs11_softtoken_extra.so.1`soft_rsa_decrypt pkcs11_softtoken_extra.so.1`soft_rsa_decrypt_common+0x94 pkcs11_softtoken_extra.so.1`C_Decrypt+0x164 libpkcs11.so.1`meta_do_operation+0x27c libpkcs11.so.1`meta_Decrypt+0x4c libpkcs11.so.1`C_Decrypt+0xcc libnss3.so`PK11_PrivDecryptPKCS1+0x1ac libssl3.so`ssl3_HandleRSAClientKeyExchange+0xe4 libssl3.so`ssl3_HandleClientKeyExchange+0x154 libssl3.so`ssl3_HandleHandshakeMessage+0x440 libssl3.so`ssl3_HandleHandshake+0x11c libssl3.so`ssl3_HandleRecord+0x5e8 libssl3.so`ssl3_GatherCompleteHandshake+0x5c libssl3.so`ssl_GatherRecord1stHandshake+0x30 libssl3.so`ssl_Do1stHandshake+0xec libssl3.so`ssl_SecureRecv+0x1c8 libssl3.so`ssl_Recv+0x9c libns-httpd40.so`__1cNDaemonSessionDrun6M_v_+0x2dc libnsprwrap.so`ThreadMain+0x1c libnspr4.so`_pt_root+0xe8 13.6ustack() FOR AES operations /probefunc == "yf_aes256_cbc_encrypt" / When PKCS#11bypass is disabled (allow-bypass is 0) pkcs11_softtoken_extra.so.1`yf_aes256_cbc_encrypt pkcs11_softtoken_extra.so.1`aes_block_process_contiguous_whole_blocks+0xb4 pkcs11_softtoken_extra.so.1`aes_crypt_contiguous_blocks+0x1cc pkcs11_softtoken_extra.so.1`soft_aes_encrypt_common+0x22c pkcs11_softtoken_extra.so.1`C_EncryptUpdate+0x10c libpkcs11.so.1`meta_do_operation+0x1fc libpkcs11.so.1`meta_EncryptUpdate+0x4c libpkcs11.so.1`C_EncryptUpdate+0xcc libnss3.so`PK11_CipherOp+0x1a0 libssl3.so`ssl3_CompressMACEncryptRecord+0x264 libssl3.so`ssl3_SendRecord+0x300 libssl3.so`ssl3_FlushHandshake+0x54 libssl3.so`ssl3_SendFinished+0x1fc libssl3.so`ssl3_HandleFinished+0x314 libssl3.so`ssl3_HandleHandshakeMessage+0x4ac libssl3.so`ssl3_HandleHandshake+0x11c libssl3.so`ssl3_HandleRecord+0x5e8 libssl3.so`ssl3_GatherCompleteHandshake+0x5c libssl3.so`ssl_GatherRecord1stHandshake+0x30 libssl3.so`ssl_Do1stHandshake+0xec Shows thatlibnss3.so is calling C_* functions of libpkcs11.so which is callingfunctions of pkcs11_softtoken_extra.so However when PKCS#11bypass is disabled (allow-bypass is 1) thisstack isn't getting called. 14.LIST OF ALL THE PROBES MATCHED BY D SCRIPT FOR REFERENCE # ./t4crypto.d-p 18225 -l ID PROVIDERMODULE FUNCTION NAME ... 55720 pid18225libmd_psr.so.1 yf_md5_instruction_present entry 55721 pid18225libmd_psr.so.1 yf_sha256_instruction_present entry 55722 pid18225libmd_psr.so.1 yf_sha512_instruction_present entry 55723 pid18225libmd_psr.so.1 yf_sha1_instruction_present entry 55724 pid18225libmd_psr.so.1 yf_sha256 entry 55725 pid18225libmd_psr.so.1 yf_sha256_multiblock entry 55726 pid18225libmd_psr.so.1 yf_sha512 entry 55727 pid18225libmd_psr.so.1 yf_sha512_multiblock entry 55728 pid18225libmd_psr.so.1 yf_sha1 entry 55729 pid18225libmd_psr.so.1 yf_sha1_multiblock entry 55730 pid18225libmd_psr.so.1 yf_md5 entry 55731 pid18225libmd_psr.so.1 yf_md5_multiblock entry 55732pid18225 pkcs11_softtoken_extra.so.1 yf_aes_instructions_presententry 55733 pid18225pkcs11_softtoken_extra.so.1 rijndael_key_setup_enc_yf entry 55734pid18225 pkcs11_softtoken_extra.so.1 yf_aes_expand128 entry 55735pid18225 pkcs11_softtoken_extra.so.1 yf_aes_encrypt128 entry 55736pid18225 pkcs11_softtoken_extra.so.1 yf_aes_decrypt128 entry 55737pid18225 pkcs11_softtoken_extra.so.1 yf_aes_expand192 entry 55738pid18225 pkcs11_softtoken_extra.so.1 yf_aes_encrypt192 entry 55739pid18225 pkcs11_softtoken_extra.so.1 yf_aes_decrypt192 entry 55740pid18225 pkcs11_softtoken_extra.so.1 yf_aes_expand256 entry 55741pid18225 pkcs11_softtoken_extra.so.1 yf_aes_encrypt256 entry 55742pid18225 pkcs11_softtoken_extra.so.1 yf_aes_decrypt256 entry 55743pid18225 pkcs11_softtoken_extra.so.1 yf_aes128_load_keys_for_encryptentry 55744pid18225 pkcs11_softtoken_extra.so.1 yf_aes192_load_keys_for_encryptentry 55745pid18225 pkcs11_softtoken_extra.so.1 yf_aes256_load_keys_for_encryptentry 55746pid18225 pkcs11_softtoken_extra.so.1 yf_aes128_ecb_encrypt entry 55747pid18225 pkcs11_softtoken_extra.so.1 yf_aes192_ecb_encrypt entry 55748pid18225 pkcs11_softtoken_extra.so.1 yf_aes256_ecb_encrypt entry 55749pid18225 pkcs11_softtoken_extra.so.1 yf_aes128_cbc_encrypt entry 55750pid18225 pkcs11_softtoken_extra.so.1 yf_aes192_cbc_encrypt entry 55751pid18225 pkcs11_softtoken_extra.so.1 yf_aes256_cbc_encrypt entry 55752pid18225 pkcs11_softtoken_extra.so.1 yf_aes128_ctr_crypt entry 55753pid18225 pkcs11_softtoken_extra.so.1 yf_aes192_ctr_crypt entry 55754pid18225 pkcs11_softtoken_extra.so.1 yf_aes256_ctr_crypt entry 55755pid18225 pkcs11_softtoken_extra.so.1 yf_aes128_cfb128_encrypt entry 55756pid18225 pkcs11_softtoken_extra.so.1 yf_aes192_cfb128_encrypt entry 55757pid18225 pkcs11_softtoken_extra.so.1 yf_aes256_cfb128_encrypt entry 55758pid18225 pkcs11_softtoken_extra.so.1 yf_aes128_load_keys_for_decryptentry 55759pid18225 pkcs11_softtoken_extra.so.1 yf_aes192_load_keys_for_decryptentry 55760pid18225 pkcs11_softtoken_extra.so.1 yf_aes256_load_keys_for_decryptentry 55761pid18225 pkcs11_softtoken_extra.so.1 yf_aes128_ecb_decrypt entry 55762pid18225 pkcs11_softtoken_extra.so.1 yf_aes192_ecb_decrypt entry 55763pid18225 pkcs11_softtoken_extra.so.1 yf_aes256_ecb_decrypt entry 55764pid18225 pkcs11_softtoken_extra.so.1 yf_aes128_cbc_decrypt entry 55765pid18225 pkcs11_softtoken_extra.so.1 yf_aes192_cbc_decrypt entry 55766pid18225 pkcs11_softtoken_extra.so.1 yf_aes256_cbc_decrypt entry 55767pid18225 pkcs11_softtoken_extra.so.1 yf_aes128_cfb128_decrypt entry 55768pid18225 pkcs11_softtoken_extra.so.1 yf_aes192_cfb128_decrypt entry 55769pid18225 pkcs11_softtoken_extra.so.1 yf_aes256_cfb128_decrypt entry 55771 pid18225pkcs11_softtoken_extra.so.1 yf_des_instructions_present entry 55772 pid18225pkcs11_softtoken_extra.so.1 yf_des_expand entry 55773 pid18225pkcs11_softtoken_extra.so.1 yf_des_encrypt entry 55774 pid18225pkcs11_softtoken_extra.so.1 yf_mpmul_present entry 55775 pid18225pkcs11_softtoken_extra.so.1 yf_montmul_present entry 55776 pid18225pkcs11_softtoken_extra.so.1 mm_yf_montmul entry 55777 pid18225pkcs11_softtoken_extra.so.1 mm_yf_montsqr entry 55778 pid18225pkcs11_softtoken_extra.so.1 mm_yf_restore_func entry 55779 pid18225pkcs11_softtoken_extra.so.1 mm_yf_ret_from_mont_func entry 55780 pid18225pkcs11_softtoken_extra.so.1 mm_yf_execute_slp entry 55781 pid18225pkcs11_softtoken_extra.so.1 big_modexp_ncp_yf entry 55782 pid18225pkcs11_softtoken_extra.so.1 big_mont_mul_yf entry 55783 pid18225pkcs11_softtoken_extra.so.1 mpmul_arr_yf entry 55784 pid18225pkcs11_softtoken_extra.so.1 big_mp_mul_yf entry 55785 pid18225pkcs11_softtoken_extra.so.1 mpm_yf_mpmul entry 55786 pid18225libns-httpd40.so nsapi_rsa_set_priv_fn entry ... 55795 pid18225libnss3.so prepare_rsa_priv_key_export_for_asn1 entry 55796 pid18225libresolv.so.2 sunw_dst_rsaref_init entry 55797 pid18225libnssutil3.so NSS_Get_SEC_UniversalStringTemplate entry ... 55813pid18225 libsoftokn3.so prepare_low_rsa_priv_key_for_asn1 entry 55814pid18225 libsoftokn3.so rsa_FormatOneBlock entry 55815pid18225 libsoftokn3.so rsa_FormatBlock entry 55816pid18225 libnssdbm3.so lg_prepare_low_rsa_priv_key_for_asn1 entry 55817pid18225 libfreebl_32fpu_3.so rsa_build_from_primes entry 55818pid18225 libfreebl_32fpu_3.so rsa_is_prime entry 55819pid18225 libfreebl_32fpu_3.so rsa_get_primes_from_exponents entry 55820pid18225 libfreebl_32fpu_3.so rsa_PrivateKeyOpNoCRT entry 55821pid18225 libfreebl_32fpu_3.so rsa_PrivateKeyOpCRTNoCheck entry 55822pid18225 libfreebl_32fpu_3.so rsa_PrivateKeyOpCRTCheckedPubKey entry 55823pid18225 pkcs11_kernel.so.1 key_gen_rsa_by_value entry 55824pid18225 pkcs11_kernel.so.1 get_rsa_private_key entry 55825pid18225 pkcs11_kernel.so.1 get_rsa_public_key entry 55826pid18225 pkcs11_softtoken_extra.so.1 soft_rsa_encrypt entry 55827pid18225 pkcs11_softtoken_extra.so.1 soft_rsa_decrypt entry 55828pid18225 pkcs11_softtoken_extra.so.1 soft_rsa_crypt_init_common entry 55829pid18225 pkcs11_softtoken_extra.so.1 soft_rsa_encrypt_common entry 55830pid18225 pkcs11_softtoken_extra.so.1 soft_rsa_decrypt_common entry 55831pid18225 pkcs11_softtoken_extra.so.1 soft_rsa_sign_verify_init_commonentry 55832pid18225 pkcs11_softtoken_extra.so.1 soft_rsa_sign_common entry 55833pid18225 pkcs11_softtoken_extra.so.1 soft_rsa_verify_common entry 55834pid18225 pkcs11_softtoken_extra.so.1 generate_rsa_key entry 55835pid18225 pkcs11_softtoken_extra.so.1 soft_rsa_genkey_pair entry 55836pid18225 pkcs11_softtoken_extra.so.1 get_rsa_sha1_prefix entry 55837pid18225 pkcs11_softtoken_extra.so.1 soft_rsa_digest_sign_commonentry 55838pid18225 pkcs11_softtoken_extra.so.1 soft_rsa_digest_verify_commonentry 55839pid18225 pkcs11_softtoken_extra.so.1 soft_rsa_verify_recover entry 55840pid18225 pkcs11_softtoken_extra.so.1 rsa_pri_to_asn1 entry 55841pid18225 pkcs11_softtoken_extra.so.1 asn1_to_rsa_pri entry 55842pid18225 pkcs11_softtoken_extra.so.1 soft_encrypt_rsa_pkcs_encodeentry 55843pid18225 pkcs11_softtoken_extra.so.1 soft_decrypt_rsa_pkcs_decodeentry 55844pid18225 pkcs11_softtoken_extra.so.1 soft_sign_rsa_pkcs_encode entry 55845pid18225 pkcs11_softtoken_extra.so.1 soft_verify_rsa_pkcs_decodeentry 55770 profiletick-1sec

Configuring Oracle iPlanet Web Server / Oracle Traffic Director to use crypto accelerators on T4-1 servers Jyri had written a technical article on ConfiguringSolaris Cryptographic Framework and Sun...

Interesting articles and blogs on SPARC T4

Interesting articles and blogs on SPARC T4 processor I have consolidated all the interesting information I could get on SPARC T4 processor and its hardware cryptographic capabilities.  Hope its useful. 1. Advantages of SPARC T4 processor Most important points in this T4 announcement are : "The SPARC T4 processor was designed from the ground up for high speed security and has a cryptographic stream processing unit (SPU) integrated directly into each processor core. These accelerators support 16 industry standard security ciphers and enable high speed encryption at rates 3 to 5 times that of competing processors. By integrating encryption capabilities directly inside the instruction pipeline, the SPARC T4 processor eliminates the performance and cost barriers typically associated with secure computing and makes it possible to deliver high security levels without impacting the user experience." Data Sheet has more details on these : "New on-chip Encryption Instruction Accelerators with direct non-privileged support for 16 industry-standard cryptographic algorithms plus random number generation in each of the eight cores: AES, Camellia, CRC32c, DES, 3DES, DH, DSA, ECC, Kasumi, MD5, RSA, SHA-1, SHA-224, SHA-256, SHA-384, SHA-512" I ran "isainfo -v" command on Solaris 11 Sparc T4-1 system. It shows the new instructions as expected : $ isainfo -v 64-bit sparcv9 applications crc32c cbcond pause mont mpmul sha512 sha256 sha1 md5 camellia kasumi des aes ima hpc vis3 fmaf asi_blk_init vis2 vis popc 32-bit sparc applications   crc32c cbcond pause mont mpmul sha512 sha256 sha1 md5 camellia kasumi des aes ima hpc vis3 fmaf asi_blk_init vis2 vis popc v8plus div32 mul32 2. Dan Anderson's Blog have some interesting points about how these can be used : "New T4 crypto instructions include: aes_kexpand0, aes_kexpand1, aes_kexpand2,  aes_eround01, aes_eround23, aes_eround01_l, aes_eround_23_l, aes_dround01, aes_dround23, aes_dround01_l, aes_dround_23_l. Having SPARC T4 hardware crypto instructions is all well and good, but how do we access it ? The software is available with Solaris 11 and is used automatically if you are running Solaris a SPARC T4.  It is used internally in the kernel through kernel crypto modules. It is available in user space through the PKCS#11 library." 3. Dan Anderson's Blog on Where's the Crypto Libraries? Although this was written in 2009 but still is very useful "Here's a brief tour of the major crypto libraries shown in the digraph: The libpkcs11 library contains the PKCS#11 API (C_\*() functions, such as C_Initialize()). That in turn calls library pkcs11_softtoken or pkcs11_kernel, for userland or kernel crypto providers. The latter is used mostly for hardware-assisted cryptography (such as n2cp for Niagara2 SPARC processors), as that is performed more efficiently in kernel space with the "kCF" module (Kernel Crypto Framework). Additionally, for Solaris 10, strong crypto algorithms were split off in separate libraries, pkcs11_softtoken_extra libcryptoutil contains low-level utility functions to help implement cryptography. libsoftcrypto (OpenSolaris and Solaris Nevada only) implements several symmetric-key crypto algorithms in software, such as AES, RC4, and DES3, and the bignum library (used for RSA). libmd implements MD5, SHA, and SHA2 message digest algorithms" 4. Dan Anderson's Blog on  How to tell if SPARC T4 crypto is being used? 5. Difference in T3 and T4 Diagram in this blog is good and self explanatory. Jeff's blog also highlights the differences  "The T4 servers have improved crypto acceleration, described athttps://blogs.oracle.com/DanX/entry/sparc_t4_openssl_engine.It is "just built in" so administrators no longer have to assign crypto accelerator units to domains - it "just happens". Every physical or virtual CPU on a SPARC-T4 has full access to hardware based crypto acceleration at all times. .... For completeness sake, it's worth noting that the T4 adds more crypto algorithms, and accelerates Camellia, CRC32c, and more SHA-x." 6. About performance counters In this blog, performance counters are explained : "Note that unlike T3 and before, T4 crypto doesn't require kernel modules like ncp or n2cp, there is no visibility of crypto hardware with kstats or cryptoadm. T4 does provide hardware counters for crypto operations. You can see these using cpustat: cpustat -c pic0=Instr_FGU_crypto 5 You can check the general crypto support of the hardware and OS with the command "isainfo -v". Since T4 crypto's implementation now allows direct userland access, there are no "crypto units" visible to cryptoadm.  " For more details refer Martin's blog as well. 7. How to turn off  SPARC T4 or Intel AES-NI crypto acceleration  I found this interesting blog from Darren about how to turn off  SPARC T4 or Intel AES-NI crypto acceleration. "One of the new Solaris 11 features of the linker/loader is the ability to have a single ELF object that has multiple different implementations of the same functions that are selected at runtime based on the capabilities of the machine. The alternate to this is having the application coded to call getisax(2) system call and make the choice itself. We use this functionality of the linker/loader when we build the userland libraries for the Solaris Cryptographic Framework (specifically libmd.so and libsoftcrypto.so) The Solaris linker/loader allows control of a lot of its functionality via environment variables, we can use that to control the version of the cryptographic functions we run. To do this we simply export the LD_HWCAP environment variable with values that tell ld.so.1 to not select the HWCAP section matching certain features even if isainfo says they are present. This will work for consumers of the Solaris Cryptographic Framework that use the Solaris PKCS#11 libraries or use libmd.so interfaces directly. For SPARC T4 : export LD_HWCAP="-aes -des -md5 -sha256 -sha512 -mont -mpul" .. For Intel systems with AES-NI support: export LD_HWCAP="-aes"" Note that LD_HWCAP is explained in http://docs.oracle.com/cd/E23823_01/html/816-5165/ld.so.1-1.html "LD_HWCAP, LD_HWCAP_32, and LD_HWCAP_64 -  Identifies an alternative hardware capabilities value... A “-” prefix results in the capabilities that follow being removed from the alternative capabilities." 8. Whitepaper on High Performance Security For Oracle Database and Fusion Middleware Applications using SPARC T4 This whitepaper on "High Performance Security For Oracle Database and Fusion Middleware Applications using SPARC T4 explains more details. It has DTrace scripts which may come in handy : "To ensure the hardware-assisted cryptographic acceleration is configured to use and working with the security scenarios, it is recommended to use the following Solaris DTrace script. " #!/usr/sbin/dtrace-s pid$target::*rsa*:entry, pid$target::*yf*:entry {     @ops[probemod,probefunc] = count(); } tick-1sec {     printa(@ops);     trunc(@ops); } Note that I have slightly modified the D Script to have *rsa* and to make it work for both Solaris 10 and 11  as per recommendations from Chi-Chang Lin. For Solaris 11, the T4 optimization is implemented in libsoftcrypto.so while it is in pkcs11_softtoken_extra.so for Solaris 10. So just add these two probes for Solaris 10 : 9. References SPARC T4 announcement http://www.oracle.com/us/corporate/features/sparc-t4-announcement-494846.html   SPARC T4-1 SERVER Data Sheet http://www.oracle.com/us/products/servers-storage/servers/sparc-enterprise/t-series/sparc-t4-1-ds-487858.pdf High Performance Security For Oracle Database and Fusion Middleware Applications using SPARC T4 http://www.oracle.com/technetwork/server-storage/sun-sparc-enterprise/documentation/o12-021-t4security-1577047.pdfDan Anderson's blogs :SPARC T4 OpenSSL Engine  https://blogs.oracle.com/DanX/entry/sparc_t4_openssl_engineWhere's the Crypto Libraries? https://blogs.oracle.com/DanX/entry/where_s_the_crypto_libraries How to tell if SPARC T4 crypto is being used? https://blogs.oracle.com/DanX/entry/how_to_tell_if_sparc Darren Moffet's blog on  "HOWTO Turn off SPARC T4 or Intel AES-NI crypto acceleration https://blogs.oracle.com/darren/entry/howto_turn_off_sparc_t4 ld.so.1 man page  http://docs.oracle.com/cd/E23823_01/html/816-5165/ld.so.1-1.html Unleash the Power of Cryptography on SPARC T4 By B.Koch https://blogs.oracle.com/hardware/entry/unleash_the_power_of_cryptographyT4 Crypto Cheat Sheet By Stefan Hinker https://blogs.oracle.com/cmt/entry/t4_crypto_cheat_sheetT4 Performance Counters explained By Martin Müller https://blogs.oracle.com/martinm/entry/t4_performance_counters_explained What happened to the MAUs on T4? By Jeff  https://blogs.oracle.com/jsavit/entry/no_mau_required_on_a

Interesting articles and blogs on SPARC T4 processor I have consolidated all the interesting information I could get on SPARC T4 processor and its hardware cryptographic capabilities.  Hope its useful. 1...

How to export ECC key and Cert from NSS DB and import into JKS keystore and Oracle Wallet

How to export ECC key and Cert from NSS DB and import into JKS keystore and Oracle Wallet In this blog I will write about how to extract a cert and key from NSS Db and import it to a JKS Keystore and then import that JKS Keystore into Oracle Wallet. 1. Set Java Home I pointed it to JRE 1.6.0_22 $ export JAVA_HOME=/usr/java/jre1.6.0_22/ 2. Create a self signed ECC cert in NSS DBI created NSS DB with self signed ECC certificate. If you already have NSS Db with ECC cert (and key) skip this step. $export NSS_DIR=/export/home/nss/ $$NSS_DIR/certutil -N -d . $$NSS_DIR/certutil -S -x -s "CN=test,C=US" -t "C,C,C" -n ecc-cert -k ec -q nistp192 -d . 3. Export ECC cert and key using pk12util Use NSS tool pk12util to export this cert and key into a p12 file      $$NSS_DIR/pk12util -o ecc-cert.p12 -n ecc-cert -d . -W password 4. Use keytool to create JKS keystore and import this p12 file 4.1 Import p12 file created above into a JKS keystore $JAVA_HOME/bin/keytool -importkeystore -srckeystore ecc-cert.p12 -srcstoretype PKCS12 -deststoretype JKS -destkeystore ecc.jks -srcstorepass password -deststorepass password -srcalias ecc-cert -destalias ecc-cert -srckeypass password -destkeypass password -v But if an error as shown is encountered, keytool error: java.security.UnrecoverableKeyException: Get Key failed: EC KeyFactory not availablejava.security.UnrecoverableKeyException: Get Key failed: EC KeyFactory not available        at com.sun.net.ssl.internal.pkcs12.PKCS12KeyStore.engineGetKey(Unknown Source)        at java.security.KeyStoreSpi.engineGetEntry(Unknown Source)        at java.security.KeyStore.getEntry(Unknown Source)        at sun.security.tools.KeyTool.recoverEntry(Unknown Source)        at sun.security.tools.KeyTool.doImportKeyStoreSingle(Unknown Source)        at sun.security.tools.KeyTool.doImportKeyStore(Unknown Source)        at sun.security.tools.KeyTool.doCommands(Unknown Source)        at sun.security.tools.KeyTool.run(Unknown Source)        at sun.security.tools.KeyTool.main(Unknown Source)Caused by: java.security.NoSuchAlgorithmException: EC KeyFactory not available        at java.security.KeyFactory.<init>(Unknown Source)        at java.security.KeyFactory.getInstance(Unknown Source)        ... 9 more 4.2 Create a new PKCS11 provider If you didn't get an error as shown above skip this step. Since we already have NSS libraries built with ECC, we can create a new PKCS11 provider Create ${java.home}/jre/lib/security/nss.cfg as follows: name = NSS     nssLibraryDirectory = ${nsslibdir}    nssDbMode = noDb    attributes = compatibility where nsslibdir should contain NSS libs with ECC support. Add the following line to ${java.home}/jre/lib/security/java.security :     security.provider.9=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg Note that those who are using Oracle iPlanet Web Server or Oracle Traffic Director, NSS libs built with ECC are in <ws_install_dir>/lib or <otd_install_dir>/lib. 4.3. Now keytool should work Now you can try the same keytool command and see that it succeeds : $JAVA_HOME/bin/keytool -importkeystore -srckeystore ecc-cert.p12 -srcstoretype PKCS12 -deststoretype JKS -destkeystore ecc.jks -srcstorepass password -deststorepass password -srcalias ecc-cert -destalias ecc-cert -srckeypass password -destkeypass password -v [Storing ecc.jks] 5. Convert JKS keystore into an Oracle Wallet You can export this cert and key from JKS keystore and import it into an Oracle Wallet if you need using orapki tool as shown below. Make sure that orapki you use supports ECC. Also for ECC you MUST use "-jsafe" option. $ orapki wallet create -pwd password  -wallet .  -jsafe $ orapki wallet jks_to_pkcs12 -wallet . -pwd password -keystore ecc.jks -jkspwd password -jsafe AS $orapki wallet display -wallet . -pwd welcome1  -jsafeOracle PKI Tool : Version 11.1.2.0.0Copyright (c) 2004, 2012, Oracle and/or its affiliates. All rights reserved.Requested Certificates:User Certificates:Subject:        CN=test,C=USTrusted Certificates:Subject:        OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=USSubject:        CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=USSubject:        OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=USSubject:        OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=USSubject:        CN=test,C=US As you can see our ECC cert in the wallet. You can follow the same steps for RSA certs as well. 6. References http://icedtea.classpath.org/bugzilla/show_bug.cgi?id=356 http://old.nabble.com/-PATCH-FOR-REVIEW-%3A-Support-PKCS11-cryptography-via-NSS-p25282932.html http://www.mozilla.org/projects/security/pki/nss/tools/pk12util.html

How to export ECC key and Cert from NSS DB and import into JKS keystore and Oracle Wallet In this blog I will write about how to extract a cert and key from NSS Db and import it to a JKS Keystore and...

Using tshark to debug SSL connections

Jyri had explained in his blog how to use ssldump to debug SSL connections. We can also use tshark. On my Linux server, tshark is installed in /usr/sbin/tshark. Support team guys need these steps for finding out what is happening. First try to reproduce the problem in a test environment with self-signed certificate and follow the steps given in this blog. I started Oracle iPlanet Web Server 7.0 instance on IP lets say 11.111.111.111 and port 15000. Exporting Private Key from NSS DB In NSS Database, I have a Server Certificate named "Server-Cert" as shown below. $ cd <WS_install-root>/https-<instance>/config $ ../../bin/certutil -L -d .Certificate Nickname                         Trust Attributes                                             SSL,S/MIME,JAR/XPIServer-Cert                                     u,u,u First use pk12util to extract server certificate and its key into a file "server.keycert". $ ../../bin/pk12util -o server.keycert -n "Server-Cert" -d . Enter Password or Pin for "NSS Certificate DB": nssdbpassword Enter password for PKCS12 file: pkcs12passwordRe-enter password: pkcs12password pk12util: PKCS12 EXPORT SUCCESSFUL then I use openssl to get just the RSA private key $ openssl pkcs12 -nodes -in server.keycert -out key.pem -nocerts -nodesEnter Import Password: pkcs12passwordMAC verified OK  $ rm server.keycert If you look at the file, its contents are like : $ cat key.pemBag Attributes    friendlyName: Server-Cert    localKeyID: ...Key Attributes: <No Attributes>-----BEGIN RSA PRIVATE KEY-----...-----END RSA PRIVATE KEY----- Edit the file key.pem manually and remove the first 4 lines. Now the file starts with line "-----BEGIN RSA PRIVATE KEY-----" and end with "-----END RSA PRIVATE KEY-----" Note that we should be very careful with this key as its not so safe to leave it unprotected. You can protect it by another password if you like. I would prefer if wireshark can take NSS DB or Oracle Wallets as input directly. Running tshark Now as root run tshark $ /usr/sbin/tshark -o "ssl.desegment_ssl_records: TRUE" \ -o "ssl.desegment_ssl_application_data: TRUE" \ -o "ssl.keys_list:11.111.111.111,15000,http,key.pem" \ -o "ssl.debug_file:ssldebug.log" \ -f "tcp port 15000" \ -R "ssl" \ -V -x 2>&1 | tee tshark.log when I had not given IP address in ssl.key_list, it wasn't associating key to some of my connections. Note that I used capture filter "tcp port 15000" and display filter "ssl". I used -V to show more verbose output and I also used -x to get both hex and ASCII dumps. You can try your own options. Now send a request through a browser to https://11.111.111.1111:15000/index.html, close the browser and after a while, press control c on the window where tshark is running and kill it. Delete the private key file key.pem. ssldebug.log should have a message that says key was loaded successfully $ grep -i "private key" ssldebug.log Private key imported: KeyID ...ssl_init private key file key.pem successfully loaded Note that ssldebug.log MUST NOT contain any error messages about key not being used etc. Now look at tshark.log, look for "Secure Socket Layer" sections one such section is shown below : Secure Socket Layer  SSL Record Layer: Handshake Protocol: Client Hello      Content Type: Handshake (22)      Version: TLS 1.0 (0x0301)      Length: 168      Handshake Protocol: Client Hello          Handshake Type: Client Hello (1)          Length: 164          Version: TLS 1.0 (0x0301)          Random              gmt_unix_time: Mar  2, 2012 00:01:26.000000000              random_bytes: .......           Session ID Length: 0          Cipher Suites Length: 72          Cipher Suites (36 suites)            Cipher Suite: Unknown (0x00ff)            Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)            Cipher Suite: TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (0x0088)... In the end you can see SSL data being decrypted : Decrypted SSL data (1 bytes):0000  48                                                H  Decrypted SSL data (225 bytes):0000  54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d 0a   TTP/1.1 200 OK.....0040  46 72 69 2c 20 30 32 20 4d 61 72 20 32 30 31 32   Fri, 02 Mar 20120050  20 30 39 3a 31 32 3a 32 38 20 47 4d 54 0d 0a 4c    09:12:28 GMT..L0060  61 73 74 2d 6d 6f 64 69 66 69 65 64 3a 20 57 65   ast-modified: We0070  64 2c 20 32 39 20 46 65 62 20 32 30 31 32 20 31   d, 29 Feb 2012 10080  31 3a 33 38 3a 31 39 20 47 4d 54 0d 0a 43 6f 6e   1:38:19 GMT..Con0090  74 65 6e 74 2d 6c 65 6e 67 74 68 3a 20 31 39 0d   tent-length: 19.00a0  0a 45 74 61 67 3a 20 22 31 33 2d 34 66 34 65 30   .Etag: "13-4f4e000b0  65 32 62 22 0d 0a 41 63 63 65 70 74 2d 72 61 6e   e2b"..Accept-ran00c0  67 65 73 3a 20 62 79 74 65 73 0d 0a 0d 0a 54 68   ges: bytes....Th00d0  69 73 20 69 73 20 69 6e 64 65 78 2e 68 74 6d 6c   is is index.html00e0  0a                                                . This log shows different stages of SSL $grep "Handshake Protocol" tshark.log     Handshake Protocol: Client Hello    Handshake Protocol: Server Hello    Handshake Protocol: Certificate    Handshake Protocol: Server Hello Done    Handshake Protocol: Client Key Exchange    Handshake Protocol: Finished    Handshake Protocol: Finished Exporting Private Key from Wallet If your product uses Oracle wallet instead of NSS DB, to extract the key and certificate from the Wallet you can use openssl command as shown below $openssl pkcs12 -in ewallet.p12 -passin pass:walletpassword -out ewallet.txt -nodes MAC verified OK If you look at this file it has  "-----BEGIN RSA PRIVATE KEY-----" and "-----END RSA PRIVATE KEY-----". $cat ewallet.txt Bag Attributes     localKeyID: ... Key Attributes: <No Attributes> -----BEGIN RSA PRIVATE KEY----- ... -----END RSA PRIVATE KEY----- Bag Attributes     localKeyID: ... subject=/C=US/CN=*.oracle.com issuer=/C=US/CN=root -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ... Edit this file and copy only the lines starting with " -----BEGIN RSA PRIVATE KEY-----" and ending with " -----END RSA PRIVATE KEY-----" into a new file key.pem. Rest of the steps remain the same. I wanted to check if we are getting "close notify" in a connection, I saw in the presentation http://www.powershow.com/view/29ec1-OWNkM/SSL_Troubleshooting_with_Wireshark_and_Tshark_flash_ppt_presentation useful commands to get a particular field in tshark: $tshark -G fields | fgrep "ssl." and hence used $tshark -R "ssl.alert_message" References http://wiki.wireshark.org/SSL https://forums.oracle.com/forums/thread.jspa?threadID=830575 http://www.powershow.com/view/29ec1-OWNkM/SSL_Troubleshooting_with_Wireshark_and_Tshark_flash_ppt_presentation

Jyri had explained in his blog how to use ssldump to debug SSL connections. We can also use tshark. On my Linux server, tshark is installed in /usr/sbin/tshark. Support team guys need these steps for...

More about PKCS11 Bypass in Oracle iPlanet Web Server 7.0

More about PKCS11 Bypass in Oracle iPlanet Web Server 7.0 Jyri's blog explains the concepts about PKCS11 Bypass in Oracle iPlanet Web server 7.0.     By default in Oracle iPlanet Web Server 7.0, PKCS11Bypass is enabled. To know if PKCS11 Bypass is actually enabled or disabled in your server instance, run the server instance in <log-level>fine</log-level> and check the error log for lines containing the words "PKCS11 bypass".  When PKCS11 Bypass is enabled When PKCS11 Bypass is disabled server.xml <pkcs11>    <allow-bypass>true</allow-bypass></pkcs11> <pkcs11>    <allow-bypass>false</allow-bypass></pkcs11> Error log fine: PKCS#11 bypass is enabled fine: PKCS#11 bypass is disabled     Even though PKCS11 Bypass is enabled in server.xml, it is possible that that check of "SSL_CanBypass" fails in that case PKCS11 Bypass is not enabled. So its essential to check error log contents.     Lets use DTrace scripts to see what's going on at function call level when PKCS11 Bypass is enabled or disabled. Lets analyse the scenario where AES cipher suite is negotiated in SSL Handshake. We know that  "AES_Encrypt" will be called in that case. So we write a script to print stack when "AES_Encrypt" function is called. #!/usr/sbin/dtrace -s#pragma D option quiet pid$1::AES_Encrypt*:entry{    printf("thread %d:  stack is : \n", tid);    ustack();} Note that if in SSL Handshake, others cipher suites were negotiated,  for example if RC4 is negotiated, "RC4_Encrypt" function will be called instead of "AES_Encrypt" and so on... Ideally we need a script with the full list of freebl algorithms but for our simple testing this will serve the purpose. We run this D script and pass the the highest webservd pid as the first argument. Now send a (HTTPS) request via browser to the Web Server instance, here is the stack we get in both the cases (when PKCS11 Bypass is enabled and disabled). User stack when PKCS11 Bypass is enabled User stack when PKCS11 Bypass is disabled* #./ssl.d 22437 thread 17: stack is : libsoftokn3.so`AES_Encrypt libssl3.so`ssl3_CompressMACEncryptRecord+0x5a8 libssl3.so`ssl3_SendRecord+0x38c libssl3.so`ssl3_FlushHandshake+0x1cc libssl3.so`ssl3_SendFinished+0x448 libssl3.so`ssl3_HandleFinished+0x5a4 libssl3.so`ssl3_HandleHandshakeMessage+0x8d0 libssl3.so`ssl3_HandleHandshake+0x2d8 libssl3.so`ssl3_HandleRecord+0xb60 libssl3.so`ssl3_GatherCompleteHandshake+0x110 libssl3.so`ssl_GatherRecord1stHandshake+0xd0 libssl3.so`ssl_Do1stHandshake+0x308 libssl3.so`ssl_SecureRecv+0x230 libssl3.so`ssl_Recv+0x124 libnspr4.so`PR_Recv+0x48 libns-httpd40.so`int DaemonSession::GetConnection()+0x470 libns-httpd40.so`void DaemonSession::run()+0xdc libnsprwrap.so`void Thread::run_()+0x28 #./ssl.d 22469 thread 17: stack is : libsoftokn3.so`AES_Encrypt libsoftokn3.so`NSC_EncryptUpdate+0x490 libnss3.so`PK11_CipherOp+0x28c libssl3.so`ssl3_CompressMACEncryptRecord+0x5a8 libssl3.so`ssl3_SendRecord+0x38c libssl3.so`ssl3_FlushHandshake+0x1cc libssl3.so`ssl3_SendFinished+0x448 libssl3.so`ssl3_HandleFinished+0x5a4 libssl3.so`ssl3_HandleHandshakeMessage+0x8d0 libssl3.so`ssl3_HandleHandshake+0x2d8 libssl3.so`ssl3_HandleRecord+0xb60 libssl3.so`ssl3_GatherCompleteHandshake+0x110 libssl3.so`ssl_GatherRecord1stHandshake+0xd0 libssl3.so`ssl_Do1stHandshake+0x308 libssl3.so`ssl_SecureRecv+0x230 libssl3.so`ssl_Recv+0x124 libnspr4.so`PR_Recv+0x48 libns-httpd40.so`int DaemonSession::GetConnection()+0x470 libns-httpd40.so`void DaemonSession::run()+0xdc libnsprwrap.so`void Thread::run_()+0x28 *Note that we see calls to NSS softoken starting with NSC_.  If we use libpkcs11.so, the names of the symbols to look might be different, i.e. C_Encrypt, C_Decrypt. From the above results we find that  When PKCS11 Bypass is enabled,  function "ssl3_CompressMACEncryptRecord" (and others) in libssl3.so directly call "AES_Encrypt" in the same library (i.e. libssl3.so). When PKCS11 Bypass is disabled,  it calls "PK11_CipherOP" function in libnss3.so which then calls "NSC_EncryptUpdate" function in libsofttoken3.so which in turn calls "AES_Encrypt" function in libsofttoken3.so.  So by enabling PKCS11 Bypass we are eliminating two layers of function calls and that's why its faster.  References http://blogs.oracle.com/jyrivirkki/entry/pkcs_11_and_ssl_performance

More about PKCS11 Bypass in Oracle iPlanet Web Server 7.0 Jyri's blog explains the concepts about PKCS11 Bypass in Oracle iPlanet Web server 7.0.     By default in Oracle iPlanet Web Server...

Which ciphers are enabled in Oracle iPlanet Web Server 7.0 instance? and how do I find information about ciphers that are actually used at run time ?

Which ciphers are enabled in Oracle iPlanet Web Server 7.0 instance? and how do I find information about ciphers that are actually used at run time ? A lot of people ask me how do I know which ciphers are enabled in Oracle iPlanet Web Server 7.0.  The list of ciphers and whether they are enabled or disabled is given in the table http://download.oracle.com/docs/cd/E19146-01/821-0794/gcfbv/index.html . This may slightly vary from update release to another.  The best way to know this is to change <log-level> in server.xml from "info" to "finest" and start the server instance. You will see these log messages at server startup which will tell  you which cipher was enabled or disabled. >...fine: Initializing "NSS Generic Crypto Services" PKCS #11 tokenfine: Initializing "internal" PKCS #11 token....fine: enabling cipher (cert: RSA, auth: RSA, kea: RSA, enc: RC4, mac: MD5, key bits: 128): SSL_RSA_WITH_RC4_128_MD5fine: enabling cipher (cert: RSA, auth: RSA, kea: RSA, enc: RC4, mac: SHA1, key bits: 128): SSL_RSA_WITH_RC4_128_SHAfine: enabling cipher (cert: RSA, auth: RSA, kea: RSA, enc: 3DES, mac: SHA1, key bits: 112): SSL_RSA_WITH_3DES_EDE_CBC_SHA...fine: disabling cipher (cert: RSA, auth: RSA, kea: ECDHE, enc: 3DES, mac: SHA1, key bits: 112): TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHAfine: disabling cipher (cert: RSA, auth: RSA, kea: ECDHE, enc: AES, mac: SHA1, key bits: 256): TLS_ECDHE_RSA_WITH_AES_256_CBC_SHAfine: enabling cipher (cert: RSA, auth: RSA, kea: RSA, enc: AES, mac: SHA1, key bits: 128): TLS_RSA_WITH_AES_128_CBC_SHAfine: enabling cipher (cert: RSA, auth: RSA, kea: RSA, enc: AES, mac: SHA1, key bits:  256): TLS_RSA_WITH_AES_256_CBC_SHAfine: SSLv3/TLS is enabled and 18 SSLv3/TLS ciphers are enabledfine: 0 export ciphers enabledfine: PKCS#11 bypass is enabledfine: 1 RSA certificate(s) present, 6 suitable cipher(s) enabledfine: 0 ECC certificate(s) present, 12 suitable cipher(s) enabled Or if you are familir with Admin CLI you can use the following CLIwadm>list-ciphers --config=<config> --http-listener=<listener> --verbose --all For more information refer :http://docs.oracle.com/cd/E19146-01/821-0792/list-ciphers-1/index.htmlHere is my blog about how to use Dtrace to collect information about ciphers used in the connection : http://blogs.oracle.com/meena/entry/dtarce_script_to_collect_information You can also modify server.xml to print cipher in access log : <access-log>    <file>../logs/access</file>    <format>%Ses->client.ip% - %Req->vars.auth-user% [%SYSDATE%] "%Req->reqpb.clf-request%" %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% %Ses->client.cipher%</format></access-log> Access log will have an new cipher entry in the last column of each row. For example it may show "AES-256","RC4" etc.

Which ciphers are enabled in Oracle iPlanet Web Server 7.0 instance? and how do I find information about ciphers that are actually used at run time ? A lot of people ask me how do I know which ciphers...

HTTPS Oracle iPlanet Web Server 7.0 Reverse Proxy Server and HTTP Origin Server

 HTTPS Oracle iPlanet WebServer 7.0 Reverse Proxy Server and HTTP Origin Server Origin Server <--- HTTP ---> Reverse Proxy Server <---HTTPS ---> client/Browser There are various SSL and non SSL configurations we can have forReverse Proxy and Origin Servers OriginServer <--- HTTP ---> Reverse Proxy Server <--- HTTP --->client/Browser Origin Server <-- HTTP ---> Reverse Proxy Server <--HTTPS --> client/Browser i.e. Reverse proxy as SSL termination Endpoint OriginServer <-- HTTPS ---> Reverse Proxy Server <-- HTTP -->client/Browser OriginServer <-- HTTPS ---> Reverse Proxy Server <-- HTTPS -->client/Browser In this blog I will try out SCENARIO 2 -reverse proxy server as SSL termination end point. For this I have setup two Oracle iPlanet Web Server 7.0 update 11instances. One acting as a reverse proxy (instance name rps on port 8080) and the otherorigin server(instance name origs on port 4444). 1. Enable SSL on Reverse Proxy Server 1.1 Install Server Certificate in Reverse Proxy Server instance Icreated self signed server certificate in reverse proxy server. UseAdmin Server CLI to create a self-signed certificate (recommened) $./bin/wadm --user=adminPlease enter admin-user-password>Connected to localhost:8989Oracle iPlanet Web Server 7.0.11 B03/11/2011 08:38wadm>wadm> list-configsrpswadm>create-selfsigned-cert --config=rps --server-name=www.rps.com --nickname=Server-Cert-RPwadm>deploy-config rps Or you can use certutil followed by pull-config . $cd <reverse-proxy-install-dir>/https-rps/config $../../bin/certutil -N -d . (if DBs do not exist already) $../../bin/certutil -S -d . -n Server-Cert-RP -s "CN=www.rps.com" -x -t "CT,CT,CT" Verify it with certutil that the certificate got installed : $../../bin/certutil -L -d <reverse-proxy-install-dir>/https-rps/config Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPIServer-Cert-RP u,u,u 1.2 Enable SSL in http-listener. Set the server certificatenickname (if its different from "Server-Cert").  Use "set-ssl-prop" Admin Server CLI to enable SSL forthis listener, set the server certificate nickname and then run deployconfig CLI. wadm> list-http-listeners --config=rpshttp-listener-1wadm> set-ssl-prop --config=rps --http-listener=http-listener-1 server-cert-nickname=Server-Cert-RP enabled=truewadm>deploy-config rps server.xml should get modified to look like <http-listener>     <name>http-listener-1</name>     <port>8080</port>     <server-name>www.rps.com</server-name>     <default-virtual-server-name>rps</default-virtual-server-name>    <ssl> <server-cert-nickname>Server-Cert-RP</server-cert-nickname> </ssl></http-listener> 1.3 Run create-reverse-proxy CLI from Administration server wadm> list-virtual-servers --config=rpsrpswadm>create-reverse-proxy --config=rps --vs=rps --uri-prefix=/ --server=http://www.origs.com:4444wadm>deploy-config rps rps.obj.conf gets modified as shown below : <Object name="default"> AuthTrans fn="match-browser" browser="*MSIE*" ssl-unclean-shutdown="true" NameTrans fn="map" from="/" name="reverse-proxy-/" to="http:/" ... </Object> <Object ppath="http:*"> Service fn="proxy-retrieve" method="*" </Object> <Object name="reverse-proxy-/"> Route fn="set-origin-server" server="http://www.origs.com:4444" </Object> In rps.obj.conf I have configured reverse proxy in such a way that allrequests are redirected to origin server. In real world situation you can if you want redirect only certainrequests depending on your requirements. 1.4 OPTIONAL : Change access log format in Reverse ProxyServer Modify access log format in Reverse Proxy Serverto contain %Ses->client.cipher% and %Ses->client.ssl-id% wadm> get-access-log-prop --config=rpsenabled=truefile=../logs/accessformat=%Ses->client.ip% - %Req->vars.auth-user% [%SYSDATE%] "%Req->reqpb.clf-request%"  %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length%mode=textwadm> enable-access-log --file="../logs/access" --config="rps" --format="%Ses->client.ip% - %Req->vars.auth-user% \[%SYSDATE%\] \\"%Req->reqpb.clf-request%\\" %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% %Ses->client.cipher% %Ses->client.ssl-id%wadm>deploy-config rps Or manually add in server.xml the log format <access-log> <file>../logs/access</file> <format> %Ses->client.ip% - %Req->vars.auth-user% [%SYSDATE%] "%Req->reqpb.clf-request%" %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% %Ses->client.cipher% %Ses->client.ssl-id% </format> </access-log> Note the contents in between <format></format> shouldbe in one line I have put it in 2 lines so that its easy to see. 1.5  (OPTIONAL) Create a file test.html file in origin server : $cat ../docs/test.html This is test.html 1.6 Send a request through browser to Reverse Proxy Server on URI /test.html. Start the origin server and reverse proxy server instances. Send a request via the browser to reverse proxy server on https://www.rps.com:8080/test.html. Origin should send the content to reverse proxy serverw hcih should send it to the browser. We can check the entries in both the access logs of reverse proxyserver and origin server to see what's happening. Note that browser may give a warning that this reverse proxy server isnot issued by a valid CA. That's ok because its a self signedcertificate. If you install a certificate from trusted CAs this messagewill not come up. When you check access log entries of reverse porxy server : $cat accessformat=%Ses->client.ip% - %Req->vars.auth-user% [%SYSDATE%]"%Req->reqpb.clf-request%" %Req->srvhdrs.clf-status%%Req->srvhdrs.content-length% %Ses->client.cipher% %Ses->client.ssl-id%xxx.xxx.xxx.xx1 - - [19/Aug/2011:14:02:51 +0530] "GET /test.html HTTP/1.1" 200 18 AES-256 Ux0aq03pRHCNZaDxLX1mrBKzmM7ac4YUAspbTX5s8pw= Its printing the ciphers used and the SSL session id. When you check access log entries of originserver : $cat .access format=%Ses->client.ip% - %Req->vars.auth-user% [%SYSDATE%] "%Req->reqpb.clf-request%" %Req->srvhdrs.clf-status%%Req->srvhdrs.content-length% xxx.xxx.xxx.xx2 - - [19/Aug/2011:13:48:20 +0530] "GET /test.html HTTP/1.1" 200 18 References ConfiguringReverse Proxy in Sun Java System Web Server 7.0 Configuringreverse proxy in Sun Java System Web Server 7.0 when origin server isSSL enabled TwoWAY SSL in Sun Web Server 7.0 reverse proxy server and originserver Abouttrust flags of certificates in NSS database that can be modified bycertutil http://forums.sun.com/thread.jspa?threadID=5397719 http://forums.sun.com/thread.jspa?threadID=5373182 http://forums.sun.com/thread.jspa?threadID=5359313

 HTTPS Oracle iPlanet Web Server 7.0 Reverse Proxy Server and HTTP Origin Server Origin Server <--- HTTP ---> Reverse Proxy Server <--- HTTPS ---> client/Browser There are various SSL and non SSL...

SNI and bench marking tools - ab and siege

SNI and bench marking tools - ab and siege I wanted to do some performance measurements on some SNI server using some too. I evaluated two tools. 1. "ab" (Apache HTTP server benchmarking tool) So I have to build "ab" so that it takes HTTPS URL and not just HTTP URL and sends TLS SNI extension in SSL handshake. 1.1. Download OpenSSL and Apache source code I downloaded OpenSSL source code (openssl-1.0.0d.tar) from http://www.openssl.org/source/ and Apache source code from http://httpd.apache.org/ (httpd-2.3.11-beta.tar and httpd-2.3.11-beta-deps.tar). But I had to make the following two changes in Apache code. 1.2. Modify configure.in $diff configure.in configure.in.ORIGINAL611,614d610< if test "$enable_ssl" != "no"; then<   APR_ADDTO(DEFS, "-DAB_USE_SSL")< fi< I took these changes from http://www.mail-archive.com/dev@httpd.apache.org/msg25661.html 1.3. Modify support/ab.c First I tried  calling the function SSL_set_tlsext_host_name(c->ssl, host_field); but it gave undefined symbol error, so I used SSL_ctrl function instead. $diff ab.c ab.c.orig184d183< #include <openssl/tls1.h> /\* for TLSEXT_NAMETYPE_host_name \*/1182d1180<1244,1245d1241<         SSL_ctrl(c->ssl, SSL_CTRL_SET_TLSEXT_HOSTNAME, TLSEXT_NAMETYPE_host_name, host_field);< 1.4. Building and Installing OpenSSL and Apache I built and installed OpenSSL and Apache as given in http://www.linuxquestions.org/questions/linux-server-73/openssl-support-for-sni-and-tls-799387/#10 OpenSSL : $./config --prefix=/usr/local --openssldir=/usr/local/openssl enable-tlsext shared$make && make install Apache : $LDFLAGS=-L/usr/local/lib CPPFLAGS=-I/usr/local/include/ ./configure --enable-so --enable-ssl --enable-rewrite --enable-unique-id --with-ssl=/usr/local/$make && make install 1.5. Send a test request using "ab" and confirm using ssltap Set LD_LIBRARY_PATH to the OpenSSL directory (containing libssl.so) :     $export LD_LIBRARY_PATH=/usr/local/lib/:$LD_LIBRARY_PATH Confirm that ab -help shows "http[s]" in the usage as shown below :    $/usr/local/apache2/bin/ab -help    Usage: ./ab [options] [http[s]://]hostname[:port]/path Now send a single request and route it to the server using ssltap to confirm if "ab" is working fine : $./ab -n 1 -c 1 -f TLS1 https://www.foo.com:1924/abc.html ssltap output shows that the server name "www.foo.com"  was sent in SSL Handshake : $ssltap -s -l -p 1924 foo.com:port--> [ (230 bytes of 225) SSLRecord { [Thu Mar 31 19:43:21 2011]    type    = 22 (handshake)    version = { 3,1 }    length  = 225 (0xe1)    handshake {       type = 1 (client_hello)       length = 221 (0x0000dd)          ClientHelloV3 {             client_version = {3, 1}             random = {...}             session ID = {                 length = 0                 contents = {...}             }             cipher_suites[46] = { ...             } ...             extensions[88] = {  extension type server_name, length [16] = {   0: 00 0e 00 00  ... 2e 63 6f 6d  | .....www.foo.com } ... 2. siege Downloaded  siege-2.70.tar.gz from ftp://ftp.joedog.org/pub/siege/siege-2.70.tar.gz $gunzip siege.tar.gz $tar -xvf siege.tar $cd siege-2.70 Make these code changes $diff client.c client.c.orig292c292<     if (SSL_initialize(C, U->hostname)==FALSE) {--->     if (SSL_initialize(C)==FALSE) {$diff ssl.h ssl.h.orig52c52< BOOLEAN SSL_initialize(CONN \*C, const char \*servername);---> BOOLEAN SSL_initialize(CONN \*C);$diff ssl.c ssl.c.orig43d42< #include <tls1.h>67c66< SSL_initialize(CONN \*C, const char \*servername)---> SSL_initialize(CONN \*C)137,138d135<   SSL_ctrl(C->ssl, SSL_CTRL_SET_TLSEXT_HOSTNAME,<            TLSEXT_NAMETYPE_host_name, servername); Build and install siege : $./configure --with-ssl=/usr/local/ $make $make install $export LD_LIBRARY_PATH=/usr/local/lib/:$LD_LIBRARY_PATH Run siege $/usr/local/bin/siege -c 10 -t1M https://www.foo.com:3333/index.html you can confirm that siege sent SNI TLS extension using ssltap. 3. References http://www.linuxquestions.org/questions/linux-server-73/openssl-support-for-sni-and-tls-799387/#10 http://stackoverflow.com/questions/5113333/how-to-implement-server-name-indicationsni-on-openssl-in-c-or-c-are-there-an http://www.mail-archive.com/dev@httpd.apache.org/msg25661.html http://www.openssl.org/news/changelog.html http://www.techrepublic.com/article/test-your-web-server-lay-siege-to-it/5171727 http://www.joedog.org/index/siege-manual

SNI and bench marking tools - ab and siege I wanted to do some performance measurements on some SNI server using some too. I evaluated two tools. 1. "ab" (Apache HTTP server benchmarking tool) So I have...

What's New in NSS 3.12.6 - SSL3 & TLS Renegotiation Indication Extension - NSS flags NSS_REQUIRE_SAFE_RENEGOTIATION and NSS_SSL_ENABLE_RENEGOTIATION

What's New in NSS 3.12.6 - SSL3 & TLS Renegotiation IndicationExtension - NSS flags NSS_REQUIRE_SAFE_RENEGOTIATION andNSS_SSL_ENABLE_RENEGOTIATION I tried my hands on various TLS renegotiation scenarios by setting these twoflags NSS_REQUIRE_SAFE_RENEGOTIATION and NSS_SSL_ENABLE_RENEGOTIATIONusing sample SSL Server built with NSS 3.12.6 (and test case requiresrenegotiation). As per NSS3.12.6 release notes for NSS 3.12.6 (or above) the default valuesare NSS_SSL_ENABLE_RENEGOTIATION = SSL_RENEGOTIATE_REQUIRES_XTN and NSS_SSL_REQUIRE_SAFE_NEGOTIATION = PR_FALSE. Sent requests using test tool tstclnt of NSS 3.12.6, NSS 3.12.4 and 3.12.5 with theseflags set in server : Note that in NSS 3.12.5 (in this renegotiation is disabled) so re-negotiations fail in all cases as expected. Server has flag NSS_SSL_ENABLE_RENEGOTIATION = Server has flag NSS_REQUIRE_SAFE_RENEGOTIATION = Controls whether safe renegotiation indication is required for initial handshake. If TRUE, a connection will be dropped at initial handshake if the peer server or client does not support safe renegotiation. Client version NSS 3.12.6 or above - client supports safe renegotiation client version NSS 3.12.4 - older clients - client doesn't support safe renegotiation client version NSS 3.12.5 - renegotiation disabled - client doesn't support safe renegotiation SSL_RENEGOTIATE_NEVER (0) Never allow renegotiation TRUE FAILURE -SSL_ERROR_RENEGOTIATION_NOT_ALLOWED FAILURE -SSL_ERROR_HANDSHAKE_NOT_COMPLETED FAILURE - SSL_ERROR_HANDSHAKE_NOT_COMPLETED FALSE FAILURE -SSL_ERROR_RENEGOTIATION_NOT_ALLOWED  FAILURE- SSL_ERROR_RENEGOTIATION_NOT_ALLOWED FAILURE - SSL_ERROR_RENEGOTIATION_NOT_ALLOWED SSL_RENEGOTIATE_UNRESTRICTED(1) Server and client are allowed to renegotiate without any restrictions. TRUE SUCCESS FAILURE -SSL_ERROR_HANDSHAKE_NOT_COMPLETED FAILURE - SSL_ERROR_HANDSHAKE_NOT_COMPLETED FALSE SUCCESS SUCCESS FAILURE- PR_END_OF_FILE SSL_RENEGOTIATE_REQUIRES_XTN(2) (Default in NSS 3.12.6 or above) Only allows renegotiation if the peer's hello bears the TLS renegotiation_info extension. This is the safe renegotiation. TRUE SUCCESS FAILURE -SSL_ERROR_HANDSHAKE_NOT_COMPLETED FAILURE - SSL_ERROR_HANDSHAKE_NOT_COMPLETED FALSE(Default in NSS 3.12.6 or above) SUCCESS FAILURE- SSL_ERROR_RENEGOTIATION_NOT_ALLOWED FAILURE- PR_END_OF_FILE SSL_RENEGOTIATE_TRANSITIONAL(3) Disallows unsafe renegotiation in server sockets only, but allows clients to continue to renegotiate with vulnerable servers. TRUE SUCCESS FAILURE -SSL_ERROR_HANDSHAKE_NOT_COMPLETED FAILURE - SSL_ERROR_HANDSHAKE_NOT_COMPLETED FALSE SUCCESS FAILURE- SSL_ERROR_RENEGOTIATION_NOT_ALLOWED FAILURE - PR_END_OF_FILE What these error codes mean Error ErrorNumber ErrorText SSL_ERROR_RENEGOTIATION_NOT_ALLOWED -12176 "Renegotiationis not allowed on this SSL socket." SSL_ERROR_HANDSHAKE_NOT_COMPLETED -12202 "CannotinitiateanotherSSL handshake until current handshake is complete." PR_END_OF_FILE_ERROR -5938 "Encounteredend of file" Here ismy Serverprogram Here is my Makefile : all: rm -rf server server.o CC -o server -I/export1/NSS_3.12.6/SunOS5.10_OPT.OBJ/include -L/export1/NSS_3.12.6/SunOS5.10_OPT.OBJ/lib -lnspr4 -lplc4 -lplds4 -lnss3 -lssl3 server.cpp Created Server certificate using certutil as shown below: $certutil -N -d . $certutil -S -n Server-Cert -s "CN=test.com" -x -t "CT,CT,CT" -d . I sent request using sslreq.txt GET /test.html HTTP/1.0end Here are the sample client and server Error logs , ssltap output in these 4 distinctive results References NSS 3.12.6 release notes http://blogs.sun.com/jyrivirkki/entry/web_server_7_and_the http://blogs.sun.com/jyrivirkki/entry/more_thoughts_on_web_server

What's New in NSS 3.12.6 - SSL3 & TLS Renegotiation Indication Extension - NSS flags NSS_REQUIRE_SAFE_RENEGOTIATION and NSS_SSL_ENABLE_RENEGOTIATION I tried my hands on various TLS renegotiation...

What's new in NSS 3.12.\* - Server Name Indication (SNI) callback

What's new in NSS 3.12.\* - Server Name Indication (SNI) callback When a request comes in to a web server, it sends the domain name in the Host header. "GET /test.html HTTP/1.0 Host: abc.com" by the time any web server parses it, its too late to send the appropriate certificate as it was already sent in the SSL Handshake. As per RFC4366, we now have a new extension in Handshake with which client can tell server the required domain. So the server can send appropriate certificate in the handshake. In a Web Server, we can configure only one certificate of a type (ECC/RSA etc.) per HTTP listener using SSL_ConfigureSecureServer(fd...) call. If the server has to be used for one more domain, we need to regenerate the certificate with that domain in Subject Alternate Names. But with Server Name Indication (SNI) extension, we can register, more than one certificates (of a particular type) per HTTP listener. In NSS 3.12.6 a new SNI callback was added as a part of bug 360421. Any Server which uses NSS can set this callback function (similar to what is shown in selfserv.c#1709) and write its implementation to send appropriate SNI certificate at runtime. I used two binaries bundled with NSS selfserv and tstclnt to see what is happening. (I have used NSS 3.12.8) Create two server certificates "www.foo.com" and "www.bar.com" $certutil -N -d . $certutil -S -x -n www.foo.com -s "CN=www.foo.com" -t CTu,u,u -d . $certutil -S -x -n www.bar.com -s "CN=www.bar.com" -t CTu,u,u -d . start the server $selfserv -D -B -s -p 4443  -n www.foo.com -r -a www.bar.com -d . Where -D => disable Nagle delays in TCP -B => bypasses the PKCS11 layer -s => disable SSL socket locking -n  rsa_cert_nickname-a is used to configure server for SNI. [-a sni_cert_nickname] -r => request, not require, cert on initial handshake. create sslreq.cat $cat > req.dat   GET /test.html HTTP/1.0 Now send a request asking for domain "www.bar.com" $tstclnt -p 4443 -h www.foo.com -f -d  . -n www.foo.com  -2 -a www.bar.com  < sslreq.dat where-n => Nickname of key & cert for client auth (use www.foo.com  for now)-a =>Send different SNIname. [-a 1st_handshake_sni_cert_name] We get :  subject DN: CN=www.bar.comissuer  DN: CN=www.bar.com 0 cache hits; 1 cache misses, 0 cache not reusable 0 stateless resumes HTTP/1.0 200 OK Server: Generic Web Server Date: Tue, 26 Aug 1997 22:10:05 GMT Content-type: text/plain GET /test.html HTTP/1.0 EOF As you can see, the server returned the certificate for "www.bar.com" To know what's happening look at the ssltap output (use -c z in selfserv and tstclnt both): $ssltap -l -s -p 1925 foo:4443 Connection #1 [Mon Jan 17 14:51:12 2011] Connected to foo:4443 --> [ (74 bytes of 69) SSLRecord { [Mon Jan 17 14:51:12 2011]    type    = 22 (handshake)    version = { 3,1 }    length  = 69 (0x45)    handshake {       type = 1 (client_hello)       length = 65 (0x000041)          ClientHelloV3 {             client_version = {3, 1}             random = {...}             session ID = {                 length = 0                 contents = {...}             }             cipher_suites[2] = {                 (0x00ff) ????/????????/?????????/???                 (0x0002) SSL3/RSA/NULL/SHA             }             compression[1] = {                 (00) NULL             } extensions[20] = { extension type server_name, length [16] = {    0: 00 0e 00 00  0b 77 77 77  2e 62 61 72  2e 63 6f 6d  | .....www.bar.com               }             }          }    } } ] <-- [ (590 bytes of 585) SSLRecord { [Mon Jan 17 14:51:12 2011]    type    = 22 (handshake)    version = { 3,1 }    length  = 585 (0x249)    handshake {       type = 2 (server_hello)       length = 81 (0x000051)          ServerHello {             server_version = {3, 1}             random = {...}             session ID = {                 length = 32                 contents = {...}             }             cipher_suite = (0x0002) SSL3/RSA/NULL/SHA             compression method = (00) NULL             extensions[9] = {               extension type 65281, length [1] = {    0: 00                    | .               }               extension type server_name, length [0]             }          }       type = 11 (certificate)       length = 430 (0x0001ae)          CertificateChain {             chainlength = 427 (0x01ab)             Certificate {                size = 424 (0x01a8)                data = { saved in file 'cert.001' }             }          }       type = 13 (certificate_request)       length = 58 (0x00003a)          CertificateRequest {             certificate types[3] = { 01 02 40 }             certificate_authorities[52] = { CN=www.bar.com   CN=www.foo.com             }          }       type = 14 (server_hello_done)       length = 0 (0x000000)    } } ] --> [ (754 bytes of 702, with 47 left over) SSLRecord { [Mon Jan 17 14:51:12 2011]    type    = 22 (handshake)    version = { 3,1 }    length  = 702 (0x2be)    handshake {       type = 11 (certificate)       length = 430 (0x0001ae)          CertificateChain {             chainlength = 427 (0x01ab)             Certificate {                size = 424 (0x01a8)                data = { saved in file 'cert.002' }             }          }       type = 16 (client_key_exchange)       length = 130 (0x000082)          ClientKeyExchange {             message = {...}          }       type = 15 (certificate_verify)       length = 130 (0x000082)    } } (754 bytes of 1, with 41 left over) SSLRecord { [Mon Jan 17 14:51:12 2011]    type    = 20 (change_cipher_spec)    version = { 3,1 }    length  = 1 (0x1) } (754 bytes of 36) SSLRecord { [Mon Jan 17 14:51:12 2011]    type    = 22 (handshake)    version = { 3,1 }    length  = 36 (0x24)    handshake {       type = 20 (finished)       length = 12 (0x00000c)          Finished {             verify_data = {...}          }    }       MAC = {...} } ] <-- [ (47 bytes of 1, with 41 left over) SSLRecord { [Mon Jan 17 14:51:12 2011]    type    = 20 (change_cipher_spec)    version = { 3,1 }    length  = 1 (0x1) } (47 bytes of 36) SSLRecord { [Mon Jan 17 14:51:12 2011]    type    = 22 (handshake)    version = { 3,1 }    length  = 36 (0x24)    handshake {       type = 20 (finished)       length = 12 (0x00000c)          Finished {             verify_data = {...}          }    }       MAC = {...} } ] --> [ (50 bytes of 45) SSLRecord { [Mon Jan 17 14:51:12 2011]    type    = 23 (application_data)    version = { 3,1 }    length  = 45 (0x2d)    0: 47 45 54 20  2f 74 65 73  74 2e 68 74  6d 6c 20 48  | GET /test.html H   10: 54 54 50 2f  31 2e 30 0a  0a                        | TTP/1.0..       MAC = {...} } ] <-- [ (196 bytes of 164, with 27 left over) SSLRecord { [Mon Jan 17 14:51:12 2011]    type    = 23 (application_data)    version = { 3,1 }    length  = 164 (0xa4)    0: 48 54 54 50  2f 31 2e 30  20 32 30 30  20 4f 4b 0d  | HTTP/1.0 200 OK.   10: 0a 53 65 72  76 65 72 3a  20 47 65 6e  65 72 69 63  | .Server: Generic   20: 20 57 65 62  20 53 65 72  76 65 72 0d  0a 44 61 74  |  Web Server..Dat   30: 65 3a 20 54  75 65 2c 20  32 36 20 41  75 67 20 31  | e: Tue, 26 Aug 1   40: 39 39 37 20  32 32 3a 31  30 3a 30 35  20 47 4d 54  | 997 22:10:05 GMT   50: 0d 0a 43 6f  6e 74 65 6e  74 2d 74 79  70 65 3a 20  | ..Content-type:   60: 74 65 78 74  2f 70 6c 61  69 6e 0d 0a  0d 0a 47 45  | text/plain....GE   70: 54 20 2f 74  65 73 74 2e  68 74 6d 6c  20 48 54 54  | T /test.html HTT   80: 50 2f 31 2e  30 0a 0a 45  4f 46 0d 0a  0d 0a 0d 0a  | P/1.0..EOF......       MAC = {...} } (196 bytes of 22) SSLRecord { [Mon Jan 17 14:51:12 2011]    type    = 21 (alert)    version = { 3,1 }    length  = 22 (0x16)    warning: close_notify       MAC = {...} } ]  Note that the client sent "www.bar.com" to the server in the extension. Also that server sends the certificate (cert.001) i.e. "www.bar.com"  to the client.   $openssl x509  -in cert.001 -text -inform DERCertificate:    Data:        Version: 3 (0x2)        Issuer: CN=www.bar.com        Subject: CN=www.bar.com And the client sends the certificate (cert.002) i.e. "www.foo.com"  to the server. $openssl x509  -in cert.002 -text -inform DERCertificate:    Data:        Version: 3 (0x2)        Issuer: CN=www.foo.com        Subject: CN=www.foo.com Now send a request asking for domain "www.domain-not-found.com", the server is unable to find a certificate registered for such a domain and hence returns error. This behavior will vary from server to server depending on the implementation.  $tstclnt -p 4443 -h www.foo.com -f -d  . -n www.foo.com  -2 -a "www.domain-not-found.com"  < sslreq.dat  We get the error :  tstclnt: write to SSL socket failed: SSL peer has no certificate for the requested DNS name For this case, ssltap output shows an "unrecognize_name" alert (use -c z in selfserv and tstclnt): $ssltap -l -s -p 1925 foo:4443 Connection #1 [Mon Jan 17 14:49:28 2011] Connected to foo:4443 --> [ (87 bytes of 82) SSLRecord { [Mon Jan 17 14:49:28 2011]    type    = 22 (handshake)    version = { 3,1 }    length  = 82 (0x52)    handshake {       type = 1 (client_hello)       length = 78 (0x00004e)          ClientHelloV3 {             client_version = {3, 1}             random = {...}             session ID = {                 length = 0                 contents = {...}             }             cipher_suites[2] = {                 (0x00ff) ????/????????/?????????/???                 (0x0002) SSL3/RSA/NULL/SHA             }             compression[1] = {                 (00) NULL             }             extensions[33] = {               extension type server_name, length [29] = {  0: 00 1b 00 00  18 77 77 77  2e 64 6f 6d  61 69 6e 2d| .....www.domain- 10: 6e 6f 74 2d  66 6f 75 6e  64 2e 63 6f  6d  | not-found.com               }             }          }    } } ] <-- [ (7 bytes of 2) SSLRecord { [Mon Jan 17 14:49:28 2011]    type    = 21 (alert)    version = { 3,1 }    length  = 2 (0x2)    fatal: unrecognized_name } ] Looks like j2SE SNI client side support was integrated in J2SE 7 b118 CR6985179 To support Server Name Indication extension for JSSE client.  Browsers that support TLS SNI Refer http://en.wikipedia.org/wiki/Server_Name_Indication#Support for a full list. Some of these browsers are : Internet Explorer 7 or later, on Windows Vista or higher. Does not work on Windows XP, even Internet Explorer 8. Mozilla Firefox 2.0 or later Google Chrome (Vista or higher. XP on Chrome 6 or newer. OS X 10.5.7 or higher on Chrome 5.0.342.1 or newer) Safari 2.1 or later (Mac OS X 10.5.6 or higher and Windows Vista or higher) References SNI Wikipedia RFC 4366 selfserv.c SSL_ConfigurSecureServer SNI BUG

What's new in NSS 3.12.\* - Server Name Indication (SNI) callback When a request comes in to a web server, it sends the domain name in the Host header. "GET /test.html HTTP/1.0 Host: abc.com" by the time...

What's new in NSS 3.12.\*- New Shared DBs

NSS has introduced NEW Shared Databases based on SQLite in NSS 3.12.x. Motivation for introducing these new Shared Databases were :ability to edit NSS databases without stopping server and servers want to share DBs. Advantages of these Shared DBs Some experts from https://wiki.mozilla.org/NSS_Shared_DB"NSS has been using an old version of the Berkeley Database as its database engine (commonly described in NSS documents as "DBM") has a number of limitations. One of the most severe limitations concerns the number of processes that may share a database file. While any process has a DBM file open for writing, NO other process may access it in any way. Multiple processes may share a DBM database ONLY if they ALL access it READ-ONLY. Processes cannot share a DBM database file if ANY of them wants to update it.Applications that want to share databases have resorted to these strategies: \* Synchronized updates, with application down time : The applications share the database read-only. If any update is desired, all the applications are shut down, and a database update program performs the update, then all the applications are restarted in read-only mode. This results in undesirable downtime and desired database changes are delayed until the next interval in which such downtime is acceptable."..."The new databases will be called 'shareable' databases. They may or may not be shared by multiple processes, but they are all capable of being shared. "... Shareable DB files : cert9.db, key4.db, pkcs11.txt Are based on SQLite3 Allows read write by multiple simultaneous processes Not enabled by default, must be enabled. To be enable it, use environment variable NSS_DEFAULT_DB_TYPE or add "sql:" prefix in configDir. certutil utility program can do upgrade i.e. convert old DB (cert8.db) to new DB(cert9.db). For automatic upgrade, NSS must be told to use SQL AND the databases must be opened for reading AND writing. If NSS is not told to use SQL, or if the databases are opened READ-ONLY, then no automatic upgrade takes place. Different types of NSS DBs \* Directory name string prefix "sql:", "dbm:", "extern:", "multiaccess:" OR\* Environment variable NSS_DEFAULT_DB_TYPE "sql", "dbm", "extern"where "dbm" is the old default Berkeley DB (cert8). "sql" is the new sqlite3 (cert9) DB. "extern" and "multiaccess" are now obsolete. How to upgrade from cert8.db to cert9.db You can either use environment variables or use sql: prefix in database directory parameter of certutil: $export NSS_DEFAULT_DB_TYPE=sql $certutil -K -d /tmp/nss -XOR$certutil -K -d sql:/tmp/nss -X When you upgrade these are the files you get : key3.db -> key4.db cert8.db -> cert9.db secmod.db -> pkcs11.txtThe contents of the pkcs11.txt files are basically identical to the contents of the old secmod.db, just not in the old Berkeley DB format. If you run the command "$modutil -dbdir DBDIR -rawlist" on an older secmod.db file, you should get output similar to what you see in pkcs11.txt. What needs to be done in programs / C code Either add  environment variable NSS_DEFAULT_DB_TYPE "sql" NSS_Initialize call in https://developer.mozilla.org/en/NSS_Initialize takes this "configDir" parameter as shown below :    NSS_Initialize(configDir, "", "", "secmod.db", NSS_INIT_READONLY); For cert9.db, change this first parameter to "sql:" + configDir (like "sql:/tmp/nss/") i.e. prefix "sql:" in the directory name where these NSS Databases exist.This code will work with cert8.db as well if cert9.db is not present. References https://wiki.mozilla.org/NSS_Shared_DB_Samples https://wiki.mozilla.org/NSS_Shared_DB_Howto https://wiki.mozilla.org/NSS_Shared_DB http://www.mail-archive.com/dev-tech-crypto@lists.mozilla.org/msg05282.html

NSS has introduced NEW Shared Databases based on SQLite in NSS 3.12.x. Motivation for introducing these new Shared Databases were : ability to edit NSS databases without stopping server and servers...

What's new in NSS 3.12.\*- PKIX - Bridged CA, revocation flags and Policies

What's new in NSS 3.12.\* - PKIX - Bridge CA, revocation flags and Policies Network Security Services (NSS) 3.12 has a new feature: libpkix: an RFC 3280 Compliant Certificate Path Validation Library Some terms PKIX "PKIX" refers to all these extensions and enhancements documented in RFC 5280. These may be grouped into three major categories: Bridge CA revocation flags policies CRL Distribution Points Refer RRFC section 4.2.1.13. The CRL distribution points extension identifies how CRL information is obtained. PKI As per RFC 4158 Public Key Infrastructure (PKI) is the set of hardware, software, personnel, policy, and procedures used by a CA to issue and manage certificates. cross-certificate As per RFC 4158, cross-certificate is a certificate issued by one CA to another CA for the purpose of establishing a trust relationship between the two CAs. As per RFC 5280, A cross-certificate is a certificate issued by one CA to another CA that contains a CA signature key used for issuing certificates. PKI structures Lets discuss the three PKI structures : Hierarchical, Mesh style structure, Bridge structure Hierarchical PKI structure In hierarchical PKI structure, a certificate chain starting with a Web Server "Server" certificate would lead to an "intermediate CA", then to a "root" CA whose certificate is present in the web browser. It looks something like      Root CA1         |           Intermediate CA1         |                    EE1 So we can have only one possible certificate chain for any leaf certificate : EE1 -> Intermediate CA1 -> Root CA1. Mesh style PKI structure In a mesh style PKI, CA's authenticate each other and creates certificates for one other. CA1 certificate for CA2 and CA2 creates certificate for CA1. In case there are 'n' CAs the number of certificates will be n2 Example of Mesh style PKI    CA1 ------ CA2       \\            /         CA3 As per RFC 4158, the CAs in this environment have peer relationships; they are neither superior nor subordinate to one another. In a mesh, CAs in the PKI cross-certify. That is, each CA issues a certificate to, and is issued a certificate by, peer CAs in the PKI.  A mesh PKI that is fully cross-certified is called a full mesh. However, it is possible to architect and deploy a mesh PKI with a mixture of uni-directional and bi- directional cross-certifications (called a partial mesh).  Partial meshes may also include CAs that are not cross-certified with other CAs in the mesh. Bridge PKI structure In a bridged PKI, a certificate chain starting with a Web Server "Server" Certificate might lead to Intermediary Bridge CA certificate, then to possible two different trusted Anchors. Example of Bridge PKI          CA1                               CA2         |             \\                        /        |                  \\                /   ICA(keypair1)   Bridge CA  (keypair2- cert1 from CA1 and cert2 from CA2)                                   |                                                EE As you can see "Bridge CA" has been issuedcertificate both by trust anchor "CA1" and trust anchor "CA2". Both these certificates have the same DN. So we can get two certificate chains for a single certificate: EE -> Bridge CA -> CA1 and EE -> Bridge CA -> CA2. As per RFC 4158, each PKI only cross-certifies with one other CA (i.e., the Bridge CA), and the Bridge CA cross-certifies only once with each participating PKI.  As a result, the number of cross certified relationships in the bridged environment grows linearly with the number of PKIs whereas the number of cross-certified relationships in mesh architectures grows exponentially. Bridge CA Brige CA is a CA which has more than one certificates from different issuers. A single intermediate CA entity can have any number of certificates from different CAs. Each of these identities  all share the same DN. As per RFC4158, A BrideCA is a nexus to establish trust paths among multiple PKIs. The Bridge CA cross-certifies with one CA in each participating PKI. In NSS 3.11.\* certificate verification function were CERT_VerifyCertNow CERT_VerifyCert CERT_VerifyCertificate CERT_VerifyCertificateNow CERT_VerifyCACertForUsage In NSS 3.12.\* a new certificate verification function CERT_PKIXVerifyCert as added which should be used to replace above mentioned calls. Scenario show casing the difference between old NSS call CERT_VerifyCertNow and the new PKIX call CERT_PKIXVerifyCert I wrote a sample program(server.cpp) . It runs on a certificate nicknamed "EE" (generated by Bridge CA),  both the old NSS call CERT_VerifyCertNow and the corresponding new PKIX call CERT_PKIXVerifyCert. I wrote a script run-tests.sh which creates CA1, CA2 and 2 certificates for Bridge CA and EE cert. Then it modifies trust flags of CA1 and CA2 and runs server program on it.  When we set a C trust flag on a certificate - It means that this certificate should be treated like aCA certificate but also should be treated as root certificate we trust. When NSS clients/server validates certificate chain we can stop right there. Because CA1 and CA2 root CA certificates are self-signed certificates, if we do not add "C" trust flags in them, we get SEC_ERROR_UNTRUSTED_ISSUER error. When we remove "C" the trust flag of either of the CA certificates CA1 or CA2 and the test case starts failing, we know that the server program uses that chain. When we mark the trust flags of either of the CA certificates CA1 or CA2 as "C" and the test starts to pass so we know that the server program uses that chain. You can see only ONE of CA1 or CA2 chain works in old CERT_VerifyCertNow() but in the new call CERT_PKIXVerifyCert() both the CA1 and CA2 chains work. Look at the last few lines of output of the script run-tests.sh to confirm that. $sh run-tests.sh ...===================================== Running tests with trust flags - CA2-trusted CA1-trusted certutil ouput is Certificate Nickname              Trust Attributes CA1                               CTu,Cu,Cu CA2                               CTu,Cu,Cu Bridge                              u,u,u Bridge                              u,u,u EE                                  u,u,u CERT_PKIXVerifyCert(EE) - PASSED. CERT_VerifyCertNow(EE) - PASSED. ===================================== Running tests with trust flags - CA2-not trusted CA1-trusted certutil ouput is Certificate Nickname              Trust Attributes Bridge                                u,u,u Bridge                                u,u,u EE                                    u,u,u CA1                                  Cu,u,u CA2                                   u,u,u CERT_PKIXVerifyCert(EE) - PASSED. ERROR: CERT_VerifyCertNow(EE) - FAILED !! ( as it uses chain EE-> Bridge ->CA2, CA2 is not trusted due to absence of flag "C")       Error = -8172   SEC_ERROR_UNTRUSTED_ISSUER Peer's certificate issuer has been marked as not trusted by the user. ===================================== Running tests with trust flags - CA2-trusted CA1-not trusted certutil ouput is Certificate Nickname                Trust Attributes Bridge                                  u,u,u Bridge                                  u,u,u EE                                      u,u,u CA2                                    Cu,u,u CA1                                     u,u,u CERT_PKIXVerifyCert(EE) - PASSED. CERT_VerifyCertNow(EE) - PASSED. (as it uses chain EE->Bridge -> CA2 and CA2 is trusted due to flag "C") ===================================== PKIX and revocation flags The function CERT_PKIXVerifyCert gives the server much greater control over how the certificate validation is done.For example, it allows the server to decide what to do about revocation checking for leaf certificates or intermediate CA certs in a certificate chain. whether to use CRLs or OCSP orboth, and in which order of preference. whether to use local CRLs or OCSP or fetch it from network or both. whether to skip it, or attempt it but not require it, or to absolutely require it. whether to ask it to attempt to fetch missing certificates from certificate chains,and/or to fetch CRLs, and/or to fetchOCSP responses. whether to impose any certificate policy requirements on certificates. CERT_PKIxVerifyCert takes an array of inputparameters for tuning these. For details refer http://mxr.mozilla.org/security/source/security/nss/lib/certdb/certt.h#887 http://mxr.mozilla.org/security/source/security/nss/lib/certdb/certt.h#1010 PKIX and Policies Refer RFC520 section 4.2.1.4 for information about policies. When verifying a certificate chain (using CERT_PKIXVerifyCert()) it is possible to pass as input, a givenpolicy oid that must be met. The certification validation will fail if the requestedpolicy is not in place. It is also possible to request a listof all policies(OIDs) which were found to be valid.(a list of OIDs). Thanx to Jyri and Nelson for helping me understand these concepts.  References http://en.wikipedia.org/wiki/Certification_path_validation_algorithm http://tools.ietf.org/html/rfc5280 CERT_VerifyCertNow https://developer.mozilla.org/en/NSS_Certificate_Functions#CERT_VerifyCertNow CERT_VerifyCertNow source codehttp://mxr.mozilla.org/security/source/security/nss/lib/certhigh/certvfy.c#1538 CERT_PKIXVerifyCert source code http://mxr.mozilla.org/security/source/security/nss/lib/certhigh/certvfypkix.c#2015 http://www.mozilla.org/projects/security/pki/nss/nss-3.12/nss-3.12-release-notes.html https://bugzilla.mozilla.org/show_bug.cgi?id=294531 http://tools.ietf.org/html/rfc4158 http://groups.google.com/group/mozilla.dev.security/browse_thread/thread/6830a8566de24547/80677b295869cafa Makefile to compile http://blogs.sun.com/meena/resource/server.cpp SECURITY_DIR=/export2/NSS_3.12.8/all: CC -g -I$(SECURITY_DIR)/include -L$(SECURITY_DIR)/lib -lnspr4 -lnss3 server.cpp -o server And then run http://blogs.sun.com/meena/resource/run-tests.sh.txt

What's new in NSS 3.12.\* - PKIX - Bridge CA, revocation flags and Policies Network Security Services (NSS) 3.12 has a new feature: libpkix: an RFC 3280 Compliant Certificate Path Validation Library Some...

What's new in NSS 3.12.\* - Transport Layer Security (TLS) Session Resumption without Server-Side State

What's new in NSS 3.12.\* - Transport Layer Security (TLS) Session Resumption without Server-Side State NSS 3.12.\* has this new feature "Transport Layer Security (TLS) Session Resumption without Server-Side State". It is not enabled by default. It is defined in RFC5077 . This RFC defines mechanisms to resume a Transport Layer Security (TLS) sessionwithout requiring session-specific state at the TLS server. This mechanism is useful in the following situations: servers that handle a large number of transactions from differentusers servers that desire to cache sessions for a long time ability to load balance requests across servers embedded servers with little memory Lets look at these two cases described in RFC 1. Full Handshake Issuing New Session Ticket Client Server ClientHello (empty SessionTicket extension) ----> <---- ServerHello (empty SessionTicket extension)Certificate\* ServerKeyExchange\* CertificateRequest\* ServerHelloDone Certificate\* ClientKeyExchange CertificateVerify\* [ChangeCipherSpec] Finished ----> <---- NewSessionTicket [ChangeCipherSpec] Finished Application Data <---> Application Data 2. Abbreviated Handshake Using New Session Ticket Client Server ClientHello (SessionTicketextension) ----> <---- ServerHello (empty SessionTicket extension) NewSessionTicket [ChangeCipherSpec] Finished [ChangeCipherSpec] Finished ----> Application Data <---> ApplicationData I have used NSS 3.12.7 in this blog. Observing Two Connections Create a certificate $NSS_DIR/bin/certutil -N -d . $NSS_DIR/bin/certutil -S -n test -s "CN=test" -x -t "CT,CT,CT" -d . For ease of use I will use the same certificate for selfserv and strsclnt. I have used -c z (SSL_RSA_WITH_NULL_SHA) NULL cipher to see what's happening as Jyri had used in one of his blogs. For client we use a tool in NSS "strsclnt" with options "-c 2" (2 connections or more) and "-u. Note that "-u" option means enable the session ticket extension. In one window we run server $NSS_DIR/bin/selfserv -p 8444 -d . -D -B -s -n test -u -c z In another window we run the client strsclnt : $NSS_DIR/bin/strsclnt -p 1924 -c 2 -o -d . -n test -u -2 test -C z strsclnt: -- SSL: Server Certificate Validated. strsclnt: 0 cache hits; 1 cache misses, 0 cache not reusable           0 stateless resumes strsclnt: 1 cache hits; 1 cache misses, 0 cache not reusable           1 stateless resumes Here is the ssltap output for this : $ssltap -l -p 1924 -s test:8444Connection #1 [Tue Nov 2 20:03:16 2010]Connected to test:8444--> [(75 bytes of 70)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 22 (handshake) version = { 3,1 } length = 70 (0x46) handshake { type = 1 (client_hello) length = 66 (0x000042) ClientHelloV3 { client_version = {3, 1} random = {...} session ID = { length = 0 contents = {...} } cipher_suites[2] = { (0x00ff) ????/????????/?????????/??? (0x0002) SSL3/RSA/NULL/SHA } compression[1] = { (00) NULL } extensions[21] = { extension type server_name, length [13] = { 0: 00 0b 00 00 08 ... | .....test } extension type session_ticket, length [0] } } }}]<-- [(526 bytes of 521)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 22 (handshake) version = { 3,1 } length = 521 (0x209) handshake { type = 2 (server_hello) length = 85 (0x000055) ServerHello { server_version = {3, 1} random = {...} session ID = { length = 32 contents = {...} } cipher_suite = (0x0002) SSL3/RSA/NULL/SHA compression method = (00) NULL extensions[13] = { extension type 65281, length [1] = { 0: 00 | . } extension type session_ticket, length [0] extension type server_name, length [0] } } type = 11 (certificate) length = 424 (0x0001a8) CertificateChain { chainlength = 421 (0x01a5) Certificate { size = 418 (0x01a2) data = { saved in file 'cert.001' } } } type = 14 (server_hello_done) length = 0 (0x000000) }}]--> [(186 bytes of 134, with 47 left over)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 22 (handshake) version = { 3,1 } length = 134 (0x86) handshake { type = 16 (client_key_exchange) length = 130 (0x000082) ClientKeyExchange { message = {...} } }}(186 bytes of 1, with 41 left over)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 20 (change_cipher_spec) version = { 3,1 } length = 1 (0x1)}(186 bytes of 36)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 22 (handshake) version = { 3,1 } length = 36 (0x24) handshake { type = 20 (finished) length = 12 (0x00000c) Finished { verify_data = {...} } } MAC = {...}}]<-- [(224 bytes of 172, with 47 left over)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 22 (handshake) version = { 3,1 } length = 172 (0xac) handshake {type = 4 (new_session_ticket) length = 168 (0x0000a8) NewSessionTicket { ticket_lifetime_hint = Sat, 03-Jan-1970 00:00:00 GMT ticket = { length = 162 contents = {...} } } }}(224 bytes of 1, with 41 left over)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 20 (change_cipher_spec) version = { 3,1 } length = 1 (0x1)}(224 bytes of 36)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 22 (handshake) version = { 3,1 } length = 36 (0x24) handshake { type = 20 (finished) length = 12 (0x00000c) Finished { verify_data = {...} } } MAC = {...}}]--> [(46 bytes of 41)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 23 (application_data) version = { 3,1 } length = 41 (0x29) 0: 47 45 54 20 2f 61 62 63 20 48 54 54 50 2f 31 2e | GET /abc HTTP/1. 10: 30 0d 0a 0d 0a | 0.... MAC = {...}}]<-- [(192 bytes of 160, with 27 left over)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 23 (application_data) version = { 3,1 } length = 160 (0xa0) 0: 48 54 54 50 2f 31 2e 30 20 32 30 30 20 4f 4b 0d | HTTP/1.0 200 OK. 10: 0a 53 65 72 76 65 72 3a 20 47 65 6e 65 72 69 63 | .Server: Generic 20: 20 57 65 62 20 53 65 72 76 65 72 0d 0a 44 61 74 | Web Server..Dat 30: 65 3a 20 54 75 65 2c 20 32 36 20 41 75 67 20 31 | e: Tue, 26 Aug 1 40: 39 39 37 20 32 32 3a 31 30 3a 30 35 20 47 4d 54 | 997 22:10:05 GMT 50: 0d 0a 43 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 20 | ..Content-type: 60: 74 65 78 74 2f 70 6c 61 69 6e 0d 0a 0d 0a 47 45 | text/plain....GE 70: 54 20 2f 61 62 63 20 48 54 54 50 2f 31 2e 30 0d | T /abc HTTP/1.0. 80: 0a 0d 0a 45 4f 46 0d 0a 0d 0a 0d 0a | ...EOF...... MAC = {...}}(192 bytes of 22)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 21 (alert) version = { 3,1 } length = 22 (0x16) warning: close_notify MAC = {...}}]Read EOF on Server socket. [Tue Nov 2 20:03:16 2010]Read EOF on Client socket. [Tue Nov 2 20:03:16 2010]Connection 1 Complete [Tue Nov 2 20:03:16 2010] You can see that the server sends the session ticket in the first connection. Hence connection 1 is same as "Full Handshake Issuing New Session Ticket" in the table above. Connection #2 [Tue Nov 2 20:03:16 2010] Connected to test:8444--> [(269 bytes of 264)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 22 (handshake) version = { 3,1 } length = 264 (0x108) handshake { type = 1 (client_hello) length = 260 (0x000104) ClientHelloV3 { client_version = {3, 1} random = {...} session ID = { length = 32 contents = {...} } cipher_suites[2] = { (0x00ff) ????/????????/?????????/??? (0x0002) SSL3/RSA/NULL/SHA } compression[1] = { (00) NULL } extensions[183] = { extension type server_name, length [13] = { 0: 00 0b 00 00 08 ... | .....test } extension type session_ticket, length [162] = { 0: 4e 53 53 21 27 9b e0 2e 14 46 9c 67 e0 31 b6 b9 | NSS!'....F.g.1.. 10: fd 97 31 0a 73 25 0f 70 de 84 89 ac 2b d9 d0 af | ..1.s%.p....+... 20: 00 60 4d 99 7c dd fc 0b 67 10 dd 61 66 55 55 88 | .`M.|...g..afUU. 30: db 59 ce f8 46 95 6b b9 09 ad 7b 95 7c cb 6a dd | .Y..F.k...{.|.j. 40: 17 b6 31 43 df 26 13 6a 15 69 4a 3a f2 9e 6c 43 | ..1C.&.j.iJ:..lC 50: b3 5f 60 4e fc e7 d1 dd 0d 97 d6 a6 cc c7 89 f7 | ._`N............ 60: a4 e4 17 54 23 0f 14 fc e7 c7 5e 7f 5d 00 56 c4 | ...T#.....\^].V. 70: e0 c3 3c cb d1 48 3c 29 b1 26 89 d2 ac d4 2f 14 | ..<..H<).&..../. 80: 99 f5 49 a5 87 2a a5 5b ac c8 a4 df d4 c4 f9 b8 | ..I..\*.[........ 90: 93 55 bf 2f ff 9e 7d e3 0d e2 51 56 c2 30 66 92 | .U./..}...QV.0f. a0: d5 17 | .. } } } }}]<-- [(133 bytes of 81, with 47 left over)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 22 (handshake) version = { 3,1 } length = 81 (0x51) handshake { type = 2 (server_hello) length = 77 (0x00004d) ServerHello { server_version = {3, 1} random = {...} session ID = { length = 32 contents = {...} } cipher_suite = (0x0002) SSL3/RSA/NULL/SHA compression method = (00) NULLextensions[5] = { extension type 65281, length [1] = { 0: 00 | . } } } }}(133 bytes of 1, with 41 left over)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 20 (change_cipher_spec) version = { 3,1 } length = 1 (0x1)}(133 bytes of 36)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 22 (handshake) version = { 3,1 } length = 36 (0x24) handshake { type = 20 (finished) length = 12 (0x00000c) Finished { verify_data = {...} } } MAC = {...}}]--> [(93 bytes of 1, with 87 left over)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 20 (change_cipher_spec) version = { 3,1 } length = 1 (0x1)}(93 bytes of 36, with 46 left over)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 22 (handshake) version = { 3,1 } length = 36 (0x24) handshake { type = 20 (finished) length = 12 (0x00000c) Finished { verify_data = {...} } } MAC = {...}}(93 bytes of 41)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 23 (application_data) version = { 3,1 } length = 41 (0x29) 0: 47 45 54 20 2f 61 62 63 20 48 54 54 50 2f 31 2e | GET /abc HTTP/1. 10: 30 0d 0a 0d 0a | 0.... MAC = {...}}]<-- [(192 bytes of 160, with 27 left over)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 23 (application_data) version = { 3,1 } length = 160 (0xa0) 0: 48 54 54 50 2f 31 2e 30 20 32 30 30 20 4f 4b 0d | HTTP/1.0 200 OK. 10: 0a 53 65 72 76 65 72 3a 20 47 65 6e 65 72 69 63 | .Server: Generic 20: 20 57 65 62 20 53 65 72 76 65 72 0d 0a 44 61 74 | Web Server..Dat 30: 65 3a 20 54 75 65 2c 20 32 36 20 41 75 67 20 31 | e: Tue, 26 Aug 1 40: 39 39 37 20 32 32 3a 31 30 3a 30 35 20 47 4d 54 | 997 22:10:05 GMT 50: 0d 0a 43 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 20 | ..Content-type: 60: 74 65 78 74 2f 70 6c 61 69 6e 0d 0a 0d 0a 47 45 | text/plain....GE 70: 54 20 2f 61 62 63 20 48 54 54 50 2f 31 2e 30 0d | T /abc HTTP/1.0. 80: 0a 0d 0a 45 4f 46 0d 0a 0d 0a 0d 0a | ...EOF...... MAC = {...}}(192 bytes of 22)SSLRecord { [Tue Nov 2 20:03:16 2010] type = 21 (alert) version = { 3,1 } length = 22 (0x16) warning: close_notify MAC = {...}}]Read EOF on Server socket. [Tue Nov 2 20:03:16 2010]Read EOF on Client socket. [Tue Nov 2 20:03:16 2010]Connection 2 Complete [Tue Nov 2 20:03:16 2010] In the second connection, the client has used the same ticket and that's why it doesn't have to go through the full handshake and is not resending its certificate chain etc. Connection 2 is same as "Abbreviated Handshake Using New Session Ticket" in the table above but only with one difference that the server doesn't send a new ticket. When session ticket extension is enabled, session information is taken from the extension data. However, if the extension is not used, SSL layer obtains session info from the internal session cache. If the client comes with a valid session id, session is restarted with out key exchange. NSS Code If you look at selfserv.c http://mxr.mozilla.org/security/source/security/nss/cmd/selfserv/selfserv.c#1695, it uses the call SSL_OptionSet(model_sock, SSL_ENABLE_SESSION_TICKETS, PR_TRUE); Same is true for strsclnt http://mxr.mozilla.org/security/source/security/nss/cmd/strsclnt/strsclnt.c#1237 Reference RFC5077 Transport Layer Security (TLS) Session Resumption without Server-Side State

What's new in NSS 3.12.\* - Transport Layer Security (TLS) Session Resumption without Server-Side State NSS 3.12.\* has this new feature "Transport Layer Security (TLS) Session Resumption without...

Using Kerberos as Authentication Database in Oracle iPlanet Web Server on Solaris 10

Using Kerberos Authentication Database in Oracle iPlanet WebServer on Solaris 10     This article describes how to useKerberos authentication database inOracle iPlanet Web Server.  In this article, KDC and Web Server aresetup on the same host (serverhost.your.domain.com),Kerberos domain is YOUR.DOMAIN.COM,DNS Domain is your.domain.comand client is on host : clienthost.your.domain.com.Both these KDC/Web Server and client machines have Solaris 10. Makesure you configure DNS properly on KDC, server and clientmachines.  "/etc/hosts" should have KDC hostname and it must be the same on allmachines. You can verify by #getent hosts serverhost.your.domain.com <ip-address> serverhost.your.domain.com            Note that thefirst entry is of the form hostname.domainnot just hostname. Clock Synchronization All hosts that participate in theKerberos authentication system musthave their internal clocks synchronized within a specified maximumamountof time. Known as clock skew, this feature provides anotherKerberos security check. If the clock skew is exceeded between any oftheparticipating hosts, requests are rejected. The default value for themaximum clock skew is 300 seconds(five minutes). One way to synchronize all the clocks isto use the Network Time Protocol(NTP) software. See SynchronizingClocks Between KDCs and Kerberos Clients for more information. Or you can also use rdate from client host as shown below [clienthost]# rdate serverhost.your.domain.com 1. Configure Kerberos master KDC on Solaris 10 Install Solaris 10 with these options Enable Kerberos: Yes Kerberos default realm: YOUR.DOMAIN.COM Kerberos Admin Server and KDC : serverhost.your.domain.com \* even if you do not we can edit configuration files manually. 1.1 Modify Kerberos configuration files            Modify Kerberos configuration files as shown in the tables below. /etc/krb5/krb5.conf [libdefaults] default_realm = YOUR.DOMAIN.COM[realms] YOUR.DOMAIN.COM = { kdc = serverhost.your.domain.com admin_server = serverhost.your.domain.com }[domain_realm] .your.domain.com = YOUR.DOMAIN.COM[logging] default = FILE:/var/krb5/kdc.log kdc = FILE:/var/krb5/kdc.log kdc_rotate = { period = 1d versions = 10 }[appdefaults] kinit = { renewable = true forwardable= true } gkadmin = { help_url = ... } /etc/krb5/kdc.conf [kdcdefaults] kdc_ports = 88,750[realms] YOUR.DOMAIN.COM = { profile = /etc/krb5/krb5.conf database_name = /var/krb5/principal admin_keytab = /etc/krb5/kadm5.keytab  acl_file = /etc/krb5/kadm5.acl kadmind_port = 749 max_life = 8h 0m 0s  max_renewable_life = 7d 0h 0m 0s default_principal_flags = +preauth } /etc/krb5/kadm5.acl \*/admin@YOUR.DOMAIN.COM \* 1.2 Start dns/client service on MasterKDC For Kerberos to work dns/client servicemust be started before you start any other Kerberos daemons. Edit /etc/resolv.confto havenameserver entries andthen enable dns/clientservice using svcadm command as shown below. # svcadm-v enable-s dns/client svc:/network/dns/client:defaultenabled. 1.3 Create principal database on masterKDC                 Create principal database using kdb5_util. # kdb5_util create -s Initializing database'/var/krb5/principal' for realm'YOUR.DOMAIN.COM', master key name'K/M@YOUR.DOMAIN.COM' You will be prompted for thedatabase Master Password. It is important that you NOTFORGET this password. Enter KDC database masterkey: <--- Enter password here Re-enter KDC database masterkey to verify: <---Enter the same password again  List the principals which gotcreated by default by listprincscommand as shown below # kadmin.local kadmin.local: listprincs K/M@YOUR.DOMAIN.COM changepw/serverhost.your.domain.com@YOUR.DOMAIN.COM kadmin/changepw@YOUR.DOMAIN.COM kadmin/history@YOUR.DOMAIN.COM kadmin/serverhost.your.domain.com@YOUR.DOMAIN.COM kiprop/serverhost.your.domain.com@YOUR.DOMAIN.COM krbtgt/YOUR.DOMAIN.COM@YOUR.DOMAIN.COM Make sure you have these threeentriesshown in green color. 1.4 Add atleast one AdministrativePrincipal to Kerberosdatabase The administrative principals createdhereshould be the ones that were added to the ACL file. # kadmin.local kadmin.local: addprincadmin/admin@YOUR.DOMAIN.COM WARNING: no policyspecified foradmin/admin@YOUR.DOMAIN.COM; defaulting to no policy Enter password for principal"admin/admin@YOUR.DOMAIN.COM": <- Enter password here Re-enter password forprincipal "admin/admin@YOUR.DOMAIN.COM": <-Enter the same password again Principal"admin/admin@YOUR.DOMAIN.COM" created. 1.5 Create a keytabfile for the kadmind service. Create thekadmin keytab with entries for these principals # kadmin.local kadmin.local: ktadd -k/etc/krb5/kadm5.keytab kadmin/serverhost.your.domain.com               changepw/serverhost.your.domain.com               kadmin/changepw This command sequence creates a specialkeytab file with principalentriesfor kadmin/<FQDN> and changepw/<FQDN>.These principals are needed for the kadmind service andfor passwords to be changed. Note that when the principal instance is ahostname, the FQDN must be specified in lowercase letters, regardless ofthe caseof the domain name in the /etc/resolv.conf file. The kadmin/changepwprincipal is used to change passwords from clientsthat are not running a Solaris release. Use klist toinspect keytab file # klist-k /etc/krb5/kadm5.keytab Keytabname:FILE:/etc/krb5/kadm5.keytab KVNOPrincipal ---------------------------------------------------------   3kadmin/serverhost.your.domain.com@YOUR.DOMAIN.COM   3kadmin/serverhost.your.domain.com@YOUR.DOMAIN.COM   3kadmin/serverhost.your.domain.com@YOUR.DOMAIN.COM   3kadmin/serverhost.your.domain.com@YOUR.DOMAIN.COM   3kadmin/changepw@YOUR.DOMAIN.COM   3kadmin/changepw@YOUR.DOMAIN.COM   3kadmin/changepw@YOUR.DOMAIN.COM   3kadmin/changepw@YOUR.DOMAIN.COM   3changepw/serverhost.your.domain.com@YOUR.DOMAIN.COM   3changepw/serverhost.your.domain.com@YOUR.DOMAIN.COM   3changepw/serverhost.your.domain.com@YOUR.DOMAIN.COM   3changepw/serverhost.your.domain.com@YOUR.DOMAIN.COM 1.6 Create /etc/krb5/krb5.keytab Some programs like telnet by defaultlook for /etc/krb5/krb5.keytabso we create a softlink as shown below #ln-s/etc/krb5/kadm5.keytab/etc/krb5/krb5.keytab 1.7 Start Kerberos Daemons on Master KDC Use svcadm command to start Kerberosdaemons krb5kdc and kadmin as shown below # svcadm-v enable -s krb5kdc svc:/network/security/krb5kdc:defaultenabled. # svcadm -v enable -skadmin svc:/network/security/kadmin:defaultenabled. Also, make sure that ktkt_warn service is alsoonline. # svcs| grep security online      Apr_17   svc:/network/security/ktkt_warn:default online       0:37:21svc:/network/security/krb5kdc:default online       0:37:26svc:/network/security/kadmin:default                Use svcadm command to start time service s as well #svcadm-v enable -s time:streamtime:dgram svc:/network/time:streamenabled. svc:/network/time:dgramenabled. 1.8Create host and HTTP principalonKDC, and extract its keys into keytab Create host and HTTP principal on KDC and extract them to keytab file as shown below # kadmin -p admin/adminkadmin: addprinc -randkey host/serverhost.your.domain.comWARNING: no policy specified for host/serverhost.your.domain.com@YOUR.DOMAIN.COM; defaulting to no policyPrincipal "host/serverhost.your.domain.com@YOUR.DOMAIN.COM" created.kadmin: addprinc -randkey HTTP/serverhost.your.domain.comWARNING: no policy specified for HTTP/serverhost.your.domain.com@YOUR.DOMAIN.COM; defaulting to no policyPrincipal "HTTP/serverhost.your.domain.com@YOUR.DOMAIN.COM" created. kadmin: ktadd -k /etc/krb5/kadm5.keytab host/serverhost.your.domain.com kadmin: ktadd -k/etc/krb5/kadm5.keytab HTTP/serverhost.your.domain.com \*Note that when the principal instance is a host name, the FQDNmust be specified in lowercase letters. 1.9 Create a user "testuser" and add itto KDC                 For testing, add a user "testuser", set its password and add thisprincipal to KDC. # useradd-u400 testuser #passwdtestuser ... #kadmin -p admin/admin kadmin: addprinctestuser WARNING:no policy specified for testuser@YOUR.DOMAIN.COM; defaulting to nopolicy Enter password forprincipal"testuser@YOUR.DOMAIN.COM": <- Enter password here Re-enter password forprincipal"testuser@YOUR.DOMAIN.COM": <-Enter the same password again Principal"testuser@YOUR.DOMAIN.COM" created. 1.10 Configure slave KDCs as given in "Solaris 10 - SystemAdministration Guide: Security Services" (optional) But for simplicity this can be skipped. 2. Oracle iPlanet Web Server Configuration 2.1 Install Oracle iPlanet Web Server onthe machinewhere KDC is installed. In this case, for simplicity, Oracle iPlanet web server is installed on the same machine as master KDC. 2.2 Copy keytab file and make sure thatuser running Web Server Instance has file system permissions If Web Server instance is running as"webservd"(or any userother than root), it doesn't have permissions to read /etc/krb5/kadm5.keytab.So copy the file as shown below and change its owner. # cdhttps-<instance>/config# cp /etc/krb5/kadm5.keytab kadm5.keytab # chown webservd:webservd kadm5.keytab 2.3 Create Kerberos AuthenticationDatabase in Web Server Tocreate an authentication database through Administration CLI, executethe following wadm commands. wadm>create-kerberos-authdb --config=test --service-name=HTTP my-kerberos wadm> set-config-prop--config=test krb5-keytab-file=kadm5.keytab \* Refer the latestWeb Server documentation for details. After this, server.xml should have these new entries : <auth-db> <enabled>true</enabled> <name>my-kerberos</name> <url>kerberos</url> <property> <name>servicename</name> <value>HTTP</value> </property> </auth-db> <krb5-keytab-file>kadm5.keytab</krb5-keytab-file> 2.4 Add the following ACL in ACL file            Using Administration GUI/CLI, add ACL for /krb uri so that onlyauthenticates users are allowed access.            ACL file should contain ACL like this : acl "uri=/krb/"; authenticate (user, group) {     method = "gssapi";     database ="my-kerberos"; }; deny (all) user="anyone"; allow (all) user="all"; You can set last allow ACE as user="testuser@YOUR.DOMAIN.COM"instead of user="all". 2.5 Add test.html in krb directory indocument root directory 3. Configure Kerberos Client       On client host (in this case it isclienthost.your.domain.com) 3.1 Copy /etc/krb5/krb5.conffile from KDC. 3.2 Obtain Kerberos ticket-granting ticket for "testuser" principal: Add user testuser and request a ticketfor that user [clienthost]# useradd -u 400 testuser [clienthost]# passwd testuser ... [clienthost]# kinit testuser Check that the ticket existswith klist command as shown below [clienthost]# klist Ticket cache:FILE:/tmp/krb5cc_0Default principal: testuser@YOUR.DOMAIN.COM Validstarting               Expires               Service principal07/01/10 01:06:48  07/01/10 09:06:48 krbtgt/YOUR.DOMAIN.COM@YOUR.DOMAIN.COM        renew until 07/08/10 01:06:4807/01/10 01:09:03  07/01/10 09:06:48 host/serverhost.your.domain.com@YOUR.DOMAIN.COM        renew until 07/08/10 01:06:48 3.3 Testing if Kerberos was setupproperly (optional) 3.3.1 Check for these entries inpam.conf. Needed just to testiftelnet is kerberized.  Entriesin pam.conf ktelnet auth required pam_unix_cred.so.1ktelnet auth binding pam_krb5.so.1ktelnet auth required pam_unix_auth.so.1 other account requisitepam_roles.so.1 other account requiredpam_unix_account.so.1 other session requiredpam_unix_session.so.1 other password requiredpam_dhkeys.so.1 other password requisitepam_authtok_get.so.1 other password requisitepam_authtok_check.so.1 other password requiredpam_authtok_store.so.1 3.3.2 Make sure that telnetd is runningon server. 3.3.3 Testing with telnet [clienthost]telnet -a -F -ltestuser serverhost.your.domain.com Trying<ip-address-of-server>... Connected toserverhost.your.domain.com. Escape character is '\^]'. [ Kerberos V5 accepts you as``testuser@YOUR.DOMAIN.COM'' ] [ Kerberos V5 acceptedforwarded credentials ] ... The above should log in without password prompt. This shows kerberos is setup properly. 3.4 Settings in Firefox/Mozilla Browser Open Mozilla/Firefox browser. On the url type ">about:config,set network.negotiate-auth.delegation-urisand network.negotiate-auth.trusted-uristohttp://serverhost.your.domain.com,https://serverhost.your.domain.com 3.5 Access WebServer Content usingMozilla/Firefox Browser On Mozilla/Firefox, access http://serverhost.your.domain.com/krb/test.html.This should bring up the test.html page On the command prompt type #kdestroy. This should destroy the existing ticket issued to testuser On Mozilla/Firefox now access http://serverhost.your.domain.com/krb/test.html. This shouldbring up Unauthorized page. Enable security in Web Server and follow the same for https://.... This is what is happening between the Browser and Web Server, when /test.html is accessed. (Ignoring Browser->KDC and Web Server -> KDC interactions) Browserto Web Server WebServer to Browser GET /test.html HTTP/1.1Host: ...... HTTP/1.1401 UnauthorizedServer: Sun-Java-System-Web-Server/7.0 Www-authenticate: Negotiate... <HTML><HEAD><TITLE>Unauthorized</TITLE></HEAD>... GET/test.html HTTP/1.1Host: ... Authorization: Negotiate <gssapi-data>... HTTP/1.1200 OKServer: Sun-Java-System-Web-Server/7.0 Www-authenticate: Negotiate <gssapi-data>Content-type: text/html... This is test.html Browser sends a GET request.  Web server returns a 401 Unauthorized responsewith Header "Www-authenticate"with value "Negotiate". Browser sends the same request with Header "Authorization" with value "Negotiate <gssapi-data>" Web Server returns a 200 OK response with Header "Www-authenticate" withvalue "Negotiate <gssapi-data>" and the contents of the file user has asked for. Access log of Web Server Instance showstwo entries as shown below format=%Ses->client.ip% -%Req->vars.auth-user% [%SYSDATE%] "%Req->reqpb.clf-request%"%Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% <client-ip-address>--  [<date-and-time>] "GET /test.html HTTP/1.1" 401 223 <client-ip-address>- testuser@YOUR.DOMAIN.COM [<date-and-time>] "GET /test.html HTTP/1.1" 200 18 4. References Blog by Slava Leanovich "Start playing with Kerberos" http://blogs.sun.com/vl/entry/start_playing_with_kerberos Solaris 10 System Administration Guide: Security Services http://docs.sun.com/app/docs/doc/816-4557/seamtm-1?l=en&a=view Working With the Authentication Database in Sun Java System WebServer 7.0 Update 8 Administrator’s Guide http://docs.sun.com/app/docs/doc/821-1496/gczym?l=en&a=view Kerberos V5 Install guide from MIT http://web.mit.edu/Kerberos/krb5-1.8/krb5-1.8.2/doc/krb5-install.html Chandra's QA test specification document

Using Kerberos Authentication Database in Oracle iPlanet Web Server on Solaris 10     This article describes how to use Kerberos authentication database in Oracle iPlanet Web Server.  In this article,...

More on Intrusion Detection

I found that experimental Intrusion Detection module as explained in my previous blog doesn't work as expected if an external plugin's AuthTrans SAF is added in obj.conf request processing and if that SAF returns REQ_PROCEED. This may be a rarely happen in customer deployments. Will try to fix it in next update release/ next major release and will let you know when it is fixed. My id.conf : SecRuleEngine on SecRequestBodyAccess onSecRule REQUEST_BODY "junk" case 1: I created a dummy plugin having AuthTrans function myauth1; which just returns REQ_NOACTION it works fine. (look at<ws7-install-dir>/samples/nsapi/ for examples of how to create a plugin) #ifdef XP_WIN32 #define NSAPI_PUBLIC __declspec(dllexport) #else /\* !XP_WIN32 \*/ #define NSAPI_PUBLIC #endif /\* !XP_WIN32 \*/ #include "nsapi.h" extern "C" NSAPI_PUBLIC int myauth1(pblock \*pb, Session \*sn, Request \*rq) { return REQ_NOACTION; } Added in Magnus.conf Init fn="load-modules" shlib="myauth.so" funcs="myauth1" Error logs in that case show : ... ... func_exec reports: executing fn="match-browser" browser="\*MSIE\*" ssl-unclean-shutdown="true" Directive="AuthTrans" magnus-internal="1" ... func_exec reports: fn="match-browser" browser="\*MSIE\*" ssl-unclean-shutdown="true" Directive="AuthTrans" magnus-internal="1" returned -2 (REQ_NOACTION) ... func_exec reports: executing fn="myauth1" Directive="AuthTrans" ... func_exec reports: fn="myauth1" Directive="AuthTrans" returned -2 (REQ_NOACTION) ... func_exec reports: executing fn="magnus-internal/secrule-filters-insert" ... func_exec reports: fn="magnus-internal/secrule-filters-insert" returned -2 (REQ_NOACTION) ... func_exec reports: executing fn="ntrans-j2ee" name="j2ee" Directive="NameTrans" ... case 2: When I change this AuthTrans SAF to return REQ_PROCEED, it doesn't work as expected: #ifdef XP_WIN32 #define NSAPI_PUBLIC __declspec(dllexport) #else /\* !XP_WIN32 \*/ #define NSAPI_PUBLIC #endif /\* !XP_WIN32 \*/ #include "nsapi.h" extern "C" NSAPI_PUBLIC int myauth2(pblock \*pb, Session \*sn, Request \*rq) { return REQ_PROCEED; } Added in Magnus.conf Init fn="load-modules" shlib="myauth.so" funcs="myauth2" Error logs in that case shows : ... func_exec reports: executing fn="match-browser" browser="\*MSIE\*" ssl-unclean-shutdown="true" Directive="AuthTrans" magnus-internal="1" ... func_exec reports: fn="match-browser" browser="\*MSIE\*" ssl-unclean-shutdown="true" Directive="AuthTrans" magnus-internal="1" returned -2 (REQ_NOACTION) ... func_exec reports: executing fn="myauth2" Directive="AuthTrans" ... func_exec reports: fn="myauth2" Directive="AuthTrans" returned 0 (REQ_PROCEED) ... func_exec reports: executing fn="ntrans-j2ee" name="j2ee" Directive="NameTrans ... Note fn="magnus-internal/secrule-filters-insert" is not getting executed here. You can add this secrule-filters-insert SAF above your ExternalPluginAuthTransSAF function: <Object name="default"> AuthTrans fn="match-browser" browser="\*MSIE\*"ssl-unclean-shutdown="true" AuthTrans fn="magnus-internal/secrule-filters-insert" AuthTrans fn="ExternalPluginAuthTransSAF" NameTrans fn="ntrans-j2ee" name="j2ee" ... </Object> This will work fine when ExternalPluginAuthTransSAF function returns REQ_PROCEED but when it returns REQ_NOATCION, these filters will be added twice. You can make a dynamic library of myauth2 plugin as shown above and put it below "ExternalPluginAuthTransSAF" <Object name="default"> AuthTrans fn="match-browser" browser="\*MSIE\*"ssl-unclean-shutdown="true" AuthTrans fn="magnus-internal/secrule-filters-insert" AuthTrans fn="ExternalPluginAuthTransSAF" AuthTrans fn="myauth" NameTrans fn="ntrans-j2ee" name="j2ee" ... </Object>

I found that experimental Intrusion Detection module as explained in my previous blog doesn't work as expected if an external plugin's AuthTransSAF is added in obj.conf request processing and if that...

About LDAP connections in Sun Web Server 7.0

About LDAP connections in Sun Web Server 7.0  When I have in my server.xml confiured an Sun LDAP server as given below  :   <auth-db>    <name>ldap</name>    <url>ldap://localhost:1369/o%3DTestCentral</url>    <property>        <name>binddn</name>        <value>cn=Directory Manager</value>    </property>    <property>        <name>bindpw</name>        <value>...</value><encoded/>    </property>  </auth-db> When I have ACL set acl "uri=/";deny (all) user="anyone"; allow (all) user="all"; When I send a request via browser as user "alpha" and its correct password. First it binds as the user specified in server.xml "binddn" property ( in this case "cn=Directory Manager") Then sends a LDAP search query with filter "uid=alpha". Third step it does is it binds as that user "alpha" with the password user specified. If bind is successful, access is allowed. Here are the entries I get in LDAP server access logs: [07/Oct/2009:13:16:16 +051800] conn=2 op=-1 msgId=-1 - fd=34 slot=34 LDAP connection from 127.0.0.1 to 127.0.0.1[07/Oct/2009:13:16:16 +051800] conn=2 op=0 msgId=1 - BIND dn="cn=Directory Manager" method=128 version=3[07/Oct/2009:13:16:16 +051800] conn=2 op=0 msgId=1 - RESULT err=0 tag=97 nentries=0 etime=0 dn="cn=directory manager"[07/Oct/2009:13:16:16 +051800] conn=2 op=1 msgId=2 - SRCH base="o=testcentral" scope=2 filter="(uid=alpha)" attrs="c"[07/Oct/2009:13:16:16 +051800] conn=2 op=1 msgId=2 - RESULT err=0 tag=101 nentries=1 etime=0[07/Oct/2009:13:16:16 +051800] conn=2 op=2 msgId=3 - BIND dn="uid=alpha,o=TestCentral" method=128 version=3[07/Oct/2009:13:16:16 +051800] conn=2 op=2 msgId=3 - RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=alpha,o=testcentral" To understand why we see these three entries, I started debugging Web Server from LASUserEval function and came at get_is_valid_password_ldap function. Note that line numbers in Open Web Server may differ but the concepts are the same : t@16 (l@16) stopped in get_is_valid_password_ldap at line 455 in file "ldapacl.cpp" 467 if ((rv = ld->find_userdn(raw_user, basedn, &userdn)) == LDAPU_SUCCESS) {(dbx) p filterfilter = "uid=alpha"... t@16 (l@16) stopped in LdapSession::find_userdn at line 1253 in file "LdapOps.cpp" 1253 retval = find(base, LDAP_SCOPE_SUBTREE, filter, attrs, 1 /\* no attrs \*/, res);(dbx) s...t@16 (l@16) stopped in LdapSession::find at line 1174 in file "LdapOps.cpp" 1174 res = search(base, scope, filter, (char \*\*)attrs, attrsonly, 0);(dbx) p basebase = 0x6ed6048 "o=TestCentral"(dbx) p filterfilter = 0xfbfbc98c "uid=alpha"...(dbx) nt@16 (l@16) stopped in LdapSession::search at line 289 in file "LdapSession.cpp" 289 rv = bindAsDefault();(dbx) st@16 (l@16) stopped in LdapSession::bindAsDefault at line 235 in file "LdapSession.cpp" 235 int msg_id = ldap_simple_bind(session, ldapRealm->getBindName(), ldapRealm->getBindPwd());(dbx) p msg_idmsg_id = 1...(dbx) nt@16 (l@16) stopped in LdapSession::bindAsDefault at line 240 in file "LdapSession.cpp" 240 int ret = ldap_result( session, msg_id, 0, &time_out, &res);(dbx) p retret = 97... As we can see Web Server first tries to bindAsDefault ie. bind as cn=Directory Manager. by looking at LDAP Server logs, we confirm this. [07/Oct/2009:14:52:03 +051800] conn=4 op=-1 msgId=-1 - fd=34 slot=34 LDAP connection from 127.0.0.1 to 127.0.0.1[07/Oct/2009:14:52:03 +051800] conn=4 op=0 msgId=1 - BIND dn="cn=Directory Manager" method=128 version=3[07/Oct/2009:14:52:03 +051800] conn=4 op=0 msgId=1 - RESULT err=0 tag=97 nentries=0 etime=0 dn="cn=directory manager" When bind is successful, now it tries LDAP search on basedn="o=TestCentra" with filter "uid=alpha" ...(dbx) nt@16 (l@16) stopped in LdapSession::search at line 296 in file "LdapSession.cpp" 296 rv = lastoprv = ldap_search_ext_s(session, base, scope, filter,297 attrs, attrsonly,298 NULL, NULL, &time_out, LDAP_NO_LIMIT, &search_results);(dbx) p basebase = 0x6ed6048 "o=TestCentral"(dbx) p scopescope = 2(dbx) p filterfilter = 0xfbfbc98c "uid=alpha"...(dbx) nt@16 (l@16) stopped in LdapSession::find at line 1178 in file "LdapOps.cpp" 1178 break;(dbx) nt@16 (l@16) stopped in LdapSession::find at line 1187 in file "LdapOps.cpp" 1187 numEntries = res->entries();(dbx) nt@16 (l@16) stopped in LdapSession::find at line 1189 in file "LdapOps.cpp" 1189 if (numEntries == 1) {(dbx) p numEntriesnumEntries = 1(dbx) n...(dbx) nt@16 (l@16) stopped in LdapSession::find_userdn at line 1278 in file "LdapOps.cpp" 1278 \*dn = entry->DN();(dbx) p \*dn\*dn = 0x2d87b0 "uid=alpha,o=TestCentral"(dbx)... Looking at LDAP server access logs it has created this "SRCH" entry : [07/Oct/2009:14:55:51 +051800] conn=4 op=1 msgId=2 - SRCH base="o=testcentral" scope=2 filter="(uid=alpha)" attrs="c"[07/Oct/2009:14:55:51 +051800] conn=4 op=1 msgId=2 - RESULT err=0 tag=101 nentries=1 etime=0 So at the end of find_userdn function, we get our userdn "uid=alpha,o=TestCentral". Coming back to debugger (dbx) p userdnuserdn = 0x2d87b0 "uid=alpha,o=TestCentral"(dbx)t@16 (l@16) stopped in get_is_valid_password_ldap at line 531 in file "ldapacl.cpp" 531 rv = ld->userdn_password(userdn, raw_pw);(dbx) s...(dbx) nt@16 (l@16) stopped in LdapSession::userdn_password at line 1052 in file "LdapOps.cpp" 1052 int msgid = ldap_simple_bind(session, userdn, password);(dbx) p userdnuserdn = 0x2d87b0 "uid=alpha,o=TestCentral"...(dbx) nt@16 (l@16) stopped in LdapSession::userdn_password at line 1058 in file "LdapOps.cpp" 1058 rc = ldap_result(session, msgid, 0, &zerotime, &res );(dbx) n(dbx) p rcrc = 97... As we can see Web Server tries to bind to LDAP server with userdn "uid=alpha,o=TestCentral" and we can confirm that by looking at LDAP server logs : [07/Oct/2009:15:00:37 +051800] conn=4 op=2 msgId=3 - BIND dn="uid=alpha,o=TestCentral" method=128 version=3[07/Oct/2009:15:00:37 +051800] conn=4 op=2 msgId=3 - RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=alpha,o=testcentral"Also refer Jyri's blog for troubleshooting

About LDAP connections in Sun Web Server 7.0  When I have in my server.xml confiured an Sun LDAP server as given below  : <auth-db> <name>ldap</name> ...

Two WAY SSL in Sun Web Server 7.0 reverse proxy server and origin server

Two Way SSL in Oracle iPlanet Web Server 7.0 reverse proxy server and origin server Origin Server <--- HTTPS ---> Reverse Proxy Server <--- HTTPS ---> client/Browser There are various SSL and non SSL configurations we can have for Reverse Proxy and Origin Servers Origin Server <--- HTTP ---> Reverse Proxy Server <--- HTTP ---> client/Browser Origin Server <-- HTTP ---> Reverse Proxy Server <-- HTTPS --> client/Browser i.e. Reverse proxy SSL termination End point Origin Server <-- HTTPS ---> Reverse Proxy Server <-- HTTP --> client/Browser Origin Server <-- HTTPS ---> Reverse Proxy Server <-- HTTPS --> client/Browser In this blog I will try out SCENARIO 4 - with and without enabling client-auth in reverse proxy server and origin server instance, with and without ssl-client-config SAF. For this I have setup two Oracle iPlanet Web Server 7.0 update 9 instances. One acting as a reverse proxy (instance name rps) and the other origin server(instance name origs) that serves the request. This blog is split into the following four sub parts Enable SSL on both Origin Server and Reverse Proxy Server AND no client authentication anywhere - Origin Server sends its Server certificate to Reverse Proxy Server. enable client-auth on Reverse Proxy Server - client/browser sends its certificate to Reverse Proxy Server AND Origin Server sends its certificate to Reverse Proxy server. enable client-auth on Origin Server, set ssl-client-config SAF on Reverse Proxy Server(OPTIONAL) - Origin Server and Reverse Proxy Server both send their certificates to each other. enable client-auth on Origin Server and Reverse Proxy Servers, set ssl-client-config SAF on Reverse Proxy Server(OPTIONAL) - client/browser sends its certificate to Reverse Proxy Server AND Origin Server and Reverse Proxy Server send certificates to each other. 1. Enable SSL on both Origin Server and Reverse Proxy Server, no client authentication anywhere - Origin Server sends its Server certificate to Reverse Proxy Server. 1.1 Install Server Certificate in Reverse Proxy Server instance I created self signed certificate in reverse proxy server. You can use Admin Server CLI to create a self-signed certificate wadm>create-selfsigned-cert --config=rps --server-name=www.rps.com --nickname=Server-Cert-RPwadm>deploy-config rps Or you can use certutil followed by pull-config $cd <reverse-proxy-install-dir>/https-rps/config $../../bin/certutil -N -d . (if DBs do not exist already) $../../bin/certutil -S -d . -n Server-Cert-RP -s "CN=www.rps.com" -x -t "CT,CT,CT" Verify it with certutil : $../../bin/certutil -L -d <reverse-proxy-install-dir>/https-rps/config Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPIServer-Cert-RP CTu,Cu,Cu 1.2 Install Server Certificate in Origin Server instance I created self signed certificate in origin server. You can use Admin Server CLI to create a self signed certificate wadm>create-selfsigned-cert --config=origs --server-name=www.origs.com --nickname=Server-Cert-OSwadm>deploy-config origs Or you can use certutil followed by pull-config $cd <origin-server-install-dir>/https-rps/config $../../bin/certutil -N -d . (if DBs do not exist already) $../../bin/certutil -S -d . -n Server-Cert-OS -s "CN=www.origs.com" -x -t "CT,CT,CT" Verify it with certutil that the certificate exists : $../../bin/certutil -L -d <origin-server-install-dir>/https-origs/configCertificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert-OS CTu,Cu,Cu 1.3 Changes made in Reverse Proxy Web Server Instance 1.3.1 Run create-reverse-proxy CLI from Administration server wadm>create-reverse-proxy --config rps --vs rps --uri-prefix=/ --server=https://www.origs.com:4444 \*obj.conf gets modified as shown below : <Object name="default"> AuthTrans fn="match-browser" browser="\*MSIE\*" ssl-unclean-shutdown="true" NameTrans fn="map" from="/" name="reverse-proxy-/" to="/" ... </Object> <Object ppath="\*"> Service fn="proxy-retrieve" method="\*" </Object> <Object name="reverse-proxy-/"> Route fn="set-origin-server" server="https://www.origs.com:4444" </Object> In obj.conf I have configured reverse proxy in such a way that all requests are redirected to origin server. In real world situation you can if you want redirect only certain requests depending on your requirements. 1.3.2 Enable SSL in http-listener, set the server certificate nickname (if its different from "Server-Cert").  You can use Admin Server CLI to do these steps. Enable SSL for this listener, set the server certificate nickname and deploy config wadm> set-ssl-prop --config=rps --http-listener=http-listener-1 server-cert-nickname=Server-Cert-RP enabled=truewadm>deploy-config rps server.xml should get modified to look like <http-listener>     <name>http-listener-1</name>     <port>3333</port>     <server-name>www.rps.com</server-name>     <default-virtual-server-name>rps</default-virtual-server-name>    <ssl> <server-cert-nickname>Server-Cert-RP</server-cert-nickname> </ssl></http-listener> 1.4 Changes made in Origin Server Web Server Instance 1.4.1 Enable SSL in http-listener, set server certificate nickname (if its different from "Server-Cert") . You can use Admin Server CLI to do these steps. Enable SSL for this listener and set the server certificate nickname and deploy config. wadm> set-ssl-prop --config=origs --http-listener=http-listener-1       server-cert-nickname=Server-Cert-OS enabled=true wadm> deploy-config origs server.xml should get modified to look like <http-listener>     <name>http-listener-1</name>     <port>4444</port>     <server-name>www.origs.com</server-name>     <ssl>         <server-cert-nickname>Server-Cert-OS</server-cert-nickname>     </ssl>     <default-virtual-server-name>origs</default-virtual-server-name> </http-listener> 1.4.2 Create a test.html file in origin server : $cat ../docs/test.html This is test.html 1.5 Export Origin Server CA certificate and import it into Reverse Proxy Server NSS DBs $../../bin/certutil -L -d https-origs/config -n Server-Cert-OS -a | tee /tmp/os.cert $../../bin/certutil -A -n Server-Cert-OS -a -i /tmp/os.cert -d https-rps/config -t "CT,CT,CT" Reverse Proxy Server should have two certificates $../../bin/certutil -L -d . Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPIServerCertRP CTu,Cu,CuServerCertOS CT,C,C 1.6 Send a request through browser to Reverse Proxy Server test.html. Origin should serve the request. You can see the request is being served from origin server. We can check the entries in both the access logs of reverse proxy server and origin server to confirm. Note that browser may give a warning that this reverse proxy server is not issued by a valid CA. That's ok because its a self signed certificate. If you install a certificate from trusted CAs this message will not come up. 1.7 Attach ssltap to confirm the SSL communication I have attached ssltap in between origin server and reverse proxy server $ssltap -l -p 1924 -s origs:4444 Connection #1 [Mon Nov  8 13:46:48 2010] Connected to origs:4444--> [ (client to server) recordLen = 75 bytes (75 bytes of 75) [Mon Nov  8 13:46:48 2010] [ssl2]  ClientHelloV2 {            version = {0x03, 0x01}            cipher-specs-length = 48 (0x30)            sid-length = 0 (0x00)            challenge-length = 16 (0x10)            cipher-suites = {                 (0x010080) SSL2/RSA/RC4-128/MD5                 (0x030080) SSL2/RSA/RC2CBC128/MD5                 (0x0700c0) SSL2/RSA/3DES192EDE-CBC/MD5                 (0x060040) SSL2/RSA/DES56-CBC/MD5                 (0x020080) SSL2/RSA/RC4-40/MD5                 (0x040080) SSL2/RSA/RC2CBC40/MD5                 (0x000004) SSL3/RSA/RC4-128/MD5                 (0x00feff) SSL3/RSA-FIPS/3DESEDE-CBC/SHA                 (0x00000a) SSL3/RSA/3DES192EDE-CBC/SHA                 (0x00fefe) SSL3/RSA-FIPS/DES-CBC/SHA                 (0x000009) SSL3/RSA/DES56-CBC/SHA                 (0x000064) TLS/RSA-EXPORT1024/RC4-56/SHA                 (0x000062) TLS/RSA-EXPORT1024/DES56-CBC/SHA                 (0x000003) SSL3/RSA/RC4-40/MD5                 (0x000006) SSL3/RSA/RC2CBC40/MD5                 (0x0000ff) ????/????????/?????????/???                 }            session-id = { }            challenge = { 0x4fad 0xcb84 0x9b5a 0x07a1 ... } } ]<-- [ (server to client) (518 bytes of 513) SSLRecord { [Mon Nov  8 13:46:48 2010]    type    = 22 (handshake)    version = { 3,1 }    length  = 513 (0x201)    handshake {       type = 2 (server_hello)       length = 77 (0x00004d)          ServerHello {             server_version = {3, 1}             random = {...}             session ID = {                 length = 32                 contents = {...}             }             cipher_suite = (0x0004) SSL3/RSA/RC4-128/MD5             compression method = (00) NULL             extensions[5] = {               extension type 65281, length [1] = {    0: 00       | .               }             }          }       type = 11 (certificate)       length = 424 (0x0001a8)          CertificateChain {             chainlength = 421 (0x01a5)             Certificate {                size = 418 (0x01a2)                data = { saved in file 'cert.001' }             }          }       type = 14 (server_hello_done)       length = 0 (0x000000)    } } ] ... As you can see Origin Server sent one certificate cert.001 to Client(Reverse Proxy Server). $openssl x509 -in cert.001 -text -inform DERCertificate: Data:     Version: 3 (0x2) ...     Signature Algorithm: sha1WithRSAEncryption     Issuer: CN=www.origs.com     Validity         Not Before: Nov  8 08:09:20 2010 GMT         Not After : Feb  8 08:09:20 2011 GMT     Subject: CN=www.origs.com ... 2. Enable SSL on both Origin Server and Reverse Proxy Server, enable client-auth on Reverse Proxy Server - client/browser sends its certificate to Reverse Proxy Server AND Origin Server sends its certificate to Reverse Proxy server. Follow all the steps as step 1. 2.1 Enable client authentication on reverse proxy server  Just change the step # 1.4.1, set client authentication as "required" also: wadm> set-ssl-prop --config=rps --http-listener=http-listener-1 server-cert-nickname=Server-Cert-RP enabled=true client-auth=required server.xml of reverse proxy server should get modified to look like <http-listener> ...  <ssl> <server-cert-nickname>Server-Cert-RP</server-cert-nickname>   <client-auth>required</client-auth> </ssl></http-listener> 2.2 Test Client I used to send request to reverse proxy server I used "tstclnt which is bundled with NSS-NSPR binaries in bin directory. Created a test request file $cat > sslreq.datGET /test.html HTTP/1.0\^M \^M Create a client certificate and import it s CA as trusted CA (flag "C") in Reverse Proxy Instance NSS DBs. For faster results, I created client certificate in Reverse Proxy NSS DB itself $../bin/certutil -S -d https-rps/config -n ClientCert -s "CN=alpha" -x -t "CT,CT,CT" Sent request to the reverse proxy server. $tstclnt -h www.rps.com -p 3333  -n ClientCert -d https-rps/config -c n -v -o -f < sslreq.dat ... tstclnt: SSL version 3.1 using 128-bit RC4 with 160-bit SHA1 MAC tstclnt: Server Auth: 1024-bit RSA, Key Exchange: 1024-bit RSA subject DN: CN=www.rps.com issuer  DN: CN=www.rps.com ... HTTP/1.1 200 OK ... This is test.html... You can see the request is being served from origin server. I have used in this test client cipher "n" which is SSL3 RSA WITH RC4 128 SHA. I can see that cipher at the time of server startup when run in <log-level>finest</log-level>. You can verify that the ssltap output between origin Server and Reverse Proxy Server of this is same as section #1. 3. Enable SSL on both Origin Server and Reverse Proxy Server, enable client-auth on Origin Server, set ssl-client-config SAF on Reverse Proxy Server(OPTIONAL) - Origin Server and Reverse Proxy Server both send their certificates to each other. About ssl-client-config SAF Refer http://docs.sun.com/app/docs/doc/821-1827/gdhqy?l=en&a=view By default reverse proxy server doesn't send the certificate to the origin server. To send the Reverse Proxy Server certificate to the Origin Server, use "ssl-client-config" SAF and set "client-cert-nickname" parameter that takes the nickname of the client certificate to present to the Origin Server. Follow all the steps as section 1. 3.1 OPTIONAL : Add ssl-client-config SAF in obj.conf of Reverse Proxy Server - to set the nickname of the certificate it will send to Origin Server Modify obj.conf to have ssl-client-config SAF ObjectType fn="ssl-client-config" client-cert-nickname="Server-Cert-RP" Reverse Proxy Server will send any appropriate certificate it can find if this SAF ssl-client-config is not present. 3.2 Enable client-auth on Origin Server in which case Reverse Proxy Server will send certificate to Origin Server Just change the step #s 1.4.1 and set client authentication as "required" or "optional" on Origin Server : wadm> set-ssl-prop --config=origs --http-listener=http-listener-1 server-cert-nickname=Server-Cert-OS enabled=true client-auth=required server.xml of origin server should get modified to look like <http-listener>... <ssl><server-cert-nickname>Server-Cert-OS</server-cert-nickname> <client-auth>required</client-auth> </ssl> </http-listener> 3.3 Export Reverse Proxy Server CA certificate and import it into Origin Server NSS DB $../../bin/certutil -L -d https-rps/config -n Server-Cert-RP -a | tee /tmp/rp.cert $../../bin/certutil -A -n Server-Cert-RP -a -i /tmp/rp.cert -d https-origs/config -t "CT,CT,CT" Origin Server also should now have two certificates : $../../bin/certutil -L -d . Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPIServerCertOS CTu,Cu,CuServerCertRP CT,C,C 3.4 Send a test request and attach ssltap to see what's going on Now send a request through the browser. It should serve the page. Using ssltap we confirm that two certificates are being sent. In this case, Client is reverse proxy server and server is origin server. $ssltap -p 1924 -l -s origs:4444Connection #1 [Mon Nov 8 16:26:07 2010]Connected to origs:4444--> [ (client to server)recordLen = 75 bytes(75 bytes of 75) [Mon Nov 8 16:26:07 2010] [ssl2] ClientHelloV2 { version = {0x03, 0x01} cipher-specs-length = 48 (0x30) sid-length = 0 (0x00) challenge-length = 16 (0x10) cipher-suites = { (0x010080) SSL2/RSA/RC4-128/MD5 (0x030080) SSL2/RSA/RC2CBC128/MD5 (0x0700c0) SSL2/RSA/3DES192EDE-CBC/MD5 (0x060040) SSL2/RSA/DES56-CBC/MD5 (0x020080) SSL2/RSA/RC4-40/MD5 (0x040080) SSL2/RSA/RC2CBC40/MD5 (0x000004) SSL3/RSA/RC4-128/MD5 (0x00feff) SSL3/RSA-FIPS/3DESEDE-CBC/SHA (0x00000a) SSL3/RSA/3DES192EDE-CBC/SHA (0x00fefe) SSL3/RSA-FIPS/DES-CBC/SHA (0x000009) SSL3/RSA/DES56-CBC/SHA (0x000064) TLS/RSA-EXPORT1024/RC4-56/SHA (0x000062) TLS/RSA-EXPORT1024/DES56-CBC/SHA (0x000003) SSL3/RSA/RC4-40/MD5 (0x000006) SSL3/RSA/RC2CBC40/MD5 (0x0000ff) ????/????????/?????????/??? } session-id = { } challenge = { 0xe793 0x10b4 0xf9b2 0x826d 0... }}]<-- [ (server to client)(577 bytes of 572)SSLRecord { [Mon Nov 8 16:26:07 2010] type = 22 (handshake) version = { 3,1 } length = 572 (0x23c) handshake { type = 2 (server_hello) length = 77 (0x00004d) ServerHello { server_version = {3, 1} random = {...} session ID = { length = 32 contents = {...} } cipher_suite = (0x0004) SSL3/RSA/RC4-128/MD5 compression method = (00) NULL extensions[5] = { extension type 65281, length [1] = { 0: 00 | . } } } type = 11 (certificate) length = 424 (0x0001a8) CertificateChain { chainlength = 421 (0x01a5) Certificate { size = 418 (0x01a2) data = { saved in file 'cert.001' } } } type = 13 (certificate_request) length = 55 (0x000037) CertificateRequest { certificate types[3] = { 01 02 40 } certificate_authorities[49] = { CN=www.rps.com CN=www.origs.com } } type = 14 (server_hello_done) length = 0 (0x000000) }}]--> [ (client to server)(750 bytes of 702, with 43 left over)SSLRecord { [Mon Nov 8 16:26:07 2010] type = 22 (handshake) version = { 3,1 } length = 702 (0x2be) handshake { type = 11 (certificate) length = 430 (0x0001ae) CertificateChain { chainlength = 427 (0x01ab) Certificate { size = 424 (0x01a8) data = { saved in file 'cert.002' } } } type = 16 (client_key_exchange) length = 130 (0x000082) ClientKeyExchange { message = {...} } type = 15 (certificate_verify) length = 130 (0x000082) }}(750 bytes of 1, with 37 left over)SSLRecord { [Mon Nov 8 16:26:07 2010] type = 20 (change_cipher_spec) version = { 3,1 } length = 1 (0x1)}(750 bytes of 32)SSLRecord { [Mon Nov 8 16:26:07 2010] type = 22 (handshake) version = { 3,1 } length = 32 (0x20) < encrypted >}]... This shows that : origin server sends the certificate(cert.001) to the reverse proxy server and reverse proxy sends the certificate(cert.002) to the origin server.  Origin Server sent certificate cert.001(Server-Cert-OS) to Client(Reverse Proxy). and Client(Reverse Proxy Server) sent certificate cert.002(Server-Cert-RP) to Origin Server. $openssl x509 -in cert.001 -text -inform DERCertificate: Data:    Version: 3 (0x2) ...     Signature Algorithm: sha1WithRSAEncryption     Issuer: CN=www.origs.com     Validity         Not Before: Nov  8 08:09:20 2010 GMT         Not After : Feb  8 08:09:20 2011 GMT     Subject: CN=www.origs.com ... $openssl x509 -in cert.002 -text -inform DERCertificate: Data: Version: 3 (0x2)... Signature Algorithm: sha1WithRSAEncryption Issuer: CN=www.rps.com Validity Not Before: Nov 8 08:09:50 2010 GMT Not After : Feb 8 08:09:50 2011 GMT Subject: CN=www.rps.com...  4. Enable SSL on both Origin Server and Reverse Proxy Server, enable client-auth on Origin Server and Reverse Proxy Servers, set ssl-client-config SAF on Reverse Proxy Server(OPTIONAL) - client/browser sends its certificate to Reverse Proxy Server AND Origin Server and Reverse Proxy Server send certificates to each other Follow all the steps as step 1. 4.1 Enable client-auth on reverse proxy server  Just change the step # 1.4.1, set client authentication as "required" also: wadm> set-ssl-prop --config=rps --http-listener=http-listener-1 server-cert-nickname=Server-Cert-RP enabled=true client-auth=required server.xml of reverse proxy server should get modified to look like <http-listener> ...  <ssl> <server-cert-nickname>Server-Cert-RP</server-cert-nickname>   <client-auth>required</client-auth> </ssl></http-listener> 4.2  OPTIONAL : Add ssl-client-config SAF in obj.conf of Reverse Proxy Server - to set the nickname of the certificate it will send to Origin Server  Modify obj.conf to have ssl-client-config SAF  ...ObjectType fn="ssl-client-config" client-cert-nickname="Server-Cert-RP"... Reverse Proxy Server will send any appropriate certificate it can find if this SAF ssl-client-config is not present.  4.3  Enable client-auth on Origin Server in which case Reverse Proxy Server will send  certificate to Origin Server Just change the step #s 1.4.1 and set client authentication as "required" or "optional" on Origin Server : wadm> set-ssl-prop --config=origs --http-listener=http-listener-1 server-cert-nickname=Server-Cert-OS enabled=true client-auth=required server.xml of origin server should get modified to look like <http-listener>... <ssl><server-cert-nickname>Server-Cert-OS</server-cert-nickname> <client-auth>required</client-auth> </ssl> </http-listener> 4.4  Export Reverse Proxy Server CA certificate and import it into Origin Server NSS DBs $../../bin/certutil -L -d https-rps/config -n Server-Cert-RP -a | tee /tmp/rp.cert $../../bin/certutil -A -n Server-Cert-RP -a -i /tmp/rp.cert -d https-origs/config -t "CT,CT,CT" Verify that Origin Server also should now have two certificates $../../bin/certutil -L -d . Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPIServerCertOS CTu,Cu,CuServerCertRP CT,C,C 4.5 OPTIONAL : Changing Log Settings in server.xml :I have also modified access log format both in Reverse Proxy and in Origin Server <access-log>...    <format> %Ses->client.ip% "%Req->reqpb.clf-request%" %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% %Ses->client.cipher% %Req->vars.auth-cert% %Req->headers.proxy-auth-cert% </format></access-log> Note the contents in between <format></format> should be in one line I have put it in 2 lines so that its easy to see. 4.6 : Test Client used to send Request to Reverse Proxy ServerI used "tstclnt" which is bundled with NSS-NSPR binaries in bin directory. Created a test request file $cat > sslreq.datGET /test.html HTTP/1.0\^M \^M Create a client certificate and import it s CA as trusted CA (flag "C") in Reverse Proxy Instance NSS DBs. For faster results, I created client certificate in Reverse Proxy NSS DB itself $./bin/certutil -S -d https-rps/config -n ClientCert -s "CN=alpha" -x -t "CT,CT,CT" Now send a request to the Reverse Proxy Server $tstclnt -h www.rps.com -p 3333  -n ClientCert -d https-rps/config -c n -v -o -f < sslreq.dat ... tstclnt: SSL version 3.1 using 128-bit RC4 with 160-bit SHA1 MAC tstclnt: Server Auth: 1024-bit RSA, Key Exchange: 1024-bit RSA subject DN: CN=www.rps.com issuer  DN: CN=www.rps.com ... HTTP/1.1 200 OK ... This is test.html... You can see the request is being served from origin server. I have used in this test client cipher "n" which is SSL3 RSA WITH RC4 128 SHA. I can see that at the time of server startup when run in <log-level>finest</log-level>. You can use other ciphers as well. Access log of Reverse Proxy Web Server shows ONE certificate in auth-cert vars pblock $cat ../logs/accessformat=%Ses->client.ip% "%Req->reqpb.clf-request%" %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% %Ses->client.cipher% %Req->vars.auth-cert%%Req->headers.proxy-auth-cert% xxx.xxx.xxx.xxx "GET /test.html HTTP/1.0" 200 18RC4MIIBmDCCAQGgAwIBAgIFAJKp8cowDQYJKoZIhvcNAQEFBQAwEDEOMAwGA1UEAxMF...RiyrKq9SuMfqI8b++8613QMSJYwdVCFi1DrDGw==- Note cipher is RC4. You can verify that in Reverse Proxy Server instance, the certificate in pblock vars->auth-cert is the Client Certificate. $certutil -L -d https-rps/config -n ClientCert -a-----BEGIN CERTIFICATE-----MIIBmDCCAQGgAwIBAgIFAJKp8cowDQYJKoZIhvcNAQEFBQAwEDEOMAwGA1UEAxMF...RiyrKq9SuMfqI8b++8613QMSJYwdVCFi1DrDGw==-----END CERTIFICATE----- Access log of origin server  shows one certificate in vars->auth-cert and ONE certificate in Headers pblock in proxy-auth-cert: $cat ../logs/accessformat=%Ses->client.ip% "%Req->reqpb.clf-request%"%Req->srvhdrs.clf-status% %Req->srvhdrs.content-length%%Ses->client.cipher%%Req->vars.auth-cert%%Req->headers.proxy-auth-cert% xxx.xxx.xxx.xxx "GET /test.html HTTP/1.1" 200 18RC4 MIIBpDCCAQ2gAwIBAgIFAJKQxLEwDQYJKoZIhvcNAQEFBQAwFjEUMBIGA1UEAxML...L0wNBvqTkA7a4nagsj0UFRCUxfL1AjMRDJp6v7BGQRBcx2nrvgb+Bg== MIIBmDCCAQGgAwIBAgIFAJKp8cowDQYJKoZIhvcNAQEFBQAwEDEOMAwGA1UEAxMF...RiyrKq9SuMfqI8b++8613QMSJYwdVCFi1DrDGw== You can verify that in the Origin Server instance, the certificate in pblock vars->auth-cert is the Reverse Proxy Server Certificate and the certificate in pblock headers->proxy-auth-cert is the Client Certificate(browser/tstclnt sent to Reverse Proxy Server) . You can use certutil $certutil -L -d https-rps/config -n Server-Cert-RP -a-----BEGIN CERTIFICATE-----MIIBpDCCAQ2gAwIBAgIFAJKQxLEwDQYJKoZIhvcNAQEFBQAwFjEUMBIGA1UEAxML...L0wNBvqTkA7a4nagsj0UFRCUxfL1AjMRDJp6v7BGQRBcx2nrvgb+Bg==-----END CERTIFICATE-----OR you can use openssl to confirm$openssl s_client -connect www.rps.com:3333 -showcerts...Certificate chain 0 s:/CN=www.rps.com i:/CN=www.rps.com-----BEGIN CERTIFICATE-----MIIBpDCCAQ2gAwIBAgIFAJKQxLEwDQYJKoZIhvcNAQEFBQAwFjEUMBIGA1UEAxML...L0wNBvqTkA7a4nagsj0UFRCUxfL1AjMRDJp6v7BGQRBcx2nrvgb+Bg==-----END CERTIFICATE-----...  ssltap output between origin server and reverse proxy server will be same as section #3 above. References forward-auth-cert SAF Configuring Reverse Proxy in Sun Java System Web Server 7.0 Configuring reverse proxy in Sun Java System Web Server 7.0 when origin server is SSL enabled About trust flags of certificates in NSS database that can be modified by certutil http://forums.sun.com/thread.jspa?threadID=5397719 http://forums.sun.com/thread.jspa?threadID=5373182 http://forums.sun.com/thread.jspa?threadID=5359313 ssl-client-config

Two Way SSL in Oracle iPlanet Web Server 7.0 reverse proxy server and origin server Origin Server <--- HTTPS ---> Reverse Proxy Server <--- HTTPS ---> client/Browser There are various SSL and non SSL...

Building Open Web Server on OpenSolaris SPARC

I tried building Open Web Server on OpenSolaris SPARC. Install package as given in http://wikis.sun.com/display/wsFOSS/Checkout+and+Build+Instructions SUNWmozldap, SUNWxercesc, SUNWxalanc in http://src.opensolaris.org/source/xref/webstack/ were probably not built for 64 bit. I have only tested on OpenSolaris x86 32 bit. They do not work on SPARC yet.  Building Mozilla LDAP C SDK for 64 bit Run this script it builds and puts Mozilla LDAP C SDK and puts it in /usr/local/include/mozldap/, /usr/local/lib/mozldap, /usr/local/lib/mozldap/64 #!/bin/sh #building 32 bit cdcvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co -P -rLDAPCSDK_6_0_5_RTM DirectorySDKSourceC cd mozilla/directory/c-sdk ./configure --with-sasl --with-nspr-inc=/usr/include/mps --with-nspr-lib=/usr/lib/mps --with-nspr --with-nss-inc=/usr/include/mps --with-nss-lib=/usr/lib/mps --with-nss gmake cd ../../dist/ sudo mkdir -p /usr/local/include/mozldap /usr/local/lib/mozldap sudo cp public/ldap/\* /usr/local/include/mozldap/ sudo cp lib/\* /usr/local/lib/mozldap/ cd mv mozilla mozilla32#building 64 bit cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co -P -rLDAPCSDK_6_0_5_RTM DirectorySDKSourceC cd mozilla/directory/c-sdk ./configure --with-sasl --with-nspr-inc=/usr/include/mps --with-nspr-lib=/usr/lib/mps/64 --with-nspr --with-nss-inc=/usr/include/mps --with-nss-lib=/usr/lib/mps/64 --with-nss --enable-64bit gmake cd ../../dist/ sudo mkdir -p /usr/local/lib/mozldap/64 sudo cp lib/\* /usr/local/lib/mozldap/64 cd mv mozilla mozilla64 Building Xerces and Xalanc for 64 bit Build Xerces C and Xalan C 32 bit and 64 bit and put it in /usr/local/ area. Build Xerces C 2.6 from http://xerces.apache.org/xerces-c/build-2.html Build Xalan C  1.9 from http://xml.apache.org/xalan-c/build_instruct.html Check Out Open Web Server hg clone ssh://anon@hg.opensolaris.org/hg/webstack/webserver Patch these diffs Building Open Web Server gmake BUILD_VARIANT=OPTIMIZEDIf you want 64 bit support also gmake BUILD_VARIANT=OPTIMIZED BUILD64=1 Installing Open Web Server gmake BUILD_VARIANT=OPTIMIZED install If you want 64 bit also gmake BUILD_VARIANT=OPTIMIZED install BUILD64=1  Go to work/B1/\*/https-test/config directory. To run Open Web Server in 64 bit you need to add <platform>64</platfrom> in server.xml "server" element. If you get error that looks like ld: fatal: file /usr/lib/64/libsqlite3.so: version `SQLITE_3' does not exist:        required by file /usr/lib/mps/64/libsoftokn3.so$mv /usr/lib/64/libsqlite3.so /usr/lib/64/libsqlite3.so.BACK References http://wikis.sun.com/display/wsFOSS/Checkout+and+Build+Instructions http://forums.sun.com/thread.jspa?messageID=10810336#10810336

I tried building Open Web Server on OpenSolaris SPARC. Install package as given in http://wikis.sun.com/display/wsFOSS/Checkout+and+Build+Instructions SUNWmozldap, SUNWxercesc, SUNWxalanc in http://src....

Running Open Web Server on FreeBSD

Running Open Web Server on FreeBSD Downloaded Free BSD fromftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/ISO-IMAGES/7.2/7.2-RELEASE-i386-dvd1.iso.gz and installed it. For reference, see http://wikis.sun.com/display/wsFOSS/Checkout+and+Build+Instructions To search for a package, I used cd /usr/ports; make searchname="mercurial" and so on. First make sure that you already have these components GNU make 3.81 /usr/local/bin/gmake C/C++ Compiler 4.2.1 /usr/bin/g++ and gcc Zlib  /usr/lib/libz.so already installed Perl 5.8.9 /usr/bin/perl CVS  1.11.22.1 /usr/bin/cvs Install/Build these Components Mercurial cd /usr/ports/devel/mercurial; make install Java SE 5 or 6 Due to license problems had to manually download and puttingit in /usr/ports/distfiles For JDK 5 : cd /usr/posts/java/jdk15; make install and download thefollowing files into /usr/ports/distfiles http://www.java.net/download/tiger/tiger_u16/jdk-1_5_0_16-fcs-bin-b02-jrl-28_may_2008.jar http://www.java.net/download/tiger/tiger_u16/jdk-1_5_0_16-fcs-src-b02-jrl-28_may_2008.jar tzupdater-1_3_12-2009a.zip fromhttp://www.filewatcher.com/m/tzupdater-1_3_12-2009a.zip.261842.0.0.html Java(TM) Cryptography Extension (JCE) Unlimited StrengthJurisdiction Policy Files 5.0 - jce_policy-1_5_0.zip http://www.eyesbeyond.com/freebsddom/java/jdk15.html -bsd-jdk15-patches-9.tar.bz2 For JDK 6 : cd /usr/posts/java/diablo-jdk16; make install and download the following files into /usr/ports/distfiles Java Cryptography Extension (JCE) Unlimited StrengthJurisdiction Policy Files 6 jce_policy-6.zip Java SE Timezone Updater 1.3.15 -tzupdater-1_3_15-2009g.zip http://www.freebsdfoundation.org/cgi-bin/download?download=diablo-caffe-freebsd7-i386-1.6.0_07-b02.tar.bz2 Ant  cd /usr/ports/devel/apache-ant; make install NSPR cd /usr/ports/devel/nspr; make install NSS cd /usr/ports/security/nss; make install Xerces C++ cd /usr/ports/textproc/xerces-c2; make install cd /usr/ports/textproc/xerces-c2-devel; make install selected Use ICU transcoder option. Build thread safe versionof library.Got an error "/usr/bin/ld: can not find-lgcc_pic" For this we need to manually modify "files/filepatch-src-xercesc-Makefile.incl" and change "-lgcc_pic" to "-lgcc" Xalan C++ cd/usr/ports/textproc/xalan; make install Selected INMEN andTRANSCODER_ICU options. PCRE cd /usr/ports/devel/pcre; make install ICU cd /usr/ports/devel/icu; make install SASL cd /usr/ports/security/cyrus-sasl2; make install Mozilla LDAP C SDK\* cd /usr/ports/convertors/libiconv; make install cd /usr/ports/convertors/iconv; make install cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co -P -rLDAPCSDK_6_0_5_RTM DirectorySDKSourceC cd mozilla Add this patch Index: directory/c-sdk/configure===================================================================RCS file: /cvsroot/mozilla/directory/c-sdk/configure,vretrieving revision 5.65diff -u -r5.65 configure--- directory/c-sdk/configure   17 Sep 2007 17:46:23 -0000      5.65+++ directory/c-sdk/configure   22 Jul 2009 07:27:11 -0000@@ -4362,12 +4362,7 @@ EOF     CFLAGS="$CFLAGS $(DSO_CFLAGS) -ansi -Wall"-    MOZ_OBJFORMAT=`test -x /usr/bin/objformat && /usr/bin/objformat || echo aout`-    if test "$MOZ_OBJFORMAT" = "elf"; then-        DLL_SUFFIX=so-    else-        DLL_SUFFIX=so.1.0-    fi+   DLL_SUFFIX=so     DSO_CFLAGS=-fPIC     DSO_LDOPTS=-Bshareable     ;;Index: directory/c-sdk/ldap/libraries/libldap/Makefile.in===================================================================RCS file: /cvsroot/mozilla/directory/c-sdk/ldap/libraries/libldap/Makefile.in,vretrieving revision 5.23diff -u -r5.23 Makefile.in--- directory/c-sdk/ldap/libraries/libldap/Makefile.in  20 Jun 2007 17:57:11 -0000      5.23+++ directory/c-sdk/ldap/libraries/libldap/Makefile.in  22 Jul 2009 07:27:12 -0000@@ -267,6 +267,10 @@ EXTRA_LIBS = -L$(dist_libdir) -l$(LBER_LIBNAME) -pthread endif+ifeq ($(OS_ARCH), FreeBSD)+EXTRA_LIBS += -L$(dist_libdir) -l$(LBER_LIBNAME) -L/usr/lib -lcompat+endif+ ifeq ($(HAVE_SASL), 1) EXTRA_LIBS += $(SASL_LINK) endif cd directory/c-sdk ./configure --with-sasl-inc=/usr/local/include/sasl        --with-sasl-lib=/usr/local/lib          --with-nspr-lib=/usr/local/lib --with-nspr-inc=/usr/local/include/nspr/ --with-nspr          --with-nss-lib=/usr/local/lib/nss --with-nss-inc=/usr/local/include/nss/nss/ --with-nss gmake cd ../../dist/ sudo mkdir /usr/local/include/mozldap /usr/local/lib/mozldap sudo cp public/ldap/\* /usr/local/include/mozldap/ sudo cp lib/\* /usr/local/lib/mozldap/ \*Note  There is one /usr/ports/net/ldapsdk. It downloads and builds ldapsdk_12311998.tar.gz. But I can see it in ancient now from mozilla site http://ftp.mozilla.org/pub/mozilla.org/directory/c-sdk/ancient/ So I built Mozilla C SDK myself. Hack IN NSPR One manual hack you need to do in "/usr/local/include/prinet.h"  #ifdefined(FREEBSD) ||defined(BSDI) || defined(QNX)#include <rpc/types.h> /\* the only place that definesINADDR_LOOPBACK \*/#endif replace it by #ifndefINADDR_LOOPBACK #defineINADDR_LOOPBACK        (u_long)0x7F000001 #endif Building Open Web Server /usr/local/bin/hg clonessh://anon@hg.opensolaris.org/hg/webstack/webserver cd webserver DOWNLOAD AND PATCH THESE DIFFS gmake BUILD_VARIANT=OPTIMIZED gmake BUILD_VARIANT=OPTIMIZED install Go to work/FreeBSD7.2-RELEASE_OPT.OBJ/https-test/bin/ directory and start the server instance using startserv script. For the last two gmake commands, you can add WS_INSTALL_ROOT=/opt/ws if you want the server instance to be installed in /opt/ws/https-test References: http://wikis.sun.com/display/wsFOSS/Checkout+and+Build+Instructions http://www.freebsd.org/cgi/cvsweb.cgi/ports/textproc/xerces-c2-devel/files/patch-src-xercesc-Makefile.incl?rev=1.2;content-type=text%2Fplain https://bugzilla.mozilla.org/show_bug.cgi?id=505418 https://bugzilla.mozilla.org/show_bug.cgi?id=504248 http://www.freebsd.org/cgi/query-pr.cgi?pr=136984

Running Open Web Server on FreeBSD Downloaded Free BSD from ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/ISO-IMAGES/7.2/7.2-RELEASE-i386-dvd1.iso.gz and installed it. For reference, see http://wikis.su...

Running Open Web Server on MacOS

Running Open Web Server on MacOS  I tried to run Open Web Server on iMac with Mac OS X 10.5.7 on it. Install http://svn.macports.org/repository/macports/downloads/MacPorts-1.7.1/MacPorts-1.7.1-10.5-Leopard.dmg as per http://www.macports.org/install.php Already existing Components To build Open Web Server, first we need to have a few components. I already have /usr/bin/make, JAVA SE(/usr/bin/javac etc.), /usr/bin/ant, C/C+ compiler(/usr/bin/gcc and /usr/bin/g++),/usr/bin/perl, /usr/bin/cvs, /usr/lib/libz.\*, /usr/lib/libsasl2.\*, installed. Double check these component versions as per the table given in http://wikis.sun.com/display/wsFOSS/Checkout+and+Build+Instructions#CheckoutandBuildInstructions-RequiredLibrariesandTools Install Mercurial, NSS, NSPR, Xerces C++, Xalan C++, PCRE, ICU sudo port install mercurial nspr nss pcre icu xercesc xalanc Building and Installing Mozilla LDAP C SDK cvs -d:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co -P-rLDAPCSDK_6_0_5_RTM DirectorySDKSourceC cd mozilla/directory/c-sdk ./configure --with-sasl --with-nspr-inc=/opt/local/include/nspr --with-nspr-lib=/opt/local/lib/nspr --with-nspr --with-nss-inc=/opt/local/include/nss --with-nss-lib=/opt/local/lib/nss --with-nss make cd ../../dist/ sudo mkdir /usr/local/include/mozldap /usr/local/lib/mozldap sudo cp public/ldap/\* /usr/local/include/mozldap/ sudo cp lib/\* /usr/local/lib/mozldap/ Building Open Web Server cd hg clonessh://anon@hg.opensolaris.org/hg/webstack/webserver cd webserver Download and apply this patch make make install cdwork/Darwin9.7.0_DBG.OBJ/https-test/bin Manually substitute in startservscript, LD_LIBARAY_PATHto DYLD_LIBRARY_PATHbefore starting the server. Make similar changes in all the scripts like stopserv. ./startserv As you can see the server starts up on port 8080. Note that I am using "_xpatomic_locked.h" for now. Their performance needs to be optimized using functions in /usr/include/libkern/OSAtomic.h. NSPR's PR_GetLibraryFilePathname dumps if called with NULL as first argument on MAC so added a hack of PR_GetLibraryFilePathname("ns-httpd40"..). Some filenames are ugly like defines___.mk will fix it later. If you see entries in error log like : failure : HTTP3360: connection limit (1) exceeded.PollManager::RequestReservation() keep-alive subsystem full Run Web Server in finest log level ( set <log-level>finest</log-level> in server/xml). And look for messages like fine: operating system file descriptor limit is 256fine: allocating 1 file descriptors to the connection queue, 1 filedescriptors to keep-alive connections, and 1 file descriptors to thefile cache fine: 130 connection maximum (1 queued, 128 active, 1 keep-alive) fine: HTTP3066: HTTP keep-alive subsystem will accomodate up to1 connections $ulimit -aopen files                      (-n) 256... Now I changed it to 1024 :$ulimit -n 1024 $./https-test/bin/startservfine: operating system file descriptor limit is 1024fine: allocating 128 file descriptors to the connection queue, 128 filedescriptors to keep-alive connections, and 8 file descriptors to thefile cachefine: 384 connection maximum (128 queued, 128 active, 128 keep-alive) fine: HTTP3066: HTTP keep-alive subsystem will accomodate up to 128connections References Build instructions for Open Web Server https://bugzilla.mozilla.org/show_bug.cgi?id=504893 http://developer.apple.com/documentation/Darwin/Reference/ManPages/man3/barrier.3.html http://devworld.apple.com/technotes/tn2002/tn2071.html http://lists.apple.com/archives/unix-porting/2002/Sep/msg00021.html http://dev.eclipse.org/newslists/news.eclipse.tools.cdt/msg16863.html http://blogs.sun.com/Janice/entry/http3360_connection_limit_1_exceeded

Running Open Web Server on MacOS  I tried to run Open Web Server on iMac with Mac OS X 10.5.7 on it. Install http://svn.macports.org/repository/macports/downloads/MacPorts-1.7.1/MacPorts-1.7.1-10.5-Leopa...

Installing Sun Java System Web Server 7.0 on CentOS 5.3 or Fedora 10-11 or Ubuntu

Installing Sun Java System Web Server 7.0 on CentOS 5.3 or Fedora 10/11 or Ubuntu In my free time, I just tried out Sun Java System Web Server 7.0 update 5 on CentOS 5.3 and Fedora10 and 11 and Ubuntu. Joe Mccabe has also written a similar blog on this Download Sun Java System Web Server 7.0 for Linux and extract the contents of tar.gz file $gunzip  sjsws-7_0u5-linux-i586.tar.gz; tar -xvf sjsws-7_0u5-linux-i586.tar Run installer, it will fail $./setup error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or directory CentOS To fix this problem  install "compat-libstdc++-33" package as shown below $sudo yum -y install compat-libstdc++-33 Now run setup, it will work fine. Fedora 10/11Install "compat-libstdc++-33" package as shown below $sudo yum -y install compat-libstdc++-33 There is one more problem in Fedora 11 but is I heard fixed in 7.0 update 6 (onwards). If you get error message like lib/libfreebl3.so: version `NSSRAWHASH_3.12.3' not found (required by /lib64/libcrypt.so.1) You need workaround as given in https://forums.oracle.com/forums/thread.jspa?threadID=2212337 Ubuntu You need to install libstdc++5 as shown below $sudo apt-get install libstdc++5 If you get an error which has something that looks like /bin/domainname not found, you need to install $sudo apt-get install nis I have tested on Ubuntu 9.04. *Note that Sun Java System Web Server 7.0 update 5 or 6 is not officially certified or supported on CentOS or Fedora or Ubuntu, but here's how you can make it work. References https://wikis.oracle.com/display/WebServer/Installing+on+Ubuntu http://jmccabe.org/blog/CentOS_WebServer_Install http://ubuntuforums.org/showthread.php?t=855603 https://forums.oracle.com/forums/thread.jspa?threadID=2212337 https://blogs.oracle.com/kkranz/entry/installing_sun_java_system_web Outdated links : http://forums.sun.com/thread.jspa?messageID=10769043#10769043 http://www.sun.com/download/index.jsp?cat=Web%20%26%20Proxy%20Servers&tab=3&subcat=Web%20Servers http://forums.sun.com/thread.jspa?threadID=5396968 http://blogs.sun.com/kkranz/entry/installing_sun_java_system_web

Installing Sun Java System Web Server 7.0 on CentOS 5.3 or Fedora 10/11 or Ubuntu In my free time, I just tried out Sun Java System Web Server 7.0 update 5 on CentOS 5.3 and Fedora10 and 11 and Ubuntu....

About trust flags of certificates in NSS database that can be modified by certutil

Managing Certificate Trust flags in NSS Database We can modify certificate trust flags using certutil. But before we do so we must know more about these trust flags. Here are my notes about trust flags from  Nelson Bolyard 's Brown bag: There are three available trust categories for each certificate, expressed in this order: "SSL, email, object signing". In each category position use zero or more of the following attribute codes c,C,p,PT,u,w. The attribute codes for the categories are separated by commas.  These trust flags are allow overrides, they tell NSS that it is ok touse certificates for this purpose even though they may not seem like itis. There are two categories : Validity override and trust overrides. Flags that apply to CA Certificatesc - its for validity override - It tells even though this certificate doesn't look like a CAcertificate - In version 1 certificates like old root CA certificates (that predated X509 v3) its necessary to set this.C trusted CA flag implies c - This certificate should be treated like aCA certificate but also should be treated as root certificate we trust.By default roots arenot trusted. We can take any certificate and mark it with a  "C" so itwill be treated as aroot CA certificate. When NSS clients/server validates certificate chain we can stop right there. NSSwill build chain to send out as far as it reaches a root CA i.e. whenit sees "C" flag. So intermediate CA certificates should not have this trustflag "C". Flags that apply to end entity certificates like Server certificates or Clientcertificates. certificates that are not CA certificates : p - valid peer flag - even though the certificate doesn't look like apeer cert, treat it like a peer cert.P - Trusted Peer flag - implies p - This is a peer certificate & I want it take it at face value. Iwant uto trust this cert. Don't even bother to look & see if is issued bythe issuer that you know and we are going to trust this certificate justbyitself and so in the world of self signed server certificates its sometimesnecessary to set this trusted peer flag in the client so the clientwill trust the certificate when it comes from the server. T - special trust flag that is used in SSL Column only, it is used onlyonthe server. It is not used on the client. It tells the server thatthis certificate is one whose name it should send out when it requestsclientauthentication. When a server requests a client to authenticate itselfwith acert, the server sends out a list of names of certificates that are issuersfrom whom it is willing to accept certificates. It figures out thenames it should sent out because it looks for certificates with this "T" flag,those are the names it sends out to the remote client. Flag that can not be set by certutilu - User flag - This is  not a trust flag and this is not a flag that youcan set withcertutil, this is a dynamic flag it says that NSS has discovered thatNSS has the private key associated with this cert. That's essential forserver certificate. If you are a server and you are trying to send out yourserver certificate and its chain you have to have the private key associatedwith this server cert. This is something to look for.

Managing Certificate Trust flags in NSS Database We can modify certificate trust flags using certutil. But before we do so we must know more about these trust flags. Here are my notes about trust...

Using DTrace in Sun Java System Web Server - part 2

Using DTrace in Sun Java System Web Server - part 2 In my previous blog I was just tracing the flow of request processing, in this blog I have tried to see how much time it takes when <If>/ <Client> tags etc. are evaluated. Add in obj.conf the following lines to disable accelerator cache AuthTrans fn="log" level="fine" message="Accelerator cache disabled" Add one "If" condition in obj.conf <If $uri =~ "/test.html"> Service method="(GET|HEAD|POST)" type="\*~magnus-internal/\*"fn="send-file" </If> Check webservd's pid $ps -eaf | grep webservd | grep highest pid Run this DTrace script, send a request to GET /test.html HTTP/1.0 via telnet from another window. Here is the output : $./ws.d highest-pidthread 24: Added connection 0x29b3228 to connectionQ thread 29: Removed connection 0x29b3228 from connectionQ. Time spent in Q 81250 nanosecondsthread 29: Processing for absolute path /test.html client ip xxx.xxx.xxx.xxxthread 29: processing objects for uri /test.htmlthread 29: object-check calledthread 29: object-check finished. Time spent 8584 nanosecondsthread 29: Calling saf match-browserthread 29: saf match-browser returned -2. Time spent is 66916 nanosecondsthread 29: object-check calledthread 29: object-check finished. Time spent 2917 nanosecondsthread 29: Calling saf ntrans-j2eethread 29: saf ntrans-j2ee returned -2. Time spent is 26084 nanosecondsthread 29: object-check calledthread 29: object-check finished. Time spent 2417 nanosecondsthread 29: Calling saf pfx2dirthread 29: saf pfx2dir returned -2. Time spent is 21500 nanosecondsthread 29: object-check calledthread 29: object-check finished. Time spent 3500 nanosecondsthread 29: Calling saf uri-cleanthread 29: saf uri-clean returned 0. Time spent is 21750 nanosecondsthread 29: object-check calledthread 29: object-check finished. Time spent 2500 nanosecondsthread 29: Calling saf find-pathinfothread 29: saf find-pathinfo returned -2. Time spent is 10750 nanosecondsthread 29: object-check calledthread 29: object-check finished. Time spent 2250 nanosecondsthread 29: Calling saf find-index-j2eethread 29: saf find-index-j2ee returned -2. Time spent is 8917 nanosecondsthread 29: object-check calledthread 29: object-check finished. Time spent 2250 nanosecondsthread 29: Calling saf find-indexthread 29: saf find-index returned -2. Time spent is 9000 nanosecondsthread 29: object-check calledthread 29: object-check finished. Time spent 2750 nanosecondsthread 29: Calling saf type-j2eethread 29: saf type-j2ee returned 0. Time spent is 11584 nanosecondsthread 29: object-check calledthread 29: object-check finished. Time spent 2334 nanosecondsthread 29: Calling saf type-by-extensionthread 29: saf type-by-extension returned 0. Time spent is 38750 nanosecondsthread 29: object-check calledthread 29: object-check finished. Time spent 2833 nanosecondsthread 29: Calling saf force-typethread 29: saf force-type returned 0. Time spent is 10833 nanosecondsthread 29: cond-eval calledthread 29: expr-eval calledthread 29: expr-compiled-re-eval subject /test.html match 1 pattern /test.html, return value 1thread 29: expr-eval finished. Time spent 65584 nanosecondsthread 29: cond-eval finished. Time spent 111500 nanosecondsthread 29: Calling saf send-filethread 29: saf send-file returned 0. Time spent is 102583 nanosecondsthread 29: object-check calledthread 29: object-check finished. Time spent 3166 nanosecondsthread 29: Calling saf flex-logthread 29: saf flex-log returned 0. Time spent is 47833 nanoseconds This shows how much time was spent in each SAF, when connection was put in connection queue and when it was removed and which regular expression in obj.conf was evaluated, how much time was spent, what was the result. Attached are the cvs diffs and ws.d script  Summary These DTrace probes will help us to dynamically find out Which request processing stages were called for a request,  how much time each request processing stage took for a particular request. When was a connection put into connection queue (Requests in Web server are accepted by acceptor threads and put in connection queue and picked up by Worker threads) so if the requests sit for a long time in connection queue, then worker threads settings in server.xml configuration file should be increased accordingly. Which regular expression in obj.conf was evaluated, how much time was spent, what was the result.

Using DTrace in Sun Java System Web Server - part 2 In my previous blog I was just tracing the flow of request processing, in this blog I have tried to see how much time it takes when <If>/...

Tracing the flow of request processing using DTrace in Sun Java System Web Server

Tracing the flow of request processing using DTrace in Sun Java System Web Server I am trying to add basic DTrace probes in Sun Java System Web Server 7.0.I will start by adding probes that let us know which SAF got executedat run time. Let me know if you have any ideas. I start the server instance and run this script ws.d and send a few requests : $./ws.d highest-pid thread 29: Calling saf match-browserthread 29: saf match-browser returned -2thread 29: Calling saf ntrans-j2eethread 29: saf ntrans-j2ee returned -2thread 29: Calling saf pfx2dirthread 29: saf pfx2dir returned -2thread 29: Calling saf uri-cleanthread 29: saf uri-clean returned 0thread 29: Calling saf find-pathinfothread 29: saf find-pathinfo returned -2thread 29: Calling saf find-index-j2eethread 29: saf find-index-j2ee returned -2thread 29: Calling saf find-indexthread 29: saf find-index returned -2thread 29: Calling saf type-j2eethread 29: saf type-j2ee returned 0thread 29: Calling saf type-by-extensionthread 29: saf type-by-extension returned 0thread 29: Calling saf force-typethread 29: saf force-type returned 0thread 29: Calling saf send-filethread 29: Calling saf insert-filterthread 29: saf insert-filter returned -2thread 29: saf send-file returned 0thread 29: Calling saf flex-logthread 29: saf flex-log returned 0thread 32: Calling saf match-browserthread 32: saf match-browser returned -2thread 32: Calling saf ntrans-j2eethread 32: saf ntrans-j2ee returned -2thread 32: Calling saf pfx2dirthread 32: saf pfx2dir returned -2thread 32: Calling saf uri-cleanthread 32: saf uri-clean returned 0thread 32: Calling saf find-pathinfothread 32: saf find-pathinfo returned -2thread 32: Calling saf find-index-j2eethread 32: saf find-index-j2ee returned -2thread 32: Calling saf find-indexthread 32: saf find-index returned -2thread 32: Calling saf type-j2eethread 32: saf type-j2ee returned 0thread 32: Calling saf type-by-extensionthread 32: saf type-by-extension returned 0thread 32: Calling saf force-typethread 32: saf force-type returned 0thread 32: Calling saf send-filethread 32: Calling saf insert-filterthread 32: saf insert-filter returned -2thread 32: saf send-file returned 0thread 32: Calling saf flex-logthread 32: saf flex-log returned 0 If you want to call this only for a particular SAF lets say send-file modify ws.d to ws-saf.d $./ws-saf.d 16186thread 29: Calling saf send-filethread 29: saf send-file returned 0thread 29: Calling saf send-filethread 29: saf send-file returned 0thread 29: Calling saf send-filethread 29: saf send-file returned 0thread 29: Calling saf send-filethread 29: saf send-file returned 0thread 29: Calling saf send-filethread 29: saf send-file returned 0thread 29: Calling saf send-filethread 29: saf send-file returned 0thread 29: Calling saf send-filethread 29: saf send-file returned 0... I added these 2 files and these cvs diffs iniplanet/ias/server/src/cpp/iws/netsite/lib/frame/ directory andcompiled Web Server gmake all publish; cd ../httpd/src; gmake clobberall publish. Note that I have kept DTrace probes signature as someimportantparameters, followed by Pblock, Session, Request structure pointersjust incase at any point of time we need to find out values of some of these. sjsws.dsjsws.hdiff.txt

Tracing the flow of request processing using DTrace in Sun Java System Web Server I am trying to add basic DTrace probes in Sun Java System Web Server 7.0.I will start by adding probes that let us know...

Running Sun Web Server 7.0 in FIPS mode

Running Sun Java System Web Server 7.0 in FIPS mode Jyri's blog  http://blogs.sun.com/jyrivirkki/entry/fips_140_certification talks about FIPS certification. In this blog I will show how to run Web Server 7.0 in FIPS mode. Its a piece of cake. Create a Web Server instance called https-test on test.sun.com. $cd /export1/ws/https-test/configEdit server.xml to have <ssl/> element in the listener. $rm \*.db Initliaze NSS Database with password "test" lets say. This is necessary because if you enable FIPS using modutil, it won't allow us to use NSS database without password. $../../bin/certutil -N -d . Enter a password which will be used to encrypt your keys. The password should be at least 8 characters long, and should contain at least one non-alphabetic character. Enter new password: test Re-enter password: test Create a self-signed certificate (for ease of use). You can also generate CSR and get server certificate from CA and install it in Web Server. $../../bin/certutil -S -n "Server-Cert"-s "CN=test.sun.com" -x -t "CP,CP,CP" -d . Enter Password or Pin for "NSS Certificate DB": test Enter Password or Pin for "NSS Certificate DB": test A random seed must be generated that will be used in the creation of your key.  One of the easiest ways to create a random seed is to use the timing of keystrokes on a keyboard. To begin, type keys on the keyboard until this progress meter is full.  DO NOT USE THE AUTOREPEAT FUNCTION ON YOUR KEYBOARD! Continue typing until the progress meter is full: |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*| Finished.  Press enter to continue: Generating key.  This may take a few moments... Start the server it works. $../bin/startserv Sun Java System Web Server 7.0U4 B10/20/2008 12:12 Please enter the PIN for the "internal" token: test info: CORE5076: Using [Java HotSpot(TM) Server VM, Version1.5.0_15] from [Sun Microsystems Inc.] info: HTTP3072: http-listener-1:https://test.sun.com:3333 ready to accept requests info: CORE3274: successful server startup $../bin/stopserv server has been shutdown Enable FIPS using modutil : $su Password: #/usr/sfw/bin/modutil -fips true  -dbdir . WARNING: Performing this operation while the browser is runningcould cause corruption of your security databases. If the browser iscurrently running, you should exit browser before continuing this operation. Type 'q <enter>' to abort, or <enter> to continue: Using database directory .... FIPS mode enabled. # control D to come back to normal userStart the server. It works !! $../bin/startserv Sun Java System Web Server 7.0U4 B10/20/2008 12:12 Please enter the PIN for the "NSS FIPS 140-2 Certificate DB"token: test security: CORE1296: Token associated with server certificate[Server-Cert] cannot support PKCS#11 bypass security: CORE1295: PKCS#11 bypass has been disabled because thecurrent configuration cannot support bypass info: CORE5076: Using [Java HotSpot(TM) Server VM, Version1.5.0_15] from [Sun Microsystems Inc.] info: HTTP3072: http-listener-1:https://test.sun.com:3333 ready to accept requests info: CORE3274: successful server startup Note If you are using Admin GUI/CLI you need to use pull-config CLI to pull these changes into Admin Server. If you're running in FIPS mode, you need to set a password. It's a FIPS requirement that needs to be enforced when FIPS is enalbed by any FIPS compliant module. In FIPS compliant mode you're password also should be of the form: - The password must be at least 7 characters long.- The password must consist of characters from three or more character classes.  We define five character classes: digits (0-9), ASCII lowercase letters, ASCII uppercase letters, ASCII non-alphanumeric characters (such as space and punctuation marks), and non-ASCII characters.  If an ASCII uppercase letter is the first character of the password, the uppercase letter is not counted toward its character class.  Similarly, if a digit is the last character of the password, the digit is not counted toward its character class. If you are interested in FIPS compliance please review the security policy: http://csrc.nist.gov/groups/STM/cmvp/documents/140-1/140sp/140sp814.pdf

Running Sun Java System Web Server 7.0 in FIPS mode Jyri's blog  http://blogs.sun.com/jyrivirkki/entry/fips_140_certificationtalks about FIPS certification. In this blog I will show how to run Web...

DTrace script to collect information about cipher suites used

DTrace script to collect information about cipher suites used Given below is a dtrace script I have to trace SSL calls. Running this script on a Web Server instance (32 bit ) pid lets say in our case is 9149. Sending some SSL requests on to this server : There are two ways to run this log=normal and log=verbose. Pressing control C returns the statistical data. $./ssltop.d 9149 log=normal t@26: 2008 Oct 30 16:50:56: 129.158.224.109 Connection created t@26: 2008 Oct 30 16:50:56: ListenSocket::accept called t@30: 2008 Oct 30 16:50:56: 129.158.224.109 Negotiated cipher RC4 t@33: 2008 Oct 30 16:51:06: 129.158.224.109 Negotiated cipher AES-256 t@32: 2008 Oct 30 16:51:06: 129.158.224.109 Negotiated cipher AES-256 t@26: 2008 Oct 30 16:51:06: 129.158.224.109 Connection created t@26: 2008 Oct 30 16:51:06: ListenSocket::accept called t@33: 2008 Oct 30 16:51:06: 129.158.224.109 Negotiated cipher AES-256 t@32: 2008 Oct 30 16:51:06: 129.158.224.109 Negotiated cipher AES-256 t@33: 2008 Oct 30 16:51:06: 129.158.224.109 Negotiated cipher AES-256 t@32: 2008 Oct 30 16:51:06: 129.158.224.109 Negotiated cipher AES-256 t@33: 2008 Oct 30 16:51:06: 129.158.224.109 Negotiated cipher AES-256 t@32: 2008 Oct 30 16:51:06: 129.158.224.109 Negotiated cipher AES-256 t@33: 2008 Oct 30 16:51:06: 129.158.224.109 Negotiated cipher AES-256 t@32: 2008 Oct 30 16:51:06: 129.158.224.109 Negotiated cipher AES-256 t@33: 2008 Oct 30 16:51:06: 129.158.224.109 Negotiated cipher AES-256 t@32: 2008 Oct 30 16:51:06: 129.158.224.109 Negotiated cipher AES-256 t@30: 2008 Oct 30 16:51:06: 129.158.224.109 Negotiated cipher AES-256 \^C SSL Functions Called -------------------- count      Function SSL Ciphers used -------------------- count      cipher suite 1          RC4 13         AES-256  Running in verbose mode gives more information : $./ssltop.d 9149 log=verbose t@32: 2008 Oct 30 16:50:13: Entered ssl3_GatherAppDataRecord ... t@32: 2008 Oct 30 16:50:13: Entered ssl3_GatherCompleteHandshake ... t@32: 2008 Oct 30 16:50:13: Entered ssl3_GatherData ... t@35: 2008 Oct 30 16:50:13: Entered ssl3_GatherAppDataRecord ... t@35: 2008 Oct 30 16:50:13: Entered ssl3_GatherCompleteHandshake ... t@35: 2008 Oct 30 16:50:13: Entered ssl3_GatherData ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_HandleRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ClientAuthTokenPresent ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ComputeRecordMAC ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_BumpSequenceNumber ... t@32: 2008 Oct 30 16:50:14: 129.158.224.109 Negotiated cipher AES-256 t@32: 2008 Oct 30 16:50:14: Entered ssl3_SendApplicationData ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_SendRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ClientAuthTokenPresent ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_CompressMACEncryptRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ComputeRecordMAC ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_BumpSequenceNumber ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_SendApplicationData ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_SendRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ClientAuthTokenPresent ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_CompressMACEncryptRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ComputeRecordMAC ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_BumpSequenceNumber ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_GatherAppDataRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_GatherCompleteHandshake ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_GatherData ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_HandleRecord ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_ClientAuthTokenPresent ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_ComputeRecordMAC ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_BumpSequenceNumber ... t@35: 2008 Oct 30 16:50:14: 129.158.224.109 Negotiated cipher AES-256 t@35: 2008 Oct 30 16:50:14: Entered ssl3_SendApplicationData ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_SendRecord ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_ClientAuthTokenPresent ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_CompressMACEncryptRecord ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_ComputeRecordMAC ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_BumpSequenceNumber ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_GatherAppDataRecord ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_GatherCompleteHandshake ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_GatherData ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_HandleRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ClientAuthTokenPresent ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ComputeRecordMAC ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_BumpSequenceNumber ... t@32: 2008 Oct 30 16:50:14: 129.158.224.109 Negotiated cipher AES-256 t@32: 2008 Oct 30 16:50:14: Entered ssl3_SendApplicationData ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_SendRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ClientAuthTokenPresent ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_CompressMACEncryptRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ComputeRecordMAC ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_BumpSequenceNumber ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_GatherAppDataRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_GatherCompleteHandshake ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_GatherData ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_HandleRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ClientAuthTokenPresent ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ComputeRecordMAC ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_BumpSequenceNumber ... t@32: 2008 Oct 30 16:50:14: 129.158.224.109 Negotiated cipher AES-256 t@32: 2008 Oct 30 16:50:14: Entered ssl3_SendApplicationData ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_SendRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ClientAuthTokenPresent ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_CompressMACEncryptRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ComputeRecordMAC ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_BumpSequenceNumber ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_GatherAppDataRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_GatherCompleteHandshake ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_GatherData ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_HandleRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ClientAuthTokenPresent ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ComputeRecordMAC ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_BumpSequenceNumber ... t@32: 2008 Oct 30 16:50:14: 129.158.224.109 Negotiated cipher AES-256 t@32: 2008 Oct 30 16:50:14: Entered ssl3_SendApplicationData ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_SendRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ClientAuthTokenPresent ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_CompressMACEncryptRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ComputeRecordMAC ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_BumpSequenceNumber ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_GatherAppDataRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_GatherCompleteHandshake ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_GatherData ... ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_BumpSequenceNumber ... t@32: 2008 Oct 30 16:50:14: 129.158.224.109 Negotiated cipher AES-256 t@32: 2008 Oct 30 16:50:14: Entered ssl3_SendApplicationData ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_SendRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ClientAuthTokenPresent ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_CompressMACEncryptRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_ComputeRecordMAC ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_BumpSequenceNumber ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_GatherAppDataRecord ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_GatherCompleteHandshake ... t@32: 2008 Oct 30 16:50:14: Entered ssl3_GatherData ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_HandleRecord ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_ClientAuthTokenPresent ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_ComputeRecordMAC ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_BumpSequenceNumber ... t@35: 2008 Oct 30 16:50:14: 129.158.224.109 Negotiated cipher AES-256 t@35: 2008 Oct 30 16:50:14: Entered ssl3_SendApplicationData ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_SendRecord ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_ClientAuthTokenPresent ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_CompressMACEncryptRecord ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_ComputeRecordMAC ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_BumpSequenceNumber ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_GatherAppDataRecord ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_GatherCompleteHandshake ... t@35: 2008 Oct 30 16:50:14: Entered ssl3_GatherData ... \^C SSL Functions Called -------------------- count      Function 13         ssl3_HandleRecord 15         ssl3_CompressMACEncryptRecord 15         ssl3_GatherAppDataRecord 15         ssl3_GatherCompleteHandshake 15         ssl3_GatherData 15         ssl3_SendApplicationData 15         ssl3_SendRecord 28         ssl3_BumpSequenceNumber 28         ssl3_ClientAuthTokenPresent 28         ssl3_ComputeRecordMAC SSL Ciphers used -------------------- count      cipher suite 13         AES-256 Dtrace Script : #!/usr/sbin/dtrace -s #pragma D option quiet BEGIN {      log = $2; }  pid$1::ssl2_*:entry,  pid$1::ssl_GatherRecord1stHandshake:entry,  pid$1::sendRSAClientKeyExchange:entry,  pid$1::sendDHClientKeyExchange:entry,  pid$1::ssl3_*:entry,  pid$1::SSL3_*:entry / log == "log=verbose" / {    printf("t@%d: %-20Y: Entered %s ...\n", tid, walltimestamp, probefunc);    @count_table[probefunc] = count() ; } pid$1::*Listen*accept*:entry {     printf("t@%d: %-20Y: ListenSocket::accept called\n ", tid, walltimestamp); } pid$1::*HttpRequest*UnacceleratedRespond*:entry { /*   strcpy(rqSn.sn.inbuf->address, clientIP) */     self->getClientIP = 1; } pid$1::strcpy:entry / self->getClientIP == 1 / {     self->ip =  copyinstr(arg0);     self->getClientIP = 0; } pid$1::*Connection*create*:entry {    self->connection = arg0; } pid$1::*Connection*create*:return {    self->ip = copyinstr(self->connection+8+112+112);    printf("t@%d: %-20Y: %s Connection created\n", tid, walltimestamp, self->ip); } pid$1::SSL_SecurityStatus:entry {    self->getPblock = 1;    /* first pblock call in this function is setting cipher */ } pid$1::*pblock_kvinsert:entry / self->getPblock == 1 / {     self->getPblock = 0;     self->cipher = copyinstr(arg1);     printf("t@%d: %-20Y: %s Negotiated cipher %s\n", tid, walltimestamp,            (self->ip != 0? self->ip : ""),            (self->cipher != 0? self->cipher : ""));     @count_cipher_freq[(self->cipher != 0?self->cipher:" ")] = count(); } END {         printf("SSL Functions Called\n");         printf("--------------------\n");         printf("%-10s %-25s\n", "count", "Function");         printa("%@-10u %-25s\n", @count_table);         printf("SSL Ciphers used\n");         printf("--------------------\n");         printf("%-10s %-25s\n", "count", "cipher suite");         printa("%@-10u %-25s\n", @count_cipher_freq); }

DTrace script to collect information about cipher suites used Given below is a dtrace script I have to trace SSL calls. Running this script on a Web Server instance (32 bit ) pid lets say in our case...

Understanding Oralce iPlanet/Sun Java System Web Server 7.0 - for developers

Here are things a developer working on Oracle Iplanet Web Server 7.0 or writing a NSAPI plugin should know   The latest Oracle iPlanet Web Server(formerly known as Sun Java System Web Server) 7.0 latest update download location : http://www.oracle.com/technetwork/java/webtier/downloads/index.htmlThe latest Sun Java System Web Server 7.0 update 9 documentation : http://download.oracle.com/docs/cd/E18958_01/index.htm Bookmark NSAPI configuration guide :http://docs.sun.com/app/docs/doc/820-4843and Administrator's Configuration FileReference http://docs.sun.com/app/docs/doc/820-4841 (which hasinterfaces) Some examples on how to write a plugin for Oracle iPlanet Web Server 7.0 <ws-install-dir>/samples/nsapi has some examples about how to write a custom plugin. Let me start by explaining the basics of our Web Server first in this blog  To add your own .so dynamic library  In magnus.conf configuration file we can load our own .so's  as shown below : #cat magnus.confInit fn="load-modules" shlib="libj2eeplugin.so" Different request and response processing stages in Oralce iPlanet Web Server 7.0 In Sun Java System Web Server 7.0 there are different stages AuthTrans, NameTrans, PathCheck, ObjectType, Service, Output, Input, Error as given in obj.conf.  More info inhttp://docs.sun.com/app/docs/doc/820-4841/abvag?a=view . We can even makeown function that can be executed in a different stages and add it herein this file. cat obj.conf ... <Object name="default"> AuthTrans fn="match-browser" browser="\*MSIE\*"ssl-unclean-shutdown="true" NameTrans fn="ntrans-j2ee" name="j2ee" NameTrans fn="pfx2dir" from="/mc-icons"dir="/opt/ws/lib/icons"name="es-internal" PathCheck fn="uri-clean" PathCheck fn="check-acl" acl="default" PathCheck fn="find-pathinfo" PathCheck fn="find-index-j2ee" PathCheck fn="find-index"index-names="index.html,home.html,index.jsp" ObjectType fn="type-j2ee" ObjectType fn="type-by-extension" ObjectType fn="force-type" type="text/plain" Service method="(GET|HEAD)" type="magnus-internal/directory"fn="index-common" Service method="(GET|HEAD|POST)" type="\*~magnus-internal/\*"fn="send-file" Service method="TRACE" fn="service-trace" Error fn="error-j2ee" AddLog fn="flex-log" </Object> <Object name="j2ee"> Service fn="service-j2ee" method="\*" </Object> ... <Object name="compress-on-demand"> Output fn="insert-filter" filter="http-compression" </Object> Filters In Web Server we can insert filters like NSPR I/O layer. Filters enablefunctions to intercept and possiblymodify data sent to and from the server. More info is in :http://docs.sun.com/app/docs/doc/820-4843/abvek?a=view . There are twotypes of filters Input and Output. Some important Structures in Oracle iPlanet Web Server 7.0 Sun Java System Web Server 7.0 has Requeststructurehttp://docs.sun.com/app/docs/doc/820-4843/abvmz?l=en&a=view . It has 4 pblock s which stores data in key andvalue pairs. More information is given in http://docs.sun.com/app/docs/doc/820-4843/abvmv?l=en&a=view typedef struct{ \*/Server working variables \*/ pblock \*vars; /\* The method, URI, and protocol revision of this request \*/ pblock \*reqpb; /\* Protocol specific headers \*/ int loadhdrs; pblock \*headers; /\* Server's response headers \*/ int senthdrs; pblock \*srvhdrs; /\* The object set constructed to fulfill this request \*/ httpd_objset \*os;} Request; We can print these pblocks by INTpblock_pblock2str(rq-><pblockname>, NULL);. All requestheaders will be parsed and put into headers pblock. Responseheaders will be put into srvhdrs pblock. Memory Management Everyrequest has its own memory pool which gets freed automatically. So weshould call MALLOC , FREE, STRDUP etc. rather than malloc, free, andstrdup.

Here are things a developer working on Oracle Iplanet Web Server 7.0 or writing a NSAPI plugin should know   The latest Oracle iPlanet Web Server(formerly known as Sun Java System Web Server) 7.0...

Intrusion detection in Sun Java System Web Server 7.0 update 2 - in experimental stages

Intrusion detection in Sun Java System Web Server 7.0 update 2 - in experimental stages I have introduced an experimental untested intrusion detection featurein Web Server 7.0 update 2. It iscurrently an unsupported feature. Basically we can add inserver.xml a file name which contains ModSecurity ruleset. Note that this is an experimentalfeature so please give me feedback about your experiences. Additions to server.xml Element PossibleValues Description <config-file> Text Thiselement may be present at the virtual-serverlevel as well as at the serverlevel. Points to a file containing ModSecurity rules. As with all filepaths in server.xml it may be an absolute path or a relative path, inwhich case it is relative to the configdirectory. The file name component may contain wildcard characters tospecify multiple files within the given directory. Multiple config-fileelements may be present as well. Additions to obj.confA new AuthTrans SAF, secrule-config,is introduced to control the behavior of the ModSecurity engine. Please make sure that this is the first (topmost) AuthTrans directive in obj.conf. The following table describes parameters for the secrule-config function. Parameter Description engine               (Optional) Indicates how SecRule directives areprocessed at request time. "on" indicates that the directives should be applied. "off" indicates that the directives should not be applied. "detection only" indicates that thedirectives should be evaluated but the result ofthe evaluation should not be enforced. The default value is what is set by SecRuleEngine directive (ifany) in configuration file(s) specified by <config-file> element. IfSecRuleEnginedirective is not present, it is "off". process-request-body (Optional) Indicates whetherrequest bodies are processed when evaluating SecRule directives. Whenrequest body processing is enabled, the server will buffer the entire request body in memory, upto the limit defined by SecRequestBodyInMemoryLimitdirective (if any) in configuration file(s) specified by <config-file> element. IfSecRequestBodyInMemoryLimitdirective is not present, it is "131072"."on" indicates that request bodies should be processed. "off" indicates that response bodiesshould not be processed. The default value is what is set by SecRequestBodyAccess directive(if any) in configuration file(s) specified by <config-file> element. IfSecRequestBodyAccessdirective is not present, it is "off". process-response-body (Optional) Indicates whetherresponse bodies areprocessed when evaluating SecRuledirectives. When response body processing isenabled, the server will buffer the entire response body in memory, upto the limit defined by SecResponseBodyLimitdirective (if any) in configuration file(s) specified by <config-file> element. IfSecResponseBodyLimitdirective is not present, it is "524288". "on" indicates that response bodiesshould be processed. "off" indicates that response bodies should not be processed. The default value is what is set by SecResponseBodyAccess directive (if any) in configuration file(s) specified by <config-file> directive.If SecResponseBodyAccessdirective is not present, it is "off". Example # Disable SecRule processing inthe /docs directory <Object ppath="/docs/\*"> AuthTrans fn="secrule-config"engine="off" </Object> SecRuleEngine, SecRequestBodyAccess and SecRequestBodyAccess will stillwork in the files specified in<config-file> in server.xml. Sample :I changed server.xml to have a new config-file element as shownbelow:   <virtual-server>    <config-file>sample.conf</config-file>    <name>test</name>    <host>...</host>    <http-listener-name>http-listener-1</http-listener-name>   </virtual-server> and added this file in <ws7.0u2-server-install-dir>/https-test/configdirectory $catsample.conf SecRuleEngine On # Request related SecRequestBodyAccess On # default limit is 128 KB (131072) SecRequestBodyInMemoryLimit 10000 # Variables SecRule REQUEST_HEADERS"request_headers_match" SecRule REQUEST_HEADERS_NAMES"request_headers_names_match" SecRule HTTP_xxx"request_headers_xxx_match" "phase:1" SecRule REQUEST_HEADERS:yyy"request_headers_yyy_match"  "phase:1" SecRule REQUEST_HEADERS:/zzz/"request_headers_zzz_match" "phase:1" SecRule ARGS "args_match" SecRule ARGS_NAMES"args_names_match" SecRule REQUEST_BODY"request_body_match" SecRule ARGS_COMBINED_SIZE "@gt1000" SecRule ENV "env_match" SecRule QUERY_STRING"query_string_match" SecRule REQUEST_COOKIES"request_cookies_match" SecRule REQUEST_COOKIES_NAMES"request_cookies_name_match" SecRule REQUEST_HEADERS_NAMES"x-aaaaaa.\*" "rev:1,severity:2,msg:'Oops not allwed'" #Do not match lowercase SecDefaultAction "deny" SecRule HTTP_User-Agent"Internet-exprorer" SecRule HTTP_User-Agent "Mosiac1\\.\*" SecRule REQUEST_BODY "X-AAAAAA.\*" SecRule HTTP_Referer "Powered byGravity Board" "id:350000,rev:1,severity:2,msg:'Gravity Board GoogleRecon attempt'" SecRule REQUEST_URI|REQUEST_BODY"select.\*from.\*information_schema\\.tables" I started the server. I get this warning at the time of server start up [11/Apr/2008:12:28:26] info (28100): CORE1116: Sun Java System Web Server 7.0U3 B04/04/2008 12:38 [11/Apr/2008:12:28:27] warning (28101): HTTP3359: An unsupported element config-file is being used. The server may not operate correctly if unsupported features are used.[11/Apr/2008:12:28:29] info (28101): CORE5076: Using [Java HotSpot(TM) Server VM, Version 1.5.0_12] from [Sun Microsystems Inc.][11/Apr/2008:12:28:43] info (28101): HTTP3072: http-listener-1: http://test.sun.com: ready to accept requests[11/Apr/2008:12:28:43] info (28101): CORE3274: successful server startup Now I send these two requests :$telnet0 1894 GET / HTTP/1.0 foo: request_headers_match HTTP/1.1 403 Forbidden Server:Sun-Java-System-Web-Server/7.0 Date: Fri, 11 Apr 2008 07:01:58GMT Content-length: 142 Content-type: text/html Connection: close <HTML><HEAD><TITLE>Forbidden</TITLE></HEAD> <BODY><H1>Forbidden</H1> Your client is not allowed toaccess the requested object. </BODY></HTML> $telnet0 1894 GET / HTTP/1.0 foo: bar HTTP/1.1 200 OK Server:Sun-Java-System-Web-Server/7.0 Date: Fri, 11 Apr 2008 07:02:07GMT Content-type: text/html Last-modified: Fri, 04 Apr 200814:24:09 GMT Content-length: 12038 Etag: "2f06-47f63a09" Accept-ranges: bytes Connection: close <!DOCTYPE HTML PUBLIC"-//W3C//DTD HTML 4.01 Transitional//EN"> <html> ....   </body> </html> The first request got denied as the header value matched with "SecRule REQUEST_HEADERS"request_headers_match"" rule in sample.conf as shown below : $tail-f ../logs/errors [11/Apr/2008:12:31:58] security(28166): for host 127.0.0.1 trying to GET /index.html while trying toGET /, HTTP8005: SecRule directive in file/export1/ws/https-test/config/ms.conf at line 7 matched returningstatus code 403 Note that thisis an untested feature and may have bugs. Please let me know if youfind any. Currently Supported ModSecurity Directives Due to time constraints not all of the ModSecurity directives aresupported in this release. This section documents the supported subset.Note that the supported rules are based on ModSecurity 2.0. The terms and interfaces given below are taken from ModSecurity 2.0documentation http://www.modsecurity.org/documentation/modsecurity-apache/2.0.0-rc-4/modsecurity2-apache-reference.html Some of these keywords like VARIABLESetc. are abstract quantities and not elements. Directive Values Description SecRuleEngine On Off DetectionOnly server initializationDefault value is "off" Directive Values Description SecRule VARIABLES"[@OPERATOR] Text regular expression or parameters to pass to the operator "[ACTIONS] VARIABLES [&!]VARIABLE[:/regular-expression/]| [&!]VARIABLE[:name]| [&!]VARIABLE[:regular-expression]... & should count the number ofvariables in the array. ! x|!x:y examine all x but yshouldnot be checked. | concatenate variables :name a particular value :/regular_expression/or :'/regular_expression/' matches regular expression Values Description VARIABLE REQUEST_HEADERS REQUEST_HEADERS_NAMES REQUEST_HEADERS:yyyor REQUEST_HEADERS:/yyy/ Where yyy is any applicablerequest header name. ARGS Contains request body if SecRequestBodyAccessis set to on. ARGS_NAMES ARGS_COMBINED_SIZE AUTH_TYPE ENV QUERY_STRING HTTP_yyy TIME,TIME_EPOCH, TIME_DAY, TIME_HOUR, TIME_MIN,  TIME_SEC,TIME_WDAY, TIME_MON, TIME_YEAR REMOTE_HOST,REMOTE_PORT, REMOTE_USER,REMOTE_ADDRESS PATH_INFO SERVER_NAME,SERVER_PORT, SERVER_ADDR SCRIPT_BASENAME,SCRIPT_FILENAME REQUEST_COOKIES REQUEST_COOKIES_NAMES REQUEST_FILENAME REQUEST_BASENAME REQUEST_BODY REQUEST_LINE REQUEST_METHOD REQUEST_PROTOCOL REQUEST_URI,REQUEST_URI_RAW RESPONSE_BODY RESPONSE_STATUS RESPONSE_HEADERS RESPONSE_HEADERS_NAMES RESPONSE_PROTOCOL Values OPERATOR rx eq ge gt le lt validateByteRange  Values Description ACTIONS ACTION[:xxx],ACTION[:xxx] ...  Values ACTION allow msg id rev severity log deny status phase t skip chain Values Description phase [1..4] phase:1 - Request headers stage phase:2 - Request body stage phase:3 - Response headers stagephase:4 - Response body stage Defaultvalue is phase:2Note that operations on "phase:5 (Logging stage)" are not supported. Ifencountered, these are ignored and a log message is recorded. Values Description t lowercase Transformation functions to performon the variables before operatoris executed (this includes request body) urlDecode none compressWhitespace removeWhitespace replaceNulls removeNulls Directive Values Description SecDefaultAction ACTIONS For a SecRule, if the previous SecDefaultAction directive ispresent, those actions takes into effect. If none of these SecDefaultActiondirectives are present before a SecRule(in that file or files loaded before it), default SecDefaultActiondirective withACTIONS "log,deny,status:403,phase:2,t:replaceNulls,t:compressWhitespace,t:lowercase"is internally added. For the following directives, in case the multiple directives arepresent (in one or multiple files), last directive's value takes theprecedence. Directive Values Description SecRequestBodyAccess On Off Whether the server should parserequest body or not.Default value is "off" Directive Values Description SecRequestBodyInMemoryLimit integer Configures the maximum request body size server will store inmemory. By default the limit is 128 KB (131072) Directive Values Description SecResponseBodyAccess OnOff Whether the server should parseresponse body or not. Default value is "off" Directive Values Description SecResponseBodyLimit integer Configures the maximum response body size that will beaccepted for buffering. Anything over this limit will be rejected withstatus code 500 Internal Server Error. This setting will not affect theresponses with MIME types that are not marked for buffering. By defaultthis limit is configured to 512 KB. (524288) Directive Values Description SecResponseBodyMimeType strings Configures which MIME types areto be considered for response body buffering. The default value is text/plain text/html. Directive Values Description SecResponseBodyMimeTypesClear - Clears the list of MIME types considered for response bodybuffering, allowing to start populating the list from scratch. Please look at my next blog on this topic also.

Intrusion detection in Sun Java System Web Server 7.0 update 2 - in experimental stages I have introduced an experimental untested intrusion detection feature in Web Server 7.0 update 2. It iscurrently...

Using HTTP compression to speed up content delivery in Sun Java System Web Server 7.0 update 2

Using HTTP compression to speed up content delivery in Sun JavaSystem Web Server 7.0 update 2 If you are looking for faster web page downloads, you can use HTTPcompression feature which compresses your content to speed it up. It is especially beneficial forlow bandwidth connections. This also reduces the number of bytestransferred and improves the overall server performance.How it Works Browsers that support compressed content send an Accept-encoding header withvalue gzip, deflate. Our Web Server sees the header and chooses to provide compressedcontent, and sends Content-encoding:gzip response header. The browser on seeing this header tries todecompress the content and renders it. Now the question is how to enable HTTP compression feature can beconfigured in obj.conf. Let me explain that in detail in this blog. HTTP Compression of Static Files    For dynamic compression of static files in  SunJava System Web Server7.0 Update 2, we need to use compress-fileSAF with find-compressedSAF.     The compress-filefunction creates the compressed file in the subdirectory specified ifthe file size is in between min-sizeand max-size the firsttime a request is sent on the URI. If Web Server needs to compress thatstatic file for everystatic request then it may not really improve performance. It willreduce thenetwork traffic but will cause more CPU usage. So content compressionwill only happen once or until the uncompressed original file ismodified. If check-age isset to true, itrecreates the compressed file if the compressed version is not asrecent as the non-compressed version. The find-compressedfunction checks if a compressedversion of the requested file is available. If the following conditionsare met, find-compressed changesthe path to point to the compressed file: A compressed version is available. The compressed version is as recent as the non-compressedversion. The client supports compression. If the HTTP method is GET or HEAD. Example    This is an example of how server compresses htmlfiles and places themin .compressed directory (relative to the original file location).Modify the default object of obj.conf as shown below <Objectname="default"> NameTrans fn="assign-name"from="\*.html" name="find-compressed" ... Service method=(GET|HEAD|POST)type=\*~magnus-internal/\* fn=compress-file subdir=".compressed-files" Service method=(GET|HEAD|POST)type=\*~magnus-internal/\* fn=send-file ... </Object> <Objectname="find-compressed"> PathCheck fn="find-compressed" </Object>     However, if you plan to put precompressedcontent in the location where the original file is present yourself,you can just use find-compressedSAF . A compressed version of a file must havethe same file name as the non-compressed version but with a .gzsuffix.For example, the compressed version of a file named /httpd/docs/index.htmlwould be named /httpd/docs/index.html.gz. Tocompress files, you can use the freely available gzip program. <Object name="default">NameTrans fn="assign-name" from="\*.html" name="find-compressed"...</Object><Object name="find-compressed">PathCheck fn="find-compressed"</Object> HTTP Compression of Dynamic Content     The http-compressionfilter compressesoutgoing  dynamic content such as the output from SHTMLpages, CGI programs, or pages created with JavaServer PagesTM(JSPTM) technology thatneed to be interpreted by the server.Example Output fn="insert-filter" type="text/\*" filter="http-compression" vary="on" compression-level="9" In this example, type="text/\*" restricts compressionto documents that have a MIME type of text/\* (for example, text/ascii,text/css, text/html,and so on). Suppose you want to compress only JSPs. Modify the default object ofobj.conf as shown below<Objectname="default"> NameTrans fn="assign-name"from="\*.jsp" name="compress-on-demand" ... </Object> <Objectname="compress-on-demand"> Output fn="insert-filter"filter="http-compression" </Object> ParametersHere is a list of parameters you can use if you want to tune thesefunctions Parameters for compress-file subdir (Optional) This should be a directory name only relative to thedirectoryin which the original non-compressed file is located.  To overwrite a pre-compressed compressed file lying indocroot  set the subdir to ".". The default value  is "." will overwrite any precompressedgzfiles if any. check-age           (Optional) The values can be trueor false.       The defaultvalue is true. vary                (Optional) Thevalues can be true or false.       The default value is true. compression-level(Optional) The values can be 1 to 9.                 The default value is 6. min-size            (Optional) The values can be 0to INT_MAX.          The default value is 256. max-size        (Optional) Thevalues can be min-size to INT_MAX.The default value is 1048576. Parameters for find-compressed check-age (Optional)  Specifies whether to check if the compressedversionis olderthan the non-compressed version.  The values can be yes or no. By default,the value is set to yes. If set to yes, the compressed version willnot be selected if it is older than the non-compressed version.  If set to no, the compressed version isalways selected, even if it is older than the non-compressed version. vary  (Optional) Specifies whether to insert a Vary:Accept-Encoding header. The values can be yes or no. By default,the value is set to yes. If set to yes, a Vary: Accept-Encodingheader is always inserted when a compressed version of a file isselected.  If set to no, a Vary: Accept-Encodingheader is never inserted.  Parameters for http-compression vary  Controls whether the filter inserts a Vary:Accept-encoding header. If vary is absent, the default value is yes.yestells the filter to insert a Vary: Accept-encoding headerwhen it compresses content.  no tells the filter to neverinsert a Vary: Accept-encoding header. fragment-size Size in bytes of the memory fragment usedby the compression libraryto control how much to compress at a time. The default value is 8096. compression-level Controls the compression level usedby the compression library. Validvalues are from 1 to 9. Avalue of 1 results in the best speed. A valueof 9 results in the bestcompression. The default value is 6. window-size  Controls an internal parameter of thecompression library. Valid valuesare from 9 to 15. Higher values result in better compression at theexpenseof memory usage. The default value is 15. memory-level  Controls how much memory is used by thecompression library. Valid valuesare from 1 to 9. A value of 1 uses the minimum amount of memory but isslow. A value of 9 uses the maximum amount of memory for optimalspeed. Thedefaultvalue is 8. Using wadm (Administration CLI) to configure these HTTPcompression featuresYou can use the following wadm commands to configure these. For compression of static content i.e. find-compressed and compress-file functions usewadm>enable-precompressed-content Usage:enable-precompressed-content --help|-?   or  enable-precompressed-content [--echo] [--no-prompt] [--verbose][--uri-pattern=pattern] [--no-vary-header] [--no-age-check][--compress-file] [--sub-dir=sub-directory] [--compression-level=1-9][--min-size=size-in-bytes] [--max-size=size-in-bytes]--config=config-name --vs=vs-name CLI014 vs is a required option. wadm> For compression of dynamic content i.e. http-compression function usewadm>enable-on-demand-compression Usage:enable-on-demand-compression --help|-?   or  enable-on-demand-compression [--echo] [--no-prompt] [--verbose][--uri-pattern=pattern] [--no-vary-header] [--fragment-size=size][--compression-level=1-9] --config=config-name --vs=vs-name CLI014 vs is a required option. You can also try experimenting with other related CLIs get-precompressed-content-prop get-on-demand-compression-prop disable-precompressed-content disable-on-demand-compression How do I test if my configuration is working or not    You can use telnet to submit HTTP requests,impersonating a webbrowser. The stuff in bold is the stuff I typed.  Here'san example on a Solaris machine testserver,where I have a Web Server instance running on port 2600:$ telnet testserver 2600Trying 1.2.3.4... Connected to testserver. Escape character is '\^]'. GET / HTTP/1.0 [press enter twice] HTTP/1.1 200 OKServer:Sun-Java-System-Web-Server/7.0Date: Wed, 06 Feb 2008 12:09:45GMTContent-type: text/htmlLast-modified: Fri, 25 Jan 200800:12:38 GMT Content-length: 355 Connection: close <html>...</html>Connection to testserver closed by foreign host.$ I'll try that same HTTP request again, but this time indicate that Iwant a compressed response: $telnet testserver 2600Trying 1.2.3.4...Connected to testserver. Escape character is '\^]'. GET / HTTP/1.0 [press enter] Accept-encoding: gzip [press enter twice] HTTP/1.1 200 OKServer: Sun-Java-System-Web-Server/7.0Date: Wed, 06 Feb 2008 12:09:45 GMTContent-type: text/htmlLast-modified: Fri, 25 Jan 2008 00:12:38 GMTContent-encoding: gzipVary: accept-encodingConnection: close MPËNÄ0...ó²õã\\åConnection to testserver closed by foreign host.$ When I ask for a compressed response, the server responds with aContent-encoding: gzipheader and returns compressed response instead of the html. \*\*Note that compress-file SAF is not available in Web Server 6.1, in that case you have to put pre compressed content into document root for find-compressed to work. References Web Server software forum questions Compressing Web Content with mod_gzip and mod_deflate Best Practices for Speeding Up Your Web Site HTTP Compression Web Server 7.0 update 2 documentation about find-compressed function Web Server 7.0 update 2 documentation about http-compression function my previous blog Dynamic Compression Of Static Files

Using HTTP compression to speed up content delivery in Sun Java System Web Server 7.0 update 2 If you are looking for faster web page downloads, you can use HTTPcompression feature which compresses...

Using built-in hardware crypto accelerators of SPARC Enterprise T5120 server (powered by UltraSPPARC T2 i.e. Niagara2 processor) in Oracle iPlanet Web Server 7.0

Using built-in hardware crypto accelerators of SPARC Enterprise T5120 server (powered by UltraSPARC T2 i.e. Nigara2 processor) in Oracle iPlanet Web Server 7.0 update 9 In my previousblog I talked about SCF framework and Sun Java System Web Server7.0 in general. This time I tried to make use of built-in hardwarecrypto accelerators of SPARC Enterprise T5120 server (powered by Ultra SPARC-T2 i.e. Niagara 2 processor) in Oracle iPlanet Web Server 7.0 update 9. T5120 server has intergrated onboard cryptographic acceleration supporting 10 embedded security industry-standard ciphers including DES, 3DES, AES, RC4, SHA1, SHA256, MD5, RSA to 2048 key, ECC, and CRC32. Here is what I did : Step 1 : Go to Webserver installation directory and start adminserver # ./admin-server/bin/startserv Step 2.1 : Go to Web Server 7.0 instance's configdirectory and perform these manual steps # cd https-foo.com/config/ 2.2) First move the existing database to another directory # mv /.sunw ./sunw.old Setpin # pktool setpinCreate new passphrase: type-password-hereRe-enter new passphrase: type-password-herePassphrase changed. 2.3) List the current PKCS#11modules # ../../bin/modutil-list -dbdir .Listing of PKCS #11 Modules-----------------------------------------------------------   1. NSS Internal PKCS#11 Module         slots: 2 slots attached        status: loaded         slot: NSS Internal Cryptographic Services        token: NSS Generic Crypto Services         slot: NSS User Private Key and Certificate Services        token: NSS Certificate DB   2. Root Certs        library name: libnssckbi.so         slots: 1 slot attached        status: loaded         slot: NSS Builtin Objects        token: Builtin Object Token----------------------------------------------------------- 2.4) Add SCF module # ../../bin/modutil -dbdir . -add "Solaris Crypto Framework"-libfile /usr/lib/libpkcs11.so -mechanisms RSA...Module "Solaris CryptoFramework" added to database. 2.5) Enable SCF module # ../../bin/modutil -enable"Solaris Crypto Framework" -dbdir ....Slot "Sun Metaslot" enabled.Slot "n2cp/0 Crypto Accel Bulk 1.0" enabled.Slot "ncp/0 Crypto Accel Asym1.0" enabled.Slot "n2rng/0 SUNW_N2_Random_Number_Generator" enabled. 2.6) List modules to make sure add and enable stuff above succeeded. # ../../bin/modutil -list -dbdir .Listing of PKCS #11 Modules-----------------------------------------------------------  1. NSS Internal PKCS#11 Module        slots: 2 slots attached       status: loaded        slot: NSS Internal Cryptographic Services       token: NSS Generic Crypto Services        slot: NSS User Private Key and Certificate Services       token: NSS Certificate DB 2. Solaris Crypto Framework        library name: /usr/lib/libpkcs11.so         slots: 4 slots attached        status: loaded         slot: Sun Metaslot        token: Sun Metaslot         slot: n2cp/0 Crypto Accel Bulk 1.0        token: n2cp/0 Crypto Accel Bulk 1.0         slot: ncp/0 Crypto Accel Asym 1.0        token: ncp/0 Crypto Accel Asym 1.0         slot: n2rng/0 SUNW_N2_Random_Number_Generator        token: n2rng/0 SUNW_N2_RNG  3. Root Certs        library name: libnssckbi.so         slots: 1 slot attached        status: loaded         slot: NSS Builtin Objects        token: Builtin Object Token ----------------------------------------------------------- 2.7) List cryptoadm providers and their mechanisms: # cryptoadm list -pUser-level providers:=====================/usr/lib/security/$ISA/pkcs11_kernel.so: all mechanisms are enabled. random is enabled./usr/lib/security/$ISA/pkcs11_softtoken_extra.so: all mechanisms are enabled. random is enabled.Kernel software providers:==========================des: all mechanisms are enabled.aes256: all mechanisms are enabled.arcfour2048: all mechanisms are enabled.blowfish448: all mechanisms are enabled.sha1: all mechanisms are enabled.sha2: all mechanisms are enabled.md5: all mechanisms are enabled.rsa: all mechanisms are enabled.swrand: random is enabled.Kernel hardware providers:==========================n2cp/0: all mechanisms are enabled.ncp/0: all mechanisms are enabled.n2rng/0: all mechanisms are enabled. random is enabled. 2.9) Disable the following mechanisms in User Level Providers #  cryptoadmdisable provider=/usr/lib/security/\\$ISA/pkcs11_softtoken_extra.so \\                mechanism=CKM_SSL3_PRE_MASTER_KEY_GEN,CKM_SSL3_MASTER_KEY_DERIVE,CKM_SSL3_KEY_AND_MAC_DERIVE,\\                       CKM_SSL3_MASTER_KEY_DERIVE_DH,CKM_SSL3_MD5_MAC,CKM_SSL3_SHA1_MAC 2.10) List to make sure these were disabled # cryptoadm list -pUser-level providers:=====================/usr/lib/security/$ISA/pkcs11_kernel.so: all mechanisms are enabled. random is enabled./usr/lib/security/$ISA/pkcs11_softtoken_extra.so: all mechanisms are enabled, except CKM_SSL3_SHA1_MAC,CKM_SSL3_MD5_MAC,CKM_SSL3_MASTER_KEY_DERIVE_DH,CKM_SSL3_KEY_AND_MAC_DERIVE,CKM_SSL3_MASTER_KEY_DERIVE,CKM_SSL3_PRE_MASTER_KEY_GEN. random is enabled.Kernel software providers:==========================des: all mechanisms are enabled.aes256: all mechanisms are enabled.arcfour2048: all mechanisms are enabled.blowfish448: all mechanisms are enabled.sha1: all mechanisms are enabled.sha2: all mechanisms are enabled.md5: all mechanisms are enabled.rsa: all mechanisms are enabled.swrand: random is enabled.Kernel hardware providers:==========================n2cp/0: all mechanisms are enabled.ncp/0: all mechanisms are enabled.n2rng/0: all mechanisms are enabled. random is enabled. 2.11) a) Now run the following Web Server CLI commands # ../../bin/wadm --user=admin Please enteradmin-user-password> type-admin-server-password-here wadm> list-configs foo wadm> pull-config--config=foo foo CLI201 Command 'pull-config' ransuccessfully wadm> list-tokens--config=foo internal SunMetaslot wadm>  create-selfsigned-cert --config=foo--server-name=foo --nickname=Server-Cert --token="Sun Metaslot" Please enter token-pin> type-password-here CLI201 Command'create-selfsigned-cert' ran successfully wadm> list-http-listeners --config=foo http-listener-1 wadm> set-ssl-prop --config=foo--http-listener=http-listener-1 server-cert-nickname="SunMetaslot:Server-Cert" enabled=true CLI201 Command 'set-ssl-prop'ran successfully wadm>  deploy-config foo CLI201 Command 'deploy-config'ran successfully wadm> \*\* 2.11) b) If you are using older version of Web Server and you do not have Admin CLI, you can use the following command to create the self signed certificate # ../../bin/certutil -S -x -n "Server-Cert" -t "u,u,u" -s "CN=foo" -d . -h "Sun Metaslot" Enter Password or Pin for "Sun Metaslot":type-password-here...Continue typing until the progress meter is full: |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*| Finished. Press enter to continue:      Generating key. This may take a few moments...    #2.11) c) OR you can create a self-signed certificate in NSS DB and export it and import it into Sun Metaslot : # ../../bin/certutil -S -x -n "Server-Cert" -t "u,u,u" -s "CN=foo" -d . Enter Password or Pin for "Sun Metaslot":type-password-here...Continue typing until the progress meter is full: |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*| Finished. Press enter to continue: Generating key. This may take a few moments...# # ../../bin/pk12util  -o key-cert-data.pk12 -n "Server-Cert" -d .Enter password for PKCS12 file: type-a-new-password-here-or-press-enterRe-enter password: type-a-new-password-here-or-press-enterpk12util: PKCS12 EXPORT SUCCESSFUL # ../../bin/pk12util -i key-cert-data.pk12 -d . -h "Sun Metaslot"Enter Password or Pin for "Sun Metaslot": type-password-hereEnter password for PKCS12 file: type-the-new-password-herepk12util: PKCS12 IMPORT SUCCESSFUL 2.12) Now manually double check if the certificate exists # ../../bin/certutil -L -d . -h "SunMetaslot" Certificate Nickname                Trust Attributes                                   SSL,S/MIME,JAR/XPIEnter Password or Pin for "SunMetaslot": type-password-here SunMetaslot:Server-Cert                      u,u,u Check that server.xml contains server-cert-nickname element  <http-listener>    <name>http-listener-1</name>    <port>80</port>    <server-name>foo</server-name>    <default-virtual-server-name>foo</default-virtual-server-name>    <ssl>      <server-cert-nickname>SunMetaslot:Server-Cert</server-cert-nickname>    </ssl>   </http-listener> 2.13) Start the server #../bin/startserv Oracle iPlanet Web Server 7.0.9 B07/04/2010 02:15 Please enter the PIN for the"Sun Metaslot" token: type-password-hereinfo: CORE5076: Using [Java HotSpot(TM) Server VM, Version 1.6.0_20] from [Sun Microsystems Inc.]info: HTTP3072:http-listener-1: https://foo:80ready to accept requests info: CORE3274: successfulserver startup 2.14) Check the statistics using kstat # kstat -n ncp0 | grep rsa         rsagenerate                     6        rsaprivate                      18        rsapublic                       15 Send a request through the browser (with a cipher containing RSA) to this server https://foo:80/test.html, it should show test.html I used tstclnt (a client bundled with NSS) to send request to the server using cipher c i.e. SSL3 RSA WITH RC4 128 MD5. # tstclnt -h accel -p 81 -d . -n Server-Cert  -T -f -o -v  -c c < req.txttstclnt: connecting to accel:81 (address=10.133.169.154)...       tstclnt: SSL version 3.0 using 128-bit RC4 with 128-bit MD5 MAC      tstclnt: Server Auth: 1024-bit RSA, Key Exchange: 1024-bit RSA      ...      HTTP/1.1 200 OK      Server: Oracle-iPlanet-Web-Server/7.0      Content-type: text/html      Last-modified: Mon, 10 Jan 2011 09:45:03 GMT      Etag: "12-4d2ad51f"      Accept-ranges: bytes      Content-length: 18      Date: Mon, 10 Jan 2011 09:45:23 GMT      Connection: close      This is test.html      ... Now run kstat again,  if it shows an increase that means we are ok. # kstat -n ncp0 | grep -i rsa      rsagenerate                     6      rsaprivate                      18    rsapublic                       16 Note : If 1. Web server does not present the Intermediate CA certificates installed as Server Certificate Chain to the browser and that causes the certificate validation by the browser to fail.or 2. Client authentication fails with the following error message in the errors log .  Root CA cert has been installed to the certificate database. failure (16670): HTTP3068: Error receiving request from 123.45.67.897(SEC_ERROR_UNKNOWN_ISSUER: Peer's certificate is signed by an unknown issuer) These two issues are caused by the /.sunw directory not being accessible by the web server running user "webservd". That directory has permissions 0700 and is owned by root. Web Server starts up as root and then changes (using setuid) to user "webservd".  Solution to this is 1) Have the web server running as root2) Open up the permission on /.sunw so that it is readable by the web server running user3) Set  the environment variable SOFTTOKEN_DIR to point to some directory that is owned by webservd before the web server is started. The SCF will then access the files in $SOFTTOKEN_DIR/pkcs11_softoken/ during execution. References Configuring Solaris Cryptographic Framework and Sun Java System Web Server 7 on Systems With UltraSPARC T1 Processors - Jyri J. Virkki, September 2006 Using the Cryptographic Accelerator of the UltraSPARC T1 Processor (March 2006)  -by Ning Sun, Pallab Bhattacharya Picture of Sun Crypto accelerator 1000 cpustat and T4 crypto cheat sheet https://blogs.oracle.com/cmt/entry/t4_crypto_cheat_sheet \*\*If you get an error in list-configs command like : "CLI104 Unable to communicate with the administration server: No such file or directory" then you are probably seeing bug "6606384 SCF consumers crash after mechanisms are disabled using cryptoadm when using libumem". Upgrade to Solaris 10 update 9 or apply patches. core dump should look like # mdb ../../admin-server/config/core.18103>::stacklibc.so.1`_lwp_kill+8(6, 0, 20f04, ff34ba3c, ff36a000, ff36abdc)libumem.so.1`umem_do_abort+0x1c(44, e4ced4e8, 6, 20e40, ff356ad8, 0)libumem.so.1`umem_err_recoverable+0x7c(ff357b54, a, 20d38, 0, ff36d0e8, ff357b5f)libumem.so.1`process_free+0x114(12d3ee8, 1, 0, 3e3a1000, 1ec08, ff)libpkcs11.so.1`pkcs11_slottable_delete+0x158(12d3ee8, a11628, a11628, f97c6bb0, 1, 1)libpkcs11.so.1`pkcs11_fini+0x4c(f97c6b8c, 1, f97ae1c8, f97c6000, 17aac, f97c6b84)libc.so.1`_postfork_child_handler+0x30(1d18, fd543800, 1c00, 4, fba5ca00, fd543800)libc.so.1`fork+0x144(0, 2, 0, 3c, fd543800, fba5ca00)libns-httpd40.so`CHILDEXEC_ERR ChildExec::_startListener()+0x19c(...)libns-httpd40.so`CHILDEXEC_ERR ChildExec::PerformListenerOp(ListenerOp,int&)+0x14(...)libns-httpd40.so`CHILDEXEC_ERR ChildExec::StartListener(int&)+0x48(...)libns-httpd40.so`CHILDEXEC_ERR ChildExec::initialize(int&)+0xa8(...)libns-httpd40.so`PRStatus cgistub_child_exec_init()+0x2d0(...)libns-httpd40.so`PRStatus cgistub_init()+0x58(...)...

Using built-in hardware crypto accelerators of SPARC Enterprise T5120 server (powered by UltraSPARC T2 i.e. Nigara2 processor) in Oracle iPlanet Web Server 7.0 update 9 In my previous blog I talked...

Configuring reverse proxy in Sun Java System Web Server 7.0 when origin server has SSL enabled

Configuring reverse proxy in Oracle iPlanet Web Server 7.0 when origin server has SSL enabled. Origin Server <-- HTTPS ---> Reverse Proxy Server <-- HTTP -->  client/Browser There are various SSL and non SSL configurations we can have for Reverse Proxy and Origin Servers Origin Server <--- HTTP ---> Reverse Proxy Server <--- HTTP ---> client/Browser Origin Server <-- HTTP ---> Reverse Proxy Server <-- HTTPS --> client/Browser i.e. Reverse proxy SSL termination End point Origin Server <-- HTTPS ---> Reverse Proxy Server <-- HTTP --> client/Browser Origin Server <-- HTTPS ---> Reverse Proxy Server <-- HTTPS --> client/Browser In my last blog I explained configuration of a simple non SSL reverse proxy(i.e. scenario 1). In this blog I have tried to set up a SCENARIO 3 shown above, where a non SSL Oracle iPlanet Web Server 7.0 tries to connect to origin server which is SSL enabled Creating SSL enabled origin server If you already have an SSLenabled origin server you can skip this. For ease of use I have used Oracle iPlanet Web Server 7.0 as origin server also. Start the administrationserver, and go to wadm >./wadm --user=admin Please enteradmin-user-password> \*\*\*\* Create a self signed certificate wadm>create-selfsigned-cert --config=test --server-name=www.test.com --nickname=OriginServerServer-Cert Optional : Create a HTTP listener (or use existing one) wadm> create-http-listener --listener-port=8888 --config=test --server-name=www.test.com --default-virtual-server-name=www.test.com mylistener Enable SSL for this listener and set the server certificate nickname wadm> set-ssl-prop --config=test --http-listener=mylistenerserver-cert-nickname=OriginServerServerCert enabled=true Deploy the changes wadm> deploy-config test Start this origin server instance. Settings in Web Server 7.0 instance Lets say we want to forward allrequests to /xyz to the origin server. Go the Web Server instance config directory and modify the obj.conf as given below <Object name="default"> AuthTrans fn="match-browser"browser="\*MSIE\*" ssl-unclean-shutdown="true" NameTrans fn="ntrans-j2ee" name="j2ee" NameTrans fn="pfx2dir" from="/mc-icons" dir="/export2/mv/lib/icons" name="es-internal" NameTrans fn="map" from="/xyz" name="reverse-proxy-/" to="/xyz" PathCheck fn="uri-clean" ... </Object> <Object ppath="\*"> Service fn="proxy-retrieve" method="\*" </Object><Object name="reverse-proxy-/">Route fn="set-origin-server" server="https://test.sun.com:8888" </Object> \*\*Note that I have given manual steps here. In my last blog I have given Administration CLI steps. Lets say for this instance server.xml has <port>8080</port>. Make sure that the originserver is up and running.Start the server and access http://test.sun.com:8080/xyz/should show you xyz directory in the docroot of the origin server. Troubleshooting In case we get a Gateway Timeout errorand in error logs we see some error like [23/Jul/2007:16:44:11] failure(27927): for host .... trying to GET ...., service-http reports:HTTP7758: error sending request(SEC_ERROR_UNTRUSTED_ISSUER: Client certificate is signed by anuntrusted issuer.) We get this error because the origin server's certificate was not issued by a trusted CA. It means we need to export CA certificate of the origin server instance and import it into Web Server instance. In this case, Origin Server certificate (nickname OriginServerServerCert) is the CA for itself so we import that certificate. This step is not required if the Origin Server's certificate chain ends at a root CA Certificate which is a trusted CA and is present in built-in root CA certificate DB. Export the origin server's CA certificate Go to <server-instance>/config directory of the origin server, and list certificates and then use pk12util to export thecertificate. $../../bin/pk12util -o /tmp/exported.crt -n OriginServerServerCert -d . Import the origin server CA certificate in server instance configdirectory Initialize NSS Database To import certificate in serverinstance config directory you have to first initialize the NSS database. Note that if this Web Server instance is SSL enabled you can skip thisNSS database intialization part. $../../bin/certutil -N -d . Import the certificate Lets say the file /tmp/exported.crt contained theCA cert of the origin server, import that to NSS database. $ ../../bin/pk12util -i /tmp/exported.crt -d . Confirm by listing certs using certutil $../../bin/certutil -L -d . OriginServerServerCert          u,u, Modify trust flags if required (if its a self signed cert) You can see that the certificatesimported doesn't contain 'CT' trust flags. $../../bin/certutil -M -n OriginServerServer-Cert -t 'CTu,CTu,CTu' -d . Now u can confirm $../../bin/certutil -L -d .OriginServerServerCert       CTu,CTu,CTu Restart the server instance and things should work fine now.

Configuring reverse proxy in Oracle iPlanet Web Server 7.0 when origin server has SSL enabled. Origin Server <-- HTTPS ---> Reverse Proxy Server <-- HTTP -->  client/Browser There are various SSL and...

Troubleshooting memory leaks and memory corruptions in Sun Java System Web Server 7.0

Troubleshooting memory leaks and memory corruptions in Sun Java System Web Server 7.0In this blog I am trying to show how to use libumem and watchmalloc to find more information about memory leaks and memory corruptions in Sun Java System Web Server 7.0. Using libumem to find memory leaks and memory corruptions 1) Disable pools Add this line in magnus.conf Init fn="pool-init"disable="true"Note this is not a supported public interface. 2) Set environment variable UMEM_DEBUG $export UMEM_DEBUG="default" Refer man pages for umem_debug (3MALLOC) for more details. 3)  Start the Web Server instance $./bin/startserv 4) Note down the pid of the webservd process $ps-eaf |grep webservdYou will see two webservdprocesses, note down the highest pid. 5) Dump the initial core $gcore-o core.pid.start pid 6) Run tests , send some requests which cause memory corruption orleaks. 7) Dump core again gcore-o core.pid.end pid 8) Compare memory allocated between these two should show theleaks. $mdbcore.pid.start   >::findleaks -d ! cat >findleaks.start.log( Or you can also run $echo ::findleaks -d |mdb core.pid.start > findleaks.start.log ) $mdb core.pid.end >::findleaks -d ! cat >findleaks.end.log compare these two logs manually to seeif there are leaks. If these logs are unreadable you can use Sun studio's c++filt or in mdb give this command as shown below > $GC++ symbol demangling enabled> 9) For memory corruptions, try umem_status, bufctl_audir and umem_verify commands in the last core: $mdbcore.pid.end >::umem_status...If it shows a corrupted buffer try looking at its contents (For example >26a3640/40X)or >>bufferaddress::bufctl_audit (For example >26a4618::bufctl_audit) umem_status command, also shows if a buffer is accessed after its already freed. For example, once it showed me > ::umem_statusStatus: ready and activeConcurrency: 2Logs: (inactive)Message buffer:umem allocator: buffer modified after being freedmodification occurred at offset 0x8 (0xdeadbeefdeadbeef replaced by 0xde20be20de20be20)... You can also use umem_verify command to see if one of the umem caches has a corrupted buffer. Using watchmalloc for memory corruptions 1) Disable pools Add the following line in magnus.conf Init fn="pool-init" disable="true" 2) Add LD_PRELOAD watchmalloc with MALLOC_DEBUG=WATCH in startscript , comment out libumem and libCld portions $diff -u startservstartserv.watchmalloc ---startserv     Wed Dec  7 14:03:40 2005 +++ startserv.watchmalloc    Thu Mar 15 13:19:06 2007 @@ -9,6 +9,9 @@  SERVER_BIN_DIR="/space/wsDec7/iplanet/ias/server/work/B1/SunOS5.8_DBG.OBJ/bin"  SERVER_LIB_DIR="/space/wsDec7/iplanet/ias/server/work/B1/SunOS5.8_DBG.OBJ/lib"  SERVER_BIN=webservd-wdog +LIBWMC=/usr/lib/watchmalloc.so.1 +LD_PRELOAD_32="${LIBWMC}${LD_PRELOAD}"; export LD_PRELOAD_32 +MALLOC_DEBUG="WATCH"; exportMALLOC_DEBUG  # Add path to serverbinaries to PATH  PATH="${SERVER_BIN_DIR}:${SERVER_LIB_DIR}:/bin:${PATH}";export PATH @@ -47,26 +50,6 @@  # Add instance-specificinformation to SHLIB_PATH for HP-UX  SHLIB_PATH="${SERVER_LIB_PATH}:${SERVER_JVM_LIBPATH}:${SHLIB_PATH}";export SHLIB_PATH -# Preload libumem to improveperformance on Solaris 10 -LIBUMEM_32=/usr/lib/libumem.so -if [ -f "${LIBUMEM_32}" ] ; then -    if [ `uname-r | sed s/\\\\\\.//` -ge 510 ] ; then -       LD_PRELOAD_32="${LIBUMEM_32} ${LD_PRELOAD_32}"; export LD_PRELOAD_32 -    fi -fi -LIBUMEM_64=/usr/lib/64/libumem.so -if [ -f "${LIBUMEM_64}" ] ; then -    if [ `uname-r | sed s/\\\\\\.//` -ge 510 ] ; then -       LD_PRELOAD_64="${LIBUMEM_64} ${LD_PRELOAD_64}"; export LD_PRELOAD_64 -    fi -fi - -# Preload libCld to resolve-compat=4/-compat=5 C++ ABI issues on Solaris -LIBCLD="${SERVER_LIB_DIR}/libCld.so" -if [ -f "${LIBCLD}" ] ; then -   LD_PRELOAD_32="${LIBCLD} ${LD_PRELOAD_32}"; export LD_PRELOAD_32 -fi -  if [ $# -eq 0 ] ; then     COMMAND=--start;  elif [ "$1" = "-i" ] ; then 3) Comment out compat4/compat5NSAPI Init plugins in magnus.conf if they exist (mostprobably not) 4) Start the Web Server instance $./bin/startservThis makes the server extremely slow. 5) Run tests, send requests to the server, The server may crash at proper placesrather than random ones. In general if you want to see what's happening when you send a request you can try using truss :truss -o truss.log -u "\*" -d -fael -rall -vall -wall -p <pid> More information about this is in http://docs.sun.com/app/docs/doc/816-5165/truss-1?l=ja&a=view Note that -u option will show the user-level function call tracing. Links: http://sunsolve.sun.com/search/document.do?assetkey=1-9-70641 http://developers.sun.com/solaris/articles/libumem_library.html

Troubleshooting memory leaks and memory corruptions in Sun Java System Web Server 7.0 In this blog I am trying to show how to use libumem and watchmalloc to find more information about memory leaks and...

Troubleshooting Web Server crashes : enabling core dumps

Troubleshooting Web Server crashes : enabling core dumps I get a lot of questions regarding what to do when Web Server instancecrashes as shown by error logs and you do not find the core file. The core file will be dumped in the Web Server instance's config directory. For example, if the Web Server is installed in /opt/SUNWwbsvr directory, core file will be /opt/SUNWwbsvr/https-hostname/config/core. If you are using SSL, core dumps will be disabled by default. You can force them on by setting the SSL_DUMP environment variable before starting Web Server: $ SSL_DUMP=1$ export SSL_DUMP$ ./startCheck coreadm(1M) and ulimit(1) to see if Solaris will allow processes to dump core. Use the pstack(1M) command to obtain a stack trace. You can use the same coreadm (core file administration) command to set appropriate values. $coreadm global core file pattern: global core file content: default init core file pattern: core.%p initcore file content: default global core dumps: disabled per-process core dumps: enabled global setid core dumps: disabled per-process setid core dumps: disabled global core dump logging: disabled Note that I have set core file pattern to core.%p instead of the usual core . If a process with pidlets say 1000 dumps core, it will generate a core file with name core.1000 to avoid overriding in case the server dumps multiple core files. But this is not necessary.If your Operating System is Linux, make sure that you set ulimit to unlimited before starting the server.$ulimit -c unlimited or $ulimit -S -c unlimited oror edit /etc/security/limits.conf and followed by editing /etc/profileTo get core dumps on Windows, first make drwatson as a default debugger: C:\\WINDOWS\\system32>drwtsn32.exe -iTo change various settings of this drwatson, C:\\WINDOWS\\system32>drwtsn32.exeThis opens a window where you can specify where to dump the core.To set back the default debugger to MSVC, change registry key HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\AeDebug to "C:\\Program Files\\Microsoft Visual Studio\\Common\\MSDev98\\Bin\\msdev.exe" -p %ld -e %ldSet Auto to 0.Set UserDebuggerHotKey to 0. Some more suggested reading http://publib.boulder.ibm.com/httpserv/ihsdiag/coredumps.html http://developers.sun.com/solaris/articles/DebugLibraries/DebugLibraries_content.html  - The article shows how to use elfdump -p and dbx to check corefiles.

Troubleshooting Web Server crashes : enabling core dumps I get a lot of questions regarding what to do when Web Server instance crashes as shown by error logs and you do not find the core file.The...

Denial of Service (DoS) Prevention By Request Timeout in Sun Java System Web Server 7.0

Denial of Service (DoS) Prevention By Request Timeout in Sun Java System Web Server 7.0 Check out the new improvements we made in Sun Java System Web Server 7.0. In this blog I will talk about Denial Of Service (DoS) Prevention "Request Timeout" enhancements.We have introduced two more timeouts in the server.xml's <http> element in addition to the existing <io-timeout>. They are <request-header-timeout>and <request-body-timeout>. If you are a Web Server Administrator and you want to limit users to be sending all request headers in the first 10 minutes of the connection and request body in the next one hour, you can set these two parameters in server.xml like ... <http> ...    <request-header-timeout>600</request-header-timeout>    <request-body-timeout>3600</request-body-timeout> </http> ... All other connections which last longer will be disconnected by theserver automatically. Other blogs on this topic are : Web Server 7.0 Request Limiting  http://blogs.oracle.com/jyrivirkki/entry/web_server_7_0_request More on Request Limiting http://blogs.oracle.com/jyrivirkki/entry/more_on_request_limiting Web Server 7.0 Concurrency Limiting http://blogs.oracle.com/jyrivirkki/entry/web_server_7_0_concurrency

Denial of Service (DoS) Prevention By Request Timeout in Sun Java System Web Server 7.0 Check out the new improvements we made in Sun Java System Web Server 7.0. In this blog I will talk about Denial...

Directory listing in Sun Java System Web Server 7.0

Directory listing in Sun Java System Web Server 7.0 Been getting a lot of questions about directory listing in Sun Java SystemWeb Server 7.0 I have setup a Sun Java System Web Sever 7.0. I am not able see directory contents. I get a popup saying "Authentication Required". Under "Content Handling"->"General"->"Directory Listing", I tried setting "Listing Type" to 'Fancy' & 'Simple'.  But that did not help (I saved and re-deployed the server instance). Here is the solution. Check in obj.conf configuration file if "index-common" service SAF is present:Service method="(GET|HEAD)" type="magnus-internal/directory" fn="index-common"Directory listing is not enabled by default. If you look at the default ACL file default.acl, (assuming you do not have any other other VS specific ACL file) version 3.0;acl "default";authenticate (user, group) {  prompt = "Sun Java System Web Server";};allow (read, execute, info) user = "anyone";allow (list, write, delete) user = "all"; That shows "list" right is allowed to"all" (authenticated users only). And for directory listing, you need "list" rights. That means only authenticated users can see directory lists. You can move this "list" right to "anyone" so that evenunauthenticated users can see the directory lists.So here is what the changed ACEs should look like : allow (read, execute, info, list) user = "anyone";allow (write, delete) user = "all"; Two more minor tips I would like to add 1. If you want to change the width of the columns of filename, last modified time, size, description in directory listing, add "cindex-init" directive in magnus.conf. For example Init fn="cindex-init" widths="50,5,5,20" 2. If you want to change the directory listing to "simple" style where you will only see the list of filenames, you can change "index-common" to "index-simple" as shown below Service method="(GET|HEAD)" type="magnus-internal/directory" fn="index-simple"

Directory listing in Sun Java System Web Server 7.0 Been getting a lot of questions about directory listing in Sun Java System Web Server 7.0I have setup a Sun Java System Web Sever 7.0. I am not...

Creating Authentication Databases in Sun Java System Web Server 7.0

Creating Authentication Databases in Sun Java System Web Server 7.0 I tried out creating different authentication databases (keyfile, digestfile, LDAP, PAM) via Administration CLIs in Sun Java System Web Server 7.0. Writing it down in a blog. I went to server installation root and start Administration server and then started wadm. ./admin-server/bin/startserv ./bin/wadm--user=admin Pleaseenter admin-user-password>\*\*\* wadm> I created a file authentication database of type "keyfile" in config "test" and in virtual server "test". wadm> create-file-authdb--vs=test --config=test --path=/space/mykeyfile mykeyfile CLI201Command 'create-file-authdb' ran successfully Then created a file authentication database of type "digest", added "--syntax=digestfile" in the above command. wadm> create-file-authdb--vs=test --config=test --syntax=digestfile--path=/space/mydigestfile mydigestfile CLI201Command 'create-file-authdb' ran successfully To create authentication database of type PAM, I used "create-pam-authdb" CLI, wadm>create-pam-authdb--vs=test --config=test mypamauthdb CLI201 Command 'create-pam-authdb' ran successfully Note that PAM realm and PAM auth-db's are only supported on Solaris 9 and 10 and the server instance must be running as root. Change in server.xml <user>webservd</user> to <user>root</user> To add authentication database of type LDAP, I used "create-ldap-authdb" CLI. This CLI does not create LDAP database, it only configures it. I used an already existing Directory (LDAP) server located in server "test.sun.com", on port 389, with root suffix "o=TestCentral", bind dn "cn=Directory Manager", wadm> create-ldap-authdb --vs=test--config=test --bind-dn="cn=Directory Manager"--ldap-url=ldap://test.sun.com:389/o=TestCentral --config=testmyldapauthdb Please enter bind-password>\*\*\* CLI201 Command'create-ldap-authdb' ran successfully Note that if I had to add an LDAP server with SSL, all I had to do is change the url prefix from ldap:// to ldaps:// i.e. make LDAP url ldaps://test.sun.com:443/o=TestCentral instead. If CA of LDAP server is not a trusted CA (like Verisign etc.) then I would have to import LDAP Server's CA certificate into Web Server Instance's NSS database as well as in Web Server's admin-server's NSS database. Listed the authentication databases to check whether the databases were created successfully. wadm> list-authdbs--vs=test --config=test --all mykeyfile      keyfilemydigestfile   digestfilemypamauthdb    pammyldapauthdb   ldap Added a user "user1" in "mykeyfile" authentication database. wadm> create-user--authdb=mykeyfile --user-password=\*\*\* --vs=test --config=test user1 CLI201Command 'create-user' ran successfullySimilarly we can add users in other databases also, but I am skipping that part in this blog.List users to make sure everything is all right. wadm> list-users --config=test --vs=test--authdb=mykeyfile --all user1  - After I was done with all my changes, I deployed theconfiguration, wadm> deploy-config CLI201Command 'deploy-config' ran successfully I double checked that "user1" exists in "mykeyfile" >cat /space/mykeyfile user1;{SSHA}\*\*\*;Also I made sure that server.xml had all these auth-db entries : >cat server.xml    <virtual-server>    <name>test</name> ...    <auth-db>      <name>mykeyfile</name>      <url>file</url>      <property>        <name>keyfile</name>        <value>/space/mykeyfile</value>      </property>      <property>        <name>syntax</name>        <value>keyfile</value>      </property>    </auth-db>     <auth-db>      <name>mydigestfile</name>      <url>file</url>      <property>        <name>digestfile</name>        <value>/space/mydigestfile</value>      </property>      <property>        <name>syntax</name>        <value>digest</value>      </property>    </auth-db>    <auth-db>      <name>mypamauthdb</name>      <url>pam</url>    </auth-db>    <auth-db>      <name>myldapauthdb</name>      <url>ldap://test.sun.com:389/o%3dTestCentral</url>      <property>        <name>bindpw</name>        <value>\*\*\*</value>        <encoded>true</encoded>      </property>      <property>        <name>binddn</name>        <value>cn=Directory Manager</value>      </property>    </auth-db> ... I went to "https-test/config" directory and added an ACL manually in the end of the virtual server's ACL file (in this case it is default.acl) which allows only "user1" access. I could have done this from wadm also but I forgot to do so at that time. > tail -7 default.acl acl"uri=/"; authenticate(user,group) {        prompt = "Sun Java System Web Server";        database = "mykeyfile"; }; deny (all)user = "anyone"; allow(all) user = "user1";Note that database I have added is "mykeyfile" and should be the same as the name we specified during database creation. Started the instance and sent a request with "user1", access logs showed that "user1" has been authenticated successfully. $tail -f https-test/logs/access 123.456.78.90- user1 [19/Jan/2007:15:00:44 +0530] "GET /a.txt HTTP/1.1" 200 14 NOTE THAT SERVER RESTART IS REQUIRED WHEN YOU ADD A NEW DIGESTFILE/KEYFILE AUTHENTICATION DATABASE.

Creating Authentication Databases in Sun Java System Web Server 7.0 I tried out creating different authentication databases (keyfile, digestfile, LDAP, PAM) via Administration CLIs in Sun Java System...

Disabling TRACE in Sun Java System Web Server 7.0

Disabling TRACE in Sun Java System Web Server 7.0In Sun Java System Web Server 7.0 or Sun ONE Web Server 6.1, comment the TRACE service in obj.conf. #Service method="TRACE" fn="service-trace" For releases prior to Sun ONE Web Server 6.1: <Client method="TRACE">AuthTrans fn="set-variable"         remove-headers="transfer-encoding"         set-headers="content-length: -1"         error="501"</Client> It is a perception that Sun Java System Web Server (Web Server) is somehow vulnerable with these methods.These methods (except for TRACE) are NOT enabled by default in the Web Server. The fact that OPTIONS request lists these methods doesn't mean they could be exploited. Web Server responds to the HTTP OPTIONS method by reporting the methods understood. It should be noted that indication that a method is understood, however, is no guarantee that a method is permitted or will be executed. By default Web Server blocks all "privileged" HTTP methods behind the Access Control Lists (ACL) system. Attempts to invoke the methods will be responded to with an HTTP 401 error code (Unauthorized) requesting credentials from the User-Agent. If valid credentials are provided, or if the default ACL is disabled, Web Server will respond with an HTTP 405 error code (Method Not Allowed). You can also set it as the first ACE in the default.acl : deny absolute (http_trace, http_put, http_delete, http_move, http_mkdir, http_rmdir)user="anyone";Related Links : http://sunsolve.sun.com/search/document.do?assetkey=1-26-50603-1 http://forum.java.sun.com/thread.jspa?threadID=5064651 http://forum.java.sun.com/thread.jspa?threadID=5161735 http://forum.java.sun.com/thread.jspa?threadID=5166155 http://unix.derkeiler.com/Mailing-Lists/HP-UX-Admin/2003-06/0082.html http://sunsolve.sun.com/search/document.do?assetkey=1-66-200171-1

Disabling TRACE in Sun Java System Web Server 7.0 In Sun Java System Web Server 7.0 or Sun ONE Web Server 6.1, comment the TRACE service in obj.conf. #Service method="TRACE" fn="service-trace"For...

Migrating JKS Keystore Entries to NSS datbase in Sun Java System Web Server 7.0 or 6.x

Migrating JKS Keystore Entries to NSS database in Sun Java System Web Server 7.0 or 6.x Migrating JKS Keystore Entries to NSS database in Sun Java System Web Server 7.0 using Administration CLI I installed Sun Java System Web Server7.0 in <server-installation>directoryand started Administration server. $ ./admin-server/bin/startserv Now I used wadm to runAdministration CLIs $ ./bin/ wadm --user=admin Please enteradmin-user-password> typed-admin-password-here wadm> I have a config named "test" wadm> list-configs test If the keystore is in file /tmp/ my-jks-key-store.jks, keystore password is storepass,key password is keypass,and NSS DB password is nsspass, Irunmigrate-jks-keycert CLI wadm> migrate-jks-keycert--config=test --keystore=/tmp/my-jks-key-store.jks Pleaseenter keystore-password> storepass Pleaseenter key-password> keypass Pleaseenter certdb-password> nsspass CLI201 Command"migrate-jks-keycert" ran successful. I confirmed that migration worked by listing thecertificates using list-certsCLI wadm> list-certs --config=test CN=test,OU=WS,O=SUN,L=BLR,ST=KA,C=INwadm> After all the changes are done, run deploy-configCLI. From server instance's config directory run certutil toconfirm that the certificates are present in NSS DB. $.${server-install-dir}/bin/certutil-L -d ${server-instance-dir}/config CN=test,OU=WS,O=SUN,L=BLR,ST=KA,C=INu,u,u Migrating JKS Keystore Entries to NSS Datbase in Sun One Web Server 6.x We have to manually migrate jks keystore to NSS Database here is what we have to do From the server instance config directory, initialize NSS DB if required $${server-install-dir}/bin/certutil-N -d ${server-instance-dir}/config Enter a password which will be usedto encrypt your keys. The password should be at least 8characters long, and should contain at least one non-alphabeticcharacter. Enter new password: nsspass Re-enter password: nsspass Use keytool from JDK greater than version 6, it has importkeystore option which convertsJKS keystore to PKCS12 format. $/share/builds/components/jdk/1.6.0_01/SunOS/bin/keytool -importkeystore -srckeystoreserver-keystore.jks -srcstoretype JKS-deststoretype PKCS12 -destkeystore server-keystore.pkcs12 Enter destination keystore password:storepass Re-enter new password: storepass Enter source keystore password: storepass Entry for alias s1as successfullyimported. Import command completed: 1 entriessuccessfully imported, 0 entries failed or cancelled Now import them into NSS database $${server-install-dir}/bin/pk12util -iserver-keystore.pkcs12 -d ${server-instance-dir}/config Enter Password or Pin for "NSSCertificateDB": nsspass Enter passwordfor PKCS12 file: storepass pk12util: PKCS12 IMPORT SUCCESSFUL Verify if the certificate was imported $${server-install-dir}/bin/certutil-L -d . CN=Server,OU=JWS,O=SUN,ST=Some-State,C=AU                   u,u,u Then we may have to set trust flags using the following command $${server-install-dir}/bin/certutil -M -t "CTu,CTu,CTu".... For migrating Apache server(OpenSSL) certificate, we can use openssl utility to export it to a PKCS#12 file $ openssl pkcs12 -export -out <output-pkcs-file> -in <openssl-server-crt-file> -inkey <openssl-server-key-file> -nodes -name <alias> And import it into SJS Web Server NSS database using pk12util utility $ ${server-install-dir}/bin/pk12util -i <exported-pkcs12-file> -d ${server-instance-dir}/config References http://java.sun.com/javase/6/docs/technotes/tools/solaris/keytool.html http://www.mozilla.org/projects/security/pki/nss/tools/certutil.html http://www.openssl.org/docs/apps/pkcs12.html For more readability, the commands I used are shown in brown and the output is shownin green.Sun Java System Web Server 7.0 Technology Preview 3 is released and isFREE download it from here.

Migrating JKS Keystore Entries to NSS database in Sun Java System Web Server 7.0 or 6.x Migrating JKS Keystore Entries to NSS database in Sun Java System Web Server 7.0 using Administration CLI I...

Solaris Cryptographic Framework and Sun Java System Web Server 7.0

Solaris Cryptographic Framework and Sun Java System Web Server 7.0 Here are my initial experiments to useexternal PKCS#11 security module Solaris CryptographicFramework in Sun Java System Web Server 7.0.  Some references I liked in this regardare "man libpkcs11", "Using the Cryptographic Accelerator of the UltraSPARC T1 Processor" and Jyri's article "Configuring Solaris Cryptographic Framework and Sun Java System Web Server 7 on Systems With UltraSPARC T1 Processors" . Special Thanx to Basant who helped me. Note that Iexecuted these commands from the serverinstance's configdirectory.  For morereadability, the commands I used are shown in brown and the outputis shown in green. Initial steps First I move .sunwdirectory $mv $HOME/.sun $HOME/.sunw.OLD Then I initialized password/pin $pktoolsetpin Enter newPIN:typed-my-password-here Re-enternew PIN:typed-my-password-here Then disabled the followingmechanisms Note that these commandsneed to be executed as root. #cryptoadmdisable provider=/usr/lib/security/\\$ISA/pkcs11_kernel.somechanism=CKM_SSL3_PRE_MASTER_KEY_GEN,CKM_SSL3_MASTER_KEY_DERIVE,CKM_SSL3_KEY_AND_MAC_DERIVE,CKM_SSL3_MASTER_KEY_DERIVE_DH,CKM_SSL3_MD5_MAC,CKM_SSL3_SHA1_MAC #cryptoadmdisable provider=/usr/lib/security/\\$ISA/pkcs11_softtoken.somechanism=CKM_SSL3_PRE_MASTER_KEY_GEN,CKM_SSL3_MASTER_KEY_DERIVE,CKM_SSL3_KEY_AND_MAC_DERIVE,CKM_SSL3_MASTER_KEY_DERIVE_DH,CKM_SSL3_MD5_MAC,CKM_SSL3_SHA1_MAC(if pkcs11_softtoken_extra.sois used, disable these mechanisms in that also) #cryptoadmlist -p user-level providers: ===================== /usr/lib/security/$ISA/pkcs11_kernel.so:all mechanisms are enabled, exceptCKM_SSL3_SHA1_MAC,CKM_SSL3_MD5_MAC,CKM_SSL3_MASTER_KEY_DERIVE_DH,CKM_SSL3_KEY_AND_MAC_DERIVE,CKM_SSL3_MASTER_KEY_DERIVE,CKM_SSL3_PRE_MASTER_KEY_GEN. /usr/lib/security/$ISA/pkcs11_softtoken.so:all mechanisms are enabled, exceptCKM_SSL3_SHA1_MAC,CKM_SSL3_MD5_MAC,CKM_SSL3_MASTER_KEY_DERIVE_DH,CKM_SSL3_KEY_AND_MAC_DERIVE,CKM_SSL3_MASTER_KEY_DERIVE,CKM_SSL3_PRE_MASTER_KEY_GEN.random is enabled. ... Registering PKCS#11 library I  have used PKCS#11 library/usr/lib/libpkcs11.so (for 64bit, it is /usr/lib/64/libpkcs11.so). The following command addedthe Solaris crypto framework module to the NSS database in the configdirectory : $../../lib/modutil-dbdir . -add "scf" -libfile /usr/lib/libpkcs11.so-mechanisms RSA ... Module"scf" added to database. $../../lib/modutil-dbdir . -enable "scf" Verified the above steps, $../../lib/modutil -dbdir . -nocertdb-list Listing ofPKCS #11 Modules -----------------------------------------------------------   1. NSS Internal PKCS #11 Module          slots: 2 slots attached         status: loaded          slot: NSS Internal Cryptographic Services         token: NSS Generic Crypto Services          slot: NSS User Private Key and Certificate Services         token: NSS Certificate DB   2. scf         library name: /usr/lib/libpkcs11.so          slots: 1 slot attached         status: loaded          slot: Sun Crypto Softtoken         token: Sun Software PKCS#11 softtoken   3. Root Certs         library name: libnssckbi.so          slots: There are no slots attached to this module         status: Not loaded -----------------------------------------------------------Note that slot "Sun Crypto Softtoken" has token"Sun Software PKCS#11 softtoken".I will be using this token in the next stages. Creating Server Certificates The normal process for requesting andinstalling certificates is used.Only with a difference, create all certificate and keys inthat security module, not using"internal" NSS database token,but using the "Sun SoftwarePKCS#11 softtoken" token instead. 1. Exporting and Importing already existing certificates using pk12util If I already had certificates in NSS database, I could have exported and imported them using pk12util $pk12util –o server.pk12 –d . –n MyCert $pk12util –i server.pk12 –d . –h “Sun Software PKCS#11 softtoken” By default, certutil / pk12util searches for databases named cert8.db and key3.db, but some of the versions of Web Server use alternate names such as https-instance-hostname-cert8.db and https-instance-hostname-key3.db in that case add -P parameter for the prefix. 2. Using certutil to create self signed server certificate I used NSS utility "certutil"to create a self signed server certificates. $../../bin/certutil-S -d . -n MyCert -s "CN=test.sun.com" -x -t "u,u,u" -h "Sun Software PKCS#11 softtoken"-5 Enter Password or Pin for "SunSoftware PKCS#11 softtoken":typed-my-password-here A randomseed must be generated that will be used in the creationof your key.  One of the easiest ways to create a randomseed is to use the timing of keystrokes on a keyboard. To begin,type keys on the keyboard until this progress meter isfull.  DO NOT USE THE AUTOREPEAT FUNCTION ON YOUR KEYBOARD! Continuetyping until the progress meter is full: |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*| Finished. Press enter to continue: Generatingkey.  This may take a few moments...                           0 - SSL Client                           1 - SSL Server                           2 - S/MIME                           3 - Object Signing                           4 - Reserved for future use                           5 - SSL CA                           6 - S/MIME CA                           7 - Object Signing CA                           Other to finish 1                           0 - SSL Client                           1 - SSL Server                           2 - S/MIME                           3 - Object Signing                           4 - Reserved for future use                           5 - SSL CA                           6 - S/MIME CA                           7 - Object Signing CA                           Other to finish 5                           0 - SSL Client                           1 - SSL Server                           2 - S/MIME                           3 - Object Signing                           4 - Reserved for future use                           5 - SSL CA                           6 - S/MIME CA                           7 - Object Signing CA                           Other to finish 9 Is this acritical extension [y/N]? y Verified that the certificate was added to the database $../../lib/certutil-L -d . -h "Sun Software PKCS#11softtoken" Enter Password or Pin for "Sun SoftwarePKCS#11 softtoken":typed-my-password-here SunSoftware PKCS#11softtoken:MyCert                       u,u,u Enable SSL for the Web Serverinstance In server.xml,  enabled sslfor http-listenerelement, and  added server certificate nickname correctly. .... <http-listener>   ... <ssl>  <enabled>true</enabled>  <server-cert-nickname>Sun Software PKCS#11softtoken:MyCert</server-cert-nickname>  </ssl> </http-listener>... Note the prefix "SunSoftware PKCS#11 softtoken:". 3. Using Administration CLI to create self signed certificate and enabling SSL Start admin-server, from <server-installation>/bindirectory,$wadm --user=admin Please enteradmin-user-password>typed-admin-server-password-here Sun Java System Web Server7.0-Technology-Preview-3 B09/20/2006 10:07 wadm>create-selfsigned-cert--config=test.sun.com --server-name=test.sun.com --nickname=MyCert--token="Sun Software PKCS#11 softtoken" ADMIN4099: Token 'SunSoftware PKCS#11 softtoken' was not found wadm>list-tokens --config=test.sun.com internal The reason for this error is I ran modutil into server instance'sconfig directory so I need to pull-config (I should have run modutilcommand from admin-server/config-store/test.sun.com/config directory toavoid this) wadm>pull-config --config=test.sun.com test.sun.com CLI201 Command 'pull-config'ran successfully wadm>list-tokens --config=test.sun.com internal SunSoftware PKCS#11 softtokenThis looks ok. wadm>create-selfsigned-cert--config=test.sun.com --server-name=test.sun.com--nickname=MyCert--token="Sun Software PKCS#11 softtoken" Pleaseenter token-pin>typed-my-password-here CLI201 Command'create-selfsigned-cert' ransuccessfully wadm>set-ssl-prop --config=test.sun.com--http-listener=http-listener-1 enabled=trueserver-cert-nickname="Sun SoftwarePKCS#11 softtoken:MyCert" CLI201Command 'set-ssl-prop' ran successfully wadm>deploy-configtest.sun.com CLI201 Command'deploy-config' ran success Now I started the Web Server, $../bin/startserv Sun JavaSystem Web Server 7.0 B09/11/2006 12:04 Pleaseenter the PIN for the "Sun Software PKCS#11 softtoken" token:typed-my-password-here info:HTTP3072: http-listener-1: https://test.sun.com:2222 ready to acceptrequests info:CORE3274: successful server startup I sent a request through the browser to https://test.sun.com:2222, andthe server served the request. More References Jyri's BigAdmin Article "Configuring Solaris Cryptographic Framework and Sun Java System Web Server 7 on Systems With UltraSPARC T1 Processors" Using the Cryptographic Accelerator of the UltraSPARC T1 Processor man libpkcs11 man cryptoadm Sun crypto accelerator 6000 user's guide has chapter on Installing and configuring with Sun Java System Web Server 6.1. Tip : If 1. Web server does not present the Intermediate CA certificates installed as Server Certificate Chain to the browser and that causes the certificate validation by the browser to fail.or 2. Client authentication fails with the following error message in the errors log .  Root CA cert has been installed to the certificate database. failure (16670): HTTP3068: Error receiving request from 123.45.67.897(SEC_ERROR_UNKNOWN_ISSUER: Peer's certificate is signed by an unknown issuer) These two issues are caused by the /.sunw directory not being accessible by the web server running user "webservd". That directory has permissions 0700 and is owned by root. Web Server starts up as root and then changes (using setuid) to user "webservd".  Solution to this is 1) Have the web server running as root2) Open up the permission on /.sunw so that it is readable by the web server running user3) Set  the environment variable SOFTTOKEN_DIR to point to some directory that is owned by webservd before the web server is started. The SCF will then access the files in $SOFTTOKEN_DIR/pkcs11_softoken/ during execution. Read my next blog Using builtin hardware accelerators of Niagara 1 (Sun Fire T 2000) server with SSL enabled Sun Java System Web Server 7.0 instance

Solaris Cryptographic Framework and Sun Java System Web Server 7.0 Here are my initial experiments to use external PKCS#11 security module Solaris CryptographicFramework in Sun Java System Web Server...

Cross Site Scripting Prevention in Sun Java System Web Server 7.0

Cross Site Scripting Prevention in Sun Java System Web Server 7.0     Check out the new improvements wemade in Sun Java System Web Server 7.0. It can be downloadedforfree from http://www.sun.com/download/index.jsp?cat=Web%20%26%20Proxy%20Servers&tab=3&subcat=Web%20Servers.In this blog I will talk about Cross Site Scripting (XSS) prevention. Obj.conf now supports a lot of features which allows you to use it alot like a programming language, which allows us to configure in ourWeb Server features similar to in ModSecurity Apache Module. The main method of preventing Cross Site Scripting (XSS) is throughentity encoding, using entities such as "&lt;".  We now have aintroduced a native input stage filter based on sed which can do XSSfiltering. This sed-requestfilter applies sed editcommands to an incoming request entity body, e.g. an uploaded file orsubmitted form. Input fn="insert-filter" ...filter="sed-request" sed="script" [sed="script" ... ] Where "script" is theactual sed script you want to run on request body.For example, if we take example of request body posted in HTML formcontaining  "<"and ">" characters. In ModSecurity in Apache serveryou have SecFilter like SecFilterEngine OnSecFilterScanPOST OnSecFilter "<(.|\\n)+>" By adding the following in obj.conf, Web Server 7.0 will encode any < and > characters. Inputfn="insert-filter" method="POST" filter="sed-request" sed="s/(<|%3c)/\\\\&lt;/gi" sed="s/(>|%3e)/\\\\&gt;/gi" \* Note that because POST bodies are usually URL-encoded, it isimportant to check for URL-encoded forms also when editing POST. "%3C" is the URL-encoded formof "<" and bodies. "%3E" is the URI-encoded form of ">". In Web Server 7.0 update 2 or 3 onwards, you can have a config filemyrules.txt as shown below SecRuleEngine OnSecRequestBodyAccess OnSecRule REQUEST_BODY "<(.|\\n)+>" I have added in server.xml <config-file>myrules.txt</config-file> I have added a simple cgi script to test my stuff. $cat https-test/docs/cgi-bin/test.pl #!/tools/ns/bin/perl5 binmode(STDOUT); binmode(STDIN); if ($ENV{'REQUEST_METHOD'}eq "POST") {    read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});     @pairs =split(/&/, $buffer); } else {     @pairs =split(/&/, $ENV{'QUERY_STRING'}); } foreach $pair (@pairs) {     ($key,$value) = split(/=/, $pair);     $value=~ tr/+/ /;     $value=~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;     $value=~ tr/\\cM/\\n/;    eval("\\$$key = \\"$value\\"");    $FORM{$key} = $value; } print "Content-Type:text/html\\n\\n"; print "CGI valuespassed\\n\\n"; if ($#pairs < 0) {     print"No CGI Variables\\n"; } else {     foreach$var (keys(%FORM)) {        print "$var $FORM{$var}\\n";     } }exit; So when we send a request without < and > , it goes through fine asshown below $telnet 0 3333 POST /cgi-bin/test.plHTTP/1.0 Content-length: 10 abcde12345 HTTP/1.1 200 OK Server:Sun-Java-System-Web-Server/7.0 Date: Wed, 16 Jul 200807:56:47 GMT Content-type: text/html Connection: close CGI values passed abcde12345 When we send request with <and > as shown belowwe get forbidden error $telnet 0 3333 POST /cgi-bin/test.pl HTTP/1.0 Content-length: 10 ab<cd>12 HTTP/1.1 403 Forbidden Server: Sun-Java-System-Web-Server/7.0 Date: Wed, 16 Jul 2008 07:57:24 GMT Content-length: 142 Content-type: text/html Connection: close <HTML><HEAD><TITLE>Forbidden</TITLE></HEAD> <BODY><H1>Forbidden</H1> Your client is not allowed to access the requested object. </BODY></HTML> When we send a request with just < it doesn't match thepattern, and hence passes : $telnet 0 3333 POST /cgi-bin/test.pl HTTP/1.0 Content-length: 10 ab<cdef12345 HTTP/1.1 200 OK Server: Sun-Java-System-Web-Server/7.0 Date: Wed, 16 Jul 2008 07:58:03 GMT Content-type: text/html Connection: close CGI values passed ab<cdef123 More details on SecRule and other related Directives supported in Web Server 7.0 update 2 onwards are in this blog.

Cross Site Scripting Prevention in Sun Java System Web Server 7.0     Check out the new improvements we made in Sun Java System Web Server 7.0. It can be downloaded for free from http://www.sun.com/dow...

Dynamic compression of static files in Sun Java System Web Server 7.0

Dynamic compression of static files in Sun Java System Web Server 7.0     Check out the new cool features in Sun JavaSystem Web Server 7.0 which can be downloaded forfree from http://www.sun.com/download/index.jsp?cat=Web%20%26%20Proxy%20Servers&tab=3&subcat=Web%20Servers .In this blog I will talk about dynamic compression of static files.     We have also implemented caching of compressed datafor static files. We have added a new service function "compress-file" which willcompress static files (if the compressed file doesn't exist) and servesit from the cache if the compressed version already exists.Lets say if I want to create .gz files on the fly for  staticfiles, all I have to do is to modify obj.conf as shown below. Use itwith find-compressed SAF.Add it before send-filefunction. <Objectname="default">... NameTrans fn="assign-name" from="\*.html" name="find-compressed"... Service method=(GET|HEAD|POST)type=\*~magnus-internal/\* fn=compress-file subdir=".compressed-files"Service method=(GET|HEAD|POST) type=\*~magnus-internal/\* fn=send-file...</Object> <Object name="find-compressed">PathCheck fn="find-compressed"</Object> Note that subdir is a directory name only relative to thedirectory in which the originalnon-compressed file is located. "." means use thesame directory as the original file to create compressed files. Parametersthis new "compress-file" SAF takes are subdir,check-age(true), vary(true), compression-level(6), min-size(256),max-size(1048576). Let me take an example, if we have a static file in the docrootstatic1.txt. #ls -al -rwxr--r-- 1 root root 1386 Aug 121:16 static1.txt Send a GET request with Accept-Encoding: gzip header,it returns the compressed content, #telnet 0 2701 Trying 0.0.0.0... Connected to 0. Escape character is '\^]'. GET /static1.txt HTTP/1.0 Accept-Encoding:gzip HTTP/1.1 200 OK Server:Sun-Java-System-Web-Server/7 Date: Mon, 01 Aug 2005 15:47:11GMT Content-length: 892 Content-type: text/plain Vary: accept-encoding Content-encoding: x-gzip Last-modified: Mon, 01 Aug 200515:47:11 GMT Accept-ranges: bytes Connection: close z`B®(¼?øKä0¢TeÑB¨÷Q$A .... O9êÓÌ iñÂ6ì¹­ ³Í§8zªè)óÍ6 We can verify that the Web Server creates a .gz file in .compressed-files subdirectory. # ls -la drwxr-xr-x 2 webservd webservd512 Aug 1 21:17 .compressed-files -rwxr-xr-x 1 root root 1386 Aug 121:16 static1.txt # ls -l .compressed-files/ total 2 -rw------- 1 webservd webservd892 Aug 1 21:17 static1.txt.gz Also note that the content length on access file is the same as the .gzfile in the .compressed-filesdirectory. #tail ../logs/access format=%Ses->client.ip% -%Req->vars.auth-user% [%SYSDATE%] "%Req->reqpb.clf-request%"%Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% abc.def.com - -[01/Aug/2005:21:17:11 +0530] "GET /static1.txt HTTP/1.0" 200 892 ...

Dynamic compression of static files in Sun Java System Web Server 7.0     Check out the new cool features in Sun Java System Web Server 7.0 which can be downloaded for free from http://www.sun.com/down...

New Feature WebDAV Access Control Protocol In Sun Java System Web Server 7.0

New Feature WebDAV Access Control Protocol In Sun Java System Web Server 7.0 There's a whole bunch of new stuff in Sun Java System Web Server 7.0, and you can download it for free from http://www.sun.com/download/index.jsp?cat=Web%20%26%20Proxy%20Servers&tab=3&subcat=Web%20Servers .Remote Content Manipulation And Remote Access ControlWhenever I create website for myself for personal use or when I need touse a website that is common to our group with more than one membermodifying the content, I always want to Create, modify or delete files on a website via a HTTPwithout actually ftp'ing the files into the machine that contains thecontent. Lock a page when I am modifying a file and prevent my teammembers from modifying it at the same time. Create directories on a website just by sending a simple HTTPcommand remotely rather than logging into the machine where web serveris installed and then creating the directory by typing "mkdir foo". Copy and move files around without actually logging into themachine that contains the content. Associating properties to a file. More than that, I also want to Control who can view, modify (access) the file I have createdrather thantelling the web site administrator to do it for me that too without aserverrestart. Finding out how many files I myself have created inside adirectory on a website. Finding out who has created a file (Who is the owner of a file). Finding out how many files I can view, modify inside a directoryon a website. Finer control of privileges like write-content, write-properties, bind, write-acl thanmerely write privileges. WebDAV and WebDAV Access Control Protocol support in Sun Java System Web Server 7Check this out, we have implemented ALL this Sun Java SystemWeb Server 7. It was an experience reading RFC and implementingit. What I found the toughest part was reusing the existinginfrastructure with bare minimum changes to the core and yet meetingthe deadlines !! Sun Java System Web Server 7 now Conforms to WebDAV protocol as defined by the RFC2518. Supports the following HTTP methods GET, HEAD, OPTIONS, PUT, LOCK, UNLOCK,MKCOL, COPY, MOVE PROPPATCH, DELETE, PROPFIND. Conforms to WebDAV AccessControl Protocol as defined bythe RFC 3744. Also supports ACL,REPORT(DAV:acl-principal-prop-set, DAV:principal-match,DAV:expand-property), PROPFIND (DAV:acl andDAV:current-user-privilege-set property). Has finer access rights like dav:all, dav:read, dav:read-acl,dav:read-current-user-privilege-set, dav:write, dav:write-acl,dav:write-properties, dav:write-content, dav:bind, dav:unbind,dav:unlock. Assuming I am a user alpha(and I have DAV:write-acl privilegeson this file). I want to allows all users in group foo,read privileges, and user alphaall privileges on a file I created, all I have to do is send an ACLrequest through a WebDAV client or telnet as follows ACL/col1/myfile.html HTTP/1.1Host: test.sun.comContent-type: text/xml; charset="utf-8"Content-Length: xxxAuthorization: xxx <?xml version="1.0" encoding="utf-8"?><D:multistatus xmlns:D="DAV:">  <D:acl>    <D:ace>      <D:principal>         <D:href>http://test.sun.com/magnus-internal/my-ldap-auth-db/groups/foo</D:href>      </D:principal>      <D:grant>       <D:privilege><D:read/></D:privilege>      </D:grant>    </D:ace>    <D:ace>      <D:principal>          <D:href>http://test.sun.com/magnus-internal/my-ldap-auth-db/users/alpha</D:href>      </D:principal>      <D:grant>       <D:privilege><D:all/></D:privilege>      </D:grant>     </D:ace>  </D:acl></D:multistatus> Explore WebDAV and WebDAV Access Control features and let us know,

New Feature WebDAV Access Control Protocol In Sun Java System Web Server 7.0  There's a whole bunch of new stuff in Sun Java System Web Server 7.0, and you can download it for free from http://www.sun....

Configuring WebDAV in Sun Java System Web Server 7.0

Configuring WebDAV in Sun Java System Web Server 7.0 Download Sun Java System Web Server 7.0 absolutely free from here . Also check out my next blog about WebDAV Access Control Protocol(RFC 3744) support in Sun Java System Web Server 7.0. OverviewWebDAV stands for "Web-based Distributed Authoring and Versioning". Itis a set of extensions to the HTTP protocol which allows users tocollaboratively edit and manage files on remote web servers. Sun JavaSystem Web Server 7.0 conforms to WebDAV protocol as defined by the RFC2518 and the WebDAV Access Control Protocol specification as defined bythe RFC 3744. In this article I will describe how to configure WebDAV in Sun Java System Web Server 7.0 via Administration CLI. Steps to configure WebDAV in Sun Java System Web Server 7.0Go to Sun Java System Web Server 7.0 installation directory.Start Admin Server. ./admin-server/bin/startservStart wadm. ./bin/wadm--user=admin Please enteradmin-user-password> Sun Java System Web Server 7.0B12/02/2005 18:02 wadm> Enabling WebDAV at instance level wadm>enable-webdav --config=test CLI201 Command "enable-webdav"ran successfully Note the new lines in magnus.conf, default.acl and server.xml asgiven below magnus.conf ...Init fn="load-modules" shlib="libdavplugin.so"funcs="init-dav,ntrans-dav,pcheck-dav,service-dav"shlib_flags="(global|now)"Init fn="init-dav" LateInit="yes" \*For Windows it is davplugin.dll and for HP-UX itis libdavplugin.sl server.xml ...<dav/>... default.acl ...acl "uri=/magnus-internal";deny (all) user = "anyone";allow (list) user = "all"; acl "dav-src";deny (all) user = "anyone"; Adding a DAV collection wadm>create-dav-collection--config=test --vs=test --uri=/dav --sourceuri=/source CLI201 Command "create-dav-collection" ran successfully To verify that the command ransuccessfully you can see the changes in server.xml and obj.conf asgiven belowserver.xml ...<dav-collection><uri>/dav</uri><source-uri>/source</source-uri></dav-collection>... obj.conf ...NameTrans fn="ntrans-dav" name="dav"...PathCheck fn="pcheck-dav"...Service fn="service-dav"method="(OPTIONS|PUT|DELETE|COPY|MOVE|PROPFIND|PROPPATCH|LOCK|UNLOCK|MKCOL|ACL|REPORT)"...<Object name="dav">PathCheck fn="check-acl" acl="dav-src"Service fn="service-dav"method="(GET|HEAD|POST|PUT|DELETE|COPY|MOVE|PROPFIND|PROPPATCH|LOCK|UNLOCK|MKCOL|ACL|REPORT)"</Object> Now set appropriate ACLs in default.acl or create a new ACL file forthis virtual server and set ACLs in it. Make sure that the users haveappropriate "rwx" permissions in docroot as required.Note that ntrans-dav directive has to be the first Nametrans directive (before ntrans-j2ee directive) and pcheck-dav directive has to be the last PathCheck directive (after find-index-j2ee directive). Difference Between Source Uri and Content UriSuppose we crate a dav-collection with content uri "/dav" and source uri as "/source".Sending a request, GET /dav/test.jsp HTTP/1.0will return the output of the jsp Hello World!! Sending a request, GET /source/test.jsp HTTP/1.0will return the source of the jsp < % System.out.println("HelloWorld!!"); % > Advanced CLIs Listing DAV collections Normal listingwadm>list-dav-collections --config=test --vs=test /dav Detailed listingwadm>list-dav-collections --config=test --vs=test -luri    source-uri    enabled ------------------------------ /dav    /source    true Setting properties at server levelwadm>set-webdav-prop --config=test max-xml-request-body-size=1024 CLI201 Command "set-webdav-prop" ran successfully Verify by getting the properties as shown in the next section. Getting properties of DAV at server levelwadm>get-webdav-prop --config=test acl-db-max-entries=10 auth-method=basic default-owner=webservd property-db-update-interval=0 lock-db-update-interval=0 max-xml-request-body-size=1024 auth-prompt=Sun Java System WebServer WebDAV max-propfind-depth=1 property-db-max-size=8192 acl-db-update-interval=0 acl-db-max-size=8192 max-expand-property-depth=3 auth-auth-db-name=keyfile max-report-response-elements=1000 enabled=true min-lock-timeout=0 Setting DAV Collection Propertieswadm>set-dav-collection-prop --config=test --vs=test --uri=/dav min-lock-timeout=60 CLI201Command "set-dav-collection-prop" ran successfully Verify this by getting the properties as shown in next section Getting properties of a DAV Collectionwadm> get-dav-collection-prop --config=test --vs=test --uri=/dav acl-db-max-entries=10 auth-method=basic property-db-update-interval=0 lock-db-update-interval=0 source-uri=/source max-xml-request-body-size=8192 auth-prompt=Sun Java System WebServer WebDAV max-propfind-depth=1 property-db-max-size=8192 acl-db-update-interval=0 acl-db-max-size=8192 max-expand-property-depth=3 max-report-response-elements=1000 uri=/dav enabled=true auth-auth-db-name=keyfile min-lock-timeout=60 Disabling DAV Collectionwadm>disable-dav-collection --config=test --vs=test --uri=/davCLI201 Command "disable-dav-collection" ran successfully Enabling DAV Collectionwadm>enable-dav-collection --config=test --vs=test --uri=/davCLI201Command "enable-dav-collection" ran successfully Deleting a DAV collectionwadm>delete-dav-collection --config=test --vs=test --uri=/davCLI201 Command "delete-dav-collection" ran successfully Disabling WebDAV at server levelwadm>disable-webdav --config=test CLI201Command "disable-webdav" ran successfully

Configuring WebDAV in Sun Java System Web Server 7.0 Download Sun Java System Web Server 7.0 absolutely free from here . Also check out my next blog about WebDAV Access Control Protocol(RFC 3744)...

Configuring Reverse Proxy in Sun Java System Web Server 7.0

Configuring Reverse Proxy in Oracle iPlanet Web Server 7.0  Origin Server <-- HTTP ---> Reverse Proxy Server <-- HTTP --> client/Browser IntroductionA reverse proxy is a proxy that appears to be a web server (origin server) to clients but in reality forwards the requests it receives to one or more origin servers. Because a reverse proxy presents itself as an origin server, clients do not need to be configured to use a reverse proxy. By configuring a given reverse proxy to forward requests to multiple similarly configured origin servers, a reverse proxy can operate as an application level software load balancer. In a typical deployment one or more reverse proxies will be deployed between the browsers and the origin servers.In this article I will describe how to configure reverse proxy in Sun Java System Web Server 7.0 via CLI. Configuring reverse proxy in Oracle iPlanet Web Server 7.0Go to Oracle iPlanet Wwb Server 7.0 installation directory.Start Admin Server ./admin-server/bin/startservStart wadm. /bin/wadm --user=adminPlease enteradmin-user-password>Sun Java System Web Server 7B12/02/2005 18:02 wadm> Then run create-reverse-proxy CLI wadm>create-reverse-proxy--config test --vs test --uri-prefix=/ --server=http://abc.sun.com:8080 where server can contain more than one comma separated URLs. obj.conf should have entries that look like #obj.conf<Object name="default">...NameTrans fn="map" from="/" name="reverse-proxy-/" to="http:/"...</Object>...<Object name="reverse-proxy-/">Route fn="set-origin-server" server="http://abc.sun.com:8080"</Object> <Object ppath="http:\*">Service fn="proxy-retrieve" method="\*"</Object> Other Advanced reverse proxy CLIs list-reverse-proxy-uris set-reverse-proxy-prop get-reverse-proxy-prop forward-reverse-proxy-header block-reverse-proxy-header list-reverse-proxy-headers Related links : Regular Expression Redirects in Web Server 7 301 Permanent Redirects in Web Server 7 Other blogs I have written on this : There are various SSL and non SSL configurations we can have for Reverse Proxy and Origin Servers Origin Server <--- HTTP ---> Reverse Proxy Server <--- HTTP ---> client/Browser Origin Server <-- HTTP ---> Reverse Proxy Server <-- HTTPS --> client/Browser i.e. Reverse proxy SSL termination End point Origin Server <-- HTTPS ---> Reverse Proxy Server <-- HTTP --> client/Browser Origin Server <-- HTTPS ---> Reverse Proxy Server <-- HTTPS --> client/Browser In this blog I tried out scenario 1.

Configuring Reverse Proxy in Oracle iPlanet Web Server 7.0  Origin Server <-- HTTP ---> Reverse Proxy Server <-- HTTP --> client/Browser Introduction A reverse proxy is a proxy that appears to be a web...

Access Control In Sun Java System Web Server 7.0 - II

Access Control In Sun JavaSystem Web Server 7.0 -IIAfter my first blog about Access Control in general, let me try to talk about some more ACL related topics.Access Rights in Access Control Lists in Sun Java System Web Server 7.0Access rights that can be used to set access control in ACL file in SunJava System Web Server 7.0 - are all read execute info write list delete http_<method> dav:read-acl dav:read-current-user-privilege-set If we want to grant write,delete,listrights ONLY to authenticated users but grant read, execute, info rights toall users (authenticated as well as unauthenticated users), we needto have an ACL like. #default.acl... acl "default"; authenticate (user, group) {   prompt = "Sun JavaSystem Web Server"; }; allow(read, execute, info)user = "anyone"; allow(list, write, delete)user = "all"; This isalready set by default in <WS_INSTALLATION_DIR>/https-<servername>/config/default.aclfile. If we want to grant write,delete,listaccess to a set of high privileged users only (users alpha,beta and gamma) andgrant read,execute,inforights to ONLY authenticated users, set the following ACLin <WS_INSTALLATION_DIR>/https-<servername>/config/default.aclfile in the end. (We can also add this in thatVirtual Server's ACL file if we are using multiple virtual servers.) #default.acl...acl "uri=/" deny (all) user="anyone"; allow(read,execute, info) user = "all"; allow(list, write, delete)user = "alpha" or user = "beta" or user = "gamma"; Note that the first ACE (ACE starting with deny)  MUSTBE added. It denies all rightsto "anyone" i.e. allusers (both authenticated and unauthenticated users). Second ACE, grants read,execute and info rights to authenticatedusers only. Third ACE,  grants list,write, deleterights to alpha, beta or gammaonly. Instead of adding individual users, we can also create agroup (letssay "premium") inauthentication database and add these three users  alpha, beta and gamma in thegroup. The new ACE will be #default.acl...acl "uri=/"deny (all) user="anyone"; allow (read,execute, info) user = "all"; allow (list, write, delete)group = "premium"; Mapping between rights and HTTP Methods Rights Maps to HTTP Methods read GET,HEAD, TRACE, OPTIONS, COPY, BCOPY write PUT,MKDIR, LOCK, UNLOCK, PROPPATCH, MKCOL, ACL, VERSION-CONTROL, CHECKOUT,UNCHECKOUT, CHECKIN, MKWORKSPACE, UPDATE, LABEL, MERGE,BASELINE-CONTROL, MKACTIVITY, BPROPPATCH execute POST,CONNECT, SUBSCRIBE, UNSUBSCRIBE, NOTIFY, POLL delete DELETE,RMDIR, MOVE, BDELETE, BMOVE info HEAD,TRACE, OPTIONS list INDEX,PROPFIND, REPORT, SEARCH, BPROPFIND Notethat to get directory listing weneed "list" rights. Note that we have HEAD, TRACE and OPTIONS map to both read and info rights. \* Will discuss dav:read-acland dav:read-current-user-privilege-set  later. We can also set these http_<method>into ACL file for finer access control. #default.acl ...acl "uri=/dirGETNotAllowed"; authenticate (user, group) {   database ="mykeyfile";   method = "basic"; }; deny (http_GET) ip = "123.45.\*"; ACL EvaluationThe server goes through the list of ACEsto determine the access permissions. Attribute pattern anyone means all users. Ifthat ACE's attribute is useror group and it does NOTcontain pattern anyone,the server first authenticates theuser. The firstACE is evaluated and it denies/allows access to the current user. Theserver then checks the second ACE in the list, and if it matches, thenextACE is used. The servercontinues down the list until it reaches the end or it reaches an absoluteACE\*. The last matchingACE determines if access is allowed or denied. \*absolute ACE is an ACEthat contains absolutekeyword. Lets take the example of a simple ACL file and letssee how they are effective for some requests. #default.acl version 3.0; acl "default";# call it ACE#1 allow (read,execute, info) user = "anyone"; # call it ACE#2 allow (list, write, delete)user = "all";...acl "uri=/file.html"; # call it ACE#3 deny (info) user = "alpha"; Request Serverreads ACEs from ACL file ServerEvaluates ACEs in this order Result HEAD / Readsall ACEs containing "all","read", "http_head" and "info"rightsapplicable for thisresource. (It reads ACE#1 in the ACLfile above) ACE#1 is evaluated, operationis allowed. HEAD request is allowed. GET / or GET /file.html Reads all ACEs containing "all", "read" and "http_get" rights applicablefor thisresource. (It reads ACE#1in the ACL file above) ACE#1 is evaluated, operationis allowed. GET request is allowed. HEAD /file.html as user alpha Readsall ACEs containing "all", "read", "http_head" and "info"rightsapplicable for thisresource. (It reads ACE#1 andACE#3 in the ACL file above) ACE#1 is evaluated, operation is allowed.\* ACE#3 isevaluated, operation is denied(for alpha). HEAD request is denied. HEAD /file.html as user beta ACE#1 is evaluated, operation is allowed.\* ACE#3 is evaluated, operation is allowed (for beta). HEAD request is allowed. HEAD /file.html\*\* ACE#1 is evaluated, operation is allowed.\* ACE#3 isevaluated, operation is denied(for unauthenticated users). HEAD request is denied. \* It continues tothe next step as "absolute"keyword is not found. \*\* Although we do not have any deny ACEs for "anyone" (unauthenticated users), having a deny or allow ACEfor a particular user implicitly denies access to unauthenticated users. Absolute ACEACL Evaluation stops if an "absolute"ACE is found. For example if we have ACEs like #default.acl version 3.0; acl "default";# call it ACE #1 allow absolute (read,execute, info) user = "anyone"; # call it ACE #2 allow (list, write, delete)user ="all"; acl "uri=/file.jsp"; # Call it ACE#3 is ignored deny (info) user="all";... For HEAD request for /file.jsp, request is allowedfor all users even though ACE#3 sets a deny.  During theevaluationof ACE#1 as it encounters absolutekeyword, it never evaluates ACE#3 and other ACEs below it. References Access Control Programmer Guide 3.x - new link http://download.oracle.com/docs/cd/E19957-01/816-5643-10/816-5643-10.pdf old link http://dlc.sun.com/pdf/816-5643-10/816-5643-10.pdf

Access Control In Sun Java System Web Server 7.0 -II After my first blog about Access Control in general, let me try to talk about some more ACL related topics. Access Rights in Access Control Lists...

Access Control In Sun Java System Web Server 7.0

Access Control In Sun Java System Web Server 7.0You can now download Sun Java System Web Server 7.0 for FREE from http://www.sun.com/download/index.jsp?cat=Web%20%26%20Proxy%20Servers&tab=3&subcat=Web%20Servers . What Is Access Control? Access control allows us to determine and manage Who (subject) can access the resourceson our website Which resources (like files ordirectories) they can access Which operations can they perform (like creating a file, POST-ingcontents to the server). We can allow or denyaccess based on: Who is making the request (forexample, user or groupusing user, groupattributes). Where the request is coming from(forexample, host or IP using ip, dnsattributes). When the request is happening (forexample, time of day or day of week using timeofday, dayofweekattributes). What type of connection is being used(for example, SSL using sslattribute). Authentication is the processof acquiring and verifying the attributes ofthe subject which help to identify the subject. For example,authentication may involve prompting the user to login with a usernameand password, and then looking in a database to verify that the user'spassword is correct. Authorization is the process ofchecking the rights (or permissions) to the server resource that areallowed for the subject. for example, a subject might be allowed readaccess but not write access to a server resource. An access control entry (ACE)specifies the rules for accessing a given server resource. There aretwo kinds of ACEs Authentication ACEdefines how subject(who is making the request) is identified. It can beused to set authorizationmethod (Digest, Basic, SSL) or database names (like default, myldap,mykeyfile). Authorization ACE definesthe rights allowed or denied for a particular subject or a group ofsubjects. ACL (Access Control List) is anordered list of ACEs. An ACL can contain both types of ACEs. We can control accessto the entire server or toparts of the server,or the files or directories on our web site. When the server gets a request for a page, the server uses the rules inthe ACL file to determine if it should grant access or not. We create ahierarchyofrules (called ACEs) to allow or deny access.Each ACE specifies whether or not the server should check the next ACEin the hierarchy. The collection of ACEs we create is called an accesscontrol list (ACL). Types Of ACLs Named ACLs URI (Uniform Resource Indicator)ACLs specify a directory or file relative to the server’sdocument root. Path ACLs specify anabsolute path to the resource they affect. Path and URI ACLs can include wildcardsat the end of the entry. Forexample: /a/b/\*.Wildcards placed anywhere except at the end of the entry willNOT work. Examples of Named ACLs Named ACLs are ACLs which have thefirst line like acl"<name>"; . It MUSThave acorresponding check-aclSAF entry in obj.conf. For example, named acl default or named ACL dav-src  for WebDAVrequests we add in Object named dav. #default.aclversion 3.0; acl"default"; authenticate (user, group) {   prompt = "Sun JavaSystem Web Server"; }; allow (read, execute, info)user = "anyone"; allow (list, write, delete)user = "anyone"; ... acl"dav-src"; deny (all) user = "anyone"; Named ACL default has a corresponding check-aclSAF in obj.conf.  Named ACL dav-srchas a corresponding check-aclSAF in obj.conf. #obj.conf<Object name=default>...PathCheck fn="uri-clean" PathCheck fn="check-acl"acl="default"... </Object>...<Object name="dav"> PathCheck fn="check-acl"acl="dav-src"Service fn="service-dav"method="(GET|HEAD|POST|PUT|DELETE|COPY|MOVE|PROPFIND|PROPPATCH|LOCK|UNLOCK|MKCOL)"</Object> Examples of URI based ACLs URI based ACLs start with uri= prefix.We can set ACLs for a particular resource or a directory and resourcesinside it. For example, we can set URI based ACL for a particular file file.html #default.acl .... acl"uri=/dir1/file.html"; authenticate (user,group) {   method = "basic";   database = "myldap"; }; deny (all) user = "anyone"; allow (all) user = "beta"; For setting a URI based ACL on a directory and all the files insidethatdirectory, we can set ACEs like #default.acl... acl "uri=/dir1/\*";allow (all) dns=".sun.com"; Or #default.acl... acl "uri=/dir1/";deny (read) user="anyone"; allow (read) group="premium" anddayofweek="Sat,Sun"; Example of Path based ACLs Path based ACLs start with  "path="prefix. #default.acl ... acl "path=/opt/Sun/Servers/docs/index.html"; deny (read) user="anyone"; allow (read)timeofday<0800 or timeofday=1700; The URI or path that preceeds the wildcard does NOT work properly ifthe URI or path information is not a directory but is, instead, a file.For example, the following ACL settings work. When /test/README is accessed,access is allowed only for the user abc. #default.acl...acl "uri=/test/\*";authenticate (user) {  prompt = "Test ACL.";};deny (all) user = "anyone";allow (all) user = "abc"; But, the following ACL settings do NOT work. When /test/README is accessed therequest is allowed to everyone. #default.acl...acl "uri=/test/READ\*";authenticate (user) {  prompt = "Test ACL.";};deny (all) user = "anyone";allow (all) user = "abc"; Only tailing "/\*" is supported in ACLfile. ACLs with patterns XX\*, \*XX,  X\* are ignored. For such scenarios, declare it as a named ACL in obj.conf. For example, #obj.conf...PathCheck fn="check-acl" fn=check-acl path="/pathtosomedir/tes\*.html"acl="acl123"... Or #obj.conf...<Object ppath="/pathtosomedir/tes\*.html" >PathCheck fn="check-acl" acl="acl123"</Object> \*\* Note that check-acl usage has a limitation as ACLs are cached in memory. As long as the same ACLs are applied for a resource it is ok. You can not use check-acl for the case where different ACLs are supposed to be added for different condition. For a particular resource we have in obj.conf ,<If $client-ip="1.1.1.1"> PathCheck fn="check-acl" acl="acl123" </If> it may or may not add this ACL depending on client's ip address. To make it work we have to disable acl-cache in server.xml (in Web Server 7.0 onwards).One tip, whenever you want to set an ACL to allow access of a resource to a small audience, first add a deny ACE that would restrict the whole audience from accessing the resource and then add specific ACEs to allow access to the resource to the smaller audience. Something like #default.acl...acl ...deny (all) user ="anyone";allow (all) user="alpha,beta,gamma"; Or if you want to add allow ACE first use "absolute" keyword. #default.acl...acl ...allow absolute (all) user="alpha,beta,gamma";deny (all) user ="anyone"; Refer my next blog for more information about this.

Access Control In Sun Java System Web Server 7.0 You can now download Sun Java System Web Server 7.0 for FREE from http://www.sun.com/download/index.jsp?cat=Web%20%26%20Proxy%20Servers&tab=3&subcat=Web...