Friday Jan 17, 2014

Inject filesystem I/O latency. Oracle Linux 5 and 6

I needed file I/O to take longer time for test purpose.
I needed latency on all files in a directory including newly created files.
I thought one of FUSE filesystem can be used here.

Filesystem in Userspace - Wikipedia, the free encyclopedia

Filesystem in Userspace (FUSE) is an operating system mechanism for Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code.

In my env, FUSE was already installed to use DBFS.

Using DBFS

dbfs_client host must have the FUSE Linux package

I haven't configured DBFS and I thought one of filesystem that stores data over the network
can become slow if I can simulate slow network.

So I tried SSHFS.
I could install "fuse-sshfs" rpm package both on Oracle Linux 5&6 via 'yum' command.
This package is on 'epel' channel. For OL6, I think there is 'rpm' just to set up this channel.
I don't remember where I found.
For OL5, I couldn't find the rpm so I googled and copy&pasted from this page.

The point of no return » Adding EPEL repository on CentOS

epel.repo file into /etc/yum.repos.d

Below is the test result.
As you can see, File I/O is now 100 times slower.
But since there could be other operations which uses loopback network, I think I will keep searching for a
safer way.

$ su -c "sshfs -o allow_other root@localhost:/root/sshfs_dir /mnt/tmp"
Password:
$ ping -c 1 localhost
PING localhost.localdomain (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=1 ttl=64 time=0.013 ms
$ time dd if=/dev/zero of=/mnt/tmp/128blocks.f bs=8192 count=128
128+0 records in
128+0 records out
1048576 bytes (1.0 MB) copied, 0.0590345 seconds, 17.8 MB/s

real    0m0.181s
user    0m0.000s
sys     0m0.003s
$ su - -c "tc qdisc add dev lo root netem delay 500ms"
Password:
$ ping -c 1 localhost
PING localhost.localdomain (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=1 ttl=64 time=1000 ms
$ time dd if=/dev/zero of=/mnt/tmp/128blocks.f bs=8192 count=128
128+0 records in
128+0 records out
1048576 bytes (1.0 MB) copied, 13.0411 seconds, 80.4 kB/s

real    0m19.084s
user    0m0.000s
sys     0m0.003s

Thursday Dec 19, 2013

perl for processing sqlplus spool output

When I wrote my bash script below 3 monthes ago,

fixed length sqlplus spool file for loading data (INOUE Katsumi @ Tokyo)

you need to salvage data from sqlplus spool file


I thought I googled enough not to reinvent the wheel.
But that wasn't enough! One of my colleague's mail lead me to another colleague's blog entry.

Convert sqlplus spool output to CSV « Tyler Muth’s Blog

perl seemed like the best fit as it’s easy, cross platform, and made for text files.

My bash script runs on Windows Cygwin so it's somewhat cross platform. But I recommend using Tyler's perl.
It's more readable and it seems it can handle many corner cases.

Now I remember that there was an internal discussion as to what's the best language to write 'one-off' script.
Kind of consensus at that time was 'perl' as far as Oracle Database is concerned because perl gets installed
under $ORACLE_HOME.

I think python is OK now in Exadata env.

About

The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today