Tuesday, July 24, 2018

XL C/C++ compiler installation on AIX 5.3,6.1 and 7.1

XL C compiler
Hello everyone today i am going to discuss how to compile and run C and C ++ program on AIX .Generally  XL C/C++ compiler is used for compiling C and C++ program on AIX platform.

While doing administration of AIX operating system one day i got mail from application development team that they need to compile C program on AIX, is there any way that we can compile on AIX.

After this i started googling and found that there is XL C compiler for compiling and running C program on AIX operating system. Also found that there is some evaluation version also available on IBM portal which I can install for limited time(60 Day) and after evaluation time expire ,organization need to purchase new license version for XL C. I informed this update to my manager and he suggested that for now install evaluation version and also check with IBM agent for license version.

So from following link i downloaded XL C/C++ compiler evaluation version and copied on required AIX LPAR. If your organization afford to pay for license and need XL C/C++ for long term, they can go for license version. In our case we first installed evaluation version, because of urgent requirement.
Evaluation period for XL C C++ was 60 days.

You can download latest evaluation version from following IBM Link.

And for downloading licensed version u need to contact IBM and same licensed version admin can download from IBM "Entitled Software Support". For downloading licensed product from IBM 
Entitled Software Support your organization must have login ID provided by IBM.

Let’s see how to install evaluation version on AIX .

I had copied XL C C++ compiler evaluation version on /tmp/compiler directory.

**Before staring installation i want to highlight below mentioned important thing .
keep ready AIX OS base CD/DVD fileset,these fileset will help you when some pre-requisite failure occur during installation of XL C compiler. In our case we wasted lot of time for finding 
pre-requisite fileset  "xlsmp.rte".

Step 1.

Execute following command from AIX LPAR
smitty easy_install

Then chose .  current directory where i stored files required for installation.

Step 2:

Choose appropriate package according to AIX version, i was configuring XL C/C++ on AIX 5.3 so i selected "vacpp-evaluation-aix53".
 For 6.1 and 7.1 admin must choose "vacpp-evaluation-aix61-aix71".

Here during installation of this evaluation version one query came in mind was, how I going to upgrade or install licensed version after evaluation version will expired ??

Answer to this query is like , AIX admin first need to buy licensed version and copy that files on AIX LPAR and choose following option while installing using "smitty easy_install".

1.licensed version on AIX V5.3, choose
For installing licensed version on AIX V6.1 or V7.1, choose
Step 3 :
From same smitty easy_install menu in step 1 select like following
AUTOMATICALLY install requisite software? Option to
yes to install the compiler automatically

change the EXTEND file systems if space needed? option to yes to
extend file systems

You must change the ACCEPT new license agreements? option to yes to
accept new license agreements

Press enter and after successful installation check whether it successfully installed or not.

also my suggestion is take preview of install before installing these fileset, so that u will get idea that any requisite fileset is missing or not.  

Press ESC +0 to exit from Smitty menu.
After this installation i thought that it get installed and i informed application team that i installed required fileset on server. But when they tried to compile there C program using xlc command they getting error message that "command not found". So after checking read me document of IBM for XL C/C++ compiler found that i missed symlink creation for "xlc" command and i need to set environment variable for specific user on AIX. So i created following 2 symlink for xlc and xlc++.

1. xlc symlink to compile C program.

ln -s /usr/vacpp/bin/xlc /usr/bin/xlc  

2.xlc++ symlink to compile C++ programm.
ln -s /usr/vacpp/bin/xlc++ /usr/bin/xlc++
Also we can add following entry in /etc/environment file like below 
Add /usr/vac/bin/ and /usr/vacpp/bin/ to your PATH environment variable
To confirm whether soft link created or not you can confirm by following command
#which xlc
#which xlc++

From above output we can confirm that symlink successfully created and environment properly set for running C and C++ code on AIX.
So let’s Test some C and C++ program.

test.c  //sample c program
Create test.c on AIX LPAR and put following C code in test.c file.
int main(void)
printf("Test program!\n");
return 0;

Compile C program using xlc
#xlc test.c -o test

Run program by executing below command.
Test program!

test1.cpp   //sample C++ programm 
create test1.cpp and copy following code in test1.cpp file.

int main()
std::cout << "Test cpp programm!\n" << std::endl;
return 0;

Compile C++ program using xlc++
#xlc++ test1.cpp -o test1

Run program
Test cpp programm!

If  IBM XL C/C++ for AIX software evaluation license expired?

there is work around suggested by IBM like below


AIX admin can add following entry in .profile file of user for workaround.
Edit /etc/profile and add following entry.

Also we can use export of environment variable "XLC_EXTEND_EVAL=yes"  

# export  XLC_EXTEND_EVAL=yes

This will extend XL C/C++ to Total 74 days. Previous evaluation period was 60 days ,this workaround will 14 more days.
So you must be thinking that what will happened after evaluation version getting expired. Answer to this was, we purchased licensed for XL C compiler and we did install/upgrade of XL C compiler after evaluation time ended.
If during installation you face any difficulty you can refer Guide to install XL C .



Friday, July 20, 2018

Java installation and configuration on AIX

Hello everyone today i am going to share how to install and configure JAVA5 on AIX. Generally JAVA required Pre-requisite for some installation .Example . TSM installation and configuration,oracle installation and configuration.

So lets see how configure JAVA on AIX.

First thing need to check was to confirm which version of JAVA was installed on AIX. execute following command to  check java version

#lslpp -l |grep -i java 

if above command not showing any output then there is no java installed on AIX .
AIX admin can install and keep multiple version of JAVA and do environment setting according to requirement of application. whether application or TSM team require 32 bit or 64 bit JAVA version.

#Java -version

Above command will show java -version currently configured on AIX .

So here i am going share method to install java 5 ,32 bit version on AIX and same method can be used to install other new version also.

In our case TSM admin need JAVA 5 ,32 bit version. So first copied required installation files on AIX LPAR.

1. First install Java5.sdk using SMIT install.
2.If in case AIX admin want to update current JAVA5 then simply copy required update fileset on current path and do smitty_update all.
3.After this, edit /etc/environment file and add java5 path .There is chances that already some PATH are present in /etc/environment file ,according to syntax of file AIX admin can add JAVA5 setting. if AIX admin want to add this setting for specific user he can do changes in .profile file of specific user and to reload .profile after changes execute command  #. .profile


AIX admin can do PATH setting according to requirement by other team,by changing PATH setting in /etc/environment variable.

4. After installation and PATH setting execute following command to check currently configured JAVA version.

#Java -version

To test whether java is properly configured or not on AIX  node ,we are going to execute following JAVA test program.

root@AIX_LPAR# cat testjavaprog.java
        public static void main(String args[])
           System.out.println("Java test program");

3. compile program  with javac:
root@AIX_LPAR#javac testjavaprog.java

4. run the program using following command
root@AIX_LPAR# java testjavaprog

Java test program

IF everything is correctly configured then after running program
output will be "Java test program".

Thanks  !!!

Friday, July 13, 2018

NIMSH service PORT 3901 preventing SAP application from starting

                                  NIMSH service Reserved PORT 3901 and SAP

Today i am going to discuss about one of the incident which stopping SAP application from starting on AIX two node cluster.
So lets see what happened  and how we resolved this issue.

At first let me introduce about setup. It was 2 node AIX 7.1 HACMP setup,which was hosting SAP application. We got 3 hrs of downtime to do reboot of these 2 node cluster,so when down time started application team stopped SAP application and informed us to proceed for AIX HACMP node reboot.Then we took configuration backup as per standard checklist before doing reboot of these server. Generally we having script which executed from remote server to take backup of OS configuration and also bring resource group offline and then we rebooted AIX HACMP node. Node is up and active in 10 minutes then we started cluster services and bring up resource group on rebooted node and done POST check after OS reboot, all was looking fine from OS side like all File system are in mounted state ,also checked ifconfig -a output, it was also correct.

We informed SAP application team to start their application but after 5-10 minutes we got reply that they not able to start their application,while starting application it throwing some error, application team try 2-3 time but result was same. Then somebody from application team suggested that let clear some SAP  processes which are occupying shared memory segment. Multiple processes from shared memory segment has been identified and cleared using command ipcs and ipcrm, also some processes from OS side killed and SAP application team retried for application restart again,but no success  :(.

what Next .................??

So from management it was decided that go for second node and try to start SAP application on 2nd node . As per their request we done resource group switch to second node and asked SAP team to start their application, but this time also no success. From application and OS end  cleared shared memory for SAP processes and killed some SAP application process from OS end but no success yet............

It was almost 2 hr 30 minutes and down time about to finish. In next 30 minutes application must be online and active to start business again . So i asked one of the guy from SAP application team ,please share the error. He shared some output but that not making any sense that, why exactly application was not starting and who was culprit. But suddenly in our conversation chat one of expert from SAP checked some log file and asked us to "check whether PORT 3901 is free or not ".
finding in log file is like below ....

Error details: "PORT 3901 in use and because this  socket BIND failed"

So we checked port availability using following command and found that 0n 3901 PORT  nimsh service  is listening.

#netstat -Aan |grep -i 3901
tcp        0      0  *.3901                 *.*                    LISTEN 
also we then find process details which are listening on this port by using lsof command.
#lsof  -i tcp :3901
command  PID    user
nimsh   6115400 root         0t0  TCP *:nimsh (LISTEN) 

We found that "nimsh" process started by root user, listening on PORT 3901,until we stop this process SAP application will not going to start.

we stopped this service by #stopsrc -s nimsh and asked SAP application team to start their application by clearing some shared memory segment for SAP processes and guess what ,we succeeded in starting SAP application  😊😊😊.
Finally we able to start SAP application before down time end.

Now SAP Team was asking OS team for RCA why "nimsh" was acquiring PORT "3901" before SAP application process ??

our answer to them was ,from analysis we found that PORT 3901 is reserved for nimsh service on AIX operating system and it is not recommended that it is used by any application or database process . We found following stanza in /etc/services of AIX node and which was indicating that this PORT is reserved for nimsh service.

#root : cat /etc/services | grep -i 3901
nimsh                    3901/tcp               # NIM Service Handler
nimsh                    3901/udp               # NIM Service Handler

Also found observation that in past while doing SAP application installation/configuration by SAP admin who didn't know that 3901 is reserved for "nimsh" service and he used that PORT to configure  SAP application. So our suggestion to SAP team was use other PORT than OS reserved PORT(like 3901) for SAP application process because of following reason :

1. In future if OS team need to start nimsh service then it definitely require PORT 3901 .

2.Whenever AIX admin reboot any node of this cluster and if SAP admin want to start application  on that node after reboot,then SAP application will not start again,because PORT 3901 will be acquired by "nimsh" process by default . So for starting their application AIX admin team again need to STOP "nimsh" every time.

Found following entry in /etc/inittab file which start nimsh service during system boot time.
nimsh:2:wait:/usr/bin/startsrc -g nimclient >/dev/console 2>&1

 lssrc -g nimclient
Subsystem         Group            PID          Status

 nimsh            nimclient        3670130      active

At last i strongly suggest that it is always recommended that application or database team should not use OS reserved PORT for their application, because AIX admin who love NIM server for doing operation on NIM client will always require "nimsh" active on AIX client node.

In our case we solved this issue by finding some workaround,but this workaround is not recommended by IBM also.

Thanks !!!

Monday, July 9, 2018

Manual Resource group movement while clstrmgrES state is ST_INIT

                                    clstrmgrES  state  ST_INIT

Today i am going to discuss clstrmgrES (cluster manager) state ST_INIT and how it affect while moving resource group from secondary node to primary node.
Let see different state of cluster manager and understand their purpose.

ST_INIT: cluster configured and down

ST_STABLE: cluster services are running with managed resources and cluster is in stable state.

while handling AIX hacmp cluster incident where one node from cluster rebooted which was primary node then resource group on primary node fail-over to secondary node.but after reboot of primary server we found that resource group was not fallback to primary node,so we started analysis on it and decided to to manual resource group switch.

while doing manual switch we followed following steps using smitty hacmp
Here for understanding purpose i am taking following node name and resource group name.

hacmp node name : aixhanode1 and aixhanode2 and 
resource group name : aixRG1

so in our scenario before reboot of aixhanode1 resource group  aixRG1 was on aixhanode1,but after reboot it failover to secondary node aixhanode2.so now client want to move aixRG1 resource group back on primary node aixhanode1.
so for manual resource group switch we followed below approach.

 Group Name  State    Node
 aixRG1      OFFLINE    aixhanode1
                    ONLINE     aixhanode2

1.smitty hacmp
 2. System Management (C-SPOC)
    3.HACMP Resource Group and Application Management
      4.Move a Resource Group to Another Node / Site
       5. Move Resource Groups to Another Node
          6.Choose RG     Here choose resource group name aixRG1
           7. Choose target node  but for step 7 we not able to see target node name in smitty panel  ???

so whats next ?? if you want to do cluster resource group switch then target node must be shown in smitty panel.after doing error analysis found that clstrmgrES  state must be ST_STABLE on both node,so after executing below command on both node we found that on primary node aixhanode1 it was in ST_INIT .

#aixhanode1#lssrc -ls clstrmgrES |grep state
Current state: ST_INIT    here is the issue this state must in ST_STABLE to move resource group from aixhanode2 to aixhanode1.

#aixhanode2#lssrc -ls clstrmgrES |grep state
Current state: ST_STABLE

so after identifying cause we simply started clstrmgrES on node aixhanode1 by using following method .

1.smitty hacmp
 2. System Management (C-SPOC)
  3.PowerHA SystemMirror Services
   4.Start Cluster Services
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
                                                     [Entry Fields]
* Start now, on system restart or both                now                                                                                                            +
  Start Cluster Services on these nodes              [aixhanode1]                                                                                                      +
* Manage Resource Groups                              Automatically                                                                                                  +
  BROADCAST message at startup?                       false                                                                                                          +
  Startup Cluster Information Daemon?                 true                                                                                                           +
  Ignore verification errors?                         false                                                                                                          +
  Automatically correct errors found during           Interactively                                                                                                  +
  cluster start?

Press enter wait for some time to command complete.

so after command completion we checked clstrmgrES and now its in stable state on aixhanode1,so now we can move resource group from aixhanode2 to aixhanode1.

#aixhanode1#lssrc -ls clstrmgrES |grep state
Current state: ST_STABLE

so we again follow below approach for moving aixRG1 to primary node .

1.smitty hacmp
 2. System Management (C-SPOC)
    3.HACMP Resource Group and Application Management
      4.Move a Resource Group to Another Node / Site
       5. Move Resource Groups to Another Node
          6.Choose RG     Here choose resource group name aixRG1
           7. Choose target node   and now we able to see traget node name "aixhanode1"

now press enter wait for some time.

after this check whether resource is online or not on primary node by clRGinfo command.

 Group Name  State    Node
 aixRG1       ONLINE   aixhanode1
              OFFLINE  aixhanode2

at last we can conclude that while moving resource group ,clstrmgrES state must be in ST_STABLE on both node to move resource group from one node to other.if its not in stable bring it in stable state and then only do resource group switch.

Thanks !!!

Saturday, July 7, 2018

XCLOCK configuration on SUSE Linux


Hello everyone ,
In my last post i shared how to do X browser configuration on AIX. Now i am going to share  one incident where roor user was unable to run XCLOCK on SUSE 12 and then how this incident was  handled by our team ??
Lets see
xclock Error   "can't open dispaly"

After doing analysis on this ticket we found that in root user home directory ".Xauthority" file wasn't present and also found that "X11Forwarding” in /etc/ssh/sshd_config not set correctly it must be "X11Forwarding yes".after doing chnages to sshd_config file refresh sshd service once.

Also we crosscheck that required packages are installed on SUSE 12 server.
required package to run Xclock:

xorg-x11-essentials xorg-x11-fonts xorg-x11-server xorg-x11-libs xorg-x11-util-devel xorg-x11-Xvnc xorg-x11-devel gcc  gcc-c++ libaio-devel compat-libldap-2_3-0 xclock libstdc++-devel libstdc++-devel-32bit libstdc++33

you can check whether package is installed or not by following command 

target_server#zypper se xorg-x11-essentials xorg-x11-fonts xorg-x11-server xorg-x11-libs xorg-x11-util-devel xorg-x11-Xvnc xorg-x11-devel gcc  gcc-c++ libaio-devel compat-libldap-2_3-0 xclock libstdc++-devel libstdc++-devel-32bit libstdc++33

if packages are not installed then install using following command 

target_server#zypper in -y xorg-x11-essentials xorg-x11-fonts xorg-x11-server xorg-x11-libs xorg-x11-util-devel xorg-x11-Xvnc xorg-x11-devel gcc  gcc-c++ libaio-devel compat-libldap-2_3-0 xclock libstdc++-devel libstdc++-devel-32bit libstdc++33

in our case packages are already present on SUSE 12 server so one more thing we observed was when we execute "xauth" command getting error message that "xauth program not found".so we checked the SUSE linux server and found that there is some symbolic link for "xauth" command so we created that using following command.

target_server#cd /usr/X11R6/
target_server# mkdir bin
target_server# ln -s /usr/bin/xauth /usr/X11R6/bin 

xauth is used for editing and displaying authorization file information.

Then using nxclient(used for running graphical Linux)  do ssh login to root user  from Jump server to target SUSE Linux server and check that " .Xauthority" file generated or not .it should be there .
LinuxJumpServer#ssh root@target_server

.Xauthority" file stores credentials in cookies and that is used by xauth program for authentication at time of X session." 
if file is there execute following command to check "MIT-MAGIC-COOKIE" cookie are there in ".Xauthority" file.do not create .Xauthority file manually because when user login from remote server using ssh ,after successful login ".Xauthority" will be genrated.

target_server#xauth list
will show "MIT-MAGIC-COOKIE".

Then execute "xclock" from root user account if it showing following clock then you have successfully configured XCLOCK on SUSE 12.


using above mentioned method , we sorted out XCLOCK incident on SUSE 12.

Thanks !!!