Saturday, June 30, 2018

Fork function failed on AIX versioned WPAR.

Hello everyone today i am going to share knowledge about the error "fork function failed".
I faced this issue on AIX 5.3  versioned WPAR on AIX 7.1 TL03 SP4.

fork function - "it create new process"

On IBM POWER8 we had created versioned WPAR of AIX here you must be thinking that why this guy didn't install directly AIX 5.3 on POWER8 BOX ??
so answer to this query is ,we cant directly install AIX 5.3 on POWER8,so first we need to install AIX 7.1 TL03 and SP4 and on that need to create AIX 5.3 versioned WPAR by using AIX 5.3 MKSYSB image,also because of database dependency ,we not able to install latest version of AIX on POWER8.

Let me answer why we created Versioned WPAR 5.3 on AIX 7.1
1st reason is that we cant directly install 5.3 on POWER 8
2nd ,we need IBM support for versioned WPAR in case of any critical issue and IBM support AIX 5.3 Versioned WPAR on AIX 7.1.

Overall setup was like below.

AIX 7.1 TL03 SP4.
on top of this we created AIX 5.3 versioned WPAR which hosting oracle 9i database for critical application.After restoration of database on AIX versioned WPAR oracle team started database and
after 30-40 minutes ,oracle team was complaining ,when they trying to login oracle account they are getting error :

"The fork function failed. Too many processes already exist"

after doing analysis of this error we increased memory of AIX LPAR and again asked DBA team to start database again and check whether error is sorted out or not.but no success getting same error again.
so what next ??

from OS end our thinking was like if we restore OS MKSYSB then no need to change any OS tuning parameter,but when we started checking parameter for "MAXUPROC" ,we found culprit which was causing this error."MAXUPROC" wasn't correctly set on AIX global LPAR and versioned WPAR  and which causing fork function failed error.on source AIX  server it was 2048 and on target it was 128 on both BASE OS and versioned WPAR.

MAXUPROC = maximum number of oracle/application process a user can able run on AIX OS.
so because of wrong setting at target after migration database throwing error of reaching maximum limit 128 process on versioned WPAR.

our solution to this was like ,first change maxuproc on AIX 7.1 GLOBAL LPAR and then do changes on AIX 5.3 versioned WPAR. Here note one thing that if AIX admin didn't set maxuproc for WPAR then non-root user inherits maximum process limit from global AIX 7.1 LPAR.

Command to change "MAXUPROC"

# chdev -l sys0 -a maxuproc=2048 
and then reboot AIX 7.1 global LPAR
and after reboot change "totalProcessess" on wpar by command,
#chwpar -R totalProcessess=2048 wparname.
then stopwpar and startwpar once .

after setting correct MAXUPROC limit we sorted out fork function failed error .

at last we can say that maxuproc limit changes according  to database and application requirement,so 
its always wise decision to refer database and application tuning document for performance related issue.

Thanks !!!!

No comments:

Post a Comment