makenemo fails in nestor

I can compile (./makenemo) in Japer, while I cannot in Nestor. With Nestor, I get the following error:

./makenemo -m ifort_hpcnestor_pisces_x -n physics_only_naa

mpif90 -o nemo.exe /home/hhayashi/nemo_eric/CONFIG/physics_only_naa/BLD/obj/nemo.o -L/home/hhayashi/nemo_eric/CONFIG/physics_only_naa/BLD/lib -l__fcm__nemo -shared-intel -mcmodel=medium -L/global/software/netcdf-4.1.3/lib -lnetcdf -lnetcdff
ld: cannot find -ltorque
fcm_internal load failed (256)
gmake: *** [nemo.exe] Error 1
gmake -f /home/hhayashi/nemo_eric/CONFIG/physics_only_naa/BLD/Makefile -j 1 all failed (2) at /g01/home/hhayashi/nemo_eric/EXTERNAL/fcm/bin/../lib/Fcm/Build.pm line 597

torque library does exist:

ldconfig -p | grep libtorque

    libtorque.so.2 (libc6,x86-64) => /opt/torque-2.5.13/lib/libtorque.so.2
    libtorque.so (libc6,x86-64) => /opt/torque-2.5.13/lib/libtorque.so

After some googling, I found the solution (ref: http://stackoverflow.com/a/16710346). I had to explicitly locate the library path for torque in the ARCH file (for %NCDF_LIB):

-L/opt/torque-2.5.13/lib -ltorque

blow up issue solved!

tracer.stat indicates undesirable increase in concentrations of some state variable(s) at the 7th time step:

         1  0.7887250195E+03
         2  0.7887190683E+03
         3  0.8475861522E+03
         4  0.9302648776E+03
         5  0.1044380477E+04
         6  0.1170169324E+04
         7  0.2228768128E+11
         8  0.2853781480E+11
         9  0.5271543990E+11
        10  0.7216373356E+11

After commenting and uncommenting source/sink terms in the equations, I discovered that this was due to fmort1 and fmort2 not property flushed into the water column:

        tra(ji,jj,1,jpgoc) = tra(ji,jj,1,jpgoc) &
                           ! source1: flushing
                           + (1.-f_p2)*flushdia*(1.-frld(ji,jj))*z_ia/e3t(ji,jj,1) &
                           ! source2: linear mortality
                           +0.
!                          + 0.7*fmort1 &
                           ! source3: quadratic mortality
!                          + fmort2


After commenting fmort1 and fmort2 terms above, tracer.stat looks more reasonable:

         1  0.7887289636E+03
         2  0.7887230246E+03
         3  0.7890159536E+03
         4  0.7902228379E+03
         5  0.7905440777E+03
         6  0.7908932570E+03
         7  0.7910913500E+03
         8  0.7913530813E+03
         9  0.7915290453E+03
        10  0.7917645388E+03

generating meteo file for amundsen

for month in `seq 1 12`;do wget --content-disposition "http://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=csv&stationID=1633&Year=2008&Month=${month}&Day=14&timeframe=1&submit= Download+Data" ;done
Other stations in the Amundsen Gulf area:

When all three my_trc components are added, the tracer.stat increases too rapidly:

tail tracer.stat
         1  0.7895863787E+03
         2  0.3124101036E+04
         3  0.1067765821E+13
         4  0.2180091013E+27
         5  0.8173235638E+41

This still occurs even I remove the icedms component:

         1  0.7895863787E+03
         2  0.3056491046E+04
         3  0.1034367164E+13
         4  0.2180090884E+27
         5  0.8173235638E+41

When I removed the chunk of code about ice ablation/accretion, the tracer.stat became normal:

         1  0.7895862638E+03
         2  0.7895870356E+03
         3  0.7913792876E+03
         4  0.7936409329E+03
         5  0.7949360900E+03

Thus, the problem is here:

         ! Lateral accretion/ablation
        IF(frld(ji,jj)-pfrld(ji,jj) > 0.) THEN ! In case of accretion, newly formed ice will have a tracer concentration of an underlying seawater value.
          icechl(ji,jj) = (1.-pfrld(ji,jj))/(1.-frld(ji,jj)+epsilon10)*icechl(ji,jj) + ((1.-frld(ji,jj))-(1.-pfrld(ji,jj)))/(1.-frld(ji,jj)+epsilon10)*trn(ji,jj,1,jpdch)
          iceno3(ji,jj) = (1.-pfrld(ji,jj))/(1.-frld(ji,jj)+epsilon10)*iceno3(ji,jj) + ((1.-frld(ji,jj))-(1.-pfrld(ji,jj)))/(1.-frld(ji,jj)+epsilon10)*trn(ji,jj,1,jpno3)
          icenh4(ji,jj) = (1.-pfrld(ji,jj))/(1.-frld(ji,jj)+epsilon10)*icenh4(ji,jj) + ((1.-frld(ji,jj))-(1.-pfrld(ji,jj)))/(1.-frld(ji,jj)+epsilon10)*trn(ji,jj,1,jpnh4)
        END IF

interesting segfault

I had a segmentation fault which is due to NaN. However, this was interesting because although the IF statement was always false, it gave me segfault because NaN would result if the IF statement was true.. 

        write(numout,*) 'hhpass: nancheck ',(1.-pfrld(ji,jj))/(1.-frld(ji,jj)+epsilon30)
        IF(frld(ji,jj)-pfrld(ji,jj) > 0.) THEN ! In case of accretion, newly formed ice will have a tracer concentration of an underlying seawater value.
          write(numout,*) 'hhpass: inside if statement'
          icechl(ji,jj) = icechl(ji,jj)/(trn(ji,jj,1,jpdch)+epsilon30)!((1.-pfrld(ji,jj))/(1.-frld(ji,jj)+epsilon30)*icechl(ji,jj) + ((1.-frld(ji,jj))-(1.-pfrld(ji,jj)))/(1.-frld(ji,jj)+epsilon30)*trn(ji,jj,1,jpdch)
        END IF


 Update: This seems to have to do with floating points.

Writing out the output for the variables in dinomenator and numerator show a particular floating point (i.e. 14):

 hhpass: pfrld    1.00000000000000
 hhpass: frld    1.00000000000000





In the example above, I tried avoid the division by zero by introducing epsilon number 30 (i.e. 1.e-30), but the division produced NaN:

hhpass: nancheck    NaN

After this, I instead introduced a epsilon number that is smaller than the floating point of the two variables (i.e. 4). Then the division produced zero:

 hhpass: nancheck   0.000000000000000E+000


Epsilon30 worked for biogeochemical tracers. But not for ice physical tracers (probably a specic floating point number is defined in LIM_SRC_2 for these tracers?)