unexpected free inode

unexpected free inode 208925, run fsck(1M)
this is happening on a Solaris 8 system.
It means that something deleted a file on the disk without going
through the filesystem driver to do it.
Some sort of raw access to the /dev/rdsk device file or some bug.
The filesystem must be umounted and fsck must be run on it. If
there is further filesystem corruption at some point it can
trigger a panic.
In the meantime, do all of the usual checks to be sure the device
is not being accessed raw. Not a swap device, not a database
devices, and so on. And audit the rest of the messages file to
see if there are any hints about hardware problems.

solaris 10 Zones with VCS

Configuration Steps

To Build a standard VCS Cluster to Support Zones, the following must be done:

a. Build Servers (Including VxVM/VCS)

b. Configure / Enable FSS Scheduler

c. Configure / Enable Default Processor Pools

d. Configure VCS Servicegroups

e. Create Zone

Build Servers (Including VCS) When building a VCS Cluster to Support Zones, the latest Solaris 10x86 Build should be used. VCS Supports Zones starting in 4.1 MP1 Release.

Enable FSS Scheduler Once Servers have been build, need to enable the FSS (Fair Share Scheduler), by executing the following commands: # dispadmin -d FSS Move All existing processes into FSS scheduler: # priocntl -s -c FSS -i all

Enable Processor pools, and default pool Create default system pool, this will be the default pool where all CPU.s are assigned unless an application has specifically requested a set number of CPUs and does not want to share with other applications (when cpu available). Execute the following commands to enable pools: # pooladm -e # poolcfg -c discover # pooladm -c Set default pools scheduler to FSS class: # poolcfg -c 'modify pool pool_default (string pool.scheduler="FSS")' # pooladm -c

Setup Veritas Volumes Prior to setting up Zones, should first setup the proper volumes/filesystems: Initialize Devices # /etc/vx/bin/vxdisksetup -I

Initialize DiskGroups # vxdg init DG#_

Initialize Volumes # vxassist -g make

Create Filesystems # mkfs -Fvxfs -olargefiles /dev/vx/rdsk//

Recommended filesystem for zone is 16g: # vxassist -g make zone_os 16g

Create zone Create Zone using create_zone script from the jumpstart server: /jumpstart_10x86v0/stdbuild/scripts/create_zone:

# mount :/jumpstart_10x86v0/stdbuild /mnt # cd /mnt/scripts # ./create_zone

Need to specify a Zone Name?

./create_zone -z [ -l localisation ] [ -c ] [ -m ] [ -s ] -p [ -i ] [-e ] -n example: ./create_zone -z pe-as1-d -l dselab -n ./create_zone -z pe-as1-d -l dselab -c 2 -n ./create_zone -z pe-as1-d -l dselab -c 2 -m 2G -n ./create_zone -z pe-as1-d -l dselab -m 2G -s 200 -p /zone_os -n -v [for debug verbose] Unless -n is used, script will only show commands used. The script will create the project for the Zone on the server where it was run. /etc/project: dseds1d:100:dse-ds1-d:::project.cpu-hares=(priv,100,none);project.pool=pool_default dseas1d:101:dse-as1-d:::project.cpu-hares=(priv,100,none);project.pool=pool_default Example: # ./create_zone -z dse-ds1-d -l dselab -s 200 -p /dse-ds1/zone_os -I 30.6.25.20 -e ce0 -n After create script completes, the rest of standard build will be applied during first zone boot. Once zone has been completely built, can include additional filesystems, (either by lofs or direct mount under /zone_os//root/ directory). Example: # zonecfg -z dse-ds1-d zonecfg:dse-ds1-d> add fs zonecfg:dse-ds1-d:fs> set dir=/dse-ds1/ora01 zonecfg:dse-ds1-d:fs> set special=/dse-ds1/ora01 zonecfg:dse-ds1-d:fs> set type=lofs zonecfg:dse-ds1-d:fs> end zonecfg:dse-ds1-d> add fs zonecfg:dse-ds1-d:fs> set dir=/dse-ds1/ora02 zonecfg:dse-ds1-d:fs> set special=/dse-ds1/ora02 zonecfg:dse-ds1-d:fs> set type=lofs zonecfg:dse-ds1-d:fs> end zonecfg:dse-ds1-d> add fs zonecfg:dse-ds1-d:fs> set dir=/dse-ds1/ora03 zonecfg:dse-ds1-d:fs> set special=/dse-ds1/ora03 zonecfg:dse-ds1-d:fs> set type=lofs zonecfg:dse-ds1-d:fs> end zonecfg:dse-ds1-d> add fs zonecfg:dse-ds1-d:fs> set dir=/dse-ds1/oraarch zonecfg:dse-ds1-d:fs> set special=/dse-ds1/oraarch zonecfg:dse-ds1-d:fs> set type=lofs zonecfg:dse-ds1-d:fs> end zonecfg:dse-ds1-d> add fs zonecfg:dse-ds1-d:fs> set dir=/dse-ds1/oratemp zonecfg:dse-ds1-d:fs> set special=/dse-ds1/oratemp zonecfg:dse-ds1-d:fs> set type=lofs zonecfg:dse-ds1-d:fs> end zonecfg:dse-ds1-d> commit Modify zone.cpu-shares (this is updated automatically if .s included with create_zone script) zonecfg:dse-ds1-d> add rctl zonecfg:dse-ds1-d:rctl> set name=zone.cpu-shares zonecfg:dse-ds1-d:rctl> add value (priv=privileged,limit=100,action=none) zonecfg:dse-ds1-d:rctl> end zonecfg:dse-ds1-d> commit After rebooting the ZONE the partitions should be mounted: # df -k Filesystem kbytes used avail capacity Mounted on / 16777216 3938350 12036933 25% / /dev 16777216 3938350 12036933 25% /dev /orape-ds1/ora01 10485760 19652 9811983 1% /orape-ds1/ora01 /orape-ds1/ora02 10485760 19651 9811985 1% /orape-ds1/ora02 /orape-ds1/ora03 10485760 19651 9811985 1% /orape-ds1/ora03 /orape-ds1/oraarch 10485760 19651 9811985 1% /orape-ds1/oraarch /orape-ds1/oratemp 10485760 19651 9811985 1% /orape-ds1/oratemp proc 0 0 0 0% /proc ctfs 0 0 0 0% /system/contract swap 5235304 268 5235036 1% /etc/svc/volatile mnttab 0 0 0 0% /etc/mnttab /usr/lib/libc/libc_hwcap2.so.1 16777216 3938350 12036933 25% /lib/libc.so.1 fd 0 0 0 0% /dev/fd swap 1048576 0 1048576 0% /tmp swap 20480 20 20460 1% /var/run If dedicating CPU.s can assign CPU pool to zone: bash-3.00# zonecfg -z dse-ds1-d zonecfg:dse-ds1-d> set pool=dse-ds1-d else, add it to global pool which should be using FSS (Faire Share Scheduler). zonecfg:dse-ds1-d> set pool=pool_default zonecfg:dse-ds1-d> commit All projects should be created on all cluster nodes and Zone, so that all shares are listed properly. Create Project / Assign users within Zone: Login to ZONE and setup projects,users,shares. projadd -c (project name can not contain - ) # projadd -c dse-ds1-d dseds1d Assign cpu shares to Project / Zone # projmod -sK "project.pool=pool_default" dseds1d # projmod -sK "project.cpu-shares=(priv,100,none)" dseds1d . Setup project in Zone, assign users to Zone. projmod -U oracle,orape01 # projmod -U oracle,orape01 dseds1d Setup VCS Configure VCS as per the usual standards/best practices. After creating Servicegroups, prior to creating ZONE resource, should first shutdown ZONE. VCS Example: Create Cluster: haclus -modify Administrators root haclus -modify ClusterName dse-cluster2 Create MultiNIC Group/Resource: hagrp -add acb-mnic hagrp -modify acb-mnic SystemList dse-clust1-da 0 dse-clust1-db 1 hagrp -modify acb-mnic AutoStart 0 hares -add MNIC_acb-mnic_bge0_bge1 MultiNICB acb-mnic hares -modify MNIC_acb-mnic_bge0_bge1 Device bge0 0 bge1 1 hares -modify MNIC_acb-mnic_bge0_bge1 NetMask "255.255.255.0" hares -modify MNIC_acb-mnic_bge0_bge1 IgnoreLinkStatus 0 Create ServiceGroup: hagrp -add dse-ds1-d hagrp -modify dse-ds1-d SystemList dse-clust1-da 0 dse-clust1-db 1 Create DiskGroup hares -add DG1_dse-ds1-d DiskGroup dse-ds1-d hares -modify DG1_dse-ds1-d DiskGroup DG1_dse-ds1-d Create Proxy Resource: hares -add PXY_dse-ds1-d_bge0_bge1 Proxy dse-ds1-d hares -modify PXY_dse-ds1-d_bge0_bge1 TargetResName MNIC_acb-mnic_bge0_bge1 Create Volume Resources hares -add VOL_DG1_dse-ds1-d_admin Volume dse-ds1-d hares -modify VOL_DG1_dse-ds1-d_admin Volume admin hares -modify VOL_DG1_dse-ds1-d_admin DiskGroup DG1_dse-ds1-d hares -add VOL_DG1_dse-ds1-d_ora01 Volume dse-ds1-d hares -modify VOL_DG1_dse-ds1-d_ora01 Volume ora01 hares -modify VOL_DG1_dse-ds1-d_ora01 DiskGroup DG1_dse-ds1-d hares -add VOL_DG1_dse-ds1-d_ora02 Volume dse-ds1-d hares -modify VOL_DG1_dse-ds1-d_ora02 Volume ora02 hares -modify VOL_DG1_dse-ds1-d_ora02 DiskGroup DG1_dse-ds1-d hares -add VOL_DG1_dse-ds1-d_ora03 Volume dse-ds1-d hares -modify VOL_DG1_dse-ds1-d_ora03 Enabled 1 hares -modify VOL_DG1_dse-ds1-d_ora03 Volume ora03 hares -modify VOL_DG1_dse-ds1-d_ora03 DiskGroup DG1_dse-ds1-d Create Mount Resources: hares -add MNT_dse-ds1-d_admin Mount dse-ds1-d hares -modify MNT_dse-ds1-d_admin MountPoint "/dse-ds1" hares -modify MNT_dse-ds1-d_admin BlockDevice "/dev/vx/dsk/DG1_dse-ds1-d/admin" hares -modify MNT_dse-ds1-d_admin FSType vxfs hares -modify MNT_dse-ds1-d_admin FsckOpt "%-y" hares -add MNT_dse-ds1-d_ora01 Mount dse-ds1-d hares -modify MNT_dse-ds1-d_ora01 MountPoint "/dse-ds1/ora01" hares -modify MNT_dse-ds1-d_ora01 BlockDevice "/dev/vx/dsk/DG1_dse-ds1-d/ora01" hares -modify MNT_dse-ds1-d_ora01 FSType vxfs hares -modify MNT_dse-ds1-d_ora01 FsckOpt "%-y" hares -add MNT_dse-ds1-d_ora02 Mount dse-ds1-d hares -modify MNT_dse-ds1-d_ora02 MountPoint "/dse-ds1/ora02" hares -modify MNT_dse-ds1-d_ora02 BlockDevice "/dev/vx/dsk/DG1_dse-ds1-d/ora02" hares -modify MNT_dse-ds1-d_ora02 FSType vxfs hares -modify MNT_dse-ds1-d_ora02 FsckOpt "%-y" hares -add MNT_dse-ds1-d_zone Mount dse-ds1-d hares -modify MNT_dse-ds1-d_zone MountPoint "/dse-ds1/zone_os" hares -modify MNT_dse-ds1-d_zone BlockDevice "/dev/vx/dsk/DG1_dse-ds1-d/zone" hares -modify MNT_dse-ds1-d_zone FsckOpt "%-y" Create ZONE Resource: hares -add Zone_dse-ds1-d Zone dse-ds1-d hares -modify Zone_dse-ds1-d ZoneName dse-ds1-d Create Links: hares -link DG1_dse-ds1-d PXY_dse-ds1-d hares -link MNT_dse-ds1-d_admin DG1_dse-ds1-d hares -link MNT_dse-ds1-d_admin VOL_dse-ds1-d_admin hares -link MNT_dse-ds1-d_ora01 MNT_dse-ds1-d_admin hares -link MNT_dse-ds1-d_ora01 VOL_DG1_dse-ds1-d_ora01 hares -link MNT_dse-ds1-d_ora02 MNT_dse-ds1-d_admin hares -link MNT_dse-ds1-d_ora02 VOL_DG1_dse-ds1-d_ora02 hares -link MNT_dse-ds1-d_ora03 MNT_dse-ds1-d_admin hares -link MNT_dse-ds1-d_ora03 VOL_DG1_dse-ds1-d_ora03 hares -link MNT_dse-ds1-d_zone MNT_dse-ds1-d_admin hares -link MNT_dse-ds1-d_zone VOL_dse-ds1-d_zone hares -link VOL_DG1_dse-ds1-d_ora01 DG1_dse-ds1-d hares -link VOL_DG1_dse-ds1-d_ora02 DG1_dse-ds1-d hares -link VOL_DG1_dse-ds1-d_ora03 DG1_dse-ds1-d hares -link VOL_DG1_dse-ds1-d_oraarch DG1_dse-ds1-d hares -link VOL_DG1_dse-ds1-d_oratemp DG1_dse-ds1-d hares -link VOL_dse-ds1-d_admin DG1_dse-ds1-d hares -link VOL_dse-ds1-d_zone DG1_dse-ds1-d hares -link Zone_dse-ds1-d MNT_dse-ds1-d_zone hares -link Zone_dse-ds1-d MNT_dse-ds1-d_ora01 hares -link Zone_dse-ds1-d MNT_dse-ds1-d_ora02 hares -link Zone_dse-ds1-d MNT_dse-ds1-d_ora03 Add a new cluster user, with group administrator privileges for the group containing the zone. # hauser -add dse-adm -priv Administrator -group dse-ds1-d Enter New Password: Enter Again: Ensure the local zone can resolve the host name of the global zone, either through DNS or through the /etc/hosts file. Make sure the global zone.s host name is in DNS or the local /etc/hosts file. Log into the local zone (zlogin ): # zlogin -C -e T dse-ds1-d [Connected to zone 'dse-ds1-d' console] dse-ds1-d console login: root Password: Sep 25 07:24:15 dse-ds1-d login[23334]: ROOT LOGIN /dev/console Last login: Mon Sep 25 07:21:24 on console Machine: dse-ds1-d, SunOS 5.10, built 21 Sep 2006 Set the environment variable VCS_HOST to the host name of the global zone. # export VCS_HOST=dse-clust1-da Issue the command /opt/VRTSvcs/bin/halogin . # /opt/VRTSvcs/bin/halogin dse-adm dse-adm (failover and do the samething for the other cluster nodes). Enable Application Agents in VCS to monitor resources within a Zone Application Agents that are monitoring resources within a Zone, need to have ContainerType and Containername attributes defined. To update existing Oracle/Sqlnet resources to support Zones: # haconf -makerw # haattr -add -static Oracle ContainerType -string -scalar Zone # hatype -modify Oracle ArgList -add ContainerName # haattr -add -static Sqlnet ContainerType -string -scalar Zone # hatype -modify Sqlnet ArgList -add ContainerName # haattr -add Oracle ContainerName # haattr -add Sqlnet ContainerName # hatype -modify Oracle ArgList -add ContainerName # hatype -modify Sqlnet ArgList -add ContainerName Example Oracle/Sqlnet definitions: # hares -add ORA_dse-ds1-d_PE10GSOL Oracle dse-ds1-d # hares -modify ORA_dse-ds1-d_PE10GSOL Critical 0 # hares -modify ORA_dse-ds1-d_PE10GSOL SID PE10GSOL # hares -modify ORA_dse-ds1-d_PE10GSOL ContainerName dse-ds1-d # hares -add SQL_dse-ds1-d_LISTENER_PE10GSOL Sqlnet dse-ds1-d # hares -modify SQL_dse-ds1-d_LISTENER_PE10GSOL Critical 0 # hares -modify SQL_dse-ds1-d_LISTENER_PE10GSOL LISTENER LISTENER_PE10GSOL # hares -modify SQL_dse-ds1-d_LISTENER_PE10GSOL ContainerName dse-ds1-d Create Application Links to Zone: # hares -link ORA_dse-ds1-d_PE10GSOL Zone_dse-ds1-d # hares -link SQL_dse-ds1-d_LISTENER_PE10GSOL Zone_dse-ds1-d Sybase RAW Devices within a Zone running on VxVM Currently, there is an issue with Zone visibility and VxVM volumes from the global zone. The following is a workaround until its addressed: Use the ls -l command to find out the major and minor numbers of the raw volume In this example, the volume u1 is in the DG1_dse-ds1-d disk group within the global zone. The raw device in the global zone that corresponds to u1 is /dev/vx/rdsk/DG1_dse-ds1-d/u1. Running the ls -l command on this device shows that the major number is 289, a nd the minor number is 45000: # ls -l /dev/vx/rdsk/DG1_dse-ds1-d crw------- 1 root root 289, 45000 Aug 24 11:13 v1 Use the mknod command to create an entry for the VxVM volume in the non global zone. In this example, the major number is 289 and the minor number is 45000: # cd /dse-ds1/zone_os/dev # mknod u1 c 289 45000 Log into the non global zone and check the device entry for the volume: # zlogin -l root dse-ds1-d [Connected to zone .dse-ds1-d. pts/6] Last login: Thu Aug 24 14:31:41 on pts/5 Sun Microsystems Inc. SunOS 5.10 Generic January 2005 # ls -l /dev/u1 crw-r--r-- 1 root root 289, 45000 Aug 24 16:10 /dev/v1 Verify that you can perform I/O with the device entry in the non global zone: # dd if=/dev/u1 of=/dev/null 2048+0 records in 2048+0 records out