Solaris Comstar client setup

iSCSI

• Linux

Create a new file called 99-zfssa.rules inside /etc/udev/rules.d with the following contents:

ACTION=="add", SYSFS{vendor}=="SUN", SYSFS{model}=="*ZFS*", ENV{ID_FS_USAGE}!="filesystem", ENV{ID_PATH}=="*-iscsi-*", RUN+="/bin/sh -c 'echo 1024 > /sys$DEVPATH/queue/max_sectors_kb'"
  
Reboot

• Windows   

Open up the Registry Editor (regedit)
Navigate to HKLM\SYSTEM\CurrentControlSet\Control\Class\{4D36E97B-E325-11CE-BFC1-08002BE10318}\<Instance Number>\Parameters
Change the following parameters to 1048576 in Decimal form:
    FirstBurstLength
    MaxBurstLength
    MaxRecvDataSegmentLength
    MaxTransferLength
Reboot

• Solaris  

# echo 'set maxphys=1048576' >> /etc/system
Reboot

Linux. how to find procceses which uses swap?

GOAL

Find proccesses which uses swap

SOLUTION

Summary by processes

#!/bin/bash

echo -n "" > /tmp/swapusage.output
for i in $( ps -e -o pid --no-headers); do
 swapusage=$(grep Swap "/proc/$i/smaps"| grep -v '\W0 kB' | awk '{s+=$2} END {print s}')
 if [ ! -z "$swapusage" ]; then  
  cmd=$(ps -p $i -o user,cmd --no-headers | cut -d" " -f 1)
  printf "%s\t%-40s\t\t\t%s\n" $i $cmd $swapusage >> /tmp/swapusage.output
 fi ;
done
echo -e "PID\tCMD\t\t\t\t\t\t\t\tSWAP kB"
echo "--------------------------------------------------------------------------------"
sort -k 3 -n /tmp/swapusage.output

Top 10 segments in swap

# ps ax | sed "s/^ *//" > /tmp/ps_ax.output
# for x in $(grep Swap /proc/[1-9]*/smaps | grep -v '\W0 kB' | tr -s ' ' | cut -d' ' -f-2 | sort -t' ' -k2 -n | tr -d ' ' | tail -20); do
    swapusage=$(echo $x | cut -d: -f3)
    pid=$(echo $x | cut -d/ -f3)
    procname=$(cat /tmp/ps_ax.output | grep "^$pid ")
    echo "============================"
    echo "Process   : $procname"
    echo "Swap usage: $swapusage kB"; done

udev. change network interface name.

To change network card name without reboot:
1) Bring the network interfaces down, then modify /etc/udev/rules.d/70-persistent-net.rules (or its equivalent)
2) re-load with udevadm control --reload-rules and finally
3) re-trigger with udevadm trigger --attr-match=subsystem=net
4) bring the network interfaces up.

Linux. Orphaned connections in CLOSE_WAIT state ( a lot of connections are in CLOSE_WAIT state )

Usually dealing with web servers we have a lot of connections in CLOSE_WAIT state. These connections hold web server threads and also consume our nofile, nproc resources described in /etc/security/limits.conf.

To prevent such situation we need to adjust some kernel variables:

• net.ipv4.tcp_keepalive_time - the interval between the last data packet sent (simple ACKs are not considered data) and the first keepalive probe; after the connection is marked to need keepalive, this counter is not used any further.
• net.ipv4.tcp_keepalive_intvl - the interval between subsequential keepalive probes, regardless of what the connection has exchanged in the meantime.
• net.ipv4.tcp_keepalive_probes - the number of unacknowledged probes to send before considering the connection dead and notifying the application layer.

The first two parameters are expressed in seconds, and the last is the pure number. This means that the keepalive routines wait for two hours (7200 secs, default value) before sending the first keepalive probe, and then resend it every 75 seconds. If no ACK response is received for 9 consecutive times, the connection is marked as broken.

You can modify and persistently save your own setting using sysctl command and file /etc/sysctl.conf like this:

sysctl -w net.ipv4.tcp_keepalive_time=120 >> /etc/sysctl.conf
sysctl -w net.ipv4.tcp_keepalive_probes=3 >> /etc/sysctl.conf 
sysctl -w net.ipv4.tcp_keepalive_intvl=20 >> /etc/sysctl.conf

Linux, 'du' and 'df' tools report different space utilization

The 'df' command reports how many disk blocks are used, whilst 'du' traverses the filesystem and reports the actual number of blocks used (directory by directory), including any relating to files in use by processes.
In most cases, space utilisation values returned from 'df' and 'du' will be consistent. However, the potential exists for a running process to delete a large file, say. In this instance, according to 'du', the large file no longer exists, so the blocks associated with the deleted file are not reported. With the process still running, and with an open file descriptor still held against the deleted file, 'df' continues to track and report all disk blocks used, including those associated with the deleted (phantom) file. In this situation, the disk space associated with the deleted file will only be relinquished back to the system when the process completely releases the deleted file's descriptor or the process terminates (either gracefully or killed).

The solution is to identify and stop (or kill) the process that continues to hold a file descriptor open for the deleted file. To do so, run the lsof command (/usr/sbin/lsof | grep deleted) as root to identify the holding process, for example:

# lsof | grep deleted

COMMAND     PID   USER    FD  TYPE  DEVICE      SIZE       NODE NAME

cannaserv  3825  canna    0u   CHR   136,0                    2 /dev/pts/0 (deleted)

vmware     4295   root    6u   REG   253,0      6770   13074503 /tmp/vmware-root/ui-4295.log (deleted)

vmware-re  4316   root    6u   REG   253,0      6770   13074503 /tmp/vmware-root/ui-4295.log (deleted)

vmnet-nat  4448   root    0u   CHR   136,0                    2 /dev/pts/0 (deleted)

vmware-se  4454   root    0u   CHR   136,0                    2 /dev/pts/0 (deleted)

gdm-binar  4506   root    0u   CHR   136,0                    2 /dev/pts/0 (deleted)

gconfd-2   5392   root   12wW  REG   253,0       609   13090818 /tmp/gconfd-root/lock/0t1188207163ut519551u0p5392r346479926k3219784492 (deleted)

vmware-vm  5822   root   57u   REG   253,0   6520832   13074477 /tmp/vmware-root/ram0 (deleted)

vmware-vm 16487   root   57u   REG   253,0  11153408   13074520 /tmp/vmware-root/ram0 (deleted)

kdeinit   17991   root   17u   REG   253,0     26712   13074524 /tmp/kde-root/khtmlcacheM7jXYb.tmp (deleted)

kdeinit   17991   root   18u   REG   253,0      5631   13074501 /tmp/kde-root/khtmlcacheZlJmda.tmp (deleted)

kdeinit   17991   root   21u   REG   253,0     44718   13074514 /tmp/kde-root/khtmlcacheH5m4lc.tmp (deleted)

In the example above, the 7th column in the output denotes the size of deleted files (in bytes). The 9th column denotes which file remains held open. The 1st and second columns denotes the process and pid that still holds the open file descriptor.

RHEL. Network post up scripts

Issue

• How do I run a script or program immediately after my network interface goes up?

Environment

• Red Hat Enterprise Linux 4, 5, 6

Resolution

• Create file /sbin/ifup-local and add the commands to run after the network is up.

An example for one physical interface:

#!/bin/bash
if [ "$1" == "eth0" ]; then
  /sbin/ethtool -G eth0 rx 4096 tx 4096
fi

For bonding and bridging, you need to reference the virtual device being started, not the underlying physical devices:

#!/bin/bash
if [ "$1" == "br0" ]; then
  /sbin/ethtool -G eth0 rx 4096 tx 4096
fi

#!/bin/bash
if [ "$1" == "bond0" ]; then
  /sbin/ethtool -G eth0 rx 4096 tx 4096
  /sbin/ethtool -G eth1 rx 4096 tx 4096
fi

• Make /sbin/ifup-local executable

chmod +x /sbin/ifup-local

Root Cause

The network control scripts in /etc/sysconfig/network-scripts allows for adding scripts that will run after the device is up.

For example, looking at the /etc/sysconfig/network-scripts/ifup-eth file, the last few lines are:

...
        echo ${dhcp6_pid[0]} > /var/run/dhcp6c_${DEVICE}.pid
    fi;
fi

if [ "${IPX}" = yes ]; then
       /etc/sysconfig/network-scripts/ifup-ipx ${DEVICE}
fi

exec /etc/sysconfig/network-scripts/ifup-post ${CONFIG} ${2}

A script /etc/sysconfig/network-scripts/ifup-post is the last item to be executed. Looking at that script:

...
if [ -x /sbin/ifup-local ]; then
    /sbin/ifup-local ${DEVICE}
fi

exit 0

Notice the last few lines search for a file called /sbin/ifup-local. If that file is found, it is executed.

The file /sbin/ifup-local is the one that is available to run programs immediately after the interface is up.

Linux. Работа с SCSI

GOAL

SCSI и Linux, в общих чертах

SOLUTION

Адресс SCSI устройств

<H>ost:      Instance of hostadapter to which device is attached
<B>us:       SCSI Bus or Channel on the hostadapter
<T>arget:    SCSI Id assigned to an individual device
<L>un:       Logical unit number on the device

Посмотреть подключенные SCSI устройства можно множеством комманд.

LSSCSI

#  lsscsi
[0:0:0:0]    disk    Single   Flash Reader     1.00  /dev/sdb
[1:0:0:0]    disk    HP       LOGICAL VOLUME   6.64  /dev/sda
[1:3:0:0]    storage HP       P410i            6.64  -
[2:0:0:0]    disk    3PARdata VV               3222  /dev/sdc
[2:0:0:254]  enclosu 3PARdata SES              3222  -
[2:0:1:0]    disk    3PARdata VV               3222  /dev/sdd
[2:0:1:254]  enclosu 3PARdata SES              3222  -
[3:0:0:0]    disk    3PARdata VV               3222  /dev/sde
[3:0:0:254]  enclosu 3PARdata SES              3222  -
[3:0:1:0]    disk    3PARdata VV               3222  /dev/sdf
[3:0:1:254]  enclosu 3PARdata SES              3222  -
[4:0:0:0]    cd/dvd  hp       DVD D  DS8D3SH   HHE8  /dev/sr0

В квадратных скобках показывается адрес устройства. Почитайте man комманды lsscsi, удобные для использования опции.

PROC

# cat /proc/scsi/scsi
Attached devices:
Host: scsi1 Channel: 03 Id: 00 Lun: 00
  Vendor: HP       Model: P410i            Rev: 6.64
  Type:   RAID                             ANSI  SCSI revision: 05
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: HP       Model: LOGICAL VOLUME   Rev: 6.64
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: Single   Model: Flash Reader     Rev: 1.00
  Type:   Direct-Access                    ANSI  SCSI revision: 00
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: 3PARdata Model: VV               Rev: 3222
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 00 Lun: 254
  Vendor: 3PARdata Model: SES              Rev: 3222
  Type:   Enclosure                        ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 01 Lun: 00
  Vendor: 3PARdata Model: VV               Rev: 3222
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 01 Lun: 254
  Vendor: 3PARdata Model: SES              Rev: 3222
  Type:   Enclosure                        ANSI  SCSI revision: 06
Host: scsi4 Channel: 00 Id: 00 Lun: 00
  Vendor: hp       Model: DVD D  DS8D3SH   Rev: HHE8
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 00
  Vendor: 3PARdata Model: VV               Rev: 3222
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi3 Channel: 00 Id: 00 Lun: 254
  Vendor: 3PARdata Model: SES              Rev: 3222
  Type:   Enclosure                        ANSI  SCSI revision: 06
Host: scsi3 Channel: 00 Id: 01 Lun: 00
  Vendor: 3PARdata Model: VV               Rev: 3222
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi3 Channel: 00 Id: 01 Lun: 254
  Vendor: 3PARdata Model: SES              Rev: 3222
  Type:   Enclosure                        ANSI  SCSI revision: 06




Информацию по SCSI адаптерам можно посмотреть в файловой системе sys

# ls -ld /sys/class/scsi_host/host*/
drwxr-xr-x 2 root root 0 2008-08-06 17:25 /sys/class/scsi_host/host0/
drwxr-xr-x 2 root root 0 2008-08-06 17:25 /sys/class/scsi_host/host1/

Используемый SCSI драйвер

# cat /sys/class/scsi_host/host[0-1]/proc_name
ata_piix
ata_piix




Если SCSI устройства подключены по протоколу FC, то инфомацию о FC адаптерах можно найти там же

# ls -l /sys/class/fc_host/host*
lrwxrwxrwx 1 root root 0 Jun 26 21:41 /sys/class/fc_host/host2 -> ../../devices/pci0000:00/0000:00:01.0/0000:0e:00.0/host2/fc_host/host2
lrwxrwxrwx 1 root root 0 Jun 26 21:41 /sys/class/fc_host/host3 -> ../../devices/pci0000:00/0000:00:01.0/0000:0e:00.1/host3/fc_host/host3

WWN FC карточки инициатора

# cat /sys/class/fc_host/host*/port_name
0x5001438024246d18
0x5001438024246d1a




WWN FC карточки таргета

# cat /sys/class/fc_remote_ports/*/port_name
0x20510002ac018827
0x21510002ac018827
0x20520002ac018827
0x21520002ac018827

Удалить/добавить SCSI устройства в ядрах версии 2.4.Х. (так же работает и для новых ядер)

## смотрим подключенные устройства
# lsscsi
[0:0:0:0]    disk    ATA      IC35L040AVVA07-0 VA2O  /dev/sda
[0:0:1:0]    disk    ATA      WDC WD400BB-32CL 05.0  /dev/sdb
[1:0:0:0]    disk    ATA      QUANTUM FIREBALL A1Y.  /dev/sdc
[1:0:1:0]    cd/dvd  JLMS     DVD-ROM XJ-HD166 DD05  /dev/sr0

## удаляем устройство
# echo "scsi remove-single-device 1 0 0 0" > /proc/scsi/scsi

# lsscsi
[0:0:0:0]    disk    ATA      IC35L040AVVA07-0 VA2O  /dev/sda
[0:0:1:0]    disk    ATA      WDC WD400BB-32CL 05.0  /dev/sdb
[1:0:1:0]    cd/dvd  JLMS     DVD-ROM XJ-HD166 DD05  /dev/sr0

## добавляем устройство
# echo "scsi add-single-device 1 0 0 0" > /proc/scsi/scsi

# lsscsi
[0:0:0:0]    disk    ATA      IC35L040AVVA07-0 VA2O  /dev/sda
[0:0:1:0]    disk    ATA      WDC WD400BB-32CL 05.0  /dev/sdb
[1:0:0:0]    disk    ATA      QUANTUM FIREBALL A1Y.  /dev/sdd
[1:0:1:0]    cd/dvd  JLMS     DVD-ROM XJ-HD166 DD05  /dev/sr0




Это же можно сделать через файловую систему SYS ( linux 2.6.XX  и выше)

# echo "1" > /sys/block/sdc/device/delete

или вот так

# echo 1 > /sys/class/scsi_host/host<H>/device/target<H>:<B>:<T>/<H>:<B>:<T>:<L>/delete




Чтобы добавить scsi устройство

echo "<B> <T> <L>" > /sys/class/scsi_host/host<H>/scan




если хотите просканировать всю scsi шину замените B T L на три тире

echo "- - -" > /sys/class/scsi_host/host0/scan

в RHEL есть скрипт rescan-scsi-bus.sh который поставляется с пакетом sg3_utils.

Иногда бывает полезным изменить уровень логирования в драйвере scsi, это можно сделать вот так

# sysctl -q -w dev.scsi.logging_level=<N>

Выдержка из scsi_logging.h

/*

* This defines the scsi logging feature. It is a means by which the user

* can select how much information they get about various goings on, and it

* can be really useful for fault tracing. The logging word is divided into

* 8 nibbles, each of which describes a loglevel. The division of things is

* somewhat arbitrary, and the division of the word could be changed if it

* were really needed for any reason. The numbers below are the only place
* where these are specified. For a first go-around, 3 bits is more than

* enough, since this gives 8 levels of logging (really 7, since 0 is always
* off).

*/

#define SCSI_LOG_ERROR_SHIFT 0

#define SCSI_LOG_TIMEOUT_SHIFT 3

#define SCSI_LOG_SCAN_SHIFT 6

#define SCSI_LOG_MLQUEUE_SHIFT 9

#define SCSI_LOG_MLCOMPLETE_SHIFT 12

#define SCSI_LOG_LLQUEUE_SHIFT 15

#define SCSI_LOG_LLCOMPLETE_SHIFT 18

#define SCSI_LOG_HLQUEUE_SHIFT 21

#define SCSI_LOG_HLCOMPLETE_SHIFT 24

#define SCSI_LOG_IOCTL_SHIFT 27

#define SCSI_LOG_ERROR_BITS 3 /* additional logging associated with errors and recovery */

#define SCSI_LOG_TIMEOUT_BITS 3 /* additional logging associated with command timeouts */

#define SCSI_LOG_SCAN_BITS 3 /* additional logging associated with device scans and discovery */

#define SCSI_LOG_MLQUEUE_BITS 3 /* additional logging associated with mid-level command queueing */

#define SCSI_LOG_MLCOMPLETE_BITS 3 /* additional logging associated with mid-level command completions */

#define SCSI_LOG_LLQUEUE_BITS 3 /* additional logging associated with low-level command queueing */

#define SCSI_LOG_LLCOMPLETE_BITS 3 /* additional logging associated with low-level command completions */

#define SCSI_LOG_HLQUEUE_BITS 3 /* additional logging associated with hi-level command queueing */

#define SCSI_LOG_HLCOMPLETE_BITS 3 /* additional logging associated with hi-level command completions */

#define SCSI_LOG_IOCTL_BITS 3 /* additional logging associated with ioctl (typ: non-data commands) */

extern unsigned int scsi_logging_level;




Troubleshooting

В тарблшутинге пригодится SCSI-Programming-HOWTO