lsusb and chroot in anaconda.. Is usbfs mounted in anaconda %post installation ?

The binary ‘/sbin/lsusb’ in a chroot-ed environment have problems running properly. I have not checked this in a manually created chroot environment or using tools like ‘mock’.

The scenario is as following :

We were trying to check the output of ‘lsusb’ in the %post section of a kickstart installation. I had specified ‘noreboot’ in the kickstart file so the machine will wait for the user to manually reboot the machine. This helps to check the logs and the situation of the machine just after the installation finishes.

After the installation and prior to the reboot, i checked in the second available terminal (Alt + F2) created by anaconda and was astonished to see that the command ‘lsusb’ does not give us the required output but an error that ‘/usr/share/hwdata/usb.ids’ is not accessible or found.

By default, i think only the ‘installation’ ie.. the %post section starts in a ‘chroot’ mode and the terminal available is not chroot-ed. So we will have to use ‘/mnt/sysimage/sbin/lsusb’. This didn’t work as expected since the ‘lsusb’ binary needs to check the file ‘/usr/share/hwdata/usb.ids’ and won’t be able to find it.

So I did a chroot from the second terminal and did an /sbin/lsusb, since /sbin in not in the ‘PATH’ by default. That too, didn’t work out. But this time it didn’t even complain anything. Just nothing at all, no output. Last time, at-least it complained it could not find something. So how do we go forward now ??? Here comes ‘strace’ to the rescue !!!

strace is of-course a really nice tool to know what system calls are made and lots of internal stuff a binary will do while being executed. But ‘strace’ is not installed by default on a RHEL5 machine, which is the case here. As most of you would know, anaconda creates a virtual file system which consists of most of the folders found under a linux main /. The location where the OS is installed is mounted under /mnt/sysimage.

Since we already have an ISO from where we have booted the machine from (DVD/CD), we are free to mount it on the filesystem, which is what we did. :

# mkdir /mnt/source

# mount -t iso9660 /dev/hdc /mnt/source

# cd /mnt/source/Server/

In case you want to know how the DVD/CD drive is detected, all you need to do is execute ‘dmesg’ in the available terminal. ie.. after pressing ‘Alt + Ctrl + F2′.

So we went forward and mounted the DVD to /mnt/source and changed the directory to /mnt/source/Server where all the rpm packages reside. Installed the package ‘strace’ using an ‘rpm -ivh’. Please note that we need to use ‘–root /mnt/sysimage’ as an option since we are installing the package to our newly installed file system which is at /mnt/sysimage. If this is not used, the installer will try to install the package to the virtual environment created in the memory.

# cd /mnt/source/Server

# rpm -ivh strace-<version>.rpm –root /mnt/sysimage

# cd

# chroot /mnt/sysimage

This will make /mnt/sysimage as the working root, ie.. where our installation was done. OK.. now for the ‘strace’ stuff.

# strace -fxvto strace.log -s 1024 /sbin/lsusb

The strace output will be saved to ‘strace.log’ which we can open up in a text editor of our choice. Opening it in ‘vi’, shows a lot of stuff such as the command run, the default language, location of libraries loaded, the environment variables etc.. In this case we would only need to be interested in the last parts, ie.. to know where the binary failed :

<snip>

15:16:17 open(“/dev/bus/usb”, O_RDONLY|O_NONBLOCK|O_DIRECTORY) = -1 ENOENT (No such file or directory) = 03067

15:16:17 open(“/proc/bus/usb”, O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 33067

15:16:17 fstat(3, {st_dev=makedev(0, 3), st_ino=4026532146, st_mode=S_IFDIR|0555, st_nlink=2, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0, st_atime=2009/09/25-15:16:17, st_mtime=2009/09/25-15:16:17, st_ctime=2009/09/25-15:16:17}) = 03067

15:16:17 fcntl(3, F_SETFD, FD_CLOEXEC) = 03067

15:16:17 getdents(3, {{d_ino=4026532146, d_off=1, d_reclen=24, d_name=”.”} {d_ino=4026531879, d_off=2, d_reclen=24, d_name=”..”}}, 4096) = 483067

15:16:17 getdents(3, {}, 4096) = 03067

15:16:17 close(3) = 03067

15:16:17 exit_group(1) = ?

</snip>

The above trace output shows how the ‘lsusb’ binary proceeded at its last time and where it failed. We can see that it went to open ‘/dev/bus/usb’, only to find that the said location does not exist. We can understand that it is a directory from the call ‘open(“/dev/bus/usb”, O_RDONLY|O_NONBLOCK|O_DIRECTORY)’.

Ok,, fine.. so what does it do next ?

As the next step, it tries to open ‘/proc/bus/usb’ and it is present, which we know since there are no ‘No such file or directory’ errors. Going further, the binary goes on to do a ‘stat’ on ‘/proc/bus/usb’. After doing an ‘fstat’, it goes to check the file descriptor using ‘fcntl’ and further goes to list the directory contents using ‘getdents’.

This is where we find the interesting output :

getdents(3, {{d_ino=4026532146, d_off=1, d_reclen=24, d_name=”.”} {d_ino=4026531879, d_off=2, d_reclen=24, d_name=”..”}}, 4096) = 48

As you can see in the above trace, it returns ‘.’ and ‘..’, which means there are nothing in /proc/bus/usb. So what we do understand is ‘lsusb’ refers /dev/bus/usb and /proc/bus/usb for its outputs.. If it was not able to find anything, strace would have given us an error which obviously would have made life much easier.

And that’s how ‘/sbin/lsusb’ failed silently.. Isn’t strace a nice tool ??

Okay, those who want to know why is this so… ‘lsusb’ needs either /mnt/sysimage/proc/bus/usb or /mnt/sysimage/dev/bus/usb display contents to work properly. Anaconda is not mounting /mnt/sysimage/proc/bus/usb with the ‘usbfs’ file system in the limited installation environment and hence ‘lsusb’ fails…

And we have a fix for that which goes into yuminstall.py in the anaconda source :

try:

    isys.mount("/proc/bus/usb", anaconda.rootPath + "/proc/bus/usb", "usbfs")

except Exception, e:

    log.error("error mounting usbfs: %s" %(e,))

This piece of python code, tries mounting /proc/bus/usb on /mnt/sysimage/proc/bus/usb as ‘usbfs. If its not possible, the code excepts an Exception error and reports “error mounting ‘usbfs’.

Device Mapper and applications ….

What is device-mapper ?

Device mapper is a modular driver for the linux kernel 2.6. It can be said as a framework which helps to create or map logical sectors of a pseudo block device to an underlying physical block device. So what device-mapper do is keep a table of mappings which equate the logical block devices to the physical block devices.

Applications such as LVM2, EVMS, software raid aka dmraid, multipathing, block encryption mechanisms such as cryptsetup etc… use device-mapper to work. All these applications excluding EVMS use the libdevmapper library to communicate with device-mapper.

The applications communicate with device-mapper’s API to create the mapping. Due to this feature, device-mapper does not need to know what LVM or dmraid is, how it works, what LVM metadata is, etc… It is upto the application to create the pseudo devices pointing to the physical volumes using one of device-mapper’s targets and then update the mapper table.

The device-mapper mapping table :

The mapping table used by device-mapper doesn’t take too much space and is a list created using a ‘btree’. A btree or a ‘Binary Search Tree‘ is a data-structure from which data can be added, removed or queried.

In order to know more on what a btree is and the concept behind it, read :

http://en.wikipedia.org/wiki/Binary_search_tree

http://en.wikipedia.org/wiki/B-tree

Types of device-mapper targets :

Applications which use device-mapper actually use one or more of its target methods to achieve their purpose. Targets can be said as a method or type of mapping used by device-mapper. The general mapping targets are :

a) Linear – Used by linear logical volumes, ie.. the default data layout method used by LVM2.

b) Striped – Used by striped logical volumes as well as software RAID0.

c) Mirror – Used by software RAID1 and LVM mirroring.

d) Crypt – Used by disk encryption utilties.

e) Snapshot – Used to take online snapshots of block devices, an example is LVM snapshot.

f) Multipath – Used by device-mapper-multipath.

g) RAID45 – Software raid using device-mapper, ie.. dmraid

h) Error – Sectors of the pseudo device mapped with this target causes the I/O to fail.

There are a few more mappings such as ‘flaky’ which is not used much.

I’ll write on how device-mapper works in LVM, in the next post…

lvcreate fails with the error “Aborting. Failed to activate new LV to wipe the start of it.”. Why ??

In case anyone out there gets an error message like “Aborting. Failed to activate new LV to wipe the start of it.” while doing an ‘lvcreate’, please check your lvm configuration (/etc/lvm/lvm.conf) once more. Most probably, a ‘volume_list’ would have been defined in there, which in turns want you to specify the ‘volume_list’ tag specified along with the lvcreate command.

Excerpt from /etc/lvm/lvm.conf:

<>

# If volume_list is defined, each LV is only activated if there is a
# match against the list.
#   “vgname” and “vgname/lvname” are matched exactly.
#   “@tag” matches any tag set in the LV or VG.
#   “@*” matches if any tag defined on the host is also set in the LV or VG
#
# volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]
volume_list = [ "VG01", "@foo.com" ]

</>

In this case, you will have to use the ‘lvcreate’ command as follows, which will create the logical volume properly.

# lvcreate –addtag @foo.com    … <following-options> ..

Why does specifying ‘@kde-software-development’ in ‘%packages’ section of the kickstart file doesn’t install the needed packages in RHEL5.4 ?

Adding the package group ‘kde-software-development’ in the %packages section for a RHEL5.4 kickstart would not install the packages which come under the said group. This is due to a clean-up done in the comps.xml file lately to remove many of the obsolete stuff. In case anyone doesn’t know about the comps.xml file, its an XML formatted file present in the ISO media in which all of the package groups, the packages under each available group, its dependencies etc.. are defined.

In the RHEL5.4 comps.xml file, even though the package-group ‘kde-software-development’ is defined, no packages are mentioned under it and so nothing gets installed quite to the annoyance of the user. Quite a bit more stuff are broken in the comps file and i think it will be cleaned up in the further versions.

How to create a bootable USB disk with RHEL image..

In this scenario, we will have to format the USB drive, so please double check if you have any valuable data on the USB disk since this would be wiped off while formatting.

Connect the USB drive to the machine. In this case, we do not need the device to be mounted so if it is mounted automatically, it must be unmounted.

You can do it using the command :

(a) # umount /dev/sd’X’ (where /dev/sd’X’ is the device file name)

If the device was detected as /dev/sdb, the command would be ‘umount /dev/sdb’. The device could be detected as a different device file on your machine. On connecting the USB, you can do a ‘dmesg | tail’ to check if the USB device is detected or not.

As the next step, we have to format the USB disk. We are using ‘fdisk’ to format the disk.

(b) # /sbin/fdisk /dev/sdb (Assuming /dev/sd is /dev/sdb)

On executing the command /sbin/fdisk, you will be presented with the prompt ‘Command (m for help):’ after a bit of details. If you select ‘m’, you will be presented with a list of options by which you can carry out the partitioning process.

From the list of options, we can see that ‘p’ is used for printing the partitioning table.

———
Command (m for help): p

Disk /dev/sdb: 1998 MB, 1998519808 bytes
62 heads, 62 sectors/track, 1015 cylinders
Units = cylinders of 3844 * 512 = 1968128 bytes
Disk identifier: 0x59c07261

Device Boot Start End Blocks Id System
/dev/sdb1 1 1936 1951456+ 6 FAT16

Command (m for help):
———

(c) If you see any partitions as above (most probably you would see one in FAT/VFAT), we will have to delete them. For deleting it, we can use the ‘d’ command in fdisk. From the output of step (b), we can see that the USB disk has one FAT16 partition.

———
Command (m for help): d
Selected partition 1

Command (m for help): p

Disk /dev/sdb: 1998 MB, 1998519808 bytes
32 heads, 63 sectors/track, 1936 cylinders
Units = cylinders of 2016 * 512 = 1032192 bytes
Disk identifier: 0x297029dc

Device Boot Start End Blocks Id System

Command (m for help):
———-

(d) Write the new partition table to the disk using ‘w’.

———-
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

———-

As the next step, we use the command ‘partprobe’ to force the kernel re-read the partition table so that we need not re-boot the machine for the partition table to be effective.

(e) # partprobe /dev/sdb

As the next step, copy the ‘diskboot.img’ file to the USB drive using the ‘dd’ command. You will be able to find the said file in the ‘isoimages’ directory of the installation disk. The ‘dd’ (disk dump) command will dump the contents bit by bit so that its an exact replica.

(f) # dd if=diskboot.img of=/dev/sdb
24576+0 records in
24576+0 records out
12582912 bytes (13 MB) copied, 2.85287 s, 4.4 MB/s

Ensure you have the entire contents on the USB drive by mounting the media to the filesystem :

# mount /dev/sdb /media/
# ls /media/
boot.msg general.msg initrd.img isolinux.bin ldlinux.sys options.msg param.msg rescue.msg snake.msg splash.lss syslinux.cfg vmlinuz

Unmount the USB disk using the command :

(g) # umount /media

Now enable the BIOS option to boot from the USB drive on the machine you intend to install, connect the drive and boot. You must see the boot prompt after this step.

How to map PCI devices in Linux ?

From the output of the command ‘lspci -n’ (The number after the colon, here ’1679′ from the below snip)

0a:04.0 0200: 14e4:1679 (rev a3)
Subsystem: 103c:703c
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium
Latency: 64 (16000ns min), Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 138
Region 0: Memory at fdef0000 (64-bit, non-prefetchable) [size=64K]
Region 2: Memory at fdee0000 (64-bit, non-prefetchable) [size=64K]

IMPORTANT:
——————-

In the above line “14e4:1679″, ’14e4′ is the UID of the manufacturer and ’1679′ is the card model or hardware ID.

The actual way to proceed is to open the pci.ids file (‘/usr/share/hwdata/pci.ids’ and ‘/lib/modules/`uname -r`/modules.pcimap’) and check for the manufacturer UID, like ’14e4′ which is the ‘Broadcom Corporation’. The file /lib/modules/`uname -r`/modules.pcimap would be more reliable since it is from the modules of the loaded kernel.

Under that, check the card model, like ’1679′ which is ‘NetXtreme BCM5715S Gigabit Ethernet’.

Under that you can also have subdivisions, so in order to pin-point a particular card you will have to use the ‘Subsystem’ value from ‘lspci’.

In this example, ‘Subsystem’ is 103c:703c, which turns out to be ‘NC326i PCIe Dual Port Gigabit Server Adapter’

Static routes in RHEL

How can we add routes for ethernet interfaces that would be persistent over reboots, in RHEL.

For each device eth existing on the machine (or for the ones you want a persistent route), create the file /etc/sysconfig/network-scripts/route-ethX.

The contents should in the format:

via dev eth
via dev eth0

Make one for each device and modify the lines above to match each device. Each line is parsed by the ifup start up scripts and is passed and appended to the “ip” command.

For example:

ip 10.0.1.0/24 via 192.168.2.4 dev eth0

For more information on methods of configuring static routes, check the “IP route section of the IP man page.

ip_conntrack module dropping packets

This time its related to an error on iptables, which can be a bit annoying. This usually can happen in really busy machines which could have a lot of TCP/IP network connections being established.

The error follows a pattern as below : (from /var/log/messages)

<snip>

kernel: ip_conntrack: table full, dropping packet.
last message repeated 9 times
kernel: printk: 289 messages suppressed.
kernel: ip_conntrack: table full, dropping packet.
kernel: printk: 399 messages suppressed.
kernel: ip_conntrack: table full, dropping packet.
kernel: printk: 407 messages suppressed.
kernel: ip_conntrack: table full, dropping packet.
kernel: printk: 311 messages suppressed.
kernel: ip_conntrack: table full, dropping packet.
kernel: printk: 488 messages suppressed.
kernel: ip_conntrack: table full, dropping packet.
kernel: printk: 470 messages suppressed.
kerrnel: ip_conntrack: table full, dropping packet.

</snip>

The module “ip_conntrack” keeps a table of all of the current connections on the system and when the table becomes full,  the problems begin (i.e., dropping packets).

So the workaround for avoiding this error, would be to increase the value in the file ‘/proc/sys/net/ipv4/ip_conntrack_max’.

# echo 655360 > /proc/sys/net/ipv4/ip_conntrack_max

This would enable the kernel to keep track of enough connections and the error would be gone.

Disable CHROOT in named

The recent versions of bind recommends the chroot environment for better security. When considering the most used popular Linux distribution ‘Red Hat’, they ship a package called ‘bind-chroot’ for the chrooting of bind.  By default the bind or named daemon runs in a chroot environment.

But users who want to run ‘named’ in the old fashion may do so by disabling the ‘chroot’ environment. The ‘chroot’ing feature can be disabled by commenting out the directive ‘ROOTDIR=/var/named/chroot’ in the file ‘/etc/sysconfig/named’ and then restarting the ‘named’ service.

‘named’ can be restarted using the command ‘/etc/init.d/named restart’.

File Counter

Hi,

Most of the scripts presented in this journal have been created while learning bash and having nothing much to do… I think its usual to get crazy ideas and work trying to implement them, especially while learning any type of coding.  This  ‘File Counter’ script came as such a  crazy idea. It was working but have not checked it recently.. should work..

This script counts the entire number of files irrespective the folders under the main directory you specify for this script to work on. ie.. It recursively  counts the files under a directory tree.

Please feel free to point out the mistakes…..

<snip>
#!/bin/bash
#################################################################
#  Counts the number of files recursively inside a directory    #
#   Coded by Vimal Kumar                                        #
#################################################################

echo ; clear
echo -e “Please enter the directory location where you want the files to be counted…\n” ; echo

read dir ; echo ;  if [ ! -d $dir ] ; then echo -e “The location you specified doesn’t exist.\n” ; exit 0;
          else cd $dir ; echo -e “Please wait for the files to be counted…..\n” ; echo ; fi

X=`ls -l | wc -l`
Y=`ls -l | grep ^d | awk ‘{print $9}’`
B=`ls -l $Y | awk ‘{print $9}’ | grep . | wc -l `
A=`expr $X + $B`

echo -e “There are a total of $A files inside the directory $dir…\n”

C=`ls -Rl | grep -v ./ | grep -v total | grep . | awk ‘{print $8}’`

echo -e “Do you want to scan the directory for the file types?\n”
echo -e “Y/N\n” ; read choice;
if [ $choice = Y ] ;
then cd $dir ; file $C | awk ‘{print $1,”=======>>”, $2}’ > $HOME/Filetype.txt;echo -e “Output saved in file Filetype.txt.\n”
elif [ $choice = N ] ; then echo -e “Thankyou $USER, Take care….\n”
else echo ; echo -e “Invalid choice buddy…\n” ; echo -e “Exiting…..Bye..\n” ;  fi

</snip>

Vimal Kumar