JAVA_OPTS="-Xms128m -Xmx1024m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
Change in run.conf
Sunday, July 25, 2010
Friday, July 2, 2010
Download link
http://java.sun.com/javase/downloads/index.jsp - Jdk
http://www.zend.com/en/products/studio/downloads - Zend Studio
http://www.zend.com/en/products/server/downloads - Zend Server
http://wiki.centos.org/Download - CentOs ISO Images
http://www.ubuntu.com/desktop/get-ubuntu/download - Ubuntu Desktop Edition
http://fedoraproject.org/get-fedora-all - Fedora ISO Images
https://glassfish.dev.java.net/ - Glassfish Download
http://hudson-ci.org/ - Hudson Download
http://www.zend.com/en/products/studio/downloads - Zend Studio
http://www.zend.com/en/products/server/downloads - Zend Server
http://wiki.centos.org/Download - CentOs ISO Images
http://www.ubuntu.com/desktop/get-ubuntu/download - Ubuntu Desktop Edition
http://fedoraproject.org/get-fedora-all - Fedora ISO Images
https://glassfish.dev.java.net/ - Glassfish Download
http://hudson-ci.org/ - Hudson Download
Wednesday, June 30, 2010
How to install ImageMagick and RMagick on CEntOS
Install dependencies first.
yum install tcl-devel -y yum install libpng-devel -y yum install libjpeg-devel -y yum install ghostscript-devel -y yum install bzip2-devel -y yum install freetype-devel -y yum install libtiff-devel -y
Download and Install ImageMagick
As root, do the following:cd /usr/local/src wget ftp://ftp.imagemagick.org/pub/ImageMagick/ImageMagick.tar.gz tar xvfz ImageMagick.tar.gz cd ImageMagick* ./configure make make install
HDFS, HBASE, ZooKeeper Cluster Configuration
[hadoop@narmada1 hbase]$ bin/start-hbase.sh
starting master, logging to /u01/hbase/bin/../logs/hbase-
localhost2: starting regionserver, logging to /u01/hbase/bin/../logs/hbase-
localhost1 : starting regionserver, logging to /u01/hbase/bin/../logs/hbase-
[hadoop@narmada1 hbase]$ /opt/jdk1.6.0_20/bin/jps
5179 SecondaryNameNode
5004 DataNode
6198 HRegionServer
5326 QuorumPeerMain
6043 HMaster
6326 Jps
4853 NameNode
Checking the Hbase cluster configuration:
[hadoop@narmada1 hbase]$ bin/hbase shell
HBase Shell; enter 'help' for list of supported commands.
Version: 0.20.4, r941076, Tue May 4 16:23:06 PDT 2010
hbase(main):001:0> create 'Test_table'
0 row(s) in 1.2080 seconds
hbase(main):002:0> describe 'Test_table'
DESCRIPTION ENABLED
{NAME => 'Test_table', FAMILIES => []} true
1 row(s) in 0.0380 seconds
hbase(main):003:0> exit
[hadoop@narmada2 bin]$ ./hbase shell
HBase Shell; enter 'help' for list of supported commands.
Version: 0.20.4, r941076, Tue May 4 16:23:06 PDT 2010
hbase(main):001:0> describe 'Test-table'
{NAME => 'Test_table', FAMILIES => []} true
1 row(s) in 0.0350 seconds
hbase(main):003:0> exit
Netbeans Installation Error
If you get below error while installing netbeans
The wizard cannot continue because of the following error: could not load wizard specified in /wizard.inf (104)
WARNING: could not delete temporary file /tmp/ismp001/1497551
WARNING: could not delete temporary file /tmp/ismp001/7317780
do the below command
Monday, June 28, 2010
Hadoop installation and Configuration
Tar files :
/root/hadoop-0.20.2.tar.gz
/root/hbase-0.20.4.tar.gz
Path :
/opt/hadoop
/opt/hbase
JAR files to place in /server/default/
/opt/hadoop/hadoop-0.20.2-
/opt/hbase/hbase-0.20.4.jar
Hadoop Configuration Changes :
1. Added JAVA_HOME path in hadoop/conf/hadoop-env.sh
2. Created Data folder to store data and added the below property in hadoop/conf/core-site.xml
3. Added below porperty in hadoop/conf/hdfs-site.xml
4. Added below property in hadoop/conf/mapred-site.xml
Formatting the name node:
You need to do this the first time you set up a Hadoop cluster. Do not format a running Hadoop filesystem, this will cause all your data to be erased.
[root@sindhu11 hadoop]# bin/hadoop namenode -format
10/06/23 09:14:48 INFO namenode.NameNode: STARTUP_MSG:
/*****************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = localhost/10.60.11.101
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build = https://svn.apache.org/repos/
******************************
10/06/23 09:14:48 INFO namenode.FSNamesystem: fsOwner=root,root,bin,daemon,
10/06/23 09:14:48 INFO namenode.FSNamesystem: supergroup=supergroup
10/06/23 09:14:48 INFO namenode.FSNamesystem: isPermissionEnabled=true
10/06/23 09:14:48 INFO common.Storage: Image file of size 94 saved in 0 seconds.
10/06/23 09:14:49 INFO common.Storage: Storage directory /usr/local/hadoop-datastore/
10/06/23 09:14:49 INFO namenode.NameNode: SHUTDOWN_MSG:
/*****************************
SHUTDOWN_MSG: Shutting down NameNode at http://localhost/10.60.11.101
******************************
Start and Stop Hadoop stand-alone node:
[root@sindhu11 hadoop]# bin/start-all.sh
When you're done, stop the daemons with:
[root@sindhu11 hadoop]# bin/stop-all.sh
Browse the web interface for the NameNode and the JobTracker; by default they are available at:
- NameNode - http://localhost:
50070/ - JobTracker - http://localhost:
50030/
HBASE Configurations:
1. Added JAVA_HOME path in hbase/conf/hbase-env.sh
2. Added the below property in hbase/conf/hbase-site.xml (to mention the hadoop server ip and port)
Start and Stop HBase Stand-alone mode:
[root@sindhu11 hadoop]# bin/start-hbase.sh
localhost: starting zookeeper, logging to /opt/hbase/bin/../logs/hbase-
starting master, logging to /opt/hbase/bin/../logs/hbase-
localhost: starting regionserver, logging to /opt/hbase/bin/../logs/hbase-
To Stop :
[root@sindhu11 hadoop]# bin/stop-hbase.sh
To connect into HBASE Shell :
[root@sindhu11 hadoop]# bin/hbase shell
HBase Shell; enter 'help' for list of supported commands.
Version: 0.20.4, r941076, Tue May 4 16:23:06 PDT 2010
hbase(main):001:0>
Type 'help' at the shells' prompt to get a list of commands. Test your running install by creating tables, inserting content, viewing content, and then dropping your tables. For example:
hbase> create "mylittletable", "mylittlecolumnfamily"
hbase> # To see the schema for you just created "mylittletable" table and its single "mylittlecolumnfamily", type
hbase> describe "mylittletable"
hbase> # To add a row whose id is "myrow", to the column "mylittlecolumnfamily:x" with a value of 'v', do
hbase> put "mylittletable", "myrow", "mylittlecolumnfamily:x", "v"
hbase> # To get the cell just added, do
hbase> get "mylittletable", "myrow"
hbase> # To scan you new table, do
hbase> scan "mylittletable"Saturday, December 19, 2009
Netstat
List The Open Ports And The Process That Owns Them
netstat -lptu
netstat command to find open ports
# netstat --listen
Display open ports and established TCP connections:
# netstat -vatn
For UDP port try following command:
# netstat -vaun
If you want to see FQDN, remove -n flag:
# netstat -vat
Display list of open ports
# lsof -i
To display all open files, use:
# lsof
To display all open IPv4 network files in use by the process whose PID is 111, use:
# lsof -i 4 -a -p 111
Get Detailed Information About Particular IP address using netstat
This is useful to find out if your server is under attack or not. You can also list abusive IP address using this method.
# netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n
1 CLOSE_WAIT
1 established)
1 Foreign
3 FIN_WAIT1
3 LAST_ACK
13 ESTABLISHED
17 LISTEN
154 FIN_WAIT2
327 TIME_WAIT
Dig out more information about a specific ip address:
# netstat -nat |grep {IP-address} | awk '{print $6}' | sort | uniq -c | sort -n
2 LAST_ACK
2 LISTEN
4 FIN_WAIT1
14 ESTABLISHED
91 TIME_WAIT
130 FIN_WAIT2Busy server can give out more information:
# netstat -nat |grep 202.54.1.10 | awk '{print $6}' | sort | uniq -c | sort -n
15 CLOSE_WAIT
37 LAST_ACK
64 FIN_WAIT_1
65 FIN_WAIT_2
1251 TIME_WAIT
3597 SYN_SENT
5124 ESTABLISHED
Get List Of All Unique IP Address
To print list of all unique IP address connected to server, enter:
# netstat -nat | awk '{ print $5}' | cut -d: -f1 | sed -e '/^$/d' | uniq
To print total of all unique IP address, enter:
# netstat -nat | awk '{ print $5}' | cut -d: -f1 | sed -e '/^$/d' | uniq | wc -l
449
Find Out If Box is Under DoS Attack or Not
If you think your Linux box is under attack, print out a list of open connections on your box and sorts them by according to IP address
# netstat -atun | awk '{print $5}' | cut -d: -f1 | sed -e '/^$/d' |sort | uniq -c | sort -n
1 10.0.77.52
2 10.1.11.3
4 12.109.42.21
6 12.191.136.3
Display Summary Statistics for Each Protocol
# netstat -s | less
# netstat -t -s | less
# netstat -u -s | less
# netstat -w -s | less
# netstat -s
Display Interface Table
You can easily display dropped and total transmitted packets with netstat for eth0:
# netstat --interfaces eth0
The command for finding if you are under DoS attacks:
# netstat -atun | awk '{print $5}' | sed -n -e '/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/p’ | sed -n -e '/ESTABLISHED/p' | sed ’s/::ffff://’ | cut -d: -f1 | sort | uniq -c | sort -n
Find
Find
Syntax
# find location test options criteria_to_match action_to_perform
Finding files
To find all files having txt extension *.txt
# find / -name *.txt
To find file location
# find / -name syslog.conf
Files owned by a user
# find / -user root
Files by type
# find / -type d
l—symbolic link
d—directory
b—block (buffered) special
c—character (unbuffered) special
p—named pipe (FIFO)
s—socket
c—character (unbuffered) special
p—named pipe (FIFO)
s—socket
File according to date and time
# find / -atime +3
-atime +3: All files that were last accessed more than 3 days ago
mtime—the time that the contents of a file were last modified
atime—the time that a file was read or accessed
ctime—the time that a file’s status was changed
-n returns less than n
+n returns greater than n
n, by itself,returns exactly n matches
Files by size(larger than 5 MB)
# find / -size +5000000c
Finding by Permission and Ownership
# find / -type f -perm a=rwx
or
# find / -type f -perm 777
Finding files and perform some action (delete)
# find . -type f -perm 777 -exec ls -l *.* {} \;
Locate command
To find the location of files & directories
# locate *.txt
To update the database
# updatedb
grub
Fix grub issues
Boot from linux DVD or CD into rescue mode
boot: linux rescue
Linux installation will be available at special directory called /mnt/sysimage
Prepare system for grub installation, chroot command allows to run rest of all *COMMAND* with root directory set to NEWROOT called /mnt/sysimage
# chroot /mnt/sysimage
# cd /boot/grub
# cd /boot/grub
Check bootloader installation location
# grep '#boot' grub.conf
boot=/dev/sda
Reinstall Grub into sda
# grub-install /dev/sda
Reboot
# reboot
Audit
Linux audit files to see who made changes to a file
Package name audit
# yum install audit
auditctl - a command to assist controlling the kernel’s audit system. You can get status, and add or delete rules into kernel audit system.
ausearch - a command that can query the audit daemon logs based for events based on different search criteria.
aureport - a tool that produces summary reports of the audit system logs.
Start audit
# /etc/init.d/auditd start
Set a watch on a file for auditing
# auditctl -w /etc/passwd -p war -k password-file
Where,
-w /etc/passwd : Insert a watch for the file system object at given path i.e. watch file called /etc/passwd
-p war : Set permissions filter for a file system watch. It can be r for read, w for write, x for execute, a for append.
-k password-file : Set a filter key on a /etc/passwd file (watch). The password-file is a filterkey (string of text that can be up to 31 bytes long). It can uniquely identify the audit records produced by the watch. You need to use password-file string or phrase while searching audit logs.
In short you are monitoring (read as watching) a /etc/passwd file for anyone (including syscall) that may perform a write, append or read operation on a file
Add a watch on "/etc/shadow" with the arbitrary filterkey "shadow-file" that generates records for "reads, writes, executes, and appends" on "shadow"
# auditctl -w /etc/shadow -k shadow-file -p rwxa
Find out who changed or accessed a file /etc/passwd
# ausearch -f /etc/passwd
or
# ausearch -f /etc/passwd -i | less
Where,
-f /etc/passwd : Only search for this file
-i : Interpret numeric entities into text. For example, uid is converted to account name
SSH
| Daemon | sshd |
| Port | 22 |
| Config | /etc/ssh/sshd_config |
Default Config Files and SSH Port
OpenSSH server configuration file
/etc/ssh/sshd_config
OpenSSH client configuration file
/etc/ssh/ssh_config
Users ssh configuration directory
~/.ssh/
Lists the public keys (RSA or DSA) that can be used to log into the user’s account
~/.ssh/authorized_keys
If the below file exists, sshd refuses to let anyone except root log in
/etc/nologin
Access controls lists that should be enforced by tcp-wrappers are defined here.
/etc/hosts.allow and /etc/hosts.deny
SSH default port : TCP 22
Use SSH Protocol 2
sshd_config file configurations
# Only allow root and xyz user to use the system via SSH, add the following to sshd_config:
AllowUsers root xyz
# Alternatively, you can allow all users to login via SSH but deny only a few users, with the following line:
DenyUsers abc foo
# Configure Idle Log Out Timeout Interval (sshd_config)
ClientAliveInterval 300
ClientAliveCountMax 0
# Disable root Login via SSH
PermitRootLogin no
# Enable a Warning Banner
Banner /etc/issue
# Change SSH Port and Limit IP Binding
Port 300
ListenAddress 192.168.1.5
ListenAddress 202.54.1.5
# Disable Empty Passwords
PermitEmptyPasswords no
# Use Log Analyzer
LogLevel INFO
Verify your sshd_config file before restarting / reloading changes:
# /usr/sbin/sshd -t
SSH allows administrators to set an idle timeout interval
Find ClientAliveInterval and set to 300 (5 minutes) as follows:
# vi /etc/ssh/sshd config
ClientAliveInterval 300
ClientAliveCountMax 0
Save and close the file. Restart sshd:
# service sshd restart
Running a remote command
# ssh username@host command1;command2
Allow SSH without password
Server – login to other system without password
Client – allows user from server without password
Server configuration
# ssh-keygen -t dsa
# cd .ssh
# scp id_dsa.pub root@server:~/.ssh/authorized_keys
Boot Process
Stages of Boot process
| System Startup | Bios |
| Stage1 Bootloader | MBR |
| Stage2 Bootloader | GRUB |
| Kernel & initrd | Linux |
| init | User-space |
- BIOS Initialization
- Boot Loader resides in MBR, load from boot partition
- Kernel Initialization Device detection, device driver initialization, Mount root filesystem read only, Loads initial process (init)
- /etc/rc.d/rc.sysinit - Ask I for interactive mode (activate selinux, set clock, enable swap, set hostname, root filesystem check, activate RAID, LVM, disk quota)
- /etc/rc.d/rc initialize default run level as per inittab file entry
- start rcX.d as per run level
- /etc/rc.d/init.d contains all service files
- /etc/rc.d/rc.local run after run level specific sctipts, common place for custom modification. Run each time system enters a run level
If root filesystem is mounted as read only in single user mode, to remount in rw
# mount –o rw,remount /
# mount –o remount –w /
File system check without prompting
# fsck –y
NFS
Network file system is both a protocol and file system for accessing and sharing file systems across a computer network using Linux.
NFS v4 is used in modern Linux distributions. It offers performance improvements, mandates strong security, and introduces a stateful protocol etc.
NFS Server configuration file
/etc/exports – file enumerates the filesystem exported through NFS
share hostname|ipaddress|domain|*(options)
options include
sync Reply to requests only after the changes have been committed to stable storage
ro exports as read-only
rw reading and writing (default)
no_root_squash allows normal access by root
noaccess prevent access
secure remote access originate from privileged port
To save new changes to export file
# exportfs -a
To remove entries from exports table
# exportfs -u
Export a directory with NFS
To export or share directory called /data
Edit the file /etc/exports, this file serves as the access control list for file systems which may be exported to NFS clients
Add config directive to exports file
/data *(rw,sync)
Restart the nfs service:
# /etc/init.d/nfs restart
or
# service nfs restart
NFS Client configuration
Client computer need to mount file system using mount command or /etc/fstab file
# mkdir /data
# mount -t nfs nfsservername|ip:/mountpoint /data
Add an entry in /etc/fstab
nfsservername|ip:/mountpoint /data nfs defaults 0 0
To list the exported filesytem
# Showmount –e hostname
Mount NFS filesystem in two different directories on same system
NFS Server : nfsserver:/data
Mount to client dirs /data and /backup
Mount command has bind option to remount part of the file hierarchy somewhere else
First mount to /home/data
# mount -t nfs4 nfsserver:/data /data
Now bind /home/data to /backup
# mount --bind /data /backup
NAS (Network attached storage) also supports NFS configuration. Access NAS server using NFS protocol
NFS Daemons, In order to use NFS you need to run portmap service and rpc.statd and rpc.lockd daemons
# chkconfig portmap on
# chkconfig nfslock on
# /etc/init.d/portmap start
# /etc/init.d/nfslock start
To access NAS from client
# mkdir /backup
# mount -o tcp 202.54.20.111:/mountpoint /backup
Add an entry in /etc/fstab
202.54.20.111:/mountpoint /backup nfs defaults 0 0
NTP
The Network Time Protocol package contains utilities and daemons that will synchronize your computer's time to Coordinated Universal Time (UTC) via the NTP protocol and NTP servers.
The NTP packageincludes ntpdate (a program for retrieving the date and time from remote machines via a network) and ntpd (a daemon which continuously adjusts system time).
NTP Server Configuration
Your NTP server contacts a central NTP server (Public Server located at ntp.org, to obtain accurate time data).
The server then allows other machines on your network to request the time data.
Sample Network
192.168.1.10 ==> NTPD Server
201.55.11.1 ==> Remote NTP server
192.168.1.0/24 ==> NTP clients
# vi /etc/ntp.conf
restrict default ignore
//Above will deny all access to any machine, server or client
restrict 202.55.11.1 mask 255.255.255.245 nomodify notrap noquery
server 202.55.11.1
//Configure NTP clients to access your NTP Server
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
//Allow NTP clients to access the Server
NTP Client configuration
# vi /etc/ntp.conf
server 192.168.1.100
Determining If NTP Is Synchronized Properly
# ntpq -p
update NTP manually, but ntp daemon should be off
# ntpdate servername
Local date & time
Date command displays the date and time of the system
# date nnddhhmm[[cc]yy][.ss]
where,
nn = month of the year (01 to 12)
dd = day of the month (01 to 31)
hh = hour of the day (00 to 23)
mm = minute of the hour (00 to 59>
cc = The first to digits of the year
yy = The last two digits of the year
.ss = The seconds
The hardware clock is the clock that runs in you PC hardware even if you disconnect it from the main power supply, this is because it has a lithium battery.
hwclock command used to set or get the time of you hardware clock
Set the clock you August 8th, 2007 at 21:08 and will tell your clock that is the local time
# hwclock --set --date='08/18/07 21:08:40' --localtime
To check
# hwclock --show
About Linux
Linus Torvalds wrote the first Linux kernel way back in 1991
Linux Principle
- Everything is file
- Configuration data stored in text file
Uniqueness of Linux
- Multitasking capability
- Multi-user capability
- Portability
Linux Architecture
- Kernel is the heart of the operating system and it controls the hardware
- The Shell acts as an interpreter between the user and the computer
- Tools are grouped into categories for certain functions, such as word processing, business applications
File System - Disk Quota
Disk space can be restricted by implementing disk quotas which alert a system admin before a user consumes too much disk space or a partition becomes full.
Configuring Disk Quotas
Enable quotas per file system by modifying /etc/fstab
# vi /etc/fstab
/dev/VG0/LV2 /home ext3 defaults,usrquota,grpquota 1 2
/dev/VG0/LV3 swap swap defaults 0 0
Remount the file system
# mount -o remount /home
Create quota database file (to generate aquota.user & aquota.group files)
# quotacheck -cug /home
option c for create quota file, u for user quota file & g for group quota file
Assigning quotas per user
# edquota username
add soft and hard values
Assigning quotas per group
# edquota -g groupname
Assigning quotas per filesystem
# edquota -t
Now enable quotas
# quotaon -vug /home
Reporting on disk quotas
# repquota /home
File System - Access Control List
Used to grant access to multiple users
Mount with ACL option
# mount –o remount,acl /directory_name
List assigned ACL
# getfacl file|directory_name
Set ACL for user
# setfacl –m u:username:rwx file
Set ACL for group
# setfacl –m g:groupname:rwx file
Set ACL only for Directory
# setfacl –m d:u:username:rwx directory_name
Remove ACL
# setfacl –x u:username file|directory_name
Entries appear in ACL
user::perm - set permission for file owner
user::username:perms - set permission for a specific user
group::perms - set permission for group that own the file
group::groupname:perm - set permission for a specific group
other::perms - set permission for all others
mask::perms - set permission for all but owner and other
File Permissions
File Permissions
- Every file is owned by a user and group
- Three Access Categories are user, group, others
- Three Permission types are read, write & execute
- suid – command run with permission of owner
- sgid – command runs with group affiliation
- sticky bit – special restriction on deletion of files other than owner
- umask 002 – list permission to withhold (default value)
Permission types
| Letter | Numeric | Permission |
| r | 4 | read |
| w | 2 | create, write & remove files |
| x | 1 | execute |
| - | 0 | no permission |
Changing file permission
# chmod [-R] mode file|directory
where Mode is
| u | user |
| g | group |
| o | others |
| a | all |
| + | grant |
| - | deny |
Changing file ownership
# chown [–R] user file|directory
# chown [-R] user:group file|directory
# chgrp [–R] group file|directory
SUID and SGID Executables
Set on an executable file cause it to run under the user/group security context of file owner/group
SGID Directories
File created in directory with SGID bit set, it belongs to the same group as the directory
Sticky Bit
Only owner can delete the file
# chmod o+t directory
T appears if directory execute permission for others is off
| Octal Mode Number | Description |
| 1000 | Sets the sticky bit |
| 2000 | Sets the setgid bit |
| 4000 | Sets the setuid bit |
Linux Read mode permissions
- Read access on a file allows you to view file
- Read access on a directory allows you to view directory contents with ls command
Write mode permissions
- Write access on a file allows you to write to file
- Write access on a directory allows you to remove or add new files
Execute mode permissions
- Execute access on a file allows to run program or script
- Execute access on a directory allows you access file in the directory
Default File permission
Umask used to withhold permission on file creation
Default User’s umask is 022
File will have 644
Directories will have 755
File Types
| Symbol | Meaning | Description |
| - | Regular | Files like word, text etc |
| d | Directory | File to hold other file types |
| b | Block | Accepts input in blocks [group of byte] e.g. Storage devices |
| c | Character | Accepts input one character [byte] at a time e.g. Serial port |
| p | Pipe | Allows sending info between apps |
| s | Socket | Allows info to be exchanged over network |
| l | Link | Additional pathname to reference |
Two types of Links
Hard link share the same inode
Soft link is more of a short cut to the original file
Executables types
1. Binary Files
2. Scripts #!/usr/bin/bash Interpreter
Determining file content
# file filename
Report use of specific file or file system
# fuser file|filesystem
Similar to fuser with more info
# lsof file|filesystem
Subscribe to:
Posts (Atom)