Oracle jre and CentOS alternatives

wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" 'http://download.oracle.com/otn-pub/java/jdk/8u20-b26/server-jre-8u20-linux-x64.tar.gz'
tar -xvzf server-jre-8u20-linux-x64.tar.gz
mkdir -p /usr/java && mv jdk1.8.0_20 /usr/java
ln -fs /usr/java/jdk1.8.0_20 /usr/java/latest
ln -fs /usr/java/jdk1.8.0_20 /usr/java/default
alternatives --install /usr/bin/java java /usr/java/latest/bin/java 2000
alternatives --set java /usr/java/latest/bin/java
Veröffentlicht unter /dev/administration | Hinterlasse einen Kommentar

ClouderaManager hive „Class org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found“

1. Select Service > Hive
2. Select Configuration Tab (View and edit)
3. Open service-wide
4. Select advanced
5. Select hive-site.xml safety valve
6. Copy paste (path to jars may differ):

<property>
<name>hive.aux.jars.path</name>
<value>/opt/cloudera/parcels/CDH-4.2.1-1.cdh4.2.1.p0.5/lib/hive/lib/hive-contrib-0.10.0-cdh4.2.1.jar,/opt/cloudera/parcels/CDH-4.2.1-1.cdh4.2.1.p0.5/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.2.1.jar,/opt/cloudera/parcels/CDH-4.2.1-1.cdh4.2.1.p0.5/lib/hive/lib/hbase.jar,/opt/cloudera/parcels/CDH-4.2.1-1.cdh4.2.1.p0.5/lib/hive/lib/zookeeper.jar</value>
<description>A comma separated list (with no spaces) of the jar files required for Hive-HBase integration</description>
</property>

7. Save changes
8. Deploy client configuration
9. Restart hive and hue services

Veröffentlicht unter /dev/administration | Hinterlasse einen Kommentar

Impala Install on CentOS 6.4

On the main impala node which also runs the state store:

curl 'http://archive.cloudera.com/impala/redhat/6/x86_64/impala/cloudera-impala.repo' > /etc/yum.repos.d/cloudera-impala.repo; echo "enabled = 1" >> /etc/yum.repos.d/cloudera-impala.repo
yum -y inastall impala-state-store impala-server impala-shell

The cloudera packages create a config directory in /etc/impala. As with all cloudera packages, alternatives is used to select the config dir for the running services:

mkdir -p /etc/impala/conf.yourConfigDir; alternatives --install /etc/impala/conf impala-conf /etc/impala/conf.yourConfigDir 40 && alternatives --set impala-conf /etc/impala/conf.yourConfigDir

Now copy some files to the config dir:

rsync -av /etc/hadoop/conf/core-site.xml /etc/impala/conf/
rsync -av /etc/hadoop/conf/hdfs-site.xml /etc/impala/conf/
rsync -av /etc/hadoop/conf/log4j.properties /etc/impala/conf/
rsync -av /etc/hive/conf/hive-site.xml /etc/impala/conf/

Change some settings in
core-site.xml:

<property> 
<name>dfs.client.read.shortcircuit</name> 
<value>true</value> 
</property> 
<property> 
<name>dfsclient.read.shortcircuit.skip.checksum</name> 
<value>false</value> 
</property>

hdfs-site.xml:

<property>
  <name>dfs.datanode.data.dir.perm</name>
  <value>755</value>
</property>

<property>
  <name>dfs.block.local-path-access.user</name>
  <value>impala,mapred,your user name here</value>
</property>

On the impala worker nodes:
Enable cloudera impala repository as described above.
Install impala-server:

yum -y inastall impala-server

Create the configuration directory and update alternatives:

mkdir -p /etc/impala/conf.yourConfigDir; alternatives --install /etc/impala/conf impala-conf /etc/impala/conf.yourConfigDir 40 && alternatives --set impala-conf /etc/impala/conf.yourConfigDir

Rsync the configuration files from your master node to the worker:

rsync -av root@your_impala_master:/etc/impala/conf/ /etc/impala/conf/

Edit /etc/default/impala to connect to your impala state store server:

IMPALA_STATE_STORE_HOST=your_impala_master
Veröffentlicht unter /dev/administration | Hinterlasse einen Kommentar

Hadoop with pydoop on CentOS 6.4

Using pydoop instead of the streaming interface and MRJob results in a massive performance boost.
I ran a simple wordcount job with a bit of regex matching on ~2GB of text data. Using the streaming interface the job took about 11 minutes on our hadoop cluster to finish. The same job took about 5 minutes using pydoop.
To install pydoop on CentOS6.4 for CDH3:

export CLASSPATH=/usr/lib/hadoop-0.20/hadoop-core.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
export JAVA_HOME=/usr/java/latest
export HADOOP_CONF_DIR=/etc/HADOOPDIR/CONFDIR
export HADOOP_HOME=/usr/lib/hadoop-0.20
yum -y install gcc
yum -y install gcc-c++
yum -y install boost-devel
yum -y install python-devel
yum -y install python-pip
yum -y install openssl-devel
/usr/bin/pip-python install argparse
/usr/bin/pip-python install importlib
/usr/bin/pip-python install jlib
/usr/bin/pip-python install pydoop

To install pydoop on CentOS6.4 for CDH4:

export CLASSPATH=/usr/lib/hadoop-0.20-mapreduce/hadoop-core.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/lib/commons-*.jar
export JAVA_HOME=/usr/java/latest
export HADOOP_CONF_DIR=/etc/HADOOPDIR/CONFDIR
export HADOOP_HOME=/usr/lib/hadoop/
yum -y install gcc
yum -y install gcc-c++
yum -y install boost-devel
yum -y install python-devel
yum -y install python-pip
yum -y install openssl-devel
/usr/bin/pip-python install argparse
/usr/bin/pip-python install importlib
/usr/bin/pip-python install jlib
/usr/bin/pip-python install pydoop

And if you want to use snappy as compression codec: maybe you are also missing a useful commandline tool for snappy. So, for your convenience (rpms were build from these sources) :
snzip-0.9.0-0.el6.x86_64.rpm
snzip-0.9.0-0.el6.src.rpm

Veröffentlicht unter /dev/administration | Hinterlasse einen Kommentar

Spring is coming ;)



Veröffentlicht unter /dev/video | Hinterlasse einen Kommentar

Black screen in HP Lo-100 virtual kvm on ELRepo.org 3.7.6 kernels [FIX]

Updated some HP Proliant-Server to a 3.7.6 Kernel. After booting, all was fine except that the Java KVM tool showed just a black login screen.
To fix this, force the kernel to boot in textmode only:

  • edit /etc/grub.conf and remove „rhgb“ and „quiet“ from the kernel boot parameters
  • add nomodeset to the kernel boot parameters
  • reboot
Veröffentlicht unter /dev/administration | Hinterlasse einen Kommentar

Sieve vacation plugin: discarding vacation response for message implicitly delivered to [FIX]

This message shows up in the log when sieve vacation is enabled for an account but the mail was addressed to an alias rather than to the original address.
This behaviour is not a bug, it’s an RFC (which in this case may read „request for chaos“ ;).

We are using a Postfix – Amavisd-New – Spamassassin – Dovecot stack and I recently ran into this problem. To fix this, change your /etc/postfix/main.cf from:

...
mailbox_command = /usr/libexec/dovecot/deliver
...

to

...
mailbox_command = /usr/libexec/dovecot/deliver -f "$SENDER" -a "$ORIGINAL_RECIPIENT"
...

hth someone…

Veröffentlicht unter /dev/administration | Verschlagwortet mit , , | 1 Kommentar

netvault 101

Description:
Running nvjobdelete on a scheduled job returns the error ‚Failed to delete job xxx: Job is active‘

Netvaults knowledgebase article 79982:

The problem comes from the user login.

Make sure the login user is part of the correct Group Membership.
This is found in Netvault ¦ Access Control ¦ Right click User and select ‚User properties‘ and open ‚Group Memberships‘ tab.
Add the user as the member of the correct group.

Also, in some cases if you double check Job Management… to see if in the „jobs“ tab… that it’s not showing up as „running“ job … if so, kill/abort the job then you should be able to delete the job from the status tab.

BUT if this doesn’t help you:

  • go to „job management“
  • open the „status“ tab
  • check if the job in question is listed as „Scheduled“
  • if so, right click on job an select „Delete Schedule“

Now you should be able to delete the job.

Veröffentlicht unter /dev/administration | Verschlagwortet mit | Hinterlasse einen Kommentar

friday night





Veröffentlicht unter /dev/video | Hinterlasse einen Kommentar

synology raid5 rampage

Last night a raid 5 array in our office storage workhorse, a synology, kicked the bucket with two failed disks. Luckily one could be resurrected from the dead. After some S.M.A.R.T. status checks on this one, all things seemed to be fine. Only one failed disk in a raid 5 should not cause any data loss…

Well, it should not but an

cat /proc/mdstat


showed the recovered drive as a spare, which it wasn't prior to the failure.
So i decided to recreate the whole array, following these steps:

Stop the raid:

mdadm --stop /dev/md2

Check the metadata version via:

mdadm --examine /dev/sdc3 | grep Version

Create the array, with the failed drive as missing:

mdadm --create --assume-clean --level=5 --raid-devices=4 metadata=0.9 /dev/md2 /dev/sda3 missing /dev/sdc3 /dev/sdd3

That went smoothly and the raid was up and running again. Sadly the ext4 filesystem has sufferd, and could not be mounted. An

fschk.ext4 /dev/md2

also yielded an error.

As the synology does not come with the nice tool dumpe2fs, I needed another way of finding the location of the backup superblocks:

mkfs.ext4 -n /dev/md2

From the manpage:

-n Causes mke2fs to not actually create a filesystem, but display what it would do if it were to create a filesystem. This can be used to determine the location of the backup superblocks for a particular filesystem, so long as the
mke2fs parameters that were passed when the filesystem was originally created are used again. (With the -n option added, of course!)

With that information I was able to run:

fschk.ext4 -b 214990848 -y -C 0 /dev/md2

This saved almost all of the data on the array... (the -C parameter gives you a nice progress bar ;)

Veröffentlicht unter /dev/administration | Hinterlasse einen Kommentar