How To Create A Wrapper Script With Customized Environment

When working in a team, everyone will have to conform and work in a same environment. But there are times that, a certain script of yours needs a different environment in order to run. An example would be, everyone in the team has their python version set to 2.7.13. But for some reason, you have a python script that will only work with python 3. Having a wrapper script that customize the environment before running your script would solve this issue. Here's how the wrapper script will look like

1 2 3 4 5 6 7

#!/bin/tcsh -f setenv BNR_PATH /some/special/path/ setenv BNR_BIN /another/special/path/bin setenv BNR_ROOT /special/root/path /path/to/your/script.py $argv:q

How To Post Codes In Blogs As Html Using Pygmentize

https://pygments.org/download/


For Command Line Help:-

>pygmentize -h
>pygmentize -H

To see all available lexer (language supported), formatter, etc ...

>pygmentize -L

To see all Options available for HTML formatter:-

>pygmentize -H formatter html

To convert lines of codes into html format so that u can post it to a blog:-

>cat report_waiverfile_errors.py | pygmentize -l python -f html -O noclasses

To include line numbers:-

>cat report_waiverfile_errors.py | pygmentize -l python -f html -O 'noclasses,linenos'

To Highlight certain lines:-

>cat report_waiverfile_errors.py | pygmentize -l python -f html -O 'noclasses,linenos,hl_lines="12 13 14 15 16 17"'

Example:-

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

#!/usr/bin/env python import os import sys #sys.path.insert(0, '/nfs/site/disks/da_infra_1/users/yltan/depot/da/infra/dmx/main/lib/python') import logging import dmx.tnrlib.waiver_file LOGGER = logging.getLogger() def main(): wf = dmx.tnrlib.waiver_file.WaiverFile() wf.load_from_file(sys.argv[1]) if __name__ == "__main__": logging.basicConfig(format='[%(asctime)s] - %(levelname)s-[%(module)s]: %(message)s', level=logging.DEBUG) main()

Fromatting Perforce Output For Simple Scripting

Perforce has a global option -ztag




If you run a normal perforce command like this

1 2 3 4

p4 files -m3 ... //depot/da/infra/dmx/main/lib/python/dmx/tnrlib/__init__.py#1 - add change 4491003 (text+kx) //depot/da/infra/dmx/main/lib/python/dmx/tnrlib/audit_check.py#57 - edit change 5716075 (text+kx) //depot/da/infra/dmx/main/lib/python/dmx/tnrlib/css/style.css#1 - branch change 4757380 (text+k)

With the -ztag ...

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

p4 -ztag files -m3 ... ... depotFile //depot/da/infra/dmx/main/lib/python/dmx/tnrlib/__init__.py ... rev 1 ... change 4491003 ... action add ... type text+kx ... time 1473325143 ... depotFile //depot/da/infra/dmx/main/lib/python/dmx/tnrlib/audit_check.py ... rev 57 ... change 5716075 ... action edit ... type text+kx ... time 1556850031 ... depotFile //depot/da/infra/dmx/main/lib/python/dmx/tnrlib/css/style.css ... rev 1 ... change 4757380 ... action branch ... type text+k ... time 1494905434

Now, say, I'd like to have perforce report out the following info in 3 columns, separated by :::, like this

1	depotFile ::: revision ::: type

All I need to do is run the following:-

1 2 3 4

p42 -ztag -F "%depotFile% ::: %rev% ::: %type%" files -m3 ... //depot/da/infra/dmx/main/lib/python/dmx/tnrlib/__init__.py ::: 1 ::: text+kx //depot/da/infra/dmx/main/lib/python/dmx/tnrlib/audit_check.py ::: 57 ::: text+kx //depot/da/infra/dmx/main/lib/python/dmx/tnrlib/css/style.css ::: 1 ::: text+k

... and here you go :)

How To Create Chart In Linux (unsing gnuplot)

Read this first

    https://stackoverflow.com/questions/327576/how-do-you-plot-bar-charts-in-gnuplot


You need to create 2 files:-

#===================
#   gnuplot.dat
#===================
0 label       100
1 label2      450
2 "bar label" 75

#===================
#   gnuplot.cmd
#===================
set boxwidth 0.5
set style fill solid
set term svg 
set output "mychart.svg"
plot "gnuplot.dat" using 1:3:xtic(2) with boxes

Now, run this command:-

    gnuplot gnuplot.cmd

This will create the output file

    mychart.svg

To view the chart

    display mychart.svg

Tutorial helps a lot 
    http://www.gnuplot.info/docs/tutorial.pdf

How To Run A Script As The Owner Of The Script

For the explanation below, just keep in mind that these assumption is true:-
- owner of file == yltan(uid=742)
- user that executes the script == icetools(uid=48102)

How to run a python script as the owner of the script.

1. Create any simple python script, eg a.py:-

#!/usr/bin/env python
import os

print "Before swap::" + str(os.getresuid())
os.system("whoami")

os.setreuid(os.geteuid(),os.getuid())

print "After Swap:" + str(os.getresuid())
os.system("whoami")

2. This is the magic wrapper script, setuid_swap.c. :-

/* This binary is intended to be a setuid script wrapper for dbRsync.pl.
  
   Example compilation on an hp machine:  gcc dbRsync.c -o dbRsync
   This compilation creates an executable called dbRsync. After compiling,
   chmod +s dbRsync (or whatever the executable is) to set the sticky bit or
   it won't work.
*/

int main(int ac, char **av) {
    execv( "/path/to/your/a.py", av );
}

3. Compile the setuid_wrap.c to wrapper_a :-

gcc setuid_swap.c -o wrapper_a

4. Change the setuid bit:-

chmod 4755 wrapper_a

5. Run wrapper_a as someone else other than the owner of the file:-

sudo su - icetools

./wrapper_a

6. ... and this is the output that you get (Running as icetools):-

Before swap::(48102, 742, 742)
icetools
After Swap:(742, 48102, 48102)
yltan

For a detail explanation of how setuid work,

https://drive.google.com/open?id=0B_HHt58thGk_MTVwQkhleWNmZE0



Alternatively, you could swap the users in the compiled code itself by doing this:-

/* This binary is intended to be a setuid script wrapper for dbRsync.pl.
  
   Example compilation on an hp machine:  gcc dbRsync.c -o dbRsync
   This compilation creates an executable called dbRsync. After compiling,
   chmod +s dbRsync (or whatever the executable is) to set the sticky bit or
   it won't work.
*/

int main(int ac, char **av) {
    int uid;
    uid = geteuid();
    setreuid(uid, uid);
    execv( "/path/to/your/a.py", av );
}

How To Diff A Directory Efficiently

diff -qr -x '*~' -x '*.swp' -x '*.pyc'  -I '$Revision:'  -I '$File:'  -I '$Header:' -I '$Change:' -I '$DateTime:' -I '$Id:' -I '$Date:' -I '$Change:' -I '$Author:' directory1 directory2

How To Write Codes In Perl For Accomodating To Simple Testings

In order to write scripts in perl which are testable, it is good to follow the following proposed method (assuming this code is inside file a.pl):-

#!/usr/bin/env perl                                                                                   

sub main
{                                                                                                  
    ### Your main code goes here           
    # ... ... ...                          
    1;                                                         
}


sub is_five
{
    my $num = shift(@_);
    if ($num == 5)
    {
        return 1;
    }
    else:
    {
        return 0;
    }
} # is_five


############################
# This loop will only be entered if this script is called explicitly, ie:-
#   $./a.pl
#
# This loop will not be entered If this file is required, ie:-
#   require 'a.pl'
############################
unless (caller)
{
    main();
}

Now, to write a test that tests the is_five() function, we can create a file call test_a.py, and write the test like this:-

#!/usr/bin/env perl

use Test::Simple tests => 2;
require "a.pl";

sub test_is_five___pass
{
    return is_five(5)
}
sub test_is_five___fail
{
    return is_five(3)
}

unless (caller)
{
    ok(test_is_five___pass());
    ok(! test_is_five___fail());
}

Debugging Flow For A Process

I've been stucked with a job which took a lot longer than what it is suppose to take to complete. A friend of mine shared his gems of tricks on how he troubleshoot it, and with a little bit of tweak to my taste, I've came up with a flow(well, almost all of it still from him :p) which I'm logging it here so that I won't forget it.


#1 Find the offending process id (_pid_)

ps -aux | grep job_name

#2 Look at the entire hierarchy of the pid and look at where it stops

pstree -pulna _pid_

#3 Look at the trace of the running program.

strace -t -s 22222 -p _pid_

#4 Look at the read/write IO activities 

cat /proc/_pid_/io

Useful Links

http://www.linux-tutorial.info/modules.php?name=MContent&pageid=84
http://chadfowler.com/blog/2014/01/26/the-magic-of-strace/

Quotes Within Quotes In Linux

I keep forgetting this every time even though I have stumbled across this so many times.

I'm now gonna stick this here so that I can refer it back and hopefully remember this for good.

http://stackoverflow.com/questions/1250079/how-to-escape-single-quotes-within-single-quoted-strings

Basically, the idea is to ....

always just replace each embedded single quote with the sequence: '\'' (that is: quote backslash quote quote) or '"'"' , which closes the string, appends an escaped single quote and reopens the string. 

 https://stackoverflow.com/a/1315213/335181


Putting the above into a perl script works wonder:

#!/usr/bin/perl -pl
s/'/'\\''/g;    ### or s/'/'"'"'/g;
$_ = qq['$_'];

Working With Perforce -G (Python Marshalled Object) Option

Most common pitfall is to directly get the output pirnted in stdout and marshal it:-

#!/usr/bin/env python
cmd = 'p4 -G changelists -m 3'
mo = subprocess.Popen(cmd, shell=True, stderr=subprocess.STDOUT, stdout=subprocess.PIPE).communicate()[0]
result = marshal.loads(mo)
pprint(result)

Nope. This won't work correctly.
You will only get the last data in the list of 3 items.

This is the correct way to do it:-

#!/usr/bin/env python
cmd = 'p4 -G changelists -m 3'
mo = subprocess.Popen(cmd, shell=True, stderr=subprocess.STDOUT, stdout=subprocess.PIPE)

result = []
try:
   while 1:
      output = marshal.load(mo.stdout)
      result.append(output)
except EOFError:
   pass
finally:
   mo.stdout.close()

pprint(result)

Reading/Writing xlsx File In Python

I'm using the python library openpyxl.
The below shows an example of how to iterate thru cells ...

#!/usr/bin/env python

import openpyxl
from pprint import pprint

wb = openpyxl.reader.excel.load_workbook('test.xlsx')
s = wb.get_sheet_by_name(name='Sheet1')

d = s.calculate_dimension()

r = s.range(d)
print "Range:" + str(r)
for row in r:
   for cell in row:
      print cell.get_coordinate()
      print cell.value

If you need help, u can use the python interactive session:-

%python
>>> import openpyxl
>>> help(openpyxl.reader.excel)
>>> wb = openpyxl.reader.excel.load_workbook('test.xlsx')
>>> help(wb)
>>> ws = wb.get_sheet_by_name(name='Sheet1')
>>> help(ws)
>>> cell = ws.cell(row=0, column=0)
>>> help(cell)

How To Kill A Bunch Of My LSF Jobs

I want to list out all my current jobs in lsf:-

%bjobs

JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
4883896 yltan RUN ice_arc_sm pg-yltan-l pg-iccf0082 *eep 11111 Mar 1 15:37
4883898 yltan RUN ice_arc_sm pg-yltan-l pg-iccf0039 *eep 11111 Mar 1 15:37
4883974 yltan RUN ice_arc_sm pg-yltan-l pg-iccf0079 *eep 11111 Mar 1 15:45
4883975 yltan RUN ice_arc_sm pg-yltan-l pg-iccf0082 *eep 11111 Mar 1 15:45
4883976 yltan PEND ice_arc_sm pg-yltan-l pg-iccf0035 *eep 11111 Mar 1 15:45
4883977 yltan PEND ice_arc_sm pg-yltan-l pg-iccf0039 *eep 11111 Mar 1 15:45

Killing All of the listed jobs 

%bjobs | awk '$1 ~ /^[0-9]/ {print $1}' | xargs bkill

Job <4883896> is being terminated
Job <4883898> is being terminated
Job <4883974> is being terminated
Job <4883975> is being terminated
Job <4883976> is being terminated
Job <4883977> is being terminated

Killing All of the listed jobs, with a few exceptions

%bjobs | awk '$1 ~ /^[0-9]/ && $1 !~ /(4883896|4883898)/ {print $1}' | xargs bkill

Job <4883974> is being terminated
Job <4883975> is being terminated
Job <4883976> is being terminated
Job <4883977> is being terminated

Book Review: jQuery Pocket Reference

After possessing all the basic information that you need to know about designing webpage in the internet about HTML, CSS and JavaScript, the next big thing that helps you make your website more dynamic, more elegant, and adding the WOW effect is jQuery + jQueryUi.











A quick explanation on what is actually jQuery:-

jQuery is the "write less, do more" JavaScript library. Its powerful features and ease of use have made it the most popular client-side JavaScript framework for the Web. This book is jQuery's trusty companion: the definitive "read less, learn more" guide to the library.
jQuery Pocket Reference explains everything you need to know about jQuery, completely and comprehensively. You'll learn how to:

• Select and manipulate document elements
• Alter document structure
• Handle and trigger events
• Create visual effects and animations
• Script HTTP with Ajax utilities
• Use jQuery's selectors and selection methods, utilities, plugins and more

The 25-page quick reference summarizes the library, listing all jQuery methods and functions, with signatures and descriptions.

This isn't a book that goes into the deep and detail points of the subject.
As the book name stated, it is just a pocket reference.

Even though as the name stated, it is still a very good book that managed to explain the basic, the fundamentals, and all the need-to-know subjects, and pack it in a very nice and non-boring way within 150+some pages.

For those programmers that are already used to reading and fiddling thru programming documentations, the official  jQuery + jQueryUi website's documentation is already very well and perfectly.... ermmm ..... documented.

With this book, it just add the punch, and get your engine moving is at smoother and faster pace.

Definitely a book which is worth the 2-weeks reading time.

Book review: Beginning HTML, XHTML, CSS and JavaScript.

Previously, I have been flirting aroung the html and css world a bit for the past few years without much seriousness.

After bumping into this book and picking up a few chapters, i now have a better understanding of not only about html and css, but also the difference between HTML n XHTML, in depth understanding of CSS, and also some badic javascript and some useful famous plugins widely used in today's internet.

This book is a very good book for beginners, like what stated in the title.

It is a very good introduction to making dynamic websites. Totally suitable for even zero internet knowledge fellas.

The sections on HTML and CSS is considered quite in depth, but not so with JavaScript. But that shouldnt be a reason to put you of from reading this book.

The introduction in JavaScript is more than enough to get us started in building a good website. Apart from that, it also introduced a few useful plugins and does point us to the correct direction for doing our own further research if we need to improve our JavaScript skills.

My personal experience, i spent almost a month flipping thru the entire book, and spent the rest of the time of my web programming time asking google. This book is good enough to give u all the knowledge needed in order for you to ask google if u need further reference.

All in all, this book is highly recommanded for those new / semi new to html/css/javascript programming.






Useful Links

• http://code.google.com/edu/submissions/html-css-javascript/