Check I can connect to internet services

I often want to check can I connect to a service running on different machines. There are a number of ways to do this. I typically default to using telnet. This isn’t ideal and overkill when I just want to check a port is open (e.g. no firewall blocking the port). I decided today to look around for alternatives and have come up with the following script:

 This uses netcat if available or bash socket connections if not. You can use the bash function as follows:

    # Download and source the script.
    eval "$(curl https://raw.githubusercontent.com/oisinmulvihill/my-bash-helpers/master/lib/tcptest.sh)"

    # Connections over HTTP
    $ tcptest google.com 80
    Connection ok to 'google.com:80'

    # Connections over HTTPS
    $ tcptest google.com 443
    Connection ok to 'google.com:443'

No try a port google won’t have open:

    $ tcptest google.com 38475
    No one is listening on 'google.com:38475'

This is really only a quick check as the connection could be slow. The tcptest will timeout after 5 seconds. The tcptest function is based on a mashup of three different stackoverflow answers:

On linux you often want to see what processes are running. I like to see a tree of parent and children processes running. Traditionally you could do to see using the “ps” command:

ps fax

1027 ?        Ssl    0:00 rsyslogd
1131 ?        S      0:00 upstart-file-bridge --daemon
1170 tty4     Ss+    0:00 /sbin/getty -8 38400 tty4
1173 tty5     Ss+    0:00 /sbin/getty -8 38400 tty5
1175 ?        Ssl    0:00 /usr/bin/python /usr/bin/salt-minion
1179 tty2     Ss+    0:00 /sbin/getty -8 38400 tty2
1180 tty3     Ss+    0:00 /sbin/getty -8 38400 tty3
1182 tty6     Ss+    0:00 /sbin/getty -8 38400 tty6
1183 ?        Ss     0:00 su - vagrant -c ipython notebook --pyl
1232 ?        Ss     0:00  \_ -su -c ipython notebook --pylab inl
1250 ?        Sl     0:01      \_ /usr/bin/python /usr/bin/ipython
1255 ?        Ss     0:00 acpid -c /etc/acpi/events -s /var/run/a
1257 ?        Ss     0:00 cron
1259 ?        Ss     0:00 atd
1273 ?        S      0:00 dnsmasq -u lxc-dnsmasq --strict-order 
1276 ?        Ss     0:00 nginx: master process /usr/sbin/nginx
1277 ?        S      0:00  \_ nginx: worker process
1278 ?        S      0:00  \_ nginx: worker process
1279 ?        S      0:00  \_ nginx: worker process
1280 ?        S      0:00  \_ nginx: worker process
1335 ?        Sl     0:00 /usr/sbin/VBoxService
1418 ?        Ssl    0:01 /usr/bin/ruby /usr/bin/puppet agent
1447 ?        Sl     0:00 ruby /usr/bin/chef-client -d -P /var/run
1474 tty1     Ss+    0:00 /sbin/getty -8 38400 tty1
1708 ?        Ss     0:00 /usr/sbin/sshd -D
3807 ?        Ss     0:00  \_ sshd: vagrant [priv]
3860 ?        S      0:00      \_ sshd: vagrant@pts/0
3861 pts/0    Ss     0:00          \_ -bash
4106 pts/0    R+     0:00              \_ ps fax

Thats ok however you have to re-run to see it updating. You could use “top” however I recently came across htop. This is very nice and shows the amount of CPU and memory usage “graphically” in the console. The one slight niggle for me are the initial settings which look a bit like:

image

You can configure it to show a tree and hide threads which I prefer:

image

The following are my settings. To keep the settings across machine set ups I copy them to the configuration file $HOME/.config/htop/htoprc

# Beware! This file is rewritten by htop when settings are changed in the interface.
# The parser is also very primitive, and not human-friendly.
fields=0 48 17 18 38 39 40 2 46 47 49 1
sort_key=46
sort_direction=1
hide_threads=1
hide_kernel_threads=1
hide_userland_threads=0
shadow_other_users=0
show_thread_names=0
highlight_base_name=0
highlight_megabytes=1
highlight_threads=0
tree_view=1
header_margin=1
detailed_cpu_time=0
cpu_count_from_zero=0
update_process_names=0
color_scheme=0
delay=15
left_meters=AllCPUs Memory Swap
left_meter_modes=1 1 1
right_meters=Tasks LoadAverage Uptime
right_meter_modes=2 2 2

Sublime Text Magic

image

One of the really handy features of most editors is to be able to search and replace. Many also allow you to use regular expressions to aid.

Sublime text, my current GUI based editor of choice, is no different. I recently remembered how to do “regex search and replace” using matches. I though I should document this so I will remember more quickly next time :)

I need to find all entries like “self.assertEquals(user.count(), 0)” and replace it with “assert user.count() ==  0”. On another line I have “        self.assertEquals(user.dump(), [])” which needs to become “assert user.dump() == []”. Doing this with “normal search and replace” would be impossible.

Enter the Regex magic. In the search and replace enable “Regular Expression” searching (Its the little button image with the “.*” in it). Then enter the expression “self.assertEquals\((.*),(.*)\)”. This tells what I’m looking for. The magic here are the two the (.*) parts. 

In the replace box I type the expression “assert \1 == \2”. The first (.*) can be refer to in the replace as \1,  the second as \2. I can now search and replace a broad range of entries that match the pattern. In this case 43 items I need to replace.

The tech interview is dead! Long live my github code…

I totally agree with this article explaining in detail why the The tech interview is dead!”. I just wish the London job scene would catch on to this. I do maybe 3+ of these a year. What hoops I’m asked to jump through tells me more about the prospective company then they realise.

I got my contract at Albion solely based on my "portfolio" of work on github. I just got an email saying “You start monday…”. I like to call this interviewing while you sleep :)

 

I started doodling while waiting for a program to pass tests. Its nerdy and simplistic view on things. But it got me thinking about how I feel about extremism, hatred and where I live.
London is my home. I’ve been living here for seven years now. It is the most diverse and fascinating places I’ve lived so far. There is always something going on. Be that interesting projects, fantastic food or things to see and do.
The 2005 terror attack on Shepherds Bush station (now called Shepherds Bush Market station), temporarily meant I had to stay at a friends house. I remember going to Starbucks then next morning. I wasn’t able to head home and had to go to work. The staff gave me free coffee and a sandwich. In a small way it showed people pulling together and not letting a such an thing hold us down.
Yesterdays events in Woolwich were really terrible. My heart goes out to the family of the victim. The world has changed a lot since September 2011. I’m no longer shocked that two people can to hold such extreme views, that make them feel butchering another human being is ok.
My concern is that we don’t let this attach divide us, as was no doubt the intention. This kind of thing, in a recession, is what fuels far right groups and their peddling of counter hate. I hope we can pull together and work to rid society of extremism in all forms.
Thats just me two cents.

I started doodling while waiting for a program to pass tests. Its nerdy and simplistic view on things. But it got me thinking about how I feel about extremism, hatred and where I live.

London is my home. I’ve been living here for seven years now. It is the most diverse and fascinating places I’ve lived so far. There is always something going on. Be that interesting projects, fantastic food or things to see and do.

The 2005 terror attack on Shepherds Bush station (now called Shepherds Bush Market station), temporarily meant I had to stay at a friends house. I remember going to Starbucks then next morning. I wasn’t able to head home and had to go to work. The staff gave me free coffee and a sandwich. In a small way it showed people pulling together and not letting a such an thing hold us down.

Yesterdays events in Woolwich were really terrible. My heart goes out to the family of the victim. The world has changed a lot since September 2011. I’m no longer shocked that two people can to hold such extreme views, that make them feel butchering another human being is ok.

My concern is that we don’t let this attach divide us, as was no doubt the intention. This kind of thing, in a recession, is what fuels far right groups and their peddling of counter hate. I hope we can pull together and work to rid society of extremism in all forms.

Thats just me two cents.

Macports and what you need.

I restored my Mac from time machine after installing a new SSD. Sadly my Macports didn’t work. When I tried to using it I got:

sudo port install coreutils

—->  Computing dependencies for coreutilsError: Unable to execute port: can’t read “build.cmd”: Failed to locate ‘make’ in path: ‘/opt/local/bin:/opt/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin’ or at its MacPorts configuration time location, did you move it?

This just meant the Developer tools weren’t installed. Although they did appear to be still there. The fix was straight forward enough. I just reinstalled "Command Line Tools (OS X Lion) for Xcode - March 2013" and I was back in action. 

New home for my blog.

Hello There,

Welcome to the new home for my SourceWeaver blog. I’m hoping to leverage Tumblr and Disqus. Tumblr is straight forward to use so I’m hoping this will let me create articles when I think of them. Rather then the extremely slow process of between contracts.

A helper for python package development

I like breaking Python development up into packages I can roll into eggs. The downside of this approach is typing “python setup.py …” a lot from the command line. To reduce typing I came up with the following Bash function. I use it a lot and would like to share with others. I place it in the public domain.

#!/bin/bash
# # Wrap python setup.py . By default it will # run a python setup.py develop. If there is a BASKET # environment variable set it will use this as the # sole source for egg dependancies. # # The function will first test for the presence of # the setup.py file. If its not present it will exit. # function sd() { if [ "$1" == "" ]; then COMMAND="develop" else COMMAND=$1 fi if [ -e setup.py ]; then if [ -z "$BASKET" ]; then # No basket defined so don't use it as a source python setup.py $COMMAND else # The BASKET if set will need to specify -i/-f. # For example: BASKET="-i http...." # For example: BASKET="-f http...." # if [ $COMMAND == "develop" ]; then # only use basket for develop. It does not work # with other options. python setup.py develop $BASKET else python setup.py $COMMAND fi fi else echo "No setup.py found here: '"`pwd`"'" fi }

You can add this to your ~/.bashrc or just use it from the command line:

source pythonsetup.sh

The ‘sd’ command willl now be available. Any command you can do to “python setup.py” can be used with “sd”.

#Example calls:


# Default (python setup.py develop):
$ sd
running develop
running egg_info
writing lib/boaconstructor.egg-info/PKG-INFO
writing eager_resources to lib/boaconstructor.egg-info/eager_resources.txt
:
etc
:
Installed /Users/omul/Dropbox/src/boaconstructor/lib
Processing dependencies for boaconstructor==0.4.0
Finished processing dependencies for boaconstructor==0.4.0



# Running tests (python setup.py test:
$ sd test
running test
running egg_info
:
etc
:
Test the utils module render which is used by the Template class. ... ok
testValueParsing (boaconstructor.tests.testboaconstructor.BoaConstructor)
Test parsing a dict entries value to recover the reference and ... ok

----------------------------------------------------------------------
Ran 15 tests in 0.010s

OK

New Mac with an old keyboard.

My favorite keyboard

 I’ve started at a new place, hurrah! I’ve got a shiny new Mac Pro to work from.The only slight niggle is that I like to use my keyboard. MacOSX doesn’t get the key-mappings right for this. I’ve managed to get it set up as I like using some know-how and a great program call KeyRemap4MacBook. This posting is more for the next job i need to do this on :)

KeyRemap4MacBook

http://pqrs.org/macosx/keyremap4macbook/

This program remaps keys on a keyboard. It is able to fix the key mappings when OSX refuses to get it right. Start KeyRemap4MacBook program and then navigate as follows from the menu.

My settings (For kinessis keyboard):

Settings -> KeyRemap4MacBook -> “For UK Keyboard users”

Checked:

  • Swap @ and double-quote (“)
  • Backslash (\) to Option_L+3 (#), Shift+Backslash(\) to Tilde (~)
  • Section (§) to Backslash (\)

To swap the left handside command and option key so its in the same “place” as a normal Mac keyboard.

  • Change Command_L Key
  • Command_L to Option_L
  • Change Optional_L Key
  • Option_L to Command_L

Success!

MacOsx reassign Home / End key

I want the Home and End key to go to the start and end of the line. The default behaviour is to got to the start and end of the document. This I find quite annoying.

  • Make the directory ~/Library/KeyBindings/ (mkdir -p ~/Library/KeyBindings/)
  • Edit the file ~/Library/KeyBindings/DefaultKeyBinding.dict (vi ~/Library/KeyBindings/DefaultKeyBinding.dict)
  • Add the following contents and save it.
{
    /* Remap Home / End to be correct :-) */
    "\UF729"  = "moveToBeginningOfLine:";    /* Home */
    "\UF72B"  = "moveToEndOfLine:";            /* End */
    /* Shift + Home: */
    "$\UF729" = "moveToBeginningOfLineAndModifySelection:"; 
    /* Shift + End:  */
    "$\UF72B" = "moveToEndOfLineAndModifySelection:";       
}
  • Restart or restart the program your using.
  • Success.

Python namespace packages

What are namespace packages?

Namespace packages are an extension of the existing Python import system. They allow:

  • Large software projects to be broken up into smaller parts.
  • Grouping functionality under a common project or company name.
  • Delivery of individual modules without needing all parts of a large project.
  • Reuse of package names which may not normally be available.

Why should I use them?

There are two typical use cases which namespace packages help with.

Use Case 1

Lets say a ficticous software company called WidgetCo had come up with their own TCP socket module. Its socket module provides extra abilities not in the standard Python version.

How would WidgetCo name the package? They can’t call it socket as this is used. Typically it might get named widgetcosocket for example. Its a bit long but it works and doesn’t conflict. Now WidgetCo goes on to develop their own web framework, database layer and maybe various business logic modules. These new modules get called widgetcoweb, widgetcodb, widgetcounderpants, widgetcostep2 and widgetcoprofit.

Is there a better way to name the packages? Yes, using namespace packages would allow WidgetCo to define a top level “namespace”. The individual modules could then be delivered under this. For example WidgetCo could delivery their modules under the “wc” namespace. The packages would then be available as:

from wc import socket
from wc import web
from wc import db
:
etc

The wc.socket, wc.web, wc.db would be delivered from eggs called wc-socket, wc-web, wc-db etc.

Use Case 2

This time the same WidgetCo company has been developing from the begining with wc as the main package. Beneath this the other packages socket, web, db, etc live. On the file system there is a single checkout of the company’s source code. After a period of time this code checkout has become quite large. It is now difficult to develop individual parts due to the checkout size and large amount of code. The companies products need some parts developing faster then others.

Is there a way to break the code up into logical packages, which preserve the imports and allows them to be individually developed? Yes, namespace packages allow you to break up a large project into logical parts. Dependancies and version numbers for the new packages can be used more effectively, to build WidgetCo’s products on specific package versions. Logical parts can be tested and developed more easily. This means development can proceed more easily with greater flexibilty. Which in turn increases productivity and the WidgetCo’s bottom line.

How do I use them?

In a previous article I showed how to create a private egg repository and how to make and egg package. I’m going to build on that reusing the setup.py. I’ve created three packages on PyPI for this article:

You can download the source for these and decompress the archive to disk. wc-db and wc-web depend on wc-socket. You could also install the eggs using easy_install and they will be download and installed into your python.

Namespace Package layout

I tend to favour the following file system layout for an egg package. To make a package for the “wc” namespace I create the following folders and files:

<wc package name socket | db | web>
|
+-- lib
| |
| +-- <wc>
| |
| +-- __init__.py : special namespace code goes in this file.
| |
| +-- <socket | db | web>
| |
| +-- package contents files
|
+-- scripts : command line tool(s)
|
+-- setup.py : distutils file configured for the namespace package.

The wc package __init__.py needs to contain the following code:

# See http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages
try:
__import__('pkg_resources').declare_namespace(__name__)
except ImportError:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

No other code should be put in this file as it can cause problems. The setuptools namespace documentation covers this in more detail. The package inside wc web, db or socket can contain any files you would normally put there.

Namspace package setup.py

To make a package you need a setup.py and to make it aware of your namespace package you need to add “namespace_package” keyword to the setup() function e.g.:

setup(
:
namespace_packages = ['wc'],
:
)

The following is a the wc-socket setup.py file I use. The files used for wc-web and wc-db are almost exactly the same save the description and dependance part. If you get the code for these you can see this in more detail on this.

This is demonstrating how to create namespace packages.

This should eggify and in theory upload to pypi without problems.

This is released under the BSD license.

Oisin Mulvihill
2010-05-31

"""
try:
from setuptools import setup, find_packages
except ImportError:
from ez_setup import use_setuptools
use_setuptools()
from setuptools import setup, find_packages

Name='wc-socket'
ProjecUrl=""
Version='1.0.0'
Author='Oisin Mulvihill'
AuthorEmail='oisinmulvihill at gmail dot com'
Maintainer=' Oisin Mulvihill'
Summary='WidgetCo socket: This is a fake socket module use to teach namespace packages .'
License='BSD License'
ShortDescription=Summary
Description=Summary

needed = [
]


# Include everything under package dir. I needed to add a __init__.py
# to each directory inside package. I did this using the following
# handy command:
#
# find lib/wc -type d -exec touch {}//__init__.py \;
#
# If new directories are added then I'll need to rerun this command.
#
EagerResources = [
'wc',
]

ProjectScripts = [
## 'scripts/runweb',
]

PackageData = {
'': ['*.*'],
}

# Make exe versions of the scripts:
EntryPoints = {
}

setup(
# url=ProjecUrl,
name=Name,
zip_safe=False,
version=Version,
author=Author,
author_email=AuthorEmail,
description=ShortDescription,
long_description=Description,
license=License,
scripts=ProjectScripts,
install_requires=needed,
include_package_data=True,
packages=find_packages('lib'),
package_data=PackageData,
package_dir = {'': 'lib'},
eager_resources = EagerResources,
entry_points = EntryPoints,
namespace_packages = ['wc'],
)

Namespace package demonstration

Recover wc-socket, wc-db and wc-web archives. Decompress them and then set the package up in development mode. Alternatively run the command:

(p2)oisin@banter [tmp]> easy_install wc-socket wc-web wc-db
Processing wc-socket
Running setup.py -q bdist_egg --dist-dir /home/oisin/src/CMDev/src/tmp/wc-socket/egg-dist-tmp-bL0ti7
Adding wc-socket 1.0.0 to easy-install.pth file

Installed /home/oisin/src/CMDev/src/tmp/p2/lib/python2.6/site-packages/wc_socket-1.0.0-py2.6.egg
Processing dependencies for wc-socket==1.0.0
Finished processing dependencies for wc-socket==1.0.0
Processing wc-web
Running setup.py -q bdist_egg --dist-dir /home/oisin/src/CMDev/src/tmp/wc-web/egg-dist-tmp-M6uT43
Adding wc-web 1.0.0 to easy-install.pth file
Installing runweb script to /home/oisin/src/CMDev/src/moneyio/tmp/p2/bin
Installing runweb script to /home/oisin/src/CMDev/src/moneyio/tmp/p2/bin

Installed /home/oisin/src/CMDev/src/tmp/p2/lib/python2.6/site-packages/wc_web-1.0.0-py2.6.egg
Processing dependencies for wc-web==1.0.0
Finished processing dependencies for wc-web==1.0.0
Processing wc-db
Running setup.py -q bdist_egg --dist-dir /home/oisin/src/CMDev/src/tmp/wc-db/egg-dist-tmp-yvuSX6
Adding wc-db 1.0.0 to easy-install.pth file
Installing rundb script to /home/oisin/src/CMDev/src/tmp/p2/bin
Installing runweb script to /home/oisin/src/CMDev/src/tmp/p2/bin

Installed /home/oisin/src/CMDev/src/tmp/p2/lib/python2.6/site-packages/wc_db-1.0.0-py2.6.egg
Processing dependencies for wc-db==1.0.0
Finished processing dependencies for wc-db==1.0.0
(p2)oisin@banter [tmp]>

The packages wc-web and wc-db depend on wc-socket. The setup.py of web and db explicity list wc-socket and a particular version string as a dependancy.

The web and db modules also provide basic command line programs runweb and rundb:

(p2)oisin@banter [tmp]> runweb
wc-socket:
----------
>>> wc.web.hello.main: hello world

(p2)oisin@banter [tmp]> rundb
wc-socket:
----------
>>> wc.db.hello.main: hello world

The socket module provides a write method that both rundb and runweb call. This is used to test the namespace package set up. This could be done manually as follows:

(p2)oisin@banter [tmp]> python
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from wc import socket
>>> from wc import web
>>> from wc import db
>>>
>>>
(p2)oisin@banter [tmp]>

Conclusion

Developing namespace packages is quite straight forward. For large scale projects is allows individual parts to developed individually and delivered under a common namespace. For small utilities it may not be worth the effort, however if your project grows its worth keeping in mind. Go forth and claim those name spaces! :)