Index

  • Planet Python now available on Gopher

    Posted on Saturday, December 13th, 2014

    In additional efforts to revive the yet to be dead Gopher protocol. I have created a Python script which I can either run manually or by using a cron job, which aggregates RSS feeds into a Gopher friendly format. This can enable browsers such as Lynx to easily read RSS feeds without requiring a separate RSS reader or by downloading huge amounts of data over HTTP. What more, is that all the articles it aggregates are all archived in Gopherspace, so they will always be available even if the blog hosting it dies. This is the advantage is not using a live feed. This act alone will immensely increase the overall content and information available over the Gopher protocol. Each day, there are more than 10 new Python articles. All these articles will also soon be indexed by the Gopher search server Veronica, adding to it's overall knowledge of information.

    Adding new RSS feeds is as simple as updating a standard Python list, so as a result, I also added other Planet feeds, such as Debian and KDE. Over time, I will add more interesting feeds that Gopher users would like to be-able to read from the comfort of a lightweight text-based browser. I have not yet decided if I will open source the script or not. I'd personally like to clean it up before the public eye sees it.

    For the moment, the script is run as a cron job once daily early in the morning hours, so while you are drinking your coffee and eating breakfast, you can open up Lynx or Overbite on your Android phone to quickly catch up on the latest Python, Debian, and KDE news.

    I will provide both the Gopher link and a link that goes through my own proxy for your convenience:

  • Building a Gopher client in Django

    Posted on Friday, December 12th, 2014

    Have you ever wanted to enable your users to access your Gopher server without the need of them installing a browser plugin, installing a standalone software, or using a third part proxy service? Now you can, by adding this simple Django app to any existing Django project. It doesn't depend on anything by my updated gopherlib.py, and the following app which I will explain here. I have already developed it and deployed it, so I will not be covering every single detail. For the most part, it's a very basic Django app that only uses function-based views to get the job done. So, this app might be good as a starting point for people new to Django, as there's nothing terribly complex going on here. You can see a fully working example here. First let's cover the urls.py:

    from django.conf.urls import patterns, url
    
    urlpatterns = patterns('djgopher.views',
        url(r'^$', 'root_menu', name='gopher_root_menu'),
        url(r'^RemoteDoc$', 'remote_document', name='gopher_remote'),
        url(r'^(?P<selector>.+)$', 'selector', name='gopher_selector'),
    )
    

    This isn't the most elegant way to grab the selector from the URL, but it gets the job done. This may have some security implications behind it, I am hoping that my code which handles this selector can't be easily exploited. Feel free to point out to me any possible security concerns and a way to fix it. Let's move onto the views.py:

    from djgopher.gopherlib import Gopher, GopherMenu, TextFile
    from django.shortcuts import render
    from django.http import HttpResponse, Http404, HttpResponseRedirect
    from PIL import Image
    from StringIO import StringIO
    
    def root_menu(req):
        g = Gopher('gopher.veroneau.net')
        menu = g.get_root_menu()
        return render(req, 'djgopher/menu.html', {'menu':menu.get_data(), 'title':'Root Menu'})
    

    Here is the first function based view, it's only task is to display the initial gopher menu to the user. I did hardcode my own Gopher server here, but for most use cases, you can actually put in 127.0.0.1 to refer to the local Gopher server. Here, we initialize the Gopher class, and obtain the root menu of that server. Then we send it off to a template which we will get into shortly.

    def selector(req, selector):
        if req.META['QUERY_STRING'] != '':
            selector += '?'+req.META['QUERY_STRING']
        gtype = selector[0]
        query = req.POST.get('query', None)
        if gtype == '7' and query is None:
            return render(req, 'djgopher/search.html', {'selector': selector, 'title':selector[1:]})
        g = Gopher('gopher.veroneau.net')
        data = g.get_selector(selector, query)
        if isinstance(data, GopherMenu):
            return render(req, 'djgopher/menu.html', {'menu':data.get_data(), 'title':selector[1:]})
        elif isinstance(data, TextFile):
            return HttpResponse('%s' % data, mimetype='text/plain')
        elif gtype == 'I':
            typ = selector[-3:].lower()
            if typ not in ('png', 'jpg', 'jpeg'):
                typ = Image.open(StringIO(data)).format.lower()
            return HttpResponse(data, mimetype='image/%s' % typ)
        elif gtype == 'g':
            return HttpResponse(data, mimetype='image/gif')
        elif gtype == '9':
            return HttpResponse(data, mimetype='application/octet-stream')
        else:
            return HttpResponse('Unhandled.', mimetype='text/plain')
    

    This function based view handles the actual rendering of the chosen selector. Here we check for a query string and append it to the selector so that the Gopher server can manage it. Then we check if the item type is a search, if so, then we display a special template if no query came in. This template is a basic form that will prompt the user for their search query. Finally, if we're still processing here, we grab our selector from the server and check what type we are handling. Technically, we should check if we support the type before even downloading any data to save bandwidth, but for now this should be fine. I believe I will update it shortly so that the item type is properly checked beforehand. Depending on the item type, we either display a menu, a text file, image, or perform a binary download. We also use the PIL library if the image file doesn't have a valid extension to make sure we serve the correct information to the browser.

    def remote_document(req):
        url = req.GET.get('uri', None)
        if url is None:
            raise Http404
        hostname, path = url.split('/',1)
        host, port = hostname.split(':')
        if path[0] != '0':
            return HttpResponseRedirect('http://gopher.floodgap.com/gopher/gw?gopher://%s:%s/%s' % (host, port, selector))
        g = Gopher(host, int(port))
        data = g.get_selector(path)
        if isinstance(data, TextFile):
            return HttpResponse('%s' % data, mimetype='text/plain')
        else:
            return HttpResponseRedirect('http://gopher.floodgap.com/gopher/gw?gopher://%s:%s/%s' % (host, port, selector))
    

    This final function based view is to enable users to retrieve remote documents, such as this one here. I only want to enable text document linking, not anything else to limit bandwidth and overall complexity. There are other clients and proxies out there that can easily grab resources from any Gopher server, so it's best to leverage those services for this.

    The Templates

    The templates are pretty dead simple, as this is a very basic client application, nothing too fancy is required to display Gopher content. Here is our base.html template:

    <html>
      <head><title>{{title}} | Django Gopher Client</title></head>
      <body>
        {{title}} | Django Gopher Client<hr/>
        {% block content %}{% endblock %}
      </body>
    </html>
    

    See? Nothing too complex, not even a stylesheet, although feel free to skin your own client all you want. Here is the template which renders a Gopher menu for us in HTML:

    {% extends "djgopher/base.html" %}
    {% load gopher %}
    
    {% block content %}
    <pre>{% for item in menu %}{% if item.type == 'i' or item.type == '3' %}{{item.type|icon:STATIC_URL}}{{item.name}}
    {% else %}<a href="{{item|selector}}">{{item.type|icon:STATIC_URL}}{{item.name}}</a>
    {% endif %}{% endfor %}</pre>
    {% endblock %}
    

    We are using pre tags here to ensure the content in the menu is rendered correctly. Lots of Gopherholes, including my own use ASCII art in the menus. Gopherpedia is another example of this. This template uses some custom template filters to enable us to dynamically place in URLs and icons using Python code. Like they say, the template is not for logic! Our final template is the search.html template:

    {% extends "djgopher/base.html" %}
    
    {% block content %}
    <form action="{% url gopher_selector selector %}" method="post">{% csrf_token %}
    Query: <input type="text" name="query"/><input type="submit" value="Go"/>
    </form>
    {% endblock %}
    

    The search template is pretty normal, just a basic HTML post form. Now, finally here is the template tag library to render those filters:

    from django import template
    from django.core.urlresolvers import reverse
    from django.utils.safestring import mark_safe
    
    register = template.Library()
    
    ICON_TYPES = {
        '0': 'text.png',
        '1': 'dir.png',
        'h': 'html.png',
        '9': 'binary.png',
        'I': 'image.png',
        'g': 'image.png',
        '7': 'search.png',
        '3': 'error.png',
        '8': 'telnet.png',
    }
    
    @register.filter
    def selector(value):
        if value['selector'][:5] == 'hURL:':
            return value['selector'][5:]
        if value['host'] != 'gopher.veroneau.net':
            uri = '%s:%s/%s' % (value['host'], value['port'], value['selector'])
            if value['type'] == '0':
                return '%s?uri=%s' % (reverse('gopher_remote'), uri)
            elif value['type'] == '8':
                return 'telnet://%s:%s/' % (value['host'], value['port'])
            else:
                return 'http://gopher.floodgap.com/gopher/gw?gopher://%s' % uri
        return reverse('gopher_selector', args=[value['selector']])
    
    @register.filter
    def icon(value, static_url):
        try:
            return mark_safe('<img src="%sdjgopher/%s"/>' % (static_url, ICON_TYPES[value]))
        except KeyError:
            return ''
    

    The first dictionary you see there is to make selecting the icon like a case-select, which is much more efficient than a ton of if statements. The first filter here, selector is used to resolve a menu item into a URL that WWW browsers can understand. Here, we check various attributes based on the Gopher standard and write an HTTP compatible URL link the user can click on. Anything from the supported Gopher server is handled internally by our selector view. Anything that has an external host is handled according to it's type. This makes it so that all Gopher menu items work regardless if the local client supports remote servers. Finally, we have the simple icon filter that does nothing more by write out a proper IMG tag with the icon for the menu item.

    And there you have it, a fully functional Gopher client all done from within a Django app! So now you have no excuse not to run your own Gopher server. For some interesting places to visit on Gopher, check out my Bookmarks.

    As mentioned in my previous post about Gopher, I will soon be making all my Python Diary content available in Gopherspace, it will contain both full HTML versions and text versions. So, if you ever wanted to try browsing a less distracting side of the Internet where advertisers and trackers can't find you, now is a great time! Over time, I am planning on developing many gateway applications for the Internets most popular online locations, such as Twitter, and various news outlets. I think I'd personally like to add RSS feeds to my Gopherspace to enable the easy reading of many popular Python blogs from Planet Python. If your on Android, be sure to check out the updated Gopher client, which can really save you on your mobile bandwidth!

  • Developing an XBMC add-on in Python

    Posted on Thursday, December 11th, 2014

    XBMC is one awesome piece of media center software that runs on all popular platforms. What's more, is that it's entirely programmable via Python. I have released a couple XBMC related Python tools and videos earlier this year, and now I wanted to dive into add-on development. The add-on I will show you today takes advantage of my gopherlib.py module. This add-on will soon be submitted to the Official XBMC Addon repository in the coming weeks for all to enjoy. Here is how the add-on looks when it's running in XBMC:

    As I mentioned in my last post about Gopher, XBMC follows the Gopher protocol pretty closely, so it was pretty easy to create an XBMC Gopher client. Here is the code which I will explain afterwords:

    import sys, xbmc, xbmcgui, xbmcplugin, urlparse, os
    from gopherlib import Gopher, GopherMenu, TextFile
    
    BASE_URL = sys.argv[0]
    addon_handle = int(sys.argv[1])
    
    xbmc.log('HANDLE: %s' % addon_handle, xbmc.LOGNOTICE)
    
    def add_directory(title, url):
        li = xbmcgui.ListItem(title, iconImage='DefaultFolder.png')
        xbmcplugin.addDirectoryItem(handle=addon_handle, url=url,
                                    listitem=li, isFolder=True)
    
    def add_text_file(title, url):
        li = xbmcgui.ListItem(title, iconImage='DefaultVideo.png')
        xbmcplugin.addDirectoryItem(handle=addon_handle, url=url,
                                    listitem=li)
    
    def add_image_file(title, url):
        li = xbmcgui.ListItem(title, iconImage='DefaultPicture.png', path=url)
        #li.setProperty('isPlayable', 'true')
        xbmcplugin.addDirectoryItem(handle=addon_handle, url=url,
                                    listitem=li)
    
    def add_info(info):
        li = xbmcgui.ListItem(info, iconImage='DefaultVideo.png')
        xbmcplugin.addDirectoryItem(handle=addon_handle, url='/', listitem=li)
    
    def text_viewer(title, text):
        WINDOW = 10147 # Text Viewer window in XBMC
        TITLEBAR = 1
        CONTENTS = 5
        xbmc.executebuiltin("ActivateWindow(%d)" % WINDOW)
        w = xbmcgui.Window(WINDOW)
        xbmc.sleep(500)
        w.getControl(TITLEBAR).setLabel(title)
        w.getControl(CONTENTS).setText(text)
    
    def image_viewer(img):
        w = xbmcgui.Window()
        w.addControl(xbmcgui.ControlImage(0,0,w.getWidth(), w.getHeight(), img, 2))
        w.doModal()
        del w
    
    xbmc.log('BASE URL: %s' % BASE_URL, xbmc.LOGNOTICE)
    xbmc.log('Arguments: %s' % sys.argv[2], xbmc.LOGNOTICE)
    
    scheme, remainder = BASE_URL.split(':', 1)
    parts = urlparse.urlparse(remainder)
    netloc, path, query = (parts[1], parts[2], parts[4])
    args = urlparse.parse_qs(query)
    
    query = args.get('query', None)
    
    xbmc.log('Path: %s' % path, xbmc.LOGNOTICE)
    
    if path == '/':
        path = '/gopher.floodgap.com:70/1'
    
    hostname, selector = path[1:].split('/', 1)
    host, port = hostname.split(':')
    
    g = Gopher(host, int(port))
    
    if selector[0] == '7' and query is None:
        keyboard = xbmc.Keyboard('', 'Input Required', False)
        keyboard.doModal()
        if keyboard.isConfirmed():
            query = keyboard.getText()
    
    data = g.get_selector(selector+sys.argv[2], query)
    
    if isinstance(data, GopherMenu):
        for item in data.get_data():
            url = 'plugin://plugin.picture.gopher/%s:%s/%s' % (item['host'], item['port'], item['selector'])
            if item['type'] == '0':
                add_text_file(item['name'], url)
            elif item['type'] == '1':
                add_directory(item['name'], url)
            elif item['type'] == 'i':
                add_info(item['name'])
            elif item['type'] == '7':
                add_directory(item['name'], url)
            elif item['type'] == 'I':
                add_image_file(item['name'], url)
        xbmcplugin.endOfDirectory(addon_handle)
    elif isinstance(data, TextFile):
        text_viewer(selector, '%s' % data)
    elif selector[0] == 'I':
        filename = selector.split('/')[-1]
        temp = xbmc.translatePath('special://temp/')
        open(temp+filename,'w').write(data)
        image_viewer(temp+filename)
        os.unlink(temp+filename)
    

    This code should be fairly easy to follow, as it basically goes from top to bottom, with a function call here and there. The first 4 functions are for generating the correct directory entries within the XBMC GUI. After, you will see 2 specialized functions used to display textual data, and another to display an image. I was unable to find a more elegant way to display resolved image URLs in XBMC. I tried using many different XBMC functions, but xbmcplugin.setResolvedUrl just wouldn't work with image files... When I implement video and audio support in the future, I will try it again. I may need to see if I can create a new VFS module for XBMC, so that the gopher:// protocol is natively supported for streaming.

    The next section here takes the arguments from XBMC and turns them into a valid Gopher URL which we can then send to my gopherlib module. Once that is out of the way, we can instantiate our Gopher class. It is here that we check if the selector is a search selector and bring up the on-screen keyboard for the user if needed. Finally we get our selector and the resulting data object.

    The final if structure here is used to either display an XBMC directory, a text file, or an image to the user.

    You can view many more screenshots and even download the prepackaged add-on in ZIP format from my Gopher server or use the Gopher proxy.

    On a side note, I also updated the Android Gopher client, since it's licensed under the BSD license, the source was available. The update now brings a more modern look and feel on newer phones and tablets, and now included a working address bar. You can grab the APK from my Gopher or Gopher proxy and side load today!

    One added benefit coming soon to Gopher users, will be the ability to browse and search all of the content on Python Diary from the comfort of Gopher. I will be developing a basic Gopher application which will be-able to access the current Python Diary database and query it. So, Python Diary will soon be available on XBMC I guess. So keep a look out for that update when it arrives.

  • Remember the Gopher protocol?

    Posted on Wednesday, December 10th, 2014

    Depending on how old you are and if you were on the Internet before the World Wide Web became the norm, you may not have heard of a rather awesome Internet protocol called Gopher. I, unfortunately never had the chance to experience Gopher in the day it ruled the information highway(at this point in time the highway wasn't super). Late last year I finally managed to into gopher and experience it for what it is. It is a rather awesome system, very easy to navigate, and is basically pick-up and play. It works using a very simple menu system concept, and every gopherhole as they are called, works the exact same. It is sort of similar to say the FTP protocol, but more descriptive and can actually run server-side scripts to perform database look-ups and such. In fact, the main search engine, which is basically the Google of Gopherspace called Veronica is still active and can search of the current Gopher spaces and any archived servers from days past.

    The reason I am bringing this up on my Python Blog, is that there are some new Gopher servers actually written in Python, and Python can obviously be used to create server-side scripts. However, this isn't the main reason why I bring this up. I bring this up on my Python blog because of a little program called XBMC. If anybody reading this recalls Gopher or has used gopher recently, you may notice a similarity to XBMC apps, where all XBMC apps are all consistent and look and function the exact same way. Each XBMC app has a root menu, and then many virtual sub directories generated by the XBMC VFS API in the apps code itself. Then, depending on what type of app you are using let it be a picture app, video app, or music app, the final content type to load will be of that type. This last tidbit is the least similar to Gopher, as gopher supports mixed content, where you can have a directory with images, music, and video files. I believe that XBMC would be a great place to have a gopher client on, since a gopher server can also act as a proxy to WWW content, it would be perfect. If you have ever seen Gopherpedia, then you will understand what I mean.

    Python used to support gopher in the standard library, but support has since been removed. I found the old standard library module and completely updated it to be more modern and along the lines of how current Python standard library modules function, and it's now class based. If your interested in checking out this module, you can obtain a copy on my Gopherhole, if your browser no longer has native gopher support you can use a gopher proxy server or configure Squid to proxy it for you. Here are both the Gopher and Gopher Proxy links. Originally, I was going to use these links to develop a more take on a Gopher client, but now I am thinking of creating an XBMC Gopher client, as I think that would be a really fun project. If you want to seamlessly explore Gopherspace, you should install the Firefox add-on or use Lynx. I once had a Lynx proxy running on my server, where you would be-able to use Lynx directly from within any HTML5 compatible web browser. I am either thinking of bringing this back or running a basic Gopher-WWW proxy, although Floodgap does have a really nice Gopher proxy server. If you want to check out my Gopherhole and see what content I have there, it can be accessed using Gopher or the Gopher Proxy. I have plenty of interesting and fun content available there.

  • Building a CARDIAC Assembler in Python

    Posted on Sunday, October 19th, 2014

    In my last article I gave a fully working example of a CARDIAC CPU simulator in Python. This article will continue down the path of creating a build toolchain for the Cardiac. Due to how the Cardiac accepts data from the outside world, namely in the form of a deck of cards or plain numbers, developing an assembler was a bit more difficult than a traditional bytecode assembler. At least when I build a binary assembler, I assemble the bytecodes in-memory. This ensures that all memory pointers are correctly pointing to the proper memory addresses, and also eliminates errors due to how information is stored. This is similar to how the MS-DOS DEBUG.EXE worked to build .COM files. It was simple and yet very effective. This tends to be the model I normally follow, as it makes debugging the virtual machine very easy. This is how I built up my simple-cpu project, and now I am in the process of enabling dynamic memory relocation to better enable the loading binary files. I also recently added a full memory map interface to it as well, so now memory mapped I/O is possible, and I plan on using that to allow my binaries to directly draw onto an SDL surface. Anyways, onward to the Cardiac Assembler, as that is the real reason your here today!

    As of this post, you can view all the source code for this post and the last one on my Python Experiments Bitbucket repo.

    The assembler is going to use just the standard Python library modules, so no fancy GNU Bison or YACC here. First, let's begin by defining what our language will look like, it's overall syntax. Here is deck1.txt turned into an Assembly program:

    # This deck1.txt converted over to ASM format.
    # Lines which start with a hash are comments.
    
    # Here we declare our variables in memory for use.
    var count = 0
    var i = 9
    
    # This is needed to add the special Cardiac bootstrap code.
    bootstrap
    
    # Program begins here.
    CLA
    STO $count
    label loop
    CLA $i
    TAC $endloop
    OUT $count
    CLA $count
    ADD
    STO $count
    CLA $i
    SUB
    STO $i
    JMP $loop
    label endloop
    HRS
    
    # End of program.
    end
    

    To make developing programs for the CARDIAC easier, the assembler is going to introduce the concept of variable storage, and labels. The first few lines here that begin with var are basically setting aside memory addresses to store these variables in question. The bootstrap statement is more of a Macro as it just writes out a basic bootstrap loader for use on CARDIAC. There is also an alternative bootloader, which uses a second stage boot program to load in your code. This has the advantage of being able to load in very large programs without needing a very large deck. I'd recommend playing around with both options when assembling your programs. The bootloader was custom built code made by me, here is an assembly listing of it:

    INP 2
    JMP 0
    
    INP 89
    INP 1
    INP 90
    CLA 89
    INP 91
    ADD
    INP 92
    STO 89
    INP 93
    CLA 98
    INP 94
    SUB
    INP 95
    STO 98
    INP 96
    TAC 3
    INP 97
    JMP 89
    INP 98
    INP 13
    INP 2
    JMP 89
    

    The bootloader is passed in the size of the deck to load, and loops over each card to load in your program, this means that your input program will no longer need to have lots of address markings. It can also be used to load in other data into a specific memory address. I am planning on updating the bootloader code to also act like a subroutine, so you could call it to load additional data into memory. Okay, so onto the actual Assembler now.

    from cmd import Cmd
    from cStringIO import StringIO
    import shlex, sys
    
    class AsmError(Exception):
        pass
    
    class Assembler(Cmd):
        """
        I am sure this could be done better with say GNU Bison or Yacc,
        but that's more complicated than needed for a simple assembler.
        """
        op_map = {
            'inp': 0,
            'cla': 1,
            'add': 2,
            'tac': 3,
            'sft': 4,
            'out': 5,
            'sto': 6,
            'sub': 7,
            'jmp': 8,
            'hrs': 9,
        }
        padding = '00'
        def configure(self):
            self.start = self.addr = None
            self.pc = 0 #: Allows us to keep track of the program pointer.
            self.var_map = {} #: This is used to keep track of variables.
            self.labels = {} #: Stores the label names, and where they point to.
            self.buffer = StringIO() #: This is our buffer where we will store the CARDIAC deck
            self.size = 0
        def emptyline(self):
            """ This is requried due to how the Python Cmd module works... """
            pass
        def unknown_command(self, line):
            self.stdout.write('*** Unknown syntax: %s\n'%line)
        @property
        def ptr(self):
            """ This will always give the proper pointer in the deck. """
            if self.addr:
                return self.addr
            return self.pc
        def write_cards(self, *card_list):
            """ Helper fuction to make life easier. """
            for card in card_list:
                self.buffer.write('%s\n' % card)
            self.pc += len(card_list) #: Increment the program pointer.
        def write_addr(self):
            """ This method will only write out the address if we're in that mode. """
            if self.addr:
                self.write_cards('0%s' % self.addr)
                self.addr +=1
            self.size +=1
    

    The first thing to notice here, is that the assembler is a subclass of Cmd, which is a command line parser. I choose to use a command-line parser and it makes things easier and can also be used interactively if I need to debug or later want to enable that feature. The first thing we declare in this class, is a dictionary called op_map, which maps our op codes to actual byte codes. This makes it efficient to write the output and also add new opcodes in the future. The first method here, configure() could have well been an override of __init__(), but I choose to just have it a separate method that I can call manually. The next 2 methods are specific to the command parser, and tell it how to handle empty lines and invalid commands. The ptr property added here is to make it transparent to the code the proper location in the deck at runtime. This is required due to how the Cardiac bootstraps code. You cannot just load in RAW code, you need to bootstrap your deck of cards, and the deck itself needs to keep track at which memory locations each of it's operations are being placed in. Since I offer 2 modes of assembly here, and also support RAW assembly, there are different ways the exact memory pointer is obtained for runtime variables and jumps. The next 2 methods are for writing specific card types, one being a normal list of cards, while the other being an address card. The address cards are only used if you assemble using bootstrap, the original Cardiac bootstrap program.

    Since this is a command processor Python subclass, next we are going to put in some commands that our source code can use to perform specific tasks:

        def do_exit(self, args):
            return True
        def do_bootstrap(self, args):
            """ Places some basic bootstrap code in. """
            self.addr = 10
            self.start = self.addr #: Updates all required address variables.
            self.write_cards('002', '800')
            self.pc = self.start
        def do_bootloader(self, args):
            """ Places a Cardiac compatible bootloader in. """
            self.write_cards('002', '800') 
            addr = 89 #: This is the start address of the bootloader code.
            for card in ('001', '189', '200', '689', '198', '700', '698', '301', '889', 'SIZE'):
                self.write_cards('0%s' % addr, card)
                addr+=1
            self.write_cards('002', '889')
            self.pc = 1
        def do_var(self, args):
            """ Creates a named variable reference in memory, a simple pointer. """
            s = shlex.split(args)
            if len(s) != 3 or s[1] != '=':
                raise AsmError('Incorrect format of the "var" statement.')
            if s[0] in self.var_map:
                raise AsmError('Variable has been declared twice!')
            value = int(s[2])
            self.var_map[s[0]] = '000'+str(value)
        def do_label(self, args):
            """ Creates a named label reference in memory, a simple pointer. """
            if args == '':
                raise AsmError('Incorrect format of the "label" statement.')
            if args in self.labels:
                raise AsmError('Label has been declared twice!')
            ptr = '00'+str(self.ptr)
            self.labels[args] = ptr[-2:]
    

    Okay, there's a fair amount to explain with this code. Firstly, the two first methods here, bootstrap and bootloader are obviously Cardiac specific, and it's doubtful that a homemade Assembler will even need such things. The bootstrap() method sets up some variables required for that mode of assembly, namely the address variables used to keep track which memory location we are at. We also write 2 basic cards, the bootstrap that Cardiac needs to load your program in a real or simulated Cardiac. The next method, bootloader is a bit more complex, it writes the same initial bootstrap required by all cardiac programs, but also writes out a second stage bootloader program which is responsible for loading in your program. Once I get to the guide about creating a compiler, the compiler will automatically choose which mode of assembly will be used based on the program being compiled.

    The next two very important methods, do_var, and do_label are responsible for controlling the variable and label system of the assembler. They make it easy to set a side a memory address for use as a variable, and to place in a label within your code so that you may JMP to it. If these didn't exist, you would be forced to keep track of memory addresses yourself, and if you update your program, you will also need to update the memory addresses... So, variables and labels keep track of all that for you, and the memory addresses are generated automatically during assembly time. Both of these methods should be fairly easy to understand what they are doing. They are storing the variable and label name into a hash along with the needed metadata.

        def default(self, line):
            """ This method is the actual lexical parser for the assembly. """
            if line.startswith('#'):
                return
            s = shlex.split(line)
            op, arg = s[0].lower(), '00'
            if len(s) == 2:
                if s[1].startswith('$'):
                    arg = '*%s' % s[1][1:]
                else:
                    arg = self.padding+s[1]
                    arg = arg[-2:]
            if op in self.op_map:
                self.write_addr()
                self.write_cards('%s%s' % (self.op_map[op], arg))
            else:
                self.unknown_command(line)
        def do_end(self, args):
            """ Finalizes your code. """
            for var in self.var_map:
                ptr = self.padding+str(self.ptr)
                self.write_addr()
                self.write_cards(self.var_map[var][-3:])
                self.labels[var] = ptr[-2:]
            if self.start:
                self.write_cards('002', '8%s' % self.start)
            self.buffer.seek(0)
            buf = StringIO()
            for card in self.buffer.readlines():
                if card[1] == '*': #: We have a label.
                    card = '%s%s\n' % (card[0], self.labels[card[2:-1]])
                elif card[:4] == 'SIZE':
                    card = '000'+str(self.size-1)
                    card = '%s\n' % card[-3:]
                buf.write(card)
            buf.seek(0)
            print ''.join(buf.readlines())
            return True
    

    Now onto the bread and butter of the assembler, these two methods here do most of the heavy lifting during the assembly process. First lets talk about default(), this method is called by the command-line parser when it cannot find the command the end-user attempted to call. Makes sense? So, here is where we place the code to check if any of those lines are assembly instructions and process them accordingly. First we filter out any comments, any line that starts with a hash is a commented line, simple as that. Next, we use the help of shlex module to parse the actual line. This module splits all the tokens into separate array elements, it even works with quotation marks, so take this line for example: echo "Example echo command" will result in only 2 array elements, the first being echo, and the second being the string. In the next line we set some needed defaults for Cardiac cards, so if there is a missing argument it will still work fine on the Cardiac. It is here that we also check the arguments to see if we are using any variables or labels and mark them accordingly in the buffer for the second pass.

    The last and final method here do_end is when you place end in your code at the very end. This method is essentially the last pass, and which generates the final assembled code. It gathers together all the variables, and generates a table of pointers to them. Then we write out the required cards to run the program the Cardiac just loaded, if we are using the original bootstrap. The bootloader doesn't use this, the bootloader manages that during runtime. Then we seek the buffer to the beginning and being our last pass through the assembled data. We check for any memory pointer references and fill in the holes as needed in the output code. If using the bootloader, it needs to be passed the size of the program code to be loaded, and this is where we fill that in so that the bootloader can locate the variable. Then finally, we display the assembled Cardiac code to standard output.

    You can find all the source for both the simulator and this assembler on my Python Experiments BitBucket page, along with an example deck that counts to 10 in both Assembly and Cardiac ready format. The code assembled by this assembler should run without error on either a real cardiac or a simulated one. You can find a simulator made in JavaScript by a Drexel University professor by following the links through this blog page here: Developing Upwards: CARDIAC: The Cardboard Computer.

    Hopefully this guide/article was helpful and inspiring to you, and I hope to have the final part of this guide/toolchain ready for consumption soon, the guide on how to build a Compiler for the Cardiac computer.

  • Building a CPU simulator in Python

    Posted on Wednesday, October 15th, 2014

    You can find the entire source listing from this article, plus more interesting code on my Python Experiments BitBucket page. Plus, the source code there will evolve and grow.

    Early this morning in my Planet Python feed, I read this really interesting article called Developing Upwards: CARDIAC: The Cardboard Computer, which is about this cardboard computer called the Cardiac. As some of my followers and readers might know, I have another project that I have been working on and evolving for the past months called simple-cpu, which I have released the source code for. I really should give that project a proper license so that others might feel interested in using it in their own projects... Anyways, hopefully I'll have that done by the time this publishes.

    After reading that article and the page it links to, I felt a bit inspired and decided to write my own simulator for it since I already have experience in writing bytecode engines. I plan on following this article up with another on how to write a Assembler for it, then shortly after how to write a compiler. So basically, throughout these articles, you will learn how to create a build toolchain for the Cardiac in Python. The simple-cpu I wrote already has a fully working Assembler, and in the game I am putting it in, already has the first steps of a working compiler. I choose to also use Cardiac as a demonstration machine due to it's pure simplicity. There are no really complex mnemonics here, each opcode only accepts a single parameter, so it's perfect as a learning tool. Furthermore, all data parameters are known to be the same, there is no need to detect if the program wants a register, a literal, or a memory address. In fact, there is really only one register, the accumulator. So lets get started. We will be making this class-based, so that the scope is contained and so that you can easily sub-class it to say add new opcodes if you want to feel adventurous. First, we will focus on the Initialization routines. This CPU is very simple, so we only need to initialize the following: CPU registers, opcodes, memory space, card reader/input, and the printer/tty/output.

    class Cardiac(object):
        """
        This class is the cardiac "CPU".
        """
        def __init__(self):
            self.init_cpu()
            self.reset()
            self.init_mem()
            self.init_reader()
            self.init_output()
        def reset(self):
            """
            This method resets the CPU's registers to their defaults.
            """
            self.pc = 0 #: Program Counter
            self.ir = 0 #: Instruction Register
            self.acc = 0 #: Accumulator
            self.running = False #: Are we running?
        def init_cpu(self):
            """
            This fancy method will automatically build a list of our opcodes into a hash.
            This enables us to build a typical case/select system in Python and also keeps
            things more DRY.  We could have also used the getattr during the process()
            method before, and wrapped it around a try/except block, but that looks
            a bit messy.  This keeps things clean and simple with a nice one-to-one
            call-map. 
            """
            self.__opcodes = {}
            classes = [self.__class__] #: This holds all the classes and base classes.
            while classes:
                cls = classes.pop() # Pop the classes stack and being
                if cls.__bases__: # Does this class have any base classes?
                    classes = classes + list(cls.__bases__)
                for name in dir(cls): # Lets iterate through the names.
                    if name[:7] == 'opcode_': # We only want opcodes here.
                        try:
                            opcode = int(name[7:])
                        except ValueError:
                            raise NameError('Opcodes must be numeric, invalid opcode: %s' % name[7:])
                        self.__opcodes.update({opcode:getattr(self, 'opcode_%s' % opcode)})
        def init_mem(self):
            """
            This method resets the Cardiac's memory space to all blank strings, as per Cardiac specs.
            """
            self.mem = ['' for i in range(0,100)]
            self.mem[0] = '001' #: The Cardiac bootstrap operation.
        def init_reader(self):
            """
            This method initializes the input reader.
            """
            self.reader = [] #: This variable can be accessed after initializing the class to provide input data.
        def init_output(self):
            """
            This method initializes the output deck/paper/printer/teletype/etc...
            """
            self.output = []
    

    Hopefully I left enough comments in the code above for you to understand exactly what's going on here. You might notice here a key difference from my simple-cpu project, is the method which opcodes are handled. I actually plan on incorporating this new way of detecting opcodes into that project in the near future. This method makes it easier for other developers of the library to extend it for their own requirements. As mentioned before, that project has been evolving as I started to understand more about how things should be done. In fact, taking on a CPU simulator project is a really incredible learning experience all on it's own. If you are truly a computer scientist, you should understand the under workings of a CPU, and how each opcode is processed at it's lowest level. Plus, developing and seeing a custom built CPU simulator made by your own imagination is a very gratifying experience. It's almost like giving birth, as this machine was entirely built by your mind alone, and seeing it work, is something magical.

    In the next part of this class, we will focus on the utility functions that we may need to call and use multiple times, these methods can also be overridden in subclasses to alter how the CPU actually functions:

        def read_deck(self, fname):
            """
            This method will read a list of instructions into the reader.
            """
            self.reader = [s.rstrip('\n') for s in open(fname, 'r').readlines()]
            self.reader.reverse()
        def fetch(self):
            """
            This method retrieves an instruction from memory address pointed to by the program pointer.
            Then we increment the program pointer.
            """
            self.ir = int(self.mem[self.pc])
            self.pc +=1
        def get_memint(self, data):
            """
            Since our memory storage is *string* based, like the real Cardiac, we need
            a reusable function to grab a integer from memory.  This method could be
            overridden if say a new memory type was implemented, say an mmap one.
            """
            return int(self.mem[data])
        def pad(self, data, length=3):
            """
            This function pads either an integer or a number in string format with
            zeros.  This is needed to replicate the exact behavior of the Cardiac.
            """
            orig = int(data)
            padding = '0'*length
            data = '%s%s' % (padding, abs(data))
            if orig < 0:
                return '-'+data[-length:]
            return data[-length:]
    

    These are the various utility functions, all of which might be overridden in a subclass. Later in this article, I will also provide an alternate source output which displays how this simulator can be built using Python Mixin classes, to make things even more pluggable. Finally, we find ourselves at the final part of code to get this simulator running, the actual processing methods:

        def process(self):
            """
            Process a single opcode from the current program counter.  This is
            normally called from the running loop, but can also be called
            manually to provide a "step-by-step" debugging interface, or
            to slow down execution using time.sleep().  This is the method
            that will also need to used if you build a TK/GTK/Qt/curses frontend
            to control execution in another thread of operation.
            """
            self.fetch()
            opcode, data = int(math.floor(self.ir / 100)), self.ir % 100
            self.__opcodes[opcode](data)
        def opcode_0(self, data):
            """ INPUT Operation """
            self.mem[data] = self.reader.pop()
        def opcode_1(self, data):
            """ Clear and Add Operation """
            self.acc = self.get_memint(data)
        def opcode_2(self, data):
            """ Add Operation """
            self.acc += self.get_memint(data)
        def opcode_3(self, data):
            """ Test Accumulator contents Operation """
            if self.acc < 0:
                self.pc = data
        def opcode_4(self, data):
            """ Shift operation """
            x,y = int(math.floor(data / 10)), int(data % 10)
            for i in range(0,x):
                self.acc = (self.acc * 10) % 10000
            for i in range(0,y):
                self.acc = int(math.floor(self.acc / 10))
        def opcode_5(self, data):
            """ Output operation """
            self.output.append(self.mem[data])
        def opcode_6(self, data):
            """ Store operation """
            self.mem[data] = self.pad(self.acc)
        def opcode_7(self, data):
            """ Subtract Operation """
            self.acc -= self.get_memint(data)
        def opcode_8(self, data):
            """ Unconditional Jump operation """
            self.pc = data
        def opcode_9(self, data):
            """ Halt and Reset operation """
            self.reset()
        def run(self, pc=None):
            """ Runs code in memory until halt/reset opcode. """
            if pc:
                self.pc = pc
            self.running = True
            while self.running:
                self.process()
            print "Output:\n%s" % '\n'.join(self.output)
            self.init_output()
    
    if __name__ == '__main__':
        c = Cardiac()
        c.read_deck('deck1.txt')
        try:
            c.run()
        except:
            print "IR: %s\nPC: %s\nOutput: %s\n" % (c.ir, c.pc, '\n'.join(c.output))
            raise
    

    As mentioned above, I will now refactor the code into Mixin classes and provide a full source output of that:

    class Memory(object):
        """
        This class controls the virtual memory space of the simulator.
        """
        def init_mem(self):
            """
            This method resets the Cardiac's memory space to all blank strings, as per Cardiac specs.
            """
            self.mem = ['' for i in range(0,100)]
            self.mem[0] = '001' #: The Cardiac bootstrap operation.
        def get_memint(self, data):
            """
            Since our memory storage is *string* based, like the real Cardiac, we need
            a reusable function to grab a integer from memory.  This method could be
            overridden if say a new memory type was implemented, say an mmap one.
            """
            return int(self.mem[data])
        def pad(self, data, length=3):
            """
            This function pads either an integer or a number in string format with
            zeros.  This is needed to replicate the exact behavior of the Cardiac.
            """
            orig = int(data)
            padding = '0'*length
            data = '%s%s' % (padding, abs(data))
            if orig < 0:
                return '-'+data[-length:]
            return data[-length:]
    
    class IO(object):
        """
        This class controls the virtual I/O of the simulator.
        To enable alternate methods of input and output, swap this.
        """
        def init_reader(self):
            """
            This method initializes the input reader.
            """
            self.reader = [] #: This variable can be accessed after initializing the class to provide input data.
        def init_output(self):
            """
            This method initializes the output deck/paper/printer/teletype/etc...
            """
            self.output = []
        def read_deck(self, fname):
            """
            This method will read a list of instructions into the reader.
            """
            self.reader = [s.rstrip('\n') for s in open(fname, 'r').readlines()]
            self.reader.reverse()
        def format_output(self):
            """
            This method is to format the output of this virtual IO device.
            """
            return '\n'.join(self.output)
        def get_input(self):
            """
            This method is used to get input from this IO device, this could say
            be replaced with raw_input() to manually enter in data.
            """
            try:
                return self.reader.pop()
            except IndexError:
                # Fall back to raw_input() in the case of EOF on the reader.
                return raw_input('INP: ')[:3]
        def stdout(self, data):
            self.output.append(data)
    
    class CPU(object):
        """
        This class is the cardiac "CPU".
        """
        def __init__(self):
            self.init_cpu()
            self.reset()
            try:
                self.init_mem()
            except AttributeError:
                raise NotImplementedError('You need to Mixin a memory-enabled class.')
            try:
                self.init_reader()
                self.init_output()
            except AttributeError:
                raise NotImplementedError('You need to Mixin a IO-enabled class.')
        def reset(self):
            """
            This method resets the CPU's registers to their defaults.
            """
            self.pc = 0 #: Program Counter
            self.ir = 0 #: Instruction Register
            self.acc = 0 #: Accumulator
            self.running = False #: Are we running?
        def init_cpu(self):
            """
            This fancy method will automatically build a list of our opcodes into a hash.
            This enables us to build a typical case/select system in Python and also keeps
            things more DRY.  We could have also used the getattr during the process()
            method before, and wrapped it around a try/except block, but that looks
            a bit messy.  This keeps things clean and simple with a nice one-to-one
            call-map. 
            """
            self.__opcodes = {}
            classes = [self.__class__] #: This holds all the classes and base classes.
            while classes:
                cls = classes.pop() # Pop the classes stack and being
                if cls.__bases__: # Does this class have any base classes?
                    classes = classes + list(cls.__bases__)
                for name in dir(cls): # Lets iterate through the names.
                    if name[:7] == 'opcode_': # We only want opcodes here.
                        try:
                            opcode = int(name[7:])
                        except ValueError:
                            raise NameError('Opcodes must be numeric, invalid opcode: %s' % name[7:])
                        self.__opcodes.update({opcode:getattr(self, 'opcode_%s' % opcode)})
        def fetch(self):
            """
            This method retrieves an instruction from memory address pointed to by the program pointer.
            Then we increment the program pointer.
            """
            self.ir = self.get_memint(self.pc)
            self.pc +=1
        def process(self):
            """
            Process a single opcode from the current program counter.  This is
            normally called from the running loop, but can also be called
            manually to provide a "step-by-step" debugging interface, or
            to slow down execution using time.sleep().  This is the method
            that will also need to used if you build a TK/GTK/Qt/curses frontend
            to control execution in another thread of operation.
            """
            self.fetch()
            opcode, data = int(math.floor(self.ir / 100)), self.ir % 100
            self.__opcodes[opcode](data)
        def opcode_0(self, data):
            """ INPUT Operation """
            self.mem[data] = self.get_input()
        def opcode_1(self, data):
            """ Clear and Add Operation """
            self.acc = self.get_memint(data)
        def opcode_2(self, data):
            """ Add Operation """
            self.acc += self.get_memint(data)
        def opcode_3(self, data):
            """ Test Accumulator contents Operation """
            if self.acc < 0:
                self.pc = data
        def opcode_4(self, data):
            """ Shift operation """
            x,y = int(math.floor(data / 10)), int(data % 10)
            for i in range(0,x):
                self.acc = (self.acc * 10) % 10000
            for i in range(0,y):
                self.acc = int(math.floor(self.acc / 10))
        def opcode_5(self, data):
            """ Output operation """
            self.stdout(self.mem[data])
        def opcode_6(self, data):
            """ Store operation """
            self.mem[data] = self.pad(self.acc)
        def opcode_7(self, data):
            """ Subtract Operation """
            self.acc -= self.get_memint(data)
        def opcode_8(self, data):
            """ Unconditional Jump operation """
            self.pc = data
        def opcode_9(self, data):
            """ Halt and Reset operation """
            self.reset()
        def run(self, pc=None):
            """ Runs code in memory until halt/reset opcode. """
            if pc:
                self.pc = pc
            self.running = True
            while self.running:
                self.process()
            print "Output:\n%s" % self.format_output()
            self.init_output()
    
    class Cardiac(CPU, Memory, IO):
        pass
    
    if __name__ == '__main__':
        c = Cardiac()
        c.read_deck('deck1.txt')
        try:
            c.run()
        except:
            print "IR: %s\nPC: %s\nOutput: %s\n" % (c.ir, c.pc, c.format_output())
            raise
    

    You can find the code for deck1.txt from the Developing Upwards: CARDIAC: The Cardboard Computer article, I used the counting to 10 example.

    Hopefully this article was inspiring to you and gives you a brief understanding on how to built class-bases modular and pluggable code in Python, and also gave you a nice introduction to developing a CPU simulator. In the next article which I hope to publish soon, will guide you through making a basic assembler for this CPU, so that you can easily and effortlessly create decks to play around with in the simulator.

  • Why Python?

    Posted on Wednesday, September 17th, 2014

    Here is an interesting subject which I would love to bring up and share with everyone. This article is not meant to compare Python with any other language, and in fact I will not be mentioning any language in this article but Python. As Everyone reading this might know by now, Python is a very diverse language. It can be used for tasks such as web development to the development of a Geographical Information System, to even computational biology and bioinformatics. Python has a vast array of fully supported GUI and web frameworks. This makes it a snap to apply the Model-View-Controller programming methodology to an entire application's ecosystem of software. The same code that is used to generate data for a GTK or Windows-based GUI application can also be applied to displaying the same data in a web application on the Internet. It is absolutely astonishing on what Python is really capable of when it comes to full-stack development. A developer really only need to know Python, and then the possibilities of what she or he can do is limitless.

    That is only the introduction to this article, and by now, regardless if you know or use Python or not, you should be at least somewhat impressed on it's capabilities. Python is so much more than a diverse language with limitless possibilities. Python is a vast community of developers from many backgrounds, ranging from computer science, to biology, to mathematicians, to even engineering. Python for many in the science community is the go-to language because of packages such as SciPy and Biopython.

    Python is normally installed by default on most Linux distributions, if not all. Python is also pre-installed on Mac OS X, making it a very universal language in terms of install base. This makes it easy to share your programs with other developers and colleagues. Due to this type of default install base on these platforms, some of the operating system specific scripts are also made in Python, and thus there are more bindings to operating system specific services that work right out of the box. This makes Python ideal for system automation and a general shell scripting replacement.

    If a developer knows Python, they are able to develop a wide array of applications, ranging from system automation, GUI applications, embedded development, web development, to even 3D modeling with Blender, all with a single language. This enables a developer to be extremely proficient in one language alone, as the developer doesn't need to worry about learning any additional languages to take on additional development tasks for their project. I personally distribute my Python knowledge throughout the Python ecosystem, I know how to build GUI applications, console application, system automation applications, network servers, and even web development. With Python, I am hardly limited to a single task, as just like Linux, Python is a general purpose. As with Linux, you aren't limited to just servers, you can deploy Linux on workstations, and embedded devices to even smartphones, as seen with Android. Knowing how to operate Linux in this day and age is starting to become critical due to the wide array of embedded devices and servers on the market today. Linux is only growing, and I believe it will play a vital role in the future of computing as we gravitate away from traditional PCs into the more mobile and embedded world. If Google and Apple have their way, PCs of the future will only be used for software development and PC gaming. I don't PC gaming dying, at least anytime soon.

    Python in the future will have a similar role to that of Linux, it will be that universal language that every computer science major will know and understand. With the Raspberry Pi, and children learning Python at a young age because of it, Python will be everywhere in our next generation of offspring. When I was a kid, BASIC was all the rage, where every home computer manufacturer had to ship with BASIC in ROM or the unit wouldn't sell as good. Python is like the new BASIC. When I grew into my teens and early adulthood, guess what was one of the more popular and widespread development kits for desktop development? If you said VisualBasic, then you would be correct. VisualBasic was huge in the late 90s and early 2000s, it was everywhere in the media and lots of organizations used it for development of internal software tools. As a result, ASP/ASP.NET used VisualBasic at the primary language at first, and is still used in .NET programming today.

    I believe Python is very critical for any developer to know and understand, or they will risk being left behind like FORTH and COBOL developers of the generation before BASIC. There are still some careers out there that require FORTH and COBOL, but for very old companies which have been around since the 70s and find it difficult to migrate away from their current solutions. If you understand how technology trends and have done research of how technology has trended since the 70s, it is difficult to not determine that Python is turning into the next hugely trending language and is only still growing. It is the language which our current offspring is learning that will be the next COBOL or BASIC in our technological world. Depending where some children grow up, you can also agree that Linux is also trending, especially in some European countries, and especially in communist countries where monopoly rule is banned. With the advent of Android, Ubuntu, Raspberry Pi, and the One-Laptop-per-Child project, children are seeing and using UNIX-like operating systems at a very young age. Also, the language shipped with the OLPC project is Python, with the entire Sugar desktop environment being made in Python. Another popular software product that teens and young adults are getting their hands on which uses Python is the Blender 3D modeling and animation software. One of my cousins has been learning Blender for her college degree as a 3D designer. In order to automate aspects of the 3D rendering software, and to enable game development, one can use Python. Python is literally everywhere, and in heavy use on popular software projects for the aspect of scripting and automation.

    In conclusion, to why to use Python should be a simple answer for any developer now. It enables a single language and code-base to be used across multiple projects easily, and it might very well be the next big language of our technological future. It is always best to be ahead of the curve and to not be left behind.

  • A JSON version of the standard Python xmlrpclib

    Posted on Thursday, August 14th, 2014

    I have always loved how easy Python made it to create simple RPC services using XML-RPC. I like how dead simple the API for both an XML-RPC server and client are. Here's an example XML-RPC Server in Python:

    from SimpleXMLRPCServer import SimpleXMLRPCServer
    
    # Create server
    server = SimpleXMLRPCServer(("localhost", 8000))
    
    # Register a function
    @server.register_function
    def hello(name='World'):
        return 'Hello %s!' % name
    
    # Run the server's main loop
    server.serve_forever()
    

    Accessing this service via Python couldn't be simpler:

    import xmlrpclib
    
    s = xmlrpclib.ServerProxy('http://localhost:8000')
    
    print s.hello()  # Outputs "Hello World!"
    print s.hello('XML-RPC') # Outputs "Hello XML-RPC!"
    

    I have been using XML-RPC a lot as a result, as I build mostly everything in Python. However, the overhead of XML parsing can be daunting for larger applications that are expected to receive many requests per minute. One technology that Python seems to be lacking in it's standard library is a simple service called JSON-RPC. Luckily I recently wrote a client library that currently only works on pure socket connections, here is the source code:

    #
    # JSON-RPC CLIENT LIBRARY inspired by the standard library xmlrpclib
    #
    # an JSON-RPC client interface for Python.
    #
    import urllib, json, socket
    
    class _Method(object):
        # some magic to bind an JSON-RPC method to an RPC server.
        # supports "nested" methods (e.g. examples.getStateName)
        def __init__(self, send, name):
            self.__send = send
            self.__name = name
        def __getattr__(self, name):
            return _Method(self.__send, "%s.%s" % (self.__name, name))
        def __call__(self, **kwargs):
            return self.__send(self.__name, kwargs)
    
    class ServerProxy(object):
        def __init__(self, uri):
            protocol, uri = urllib.splittype(uri)
            if protocol not in ('socket',):
                raise IOError, 'unsupported JSON-RPC protocol'
            self.__host, self.__handler = urllib.splithost(uri)
            self.__id = 0
        def __request(self, method, params):
            s = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM)
            s.connect(urllib.splitnport(self.__host, 80))
            self.__id +=1
            data = {"jsonrpc": "2.0", "method": method, "id":self.__id}
            if params != {}:
                data.update({'params':params})
            s.send(json.dumps(data))
            resp = json.loads(s.recv(512))
            s.close()
            return resp['result']
        def __repr__(self):
            return '<ServerProxy for %s%s>' % (self.__host, self.__handler)
        __str__ = __repr__
        def __getattr__(self, name):
            return _Method(self.__request, name)
    

    This library works in the exact same manner as the xmlrpclib shown above, expect it connects to JSON-RPC servers, such as the XBMC Media Center software Here is how you would interface with XBMC:

    #
    # A super simple XBMC JSON-RPC Interface for Python
    # Control your XBMC in a Pythonic way!
    #
    import jsonrpclib, time
    
    class XBMC(jsonrpclib.ServerProxy):
        def __init__(self, host, port=9090):
            super(XBMC, self).__init__('socket://%s:%s/' % (host, int(port)))
        def notification(self, title, message):
            return self.GUI.ShowNotification(title=title, message=message)
        def global_search(self, query):
            self.home()
            self.Input.Up()
            self.Input.Select()
            time.sleep(2) # Wait for keyboard on slower boxes.
            return self.Input.SendText(text=query)
        def weather(self):
            return self.GUI.ActivateWindow(window='weather')
        def home(self):
            return self.GUI.ActivateWindow(window='home')
        def favourites(self):
            return self.GUI.ActivateWindow(window='favourites')
        favorites = favourites
    

    All of this code, plus many more, are now available on my newest bitbucket repository aptly called Python Experiments.

  • Two-factor laptop decryption using a smartphone

    Posted on Thursday, August 7th, 2014

    This is a pet project I showed to a select few at PyCon 2014 in Montreal. I have not publicly disclosed this project to anybody else in the tech community until now. I will not be releasing the source code for this project, so do not ask. I am either planning on obtaining a patent for this, or starting a Kickstarter project to obtain the funding required to develop this into a full fledged product for multiple platforms. Currently it only works with Linux, as it was much easier to prototype using such an open and hackable operating system. However, given enough interest, I would love to port this product over to Mac OS X and Microsoft Windows platforms. I am currently looking for any feedback on this, and to see how much interest there might be for such as product.

    This product has three separate parts, which enable what I am calling Chained passphrase decryption. The third part of the chain is not mentioned in this video, nor will I talk about it at all. Basically, all parts of the chain are required to successfully construct the final encryption key that will be used to decrypt the device in question. My software does not perform any actual encryption, my software only constructs the key which is then passed on to the encryption software. On the Linux platform seen in the video, the encryption software used is LUKS, which is standard in every Linux distribution for block level encryption. This software should theoretically be usable with any software that can accept a passphrase.

    The main communication link as seen in this video is a standard authenticated bluetooth connection, from the smartphone to the laptop. The connection is only active for as long as it takes to obtain that part of the chain to construct the final passphrase. In most cases, the software is manually launched by the consumer when he or she needs to decrypt his or her device. On most secure smartphones, the consumer first needs to unlock their device with a password, PIN, or fingerprint in order to launch an application on said device. Most devices also support full encryption, securing this solution even further. With only these two chains being mentioned, there is a lot of work a hacker will need to do in order to obtain the encryption key to unlock the device. Let's review what a hacker will need to do:

    • The hacker will need physical access to both the laptop and smartphone
    • If the hacker only has the laptop, he or she can try to bruteforce the encryption, however since the actual key is a binary format, and can be very long. This might take the hacker a very long time to crack. This key isn't your everyday ASCII passphrase, which are relatively easy to crack since the human alphabet is limited.
    • If the hacker has both the laptop and smartphone, they need to unlock that smartphone using the smartphones authentication. The only way by this is to remove the NAND chip(and hope that it's not encrypted), or find a vulnerability in the smartphone's lock screen.
    • If the hacker finally has access to the devices data and can run the application which creates the bluetooth link. The hacker then is required to enter in the correct PIN on the device. This can be brute forced, but hopefully the hacker never gets to this point.
    • Finally, there is one last part of the chain, the third part, which can break the chain of trust entirely, preventing that smartphone and laptop pair from even working. I will leave this part to the imagination of the reader.

    As you can see, much thought has been put into this product to make a more secure laptop for consumers and corporate workers. Unlike existing solutions which rely on local-only measures to enforce encryption, this method takes it a step further to authenticate the end-user in question using their most personal device, their smartphone. In this day and age, everybody has a smartphone, or if you work for any large company and have a company laptop, you most likely carry a company smartphone with you at all times. So naturally, the next step of laptop encryption should use this pairing to harden laptop encryption.

    UPDATE 08/08/2014: I have been receiving various comments from the different sources which I posted a link to this. I would like to clarify that this project may or may not be open source, this variable has yet to be determined, thus is why I am not releasing the source code at this time. Some comments are pointing on the fact that bluetooth is insecure, well this might be true, so is plain text SMTP, HTTP, etc... That is why you place an encryption layer over top. For email, this is normally PGP encryption, and for HTTP, this is SSL. This bluetooth link is also encrypted, currently using AES, but this is subject to change. AES was mentioned in the original video. Hopefully that clarifies some additional left out details.

  • Hacker's Edge soon to be in Closed Beta

    Posted on Friday, May 23rd, 2014

    Does anyone recall a project I began way back in 2012 called Hacker's Edge? Originally, I was building the entire project in Django, but earlier this year after creating an ncurses based website, it got me interested in trying to build Hacker's Edge using a Python console and ncurses.

    So, for the past couple of months I have been hard at work getting this new version through the alpha stage of development. The alpha is currently publicly available for anyone to jump in and play around. This new version is no longer limited by web technologies and is only limited to what I can do in Python. At the moment, I am in the process of expanding the simple-cpu I built to create both the Black Hat and White Hat tools which will be used to hack other hosts in the game, and to better protect hosts in the game respectively. My simple-cpu has come a very long way since I first built it, and now supports a full 16-bit address space, x86-like assembly instructions, and much more. I hope to make a tutorial on using the simulator soon in stand-alone projects.

    Other implemented features are a fully working mail server system, with support for independent in-game mail server. The mail system itself uses the standard mbox API for mail storage and retrieval. A DNS system is also in place, and a host can choose which host is their DNS server. So you can imagine if a player manages to hijack a DNS server in-game the havoc that could occur. Each independent host runs it's own virtual machine and has it's own memory space mapped in memory. This allows for hosts to run their own KERNELs and other programs completely independent from the other hosts in the game world. The KERNEL can be written entirely by a player if they so choose to either better prepare their hosts from attack, or to prepare their host to attack other hosts better. There is also the possibility of players creating in-game rootkits to attack other hosts and infect them. Since the hacking and white hat in-game tools will run entirely within the virtual machine, this enables players to create torjan horses to infect other hosts easily. Since most of the in-game tools will cost in-game currency to purchase, I am assuming that some players would take advantage of this and pirate the in-game tools.

    Furthermore, the following in-game player services will also be hackable by players, if they are skilled enough to do so:

    • In-game banking services, so players can technically steal currency from other players.
    • In-game ranking and experience system, so players can try to forge themselves a better ranking or increase their experience points to level up quicker.
    • In-game shops which will sell the in-game tools, these shops may also be hacked to obtain said software for no charge.

    To make the programming/coding element more accessible to players who may not have experience in Assembly-like languages, I am also developing a high-level language which compiles into Assembly, which can then be edited and then assembled into binary.

    The project still uses Django for the front-end, and for player account management. Here is the typical workflow for a player:

    1. Real user browsers to Django website, and either signs up or logs in.
    2. Real user then accesses their player management page, and proceed to create a new player character.
    3. The Django app then registers the player character in the Hacker's Edge world, and creates them their initial host.
    4. Real user now connects to either telnet or presses Play.
    5. Real user then logs in using their Player Character login credentials created via Django.
    6. Player hopefully enjoys themselves in the virtual online hacking simulation game.

    Django still plays a critical part in this project, as it's the central hub to manage a user's player characters. This enables a user to easily have multiple player characters and switch between them at will, or play against themselves in two terminal windows. Furthermore, I am also wanting to integrate the Out-of-Character mode with the web presence. This will enable players to easily access both the forums and help center, and post questions and socialize. In Out-of-Character mode, the player character is never revealed to other users, so one can post to the forum and socialize, not knowing if you are actually getting along with a rival in-game. There is definitely a separation between in-game persona and your real-world self. The only players whom will know your real-world identity will of course be server moderators, whom are there to give a helping hand and not attack your in-game hosts or real-life self.

    Which brings me to my next subject, is that I will be looking for people who would be willing to help out with moderation. You will have a special account which will allow you to perform tasks which a regular user or player would not otherwise be-able to perform. The following special powers will be provided to those who assist in moderating:

    • Access to post into staff-only forum sections.
    • Access to all the undocumented VM API calls, to assist with building specific in-game tools.
    • Ability to see the real-world Django username of an in-game player.
    • Access to any and all in-game hosts regardless of ACL table, used to assist other players.
    • Early access to KERNEL.SYS and other critical VM source code, mainly to assist in testing and debugging.
    • Access to staff-only hosts, and admin access to many in-game hosts for moderation purposes.

    I am planning for Hacker's Edge to enter Closed Beta by the end of May, or mid-June. For those who already registered on the website a couple years back, your accounts are still there and active. Once the closed beta begins, I will be choosing from the pool of users, and sending out an email notification to them to inform them of the new Hacker's Edge experience. During closed beta, the following will be available, and be tested during the closed beta:

    • In-game Black Hat and White Hat tools.
    • In game currency, and banking system.
    • In-game experience and ranking systems.
    • Missions, which will provide the experience points, and currency to the players.
    • And more, which I cannot think of right now.

    Before the alpha ends, there will be a basic test mission given to anybody who enters the alpha game in the form of an in-game email message, with instructions on what should be done. This will be there to test out the mission mechanics before going beta.

  • PyCon 2014 Montreal

    Posted on Tuesday, April 8th, 2014

    I will be attending the PyCon in Montreal this week, and I look forward to many of the talks and blogging about as much as I possibly can while I am there. If anybody whom follows Python Diary is attending, you should be-able to find me there.

    Looking forward to all the excellent talks this year, and this is my very first PyCon, so you can just see how excited I must be!

    Everything PyCon will be properly tagged on the blog, so just bookmark that tag for the latest news directly from PyCon 2014.

  • Website made entirely using ncurses!

    Posted on Saturday, February 1st, 2014

    As an interesting proof-of-concept idea, I built an entire website, my personal profile page using nothing more than ncurses and a vt100 terminal emulator! I recently unleashed it out into the public, and there was lots of positive and nostalgic responses. This is only the first iteration of the site, and more features will be coming soon.

    The result was unexpectedly usable! Although hosting such a site is only ideal for low traffic websites as the server resources can really be used up by each forked Python process. How it works is simpler than you might imagine, it is merely a standard Python console application running directly in the browser! To obtain the vt100 emulation and access to the Python application, I used a popular remote shell tool called Shellinabox. Shellinabox is normally used to provide a Linux admin with a remote login shell via their browser to administer the server, just as he/she would while using SSH. Shellinabox can actually run any terminal program you tell it to run, and this gave birth to this idea. I first tested it by running Lynx in a web browser directly, so accessing a page would immediately display the popular Lynx browser. Amazingly, the mouse was also able to control Lynx! So... I thought, if I could make my own ncurses application which can also use a mouse, I could build an entire website using just ncurses!

    You can view it live by clicking on my name below my photo in the right bar of this website. There is also an alternative telnet interface for those whom don't have a graphical browser handy, as vt100 emulation requires a decent JavaScript engine.

    This is more of a proof-of-concept, as I am planning on building this idea further with a complete recode of my Hacker's Edge massively multiplayer online game. I had some difficulties making Hacker's Edge what I wanted it to be, and I believe turning it into a full Python console application with some ncurses components might be what I need to really make this game take off. The game will end up featuring both a web-based vt100 terminal emulation for those without a compatible Telnet client, and telnet connection support. The new idea will be to add a better network simulator, many console applications, and a fake assembly language to built fake viruses and malware to create fake botnets. Think of it as a Grand Theft Auto for CyberCrime! As you can see from my Bitbucket page, I have been creating some specialized applications, such as the CPU simulator, which will make it's way into Hacker's Edge.

  • Kevin Veroneau Consulting demo video

    Posted on Tuesday, December 10th, 2013

    In this video you can see my Consulting Services website in action to see the benefits of working with me over other independent Django contractor. The tools were of course built using Django, and have been used for over a year now successfully:

  • Have Django respond to Email in real-time

    Posted on Thursday, August 1st, 2013

    It's been a long time since I wrote a new article on Python Diary, and I do apologize for the lack of content recently. Hopefully this interesting article will make up for that.

    Have you ever wanted Django to parse an incoming Email in real-time as soon as it comes into your server? Look no further than this handy solution here. In order to use this solution, you will need to configure a local Postfix server, it doesn't need to be-able to send outbound email, only receive new messages for your domain. Although being able to send messages will make it easier to reply back to the user's whom are sending the server messages.

    You will first need to create a short Python script which Postfix will deliver the actual message to:

    import sys
    import urllib2
    
    req = urllib2.Request(url='[Django URL]', data=sys.stdin.read())
    f = urllib2.urlopen(req)
    

    This script submits the entire message unmodified directly to any web address you give the script, preferably a Django URL. Notice that I am not using the standard form encoding, as there is no need to add additional overhead when all we want to do is have Django process an entire message. The main reason why this script exists, is because this script will run normally as nobody:nogroup, so it will not be-able to easily talk with your local Django app directly using say, a management script. Sending the data directly over HTTP to the Django instance is simple and should have very little overhead and integrate easily with existing solutions.

    from django.views.decorators.csrf import csrf_exempt
    from email import message_from_string
    from django.http import HttpResponse
    
    @csrf_exempt
    def mail_parse(req):
      msg = message_from_string(req.body)
      # Process data from msg here!
      return HttpResponse("OK", content_type="text/plain")
    

    Here is the Django view which will receive the raw message data and parse it. You should read the Python documentation on the subject to learn how to extract the headers, body, and attachments you want to parse. Finally, here is what you should place in your /etc/aliases:

    api:         |/[path to script].py
    

    Be sure to run newaliases afterwards to rebuild the hash table. This alias here makes anything api@[domain] submit to your Django site. Enjoy!

  • Django bootstrap theme is ready!

    Posted on Saturday, May 25th, 2013

    You can download a project template for this now: http://www.pythondiary.com/templates/bootstraptheme

    It's finally ready for developer consumption, my Django bootstrap theme Django app. With this super simple app in your Django project, you can easily apply a site-wide Bootstrap theme in a matter of minutes. Here's an example base.html you would use in your project's templates directory:

    {% extends 'bootstrap_theme/starter.html' %}
    
    {% block title %}Bootstrap Django site{% endblock %}
    
    {% block branding %}Django Project{% endblock %}
    
    {% block content %}
    Yes Folks, this is how easy it is to implement bootstrap in Django now.  Be sure to look at the other templates available to see what <b>blocks</b> you can override.
    {% endblock %}
    

    Be sure to add bootstrap_theme to your INSTALLED_APPS, and this will work like a charm. Here is a list of available base templates you can extend:

    base.html
    This is the main template which the ones below extend to create the Bootstrap experience.
    starter.html
    This is the exact starter template from the Bootstrap documentation, it's the most basic and easiest to extend.
    fluid.html
    Similar to starter, except it allows users to sign in and out, and has a left navigation bar.
    marketing.html
    The marketing template directly from the bootstrap examples page tailored to work with Django.
    marketing_narrow.html
    From the bootstrap example page, another version of a marketing page you can try out.
    marketing_clean.html
    Yet another marketing page example, also from the bootstrap examples page.
    carousel.html
    The bootstrap carousel example made for Django, the bonus is that the Carousel is driven from the database, so it's easy to add and edit the big Carousel banner.

    Wait, there's more... There's a large amount of templatetags at your disposal as well in the bootstrap library which you can load into any template you need it in. Here's what's included:

    buttonlink url icon
    You can create bootstrap themed buttons with this tag.
    emphasis filter
    This filter will add special styling to the text, such as muted which is the default, or anything in the text- CSS components.
    abbrev filter
    This handy filter will create an abbreviation text.
    yesnoicon filter
    Depending if the value is True or False a different icon is displayed.
    ratingicon filter
    This super handy filter will create a specific amount of star icons depending on the value.
    link filter
    Pass in any model object which has a get_absolute_url and it will automatically make it click-able.
    carousel group_name
    This is used to display a database driven Carousel. The group_name parameter is which group in the database model to display for this instance.
    navbar_gradient first_color last_color
    This super handy tag will automatically generate the required CSS style needed for the navbar gradient.
    modal title modal_id form_action
    This block tag is used to generate a modal dialog box, the last 2 options are optional. The first is the title of the dialog box, the second is the unique div ID for the modal, and the last one is the URL which the form inside the modal will be posted to, if the user saves the changes.
    icon slug
    This is used to display bootstrap icons easily.
    modal_button title modal_id icon
    This is used to generate the required A tag to display a modal dialog box generated above. The first option is what to display on the button itself, the second is the div ID chosen, and the last one is an icon slug to place on the button. The last two options are optional and have reasonable defaults.

    Don't leave yet, there's still one last thing... An include-able template which can display a Django form very elegantly in bootstrap! Finally you can make your Django generated forms look great and functional with very little effort. Here's an example contact form to get you started:

              <form action="{% url 'contact-form' %}" method="post">{% csrf_token %}
                <fieldset>
                  <legend>Contact form</legend>
                  {% include "bootstrap_form.html" %}
                  <button type="submit" class="btn btn-primary">Send Message</button>
                </fieldset>
              </form>
    

    Wow, that's a lot to take in, in the next release I will be adding a bunch of ready made templates for django-registration so that you can get a new Django website up and running in a matter of hours.

About Me

My Photo
Names Kevin, hugely into UNIX technologies, not just Linux. I've dabbled with the demons, played with the Sun, and now with the Penguins.




Kevin Veroneau Consulting Services
Do you require the services of a Django contractor? Do you need both a website and hosting services? Perhaps I can help.

If you like what you read, please consider donating to help with hosting costs, and to fund future books to review.

Python Powered | © 2012-2014 Kevin Veroneau