Cleaning cruft in a Ubuntu installation

I have been running Linux on my laptops for as long as I have been owning laptops. My current one has been through quite a few versions of Ubuntu Linux. There are normally no big changes from version to version, but it steadily improves. In that process some software packages that previously were installed by default are replaced or become obsolete. Most of this software is still updated, but it is no longer present on new installations and it may have no value to me.

Every time I update to a new version of Ubuntu I have a feeling that the cruft is building up, but there is no obvious way that I have found to remove software that has no longer been found worthy of being part of the default install. So, this time I decided to write a script to identify these packages. The script takes a list of software packages to keep, i.e the software that I use directly when I use my machine. From this list it finds all the packages that must be present for my prefered software to work, and lists all the other that can in princible be removed.

In reality, it is hard to come up with a list of all the software that is required for a Ubuntu installation to work properly, so I also made the script read the newest list of default packages. Any of these that happen to be installed are also kept along with their dependencies.

The script itself, unnecessarry-packages.py can be found on Bitbucket. It does not uninstall any packages by itself, but suggest packages that you can uninstall by running apt purge PCKGNAME or similar. I put it online and write about it here in the hope that it will be useful to someone else.

I started out with the following list of packages to keep,

python3 unnecessarry-packages.py ubuntu-desktop krita thunderbird firefox ubuntu-minimal digikam gimp darktable gedit Downloads/ubuntu-16.10-desktop-amd64.manifest -v

and after some trial and error I ended up with the following by noticing packages suggested for removal that I in reality wanted to keep and adding them to the command line invocation,

python3 unnecessarry-packages.py ubuntu-desktop krita thunderbird firefox ubuntu-minimal digikam gimp darktable gedit clang bcmwl-kernel-source systemsettings enfuse ffmpegthumbs flashplugin-installer gstreamer1.0-nice gstreamer1.0-plugins-bad gstreamer1.0-plugins-bad-faad  gstreamer1.0-plugins-bad-videoparsers gstreamer1.0-plugins-ugly gstreamer1.0-plugins-ugly-amr hugin hugin-tools inkscape gimp-ufraw git  gmsh gphoto2 gstreamer1.0-libav gimp-plugin-registry  ttf-mscorefonts-installer mercurial kipi-plugins gimp-gmic sshfs Downloads/ubuntu-16.10-desktop-amd64.manifest -v

This allowed me to get rid of approximately 4 GB of cruft without any noticeable problems afterwards. Note that I did not blindly remove all packages suggested by the above final invocation, but I removed maybe 90% of them. The script is not smart, it can easily suggest that you remove your network driver, so use with caution!

Yay!

Hoarfrost / Yay!

Tagged as: Code Python Ubuntu Debian apt

3D visualization in IPython notebook

The IPython notebook is a great tool for interactive computing. Below I show a short Python example on how to work interactively with simple 3D models in the IPython notebook.

To begin with, we need a simple way to represent a 3D model in Python. We will here restrict ourselves to geometries made up of plane 2D polygons in 3D:

class PolygonCollection(object):
    def __init__(self, nodes, polygons):
        """
        A collection of plane polygons in 3D

        Nodes is a list of 3-tuples of coordinates
        Polygons is a nested list of node indices for each polygon, one
        list of indices for each polygon
        """
        self.nodes = nodes
        self.polygons = polygons

    def extents(self):
        """
        Calculate the size of the 3D polygon model
        """
        xmin, ymin, zmin = 1e100, 1e100, 1e100
        xmax, ymax, zmax = -1e100, -1e100, -1e100
        for x, y, z in self.nodes:
            xmin = min(x, xmin)
            ymin = min(y, ymin)
            zmin = min(z, zmin)
            xmax = max(x, xmax)
            ymax = max(y, ymax)
            zmax = max(z, zmax)
        return xmin, xmax, ymin, ymax, zmin, zmax

We then populate our 3D model with some polygons. The polygon <-> node/vertex connectivity is simply represented as indices into the list of nodes/vertices:

nodes = [(0.0, 0.0, 0.0), (1.0, 0.0, 0.0), (1.0, 1.0, 0.0), (0.0, 1.0, 0.0),
         (1.0, 0.0, 0.0), (1.0, 0.0, 1.0), (0.0, 0.0, 1.0),
         (0.0, 1.0, 0.0), (0.0, 1.0, 1.0), (0.0, 0.0, 1.0)]
elements = [(0, 1, 2, 3), (0, 4, 5, 6), (0, 7, 8, 9)]
polygons = PolygonCollection(nodes, elements)

IPython notebook allows us to embed images, Latex equations, HTML, Javascript and more. In this case we will use Javascript to render our 3D model. This can be done as follows:

from IPython.display import display, Javascript
js = polygons2js(polygons)
display(Javascript(js))

The end result can be seen below. In a reasonable modern browser you should be able to rotate the 3D model. If it does not work it may also be that the Javascript libraries imported to do the rendering have changed since the time when this was written. I do not believe the libraries have stable APIs just yet. As of May 2014 it should work in the latest versions of Firefox, Chrome and Internet Explorer.

The "only" thing needed to make this work is a function polygons2js(polygons) that converts the 3D model to a set of javascript commands that will run inside the notebook. This can be done quite easily with the help of JSModeler and three.js. A simple implementation will look something like this:

import random

def js_wrap(js, js_libraries):
    """
    Wrap javascript commands in code that preloads needed libraries
    """
    lines = ['jQuery.when(']
    lines += ['  jQuery.ajax({url: "%s", dataType: "script", cache: true}),' % lib
              for lib in js_libraries]
    lines += ['  jQuery.Deferred(function( deferred ){',
              '    jQuery(deferred.resolve);',
              '  })).done(function(){',
              js,
              '});']
    return '\n'.join(lines)

def polygons2js(polygons):
    # Find the size of the 3D model. We use this later to make the default camera and rotation point work
    xmin, xmax, ymin, ymax, zmin, zmax = polygons.extents()
    length = max([xmax-xmin, ymax-ymin, zmax-zmin])
    xmean = (xmin + xmax)/2
    ymean = (ymin + ymax)/2
    zmean = (zmin + zmax)/2

    # Generate the Javascript code
    # see http://kovacsv.github.io/JSModeler/documentation/tutorial/tutorial.html for details
    canvas_id = 'js3dcanvas_%d' % random.randint(0, 1e10)
    canvas_style = 'style="border: 1px solid black;" width="800" height="600"'
    js = ['var widget = jQuery(\'<canvas id="%s" %s></canvas>\');' % (canvas_id, canvas_style),
          'element.append(widget);',
          'container.show();',
          'var viewerSettings = {',
          '  cameraEyePosition : [-2.0, -1.5, 1.0],',
          '  cameraCenterPosition : [0.0, 0.0, 0.0],',
          '  cameraUpVector : [0.0, 0.0, 1.0]',
          '};',
          'var viewer = new JSM.Viewer();',
          'viewer.Start("%s", viewerSettings);' % canvas_id,
          'var body = new JSM.Body();']

    # Add node coordinates
    coords = []
    for coord in polygons.nodes:
        coords.append('[%f, %f, %f]' % ((coord[0]-xmean)/length,
                                        (coord[1]-ymean)/length,
                                        (coord[2]-ymean)/length))
        #print coord, coords[-1]
    js.append('var coords = [%s];' % ', '.join(coords))
    js.append('for (var i = 0, len = coords.length; i < len; i++) {')
    js.append('  body.AddVertex(new JSM.BodyVertex('
              'new JSM.Coord(coords[i][0], coords[i][1], coords[i][2])));')
    js.append('}')

    # Add elements
    polys = [repr(list(poly)) for poly in polygons.polygons]
    js.append('var elems = [%s];' % ', '.join(polys))
    js.append('for (var i = 0, len = elems.length; i < len; i++) {')
    js.append('  body.AddPolygon(new JSM.BodyPolygon(elems[i]));')
    js.append('}')
    js.append('var meshes = JSM.ConvertBodyToThreeMeshes(body);')
    js.append('for (var i = 0; i < meshes.length; i++) { viewer.AddMesh (meshes[i]); }')
    js.append('viewer.Draw();')

    # Wrap final js code in a library loader to make sure JSModeler and three.js are available
    js = '\n'.join(js)
    libraries =  ['http://kovacsv.github.io/JSModeler/build/three.min.js',
                  'http://kovacsv.github.io/JSModeler/build/jsmodeler.js']
    return js_wrap(js, libraries)

I have found this to be quite handy for inspecting small 3D models and modifying the interactively. The PolygonCollection class can also be updated to automatically display the 3D model in the notebook if it objcts of the class are left on the last line of an IPython input cell, or if you call display() on them. For this to work you will have to extend the PolygonCollection class with a _repr_javascript_ metod that returns polygons2js(self) (as a string). IPython will automatically look for this method and call display(Javascript(polygons._repr_javascript_())) for you.

Tagged as: Code Python IPython three.js 3D visualization

Using jsMath in MoinMoin with ReStructuredText

This is not a typical post for this blog, but it was just convenient to put the following here. Feel free to skip this post if you are reading this blog for the normal content and haven't gotten here by searching for just this.

I spent a few hours at work today researching how best to add math (displayed equations) support to the docutils ReStructuredText parser for use with the MoinMoin wiki engine. I put my findings up here for the search engines to pick up in case more people need this.

The best solution today seems to be jsMath. It is more versatile than the various LaTeX -> dvi -> png solutions (and easier to set up). It is also much better supported than MathML.

Docutils, which is the library used by MoinMoin to render ReStructuredText to html, does not yet support math natively (as of version 0.6). Fortunately it can easily be extended with new directives. The only thing you need to do is add the following code to a Python file that is executed while rendering your wiki pages. The theme module is a good place, just stick the following at the bottom of your theme's .py file:

# LaTeX support
from docutils.parsers.rst import directives, roles
from docutils import nodes

def latex_directive(name, arguments, options, content, lineno, content_offset, block_text, state, state_machine):
    latex = '\n'.join(content)
    return [nodes.raw('', '\\['+latex+'\\]', format='html')]

latex_directive.content = 1
directives.register_directive('latex', latex_directive)

def latex_role(name, rawtext, text, lineno, inliner, options={}, content=[]):
    latex = rawtext.split('`')[1].replace('\\\\','\\')  # Restore escaped backslashes
    return [nodes.raw(rawtext, '\\('+latex+'\\)', format='html')], []

roles.register_canonical_role('latex', latex_role)

This is a bit of a hack as it uses the old style function-as-directive model, but hey, it works, and I did not even know of docutils directives until today, so working is most definitely a big plus :-) It is based on this, but simplified a whole lot due to using jsMath to render the LaTeX equations.

You also need to include

<script type="text/javascript" src="/path/to/jsMath/easy/load.js"></script>

somewhere in your html code. I put it in the page header with an ugly hack by adding the script to the output around line 680 of MoinMoin/theme/__index__.py. It can (and should) probably go in your theme's .py file somewhere so that it does not disappear when MoinMoin is upgraded. I did not bother to do this for testing purposes, though.

Now you can write :latex:`e^{i\pi} + 1 = 0` in your wiki pages and it should be rendered to \(e^{i\pi} + 1 = 0\) in the final output (which seems like a small change, but believe me, it is easier this way then double backslashing the LaTeX code in the wiki page. You can also have displaymath equations with the following syntax:

.. latex::   e^{quation} g_{oes} = here

which is turned into

\[ e^{quation} g_{oes} = here \]

The \( ... \) and \[ ... \] is picked up by jsMath and turned into beautiful equations (depending on field of work and definition of beautiful ;-) ). For non-javascript enabled browsers the code is left as is, so the equation can still be deciphered by LaTeX savvy readers. See here for examples of what jsMath can do.

So, in the end it was fairly easy to get good looking math support in MoinMoin. Hopefully docutils will come with jsMath support in the future. The new upcoming 0.5 version of Sphinx seems to do, and it is based on docutils and ReST. Shinx is a new documentation power tool for Python developers. Highly recommended to make documenting code a bit more fun.

Comments are disabled on this article due to problems with spam. Somehow spammers believe that link-spamming this page with praise for the article will send them lots of readers or google-points or something ...

Tagged as: LaTeX MoinMoin Python ReStructuredText English