A Youtube Uploader - the UNIX Way!

My university's pantomime society has recently started putting a whole lot of clips of its past shows up on youTube. And since I'm the resident "computer guy", I've been given the job of transcoding/uploading them all.

Transcoding was easy enough - just install ffmpeg, write a quick bash script, and we're done. On the other hand, there didn't seem to be an obvious way to automate the upload process.

What I really wanted was a program I could use from the terminal, something I could script. Ideally, I should also be able to schedule the uploads so that they ran during the middle of the night (when I'm not using the computer and my ISP doesn't monitor bandwidth usage). Googling didn't turn up any suitable candidates - there were a few youtube upload applications, but they were all windows only. And while I do still have a windows installation on my laptop, I didn't really feel like booting a different OS just to upload a few videos. I wanted something that worked on Linux.

So I did what any good hacker would do - I've written up a quick python script to do the job. I've tried to follow the UNIX design philosophy in writing the program. Thus it doesn't transcode video. It doesn't have a gui. The scheduling is handled using at. In summary, it does just one thing (uploads videos to youtube) and tries to do it well. But the end result is a program which can be combined with others to offer a lot of flexibility, and is easy to automate using scripting.

You can get the script here. You will need to download and install Google's gdata library for python, which you can get here. You can either elect to install the library as per the installation instructions, or else you can simply extract the contents of the archive into the same directory as the uploader script. To run the script, you will need python 2.6. The reason for this is that I'm starting to get into the habit of using the print function (instead of print statement) in python 2.6 to get ready for eventually moving to python 3. However, as far as I'm aware, the gdata library only supports python 2.x. Thus the only python version capable of running the script is python 2.6. If this is a problem, the print functions could be replaced with print statements, allowing the script to run on python 2.x (with x < 6). This is left as an exercise for the reader (which is of course code for "I'm too lazy to do it, if it's a problem do it yourself" :P ).

Finally, you will also need a YouTube API key. You can generate a key here. You will need to add this key to the uploader script. Open the file "youtubeUploader.py", and replace the text "YOUR API KEY HERE" (near the top of the file) with your newly generated API key.

There's still plenty of room for improvement - for instance it would be nice to have some indication of the upload progress while files are uploading. I might add this feature at some point, although I couldn't work out how to do it from the gdata documentation. Anyway, hope someone finds the script useful (if only as a demo of how to use the gdata api).

http://dl.dropbox.com/u/1614464/youtubeUploader/youtubeUploader.tar.gz

Poll Mod v0.1.1

This is a minor update to the 0.1.0 version of my poll mod for punBB 1.2. It contains a single bugfix. Previously, if anyone had a user name containing the character "@", they were allowed to vote multiple times, but not allowed to see the poll results. This bug has now been closed.

robs_poll_mod_0.1.1

A quick script for concatenating MTS files

A small script I wrote to fix some problematic video files. In this case the problems were due to the design of the video camera I was using. Basically after recording 2GB of data, the camera was dumping whatever data it had recorded into an mts file, and then starting a new recording. While all the camera data ended up being recorded, problems arose with the second file, as the keyframe information was in the wrong place, etc. Basically, whoever designed the camera (some team at Sony) were morons.

Here's a simple script I wrote to fix their incompetence. Usage is mtscat -o output_file file1.mts file2.mts ...

Essentially does the same thing as "cat file1 file2 file3... > output_file" except that for the second and subsequent files it skips over the first 192 bytes (the header for the MTS stream)

Script is linked below:
mtscat.py

Poll Mod for PunBB 1.2.x - updated version

An update to the punBB poll mod I hacked together a year or so ago - frankly, something that should have been done a long time ago. This update is a major redesign, although it recycles much of the code from the 0.0.3 release. Looking back at the old code is a little embarrassing, to be honest, and it's clear that I hadn't spent any time thinking about its design. This release introduces many new features, but more importantly doesn't feel like it was programmed by a moron.

Changes include

• Code refactoring, more modular design, greater separation of mod code from punBB code
• Support for php4 removed - php5 is now a requirement
  
• Simplified installation procedure
• Ability to restore database if uninstalled
• Existing topics can be turned into polls
• Ability to edit, close and delete existing polls
• Poll permissions and colour scheme can be edited via the newly introduced admin console
• Ability to restrict which classes of user have permission to create polls
• n-1 polls now show approval ratings (ie percentages do not add up to 100, but results are more meaningful)
• multi-lingual support
• licence is now GPLv3, instead of LGPLv3
  
• Various performance improvements and bug fixes
  

The database schema in this release has changed from that of the 0.0.3 release. The current release, and all future releases will write the poll mod version into the database, so that the installation script will be able to update existing polls if the database schema is ever changed again. Sadly, I wasn't so foresighted when I released 0.0.3, so if upgrading from 0.0.3, the database schema will need to be updated by hand. This is not too difficult, though - just create a boolean column "closed" which defaults to 0, and a varchar(200) column named "creator" in the polls table. The following (conf_name, conf_value) pairs then need to be added to the punBB config table:

• ('rob_poll_version' , "0.1.0")
  
• ( 'rob_poll_permission_level' , '1')
• ( 'rob_poll_bg_colour' , '#FFFFFF')
• ('rob_poll_bar_colour' , "#190A8F")
• ( 'rob_poll_text_colour' , "#A0A0A0")
• ('rob_poll_text_size' , "14")

Optionally, the data type of the following columns in the polls table can be changed as follows

• question: change Text to varchar(200)
• options: change Text to varchar(2010)
• votes: change Text to varchar(110)
  

In future releases, I'll pay more attention to breaking the database schema. The version numbering convention will help in this regard - whenever I change the database schema in a backwards incompatible way, I'll update the minor version number (eg from 0.0.3 to 0.1.0)

The mod code and installation instructions are linked below. Thanks and acknowledgements go to Jan Odvarko, author of the jscolor script. This script is a javascript colour picker, which I have incorporated into the admin console - it is used to adjust the mod's colour scheme.

robs_poll_mod_0.1.0.tar.gz

Turning the AVR butterfly into an ISP

First step to building my synthesizer - constructing an AVR programmer. Due to the lack of a parallel port on my laptop, building a parallel programmer was out of the question. Hence I present here a (serial) In System Programmer (ISP).

An ISP comes with one caveat, however - the AVR 910 protocol (used for ISPing AVR devices) is too complex to simply use bitbanging over RS-232. Most programmer designs get around this by using a programmed AVR to translate messages sent over RS-232 (from the host computer) into AVR 910 (sent to the target system). Yep. They use a programmed AVR to build an AVR programmer. Can you spell circular dependency?

So in order to build an ISP, there are a few options

1. Use a parallel programmer to bitbang the AVR used in the ISP
   
2. Buy a commercial AVR programmer
3. Borrow someone else's AVR programmer

For me, (1) is ruled out due to my lack of parallel port. (2) is ruled out due to the state of my finances. And I don't know anyone who owns an AVR programmer.

There is, however, one further option, which is to turn Atmel's AVR butterfly demonstration board into an ISP. The AVR butterfly comes with several useful features, usefully for this project, a built in RS-232 level converter. Even more interestingly, it is also self programmable, due to its preloaded bootloader.

This basically means that an AVR programmer is not required to program the butterfly. Since I have a spare butterfly lying around the home (they were given out to first year electrical engineering students a couple of years back at my uni) it seems a good candidate for my ISP.

The code for the programmer, as well as the makefile and some basic schematics are here. The design is based on Atmel's AVR 910 application note, though it adds several features, notably support for a wider variety of AVR targets. It should be compatible with the "avrisp" driver in avrdude.

So happy building. Finally, I should note that I do not intend for this to be a permanent programmer - I have ported the above code to the atmega8, and will actually be building a standalone ISP later - but I will be using this butterfly programmer to bootstrap it.

Out, out damned bot!

Recently, our university club's website was used as a citation in a wikipedia article. On the plus side, this has increased our website's google pagerank quite substantially. On the downside, our small punBB based forum is now receiving much greater attention from the spambots.

So here I present two very simple modifications to punBB that have helped to stem the spambot tide. The first is integrating the captcha service provided by recaptcha into the punBB registration form. To use this service, you will need to register. You will also need to insert your recaptcha public and private keys into the mod code provided below.

The second method employs the spambot blacklist at http://www.stopforumspam.com/. It will check the email and ip address of any newly registered forum user, denying them registration if they are found to be on the blacklist.

Finally, I recommend turning on punBB's email validation feature.

While extremely simple, these measures seem to have reduced the number of spambot registrations on our forum quite substantially.

The mod code is below. As I have ranted about previously, the university's hosting server only has php4, so some parts of the code are a little kludgy (php4 lacks decent exception handling). Note also, that the code has only been tested on punBB 1.2.16, though it should work fine on any other version in the 1.2 series.

Instructions

1. Register an account with recaptcha, and then register your site's domain. Take note of your private and public keys
2. Download recaptchalib.php, and upload it to the root directory of your punBB forum.
3. Open the punBB file "register.php". Locate (around line 80) the lines
   

    else if (isset($_POST['form_sent']))
   {
   

   
   After, add the following code, remembering to insert your recaptcha private key where required:
   

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

   //Validate CAPTCHA entry require_once('recaptchalib.php'); $privatekey = "INSERT YOUR RECAPTCHA PRIVATE KEY HERE"; $resp = recaptcha_check_answer ($privatekey, $_SERVER["REMOTE_ADDR"], $_POST["recaptcha_challenge_field"], $_POST["recaptcha_response_field"]); if (!$resp->is_valid) { message ("You have failed the human test. Please re-enter the CAPTCHA. Unless you are a bot. If that is the case, then bugger off!" . "(reCAPTCHA said: " . $resp->error . ")"); } //end validate CAPTCHA entry //check the users ip and email against the www.stopforumspam.com database. This code is really awkward, since it has to run on php4. //check ip $xml_string = file_get_contents('http://www.stopforumspam.com/api?ip='.get_remote_address()); if ($xml_string){ //make sure we degrade gracefully if stopforumspam.com is offline if (strpos($xml_string, "<appears>yes</appears>")) //phps string handling is terrible, and normally we can't write code like this. This is ok, since if a match occurs, it will never be at the zeroeth character - the response from stopforumspam always begins with <response success="true"> message("You are trying to register from an ip address that has been used by a spambot. If you are a legitimate (human) user, please email us, so we can sort out your registration. If you are a spambot, then kindly die in a hole."); } //check email address $email = strtolower(trim($_POST['req_email1'])); $xml_string = file_get_contents('http://www.stopforumspam.com/api?email='.$email); if ($xml_string){ //make sure we degrade gracefully if stopforumspam.com is offline if (strpos($xml_string, "<appears>yes</appears>")) //phps string handling is terrible, and normally we can't write code like this. This is ok, since if a match occurs, it will never be at the zeroeth character - the response from stopforumspam always begins with <response success="true"> message("Your email address seems to belong to a spammer. Please die in a hole. Thank you for your co-operation."); }

   
   
4. Scroll down to about line 310, and find the following lines:
   

   1 2 3 4 5 6 7 8 9 10 11 12

   <div class="inform"> <fieldset> <legend><?php echo ($pun_config['o_regs_verify'] == '1') ? $lang_prof_reg['E-mail legend 2'] : $lang_prof_reg['E-mail legend'] ?></legend> <div class="infldset"> <?php if ($pun_config['o_regs_verify'] == '1'): ?> <p><?php echo $lang_register['E-mail info'] ?></p> <?php endif; ?> <label><strong><?php echo $lang_common['E-mail'] ?></strong><br /> <input type="text" name="req_email1" size="50" maxlength="50" /><br /></label> <?php if ($pun_config['o_regs_verify'] == '1'): ?> <label><strong><?php echo $lang_register['Confirm e-mail'] ?></strong><br /> <input type="text" name="req_email2" size="50" maxlength="50" /><br /></label> <?php endif; ?> </div> </fieldset> </div>.

   
   
   
   After, add the following code, remembering to substitute in your own recaptcha public key where required:
   

   1 2 3 4 5 6 7 8 9 10 11 12 13 14

   <div class="inform"> <fieldset> <legend>Human Test (CAPTCHA)</legend> <div class="infldset"> This is to ensure you are not a horrible spambot. Please complete the CAPTCHA below <?php //display reCAPTCHA require_once('recaptchalib.php'); $publickey = "INSERT YOUR RECAPTCHA PUBLIC KEY HERE"; echo recaptcha_get_html($publickey); ?> </div> </fieldset> </div>

   
   
5. Save the modified version of register.php, and upload it to your hosting server

If anyone has any feedback on the above modifications/wants to unleash the bile against spambots, feel free to leave a comment.

A five octave, 6 voice synthesizer

After about 2 years of cogitation, I've finally decided to do it - build a synthesizer. And finish it by the end of the year.

I'd originally intended to build an analogue synth, in the mould of the mini-moog. On further investigation, it transpired that the price/feature ratio of such a design was not particularly good. So, being a poor (and hungry) uni student, I'll be going with the cheaper (digital) option.

Basically, I want the synthesizer I design to have the following features

1. polyphony - must be able to play at least three notes simultaneously
2. the ability to split the keyboard, so that each half can be mapped to a different voice
3. pitch bending
4. analogue controls for setting the attack, levels of each harmonic, etc - I find digital controls clunky
5. 16 bit, 44kHz sound output, or better
6. Additive synthesis
   
7. Total cost to manufacture of less than $200

And now, finally, I have a (concept) design which meets all these requirements.

The design will be based around Atmel's atmega8 microcontroller. Actually, in order to meet the performance requirements, it will be based around seven of them. On the surface, using seven low performance, 8 bit micros, seems a little silly, particularly when there are plenty of excellent ARM based products out there, of similar price, but with vastly superior performance. However, there were several reasons for choosing the AVR product.

1. The atmega8 is really cheap (about $3)
   
2. Programmers for AVRs are really easy (and cheap) to build, driving down the total cost of the project (I don't own a JTAG based programmer, which is what most of the ARM based micros seem to require, and such programmers are quite expensive)
3. The documentation for the AVR family is excellent
4. AVR studio, supplied free by Atmel, is an excellent embedded IDE, and I am familiar with it
5. using multiple CPUs allows for a more modular design

So there. I've rationalized away my choice. Probably another reason would be that I'm much more familiar with AVR than ARM. On the downside, using this AVR based design means having to deal with communication and synchronization between multiple CPUs, which tends to be a process fraught with (enormous) difficulty.

But anyway, it's too late to change now (otherwise I'll never get this project finished). I'll keep udating this blog with my progress on this project, mainly as a way of keeping myself motivated. Hopefully, by this time next year there'll be plenty of photos of the finished synth up here, as well as a full set of schematics, firmware and pcb-layout.