Please check Part 1 here if you have not already. Part 2: The Payload (Python Jobs) Preparing the Data (files) Disco Distributed Filesystem (DDFS) is a great low-level component of Disco. DDFS is designed with huge data in mind, so it made more sense to use it in my experiment as opposed to any other type of storage, for example, HDFS. Moreover, we can even store job results in DDFS, which we are going to do below.
Prologue This post is my take on reviving an old project (the last commit was 3 years ago) born around 2007⁄2008 at Nokia Research Center and written in Erlang. What was exciting for me is the fact that Disco project is capable of running Python MapReduce Jobs against an Erlang core, how awesome is that! — Erlang is a synonym for parallel processing and high availability. I successfully built it though and ran a 250M records dataset which is 10GB+ in size using a Python MapReduce job that finished in 28 minutes (improved from 44 minutes) using a cluster of 3 EC2 free-tier t2.
This project is part of the Teensy LC Challenge Project 001: Blinking LED Setting up the environment The environment setup section is common to all the Teensy LC projects, so I created a single post with the required instructions to set up the Arduino IDE, Teensy Loader application, Teensyduino and Linux udev rules. Go to the complete setup guide First Usage: Blinky Program This section is also repeated in the setup guide since it’s basically the first project introduced in this quest.
Introduction Teensy LC (Low Cost) is a 32-bit microcontroller board that you can get from PJRC a company in Oregon, USA, which is owned and managed by Paul Stoffregen. It features an ARM Cortex-M0+ processor designed for low-power, low-cost devices. You can buy it and read more technical details on PJRC’s Teensy LC page. I am writing this tutorial as a complete guide for starting with the Teensy LC development and also as a reference for myself in case I need to revisit it in the future.
The issue Not so far ago, I had an issue with my Dell Inspiron 15 Series 3000 Laptop that I couldn’t connect to any WiFi network using the built-in wireless card and had to use a USB WiFi dongle. Turns out that Dell Inspiron 15 has a Broadcom series of PCI wireless cards on board. I was running Ubuntu 16.04 LTS at that time — I upgraded to 18.04 LTS (Bionic Beaver), the tips in this post would still work — and it didn’t support my WiFi card out of the box, I was almost certain it would be a driver issue even though I did some online search to get help but was guided by wrong directions by some of the online forums stating solutions like upgrading BIOS firmware, some Windows-related drivers updates and also some BIOS tweaks that worked for some people (probably they are not using Linux).
Prologue It was a great Friday, the 31st of August 2007 at 22:25:25 I etched my first PCB ever at home, it was the second year of high school and I was so excited about it. After doing some research and information gathering from some friends who happen to be great engineers. This is how I got started with electronics — believe it or not, by building my first PCB using Ferric Chloride FeCl3 solution with 60% concentration and a blank copper Printed Circuit Board (PCB).