You may have good reasons to focus on developing your disrupting app. And you may well be a very talented developer. But like me, even after years of experience as a web engineer, comes a point where we have to admit we don't know that much about security.

Unless you only use fully managed solutions to host your web applications, your startup probably has a bunch of servers you have to monitor and maintain. And unless your company has the scale to pay for real DevOps, security engineers, audits or pentests, then quite frankly, we'll mainly focus on finishing the current sprint hoping to get some traction ;)

Today I got my first Ubuntu VPS from OVH, a fresh new image ready to host the next version of my website. And it's got me thinking: I do remember it's best practice to disable SSH login by password, what else should I do ? Do I need a firewall ? Surely I don't have time to fully audit this default Ubuntu image by myself...

In this article I'll show you how to harden a default Ubuntu Server 20.04 image using existing open-source tools:

1. Inspec to identify security issues and misconfiguration
2. Ansible to automatically harden your server
3. The DevSec Hardening Framework which provides:
   - An Inspec profile
   - Ansible / Chef / Puppet recipes to enforce above Inspec profile

Quick SSH setup

Make sure your server is defined in ~/.ssh/config

Host servername
	HostName <your server ip address>
	User username
	Port 22

To ssh into the server with your ssh key without typing the password just run:

$ ssh-copy-id -i ~/.ssh/id_rsa.pub servername

Using Inspec with linux-baseline profile

On the server:

# download and install Inspec
$ wget https://packages.chef.io/files/stable/inspec/4.20.2/ubuntu/20.04/inspec_4.20.2-1_amd64.deb
$ sudo dpkg -i inspec_4.20.2-1_amd64.deb

# clone the linux-baseline profile
$ git clone https://github.com/dev-sec/linux-baseline

# run the Inspec profile
$ inspec exec linux-baseline

Inspec installation instructions

You should see a similar output:

Next thing will be to automatically apply better OS settings using Ansible and the recipes provided by the DevSec framework.

Using Ansible to harden the server

On your local machine:

# install Ansible
$ sudo apt update
$ sudo apt install software-properties-common
$ sudo apt-add-repository --yes --update ppa:ansible/ansible
$ sudo apt install ansible

# install the os and ssh hardening roles
$ ansible-galaxy install dev-sec.os-hardening
$ ansible-galaxy install dev-sec.ssh-hardening

Ansible installation instructions

We then need to write a playbook for each ansible role:

# ansible-os-hardening.yaml
- hosts: your-server
  become: true
  roles:
    - dev-sec.os-hardening
    
# ansible-ssh-hardening.yaml 
- hosts: your-server
  become: true
  roles:
    - dev-sec.ssh-hardening

Finally run these playbooks with the following commands:

$ ansible-playbook -K ansible-os-hardening.yaml
$ ansible-playbook -K ansible-ssh-hardening.yaml

Then, re-running inspec exec linux-baseline on the server should give a similar output:

Which is much better!

Final thoughts

Though I can't say I have audited the DevSec framework per-se, I hope you now have a better understanding on how you can automate your servers security to stay up-to-date whith industry best practices.

Depending on what you then run on your server, you may have to allow some ports on the Ubuntu firewall, ufw. Personally while testing CapRover I just had to run:

$ ufw allow 80,443,3000,996,7946,4789,2377/tcp; ufw allow 7946,4789,2377/udp;

Thanks for reading and take care !

Following my previous post GameDev with three.js on the modern web 🚀🎆, where I gave an overview of my three months journey into gamedev with three.js I wanted to share how I kept my code clean and organized, starting with the render loop.

One of the common culprit of developing any non trivial software project is to avoid spaghetti code. You always start with a nice single and simple page of code but soon enough it grows and grows and you have to keep it under control, you need to modularize your code, you need to organize and structure your project into a meaningful and comprehensible one.

1. How you render 3D things in three.js
2. Here comes the render loop
3. PubSubJS
4. A flexible and simple render loop
5. Final interactive demo

How you render 3D things in three.js

1. Setup the scene

You can probably skip this first part if you already touched three.js, sorry for the hello world explanation ;)

You just describe the content of your scene, with code, what you want to show on the screen. Let's take a simple example

import * as THREE from 'three'

// create a three.js scene
var scene = new THREE.Scene()

// create a camera
var camera = new THREE.PerspectiveCamera( 75, window.innerWidth / window.innerHeight, 0.1, 1000)
camera.position.set(50, 30, 50)
camera.lookAt(0, 0, 0)

// create the renderer
var renderer = new THREE.WebGLRenderer()
renderer.setSize(window.innerWidth, window.innerHeight)
document.body.appendChild(renderer.domElement)

// create a simple cube
var geometry = new THREE.BoxGeometry(20, 20, 20)
var material = new THREE.MeshLambertMaterial({color: 0x10a315 })
var cube = new THREE.Mesh(geometry, material)
scene.add(cube)

// add a light so we can see something
var light = new THREE.PointLight(0xFFFF00)
light.position.set(25, 25, 25)
scene.add(light)

2. Render the scene

We just have to call the renderer, passing it the 3D scene and the camera as arguments. It will compute the 2D representation of the 3D scene from the point of view of the camera and display that on the screen.

renderer.render(scene, camera);

You should now have rendered a cube! Here's a Codepen to show you the result.

See the Pen Hello world: a simple cube (1/3) by Maxime R. (@blaze33) on CodePen.

The catch is, it's a static image. To have an animated 3D environment at 60 fps you need to actually call renderer.render 60 times per second.

Here comes the render loop

If it's not your first three.js tutorial, you know that you have to code a loop that continuously calls renderer.render. And you know you need to use window.requestAnimationFrame() for that.

Here's the simplest example:

var mainLoop = () => {
  requestAnimationFrame(mainLoop)
  renderer.render(scene, camera)
}

mainLoop()

Now we could have a live animation of the static cube, not breathtaking, let's rotate the cube a little bit each time the loop is called to visualize that. Just add the following code in the mainLoop function

cube.rotation.x += Math.PI / 180
cube.rotation.y += Math.PI / 180
cube.rotation.z += Math.PI / 180

Pi being 180 degrees, it rotates the cube along each axis by one degree each frame. Try to add each of the three lines one by one to visualize how each axis of rotation is affected. Here's the demo:

See the Pen Hello world: a simple cube that rotates (2/3) by Maxime R. (@blaze33) on CodePen.

Maybe you see it coming now, you'll have dozens of animated 3D objects, some of them animated, others not, some appear, some disappear, some animations are fired upon user interaction, etc. The thing is: you cannot add all your logic in the mainLoop function!

The following is a presentation of how I structured my main loop function to keep it simple and flexible in droneWorld.

Let's take a detour to introduce the PubSubJS library I used to help me along the way.

PubSubJS

PubSubJS is a topic-based publish/subscribe library written in JavaScript.

The gist is, you declare a callback that will be executed when a message is published in PubSub. Why not just call the callback when needed then? Well you could have several different things happening when an event occurs, and they may not all be related. The PubSub architecture allows you to keep your code organized. Taking an example from droneWorld:

// in sounds/index.js
PubSub.subscribe('x.drones.gun.start', (msg, drone) => {
    // play sound
}

// in particles/index.js
PubSub.subscribe('x.drones.gun.start', (msg, drone) => {
    // send bullets
}

// in controls/index.js
// when mouse is clicked:
PubSub.publish('x.drones.gun.start', pilotDrone)

Now when we click the mouse, an event named x.drones.gun.start is fired with the pilotDrone object as a payload, the message subscribers are fired and a sound is played and bullets are drawn on the screen. This way the different parts of the code stay independent (e.g. you could still draw bullets without the sound system, easily) because the alternative would be to import every callback function in every module where it's needed and you quickly have a callback hell.

A flexible and simple render loop

Let's define a loops variable, it's an array containing animation functions. We initialize it with the animation we want at first.

As I implemented it, the loops array can contain functions or objects of the following form:

{
  id: 'myLoop',
  alive: true, // if false, the loop will be removed from the loops array
  loop: (timestamp, delta) => {doSomething(timestamp, delta)}
}

const rotateX = () => {
    cube.rotation.x += Math.PI / 180
}

const rotateY = () => {
    cube.rotation.y += Math.PI / 180
}

const rotateZ = () => {
    cube.rotation.z += Math.PI / 180
}

let loops = [
  rotateX,
  rotateY
]

Let's define some helpers to add or remove loops to the loops variable.

const removeLoop = (loop) => {
  loops = loops.filter(item => item.id !== loop.id)
}
// declare a subscriber to remove loops
PubSub.subscribe('x.loops.remove', (msg, loop) => removeLoop(loop))
// declare a subscriber to add a loop
PubSub.subscribe('x.loops.push', (msg, loop) => loops.push(loop))
// declare a subscriber to add a loop that will be executed first
PubSub.subscribe('x.loops.unshift', (msg, loop) => loops.unshift(loop))

const cleanLoops = () => {
  loops.forEach(loop => {
    if (loop.alive !== undefined && loop.alive === false && loop.object) {
      scene.remove(loop.object)
    }
  })
  loops = loops.filter(loop => loop.alive === undefined || loop.alive === true)
}

Let's declare stats.js here. It's a naive helper to show the FPS.

const stats = new Stats()
document.body.appendChild(stats.dom)

Let's declare a subscriber that will allow us to start and stop the animation at will.

let play = true
PubSub.subscribe('x.toggle.play', () => { play = !play })

Now we declare the mainLoop function.

let lastTimestamp = 0
var mainLoop = (timestamp) => {
  requestAnimationFrame(mainLoop)
  let delta = timestamp - lastTimestamp
  lastTimestamp = timestamp

  if (play) {
    loops.forEach(loop => {
      loop.loop ? loop.loop(timestamp, delta) : loop(timestamp, delta)
    })
    
    renderer.render(scene, camera)
  }

  cleanLoops()

  stats.update()
}

mainLoop(0)

And so we have a simple and flexible mainLoop function under 15 lines of code!

Final interactive demo

Look at the code to see how easy it is now to add or remove animations from the scene.

if (someCondition) {
  // starts the cube rotation
  PubSub.publish('x.loops.push', rotateX)
} else {
  // stops the cube rotation
  PubSub.publish('x.loops.remove', rotateX)
}

See the Pen Hello world: a simple interactive cube (3/3) by Maxime R. (@blaze33) on CodePen.

That's it! Thanks for reading!
Check droneWorld to see three.js in action:

Launch Demo Code on Github Star Follow @maxmre

Back in November 2017 I started a journey in the world of game development using the modern JavaScript toolchain and three.js. This article is kinda a summary of the useful bits accumulated during ~3months of part-time work showing what it is to start from scratch with only “classic” web development experience and to end with below game prototype. This was done in the open using only open source tools, this is not your usual Unity or Unreal tutorial ;) Let's see what can be done:

Launch Demo Code on Github Star Follow @maxmre

Detailed followups:

• Three.js & 3D interactive animations: a tutorial

Summary

1. Three.js? WebGL? What is this?
2. Assets: A free world map on AWS
3. Assets bis: textures
4. Assets ter: free 3D models
5. Shaders
6. Particles
7. Marketing
8. Missing parts
9. What next?

Three.js? WebGL? What is this?

Three.js is a library that helps you build 3D interactive environments right into your browser by leveraging the power of the WebGL standard. The rendering is hardware-accelerated and it requires no plugin and no software installation, it's just a webpage.

Assets: A free world map on AWS

A huge ressource I stumbled upon while browsing Hacker News is the terrain tiles dataset hosted by Amazon. You can get PNGs containing elevation data, like these ones:

256x256 PNGs covering the whole world at various zoom levels. It doesn't look like anything at first, you have to extract the red, green and blue values (integers between 0 and 255) of each pixel to compute the elevation value using the following formula:

(red * 256 + green + blue / 256) - 32768

I used UPNG.js to decode the images obtained via the fetch API, inside a web worker.
Let's say we build a plane with 256x256 vertices and assign the elevation value to each vertex, we can now build mountains with real world data!

Then you just have to juxtapose several terrain tiles, combining different zoom levels and you can have your own 3D version of google maps:

I believe there's a huge potential of applications having such incredible datasets available in the open, gaming being just one possible application.

Assets bis: textures

I did not use a lot of textures but a good ressource where I found free high-resolution and high quality seamless textures is poliigon. They have some free textures of wood, rocks, grass, terrain, etc.

Assets ter: free 3D models

Let's go to Sketchfab, a company co-founded by the French guy Trigrou. There's a huge collection of 3D models, some of them available under the glTF format under permissive licences. The glTF format allows for an easy import by thrre.js using its glTF importer, you get the model and its textures in one pass.

Shaders

Shaders are small programs written in GLSL (openGL Shading Language) designed to run directly, and fastly, on the graphics card allowing you to compute all sorts of graphical effects. As of 2018, webGl allows you to program two types of shaders, vertex shaders, which can manipulate the vertices of your models and fragment shaders which work on each pixel of a frame and output their color.

A good introduction to learn shaders is The book of shaders.

I used shaders to texture the terrain differently depending on the slope of the terrain for example:

The depth of field effect is also done by shaders though I mostly reused existing code in this case.

Particles

For smoke, explosions, sparks, you need a particle engine. Luckily there's one for three.js: Shader Particle Engine

Even with such libraries it might be tricky to achieve decent and efficient results but it's worth the pain!

Marketing

I used my twitter account @maxmre as a cheap devlog to build a (small) following.

I used the /r/gamedev subreddit and their weekly threads to get feedback and guidance.

I tried the itch.io platform to post droneWorld on https://maxmre.itch.io/droneworld. Though the platform is great for indie gamedevs, it wasn't a match for droneWorld as you have to post a working game to get some traction, it's not really suited for hosting a prototype under construction. Itch.io can also host your devlog but in my case that would have been redundant with what I was doing on twitter.

As I was publishing the game on https://lab.openbloc.fr/droneWorld each time I was making some progress I kept the process quite agile by having to have a working build every day.

You must be conscious that the game development process is a pretty specific topic. You may get the attention of your fellow developers but that's not necessarily where your end users will be.

Missing parts

Mobile controls, desktop controls, shaders, texturing, modeling, packaging your javascript, sound environment, game logic, gameplay, bots programming, multiplayer aspects, game feel, performance bottlenecks, what I mainly (re)discovered is that game programming covers a lot of topics, actually it's a whole industry where each part I touched can be a fulltime job inside a big team.

What next?

What topic should I cover to expand this article? Use the comments below, tweet @maxmre or write at max at openbloc.fr to let me know!

Thanks for reading :)

What is Electric Sheep ? Why would I want this thing ?

Do Androids Dream of Electric Sheep?¹

If you've never heard of Electric Sheep, it is a collaborative abstract artwork.

When computers "sleep", the Electric Sheep comes on and the computers communicate with each other by the internet to share the work of creating morphing abstract animations known as "sheep".

Anyone watching one of these computers may vote for their favorite animations using the keyboard. The more popular sheep live longer and reproduce according to a genetic algorithm with mutation and cross-over. Hence the flock evolves to please its global audience. [...]

You can design your own sheep and submit them to the gene pool. The result is a collective "android dream", blending man and machine with code to create an artificial lifeform.

Installation for Windows or Mac OS X

Just go to the download page.

Installation on Linux

There aren't precompiled binaries for Ubuntu, in fact you're told to install it from source aka. to compile it. It's been a long time I wanted to try Electric Sheep but I never got around to compile it. Eventually I did it and the following is a summary of the steps I followed.

The README instructions are pretty succinct but I managed to get it working.

1. Install system dependencies

sudo apt install subversion autoconf libtool libgtk2.0-dev libgl1-mesa-dev libavcodec-dev libavformat-dev libswscale-dev liblua5.1-0-dev libcurl4-openssl-dev libxml2-dev libjpeg8-dev libgtop2-dev libboost-dev libboost-filesystem-dev libboost-thread-dev libtinyxml-dev freeglut3-dev glee-dev
sudo apt install libwxgtk3.0-0v5 libwxgtk3.0-dev

2. Build flam3

Flam3 is a dependency of Electric Sheep, we need to install it.

git clone git@github.com:scottdraves/flam3.git
cd flam3/
./configure
make

Now make will probably fail, the fix is to edit Makefile to put the correct version of the failing rules (replaced 1.14 by 1.15 in my case):

ACLOCAL = aclocal-1.14    # -> 1.15
AUTOMAKE = automake-1.14  # -> 1.15

make

3. Build a .deb package for flam3!

checkinstall is a nice tool that we will use to replace the sudo make install step. It builds a .deb package that makes installation and uninstallation easy:

sudo apt install checkinstall
sudo checkinstall

You can fill all asked questions with the default values, except for the package version that has to be valid (I used 3.1.1). Once the .deb is built, just install it:

sudo dpkg -i flam3_3.1.1-1_amd64.deb

4. At last: build Electric Sheep

git clone git@github.com:scottdraves/electricsheep.git
cd electricsheep/client_generic/
./autogen.sh
./configure
make
sudo checkinstall
sudo dpkg -i client-generic_2.7b33-svn-1_amd64.deb

5. Enjoy!

$ electricsheep
$ electricsheep-preferences

Thanks for reading !

[1] Do Androids Dream of Electric Sheep? is a science fiction novel by American writer Philip K. Dick, first published in 1968.

Following Part 1: introduction & toolchain setup where we saw how to install the required toolchain and start a new app using the Phoenix framework, we'll see how to implement a collaborative todo app with realtime updates.

Todolists aren't something new, in fact, this post is mostly about how I used the Phoenix, React, Redux example article to get something that works.

Prerequisites: having installed Elixir and Phoenix, a reasonable understanding of how React and Redux work.

The code is available at:

0.1 Boostrapping the app

We will create a backend Elixir app and a React front-end who will communicate though Websockets.

$ mix phx.new todo_app --module Todo --app todo --no-html --no-brunch

Edit config/dev.exs with appropriate Postgresql credentials and run:

$ mix ecto.create

0.2 Creating a task API

The phoenix framework comes with code generators, we will use phx.gen.json to generate json views backed by Ecto. As we want a list of tasks for our Todo app, let's proceed:

$ mix phx.gen.json API Task tasks label:string

As you can see, the generator gives us a Model, View and a Controller. Add the ressource to the :api scope as indicated:

Run mix ecto.migrate and launch the server with mix phx.server and open a webpage localhost:4000.

Success ! We get an empty page, the database not being yet populated.

0.3 Doing a Database migration with Phoenix

As you may have noticed, the Task model does not have a field indicating its completion, that's perfect to learn how to do a database migration ! Ecto has a generator for this:

$ mix ecto.gen.migration add_tasks_completed_field

This create an empty Ecto migration in priv/repo/migrations, let's complete it:

defmodule Todo.Repo.Migrations.AddTaskCompletedField do
  use Ecto.Migration

  def change do
    alter table(:tasks) do
      add :completed, :boolean, default: false, null: false
    end
  end
end

And finally run mix ecto.migrate, that's it !

1.1 Build a front-end

The focus of this article not being on how to build a React-Redux app, I'll skip the boring part. Moreover you could download a ready todo app from the Redux examples and just adapt it.

1.2 Connect the Javascript to Elixir though Websockets

Phoenix delivers a phoenix.js well suited to its Channels.

$ yarn add phoenix

import { Socket } from 'phoenix'

export function configureChannel() {
  const socket = new Socket('ws://127.0.0.1:4000/socket');
  socket.connect();
  const channel = socket.channel('tasks');
  return channel
}

If you get 404 errors in the developer tools you may need to configure your Endpoint url host to 127.0.0.1 instead of localhost even though check origin is disabled in dev mode. God knows why, cf. this stackoverflow answer, it almost drove me crazy!

1.3 Connect your redux actions with the Elixir channel

On the client side, the idea is two sided:

• Subscribe to server messages and dispatch redux actions accordingly.
• Push messages to the server on relevant actions.

For example:

// dispatch an update when receiving one from the server
channel.on('update:todo', todo => {
  console.log('update:todo', todo);
  dispatch(updateTodoSuccess(todo));
});

// Below function is called on a component click
// We don't bother dispatching an action here, the server
// will answer with a `delete:todo` message, that will
// update the Redux store.
export function deleteTodo(todo) {
  return dispatch => {
    const payload = { id: todo.id}
    channel.push('delete:todo', payload)
  }
}

1.4 Adapt the backend to talk to the client

Once again there's a generator to create a Channel:

$ mix phx.gen.channel API
* creating lib/todo_web/channels/api_channel.ex
* creating test/todo_web/channels/api_channel_test.exs

Add the channel to your `lib/todo_web/channels/user_socket.ex` handler, for example:

    channel "api:lobby", TodoWeb.APIChannel

The created Channel contains already some code we'll adapt to our use case.

# Channels can be used in a request/response fashion
# by sending replies to requests from the client
def handle_in("ping", payload, socket) do
  {:reply, {:ok, payload}, socket}
end

# It is also common to receive messages from the client and
# broadcast to everyone in the current topic (todo:lobby).
def handle_in("shout", payload, socket) do
  broadcast socket, "shout", payload
  {:noreply, socket}
end

We have two things to do:

• Send a list of the tasks once the client connects
• Have a CRUD interface over websocket

You may think you'll get Task.list_tasks(), serialize it to JSON and send it to the client but the more pythonic way (sorry ;) ) is to use the already created JSON views to do the serialization:

def join("api:lobby", payload, socket) do
  todos = API.Task.list_tasks()
  {:ok, %{tasks: TodoWeb.TaskView.render("index.json", tasks: tasks)}, socket}
end

As for the CRUD interface:

def handle_in("new:todo", payload, socket) do
  {:ok, created} = TodoTest.Todos.create_todo(payload)
  created_json = TodoTestWeb.TodoView.render("todo.json", todo: created)
  broadcast! socket, "new:todo", created_json
  {:reply, {:ok, created_json}, socket}
end

def handle_in("update:todo", payload, socket) do
  todo = TodoTest.Todos.get_todo!(payload["id"])
  TodoTest.Todos.update_todo(todo, payload)
  broadcast! socket, "update:todo", payload
  {:noreply, socket}
end

def handle_in("delete:todo", payload, socket) do
  todo = TodoTest.Todos.get_todo!(payload["id"])
  TodoTest.Todos.delete_todo(todo)
  broadcast! socket, "delete:todo", payload
  {:noreply, socket}
end

And there you have it!

Conclusion

In short I was blown away by how easily the code generators work for you and how Phoenix integrates with websockets through channels. The backend code is very succinct and once you understand how it works, you feel quite productive.
Thanks for reading :)

Photo by Glenn Carstens-Peters on Unsplash

This tutorial started as a brain dump of my first steps in the world of Erlang, Elixir, Phoenix, etc. I thought it might be useful to share it.

Part 2 : collaborative todolist with realtime updates

0.1 What is this ?

• Erlang is a general-purpose, concurrent, functional programming language
• Elixir is a dynamic, functional language designed for building scalable and maintainable applications. Elixir runs on the Erlang virtual machine (BEAM).
• Phoenix is a web framework for Elixir.
• Hex is a package manager for the Erlang ecosystem.

Both Elixir 1.5 and Phoenix 1.3 were released in July 2017 so you may want to give it a fresh look.

0.2 Some links I found useful

• Elixir Nation centralizes the most useful resources
• The Elixir Forum the go-to place to get community support
• What kind of virtual machine is BEAM (the Erlang VM)? [stackoverflow.com]
• Hacker news comments: Elixir 1.5 released
• Hacker news comments: Phoenix 1.3 released
• Why the Elixir language has great potential
• Algolia search on Hacker News for 'Elixir'

0.3 What is this article ?

I had no previous experience with Elixir and I'll show you how to get from this:

A fresh Ubuntu 17.10 install. To this:

A Phoenix test application "hello world" running on localhost.

So there's nothing special with the rest of this article but I wouldn't say there's no value in having a step by step documentation of the toolchain setup, especially if you're new to this. I know it can be tedious to track the different steps required with 15 open tabs, not everything being always up-to-date.

In case I forgot to document some steps, just tell me and I'll update the article

1.1 Installing Erlang and Elixir

Cf. https://elixir-lang.org/install.html. For Ubuntu:

$ wget https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb && sudo dpkg -i erlang-solutions_1.0_all.deb

For Ubuntu 17.10, the Erlang artful repositories aren't yet available, so you'll have to edit /etc/apt/sources.list.d/erlang-solutions.list to point to the zesty repository (which works just fine):

$ cat /etc/apt/sources.list.d/erlang-solutions.list 
### THIS FILE IS AUTOMATICALLY CONFIGURED ###
# You may comment out this entry, but any other modifications may be lost.
deb http://binaries.erlang-solutions.com/debian zesty contrib
$ sudo apt update
$ sudo apt install esl-erlang elixir

Now we can check Elixir is correctly installed:

$ iex -v
Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [kernel-poll:false]

IEx 1.5.2

1.2 Install phoenix and create a test application

To install the new phx.new project generator, use the following command:

$ mix archive.install https://github.com/phoenixframework/archives/raw/master/phx_new.ez

Use it to generate a new project:

$ mix phx.new todoapp
$ cd todoapp
$ mix deps.get
$ cd assets && npm install && node node_modules/brunch/bin/brunch build

1.3 Install Postgresql and create a test database

Your test application is configured to use Postgresql by default. Well need to create test database and get some auth credentials to finalize the Phoenix app setup.

$ sudo apt install postgresql postgresql-contrib
# Postgresql was already installed on my machine so I may have skipped some steps.
$ createuser --interactive
Enter name of role to add: dev                 
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) y
Shall the new role be allowed to create more new roles? (y/n) n
$ createdb testapp_dev
$ psql
psql (9.6.5)
Type "help" for help.
# ALTER USER dev WITH PASSWORD 'password';
# GRANT ALL PRIVILEGES ON DATABASE testapp_dev to dev;
# \q

Then edit config/dev.exs accordingly:

username: dev,
password: password,

And run the following command:

$ mix ecto.create

For your information, Ecto is a database wrapper and language integrated query for Elixir.

1.4 Launch the Phoenix app

If everything is correctly setup, the following should sufficient:

$ mix phx.server

If you get an error message regarding inotify-tools, just install it:

$ sudo apt install inotify-tools

Finally open the website:

$ firefox http://0.0.0.0:4000/

Conclusion

Now we have setup everything to start some Elixir & Phoenix programming like in these examples:

• Phoenix, React, Redux example: real-time collaborative TODO app
• Elixir blog in 15 minutes using Phoenix framework - Step by Step

Those examples are from 2015, I'll try do something up-to-date for my next article, when I get some time... and some more inspiration :)

Thanks for reading !

Part 2 : collaborative todolist with realtime updates

Ghost is a wonderful blogging platform but what if you wanted to include something dynamic ? Nothing more easy !

<canvas id="universe"></canvas>
<script type="text/javascript"  src="https://lab.openbloc.fr/way-of-life/js/lab-bundle.js?380a213f">Enable javascript please</script>

The Ghost preview window will show you the following:

Demo of the game of life, click to add new cells:

Thanks for reading !

Photo by Max McKinnon on Unsplash

My Webassembly 101 tutorial got ~50000 page views after hitting the front page of hacker news. I got a $12 USD S3 bill in June from Amazon. In this tutorial I will show you how to get a free CDN and HTTPS everywhere with Cloudflare.

This tutorial is written for an Heroku hosted Ghost blog but it should be pretty straightforward to use it for other hosting options.

What is a CDN and why would you need it ?

A content delivery network or content distribution network (CDN) is a geographically distributed network of proxy servers and their data centers. The goal is to distribute service spatially relative to end-users to provide high availability and high performance.

In short:

1. Hosting costs reduction
2. Better performance for end users
3. Encrypted traffic for more security

Summary

1. Initial mistakes
2. Fix 1: setup Clouflare in front of your website
3. Fix 2: update all articles with lower resolution images and CDN urls
4. Fix 3: Free HTTPS
5. Cached traffic results
6. Conclusion, TLDR: 75% costs reduction
7. Bonuses

Initial mistakes

1. No CDN.
2. Having hosted high resolution images on S3.
3. No HTTPS, leading to lost referrers.

   A Referrer header is not sent by browsers if [...] an unsecured HTTP request is used and the referring page was received with a secure protocol (HTTPS).

Errors 1 and 2 led to a 12$ bill for the month of June 2017, ~100+Gb of data transferred at $0.09/Gb.

Error 3 led to bad analytics because most referrers are not set.

Fix 1: setup Clouflare in front of your website

Cloudflare has a nice on-boarding process once you register for the free account.

Main steps are:

1. Update the DNS server on you registrar with the ones Cloudflare gives you on the CDN tab:
   
   On your registrar dashboard you just have to copy paste above urls:
   
2. Setup an S3 bucket with a cdn compatible name: cdn.example.com.s3.eu-central-1.amazonaws.com and replace example.com by your website's name. See the documentation for a more detailed explanation.

If your domain is "example.com" and you want to use the CNAME "files" you'll need to make sure the S3 bucket name is "files.example.com". Amazon requires that the CNAME match the bucket name.

3. Create a cdn CNAME record for the above CDN bucket url.
   
4. After 24h the DNS have updated and I was able to access my S3 bucket content through cdn.openbloc.fr.

Fix 2: update all articles with lower resolution images and CDN urls

Pretty straightforward as the title said. Do not store 4000x3000px images ! Use a lower resolution and compress the jpeg images as much as quality permits.

Fix 3: Free HTTPS

Cloudflare offers a nice service: Cloudflare one-click SSL.

If you don't have an SSL certificate on your origin server, the available free option is Flexible SSL:

Flexible SSL

Flexible SSL encrypts traffic from Cloudflare to end users of your website, but not from Cloudflare to your origin server. This is the easiest way to enable HTTPS because it doesn’t require installing an SSL certificate on your origin. While not as secure as the other options, Flexible SSL does protect your visitors from a large class of threats including public WiFi snooping and ad injection over HTTP.

Enabling it is just one click away:

Of course, if you already have an SSL certificate on your server, go for the Full SSL option!

Cached traffic results

We got ourselves a free CDN and a free HTTPS setup without using Heroku paid SSL. I'll pay next to nothing on my next S3 bill :)

As you can see below 84% of the traffic is now cached:

Conclusion

1. June 1st-30th AWS bill: $12.32 USD with 51896 pages viewed.
2. July 1st-31th AWS bill: $0.20 USD with 3056 pages viewed.

The hosting cost went down from $0.24 USD per 1000 pages viewed to $0.06 per 1000 pages viewed. That's a 75% cost reduction on your hosting bill, for free !

Thanks for reading !

Bonuses:

1. HTTPS everywhere:

   Redirect all requests with scheme "http" to "https". This applies to all http requests to the zone.

   

2. Opportunistic Encryption

   Opportunistic Encryption allows browsers to benefit from the improved performance of HTTP/2 and SPDY by letting them know that your site is available over an encrypted connection. Browsers will continue
   to show "http" in the address bar, not "https".

3. Automatic HTTPS Rewrites

   Automatic HTTPS Rewrites helps fix mixed content by changing "http" to "https" for all resources or links on your web site that can be served with HTTPS.

   

Attention: this article is probably not researched enough for now. There'll probably never be a JavaScript boilerplate, there's simply too many of them! In the meantime you can read my other articles but if you'd like to see how to build your own little starter kit, read ahead!

Starting a new project can be tiresome because the toolchain setup can be quite time consuming. The web is full of outdated webpack tutorials, just give me something that works ! Instead of starting from scratch each time, I started to reuse the common parts of my projects. The following is the result of this process.

I present you the ES2015 (EcmaScript2015/ES6) starter kit:

The learning curve shouldn't be too steep and it is best suited to quickstart small projects. I had simplicity in mind while developing it.

Requirements

• git, a distributed version control system.
• node.js, includes npm.
• Optionally yarn.

Starting a new JavaScript project

It's very easy, simply clone and edit the project:

# clone the repository
git clone git@github.com:blaze33/es6-starter.git my-app
cd my-app
# install the npm packages of the dependencies
# if you don't have yarn installed just npm install
yarn
# start webpack-dev-server
npm run serve

Now you should have your browser open a new tab that will autoreload once you start editing the files in the src folder.

Happy coding !

Features

ES2015 starter uses:

• webpack to bundle the JavaScript app.
• node-sass and sass-loader to compile the SCSS files to CSS.
• webpack-dev-server, a development server that provides live reloading.
• Babel with preset env to compile ES2015 JavaScript down to a supported version.
• Linting check with standard.js.
• normalize.css, makes browsers render all elements more consistently and in line with modern standards.
• purify-cssremoves unused css.
• surface, a Material Design CSS only framework.

Conclusion

Ideally I would like for this starter project to be a live one that would be updated and improved over time.

Contributions, pull requests, questions and comments are welcome:
blaze33/es2015-starter

Thanks for reading !

Appendix

If you're curious how I made the above gif, I used recordmydesktop on Ubuntu 17.04. Install it with:

sudo apt install recordmydesktop

Then I edited and converted the resulting video with ffmpeg:

# install ffmpeg
sudo apt install ffmpeg
# cut the first 2 and last 2 seconds of the video
ffmpeg -ss 2 -i es2015-starter-demo.mp4 -t 88 -c copy es2015-starter-demo-cut.mp4
# generate a custom png palette to use in the final gif 
ffmpeg -y -i es2015-starter-demo-cut.mp4 -vf fps=10,scale=800:-1:flags=lanczos,palettegen palette.png
# generate the final gif
ffmpeg -i es2015-starter-demo-cut.mp4 -i palette.png -filter_complex 'fps=1,scale=800:-1:flags=lanczos,setpts=0.25*PTS[x];[x][1:v]paletteuse' es2015-starter-demo.gif

That's it !

Yesterday, my WebAssembly 101 tutorial hit a pretty sweet spot on Hacker News, raising to the #1 one and getting more than 500 points.

If you read Hacker News, you've probably seen many websites go down when they hit the front page, with people resorting to Google cache, when available, to access the content.

So when I saw that my submission started appearing on the front page after 6 or 7 upvotes (yes that's all you need!) I had a moment of doubt and went straight to Google Analytics to see how much traffic was coming. Answer: a lot !

Google Analytics five minutes after hitting the frontpage:

Sure, this is a first world problem but I never had so much traffic, my website isn't a static one, I wasn't using a CDN and I had no idea how my server would handle the load.

The Ghost platform hosted on Heroku

My first blog post was about how I set up this website using Ghost.

I'm hosted on Heroku on a single free dyno, you can't really get a smaller instance. So, how did it perform ?

To my surprise I didn't even had to scale up my app ! My sole instance flawlessly served all the requests without a single error.

At a mean of 235rpm day, we get:
235rpm.day x 60min/hour x 24hr/day = 338400 requests

At 7 requests per page for openbloc.fr (that could surely be optimized)
we get ~48000 pages viewed.

The traffic peaked at ~650rpm, or ~100 pages per minute but the CPU usage didn't go over 20% and the RAM usage as 250MB.

A big shout out to the Ghost and Heroku teams is in order.

How to benchmark your site ?

A cool tool is Apache Benchmark. On Ubuntu you can install it with:

$ sudo apt install apache2-utils

And then to request 100 pages, 10 pages at a time:

$ ab -n 100 -c 10 http://www.example.com/you-post-url/

If it takes less than a minute to complete you should be on the safe side :)

That's it !

How many people use an ad-blocker ?

Beware that the Hacker News audience is pretty tech-savvy so we should get a higher percentage not representative of the general population.

I guess I'll ask Google Analytics how many people don't use an ad-blocker ;)

27677 page viewed says GA, 48342 says my server, so I'll say:

• 43% use an ad-blocker,
• 57% don't.

Conclusion

I spent a lot of time hacking this weekend to get the WebAssembly 101 tutorial possible so I knew the content could be interesting for someone starting in WebAssembly but I would never have expected such a success, it's really overwhelming !

On a personal note, after having gone through some rough times at the beginning of this year, it feels good to be back on track !

Happy hacking !

This tutorial will guide you along the necessary steps to port a JavaScript library of the Conway's game of life to WebAssembly (wasm). This is a simple exercise that is perfect to start beyond a trivial Hello World.

I recently got interested in WebAssembly and decided to take the leap this weekend. WebAssembly is an emerging standard to enable near-native performance for web applications. Basically it's asm.js done right as stated by @kripken (Dec. 9th 2015). WebAssembly is still a moving target, with a lot of developments going on. Getting started turns out to be difficult as most of the available information is quickly becoming outdated.

I went through the awesome-wasm list which is a good starting point but still had to work two days on this to get some working code.

There's a demo of the game of life re-implemented in wasm at the end of the article :)

The following tutorial was written using Ubuntu 17.04, so your mileage may vary. Assume no knowledge of WebAssembly as I wrote this starting from scratch but I won't detail the ES6 with webpack toolchain. There's a lot more available resources for this on the web. Try finding a more up-to-date tutorial like this one.

This article has five parts:

• Setup the toolchain
• Javascript integration
• Beyond Hello World: optimizing a game of life engine
• Benchmarking (demo link is there)
• Conclusion

Setup the toolchain

The provided packages are a little outdated, I got some warnings. After spending some time installing the latest LLVM build it appeared that the easiest way was to download and install the Portable Emscripten SDK for Linux and OS X (emsdk-portable.tar.gz).
Extract the archive and open a terminal in the folder.

$ ./emsdk update
$ ./emsdk install latest

Now depending on your network speed go make yourself a coffee or read a book.

The Emscripten SDK provides the whole Emscripten toolchain (Clang, Python, Node.js and Visual Studio integration) in a single easy-to-install package, with integrated support for updating to newer SDKs as they are released.

So we should have everything necessary to start coding some WebAssembly.
Once the installation is done, activate the sdk:

$ ./emsdk activate latest
$ source ./emsdk_env.sh  # you can add this line to your .bashrc

Make some sample C file counter.c:

int counter = 100;

int count() {
    counter += 1;
    return counter;
}

Compile it to wasm with emcc:

$ emcc counter.c -s WASM=1 -s SIDE_MODULE=1 -o counter.wasm

And, tada ! We have a beautiful counter.wasm.

JavaScript integration

A standalone .wasm file won't do anything by itself, we need to load it in some client javascript code. I'm using webpack along with wasm-loader to this end. Refer to the documentation for a more vanilla JavaScript example. Ok, let's do this:

import Counter from './wasm/counter'
const wasmHelloWorld = () => {
    const counter = new Counter();
    console.log("count function result is : " + counter.exports._count());
}
window.onload = wasmHelloWorld

Loading this code in a sample html page should print 101 in the console. Except it doesn't. In Firefox 53 you should get a LinkError: import object field 'DYNAMICTOP_PTR' is not a Number instead. What went wrong? I got stuck on this an entire evening, then came StackOverflow to the rescue.

Let's get back to the code, we need to compile the C code with an optimization flag:

$ emcc counter.c -O1 -o counter.wasm -s WASM=1 -s SIDE_MODULE=1

Now when we do a new Counter(), wasm-loader calls new WebAssembly.Instance(module, importObject);

• module is a correct WebAssembly.Module instance.
• importObject is the default provided by wasm-loader which appears to not work.

The reasons are a little obscure but editing the JavaScript to the following code solves the issue:

import Counter from './wasm/counter'
const wasmHelloWorld = () => {
    const counter = new Counter({
      'env': {
        'memoryBase': 0,
        'tableBase': 0,
        'memory': new WebAssembly.Memory({initial: 256}),
        'table': new WebAssembly.Table({initial: 0, element: 'anyfunc'})
      }
    })
    console.log("count function result is : " + counter.exports._count());
}
window.onload = wasmHelloWorld

Now, reloading the webpage succeeds !

As you can see it wasn't that straightforward to get a simple hello world to work. In the following section we'll see an easier way to integrate JS and wasm.

Beyond Hello World: optimizing a game of life engine

While upping my skills in ES6, webpack, babel, etc. I made a little implementation of the game of life. The code is available at blaze33/way-of-life.

The game engine has a double loop iterating over the whole game grid at each step. Though I tried hard to keep it fast, it gets quickly slow once you increase the grid size. With our newfound WebAssembly skills it could be a nice exercise to try running the core game engine as a wasm module.

What needs to be done ?

• Re-implement the game logic in C.
• Compile the C logic to wasm.
• Expose the wasm code in the JS one.
• Have a way to interact between the C and JS code.

We won't go full WebAssembly for now and have the rendering to the canvas done in WebAssembly for now.

Compile C to WASM with some JS-glue code

Notice how we compiled the previous example with -s SIDE_MODULE=1 ? This provides a single wasm module that we have to integrate from scratch in the client code. You should know that it doesn't allow for malloc calls in the C code for example. Not really a problem for a hello world but pretty much a big no-no once you try doing more complex stuff. Fortunately you can compile the C code and have emscripten provide a wasm module AND a JS module that serves as a glue to integrate the WebAssembly in the client code. In our case, it will allow us to make malloc calls and have a way to read the allocated memory from the JS side.

The compilation is done as follow:

emcc engine.c -O3 -o engine.js -s WASM=1 -Wall -s MODULARIZE=1

By setting MODULARIZE we put all the JS output into a function. Unfortunately it's not really a JS module (AMDdefine, CommonJS nor ES6) so we'll just append export {Module as default} to engine.js, webpack will do the rest and allow us to import the Module in our ES6 client code:

import Module from './wasm/engine.js'
module = Module({wasmBinaryFile: 'wasm/engine.wasm'})

You have to specify the extension in the import as there is an engine.wasm in the same folder.
wasmBinaryFile is the url used to asynchronously fetch the wasm code, so we tell webpack to serve it using copy-webpack-plugin.

Keep this JS module in mind, we'll reuse it later.

Calling WASM functions from JavaScript

By default the C functions are not exposed by emscripten (or maybe not always, correct me if I'm wrong), we need to tell it to do so:

#include <emscripten.h>

EMSCRIPTEN_KEEPALIVE
char *init(int w, int h) {
    width = w;
    height = h;
    current = malloc(width * height * sizeof(char));
    next = malloc(width * height * sizeof(char));
    return current;
}

EMSCRIPTEN_KEEPALIVE does exactly this and we can now call module.asm._init(40, 40) if we wanted to initialize the game with a 40x40 grid.

All the exposed C funtions are available in module.asm and are prefixed with an underscore.

Accessing the wasm module memory from JS

Emscripten conveniently exposes the module memory through module.HEAP* variables. The recommended way to interact with the memory is with module.getValue and setValue. As this is slower, I'll pursue directly with accessing HEAP8, considering the state is in a char array. Beware: accessing undocumented properties will probably break in the future!

Now we have pretty much all the pieces, I worked hard to piece all this together so let's proceed to the speedup benchmarking with the demo!

Benchmarking

Benchmarking is always tricky and I shouldn't probably use this word for looking at an Hello World performance so that the following shouldn't be used to judge wasm performance. The C code is not optimized to be fast but written like the naïve JS implementation I used. That being said we can still have a look to see if the result goes faster than the JS implementation.

I did some performance profiling on Chrome 58.
This is the original JS code:

And this is the wasm code:

On average the computeNextState which took ~40ms now runs in ~15ms, not orders of magnitude faster but enough to get from ~18FPS to ~40FPS on my laptop.

The improvements were less visible on Firefox 53 as the FPS varied a lot, but it is still present.

• Demo up to 60FPS, 5 pixels per cell
• Demo up to 60FPS, 1 pixel per cell

You can play with the url options and also switch the wasm engine to the js one for comparison, have fun!

Conclusion

• Starting this was much harder than I envisioned!
• Webassembly looks really promising but the toolchain feels a bit heavy and clunky at times.
• The documentation is both sparse or too technical but that should improve over time.
• Having the emscripten glue code is really necessary for now, even if it seems to add another layer. I initially thought we could get away interfacing directly with the wasm code but I couldn't.
• Still pretty happy with the result.
• The code is available at blaze33/way-of-life

Thanks for reading ! If you liked this article you can follow me at @maxmre for future posts, or you could star the github repo, leave a comment or, you know, just ignore this internet-points-mania, I won't be mad ;)

Also thanks to Stackoverflow and Hacker News for helping me along the way!

Hacker News discussion thread.

Useful resources

• Awesome wasm
• Interacting with code (between JS and C/C++)
• MDN documentation
• Why WebAssembly is a game changer for the web — and a source of pride for Mozilla and Firefox
• Epic games Zen Garden Demo (125MB download!)

The following is not a success story. Pretty atypical for an article topic but I thought it might be of some interest to show you the work of a failed startup. Before starting your own project it's always beneficial to get some feedback from others who tried and, not always, succeeded before you.

The NewCo project

Back in 2012-2013, four people, including myself, launched a startup project called NewCo, as in the New Company. The idea was to offer a place where consumers would exchange information about the products they use and love thus empowering you to make better and faster buying decisions.

No prior experience in the web: four engineers from the same school, a very good one though, but with very little experience in what building a web startup would mean.

No traction: while I'm sure this space is still promising for new startups, in such a B2C configuration you would need a lot of traction to get things moving. We failed miserably at that, the product wasn't compelling enough to attract readers and contributors.

No money: we had the idea that once a user had the information he wanted before buying a product, we would just have to show him available resellers in order to get money through affiliate marketing. Well, this way, we made 30€ in a year.

Co-founders disagreements: trust me, don't wait one year to talk about who owns how much of the company. Maybe you don't have to incorporate right away but have at least a written agreement of some sort.

A working website: at least being engineers was useful for this.

The development was mostly done by AGASS007 and me (blaze33). The stack is old, mainly built around Django 1.4: Python 2.7, Heroku cedar-14 (upgraded from cedar-10), redis, postgresql, celery.

Advice to future entrepreneurs

1. Technology matters as long as it doesn't get in the way of all the rest.
2. Design, UX, marketing, communication matters a lot ! Don't call yourself NewCo, as in New Company, really ;)
3. People don't read, much less contribute on the Internet. You can't build your success on asking them to do stuff for you. Build it around what they would naturally do with the right tool (and build that one).
4. Choose your co-founders wisely. If your co-founder show up one day with another associate without having consulted you first, run ! Trust is everything.
5. Making money is a necessary evil, I've got a rent to pay, food to buy. The sooner you can break even, the better.

Resuscitating the app

Long story short, the original website is still running and I wanted to replace a feedback form that no longer worked by some comment about the startup story. Problem: I could no longer run the website locally and my code push were rejected by heroku. Two days of rabbitholing later I managed to rebuild and deploy the app.

Some technical details: You know what happens when you don't touch a project for 4 years: it no longer runs!

All in all there was some work, mainly updating dependencies, without breaking the app, to have it run locally first.

Pushing to heroku no longer worked too as they updated their cedar-10 stack to cedar-14 (Ubuntu 10 vs Ubuntu 14 mainly).

How to upgrade from the Cedar-10 stack to the Cedar-14 stack.

Still pushing to heroku failed. Turns out I was using an old and no longer supported python version. Adding a runtime.txt with python-2.7.13 fixed it.

Another fun thing that can happen in 4 years is that some packages are no longer available on pypi, or not at the version you need. Chasing github repos and installing from them solved it.

Opensourcing the code

The only useful thing I could still do:
newco-legacy

Conclusion

I worked one and a half year on this project when the other co-founders decided to go their own way and incorporated their own company without me but still reused all my code and simply told me to sue them if I wasn't happy.

Comes the lawyer who aptly counselled that it would cost more than I would ever get back considering they weren't making any money.

Still I'm grateful for this experience as it gave me a solid technical background that I was able to reuse by working for other more successful startups.

Thanks for reading !

Sooo... I've decided that after owning openbloc.fr for some years it was time to start sharing some of my whereabouts.

Getting Started

When I started openbloc as my personal website, though I experimented with Flask and Angular, it was mostly a static website crafted by hand.
I've spent some time investigating static website generators, hosting on S3 or on Github Pages

In the end it's still a lot of work to setup and understand those frameworks.

Comes Ghost

And then I remembered the ghost platform, which is a little bit more with "batteries included". It seems to have come a long way since 2013 with a 1.0.0 release almost ready.

I used ghost-on-heroku to deploy it.

1. Create a new S3 bucket
2. Create a new IAM role with S3 permissions
3. Use the provided ID and Key to fill the heroku form
4. Deploy !

For some reason the deploy from github did not copy the source code in the heroku container so I had to fork and clone it for this as I wanted to update the dependencies.

Adding the custom blog.openbloc.com domain

At this point ghost was only running at openbloc-blog.herokuapp.com to add a custom domain several steps are required:

1. Add a domain on the heroku interface
2. Add a CNAME record at your DNS provider using the heroku-provided DNS target.
3. Adding a Let's Encrypt certificate for a secure https connection: probably later as I would have to upgrade to a paid dyno for now.

Customizing the default theme: Casper

I've just added prism.js to do code highlighting. Also a Makefile to generate the theme zip file:

zip:
	zip -r openbloc.zip . -x .git\*

It should work with a lot of languages out of the box, here's some test with a Django model:

class BaseModel(models.Model):
    """ BaseModel
    An abstract base class with timestamps.
    Stores data with postgresql JSON field.
    """
    date_created = models.fields.DateTimeField(
        verbose_name=_("creation date"),
        auto_now_add=True,
    )
    date_modified = models.fields.DateTimeField(
        verbose_name=_("modification date"),
        auto_now=True,
    )
    data = JSONField(null=True, blank=True)

    class Meta:
        abstract = True

That's all !