Tools on When Moore's Law Ends

Status of current HDL generator

Mon, 15 Jun 2020 00:00:00 +0000

Recently, I’m trying out different HDL generator languages and tools, because

Chisel is used heavily inside SiFive, and they have developed amount of IPs including very complicated CPUs, and a very well maintained community (mainly students from Berkeley)
Many other IP companies are using their type of HDL generator to generate RTL
1. Such as ARM and Cadence. ARM’s CPU is delivered as a very well parameterized Verilog file along with a set of scripts that can take your configuration file can generate RTL/testbench/IPXACT/… accordingly in less than a minute.
HDL generator is pushed inside our team in FB, and believed to be the future of hardware design because of boost of productivity

The productivity boost of generator is coming from

shortcut.py A Command Line Script to Provide Macro

Mon, 11 Feb 2019 00:00:00 +0000

Typical usage#

`go` to some directory use short alias#

> pwd
/home/jw
> go prj0
cd /work/projects/design/master-branch/
> pwd
/work/projects/design/master-branch/

`run` a serial of commands use short alias#

> run vnc
ssh -L 5901:remote-internal-server-name:5901 jw@remote-gateway-ip
vncviewer remote-internal-server-name

Search available alias#

Here . dot is used to replace * star because zsh will try to expand * by default.

> go
prj0
> go p.
prj0
> run
vnc

Auto-correction#

shortcut.py uses difflib.get_close_matches to auto-correct the alias.

Static Site Generator

Sat, 19 Jan 2019 00:00:00 +0000

Trying to find the perfect static site generator. Used to use Pelican, because it’s written in Python. Also tried with Jekyll, the most popular candidate, because it’s used by Github. Their common problems are

Not intuitive enough. Seems like something programmer created for programmer.
There are very restricted requirements of directory structure of the content. And they doesn’t fit my understanding/requirement.

Now I’m trying Hugo, which is written in Go, and it’s really fast. The best feature I like about it is page bundles. It allows you to put articles in the same/separated directory with their own resources.

GCP (Google Cloud Platform)

Sun, 21 Oct 2018 00:00:00 +0000

Price of GCP#

Persistance Disk#

Can be used to put all the data/eda/os on it.

	Price (per month)	Price (per GB per month)
SSD 50GB	$8.50	$0.17
SSD 1TB	$174.08	$0.17
HDD 50GB	$2.00	$0.04
HDD 200GB	$8.00	$0.04
HDD 1TB	$39.76	$0.04
Snapshot 50GB	$1.30	$0.026

Always on instance (24x7)#

Can be used as working machine (support VNC) and NFS server

	vCPU	RAM (GB)	Price (per month)
f1-micro	Shared	0.60	Free
g1-small	Shared	1.70	$13.80
n1-std-1	1	3.75	$24.27
n1-std-2	2	7.50	$48.55

Instance for EDA (10 hours per week, 43.452 hours per month)#

	vCPU	RAM (GB)	Price (per month)	Price (per hour)
n1-cpu-8	8	7.2	$12.32	$0.2835
n1-std-8	8	30	$16.51
n1-mem-8	8	52	$20.58
n1-cpu-16	16	14.4	$24.65	$0.5673
n1-std-16	16	60	$33.02	$0.7599
n1-cpu-32	32	28.8	$49.29	$1.1344

At the meantime, secondhand server on unixsurplus is “E5-2620 V3 10-CORE 2.4GHz” with “64GB DDR4” and “300GB HDD” is $1,989. It’s n1-std-16 for 2617 hours, 261.7 weeks if 10 hours per week.

Docker for EDA

Mon, 01 Oct 2018 00:00:00 +0000

Dockerfile#

FROM ubuntu:16.04
COPY ./boot.sh /tmp
COPY ./hello.sh /tmp
RUN /bin/bash /tmp/boot.sh
RUN /bin/bash /tmp/hello.sh

boot.sh#

apt-get update; apt-get install -y make autoconf g++ flex bison wget
cd /tmp
wget https://www.veripool.org/ftp/verilator-4.004.tgz
tar xf verilator*.tgz
cd verilator*
./configure
make # this step will take sometime
make install

hello.sh#

cd /tmp/verilator*
cd ./example/tracing_c
make

# it will print out some log, and after finish it will generate a directory in the same directory named "logs" who needs to be saved after container exit

Docker how-to notes#

create a docker image from Dockerfile#

docker build -t "$(name):$(tag)" .; # in a directory with Dockerfile

docker run -dit --name $(name) $(name):$(tag); # -d: detach, -i: interactive, -t: tty

docker ps; # use -a to list exited docker; use "docker rm" to remove useless containers

docker exec -it $(name) /bin/bash; # run bash in guest and connect to it

docker commit -m "install/test verilator-4.004" -a "jw" $(name) $(new_name):$(new_tag); # -m: message, -a: author, with new name and tag

docker image ls; # and the image is 797MB large comparing to eda:0.0's 116MB; use "docker image rm" to remove useless docker images

save file to host#

# mount a host dir to guest /mnt/result
DATE=`date '+%Y%m%d_%H%M%S'`
mkdir result_${DATE}
docker run -it -v ${PWD}/result_${DATE}:/mnt/result eda:0.1 /bin/bash

# in guest, copy files to /mnt/result afterwards
# but the files in result_${DATE} will have permission of root:root
# the easiest way to fix the problem is in host do:
sudo chown -R $(id -u) ./result_${DATE}
sudo chown -R $(id -g) ./result_${DATE}

stop#

docker stop $(name)

Course Note of Python Design Patterns

Wed, 17 May 2017 00:00:00 +0000

The course is on Coursera

understanding design patterns#

what?
- well-known solutions for recurring problems
why?
- don’t reinvent wheels
- reuse best practices
characteristics
- language neutral
- dynamic
- incomplete by design to promote customization
types
- creational
- structual
- behavioral
OOP (object-oriented programming)
- design pattern relies on OOP
- attributes + methods
- inheritence & polymorphism
pattern context
- participants and their rols
- quality attributes: nonfunctional requirements
- forces: various factors or trade-offs to consider
- consequences: side effects by decision makers
pattern language
- name
- context
- problem
- solution
- related patterns

creational patterns#

factory#

create and return object on the fly
- flexibility of adding new classes
problem
- uncertainties in types of objects
- runtime decision of types of objects

abstract facotry#

a factory dynamically uses different factories to create objects on the fly
problem
- create a family of related objects at runtime
concepts
- abstract factory
- concrete factory (the real factory used at runtime)

singleton#

only one object from a class
problem
- cache information shared by multiple objects

Scala First Look

Sun, 14 May 2017 00:00:00 +0000

Scala introduction course on LinkedIn. Not very useful, if not using it in real project.

introduction#

short for Scalable language
object-oriented + functional programming
- everything is object
  - including numbers and functions
  - does not include primitive data types
- can pass/assign function: functional programming
seamless compatible with Java
- runs on JVM, compile to JVM
- reuse Java libraries and types
- can call any Java code
REPL interactive shell
- read, eval, print, loop

syntax#

define: var (variable) vs val (value)
- var defines variables that can be changed
  - but cannot change from one data type to another, ex. from Int to Double
- val defines variables that cannot be changed (immutable)
  - such as string, char
TIPS: worksheet
- while save, output result of every line of code to source file as comment
underscore _ is a special char in Scala, so it cannot be used in variable name

data type#

Any
- AnyVal
  - Int
  - Double
  - Boolean
  - …
- AnyRef
  - String
  - Vector
  - Array
  - …
Null
Nothing

repetition statements#

while
for
- iterate a list: for (f <- lsFruit)
  - unpacking: for ( (a, b) <- lsFruit )
- filtering
  - for (f <- lsFruit if (f.startswith("c")))
  - for (n <- 1 to 10; e = n % 2; if e == 0)
- nested for loop: for (i <- 1 to 5; j <- 1 to 5)
  - to in Scala is different from Python: 1 to 5 gives 1, 2, 3, 4, 5
  - until in Scala is the same with Python: 1 until 5 gives 1, 2, 3, 4

function vs method#

function is a complete object, that can be assigned to a variable
method is a part of a class which has a name ans signature
void function: return Unit

special function#

call by value vs call by name

def something() = {
 5 // return value
}
de callByValue(x: Int) = { // something is run once, and pass return value "5" to x
 println("x1 = " + x)
 println("x2 = " + x)
}
callByValue(something())
def callByName(x: => Int) = {
 println("x1 = " + x) // something ran once
 println("x2 = " + x) // something ran again
}
callByName(something())

anonymous function

((x: Int) => x + 1)(5) //> res0: Int = 6

high order function

var y = 5
val f = (x: Int) => (x + y) // here, y is going to use y=5 which already exists

decision#

if
match = case/switch

set and map#

both immutable
set: automatically unique
map: key to value pairs
- var m = Map("key1"->value1, "key2"->value2)
- add new item: `m += “key3”->value3
- remove item: m -= "key1"

array#

index by numbers from 0
var a = Array[String](6)

list#

immutable
concatenate use :::

class#

case class

singleton: object#

object: create a anonymous class and one of its instance

WSL (Windows Subsystem Linux) Tips

Tue, 25 Apr 2017 00:00:00 +0000

To start gnome-terminal on WSL (Windows Subsystem for Linux)#

After upgrade to Windows 10 Creators Update, reinstall WSL will have Ubuntu 16.04.2 LTS on Windows.

To reinstall WSL you should do:

> lxrun /uninstall /full
> lxrun /install

Then you will see

> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial

Some X11 apps like xclocks works fine with Xming, but “gnome-terminal” still have some problem.

Install Python Offline

Mon, 24 Apr 2017 00:00:00 +0000

In general, things like Anaconda Server are designed to make this sort of workflow easier.

Some suggested workarounds:

Reproduce your install on another machine with internet (save conda list –export to a file and conda install –file). Then use conda update on that machine to generate and download the packages. One tip that could be helpful is if you take several conda packages and tar them up into a flat tarball you can use conda to install that tarball and it will install all the packages. Or you can put the packages in the pkgs cache and use the –offline flag.
Install the newer Anaconda to a different location and copy the packages from the pkgs cache to your current folder.
Both of these are a bit clunky, but there’s no easy way to achieve this without Anaconda Server.

Python and Freemind ElementTree

Sun, 19 Jul 2015 00:00:00 +0000

我曾经自己用Python写过一个小工具来parse Freemind文件（XML格式）然后生成RestructureText和Latex格式。这个小工具的目的是为了实践我的“源文件唯一”的理念。因为我的简历（包括既往项目总结）需要保存为两种不同格式，一个为了放在个人网站上所以是HTML，另一个当然是PDF格式。

今天瞎琢磨的时候发现，其实Python对于XML文件格式支持的极好。The ElementTree XML API这个模块将XML parse成一个Tree。

Each element has a number of properties associated with it:
 a tag which is a string identifying what kind of data this element represents (the element type, in other words).
 a number of attributes, stored in a Python dictionary.
 a text string.
 an optional tail string.
 a number of child elements, stored in a Python sequence

Python的Markdown也能够utilize这个ElementTree模块。它能够将Markdown文件Parse成eTree（也能够在之前和之后加上其他的process，以callback的形式存在）。

A simple system metric collector

Mon, 01 Jan 0001 00:00:00 +0000

https://github.com/jimwang99/jimon

While experimenting my little distributed storage system built with Raspberry Pi SBCs and Ceph, some Raspberry Pi machines die now and then. My suspicion is heat, because I built them with cases, even though the cases come with attached heat sinks and fans. Therefore I built this small open-source project to help me collect system metrics periodically on Linux and send them to a server. The server store the metrics in a SQLite3 database, so that I can use a Python script to pull them out and check my small servers’ status and if they died what is the cause.

Ceph on Raspberry Pi (1) Create Storage Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Trying to follow this tutorial to create a Ceph distributed storage system in my home lab, because I’m sick of the NFS performance of my Synology NAS. https://ceph.io/en/news/blog/2022/install-ceph-in-a-raspberrypi-4-cluster/

Prepare#

Prepare Ubuntu 20.04#

Download ubuntu server 20.04 for Raspberry Pi (arm64 architecture)

sudo apt install -y python3 systemd docker.io chrony lvm2 cephadm ceph-common

Prepare USB storage#

Install Ceph use `cephadm`#

`bootstrap` on the admin host#

On admin host

Here my admin host’s IP address is 192.168.68.65

Ceph on Raspberry Pi (2) Create Block Device

Mon, 01 Jan 0001 00:00:00 +0000

Taking my home lab Ceph distributed storage system on Raspberry Pi to the next level: make it useful by creating a block device interface so that Linux system can mount it and use it.

Enable client#

Install Ceph package on client

sudo apt install ceph-common

Generate config and keyring on admin host

sudo ceph config generate-minimal-conf
sudo ceph auth get-or-create client.fs

Copy generated config file and keyring file from admin host to client. Config file is at /etc/ceph/ceph.conf, keyring file is at /etc/ceph/ceph.client.admin.keyring

Create Ceph block device#

Create pool for RBD (RADOS Block Device)#

On admin host:

Monorepo vs. Microrepo (ByteByteGo)

Mon, 01 Jan 0001 00:00:00 +0000

All the credit goes to ByteByteGo.com

Do you believe that Google, Meta, Uber, and Airbnb put almost all of their code in one repository?

This practice is called a monorepo.

Monorepo vs. Microrepo. Which is the best? Why do different companies choose different options?

Monorepo isn’t new; Linux and Windows were both created using Monorepo. To improve scalability and build speed, Google developed its internal dedicated toolchain to scale it faster and strict coding quality standards to keep it consistent.

Tools on When Moore's Law Ends

Status of current HDL generator

shortcut.py A Command Line Script to Provide Macro

Typical usage#

go to some directory use short alias#

run a serial of commands use short alias#

Search available alias#

Auto-correction#

Static Site Generator

GCP (Google Cloud Platform)

Price of GCP#

Persistance Disk#

Always on instance (24x7)#

Instance for EDA (10 hours per week, 43.452 hours per month)#

Docker for EDA

Dockerfile#

boot.sh#

hello.sh#

Docker how-to notes#

create a docker image from Dockerfile#

save file to host#

stop#

Course Note of Python Design Patterns

understanding design patterns#

creational patterns#

factory#

abstract facotry#

singleton#

Scala First Look

introduction#

syntax#

data type#

repetition statements#

function vs method#

special function#

decision#

set and map#

array#

list#

class#

singleton: object#

WSL (Windows Subsystem Linux) Tips

To start gnome-terminal on WSL (Windows Subsystem for Linux)#

Install Python Offline

Python and Freemind ElementTree

A simple system metric collector

Ceph on Raspberry Pi (1) Create Storage Cluster

Prepare#

Prepare Ubuntu 20.04#

Prepare USB storage#

Install Ceph use cephadm#

bootstrap on the admin host#

Ceph on Raspberry Pi (2) Create Block Device

Enable client#

Create Ceph block device#

Create pool for RBD (RADOS Block Device)#

Monorepo vs. Microrepo (ByteByteGo)

`go` to some directory use short alias#

`run` a serial of commands use short alias#

Install Ceph use `cephadm`#

`bootstrap` on the admin host#