Thursday, 17 January 2019

Git-Notes

To delete a branch

Local branch: git branch -d <local_branch> Remote branch: git push origin --delete <remote_branch>

To create new branch from existing branch

git checkout -b <new_branch_name> <source_branch_name>

To push my branch to remote server

git push -u origin <my_branch>

To view commit history logs

git reflog

To reset to particular commit log

git reset HEAD@{N}
*N is target number of commit log to reset to.

To show last git commit message on current checkout branch

git show --summary

To revert commits in certain branch to particular commit

git checkout <branch>
git reset --hard <commit-hash>
git push -f origin <branch>

To fetch all git branches

git pull --all

To create local branch to tracks remote branch

git checkout --track origin/john_branch
where --track is shorthand for git checkout -b [branch] [remotename]/[branch]

To reset local branch to match latest remote branch

git fetch origin
git reset --hard origin/master
git clean -fd

To rename a branch

git branch -m old_branch new_branch         # Rename branch locally    
git push origin :old_branch                 # Delete the old branch    
git push --set-upstream origin new_branch   # Push the new branch, set local branch to track the new remote

Successful git branching

Ref: https://nvie.com/posts/a-successful-git-branching-model/

Creating a feature branch

git checkout -b myfeature develop

Incorporating a finished feature on develop

git checkout develop
git merge --no-ff myfeature
.
.
.
git branch -d myfeature
git push origin develop
 
 

Using Git

Global Settings

Related Setup: https://gist.github.com/hofmannsven/6814278
Related Pro Tips: https://ochronus.com/git-tips-from-the-trenches/
Interactive Beginners Tutorial: http://try.github.io/

Reminder

Press minus + shift + s and return to chop/fold long lines!
Show folder content: ls -la

Notes

Do not put (external) dependencies in version control!

Setup

See where Git is located: which git
Get the version of Git: git --version
Create an alias (shortcut) for git status: git config --global alias.st status

Help

Help: git help

General

Initialize Git: git init
Get everything ready to commit: git add .
Get custom file ready to commit: git add index.html
Commit changes: git commit -m "Message"
Add and commit in one step: git commit -am "Message"
Remove files from Git: git rm index.html
Update all changes: git add -u
Remove file but do not track anymore: git rm --cached index.html
Move or rename files: git mv index.html dir/index_new.html
Undo modifications (restore files from latest commited version): git checkout -- index.html
Restore file from a custom commit (in current branch): git checkout 6eb715d -- index.html

Reset

Go back to commit: git revert 073791e7dd71b90daa853b2c5acc2c925f02dbc6
Soft reset (move HEAD only; neither staging nor working dir is changed): git reset --soft 073791e7dd71b90daa853b2c5acc2c925f02dbc6
Undo latest commit: git reset --soft HEAD~
Mixed reset (move HEAD and change staging to match repo; does not affect working dir): git reset --mixed 073791e7dd71b90daa853b2c5acc2c925f02dbc6
Hard reset (move HEAD and change staging dir and working dir to match repo): git reset --hard 073791e7dd71b90daa853b2c5acc2c925f02dbc6

Update & Delete

Test-Delete untracked files: git clean -n
Delete untracked files (not staging): git clean -f
Unstage (undo adds): git reset HEAD index.html
Commit to most recent commit: git commit --amend -m "Message"
Update most recent commit message: git commit --amend -m "New Message"

Branch

Show branches: git branch
Create branch: git branch branchname
Change to branch: git checkout branchname
Create and change to new branch: git checkout -b branchname
Rename branch: git branch -m branchname new_branchname or: git branch --move branchname new_branchname
Show all completely merged branches with current branch: git branch --merged
Delete merged branch (only possible if not HEAD): git branch -d branchname or: git branch --delete branchname
Delete not merged branch: git branch -D branch_to_delete

Merge

True merge (fast forward): git merge branchname
Merge to master (only if fast forward): git merge --ff-only branchname
Merge to master (force a new commit): git merge --no-ff branchname
Stop merge (in case of conflicts): git merge --abort
Stop merge (in case of conflicts): git reset --merge // prior to v1.7.4
Merge only one specific commit: git cherry-pick 073791e7

Stash

Put in stash: git stash save "Message"
Show stash: git stash list
Show stash stats: git stash show stash@{0}
Show stash changes: git stash show -p stash@{0}
Use custom stash item and drop it: git stash pop stash@{0}
Use custom stash item and do not drop it: git stash apply stash@{0}
Delete custom stash item: git stash drop stash@{0}
Delete complete stash: git stash clear

Gitignore & Gitkeep

About: https://help.github.com/articles/ignoring-files
Useful templates: https://github.com/github/gitignore
Add or edit gitignore: nano .gitignore
Track empty dir: touch dir/.gitkeep

Log

Show commits: git log
Show oneline-summary of commits: git log --oneline
Show oneline-summary of commits with full SHA-1: git log --format=oneline
Show oneline-summary of the last three commits: git log --oneline -3
Show only custom commits: git log --author="Sven" git log --grep="Message" git log --until=2013-01-01 git log --since=2013-01-01
Show only custom data of commit: git log --format=short git log --format=full git log --format=fuller git log --format=email git log --format=raw
Show changes: git log -p
Show every commit since special commit for custom file only: git log 6eb715d.. index.html
Show changes of every commit since special commit for custom file only: git log -p 6eb715d.. index.html
Show stats and summary of commits: git log --stat --summary
Show history of commits as graph: git log --graph
Show history of commits as graph-summary: git log --oneline --graph --all --decorate

Compare

Compare modified files: git diff
Compare modified files and highlight changes only: git diff --color-words index.html
Compare modified files within the staging area: git diff --staged
Compare branches: git diff master..branchname
Compare branches like above: git diff --color-words master..branchname^
Compare commits: git diff 6eb715d git diff 6eb715d..HEAD git diff 6eb715d..537a09f
Compare commits of file: git diff 6eb715d index.html git diff 6eb715d..537a09f index.html
Compare without caring about spaces: git diff -b 6eb715d..HEAD or: git diff --ignore-space-change 6eb715d..HEAD
Compare without caring about all spaces: git diff -w 6eb715d..HEAD or: git diff --ignore-all-space 6eb715d..HEAD
Useful comparings: git diff --stat --summary 6eb715d..HEAD
Blame: git blame -L10,+1 index.html

Releases & Version Tags

Show all released versions: git tag
Show all released versions with comments: git tag -l -n1
Create release version: git tag v1.0.0
Create release version with comment: git tag -a v1.0.0 -m 'Message'
Checkout a specific release version: git checkout v1.0.0

Collaborate

Show remote: git remote
Show remote details: git remote -v
Add remote origin from GitHub project: git remote add origin https://github.com/user/project.git
Add remote origin from existing empty project on server: git remote add origin ssh://root@123.123.123.123/path/to/repository/.git
Remove origin: git remote rm origin
Show remote branches: git branch -r
Show all branches: git branch -a
Compare: git diff origin/master..master
Push (set default with -u): git push -u origin master
Push to default: git push origin master
Fetch: git fetch origin
Fetch a custom branch: git fetch origin branchname:local_branchname
Pull: git pull
Pull specific branch: git pull origin branchname
Merge fetched commits: git merge origin/master
Clone to localhost: git clone https://github.com/user/project.git or: git clone ssh://user@domain.com/~/dir/.git
Clone to localhost folder: git clone https://github.com/user/project.git ~/dir/folder
Clone specific branch to localhost: git clone -b branchname https://github.com/user/project.git
Delete remote branch (push nothing): git push origin :branchname or: git push origin --delete branchname

Archive

Create a zip-archive: git archive --format zip --output filename.zip master
Export/write custom log to a file: git log --author=sven --all > log.txt

Troubleshooting

Ignore files that have already been committed to a Git repository: http://stackoverflow.com/a/1139797/1815847

Security

Hide Git on the web via .htaccess: RedirectMatch 404 /\.git (more info here: http://stackoverflow.com/a/17916515/1815847)

Large File Storage

Website: https://git-lfs.github.com/
Install: brew install git-lfs
Track *.psd files: git lfs track "*.psd" (init, add, commit and push as written above)
 

How to delete a commit from git


Using Rebase this will allow you to remove one or more consecutive commits

Example git log

Number
Hash
Commit Message
Author
1
2c6a45b
(HEAD) Adding public method to access protected method
Tom
2
ae45fab
Updates to database interface
Contractor 1
3
77b9b82
Improving database interface
Contractor 2
4
3c9093c
Merged develop branch into master
Tom
5
b3d92c5
Adding new Event CMS Module
Paul
6
7feddbb
Adding CMS class and files
Tom
7
a809379
Adding project to Git
Tom

Using Rebase

Using the git log above we want to remove the following commits; 2 & 3 (ae45fab & 77b9b82). As they are consecutive commits we can use rebase.

git rebase --onto <branch name>~<first commit number to remove> <branch name>~<first commit to be kept> <branch name>

e.g to remove commits 2 & 3 above

git rebase --onto repair~3 repair~1 repair


Delete Commit History in Git Repository

 

Follow the below steps to complete this task.
Warning: This will remove your old commit history completely, You can’t recover it again.
  • Create Orphan Branch – Create a new orphan branch in git repository. The newly created branch will not show in ‘git branch’ command.
    $ git checkout --orphan temp_branch
    
  • Add Files to Branch – Now add all files to newly created branch and commit them using following commands.
    $ git add -A
    $ git commit -am "the first commit"
    
  • Delete master Branch – Now you can delete the master branch from your git repository.
    $ git branch -D master
    
  • Rename Current Branch – After deleting the master branch, let’s rename newly created branch name to master.
    $ git branch -m master
    
  • Push Changes – You have completed the changes to your local git repository. Finally, push your changes to remote (Github) repository forcefully.
    $ git push -f origin master
    

 

Short Form of commands:

cd Repo 
rm -rf .git
git init
git remote add origin git@github.com:JohnDoe/foobar.git
git remote -v
git add --all
git commit -am "Initial commit"
git push -f origin master
 
 

 

First Method

Deleting the .git folder may cause problems in our git repository. If we want to delete all of our commits history, but keep the code in its current state, try this:

# Check out to a temporary branch:
git checkout --orphan TEMP_BRANCH

# Add all the files:
git add -A

# Commit the changes:
git commit -am "Initial commit"

# Delete the old branch:
git branch -D master

# Rename the temporary branch to master:
git branch -m master

# Finally, force update to our repository:
git push -f origin master

This will not keep our old commits history around. But if this doesn't work, try the next method below.

Second Method

# Clone the project, e.g. `myproject` is my project repository:
git clone https://github/heiswayi/myproject.git

# Since all of the commits history are in the `.git` folder, we have to remove it:
cd myproject

# And delete the `.git` folder:
git rm -rf .git

# Now, re-initialize the repository:
git init
git remote add origin https://github.com/heiswayi/myproject.git
git remote -v

# Add all the files and commit the changes:
git add --all
git commit -am "Initial commit"

# Force push update to the master branch of our project repository:
git push -f origin master

NOTE: You might need to provide the credentials for your GitHub account.

 

 

Wednesday, 24 October 2018

Linux Kernel Cross Compilation

Introduction

There are several reasons to cross-compile the Linux kernel:
  1. Compiling on a native CPU and hardware requires a wide range of devices, which is not so practical. Furthermore, the actual hardware is often not suitable for the workload of kernel compilation due to a slow CPU, small memory, etc.
  2. Hardware emulation (e.g., with qemu) can be a viable substitution if the slowness can be tolerated. However, how to setup the compilation environment is another challenge, as it requires at least a preferably Linux-flavored OS, the bash and make ecosystem, and most importantly, the toolchain.
  3. Cross-compiling on a powerful host enables quick detection of kernel compile errors and config errors, as shown in the stable queue builds project. Coupled with emulation, testing on non-native architectures becomes easier as well.
For my personal use, I would like to see the kernel build process on each architecture, capture the compilation flags of each files, collect their linking information, and finally see if I can mine some insights out of it. In this case, given the complexity of parsing the Kconfig and Makefile, probably the best way is to actually build the kernel and dump the verbose version of the build log.
Fortunately, with a modern Linux release (like Ubuntu or Fedora), all we need to cross-compile the kernel is a toolchain that can produce binaries for another CPU architecture.

Toolchain

On cross-compiling the kernel, we only need two things: binutils, and gcc, and there are generally two ways to obtain them:
  1. install pre-compiled packages
  2. build from source
Installing pre-compiled packages can be a hassle-free way if you don't want to get your hands dirty. In fact, for the following architectures: arm, aarch64, hppa, hppa64, m68k, mips, mips64, powerpc, powerpc64, s390x, sh4, sparc64, you can directly obtain them via apt-get from the official repositories. For example,
apt-get install binutils-aarch64-linux-gnu gcc-aarch64-linux-gnu
Toolchains for other architectures such as blackfin, c6x, and tile can be obtained from the Fedora repo and converted to .deb packages as shown in the tutorial in 2013.
However, in case you want to compile the toolchain from source, for whatever reasons you might have, such as using the latest version or a customized version of gcc, you might follow the following steps.
As a head up, I am running a standard Ubuntu 16.04.3 LTS release with kernel version 4.4.0. The host gcc and binutils (i.e., the gcc and binutils used to build the toolchain) are the default ones with the Ubuntu release, which is in version 5.4.0 and 2.27 respectively. However, I don't see major obstacles in applying it to other combinations of OS, gcc, and binutils versions as long as they are recent enough to build the toolchain.
Before diving into the building process, let's setup some environment variables. Part of them are for convenience reasons, while others are necessary in the build process.
export TARGET=aarch64-unknown-linux-gnu     # replace with your intended target
export PREFIX="$HOME/opt/cross/aarch64"     # replace with your intended path
export PATH="$PREFIX/bin:$PATH"
The TARGET variable should be a target triplet which is used by the autoconf system. Their valid values can be found in the config.guess script.
The last step is necessary. By adding the installation prefix to the the PATH of the current shell session, we ensure the gcc is able to detect our new binutils once we have built them.

binutils

The source code for binutils can be obtained from the GNU servers:
# for stable releases
BINUTILS_VERSION=2.29.1
wget ftp://ftp.gnu.org/gnu/binutils/binutils-$BINUTILS_VERSION.tar.xz

# for development branch
git clone git://sourceware.org/git/binutils-gdb.git
After that, modify the source code as you wish and build it with the following steps.
cd $BINUTILS_BUILD

$BINUTILS_SOURCE/configure \
  --target=$TARGET \
  --prefix=$PREFIX \
  --with-sysroot \
  --disable-nls \
  --disable-werror

make
make install
Since we have multiple targets to build, it is better to have a separate build directory for each target.
--disable-nls tells binutils not to include native language support. This is optional, but reduces dependencies and compile time.
--with-sysroot tells binutils to enable sysroot support in the cross-compiler by pointing it to a default empty directory. By default the linker refuses to use sysroots for no good technical reason, while gcc is able to handle both cases at runtime.
--disable-werror behaves exactly as the name suggests, do not add -Werror in the flag.
After installation, the binaries aarch64-unknown-linux-gnu-{as/ar/ld/...} should exist in $PATH.
These instructions apply to binutils version 2.29.1, which is the latest release at the time of writing.

gcc

Similar to binutils. the source code for gcc can be obtained from the GNU servers too:
# for stable releases
GCC_VERSION=7.2.0
wget ftp://ftp.gnu.org/gnu/gcc/gcc-$GCC_VERSION/gcc-$GCC_VERSION.tar.xz

# for development branch
git clone git://gcc.gnu.org/git/gcc.git
After that, modify the source code as you wish and build it with the following steps.
cd $GCC_SOURCE
./contrib/download_prerequisites

cd $GCC_BUILD
$GCC_SOURCE/configure \
  --target=$TARGET \
  --prefix=$PREFIX \
  --enable-languages=c,c++ \
  --without-headers \
  --disable-nls \
  --disable-shared \
  --disable-decimal-float \
  --disable-threads \
  --disable-libmudflap \
  --disable-libssp \
  --disable-libgomp \
  --disable-libquadmath \
  --disable-libatomic \
  --disable-libmpx \
  --disable-libcc1

make all-gcc
make install-gcc 
Downloading the pre-requisites (gmp, mpfr, mpc, isl) areis necessary as gcc needs these packages for compilation. This is handled seamlessly with the download_prerequisites script.
--enable-languages=c,c++ tells gcc not to build frontends for other languages like Fortran, Java, etc.
--without-headers tells gcc not to rely on any C library being present for the target.
--disable-<package> tells gcc not to build those packages as they will not be needed in kernel compilation.
More importantly, we should not simply make all as that would build way too much (for example, the libgcc, libc, libstdc++, etc), which are not needed at all. All we need is the compiler itself which can be built by make all-gcc.
Another important note is that in order for gcc to lookup the correct set of binutils, both $TARGET and $PREFIX must be exactly the same when configuring binutils and gcc. For example, aarch64-unknown-linux-gnu-gcc will lookup aarch64-unknown-linux-gnu-as in the same directory: merely putting aarch64-unknown-linux-gnu-as in $PATHis not enough.
The instructions apply to gcc version 7.2.0, which is the latest release at the time of writing.
After the whole process, the following files should present in the $PREFIX/bin directory, and also in the $PATH.
aarch64-unknown-linux-gnu-addr2line
aarch64-unknown-linux-gnu-ar
aarch64-unknown-linux-gnu-as
aarch64-unknown-linux-gnu-c++
aarch64-unknown-linux-gnu-c++filt
aarch64-unknown-linux-gnu-cpp
aarch64-unknown-linux-gnu-elfedit
aarch64-unknown-linux-gnu-g++
aarch64-unknown-linux-gnu-gcc
aarch64-unknown-linux-gnu-gcc-7.2.0
aarch64-unknown-linux-gnu-gcc-ar
aarch64-unknown-linux-gnu-gcc-nm
aarch64-unknown-linux-gnu-gcc-ranlib
aarch64-unknown-linux-gnu-gcov
aarch64-unknown-linux-gnu-gcov-dump
aarch64-unknown-linux-gnu-gcov-tool
aarch64-unknown-linux-gnu-gprof
aarch64-unknown-linux-gnu-ld
aarch64-unknown-linux-gnu-ld.bfd
aarch64-unknown-linux-gnu-nm
aarch64-unknown-linux-gnu-objcopy
aarch64-unknown-linux-gnu-objdump
aarch64-unknown-linux-gnu-ranlib
aarch64-unknown-linux-gnu-readelf
aarch64-unknown-linux-gnu-size
aarch64-unknown-linux-gnu-strings
aarch64-unknown-linux-gnu-strip

Kernel Build

With the toolchain ready, cross-compiling the kernel involves two extra steps:
  1. Find the architecture name ($KERNEL_ARCH) in kernel source tree
    • they are typically Located in arch/* in the kernel source tree.
  2. Find the configuration ($KERNEL_CONF) for each sub-arch (e.g., 32-bit vs 64-bit, big endian vs little endian, etc)
    • make ARCH=$KERNEL_ARCH help will show some hints.
    • if there is no specific machine config, the defconfig should work.
    • if there are available machine configs, choosing from an existing config seems to be more hassle free compared with doing a manual menuconfig.
Once we find the values for $KERNEL_ARCH and $KERNEL_CONF, cross-compiling the kernel is as easy as the following:
cd $KERNEL_SOURCE
make ARCH=$KERNEL_ARCH O=$KERNEL_BUILD $KERNEL_CONF

cd $KERNEL_BUILD
make ARCH=$KERNEL_ARCH CROSS_COMPILE=$TARGET- V=1 vmlinux modules
For example, in the case of building the aarch64 kernel, the command should look like the following:
cd $KERNEL_SOURCE
make ARCH=arm64 O=$KERNEL_BUILD defconfig

cd $KERNEL_BUILD
make ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- V=1 vmlinux modules
Of course we can always speed up the build process with the make -j flags.
The instructions apply to Linux kernel version 4.13.5, which is the latest release at the time of writing.

Results

In total, I have currently cross-compiled the kernel for 7 architectures and 12 sub-archs, with the procedure described above. The result is summarized in the following table:
Architecture Name $TARGET $KERNEL_ARCH $KERNEL_CONF
x86 (32-bit) i386 i386-pc-linux-gnu x86 i386_defconfig
x86 (64-bit) x86_64 x86_64-pc-linux-gnu x86 x86_64_defconfig
arm (32-bit) arm armv7-unknown-linux-gnueabi arm multi_v7_defconfig
arm (64-bit) aarch64 aarch64-unknown-linux-gnu arm64 defconfig
powerpc (32-bit) ppc powerpcle-unknown-linux-gnu powerpc pmac32_defconfig
powerpc (64-bit) ppc64 powerpc64le-unknown-linux-gnu powerpc ppc64le_defconfig
sparc (32-bit) sparc sparc-unknown-linux-gnu sparc sparc32_defconfig
sparc (64-bit) sparc64 sparc64-unknown-linux-gnu sparc sparc64_defconfig
mips (32-bit) mips mips-unknown-linux-gnu mips 32r6_defconfig
mips (64-bit) mips64 mips64-unknown-linux-gnu mips 64r6_defconfig
s390x (64-bit) s390x s390x-ibm-linux-gnu s390 default_defconfig
ia64 (64-bit) ia64 ia64-unknown-linux-gnu ia64 generic_defconfig
I have also tried building an allyesconfig kernel and succeeded in 7 architectures, as shown in the following table. Unfortunately, ia64 failed miserably with error message Error: Operand 2 of 'adds' should be a 14-bit integer (-8192-8191), and the error seems to have been there for more than 7 months.
Architecture Name $TARGET $KERNEL_ARCH $KERNEL_CONF
x86 (64-bit) x86_64 x86_64-pc-linux-gnu x86 allyesconfig
arm (32-bit) arm armv7-unknown-linux-gnueabi arm allyesconfig
arm (64-bit) aarch64 aarch64-unknown-linux-gnu arm64 allyesconfig
powerpc (64-bit) ppc64 powerpc64-unknown-linux-gnu powerpc allyesconfig
sparc (64-bit) sparc64 sparc64-unknown-linux-gnu sparc allyesconfig
mips (64-bit) mips64 mips64-unknown-linux-gnu mips allyesconfig
s390x (64-bit) s390x s390x-ibm-linux-gnu s390 allyesconfig
In case you want to try out, you can directly feed the corresponding values for $TARGET, $KERNEL_ARCH, and $KERNEL_CONF into the scripts above and test.

Wednesday, 5 September 2018

The Top Twelve Open Source Artificial Intelligence Tools

Artificial intelligence now pervades many parts of our lives and is here to stay for the foreseeable future. This bird’s eye view of the top open source AI tools serves as a reminder of how AI has already impacted our lives.
 
Imagine the digital future, in which you are dining at a restaurant. In this future scenario, right from the food served on your plate to the waiter who attends to you, everything has been arranged according to your taste and personality traits, without you being asked a single question. Imagine a world in which even the angle at which a plate is served at this restaurant or the smile of the waiter can be calculated with potential suggestions to suit your personality or mood. In such a world, if you are trying to buy a pair of shoes, you’d make a few swipes on your smartphone and all the available shoes in your favourite colour, with the accurate shape of the sole and height of heel, will show up on your screen. All this thanks to machine learning (ML) — and the possibilities are endless.

Artificial intelligence or AI has now penetrated many aspects of our lives. Let’s talk about 2018! All these voice recognition systems we use to search on Google and command our smartphones are the beginning of a new age. We have not yet understood the full potential of AI yet. A limited version of AI technology is already fitted inside our cars. Apple’s Siri, Google’s OK Google, and Amazon’s Echo services are already in use, analysing your tone and voice along with answering your query. These AI-backed services have a natural language processing system and are able to respond to the commands. You can now get directions to a location and look for a specific restaurant, all by just swiping your fingers on a screen!

Driverless cars are now on trial and the population of robots is on the rise, as they are expected to replace humans in certain fields in the very near future. You could expect robot hotel receptionists and robots to clean our houses, reducing us to couch potatoes. Oh well, let’s stick to the brighter side of things!

ClickHere to read more!