How to ssh pipe copy files from remote to local compressed

ssh  john@192.168.1.49  "cd ~/Documents && tar -cf - assignments/ | gzip -9"  >  assignments.tar.gz

# syntax explanation

ssh    username@remoteIP     "command string"

UTF8 content in Git bare clone encoding 改編碼看UTF8文件

Recently found a git repo of 13,3xx ancient chinese books in utf8 cn. // 萬多本中文古書

(可用chrome 看，可轉為繁體字；或者下載做電字書 用 calibre  做epub 放ebook reader 看  :D )

The repo just too big, multiple giga bytes.

Either download the zip file of the repo (if not care about future updates), or, can do a *minimum* clone:

$ git  clone  --bare  --depth 1  git://....

# list all files in a bare repo

$ git ls-tree --full-tree -r --name-only HEAD

# -r    recursive

By default, git will print non-ASCII file names in quoted octal notation, i.e. "\nnn\nnn...".

 # show utf8 characters    (ref:  https://stackoverflow.com/questions/22827239/how-to-make-git-properly-display-utf-8-encoded-pathnames-in-the-console-window)

$ git config core.quotepath off

list file again, now show utf8 okay

# to view one file content, say, on master branch

$ git show master:/path/to/file

optionally convert to zh_TW 繁體

$ git show master:/path/to/file |  cconv -f utf8-cn -t utf8-tw

Manjaro linux: syntax/source highlighing/coloring in vim/bash terminal

if you are a text tool user like me, syntax coloring or source code highlighting can be helpful.

# 1. use GUI add/remove software to install 2 packages:

    syntax-highlighting

    source-highlight

# 2. edit  ~/.bashrc  and add below  

# (if other linux distro, use other path of src-hilite-lesspipe.sh)

export LESSOPEN="| /usr/bin/src-hilite-lesspipe.sh %s"

export LESS=" -R "

alias less='less -m -N -g -i -J --underline-special --SILENT'

# 3. edit  ~/.vimrc  and add

if &t_Co > 2 || has("gui_running")

    syntax on

    set hlsearch

endif

* * * 

# to test

open a terminal

$ ls 

$ less test.cpp

$ less test.py

$ vim test.py

中文檔在bash出亂碼，可能unicode須轉成utf-8

 mac osx 與 linux bash 普遍支援 UTF-8 編碼，為 1~4 個bytes不定長度(variable-length) characters。例如ascii 是 1byte，中文字兩bytes。

unicode 係 定長編碼 (fixed-length)，unicode 文字檔一般在檔頭加上 Byte order mark (BOM, 字節順序標記）。這種檔在GUI 文件編輯器開無問題，但在 bash 會出亂碼，不能用cat, sed 等命令。

例如有字幕ssa檔

00000000: fffe 5b00 5300 6300 7200 6900 7000 7400  ..[.S.c.r.i.p.t.

00000010: 2000 4900 6e00 6600 6f00 5d00 0d00 0a00   .I.n.f.o.].....

檔頭32bytes 為BOM，標記含義可參考 https://zh.wikipedia.org/wiki/位元組順序記號。

解決辦法 -- 強大轉碼(convert text encoding)工具：

$ iconv -f unicode -t utf-8 in.txt  > out.txt

* * * 

順帶一提，如果要刪除每行末 ^M ，可用 sed 

s/^M//g

注意^M 是一個 character，輸入方法是 ctrl-v ctrl-m。

Search document content in Mac & Linux

 If there's one feature I like most about Mac, it must be the spotlight search 'summoned' via the keyboard shortcut:  command-space.

It searches not only by filenames, but also the content text; not only in text files, but also in RTF, DOC, etc.  It makes folder boundaries disappear, and file names unimportant.  The files indexing is automatically done in background.

In Windows, about 10 years ago,  there was a Google desktop search sidebar tool, but discontinued.

In Linux, we have multiple built-in tools, eg.   updatedb, then locate pattern for file names;  or ascii content by  grep -HERin pattern path;  or  by more search criteria,   'find' command

Ubuntu has its all-in-one search works somewhat similarly.

For other distro's like mxlinux, an option maybe a tool: search monkey.   To install:

    sudo apt update

    sudo apt upgrade

    sudo apt install searchmonkey

For usage, 3 different ways:

(A) bottom-left menu, type search.  prefilled search location   $HOME

(B) terminal command line:

    $ searchmonkey

or 

    $ searchmonkey -d directory -f filename_regex -t text_regex

eg.

    searchmonkey -d /path/to/directory -f txt -t howto

(C) the handy way, configure in thunar preference  i.e. the explorer

file preference > customized action 

select   'find files here'

origin default is catfish

replace it with

        searchmonkey -d %f

then setup keyboard shortcut for this action.

Note that it's real time search, not from pre-processed database.  Therefore, searching in a big folder takes long time.

How to Split a Subdirectory to a New Git Repository and Keep the History

 

(modified from: https://ao.ms/how-to-split-a-subdirectory-to-a-new-git-repository-and-keep-the-history/)

Say, original local path is ~/allMyCode

1) origin repo to check out the branch with all the subdirectory history, eg. master

    git checkout master

2) clone a copy

    git clone  ~/allMyCode  ~/newSubset

    cd ~/newSubset

3) in new clone, remove all but the sub directory (below is a one-liner)

    git filter-branch --prune-empty --subdirectory-filter relative/path/to/subdirectory current_branch_name

4) if the new clone is a branch other than 'master', make it the new 'master'

    git branch -m master

5) remove untracked leftovers, such as ignored/hidden files/folders (WARNING: make sure you have backup)

    git clean -fdx

6) fix remote repo URL    // assume new repo empty

    git remote set-url origin  {newRepoURL}

7) push to new repo

    git push -u origin master

Git clone to replace a dead repo

A network machine hosting the git repo was dead.

The checkout still exists in local.

Setup a new machine, planning to use it as the new host for repo.

ssh  user@remote

cd /path/to/allprojects/root

git init --bare new_proj.git

exit

# at local

git remote set-url origin  user@remote:/path/to/allprojects/root/new_proj.git

git push --all origin

Mac/Linux/Ubuntu to convert PDF to JPG

 

ref: https://stackoverflow.com/questions/43085889/how-to-convert-a-pdf-into-jpg-with-command-line-in-linux

        

# using  Image magic

    $ convert -density 300 -quality 100 input.pdf prefix.jpg     

# will give files  prefix-0.jpg, prefix-1.jpg, prefix-2.jpg ...

HOWTO convert webp to png, and from webm to mkv with ffmpeg

 # image conversion

convert input.webp output.png

# video conversion

ffmpeg -i input.webm  -c:av copy  output.mkv

bash variable string operations

$ dummy=ABCDEFGHIJ

$ echo ${#dummy}        # string length '#'
10

$ echo ${dummy:0:2}     # slice w/ offset and [length]
AB
$ echo ${dummy:5}       # default length: all
FGHIJ

$ echo ${dummy:-3}     # no space before '-', not working
ABCDEFGHIJ
$ echo ${dummy: -3}    # MUST have space, it matters
HIJ
$ echo ${dummy: -7:4}
DEFG


# pipe to other commands

$ echo $dummy | tail -c 3    # character, last 3 including '\n'
IJ

$ echo $dummy | cut -c2-5    # 'cut' to specify character positions
BCDE
$ echo $dummy | cut -c2,4-5
BDE

$ echo $dummy | grep -o ...$   # last 3 any characters
HIJ

$ echo $dummy | egrep -o '.{1,3}$'
HIJ

bash read file line-by-line

while read line || [ -n "$line" ]; do 
  line=$(echo $line | tr -d '\r')
  echo $line
done < inputfile.txt

# explain

# tr -- translate
cat file | tr -d '\r'    # delete new line

echo $companyname | tr ' ' '_'

bash script counter increment

# TL;NR
$((counter + 1))


# Example - count lines of a file, similar to bash command "wc -l file.txt" :

num=0

while read f
do
  echo "$f"
  num=$((num + 1))
done < imput.txt

echo $num

linux (eg. ubuntu) format USB and check health status

 usb drive doesn't have health data. 

First, find the drive device using df  or  lsblk

Eg. unmount the device sdc1

$ umount /dev/sdc1

Format usb drive to FAT32 on linux

    $ sudo mkfs.vfat -F 32 /dev/sdc1

Check disk for bad blocks by writing different byte patterns

WARNING!  Write mode is destructive. 

    $ sudo badblocks -w -s -o  myusb.log /dev/sdc

        # write-mode,  show progress, output to file 

# non-destructive 

    $ sudo badblocks -nvs > usberr.log /dev/sdc

# or through   fsck  or  mkfs

    # option:   -fcky    or      -fccky    # one 'c' read-only,  'cc'  read-write non-destructive

[OSX] command line 'join' failed with DOS line end 0d0a

Ever frustrated how join command did not work on some file?

To get rid of the 0x0d from DOS .CSV file, i.e. make unix 0a ending, do

# note that ^M below is entered by CTRL-V CTRL-M 

$ sed 's/^M//' in.csv > out.csv

Highlight syntax in 'less' command on source code. Colorful!!

# linux
sudo apt-get install source-highlight
# osx
brew install source-highlight
# osx only, for compatibility i have,

sudo ln -s /usr/local/Cellar/source-highlight/3.1.8_2/bin/src-hilite-lesspipe.sh /usr/local/bin/

# if the file 'src-hilite-lesspipe.sh' not included, write it without line number:

       1 #! /bin/sh

       2 

       3 for source in "$@"; do

       4     case $source in

       5         *ChangeLog|*changelog) 

       6         source-highlight --failsafe -f esc --lang-def=changelog.lang --style-file=esc.style -i "$source" ;;

       7         *Makefile|*makefile) 

       8         source-highlight --failsafe -f esc --lang-def=makefile.lang --style-file=esc.style -i "$source" ;;

       9         *.tar|*.tgz|*.gz|*.bz2|*.xz)

      10         lesspipe "$source" ;;

      11         *) source-highlight --failsafe --infer-lang -f esc --style-file=esc.style -i "$source" ;;

      12     esac

      13 done

# add the following into .bashrc

### LESS ###

# Enable syntax-highlighting in less.

# brew install source-highlight

# First, add these two lines to ~/.bashrc

export LESSOPEN="| /usr/local/bin/src-hilite-lesspipe.sh %s"

export LESS=" -R "

alias less='less -m -N -g -i -J --underline-special --SILENT'

alias more='less'


# example print-outs

# c code:

       1 /*

       2  * this is comment

       3  */

       4 #include 

       5 

       6 int main( int argc, char ** argv )

       7 {

       8     for (int i=0; i<10; i++)

       9     {

      10         printf("%d ", i);

      11     }

      12     for (int i=0; i<10; i++)

      13     {

      14         printf("%d ", i);

      15     }

      16 

      17     return 0;

      18 }

# python code

       1 #!/usr/bin/env python

       2 import sys

       3 def main(argv):

       4     for x in range( len(sys.argv)-1 ):

       5         print sys.argv[x+1]

       6     print '+'.join(sys.argv[1:])

       7 

       8 if __name__ == "__main__":

       9     main(sys.argv[1:])

sweet.  isn't it?