Github source code downloader script

Moderator: Forum moderators

Post Reply
User avatar
sc0ttman
Posts: 93
Joined: Sat Aug 22, 2020 3:55 pm
Has thanked: 4 times
Been thanked: 33 times

Github source code downloader script

Post by sc0ttman »

gh-get

Create a directory from a github repo, with the repo name and
release version (or commit hash) in the dir name.

Usage:

gh-get https://github/com/user/repo
gh-get user/repo

should create a dir called ./$repo-$version or ./$repo-$commit_hash

What's it for?

This script makes it much easier to package up stuff from GitHub as a PET file.

It does this by making one of the initial steps easier - specifically, by automating
the creation of a folder with version number, which is needed if you wanna
use new2dir, and dir2pet.

Background info

When compiling packages, you can use the command new2dir make install,
and the new2dir script will install your newly compiled program into its own
folder, ready to be packaged up as a PET.

One problem - new2dir requires a version number in the folder name of the
program being compiled, and if you download a .tar.gz from GitHub, or clone
a GitHub repo, then you don't get the version numbers in the folder name
.

Even worse, many github projects don't store a version number at all - you
need to get the latest commit hash, of the latest branch, and use that.

The script

Save as /usr/bin/gh-get

Code: Select all

#!/bin/bash
#
# Create a directory from a github repo, with the repo name and
# release version (or commit hash) in the dir name.
#
# Usage:
#
#   gh-get https://github/com/user/repo
#   gh-get user/repo
#
# should create a dir called  ./$repo-$version  or  ./$repo-$commit_hash
#

if [ ! "$1" ] || [ "$1" = "-h" ] || [ "${1:0:1}" = "-" ];then
  echo
  echo "Create a directory from a github repo, with the repo name and"
  echo "release version (or commit hash) in the dir name."
  echo
  echo "Usage:"
  echo
  echo "  gh-get https://github/com/user/repo"
  echo "  gh-get user/repo"
  echo
  echo "Either of the above commands will create a dir called "
  echo "./\$repo-\$version, or, if no release has been made, then "
  echo "./\$repo-\$commit_hash will be used instead."
  echo
  exit 1
fi

user=${1//*github.com/}
user=${user%/*}
user=${user//\//}
repo=${1##*/}
repo=${repo//.git/}
branch=
url=
has_release=false
json=$(curl -s https://api.github.com/repos/$user/$repo/releases/latest)

#echo "$json"
#exit

errmsg() {
  >&2 echo "ERROR: $1"
  sleep 1
  exit 1
}

version=$(echo "$json" | grep -oP '"tag_name": "\K(.*)(?=")' | sed 's/^v//')
[ "$version" != "" ] && has_release=true

if $has_release ;then

  url=$(echo "$json" | grep -m1 tarball | cut -f4 -d'"')
  out_file=$repo-$version.tar.gz
  curl -q -Ss -L "$url" -o $out_file || errmsg "Download failed"
  tar -xf $out_file &>/dev/null || errmsg "Unpacking failed"
  unpacked_dir=$(find -maxdepth 1 -type d | grep "\./$user\-$repo\-" )
  commit_hash=$(echo $unpacked_dir | cut -f3 -d'-')
  mv $unpacked_dir/  ${out_file//.tar.gz/} || errmsg "Creating dir with version in name, failed"

else

  for b in master main develop $2 stable lts
  do
    branch=$b
    commit_hash_long=$(git ls-remote https://github.com/${user}/${repo}.git $branch 2>/dev/null | cut -f1)
    [ -z "$commit_hash_long" ] && continue
    commit_hash=${commit_hash_long:0:7}
    url="https://github.com/$user/$repo/archive/$branch.tar.gz"
    out_file=$repo-${version:-$commit_hash}.tar.gz
    curl -q -Ss -L "$url" -o $out_file || errmsg "Download failed"
    tar -xf $out_file &>/dev/null || errmsg "Unpacking failed"
    mv $repo-$branch/  ${out_file//.tar.gz/} || errmsg "Creating dir with version name, failed"
  done

fi

# we should now have the dir called  ./$repo-$version  or  ./$repo-$commit_hash
echo "Created:  ${out_file//.tar.gz/}/"

Notes

I intend for this script to end up in Petbuild or Pkg, so we can automate the packaging of various kinds of github repos into PET packages.

Last edited by sc0ttman on Sat Nov 21, 2020 11:04 am, edited 2 times in total.
User avatar
rockedge
Site Admin
Posts: 5713
Joined: Mon Dec 02, 2019 1:38 am
Location: Connecticut,U.S.A.
Has thanked: 1992 times
Been thanked: 2097 times
Contact:

Re: Github source code downloader script

Post by rockedge »

Excellent! I really am going to like the function of creating the folder with version number automatically.

Update:
Using gh-get with the url having .git on the end if using the git clone url:

Code: Select all

root# gh-get https://github.com/AlexeyAB/darknet.git
Username for 'https://github.com': 
Password for 'https://github.com':

attempting to use the correct username and password have no effect and repeats in a loop until terminated.

without the .git

Code: Select all

root# gh-get https://github.com/AlexeyAB/darknet
Created:  darknet-darknet_yolo_v3/
User avatar
sc0ttman
Posts: 93
Joined: Sat Aug 22, 2020 3:55 pm
Has thanked: 4 times
Been thanked: 33 times

Re: Github source code downloader script

Post by sc0ttman »

Thanks.. good spot.

Should be fixed now. Updated main post.

Post Reply

Return to “Package Collections/Repositories”