Clone any Private or Public repo from Organization

Say you want to clone or backup of all github repositories under your account or specific organization or you want to contribute to an open source project. You successfully do that copy & paste repo URL and then clone them manually 😫. But there might be plenty of repos and you dont want to repeat yourself. what do you do if there plenty of repos? you will need to repeat yourself again and again 😣.

To automate this process, you can use github API together with jq processor to clone all repos with one-liner shell script.

Prerequisites

• Basic knowledge of SHELL scripting and CURL.
• Familiar with JQ.

GitHub API returns really nicely formatted JSON but there is a pretty fair amount of JSON in the response!. We’re primarily only interested to get the ssh_url, and not all the other output.

Simply we can use jq to “reformat” the JSON output from API response and include only that information. That’s done with this snippet:

1curl -s "https://api.github.com/orgs/twitter/repos?per_page=10" \
2    | jq '.[].ssh_url'

This will provide output that will look something like this:

1"git@github.com:twitter/hadoop-lzo.git"
2"git@github.com:twitter/thrift_client.git"
3"git@github.com:twitter/twurl.git"
4"git@github.com:twitter/elephant-bird.git"
5"git@github.com:twitter/ostrich.git"
6"git@github.com:twitter/scala_school.git"
7"git@github.com:twitter/finagle.git"
8"git@github.com:twitter/joauth.git"
9"git@github.com:twitter/twitter.github.io.git"
10"git@github.com:twitter/util.git"

This makes it really easy to parse out the specific information you need. Now that you’re armed with ssh_url, we need to convert the output to raw strings, not JSON texts by passing -r flag to jq then we can loop over the list and clone all repos:

1curl -s "https://api.github.com/orgs/twitter/repos?per_page=10" \
2    | jq '.[].ssh_url' -r \ # output raw strings, not JSON texts
3    | while read url; do git clone "$url"; done

Boom! You got all repos now 🔥! if you needed to clone private repos, then you will need to:

• Add &type=private on the URL.
• Create a personal access token as described here

snippet will look like this:

1curl -H "Authorization: token YOUR_ACCESS_TOKEN" -s \
2    "https://api.github.com/orgs/twitter/repos?per_page=10" \
3    | jq '.[].ssh_url' -r \ # output raw strings, not JSON texts
4    | while read url; do git clone "$url"; done

Bonus

You create a shell function inside your .bashrc or .zshrc file:

1function cloner {
2  # you can store your access token as env variable inside `.bashrc` or `.zshrc`
3  # ex: export GITHUB_TOKEN= , and replace $1 with $GITHUB_TOKEN and $2 with $1
4  curl -H "Authorization: token $1" -s \
5    "https://api.github.com/orgs/{$2}/repos?per_page=10" \
6    | jq '.[].ssh_url' -r \ # output raw strings, not JSON texts
7    | while read url; do git clone "$url"; done
8}

and use it like this:

1# with ENV variable
2cloner your-organization
3
4# without ENV variable
5cloner <your-access-token> your-organization

Did you find this useful? Buy me a coffee to give my brain a hug☕️.

Hope you liked this article. If you did, then share it. It means a lot.🙌 Thanks for reading!