Dark-Alex-17/kapow
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'develop'

Browse Source
This commit is contained in:
pancho horrillo
2019-06-28 10:33:29 +02:00
parent e4cbb96cf4 b058136eb2
commit 6af8fe7256
22 changed files with 713 additions and 251 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
-57
View File
@@ -1,57 +0,0 @@
![Kapow!](https://trello-attachments.s3.amazonaws.com/5c6edee98297dc18aa4e2b63/960x720/ff8d28fc24af11e3295afa5a9665bdc0/kapow-1601675_960_720.png)
**Kapow!** allows you to leverage the Ultimate Power™ of the UNIX® shell via HTTP.
**Kapow!** is in the process of being defined by a [specification](/spec/)
# Sample usage
## Clone the project
```bash
# clone this project
```
## Build the kapow! docker image
```bash
docker build -t bbva/kapow:0.1 /path/to/kapow/poc
```
## Build a docker image for running the nmap example
```bash
docker build -t kapow-nmap /path/to/kapow/poc/examples/nmap
```
## Run kapow
```bash
docker run \
-it \
-p 8080:8080 \
kapow-nmap
```
which will output something like this:
```
======== Running on http://0.0.0.0:8080 ========
(Press CTRL+C to quit)
Route created POST /list/{ip}
ROUTE_8ed01c48_bf23_455a_8186_a1df7ab09e48
bash-4.4#
```
## Test /list endpoint
In another terminal, try running:
```bash
curl http://localhost:8080/list/github.com
```
which will respond something like:
```
Starting Nmap 7.70 ( https://nmap.org ) at 2019-05-10 14:01 UTC
Nmap scan report for github.com (140.82.118.3)
rDNS record for 140.82.118.3: lb-140-82-118-3-ams.github.com
Nmap done: 1 IP address (0 hosts up) scanned in 0.04 seconds
```
et voilà !
# License
This project is distributed under the [Apache License 2.0](/LICENSE).
README.rst
+82
View File
@@ -0,0 +1,82 @@
.. image:: https://trello-attachments.s3.amazonaws.com/5c824318411d973812cbef67/5ca1af818bc9b53e31696de3/f51eb40412bf09c8c800511d7bbe5634/kapow-1601675_480.png
:alt: Kapow!
**Kapow!** If you can script it, you can HTTP it.
+-----------------+------------------------------------------------+
| Project site | https://github.com/BBVA/kapow |
+-----------------+------------------------------------------------+
| Issues | https://github.com/BBVA/kapow/issues |
+-----------------+------------------------------------------------+
| Specification | https://github.com/BBVA/kapow/tree/master/spec |
+-----------------+------------------------------------------------+
| Documentation | https://github.com/BBVA/kapow/tree/master/doc |
+-----------------+------------------------------------------------+
| Python versions | 3.7 or above |
+-----------------+------------------------------------------------+
CAVEAT EMPTOR
=============
**Warning!!! Kapow!** is under **heavy development** and [specification](/spec/);
the provided code is a Proof of Concept and the final version will not even
share programming language. Ye be warned.
How was born
------------
Some awesome history is coming.
Kapow! for the impatient
========================
When you need to **share** a ``command`` but **not** a complete remote ``ssh
access``, Kapow! will help you by the power of HTTP:
.. image:: https://trello-attachments.s3.amazonaws.com/5c824318411d973812cbef67/5ca1af818bc9b53e31696de3/784a183fba3f24872dd97ee28e765922/Kapow!.png
:alt: Where Kapow! lives
Kapow! allows you to write a litte script that will **serve an executable as REST
service**. This script will let you define how to connect HTTP and the Shell
using Kapow!'s shell abstractions to the HTTP world. See it to believe:
.. image:: https://github.com/BBVA/kapow/blob/develop/resources/kapow.gif?raw=true
:alt: Kapow! in action
Superpowers
-----------
Kapow! gives you:
* A very simple way to turn any shell **executable into an API**
* A **remote administration** API
* A way to define the integration in you own terms, obligations-free!
Curses
------
Kapow! can't help when:
* You need high throughput: Kapow! spawns a new executable for every HTTP call
* You must perform complex logic to attend the request: never use Kapow! if
your executables don't perform al least 90% of the hard work
* You are building a huge application
When is your best ally:
-----------------------
* Easy command + Hard API = Kapow! to the rescue
* SSH for one command? Kapow! allows you to share only that command
* Remote instrumentation of several machines? make it easy with Kapow!
The more you know
=================
If you want to know more, please follow our `documentation </doc>`_.
doc/readme.rst
+189
View File
@@ -0,0 +1,189 @@
What is Kapow!
==============
Kapow! is an adapter between the world of Pure UNIX® Shell and an HTTP service.
Some tasks are more convenient in the shell, like cloud interactions, or some
administrative tools. On the other hand, some tasks are more convenient as a
service, like DevSecOps tooling.
Kapow! lies between these two worlds, making your life easier. Maybe you wonder
about how this kind of magic can happen; if you want to know the nitty-gritty
details, just read our [spec](/spec/). Or, if you want to know how Kapow! can
help you first, let's start with a common situation.
Think about that awesome command that you use every day, something very
familiar, like ``cloudx storage ls /backups``. Then someone asks you for an
specific backup, so you ``ssh`` into the host, execute your command, possibly
``grepping`` through its output, copy the result and send it back to him.
And that's fine... for the 100 first times.
Then you decide, let's use an API for this and generate an awesome web server
with it. So, you create a project, manage its dependencies, code the server,
parse the request, learn how to use the API, call the API and deploy it
somewhere. And that's fine... until you find yourself again in the same
situation with another awesome command.
The awesomeness of UNIX® commands is infinite, so you'll be in this situation
an infinite number of times! Instead, let's put Kapow! into action.
With Kapow!, when someone asks you for an specific backup (remember your
familiar command?) you just need to create a ``.pow`` file named ``backups.pow``
that contains:
.. code-block:: bash
kapow route add /backups \
-c 'cloudx storage ls /backups | grep $(request /params/query) | response /body'
and execute it in the host with the command:
.. code-block:: bash
kapow server backups.pow
and that's it. Done. Do you like it? yes? Then let's start learning a little
more.
The mandatory Hello World (for WWW fans)
----------------------------------------
First you must create a pow file named ``hello.pow`` with the following contents:
.. code-block:: bash
kapow route add /greet -c "echo 'hello world' | response /body"
then, you must execute:
.. code-block:: bash
kapow server hello.pow
and you can check that it works as intended with good ole' ``curl``:
.. code-block:: bash
curl localhost:8080/greet
The mandatory Echo (for UNIX fans)
----------------------------------
First you must create a pow file named ``echo.pow`` with the following contents:
.. code-block:: bash
kapow route add -X POST /echo -c 'request /body | response /body'
then, you must execute:
.. code-block:: bash
kapow server echo.pow
and you can check that it works as intended with good ole ``curl``:
.. code-block:: bash
curl -X POST -d '1,2,3... testing' localhost:8080/echo
The multiline fun
-----------------
Unless you're a hardcore Perl hacker, you'll probably need to write your stuff
over more than one line.
Don't worry, we need to write several lines, too. Bash, in its magnificent
UNIX® style, provides us with the
`here-documents`_ mechanism that we can leverage precisely for this purpose.
.. _here-documents: https://www.gnu.org/software/bash/manual/bash.html#Here-Documents
Let's write a ``multiline.pow`` file with the following content:
.. code-block:: bash
kapow route add /log_and_love - <<- 'EOF'
echo "[$(date)] and stuff" >> stuff.log
echo love | response /body
EOF
and then we serve it with ``kapow``:
.. code-block:: bash
kapow server multiline.pow
Yup. As simple as that.
Sample Docker usage
===================
Clone the project
-----------------
.. code-block:: bash
# clone this project
Build the kapow! docker image
-----------------------------
.. code-block:: bash
docker build -t bbva/kapow:0.1 /path/to/kapow/poc
Build a docker image for running the nmap example
.. code-block:: bash
docker build -t kapow-nmap /path/to/kapow/poc/examples/nmap
Run kapow
---------
.. code-block:: bash
docker run \
-it \
-p 8080:8080 \
kapow-nmap
which will output something like this:
.. code-block:: bash
======== Running on http://0.0.0.0:8080 ========
(Press CTRL+C to quit)
Route created POST /list/{ip}
ROUTE_8ed01c48_bf23_455a_8186_a1df7ab09e48
bash-4.4#
Test /list endpoint
-------------------
In another terminal, try running:
.. code-block:: bash
curl http://localhost:8080/list/github.com
which will respond something like:
.. code-block:: bash
Starting Nmap 7.70 ( https://nmap.org ) at 2019-05-10 14:01 UTC
Nmap scan report for github.com (140.82.118.3)
rDNS record for 140.82.118.3: lb-140-82-118-3-ams.github.com
Nmap done: 1 IP address (0 hosts up) scanned in 0.04 seconds
et voilà !
License
=======
This project is distributed under the [Apache License 2.0](/LICENSE).
poc/Dockerfile
+16 -4
View File
@@ -1,8 +1,20 @@
FROM python:3.7-alpine
RUN apk update && apk add bash curl coreutils file
RUN pip install pipenv
COPY Pipfile Pipfile.lock /tmp/
RUN cd /tmp && pipenv install --system --deploy
COPY kapow /usr/bin
COPY bin/* /usr/bin/
WORKDIR /tmp
RUN apk upgrade --update-cache; \
apk add \
bash \
curl \
coreutils \
file; \
\
pip install pipenv; \
\
pipenv install --system --deploy;
ENTRYPOINT ["/usr/bin/kapow"]
poc/kapow → poc/bin/kapow
+95 -26
View File
@@ -26,6 +26,9 @@ import shlex
import sys
from aiohttp import web, StreamReader
import click
import requests
log = logging.getLogger('kapow')
@@ -69,17 +72,19 @@ class Connection:
return self.request.content
elif res.path == 'request/path':
return self.request.path.encode('utf-8')
elif res.path.startswith('request/match/'):
elif res.path == 'request/host':
return self.request.host.encode('utf-8')
elif res.path.startswith('request/matches/'):
return self.request.match_info[nth(2)].encode('utf-8')
elif res.path.startswith('request/param/'):
elif res.path.startswith('request/params/'):
return self.request.rel_url.query[nth(2)].encode('utf-8')
elif res.path.startswith('request/header/'):
elif res.path.startswith('request/headers/'):
return self.request.headers[nth(2)].encode('utf-8')
elif res.path.startswith('request/cookie/'):
elif res.path.startswith('request/cookies/'):
return self.request.cookies[nth(2)].encode('utf-8')
elif res.path.startswith('request/form/'):
return (await self.request.post())[nth(2)].encode('utf-8')
elif res.path.startswith('request/file/'):
elif res.path.startswith('request/files/'):
name = nth(2)
content = nth(3) # filename / content
field = (await self.request.post())[name]
@@ -109,10 +114,10 @@ class Connection:
self._status = int((await content.read()).decode('utf-8'))
elif res.path == 'response/body':
self._body.write(await content.read())
elif res.path.startswith('response/header/'):
elif res.path.startswith('response/headers/'):
clean = (await content.read()).rstrip(b'\n').decode('utf-8')
self._headers[nth(2)] = clean
elif res.path.startswith('response/cookie/'):
elif res.path.startswith('response/cookies/'):
clean = (await content.read()).rstrip(b'\n').decode('utf-8')
self._cookies[nth(2)] = clean
elif res.path == 'response/stream':
@@ -227,8 +232,8 @@ def handle_route(entrypoint, command):
shell_task = await asyncio.create_subprocess_shell(
args,
env={**os.environ,
"KAPOW_URL": "http://localhost:8080/kapow",
"KAPOW_CONNECTION": id
"KAPOW_URL": "http://localhost:8080",
"KAPOW_HANDLER_ID": id
},
stdin=asyncio.subprocess.DEVNULL)
@@ -253,7 +258,7 @@ async def get_routes(request):
return web.json_response(list(request.app.router))
async def create_route(request):
async def append_route(request):
"""Create a new Kapow! route."""
request.app.router._frozen = False
content = await request.json()
@@ -281,24 +286,23 @@ async def delete_route(request):
########################################################################
async def run_init_script(app):
async def run_init_script(app, scripts):
"""
Run the init script if given, then wait for the shell to finish.
"""
if len(sys.argv) == 1:
if not scripts:
# No script given
cmd = "/bin/bash"
elif len(sys.argv) == 2:
cmd = f"/bin/bash --init-file {sys.argv[1]}"
else:
print(f"Usage: {sys.argv[0]} <init-script>")
os._exit(1)
filenames = " ".join(shlex.quote(f) for f in scripts)
cmd = f"/bin/bash --init-file <(cat {filenames})"
shell_task = await asyncio.create_subprocess_shell(
cmd,
executable="/bin/bash",
env={**os.environ,
"KAPOW_URL": "http://localhost:8080/kapow"
"KAPOW_URL": "http://localhost:8080"
})
await shell_task.wait()
@@ -308,23 +312,88 @@ async def run_init_script(app):
async def start_background_tasks(app):
loop = asyncio.get_running_loop()
app["debug_tasks"] = loop.create_task(run_init_script(app))
app["debug_tasks"] = loop.create_task(run_init_script(app, app["scripts"]))
def kapow():
"""Start aiohttp app."""
def start_kapow_server(scripts):
app = web.Application(client_max_size=1024**3)
app.add_routes([
web.get('/kapow/routes', get_routes),
web.post('/kapow/routes', create_route),
web.delete('/kapow/routes/{id}', delete_route),
web.get('/kapow/connections/{id}/{field:.*}', get_field),
# web.post('/kapow/connections/{id}/{field:.*}', append_field),
web.put('/kapow/connections/{id}/{field:.*}', set_field),
# Control API
web.get('/routes', get_routes),
web.post('/routes', append_route), # TODO: return route index
# web.put('/routes', insert_route), # TODO: return route index
web.delete('/routes/{id}', delete_route),
# Data API
web.get('/handlers/{id}/{field:.*}', get_field),
web.put('/handlers/{id}/{field:.*}', set_field),
])
app["scripts"] = scripts
app.on_startup.append(start_background_tasks)
web.run_app(app)
########################################################################
# Command Line #
########################################################################
@click.group()
@click.pass_context
def kapow(ctx):
"""Start aiohttp app."""
pass
@kapow.command()
@click.argument("scripts", nargs=-1)
def server(scripts):
start_kapow_server(scripts)
@kapow.group()
def route():
pass
@route.command()
@click.option("-c", "--command", nargs=1)
@click.option("-e", "--entrypoint", default="/bin/sh -c")
@click.option("-X", "--method", default="GET")
@click.option("--url", envvar='KAPOW_URL')
@click.argument("url_pattern", nargs=1)
@click.argument("command_file", required=False)
def add(url_pattern, entrypoint, command, method, url, command_file):
if command:
# Command is given inline
source = command
elif command_file is None:
# No command
source = ""
elif command_file == '-':
# Read commands from stdin
source = sys.stdin.read()
else:
# Read commands from a file
with open(command_file, 'r', encoding='utf-8') as handler:
source = handler.read()
response = requests.post(f"{url}/routes",
json={"method": method,
"url_pattern": url_pattern,
"entrypoint": entrypoint,
"command": source})
response.raise_for_status()
print(response.json())
@route.command()
@click.option("--url", envvar='KAPOW_URL')
@click.argument("route-id")
def remove(route_id, url):
response = requests.delete(f"{url}/routes/{route_id}")
response.raise_for_status()
print(response.json())
if __name__ == '__main__':
kapow()
poc/bin/kroute
-69
View File
@@ -1,69 +0,0 @@
#!/usr/bin/env python
#
# Copyright 2019 Banco Bilbao Vizcaya Argentaria, S.A.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import sys
import click
import requests
@click.group()
def kroute():
pass
@kroute.command()
@click.option("-c", "--command", nargs=1)
@click.option("-e", "--entrypoint", default="/bin/sh -c")
@click.option("-X", "--method", default="GET")
@click.option("--url", envvar='KAPOW_URL')
@click.argument("url_pattern", nargs=1)
@click.argument("command_file", required=False)
def add(url_pattern, entrypoint, command, method, url, command_file):
if command:
# Command is given inline
source = command
elif command_file is None:
# No command
source = ""
elif command_file == '-':
# Read commands from stdin
source = sys.stdin.read()
else:
# Read commands from a file
with open(command_file, 'r', encoding='utf-8') as handler:
source = handler.read()
response = requests.post(f"{url}/routes",
json={"method": method,
"url_pattern": url_pattern,
"entrypoint": entrypoint,
"command": source})
response.raise_for_status()
print(response.json())
@kroute.command()
@click.option("--url", envvar='KAPOW_URL')
@click.argument("route-id")
def remove(route_id, url):
response = requests.delete(f"{url}/routes/{route_id}")
response.raise_for_status()
print(response.json())
if __name__ == '__main__':
kroute()
poc/bin/request
+1 -1
View File
@@ -16,4 +16,4 @@
# limitations under the License.
#
curl -sf ${KAPOW_URL}/connections/${KAPOW_CONNECTION}/request$1
curl -sf "${KAPOW_URL}/handlers/${KAPOW_HANDLER_ID}/request$1"
poc/bin/response
+3 -3
View File
@@ -24,17 +24,17 @@ import requests
@click.command()
@click.option("--url", envvar='KAPOW_URL')
@click.option("--connection", envvar='KAPOW_CONNECTION')
@click.option("--handler-id", envvar='KAPOW_HANDLER_ID')
@click.argument("path", nargs=1)
@click.argument("value", required=False)
def response(url, connection, path, value):
def response(url, handler_id, path, value):
if value is None:
data = sys.stdin.buffer
else:
data = value.encode('utf-8')
try:
response = requests.put(f"{url}/connections/{connection}/response{path}",
response = requests.put(f"{url}/handlers/{handler_id}/response{path}",
data=data)
except requests.exceptions.ConnectionError:
return False
poc/bin/static
+1 -1
View File
@@ -15,7 +15,7 @@ else
;;
*)
response /status 200
response /header/Content-Type "$(python -m mimetypes "$BASE/$REAL" | awk '/type:/ {print $2; exit 0}; !/type:/ {print "application/octet-stream"}')"
response /headers/Content-Type "$(python -m mimetypes "$BASE/$REAL" | awk '/type:/ {print $2; exit 0}; !/type:/ {print "application/octet-stream"}')"
response /body < "$BASE/$REAL"
esac
fi
poc/buildme
Executable
+3
View File
@@ -0,0 +1,3 @@
#!/bin/sh
docker build -t bbva/kapow:0.1 .
poc/examples/eval.pow
Regular → Executable
+1 -1
View File
@@ -16,4 +16,4 @@
# limitations under the License.
#
kroute add -X POST '/eval' -c '$($(request /body) | response /stream)'
kapow route add -X POST '/eval' -c '$($(request /body) | response /stream)'
poc/examples/eval.testme
Executable
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env sh
curl -X POST --data-binary @- http://localhost:8080/eval <<EOF
touch /tmp/kapow_was_here
EOF
echo 'Proof of success:'
ls -l /tmp/kapow_was_here
poc/examples/nmap/Dockerfile
+5 -4
View File
@@ -1,6 +1,7 @@
FROM bbva/kapow:0.1
RUN apk update && apk add nmap
RUN apk add nmap
COPY nmap.pow /tmp/
RUN cd /tmp && pipenv install --system --deploy
ENTRYPOINT ["/usr/bin/kapow"]
CMD ["/tmp/nmap.pow"]
CMD ["server", "/tmp/nmap.pow"]
poc/examples/nmap/nmap.pow
Regular → Executable
+1 -1
View File
@@ -16,4 +16,4 @@
# limitations under the License.
#
kroute add -X GET '/list/{ip}' -c 'nmap -sL $(request /match/ip) | response /body'
kapow route add -X GET '/list/{ip}' -c 'nmap -sL $(request /matches/ip) | response /body'
poc/examples/operator.pow
+11 -11
View File
@@ -16,26 +16,26 @@
# limitations under the License.
#
kroute add /list/files -c 'ls -la $(request /param/path) | response /body'
kapow route add /list/files -c 'ls -la $(request /params/path) | response /body'
kroute add /list/processes -c 'ps -aux | response /body'
kapow route add /list/processes -c 'ps -aux | response /body'
kroute add /show/cpuinfo -c 'response /body < /proc/cpuinfo'
kapow route add /show/cpuinfo -c 'response /body < /proc/cpuinfo'
kroute add /show/memory -c 'free -m | response /body'
kapow route add /show/memory -c 'free -m | response /body'
kroute add /show/disk -c 'df -h | response /body'
kapow route add /show/disk -c 'df -h | response /body'
kroute add /show/connections -c 'ss -pluton | response /body'
kapow route add /show/connections -c 'ss -pluton | response /body'
kroute add /show/mounts -c 'mount | response /body'
kapow route add /show/mounts -c 'mount | response /body'
kroute add /tail/dmesg - <<-'EOF'
response /header/Content-Type text/plain
kapow route add /tail/dmesg - <<-'EOF'
response /headers/Content-Type text/plain
dmesg -w | response /stream
EOF
kroute add /tail/journal - <<-'EOF'
response /header/Content-Type text/plain
kapow route add /tail/journal - <<-'EOF'
response /headers/Content-Type text/plain
journalctl -f | response /stream
EOF
poc/examples/pandoc/pandoc.pow
+5 -5
View File
@@ -16,11 +16,11 @@
# limitations under the License.
#
kroute add -X POST --entrypoint '/bin/zsh -c' '/convert/{from}/{to}' - <<-'EOF'
pandoc --from=$(request /match/from) \
--to=$(request /match/to) \
kapow route add -X POST --entrypoint '/bin/zsh -c' '/convert/{from}/{to}' - <<-'EOF'
pandoc --from=$(request /matches/from) \
--to=$(request /matches/to) \
--output=>(response /body) \
=(request /body)
EOF
kroute add -X GET '/formats/input' -c 'pandoc --list-input-formats | response /body'
kroute add -X GET '/formats/output' -c 'pandoc --list-output-formats | grep -v pdf | response /body'
kapow route add -X GET '/formats/input' -c 'pandoc --list-input-formats | response /body'
kapow route add -X GET '/formats/output' -c 'pandoc --list-output-formats | grep -v pdf | response /body'
poc/examples/pandoc/testme
Executable
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env sh
curl -X POST --data-binary @- http://localhost:8080/convert/markdown/man <<EOF
# This is not a pipe
1. hello
1. goodbye
EOF
poc/examples/pdfeditor/pdfeditor.pow
Regular → Executable
+2 -2
View File
@@ -16,5 +16,5 @@
# limitations under the License.
#
kroute add -X POST --entrypoint ./topdf '/editor/pdf'
kroute add / -c 'response /header/Content-Type text/html && response /body < pdfeditor.html'
kapow route add -X POST --entrypoint ./topdf '/editor/pdf'
kapow route add / -c 'response /headers/Content-Type text/html && response /body < pdfeditor.html'
poc/examples/pdfeditor/topdf
+1 -1
View File
@@ -19,7 +19,7 @@
tmpfile=$(mktemp --suffix=.pdf)
pandoc --from=$(request /form/from) --to=pdf --output=${tmpfile} -t latex =(request /form/content)
if [ $? -eq 0 ]; then
response /header/Content-Type application/pdf
response /headers/Content-Type application/pdf
response /body < ${tmpfile}
response /status 200
else
poc/examples/torrent.pow
Regular → Executable
+6 -6
View File
@@ -16,8 +16,8 @@
# limitations under the License.
#
kroute add / - <<-'EOF'
response /header/Content-Type text/html
kapow route add / - <<-'EOF'
response /headers/Content-Type text/html
response /body <<-HTML
<html>
<body>
@@ -27,8 +27,8 @@ kroute add / - <<-'EOF'
HTML
EOF
kroute add /save/magnet -e '/bin/bash -c' - <<-'EOF'
link=$(request /param/link)
kapow route add /save/magnet -e '/bin/bash -c' - <<-'EOF'
link=$(request /params/link)
[ -z $link ] && response /status 400 && exit 0
watch_folder=/tmp
@@ -37,7 +37,7 @@ kroute add /save/magnet -e '/bin/bash -c' - <<-'EOF'
echo "d10:magnet-uri${#link}:${link}e" > "meta-${BASH_REMATCH[1]}.torrent"
response /status 302
response /header/Location /torrent/list
response /headers/Location /torrent/list
EOF
kroute add /torrent/list -c 'response /body "Not Implemented Yet"'
kapow route add /torrent/list -c 'response /body "Not Implemented Yet"'
resources/kapow.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

spec/README.md
+275 -59
View File
@@ -5,45 +5,33 @@
Because we think that:
- UNIX is great and we love it
- The UNIX shell is great
- UNIX® is great and we love it
- The UNIX® shell is great
- HTTP interfaces are convenient and everywhere
- CGI is not a good way to mix them
## How?
So, how we can mix the **web** and the **shell**? Let's see...
So, how we can mix the **web** and the **shell**? Let's see...
The **web** and the **shell** are two different beasts, both packed with
history.
There are some concepts in HTTP and the shell that **resemble each other**.
```
+------------------------+-------------------------+
| HTTP | SHELL |
+--------------+------------------------+-------------------------+
| Input | POST form-encoding | Command line parameters |
| Parameters | GET parameters | Environment variables |
| | Headers | |
| | Serialized body (JSON) | |
+--------------+------------------------+-------------------------+
| Data Streams | Response/Request Body | Stdin/Stdout/Stderr |
| | Websocket | Input/Output files |
| | Uploaded files | |
+--------------+------------------------+-------------------------+
| Control | Status codes | Signals |
| | HTTP Methods | Exit Codes |
+--------------+------------------------+-------------------------+
```
| | HTTP | Shell |
|------------------------|--------------------------------------------------------------------------------|----------------------------------------------------|
| Input<br /> Parameters | POST form-encoding<br >Get parameters<br />Headers<br />Serialized body (JSON) | Command line parameters<br />Environment variables |
| Data Streams | Response/Request Body<br />Websocket<br />Uploaded files | stdin/stdout/stderr<br />Input/Output files |
| Control | Status codes<br />HTTP Methods | Signals<br />Exit Codes |
Any tool designed to give an HTTP interface to an existing shell command
**must map concepts of boths**. For example:
**must map concepts from both domains**. For example:
- "GET parameters" to "Command line parameters"
- "Headers" to "Environment variables"
- "Stdout" to "Response body"
- "stdout" to "Response body"
Kapow! is not opinionated about the different ways you can map both worlds.
Instead, it provides a concise set of tools, with a set of sensible defaults,
@@ -52,8 +40,8 @@ allowing the user to express the desired mapping in an explicit way.
### Why not tool "X"?
All the alternatives we found are **rigid** about how they match between HTTP
and shell concepts.
All the alternatives we found are **rigid** about the way they match between
HTTP and shell concepts.
* [shell2http](https://github.com/msoap/shell2http): HTTP-server to execute
shell commands. Designed for development, prototyping or remote control.
@@ -78,17 +66,17 @@ incapable in others.
* Boilerplate
* Custom code = More bugs
* Security issues (Command injection, etc)
* Security issues (command injection, etc)
* Dependency on developers
* **"A programming language is low level when its programs require attention to
the irrelevant"** *Alan Perlis*
* **There is more Unix-nature in one line of shell script than there is in ten
thousand lines of C** *Master Foo*
* *"A programming language is low level when its programs require attention to
the irrelevant."<br />&mdash;Alan Perlis*
* *"There is more Unix-nature in one line of shell script than there is in ten
thousand lines of C."<br />&mdash;Master Foo*
### Why not CGI?
* CGI is also **rigid** about how it matches between HTTP and UNIX process
* CGI is also **rigid** about how it matches between HTTP and UNIX® process
concepts. Notably CGI *meta-variables* are injected into the script's
environment; this behavior can and has been exploited by nasty attacks such as
[Shellshock](https://en.wikipedia.org/wiki/Shellshock_(software_bug)).
@@ -101,17 +89,29 @@ incapable in others.
## What?
We named it Kapow!. It is pronounceable, short and meaningless... like every
good UNIX command ;-)
good UNIX® command ;-)
TODO: Definition
TODO: Intro to Architecture
### Core Concepts
In this section we are going to define several concepts that will be used frequently throughout the spec.
#### `entrypoint`
The entrypoint definition matches *Docker*'s shell form of it.
Technically it's a string which is to be passed to the `command` (`/bin/bash -c`
by default) as the code to be interpreted or executed when attending requests.
### API
Kapow! server interacts with the outside world only through its HTTP API. Any
program making the correct HTTP request to a Kapow! server, can change its
program making the correct HTTP request to a Kapow! server can change its
behavior.
Kapow! exposes two distinct APIs, a control API and a data API, described
@@ -126,6 +126,7 @@ whole lifetime of the server.
## Design Principles
* We reuse conventions of well-established software projects, such as Docker.
* All requests and responses will leverage JSON as the data encoding method.
* The API calls responses will have two distinct parts:
* The HTTP status code (e.g., `400`, which is a bad request). The target
@@ -135,16 +136,16 @@ whole lifetime of the server.
operating the client. The human can use this information to make a
decision on how to proceed.
* All successful API calls will return a representation of the *final* state
attained by the objects which have been addressed (requested, set or
attained by the objects which have been addressed (either requested, set or
deleted).
For instance, given this request:
```
```http
HTTP/1.1 GET /routes
```
an appropiate reponse may look like this:
```
```http
200 OK
Content-Type: application/json
Content-Length: 189
@@ -169,7 +170,7 @@ Kapow! provides a way to control its internal state through these elements.
### Routes
Routes are the mechanism that allows Kapow! to find the correct program to
respond to an external event (e.g. an incomming HTTP request).
respond to an external event (e.g. an incoming HTTP request).
#### List routes
@@ -180,8 +181,24 @@ Returns JSON data about the current routes.
* **Method**: `GET`
* **Success Responses**:
* **Code**: `200 OK`<br />
**Content**: TODO
* **Sample Call**: TODO
**Content**:<br />
```json
[
{
"method": "GET",
"url_pattern": "/hello",
"entrypoint": null,
"command": "echo Hello World | response /body"
},
{
"method": "POST",
"url_pattern": "/bye",
"entrypoint": null,
"command": "echo Bye World | response /body"
}
]
```
* **Sample Call**: `$ curl $KAPOW_URL/routes`
* **Notes**: Currently all routes are returned; in the future, a filter may be accepted.
@@ -193,7 +210,7 @@ Accepts JSON data that defines a new route to be appended to the current routes.
* **Method**: `POST`
* **Header**: `Content-Type: application/json`
* **Data Params**:<br />
```
```json
{
"method": "GET",
"url_pattern": "/hello",
@@ -202,10 +219,10 @@ Accepts JSON data that defines a new route to be appended to the current routes.
}
```
* **Success Responses**:
* **Code**: `200 OK`<br />
* **Code**: `201 Created`<br />
**Header**: `Content-Type: application/json`<br />
**Content**:<br />
```
```json
{
"method": "GET",
"url_pattern": "/hello",
@@ -217,13 +234,35 @@ Accepts JSON data that defines a new route to be appended to the current routes.
* **Error Responses**:
* **Code**: `400 Malformed JSON`
* **Code**: `400 Invalid Data Type`
* **Code**: `400 Invalid Route Spec`
* **Code**: `400 Missing Mandatory Field`<br />
**Header**: `Content-Type: application/json`<br />
**Content**: `{ "mandatory_fields": ["field1", "field2", "and so on"] }`
* **Sample Call**: TODO
**Content**:<br />
```json
{
"missing_mandatory_fields": [
"url_pattern",
"command"
]
}
```
* **Sample Call**:<br />
```sh
$ curl -X POST --data-binary @- $KAPOW_URL/routes <<EOF
{
"method": "GET",
"url_pattern": "/hello",
"entrypoint": null,
"command": "echo Hello World | response /body",
"index": 0
}
EOF
```
* **Notes**:
* A successful request will yield a response containing all the effective
parameters that were applied.
* Kapow! won't try to validate the submitted command. Any errors will happen
at runtime, and trigger a `500` status code.
#### Insert a route
@@ -235,7 +274,7 @@ Accepts JSON data that defines a new route to be appended to the current routes.
* **Method**: `PUT`
* **Header**: `Content-Type: application/json`
* **Data Params**:<br />
```
```json
{
"method": "GET",
"url_pattern": "/hello",
@@ -247,7 +286,7 @@ Accepts JSON data that defines a new route to be appended to the current routes.
* **Code**: `200 OK`<br />
**Header**: `Content-Type: application/json`<br />
**Content**:<br />
```
```json
{
"method": "GET",
"url_pattern": "/hello",
@@ -259,15 +298,38 @@ Accepts JSON data that defines a new route to be appended to the current routes.
* **Error Responses**:
* **Code**: `400 Malformed JSON`
* **Code**: `400 Invalid Data Type`
* **Code**: `400 Invalid Route Spec`
* **Code**: `400 Missing Mandatory Field`<br />
**Header**: `Content-Type: application/json`<br />
**Content**: `{ "mandatory_fields": ["field1", "field2", "and so on"] }`
**Content**:<br />
```json
{
"missing_mandatory_fields": [
"url_pattern",
"command"
]
}
```
* **Code**: `400 Invalid Index Type`
* **Sample Call**: TODO
* **Code**: `400 Index Already in Use`
* **Code**: `404 Invalid Index`
* **Code**: `404 Invalid Route Spec`
* **Sample Call**:<br />
```sh
$ curl -X PUT --data-binary @- $KAPOW_URL/routes <<EOF`
{
"method": "GET",
"url_pattern": "/hello",
"entrypoint": null,
"command": "echo Hello World | response /body",
"index": 0
}
EOF
```
* **Notes**:
* Route numbering starts at zero.
* When `index` is not provided or is less than 0 the route will be inserted
first, effectively making it index 0.
* When `index` is not provided or is less than `0` the route will be inserted
first, effectively making it index `0`.
* Conversely, when `index` is greater than the number of entries on the route
table, it will be inserted last.
* A successful request will yield a response containing all the effective
@@ -278,14 +340,26 @@ Accepts JSON data that defines a new route to be appended to the current routes.
Removes the route identified by `:id`.
* **URL**: `/routes/:id`
* **URL**: `/routes/:id` # FIXME: Héctor points out that this seems inconsistent, since there are no previous mentions to route_id
* **Method**: `DELETE`
* **Success Responses**:
* **Code**: `200 OK`<br />
**Content**: TODO
**Content**:<br />
```json
{
"method": "GET",
"url_pattern": "/hello",
"entrypoint": null,
"command": "echo Hello World | response /body",
"index": 0
}
```
* **Error Responses**:
* **Code**: `404 Not Found`
* **Sample Call**: TODO
* **Sample Call**:<br />
```sh
$ curl -X DELETE $KAPOW_URL/routes/ROUTE_1f186c92_f906_4506_9788_a1f541b11d0f
```
* **Notes**:
@@ -331,6 +405,7 @@ following resource paths:
│
├─ request All information related to the HTTP request. Read-Only
│ ├──── method Used HTTP Method (GET, POST)
│ ├──── host Host part of the URL
│ ├──── path Complete URL path (URL-unquoted)
│ ├──── matches Previously matched URL path parts
│ │ └──── <name>
@@ -381,7 +456,7 @@ following resource paths:
- Comment: That would provide read-only access to the value of the request header `Content-Type`.
- Read a field from a form.
- Scenario: A request generated by submitting this form:<br />
```
```html
<form method="post">
First name:<br>
<input type="text" name="firstname" value="Jane"><br>
@@ -428,8 +503,11 @@ Returns the value of the requested resource path, or an error if the resource pa
**Code**: `400 Invalid Resource Path`<br />
**Notes**: Check the list of valid resource paths at the top of this section.
* **Code**: `404 Not Found`
* **Sample Call**: TODO
* **Notes**: TODO
* **Sample Call**:<br />
```sh
$ curl /handlers/$KAPOW_HANDLER_ID/request/body
```
* **Notes**: The content may be empty.
#### Overwrite the value of a resource
@@ -447,7 +525,10 @@ Returns the value of the requested resource path, or an error if the resource pa
* **Code**: `404 Handler Not Found`
* **Code**: `404 Name Not Found`<br />
**Notes**: Although the resource path is correct, no such name is present in the request. For instance, `/request/headers/Foo`, when no `Foo` header is not present in the request.
* **Sample Call**:
* **Sample Call**:<br />
```sh
$ curl -X --data-binary '<h1>Hello!</h1>' PUT /handlers/$KAPOW_HANDLER_ID/response/body
```
* **Notes**:
@@ -470,34 +551,169 @@ You can run it by ...
Any compliant implementation of Kapow! must provide these commands:
### `kapow`
### `kapow server`
This implements the server, XXX
This is the master command, that shows the help if invoked without args, and
runs the sub-commands when provided to it.
#### Example
```sh
$ kapow
Usage: kapow [OPTIONS] COMMAND [ARGS]...
Options:
TBD
Commands:
start starts a Kapow! server
route operates on routes
...
```
### `kapow start`
This command runs the Kapow! server, which is the core of Kapow!. If
run without parameters, it will run an unconfigured server. It can accept a path
to a `.pow` file, which is a shell script that contains commands to configure
the Kapow! server.
The `.pow` can leverage the `kapow route` command, which is used to define a route.
The `kapow route` command needs a way to reach the Kapow! server, and for that,
`kapow` provides the `KAPOW_URL` variable in the environment of the
aforementioned shell script.
Every time the kapow server receives a request, it will spawn a process to
handle it, according to the specified entrypoint, `/bin/sh -c` by default, and then
execute the specified command. This command is tasked with processing the
incoming request, and can leverage the `request` and `response` commands to
easily access the `HTTP Request` and `HTTP Response`, respectively.
In order for `request` and `response` to do their job, they require a way to
reach the Kapow! server, as well as a way to identify the current request being
served. Thus, the Kapow! server adds the `KAPOW_URL` and `KAPOW_HANDLER_ID` to the
process' environment.
#### Example
```sh
$ kapow start /path/to/service.pow
```
### `kapow route`
To serve an endpoint, you must first register it.
`kapow route` registers/deregisters a route, which maps an
`HTTP` method and a URL pattern to the code that will handle the request.
When registering, you can specify an *entrypoint*, which defaults to `/bin/sh -c`,
and an argument to it, the *command*.
To deregister a route you must provide a *route_id*.
**Notes**:
* The entrypoint definition matches *Docker*'s shell form of it.
* The index matches the way *netfilter*'s `iptables` handles rule numbering.
#### **Environment**
- `KAPOW_URL`
#### **Help**
```sh
$ kapow route --help
Usage: kapow route [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
add
remove
```
```sh
$ kapow route add --help
Usage: kapow route add [OPTIONS] URL_PATTERN [COMMAND_FILE]
Options:
-c, --command TEXT
-e, --entrypoint TEXT
-X, --method TEXT
--url TEXT
--help Show this message and exit.
```
```sh
$ kapow route remove --help
Usage: kapow route remove [OPTIONS] ROUTE_ID
Options:
--url TEXT
--help Show this message and exit.
```
#### Example
```sh
kapow route add -X GET '/list/{ip}' -c 'nmap -sL $(request /matches/ip) | response /body'
```
### `request`
Exposes the requests' resources.
#### **Environment**
- `KAPOW_URL`
- `KAPOW_HANDLER_ID`
#### Example
```sh
# Access the body of the request
request /body
```
### `response`
Exposes the response's resources.
#### **Environment**
- `KAPOW_URL`
- `KAPOW_HANDLER_ID`
#### Example
```sh
# Write to the body of the response
echo 'Hello, World!' | response /body
```
## An End-to-End Example
```sh
$ cat nmap.kpow
kapow route add -X GET '/list/{ip}' -c 'nmap -sL $(request /matches/ip) | response /body'
```
```sh
$ kapow ./nmap.kapow
```
```sh
$ curl $KAPOW_URL/list/127.0.0.1
Starting Nmap 7.70 ( https://nmap.org ) at 2019-05-30 14:45 CEST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00011s latency).
Not shown: 999 closed ports
PORT STATE SERVICE
22/tcp open ssh
Nmap done: 1 IP address (1 host up) scanned in 0.06 seconds
```
## Test Suite Notes